Educational Materials

Simplified Machine-Learning Workflow #6

Anton Antonov

Author

Anton Antonov

Title

Simplified Machine-Learning Workflow #6

Description

Semantic Analysis (Part 1)

Latent Semantic Analysis (Part 1)

A Wolfram livecoding session

Anton Antonov
December 2019

This LSA course through questions

- What data is going used?

- What is a typical workflow example?

- What are the typical applications of LSA?

- Why use LSA?

- What it the fundamental philosophical or scientific assumption for LSA?

- What is the most important and/or fundamental step of LSA?

- What is the difference between LSA and Latent Semantic Indexing (LSI)?

- What are the alternatives?
- Using Neural Networks instead?

- How is LSA used to derive similarities between two given texts?

- How is LSA used to evaluate the proximity of phrases?

- How the main dimension reduction methods compare?

SVD, NNMF, ICA

What data is going used?

Hamlet

USA State of the Union speeches

Airbnb reviews

What is a typical workflow example?

In[]:=

lsaHamlet=ToLSAMonWLCommand["create from aTextHamlet; create document term matrix;echo document term summary;apply LSI functions IDF, None, Cosine; extract 12 topics using NNMF, max steps 20, and min number of documents per term 10;echo topics table with 6 columns;echo statistical thesaurus for ghost, king, castle",True];

Context value "documentTermMatrix":

Dimensions:{223,4440}

Density:0.0111603

Number ofdocuments per termsummary



1 # documents

1st Qu	1
Median	1
Min	1
3rd Qu	2
Mean	2.48874
Max	69



topics table:

1.000	laertes
0.353	enter
0.243	ophelia
0.122	king
0.062	laer
0.030	young
0.026	gertrude
0.025	shall
0.021	queen
0.018	denmark
0.014	hamlet
0.014	call

1.000	king
0.963	enter
0.288	polonius
0.161	lord
0.023	players
0.022	father
0.016	know
0.015	oph
0.013	indeed
0.012	laertes
0.012	hath
0.011	god

1.000	oph
0.664	mark
0.535	pray
0.518	nay
0.482	ophelia
0.251	lady
0.217	sweet
0.216	queen
0.172	say
0.111	tell
0.109	know
0.109	means

1.000	hor
0.874	ham
0.573	laer
0.494	thou
0.472	let
0.443	thee
0.422	come
0.394	thy
0.361	king
0.347	father
0.339	speak
0.331	lord

1.000	hamlet
0.765	enter
0.223	horatio
0.114	players
0.046	lord
0.022	denmark
0.021	rosencrantz
0.020	ham
0.018	gentlemen
0.014	come
0.010	guildenstern
0.008	father

1.000	guildenstern
0.973	rosencrantz
0.335	enter
0.129	polonius
0.121	king
0.046	queen
0.045	ophelia
0.010	bring
0.009	hath
0.008	gentlemen
0.007	son
0.006	call

1.000	end
0.103	indeed
0.092	oph
0.090	make
0.075	ophelia
0.047	laer
0.033	king
0.028	thoughts
0.023	thy
0.021	love
0.020	makes
0.016	call

1.000	ros
0.354	lord
0.350	ham
0.344	bring
0.284	body
0.234	king
0.152	dead
0.145	sir
0.118	shall
0.100	hath
0.094	thing
0.084	guildenstern

1.000	pol
0.785	ham
0.532	lord
0.237	good
0.206	sir
0.159	mother
0.129	say
0.127	love
0.124	words
0.123	look
0.121	god
0.120	leave

1.000	queen
0.567	enter
0.429	polonius
0.231	gentleman
0.169	horatio
0.114	ophelia
0.068	denmark
0.067	king
0.061	mother
0.049	seen
0.045	scene
0.040	players

1.000	ghost
0.316	enter
0.087	father
0.077	hamlet
0.012	thy
0.008	thou
0.007	heaven
0.007	thee
0.006	poor
0.005	blood
0.005	ham
0.005	revenge

1.000	scene
0.957	elsinore
0.900	castle
0.717	room
0.246	act
0.073	state
0.068	near
0.059	polonius
0.004	queen
0.003	welcome
0.002	players
0.001	dear

statistical thesaurus:

term	statistical thesaurus entries
castle	{castle,elsinore,scene,room,act,state,near,welcome,wind,hands,gentlemen,goes}
ghost	{ghost,father,poor,heaven,blood,revenge,thy,bear,spirit,bed,away,word}
king	{king,polonius,lord,shall,know,hath,father,dead,tis,laer,think,make}

What are the typical applications of LSA?

Text feature engineering. (To be used with ML)

Document clustering.

Document summarization

In[]:=

aTextHamlet//Length

Out[]=

223

In[]:=

lsaHamlet⟹LSAMonTakeW

Out[]=

SparseArray

Specified elements: 1149

Dimensions: {223,12}

Default: 0.

Density: 0.429

Elements:

{2,4}

3.16×

-43

{2,6}

0.0722

{5,1}

1.03×

-13

{5,2}

0.0574

⋮



In[]:=

MatrixPlot[lsaHamlet⟹LSAMonTakeW]

Out[]=

In[]:=

?FeatureExtraction

Out[]=

Symbol
FeatureExtraction[{ example 1 , example 2 ,…}] generates a FeatureExtractorFunction[…] trained from the examples given.FeatureExtraction[examples,extractor] uses the specified feature extractor method.FeatureExtraction[examples,{ extractor 1 , extractor 2 ,…}] applies the extractor i in sequence to generate a feature extractor.FeatureExtraction[examples,specext] uses the extractor methods specified by ext on parts of examples specified by spec. FeatureExtraction[examples,{ spec 1  ext 1 , spec 2  ext 2 ,…}] uses the extractor methods ext i on parts of examples specified by the spec i .FeatureExtraction[examples,extractor,props] gives the feature extraction properties specified by props.

In[]:=

lsaHamlet⟹LSAMonTakeDocumentTermMatrix

Out[]=

SparseArray

Specified elements: 11050

Dimensions: {223,4440}

Default: 0

Density: 0.0112

Elements:

{1,1}

{2,996}

{2,1763}

{2,2983}

⋮



In[]:=

MatrixPlot[lsaHamlet⟹LSAMonTakeDocumentTermMatrix]

Out[]=

In[]:=

MatrixForm[(lsaHamlet⟹LSAMonTakeDocumentTermMatrix)〚30;;42,1000;;1042〛]

Out[]//MatrixForm=

	departed	depends	depriv	deprive	desert	deserve	deserved	design	desire	desires	desirous	desk	desp	desperate	desperation	despis	despite	destroy	detecting	determination	determine	device	devices	devil	devis	devise	devotion	devoutly	dew	dews	dexterity	diadem	diameter	dicers	diction	didest	dido	didst	die	died	[dies	dies	diet
id.0030	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0031	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0032	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0033	0	0	0	1	0	0	0	0	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0034	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0035	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0036	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0
id.0037	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0038	0	0	0	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0039	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0040	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0041	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.0042	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

In[]:=

aTextHamlet["id.0038"]

Out[]=

Mar. Lord Hamlet! Hor. Heaven secure him! Ham. So be it! Mar. Illo, ho, ho, my lord! Ham. Hillo, ho, ho, boy! Come, bird, come. Mar. How is't, my noble lord? Hor. What news, my lord? Mar. O, wonderful! Hor. Good my lord, tell it. Ham. No, you will reveal it. Hor. Not I, my lord, by heaven! Mar. Nor I, my lord. Ham. How say you then? Would heart of man once think it? But you'll be secret? Both. Ay, by heaven, my lord. Ham. There's neer a villain dwelling in all Denmark But he's an arrant knave. Hor. There needs no ghost, my lord, come from the grave To tell us this. Ham. Why, right! You are in the right! And so, without more circumstance at all, I hold it fit that we shake hands and part; You, as your business and desires shall point you, For every man hath business and desire, Such as it is; and for my own poor part, Look you, I'll go pray. Hor. These are but wild and whirling words, my lord. Ham. I am sorry they offend you, heartily; Yes, faith, heartily. Hor. There's no offence, my lord. Ham. Yes, by Saint Patrick, but there is, Horatio, And much offence too. Touching this vision here, It is an honest ghost, that let me tell you. For your desire to know what is between us, O'ermaster't as you may. And now, good friends, As you are friends, scholars, and soldiers, Give me one poor request. Hor. What is't, my lord? We will. Ham. Never make known what you have seen to-night. Both. My lord, we will not. Ham. Nay, but swear't. Hor. In faith, My lord, not I. Mar. Nor I, my lord- in faith. Ham. Upon my sword. Mar. We have sworn, my lord, already. Ham. Indeed, upon my sword, indeed.

Why use LSA?

Fast computations using sparse matrix algebra algorithms.

Easier representation interpretation. (From mathematical and statistical point of view.)

Different dimension reduction algorithms.

What it the fundamental philosophical or scientific assumption for LSA?

The so called distributional hypothesis.

What is the most important and/or fundamental step of LSA?


What is the difference between LSA and Latent Semantic Indexing (LSI)?


What are the alternatives?


How is LSA used to derive similarities between two given texts?

In[]:=

text=StringJoin[txt1,txt2];sentences=StringSplit[text,{".","?","!",";"}];

In[]:=

fed=FeatureExtraction[DeleteStopwords@sentences,"TFIDF","FeatureDistance"]

Out[]=

FeatureDistance

Input type: Text

Output type: NumericalVector

(length: 93)

Missing values: Forbidden

Distance function: EuclideanDistance



In[]:=

fed[txt1,txt2]

Out[]=

0.333781

How is LSA used to evaluate the proximity of phrases?


How the main dimension reduction methods compare?

SVD

In[]:=

pnts=Transpose[Table[RandomReal[NormalDistribution[RandomReal[{0.3,14}],RandomReal[{0.5,6}]],600],{3}]];pnts=pnts.RotationMatrix[π/3,{1,1,1}];

In[]:=

{U,S,V}=SingularValueDecomposition[pnts,3];

In[]:=

MatrixForm[Transpose[V]]

Out[]//MatrixForm=

-0.529181	-0.736016	-0.422194
0.825513	-0.561638	-0.0555941
-0.196202	-0.377946	0.904799

In[]:=

With[{c=15},Graphics3D[{Point[pnts],Red,Line[{{0,0,0},#/c}]&/@(S.Transpose[V])},PlotRange{{-c,c},{-c,c},{-c,c}},AxesTrue]]

Out[]=

In[]:=

qs=(#〚3〛-#〚1〛&)/@Table[Quartiles[pnts〚All,i〛],{i,Dimensions[pnts]〚2〛}];cpnts=Transpose[Table[(pnts〚All,i〛-Median[pnts〚All,i〛])/qs〚i〛,{i,Dimensions[pnts]〚2〛}]];medianPoint=Median/@Transpose[pnts];

In[]:=

cpnts=Transpose[Standardize/@Transpose[pnts]];

In[]:=

{U1,S1,V1}=SingularValueDecomposition[cpnts,3];

In[]:=

MatrixForm[Transpose[V1]]

Out[]//MatrixForm=

-0.61962	0.734069	0.277874
-0.503743	-0.100418	-0.857997
-0.601926	-0.671609	0.432003

In[]:=

With[{c=4},Graphics3D[{Point[cpnts],Red,Arrow[{{0,0,0},#/6}]&/@(S1.Transpose[V1])},PlotRange{{-c,c},{-c,c},{-c,c}},AxesTrue]]

Out[]=

NNMF

In[]:=

{W,H}=GDCLS[pnts,2];

In[]:=

MatrixForm[H]

Out[]//MatrixForm=



44.731	2.64025	10.6537
0.	16.0886	6.71657



In[]:=

With[{c=40},Graphics3D[{Point[pnts],Blue,Arrow[{{0,0,0},#}]&/@Normal[H]},PlotRange{{-c,c},{-c,c},{-c,c}},AxesTrue]]

Out[]=

ICA

In[]:=

{A,S}=ResourceFunction["IndependentComponentAnalysis"][pnts,2];

In[]:=

Dimensions[A]

Out[]=

{600,2}

In[]:=

MatrixForm[S]

Out[]//MatrixForm=



-2.19859	0.356863	-3.4822
2.2057	-3.34072	-2.33556



In[]:=

With[{c=30},Graphics3D[{Point[pnts],Blue,Arrow[{{0,0,0},#}]&/@(5*S)},PlotRange{{-c,c},{-c,c},{-c,c}},AxesTrue]]

Out[]=

References

[AA1] Anton Antonov, A monad for Latent Semantic Analysis workflows, (2019), MathematicaForPrediction at GitHub.

[AA2] Anton Antonov, Monadic Latent Semantic Analysis Mathematica package, (2017), MathematicaForPrediction at GitHub.
URL: https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicLatentSemanticAnalysis.m .

Cite this as: Anton Antonov, "Simplified Machine-Learning Workflow #6" from the Notebook Archive (2020), https://notebookarchive.org/2020-09-55sqlz6

Latent Semantic Analysis (Part 1)

This LSA course through questions

What data is going used?

What is a typical workflow example?

What are the typical applications of LSA?

Why use LSA?

What it the fundamental philosophical or scientific assumption for LSA?

What is the most important and/or fundamental step of LSA? 

What is the difference between LSA and Latent Semantic Indexing (LSI)? 

What are the alternatives?

How is LSA used to derive similarities between two given texts?

How is LSA used to evaluate the proximity of phrases?

How the main dimension reduction methods compare?

SVD

NNMF

ICA

References

What is the most important and/or fundamental step of LSA?


What is the difference between LSA and Latent Semantic Indexing (LSI)?


What are the alternatives?


How is LSA used to evaluate the proximity of phrases?
