Educational Materials

Simplified Machine-Learning Workflow #11

Anton Antonov

Author

Anton Antonov

Title

Simplified Machine-Learning Workflow #11

Description

Semantic Analysis (Part 6)

Latent Semantic Analysis (Part 6)

A Wolfram livecoding session

Anton Antonov
February 2020

Session overview

Derive a custom taxonomy over a document collection.

Clustering with the reduced dimension.

Apply LSA to Great Conversation studies.

Use LSA for translation of natural languages.

Use LSA for making or improving search engines.

Data

In[]:=

ResourceFunction["ImportCSVToDataset"]["~/MathFiles/Presentations/Live-coding sessions Latent Semantic Analysis Worflows 2019-2020/Data-breakdown.csv"]

Out[]=

Dataset name	Collection size	Description	Format	Why use it?	Source
"Hamlet" (play parts)	222 documents	The different parts in Shakespeare's play.	CSV	Small enough to digest.	GitHub
US State of Union speeches	252 documents	US presidents speeches.	JSON.zip	Good English, similar document length.	GitHub
Airbnb reviews	~525,000 documents	Airbnb reviews by customers.	CSV.gz	A lot of relatively tight descriptions.	http://insideairbnb.com/get-the-data.html
Raku documentation	399 documents	Technical documentation of Raku (Perl 6).	Pod6	Very instructive and easy to explain.	https://github.com/Raku/doc
Movie reviews	~10,000 documents	Movie reviews (with labels "positive"/"negative").	WL	Good text data available in WL.	ExampleData[{"MachineLearning", "MovieReview"}]
MNIST	~70,000 images	Images of handwritten digits.	WL	Popular image collection available in WL.	ExampleData[{"MachineLearning", "MNIST"}]
"The Idiot" (chapters)	50 documents	Chapters of the novel by Dostoevsky in both English and Russian.	JSON.zip	Show translation possibilities.	GitHub
Data science class questions	99 documents	Student questions from Data Science class teaching.	Text	Eat your own dog food.	GitHub
rstudio::conf 2019 abstracts	61 records	Abstracts of the rstudio::conf 2019 conference	JSON, CSV	For comparison representation.	GitHub
WTC-2019 abstracts	182 records	Abstracts of the Wolfram Technology Conference 2019	JSON	Eat your own dog food.	GitHub
WTC-2016-2019 abstracts	584 records	Abstracts of the Wolfram Technology Conference from 2016 to 2019	CSV	Eat your own dog food.	GitHub

In[]:=

WebImage[

"https://github.com/antononcube/SimplifiedMachineLearningWorkflows-book/tree/master/Data"

]

Out[]=

USA presidential speeches

In[]:=

url="https://github.com/antononcube/MathematicaVsR/blob/master/Data/MathematicaVsR-Data-StateOfUnionSpeeches.JSON.zip?raw=true";str=Import[url,"String"];filename=First@Import[StringToStream[str],"ZIP"];

In[]:=

aUSASpeeches=Association[Import[StringToStream[str],{"ZIP",filename,"JSON"}]];Length[aUSASpeeches]

Out[]=

233

“The Idiot”

Russian text

In[]:=

url="https://github.com/antononcube/SimplifiedMachineLearningWorkflows-book/raw/master/Data/Dostoyevsky-The-Idiot-Russian-chapters.json.zip";str=Import[url,"String"];filename=First@Import[StringToStream[str],"ZIP"];

In[]:=

aIdiotRussianChapters=Association[Import[StringToStream[str],{"ZIP",filename,"JSON"}]];Length[aIdiotRussianChapters]

Out[]=

English text

In[]:=

url="https://github.com/antononcube/SimplifiedMachineLearningWorkflows-book/raw/master/Data/Dostoyevsky-The-Idiot-English-chapters.json.zip";str=Import[url,"String"];filename=First@Import[StringToStream[str],"ZIP"];

In[]:=

aIdiotEnglishChapters=Association[Import[StringToStream[str],{"ZIP",filename,"JSON"}]];Length[aIdiotEnglishChapters]

Out[]=

Load packages

The package MonadicLatentSemanticAnalysis.m implements the LSAMon software monad. The package ROCFunctions.m is used for the classification confusion matrix plot(s).

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicLatentSemanticAnalysis.m"]

In[]:=

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/BiSectionalKMeans.m"]Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/HeatmapPlot.m"]

Deriving of custom taxonomy

We have been doing “shallow”, zero-level taxonomies all along.

How to utilize LSA for making hierarchical taxonomies?

LSAMon application

In[]:=

Length[aUSASpeeches]

Out[]=

233

In[]:=

AbsoluteTiming[lsaUSASpeeches=LSAMonUnit[aUSASpeeches]⟹LSAMonMakeDocumentTermMatrix[{},Automatic]⟹LSAMonApplyTermWeightFunctions["IDF","TermFrequency","Cosine"]⟹LSAMonExtractTopics["NumberOfTopics"24,"MinNumberOfDocumentsPerTerm"5,Method"NNMF",MaxSteps12];]

Out[]=

{62.8773,Null}

lsaUSASpeeches⟹LSAMonEchoTopicsTable["NumberOfTableColumns"12];

topics table:

1.000	jobs
0.990	tonight
0.739	americans
0.685	businesses
0.679	don
0.634	college
0.526	kids
0.497	job
0.492	parents
0.481	help
0.442	21st
0.426	families

1.000	000
0.674	veterans
0.619	1928
0.570	banks
0.506	economic
0.459	depression
0.447	agriculture
0.444	banking
0.429	cent
0.420	marketing
0.411	federal
0.389	consolidation

1.000	interstate
0.830	corporations
0.655	forest
0.522	man
0.491	islands
0.470	standpoint
0.419	canal
0.416	industrial
0.412	tariff
0.398	conditions
0.393	philippine
0.392	conference

1.000	iraq
0.876	terrorists
0.686	iraqi
0.554	terror
0.481	terrorist
0.439	tonight
0.418	afghanistan
0.358	iraqis
0.354	saddam
0.340	hussein
0.314	qaida
0.306	regime

1.000	communist
0.657	program
0.569	1953
0.546	atomic
0.490	soviet
0.457	economic
0.403	programs
0.366	korea
0.332	communists
0.309	1954
0.282	insurance
0.266	billion

1.000	tonight
0.696	jobs
0.680	spending
0.653	budget
0.439	percent
0.431	let
0.409	deficit
0.404	americans
0.392	cuts
0.380	tax
0.347	economic
0.344	cut

1.000	1942
0.927	japanese
0.702	fighting
0.679	production
0.665	democratic
0.553	economic
0.530	1946
0.523	planes
0.492	enemies
0.490	nurses
0.489	1947
0.483	today

1.000	vietnam
0.639	tonight
0.264	propose
0.232	programs
0.222	help
0.212	americans
0.204	billion
0.192	percent
0.187	crime
0.181	seventies
0.177	america
0.177	try

1.000	billion
0.966	program
0.871	budget
0.831	programs
0.696	help
0.682	space
0.613	today
0.612	economic
0.607	dollar
0.564	million
0.502	inflation
0.497	goal

1.000	gentlemen
0.373	philadelphia
0.309	amity
0.308	majesty
0.299	commissioners
0.276	punctuality
0.265	article
0.252	britannic
0.248	seat
0.245	croix
0.212	engagements
0.212	6th

1.000	isthmus
0.822	panama
0.682	colombia
0.512	canal
0.494	colombian
0.373	1903
0.371	reclamation
0.366	railway
0.326	1899
0.315	industrial
0.299	forest
0.296	consular

1.000	emancipation
0.978	1911
0.650	1899
0.638	1910
0.578	loan
0.576	122
0.554	1912
0.526	convention
0.488	1900
0.485	liberia
0.466	consular
0.456	chile

1.000	spain
0.833	chambers
0.651	french
0.636	minister
0.539	treaty
0.492	1833
0.490	bank
0.456	duties
0.444	article
0.443	france
0.438	1832
0.419	1834

1.000	silver
0.678	1890
0.669	1892
0.657	gold
0.651	cent
0.580	1893
0.536	coinage
0.510	000
0.474	1891
0.451	1894
0.407	notes
0.406	bonds

1.000	kansas
0.863	1857
0.783	paper
0.745	slavery
0.727	currency
0.713	specie
0.690	notes
0.684	1858
0.679	1859
0.649	constitution
0.609	whilst
0.550	banks

1.000	gentlemen
0.770	indians
0.746	objects
0.615	tribes
0.588	militia
0.584	burthens
0.489	object
0.479	tranquillity
0.405	frontiers
0.402	provision
0.401	legislature
0.385	conducive

1.000	cuba
0.789	1897
0.778	rebellion
0.737	gold
0.728	cable
0.690	insurgents
0.688	spain
0.644	1869
0.641	insurrection
0.603	insurgent
0.567	pensioners
0.555	acres

1.000	1827
0.678	000
0.573	colonies
0.551	1824
0.474	1822
0.434	1828
0.411	1825
0.401	1826
0.364	discriminating
0.352	parliament
0.341	intercourse
0.334	surveys

1.000	british
0.937	enemy
0.877	militia
0.806	decrees
0.795	edicts
0.756	orders
0.587	savages
0.556	frigate
0.535	whilst
0.526	council
0.486	revocation
0.475	proofs

1.000	soviet
0.516	1980
0.449	oil
0.352	nuclear
0.334	inflation
0.333	salt
0.253	afghanistan
0.247	persian
0.241	strategic
0.229	israel
0.206	commitment
0.205	energy

1.000	mexico
0.638	oregon
0.634	texas
0.569	mexican
0.543	california
0.356	1846
0.349	steamers
0.296	1847
0.288	whilst
0.287	consuls
0.280	territory
0.260	lands

1.000	billion
0.769	programs
0.691	program
0.629	percent
0.614	1974
0.610	oil
0.593	vietnam
0.563	today
0.524	energy
0.517	goals
0.508	recovery
0.485	spending

1.000	harbors
0.882	dispositions
0.855	wabash
0.854	democracy
0.847	calculate
0.836	tripoli
0.830	louisiana
0.814	vessels
0.791	1803
0.759	augmentation
0.741	upward
0.736	floating

1.000	texas
0.900	mexico
0.649	submarines
0.548	annexation
0.503	battleships
0.493	programme
0.434	hundred
0.430	cruisers
0.413	destroyers
0.404	fifteen
0.359	mexican
0.329	serviceable

In[]:=

W=lsaUSASpeeches⟹LSAMonNormalizeMatrixProduct[NormalizedLeft]⟹LSAMonTakeW

Out[]=

SparseArray

Specified elements: 4258

Dimensions: {233,24}



In[]:=

H=lsaUSASpeeches⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeH

Out[]=

SparseArray



Specified elements: 144403

Dimensions: {24,10942}

Data not saved. Save now



FindClusters

In[]:=

cls=FindClusters[AssociationThread[RowNames[W],Identity/@SparseArray[W]],12];

In[]:=

Shallow/@cls

Out[]=

{{George.Washington.1790-01-08,George.Washington.1790-12-08,George.Washington.1791-10-25,George.Washington.1792-11-06,George.Washington.1793-12-03,George.Washington.1794-11-19,George.Washington.1795-12-08,George.Washington.1796-12-07,John.Adams.1797-11-22,Thomas.Jefferson.1806-12-02,154},{John.Adams.1798-12-08,John.Adams.1799-12-03,John.Adams.1800-11-11},{Thomas.Jefferson.1801-12-08,Thomas.Jefferson.1802-12-15,Thomas.Jefferson.1803-10-17,Thomas.Jefferson.1804-11-08,Thomas.Jefferson.1805-12-03,Thomas.Jefferson.1807-10-27,Woodrow.Wilson.1920-12-07,Franklin.D..Roosevelt.1941-01-06},{Thomas.Jefferson.1808-11-08,James.Madison.1809-11-29,James.Madison.1810-12-05,James.Madison.1811-11-05,James.Madison.1812-11-04,James.Madison.1813-12-07,James.Madison.1814-09-20},{James.Monroe.1824-12-07,John.Quincy.Adams.1825-12-06,John.Quincy.Adams.1826-12-05,John.Quincy.Adams.1827-12-04,John.Quincy.Adams.1828-12-02},{Andrew.Jackson.1836-12-05,Martin.Van.Buren.1839-12-02,John.Tyler.1841-12-07,John.Tyler.1842-12-06,John.Tyler.1843-12-06,James.Knox.Polk.1848-12-05,Franklin.Pierce.1855-12-31,James.Buchanan.1857-12-08,James.Buchanan.1858-12-06,James.Buchanan.1859-12-19,3},{John.Tyler.1844-12-03,James.Knox.Polk.1845-12-02,James.Knox.Polk.1846-12-08,James.Knox.Polk.1847-12-07,Woodrow.Wilson.1913-12-02,Woodrow.Wilson.1915-12-07,Woodrow.Wilson.1918-12-02},{Abraham.Lincoln.1862-12-01,Abraham.Lincoln.1863-12-08,Chester.A..Arthur.1882-12-04,Chester.A..Arthur.1883-12-04,William.McKinley.1900-12-03,William.Howard.Taft.1910-12-06},{Theodore.Roosevelt.1903-12-07},{Franklin.D..Roosevelt.1938-01-03,Franklin.D..Roosevelt.1942-01-06,Franklin.D..Roosevelt.1943-01-07,Franklin.D..Roosevelt.1944-01-11,Franklin.D..Roosevelt.1945-01-06,Harry.S..Truman.1946-01-21,Harry.S..Truman.1947-01-06,Harry.S..Truman.1948-01-07,Harry.S..Truman.1949-01-05,Harry.S..Truman.1950-01-04},{Lyndon.B..Johnson.1965-01-04,Lyndon.B..Johnson.1966-01-12,Lyndon.B..Johnson.1967-01-10,Lyndon.B..Johnson.1969-01-14,Richard.Nixon.1970-01-22,Richard.Nixon.1971-01-22},{Jimmy.Carter.1979-01-25,Jimmy.Carter.1980-01-21,Jimmy.Carter.1981-01-16}}

Hierarchical clustering (bottom-up)


Hierarchical clustering (top-down)

In[]:=

SeedRandom[23]clRes=BiSectionalKMeans[SparseArray[H],6,{"IndexClusters","HierarchicalTree"},DistanceFunctionCosineDistance]

Out[]=

IndexClusters{{9,13,15,16,17,20},{7},{1,11},{3,4,5,21},{2,8,12,14,18,19,22},{6,10,23,24}},HierarchicalTree{{1,{2,3}},{4,{5,6}}}

In[]:=

ColumnForm[RowNames[H]〚#〛&/@clRes["IndexClusters"]]

Out[]=

{communist-program-1953,1942-japanese-fighting,vietnam-tonight-propose,soviet-1980-oil,billion-program-budget,billion-programs-program}

{iraq-terrorists-iraqi}

{jobs-tonight-americans,tonight-jobs-spending}

{000-veterans-1928,silver-1890-1892,interstate-corporations-forest,isthmus-panama-colombia}

{spain-chambers-french,gentlemen-indians-objects,1827-000-colonies,british-enemy-militia,mexico-oregon-texas,gentlemen-philadelphia-amity,harbors-dispositions-wabash}

{kansas-1857-paper,cuba-1897-rebellion,emancipation-1911-1899,texas-mexico-submarines}

What is the Great Conversation?

In[]:=

WebImage["https://en.wikipedia.org/wiki/Great_Conversation"]

Out[]=

In[]:=

WikipediaData["Great Conversation"]

Out[]=

The Great Conversation is the ongoing process of writers and thinkers referencing, building on, and refining the work of their predecessors. This process is characterized by writers in the Western canon making comparisons and allusions to the works of earlier writers. As such it is a name used in the promotion of the Great Books of the Western World published by Encyclopædia Britannica Inc. in 1952. It is also the title of (i) the first volume of the first edition of this set of books, written by Robert Maynard Hutchins, and (ii) an accessory volume to the second edition (1990), written by Mortimer J. Adler. According to Hutchins, "The tradition of the West is embodied in the Great Conversation that began in the dawn of history and that continues to the present day". Adler said, What binds the authors together in an intellectual community is the great conversation in which they are engaged. In the works that come later in the sequence of years, we find authors listening to what their predecessors have had to say about this idea or that, this topic or that. They not only harken to the thought of their predecessors, they also respond to it by commenting on it in a variety of ways.== See also ==Standing on the shoulders of giantsTranslatio studii== Notes ==== External links ==The Tradition of the West – chapter one of "The Great Conversation" onlineHutchins, Robert. "The Classic Essay for The Great Books (extended excerpt of "The Great Conversation" that comes with the Second Edition of the Great Books of the Western World)" (PDF). Archived from the original (pdf) on 2014-11-22.Great Conversation book discussion group

Great Conversation timeline(s) of LSA topics

By century

In[]:=

tempTags=First/@StringCases[Keys[aUSASpeeches],y:(DigitCharacter..)~~"-"~~(DigitCharacter..)~~"-"~~(DigitCharacter..)StringTake[y,2]];Shallow[tempTags]Union[tempTags]

Out[]//Shallow=

{17,17,17,17,17,17,17,17,17,17,223}

Out[]=

{17,18,19,20}

In[]:=

matTemp=lsaUSASpeeches⟹LSAMonRepresentDocumentTagsByTopics[tempTags]⟹LSAMonTakeValue

Out[]=

SparseArray

Specified elements: 40

Dimensions: {4,24}



In[]:=

HeatmapPlot[matTemp,DistanceFunction{None,Sort},Dendrogram{True,False},ImageSize600]

Out[]=

By president

In[]:=

tempTags=StringReplace[Keys[aUSASpeeches],{"."~~(DigitCharacter..)~~"-"~~(DigitCharacter..)~~"-"~~(DigitCharacter..)""}];Shallow[tempTags]Length[Union[tempTags]]

Out[]//Shallow=

{George.Washington,George.Washington,George.Washington,George.Washington,George.Washington,George.Washington,George.Washington,George.Washington,John.Adams,John.Adams,223}

Out[]=

In[]:=

matTemp=lsaUSASpeeches⟹LSAMonRepresentDocumentTagsByTopics[tempTags]⟹LSAMonTakeValue

Out[]=

SparseArray

Specified elements: 150

Dimensions: {42,24}



In[]:=

HeatmapPlot[matTemp,DistanceFunction{None,Sort},Dendrogram{True,False},ImageSize600]

Out[]=

LSA translations of “The Idiot”

Stop words from other languages:

In[]:=

aStopWords=Association@Import["https://raw.githubusercontent.com/stopwords-iso/stopwords-iso/master/stopwords-iso.json"];

In[]:=

stopWords=Join@@Map[aStopWords,{"en","ru"}];

In[]:=

Length[aIdiotRussianChapters]

Out[]=

In[]:=

StringLength/@aIdiotRussianChapters

Out[]=

https://ilibrary.ru/text/94/p.1/index.html24205,https://ilibrary.ru/text/94/p.2/index.html21565,https://ilibrary.ru/text/94/p.3/index.html26990,https://ilibrary.ru/text/94/p.4/index.html35505,https://ilibrary.ru/text/94/p.5/index.html34771,https://ilibrary.ru/text/94/p.6/index.html23765,https://ilibrary.ru/text/94/p.7/index.html25919,https://ilibrary.ru/text/94/p.8/index.html28715,https://ilibrary.ru/text/94/p.9/index.html20578,https://ilibrary.ru/text/94/p.10/index.html14149,https://ilibrary.ru/text/94/p.11/index.html16324,https://ilibrary.ru/text/94/p.12/index.html20400,https://ilibrary.ru/text/94/p.13/index.html23913,https://ilibrary.ru/text/94/p.14/index.html24331,https://ilibrary.ru/text/94/p.15/index.html21531,https://ilibrary.ru/text/94/p.16/index.html23177,https://ilibrary.ru/text/94/p.17/index.html26389,https://ilibrary.ru/text/94/p.18/index.html28709,https://ilibrary.ru/text/94/p.19/index.html30356,https://ilibrary.ru/text/94/p.20/index.html13236,https://ilibrary.ru/text/94/p.21/index.html30794,https://ilibrary.ru/text/94/p.22/index.html31452,https://ilibrary.ru/text/94/p.23/index.html19977,https://ilibrary.ru/text/94/p.24/index.html40143,https://ilibrary.ru/text/94/p.25/index.html28548,https://ilibrary.ru/text/94/p.26/index.html28652,https://ilibrary.ru/text/94/p.27/index.html31335,https://ilibrary.ru/text/94/p.28/index.html12982,https://ilibrary.ru/text/94/p.29/index.html38136,https://ilibrary.ru/text/94/p.30/index.html26589,https://ilibrary.ru/text/94/p.31/index.html32328,https://ilibrary.ru/text/94/p.32/index.html32187,https://ilibrary.ru/text/94/p.33/index.html31570,https://ilibrary.ru/text/94/p.34/index.html38713,https://ilibrary.ru/text/94/p.35/index.html29163,https://ilibrary.ru/text/94/p.36/index.html27495,https://ilibrary.ru/text/94/p.37/index.html31298,https://ilibrary.ru/text/94/p.38/index.html14800,https://ilibrary.ru/text/94/p.39/index.html29496,https://ilibrary.ru/text/94/p.40/index.html18678,https://ilibrary.ru/text/94/p.41/index.html24296,https://ilibrary.ru/text/94/p.42/index.html28524,https://ilibrary.ru/text/94/p.43/index.html37561,https://ilibrary.ru/text/94/p.44/index.html35105,https://ilibrary.ru/text/94/p.45/index.html36855,https://ilibrary.ru/text/94/p.46/index.html39376,https://ilibrary.ru/text/94/p.47/index.html26715,https://ilibrary.ru/text/94/p.48/index.html30272,https://ilibrary.ru/text/94/p.49/index.html31041,https://ilibrary.ru/text/94/p.50/index.html8980

In[]:=

Length@*StringSplit/@aIdiotRussianChapters

Out[]=

https://ilibrary.ru/text/94/p.1/index.html3898,https://ilibrary.ru/text/94/p.2/index.html3483,https://ilibrary.ru/text/94/p.3/index.html4405,https://ilibrary.ru/text/94/p.4/index.html5381,https://ilibrary.ru/text/94/p.5/index.html5722,https://ilibrary.ru/text/94/p.6/index.html4136,https://ilibrary.ru/text/94/p.7/index.html4293,https://ilibrary.ru/text/94/p.8/index.html4595,https://ilibrary.ru/text/94/p.9/index.html3218,https://ilibrary.ru/text/94/p.10/index.html2234,https://ilibrary.ru/text/94/p.11/index.html2807,https://ilibrary.ru/text/94/p.12/index.html3252,https://ilibrary.ru/text/94/p.13/index.html3671,https://ilibrary.ru/text/94/p.14/index.html3810,https://ilibrary.ru/text/94/p.15/index.html3453,https://ilibrary.ru/text/94/p.16/index.html3743,https://ilibrary.ru/text/94/p.17/index.html4040,https://ilibrary.ru/text/94/p.18/index.html4695,https://ilibrary.ru/text/94/p.19/index.html5295,https://ilibrary.ru/text/94/p.20/index.html2159,https://ilibrary.ru/text/94/p.21/index.html4875,https://ilibrary.ru/text/94/p.22/index.html4972,https://ilibrary.ru/text/94/p.23/index.html3077,https://ilibrary.ru/text/94/p.24/index.html6277,https://ilibrary.ru/text/94/p.25/index.html4476,https://ilibrary.ru/text/94/p.26/index.html4563,https://ilibrary.ru/text/94/p.27/index.html5042,https://ilibrary.ru/text/94/p.28/index.html2169,https://ilibrary.ru/text/94/p.29/index.html6053,https://ilibrary.ru/text/94/p.30/index.html4242,https://ilibrary.ru/text/94/p.31/index.html5452,https://ilibrary.ru/text/94/p.32/index.html5059,https://ilibrary.ru/text/94/p.33/index.html5215,https://ilibrary.ru/text/94/p.34/index.html6342,https://ilibrary.ru/text/94/p.35/index.html4747,https://ilibrary.ru/text/94/p.36/index.html4784,https://ilibrary.ru/text/94/p.37/index.html4996,https://ilibrary.ru/text/94/p.38/index.html2518,https://ilibrary.ru/text/94/p.39/index.html4749,https://ilibrary.ru/text/94/p.40/index.html3014,https://ilibrary.ru/text/94/p.41/index.html3861,https://ilibrary.ru/text/94/p.42/index.html4617,https://ilibrary.ru/text/94/p.43/index.html6059,https://ilibrary.ru/text/94/p.44/index.html5469,https://ilibrary.ru/text/94/p.45/index.html5867,https://ilibrary.ru/text/94/p.46/index.html6565,https://ilibrary.ru/text/94/p.47/index.html4284,https://ilibrary.ru/text/94/p.48/index.html4738,https://ilibrary.ru/text/94/p.49/index.html5140,https://ilibrary.ru/text/94/p.50/index.html1301

In[]:=

aChapters=MapThread[StringRiffle[{##}," "]&,{Values[aIdiotRussianChapters],Rest@Values[aIdiotEnglishChapters]}];

lsaChapters=LSAMonUnit[aChapters]⟹LSAMonMakeDocumentTermMatrix[{},stopWords]⟹LSAMonApplyTermWeightFunctions["IDF","None","Cosine"]⟹LSAMonExtractTopics["NumberOfTopics"40,"MinNumberOfDocumentsPerTerm"3,Method"NNMF",MaxSteps12];

In[]:=

lsaChapters⟹LSAMonEchoTopicsTable["NumberOfTableColumns"10,"NumberOfTerms"20]⟹LSAMonEchoStatisticalThesaurus"Words""pistol","чиновник","portrait";

topics table:

1.000	burdovsky
0.608	бурдовский
0.495	бурдовского
0.426	павлищева
0.385	племянник
0.376	article
0.376	nephew
0.375	pavlicheff
0.352	gentlemen
0.325	demand
0.300	господа
0.294	бурдовскому
0.269	господин
0.253	pavlicheff’s
0.232	boxer
0.230	господина
0.225	lebedeff’s
0.218	имеете
0.216	павлищева»
0.210	lizabetha

1.000	aglaya
1.000	аглая
0.836	пишет
0.834	прочла
0.762	вскричала
0.747	краснеть
0.728	хочу
0.715	chosen
0.697	writes
0.663	letters
0.644	candle
0.602	бежать
0.601	позвала
0.551	письма
0.541	blush
0.539	asleep
0.534	замуж
0.533	unfair
0.515	начинала
0.509	письмо

1.000	wedding
0.850	церкви
0.610	keller
0.605	church
0.588	doctor
0.572	hippolyte’s
0.550	доктора
0.494	келлер
0.483	доктор
0.416	lebedeff
0.408	nastasia
0.369	crowd
0.342	рассказывала
0.342	капитанша
0.338	филипповна
0.338	настасья
0.332	крыльца
0.288	funeral
0.285	carriage
0.282	дарьи

1.000	packet
0.627	филипповна
0.627	настасья
0.539	nastasia
0.512	сто
0.509	philipovna
0.503	katia
0.450	алексеевна
0.447	дарья
0.434	burning
0.432	daria
0.429	alexeyevna
0.365	afanasy
0.359	congratulate
0.333	возьмет
0.330	totski
0.323	ferdishenko
0.312	иль
0.302	ivanovitch
0.257	полно

1.000	николаич
0.902	pistol
0.843	duel
0.807	знаешь
0.760	пистолет
0.603	_she_
0.548	дуэль
0.512	keller
0.504	злоба
0.480	шампанское
0.454	bench
0.437	парке
0.436	_her_
0.416	parfen
0.409	uncle
0.407	павлыча
0.404	умеете
0.401	we’ll
0.379	келлер
0.374	champagne

1.000	lebedeff
0.887	генеральша
0.883	lukian
0.832	епанчиных
0.757	москву
0.646	hotel
0.631	timofeyovitch
0.630	pavlofsk
0.622	князе
0.561	uncle
0.542	обстоятельство
0.528	москве
0.527	nephew
0.508	слухи
0.504	москвы
0.488	rogue
0.458	pray
0.432	девиц
0.419	домашних
0.418	записку

1.000	knight
0.963	‘poor
0.879	knight’
0.804	лебедев
0.699	секрет
0.694	poem
0.690	lebedeff
0.690	ideal
0.611	terrace
0.561	аглая
0.551	идеал
0.536	генеральша
0.536	aglaya
0.535	дачу
0.534	draw
0.527	лизавета
0.526	прокофьевна
0.503	цыпочках
0.480	epanchin
0.463	носил

1.000	молчите
0.957	letter
0.787	письмо
0.739	princess
0.705	bielokonski
0.654	epanchins
0.632	guests
0.604	аглая
0.596	patronage
0.595	епанчиных
0.592	aglaya
0.576	deliver
0.544	lebedeff
0.542	amiable
0.500	literary
0.488	лебедев
0.488	lizabetha
0.478	society
0.472	german
0.465	белоконская

1.000	чиновник
0.551	haired
0.482	clerk
0.453	zaleshoff
0.413	pskoff
0.411	родителя
0.393	bundle
0.377	sacrilege
0.376	залёжев
0.343	сибирь
0.302	neighbour
0.231	рогожин
0.227	содержал
0.227	fur
0.226	readiness
0.219	дескать
0.214	nastasia
0.211	parfen
0.207	couldn’t
0.207	чиновника

1.000	варя
0.910	ипполит
0.894	hippolyte
0.837	ганя
0.802	gania
0.763	varia
0.739	существовало
0.738	сударь
0.738	hospitality
0.693	captain
0.597	sir
0.557	curse
0.538	молчи
0.524	милостивый
0.493	exist
0.492	государь
0.480	говори
0.465	solely
0.452	птицыну
0.443	retired

1.000	totski
0.852	афанасий
0.851	иванович
0.636	тоцкий
0.533	фердыщенко
0.440	филипповна
0.428	настасья
0.358	афанасия
0.355	totski’s
0.333	ивановича
0.324	nastasia
0.288	тоцкого
0.224	настасьи
0.201	превосходительство
0.200	филипповны
0.198	philipovna
0.184	молчавший
0.183	барыня
0.165	peter
0.164	слух

1.000	lizabetha
0.885	лизавета
0.885	прокофьевна
0.846	ипполит
0.837	prokofievna
0.785	hippolyte
0.746	evgenie
0.666	евгений
0.662	павлович
0.630	fedorovitch
0.623	превосходительство
0.594	pavlovitch
0.507	раздражительно
0.496	burdovsky
0.471	ivan
0.448	смеетесь
0.446	mockery
0.444	article
0.426	excellency
0.415	terentieff

1.000	офицер
0.906	officer
0.796	evgenie
0.715	aglaya
0.648	vauxhall
0.643	pavlovitch
0.604	евгений
0.572	воксала
0.548	аглая
0.543	павлович
0.520	капитан
0.480	band
0.429	поцеловать
0.428	публики
0.412	впереди
0.391	близ
0.390	воксал
0.388	будьте
0.384	лизавета
0.384	прокофьевна

1.000	злая
0.937	лизавета
0.937	прокофьевна
0.928	мальчишка
0.909	письмо
0.854	надежд
0.813	записку
0.765	прислала
0.752	letter
0.652	верю
0.607	nicolai
0.591	lizabetha
0.587	видишь
0.516	wicked
0.510	покажи
0.509	стыдитесь
0.509	flatter
0.508	идешь
0.508	отвечай
0.507	лжешь

1.000	varia
0.977	рогожин
0.904	ганька
0.868	настасья
0.867	филипповна
0.792	nastasia
0.720	прихожей
0.665	blackguard
0.646	you’re
0.638	сорок
0.610	рогожина
0.597	philipovna
0.597	гостиной
0.587	александровну
0.587	нину
0.586	alexandrovna
0.567	залу
0.566	gania
0.563	ганя
0.557	рогожину

1.000	совершенство
0.716	сон
0.710	letters
0.699	perfection
0.666	подпись
0.665	signature
0.633	dream
0.500	кошмар
0.422	счастлив
0.371	видение
0.357	можете
0.354	saviour
0.349	logic
0.337	сном
0.333	обиды
0.333	прочитал
0.333	писала
0.333	хитро
0.332	вспоминал
0.329	обманули

1.000	ганя
0.799	генерал
0.774	gania
0.504	бумага
0.491	каллиграф
0.489	капельку
0.487	павлищев
0.473	common
0.469	учился
0.447	pavlicheff
0.434	elizabetha
0.431	имеете
0.427	перебил
0.424	writing
0.423	занять
0.391	schneider
0.387	притом
0.385	alexandrovna
0.375	познакомиться
0.369	бумаги

1.000	ипполит
0.955	hippolyte
0.870	читать
0.724	gentlemen
0.713	beast
0.709	чтение
0.701	тайна
0.634	господа
0.614	fortnight
0.572	trees
0.565	supper
0.539	wall
0.517	sealed
0.507	сны
0.475	packet
0.460	horrible
0.440	audience
0.425	скажут
0.425	некогда
0.425	испуга

1.000	humanity
0.784	человечества
0.654	ибо
0.539	evgenie
0.517	accursed
0.500	century
0.469	закон
0.466	шестьдесят
0.443	gentlemen
0.438	champagne
0.434	железных
0.432	евгений
0.417	pavlovitch
0.413	лебедев
0.363	павлович
0.346	scientific
0.345	сравнительно
0.345	springs
0.344	hippolyte
0.344	человечеству

1.000	napoleon
0.530	emperor
0.494	rome
0.353	записки
0.342	двенадцатом
0.293	камер
0.239	eye
0.237	генерал
0.234	memoirs
0.230	свидетелем
0.224	moscow
0.199	москве
0.196	ногу
0.196	дитя
0.179	witness
0.177	франц
0.177	french
0.177	уверяет
0.173	nina
0.167	secretary

1.000	varia
0.958	ганя
0.939	варя
0.721	gania
0.595	blackguard
0.486	varvara
0.458	originality
0.453	ардалионовна
0.453	варвара
0.420	commonplace
0.380	птицына
0.374	ptitsin
0.373	ardalionovna
0.357	александровна
0.357	нина
0.350	characters
0.350	действительности
0.329	черты
0.314	порох
0.313	подло

1.000	многоуважаемый
0.848	ferdishenko
0.804	фердыщенка
0.651	лебедев
0.614	lebedeff
0.513	тимофеич
0.483	господина
0.468	suspect
0.467	лукьян
0.465	сюртук
0.430	ибо
0.374	фердыщенко
0.339	wake
0.309	подозреваете
0.287	widow
0.287	sleep
0.287	pocket
0.268	вчерашнего
0.264	собственного
0.263	седьмом

1.000	parfen
0.997	парфен
0.964	крест
0.763	старушка
0.668	картину
0.583	cross
0.538	бога
0.519	picture
0.458	верует
0.458	рогожин
0.446	neck
0.418	впечатлением
0.417	набожно
0.413	прощай
0.383	проговорив
0.356	baby
0.334	silver
0.332	faith
0.310	stairs
0.307	серебряный

1.000	павлович
0.949	евгений
0.919	evgenie
0.845	прибавьте
0.796	aglaya
0.700	женюсь
0.573	правы
0.562	публично
0.511	женщине
0.508	nastasia
0.506	события
0.481	епанчиных
0.468	wedding
0.458	philipovna’s
0.458	настасьи
0.458	филипповны
0.452	аглая
0.435	pavlovitch
0.421	marrying
0.420	невесты

1.000	petrovitch
0.537	петрович
0.517	ivan
0.399	старичок
0.302	pavlicheff
0.248	иван
0.248	faith
0.240	atheism
0.231	vase
0.172	christ
0.161	белоконская
0.156	princess
0.144	odd
0.144	bielokonski
0.140	abbot
0.138	жажды
0.138	thirst
0.136	павлищева
0.135	павлищев
0.134	губернии

1.000	ferdishenko
0.670	totski
0.629	nastasia
0.567	марфа
0.468	game
0.438	actress
0.426	ardalion
0.426	alexandrovitch
0.388	ivolgin
0.345	philipovna’s
0.309	генерал
0.297	мышкин
0.295	иволгин
0.279	gentlemen
0.275	philipovna
0.259	guests
0.252	draw
0.249	terentieff
0.243	отвечай
0.243	ass

1.000	evgenie
0.964	келлер
0.904	keller
0.868	carriage
0.841	экипаж
0.836	intrigue
0.826	евгения
0.805	павлыча
0.797	павлычу
0.705	ганя
0.606	motives
0.598	векселя
0.582	евгением
0.551	занять
0.546	точь
0.536	repentance
0.535	base
0.530	участвовал
0.525	confession
0.504	евгению

1.000	pistol
0.889	ипполит
0.889	пистолет
0.831	hippolyte
0.693	shoot
0.636	brains
0.460	евгений
0.460	evgenie
0.437	keller
0.413	blow
0.412	павлович
0.394	morality
0.381	power
0.378	ипполита
0.371	fly
0.365	келлер
0.348	фердыщенко
0.346	existence
0.327	glory
0.326	приговора

1.000	злая
0.872	accidental
0.872	федорыч
0.816	евгений
0.803	russian
0.745	либерал
0.737	fedorovitch
0.735	evgenie
0.663	alexandra
0.625	александра
0.622	crimes
0.588	прокофьевна
0.585	лизавета
0.555	факт
0.549	павлович
0.540	ivan
0.522	недостаток
0.517	прощу
0.504	вещей
0.484	русского

1.000	project
0.296	agreement
0.289	foundation
0.179	literary
0.164	access
0.164	united
0.133	terms
0.120	including
0.096	permission
0.095	laws
0.093	agree
0.089	provide
0.070	owner
0.069	limited
0.066	mission
0.066	contact
0.066	phrase
0.057	evgenie
0.057	public
0.052	count

1.000	портрет
0.890	portrait
0.747	записку
0.634	ганя
0.594	генеральша
0.545	аглая
0.520	gania
0.496	aglaya
0.492	красота
0.412	дура
0.395	столовую
0.359	кабинете
0.330	надежду
0.321	note
0.315	аделаида
0.315	letter
0.313	maman
0.310	хороша
0.309	передать
0.306	проговорился

1.000	парфен
0.395	parfen
0.383	knife
0.326	нож
0.300	семеныч
0.280	убежала
0.273	воду
0.267	знаешь
0.265	думаешь
0.258	«ты
0.253	рогожин
0.251	you’d
0.236	москве
0.222	точь
0.204	книги
0.201	говоришь
0.180	границу
0.179	неужто
0.179	сидишь
0.178	marry

1.000	aglaya
0.858	аглая
0.698	nastasia
0.546	филипповна
0.546	настасья
0.486	ivanovna
0.471	аглаи
0.443	_her_
0.439	hippolyte
0.430	letters
0.425	свидание
0.412	письма
0.399	meeting
0.375	bundle
0.374	ипполит
0.349	secretary
0.349	servant
0.342	оставим
0.335	поняла
0.328	lizabetha

1.000	карты
0.972	rogojin’s
0.782	рогожин
0.638	porter
0.620	cards
0.616	рогожину
0.608	rogojin
0.570	hotel
0.504	трактире
0.456	служанка
0.455	али
0.455	рогожина
0.436	постели
0.429	окна
0.413	parfen
0.404	darkness
0.400	ночевал
0.400	дворника
0.400	коридоре
0.399	тротуара

1.000	бумажник
0.832	purse
0.406	lebedeff
0.337	aspirations
0.336	поле
0.304	тимофеич
0.290	многоуважаемый
0.281	esteemed
0.277	нашли
0.276	лукьян
0.265	лебедев
0.252	трактире
0.247	поздравить
0.242	генералом
0.239	ибо
0.230	стула
0.208	general’s
0.207	четыреста
0.206	нашел
0.201	interview

1.000	пачку
0.814	алексеевна
0.814	дарья
0.706	афанасий
0.638	настасья
0.637	филипповна
0.475	матушка
0.471	иванович
0.433	сошла
0.419	moscow
0.398	ice
0.389	household
0.385	рогожин
0.383	сто
0.371	фердыщенко
0.366	ишь
0.364	rumours
0.359	departure
0.341	varvara
0.301	princess

1.000	настасья
0.998	филипповна
0.975	иванович
0.971	афанасий
0.815	nastasia
0.636	фердыщенко
0.610	дарья
0.592	ferdishenko
0.580	philipovna
0.563	поступок
0.504	алексеевна
0.415	алексеевны
0.412	rogojin’s
0.410	рассказа
0.399	actress
0.348	statue
0.339	salaskin
0.336	mistress
0.332	katia
0.316	очереди

1.000	pushkin
0.921	буквы
0.881	nihilists
0.837	павлищева
0.826	evgenie
0.738	чтения
0.735	пушкина
0.686	прочла
0.672	павлович
0.670	‘poor
0.656	pavlovitch
0.653	сын
0.628	евгений
0.587	knight’
0.580	поздно
0.570	племянник
0.560	такою
0.538	евгения
0.531	павловича
0.526	всеобщее

1.000	лебедев
0.763	parfen
0.618	лукьян
0.396	знаешь
0.395	павловске
0.372	emperor
0.315	павловск
0.303	ибо
0.295	pope
0.295	semeonovitch
0.293	walls
0.267	рублей
0.265	племянник
0.251	дивана
0.231	кричал
0.230	дачу
0.224	haven’t
0.222	крикнул
0.216	rogojin
0.197	сироты

1.000	дети
0.966	village
0.773	children
0.601	деревне
0.565	генеральша
0.410	детьми
0.319	teach
0.313	kissed
0.312	шнейдер
0.307	rags
0.300	abbot
0.295	аделаида
0.283	schneider
0.280	adelaida
0.272	priest
0.272	старухи
0.265	идиота
0.257	расскажу
0.252	food
0.251	считают

statistical thesaurus:

term	statistical thesaurus entries
чиновник	чиновник,haired,clerk,zaleshoff,pskoff,родителя,sacrilege,залёжев,bundle,сибирь,neighbour,fur
pistol	pistol,пистолет,shoot,duel,brains,bench,парке,park,дуэль,николаич,rays,bag
portrait	portrait,портрет,красота,столовую,кабинете,дура,надежду,записку,хороша,неприятно,проговорился,передать

In[]:=

lsaChapters⟹LSAMonEchoStatisticalThesaurus"Words""письмо","анекдот","портрет","записку","note","morality","настасья","blush";

statistical thesaurus:

term	statistical thesaurus entries
письмо	письмо,letter,прислала,получил,верю,надежд,перебила,мальчишка,лжешь,письме,стыдитесь,wicked
анекдот	анекдот,подобно,false,правы,marrying,hero,пальце,нашу,daily,comprehensible,рассказа,reports
записку	записку,note,проговорился,красота,portrait,столовую,портрет,записке,hopes,кабинете,ardalionovitch,прочесть
портрет	портрет,portrait,красота,столовую,записку,дура,кабинете,надежду,хороша,неприятно,проговорился,прочесть
настасья	настасья,филипповна,philipovna,nastasia,фердыщенко,дарья,алексеевна,сто,афанасий,иванович,настасьи,филипповны
blush	blush,позвала,влюблена,unfair,грязная,_is_,начинала,бежать,ходили,lip,unnatural,неправда
morality	morality,душ,glory,прочее,bones,existence,humility,увидят,тремя,blow,приговора,суда
note	note,проговорился,записка,спокойно,хороша,улыбнулась,ответ,записке,столовую,умеет,commission,несчастны

In[]:=

TextTranslation"портрет","Russian""English"

Out[]=

Portrait

In[]:=

TextTranslation[#,"Russian""English",Method"Google"]&/@"пистолет","записку","письмо"

Out[]=

{pistol,a note,letter}

USA speeches search engine

In[]:=

smrUSASpeeches=SMRMonUnit[]⟹SMRMonCreate[<|"terms"(lsaUSASpeeches⟹LSAMonTakeWeightedDocumentTermMatrix),"topics"(lsaUSASpeeches⟹LSAMonNormalizeMatrixProduct[NormalizedLeft]⟹LSAMonTakeW)|>]⟹SMRMonApplyNormalizationFunction["Cosine"];

In[]:=

smrUSASpeeches⟹SMRMonRecommendByProfile[{"health","care"},6]⟹SMRMonEchoValue;

value:Bill.Clinton.1994-01-251.,Bill.Clinton.1993-02-170.621755,Barack.Obama.2009-02-240.561622,George.W..Bush.2001-02-270.399469,George.W..Bush.2004-01-200.375798,Bill.Clinton.2000-01-270.362744

References

[AA1] Anton Antonov, "The Great conversations in USA presidential speeches", (2016), MathematicaForPrediction at WordPress.

[AA2] Anton Antonov, "Ingest Airbnb reviews data", (2019), Wolfram Community.

Cite this as: Anton Antonov, "Simplified Machine-Learning Workflow #11" from the Notebook Archive (2020), https://notebookarchive.org/2020-09-55t1ktr

Latent Semantic Analysis (Part 6)

Session overview

Data

USA presidential speeches

“The Idiot”

Russian text

English text

Load packages

Deriving of custom taxonomy

LSAMon application

FindClusters

Hierarchical clustering (bottom-up)

Hierarchical clustering (top-down)

What is the Great Conversation?

Great Conversation timeline(s) of LSA topics

By century

By president

LSA translations of “The Idiot”

USA speeches search engine

References

Hierarchical clustering (bottom-up)
