Educational Materials

Simplified Machine-Learning Workflow #8

Anton Antonov

Author

Anton Antonov

Title

Simplified Machine-Learning Workflow #8

Description

Semantic Analysis (Part 3)

Latent Semantic Analysis (Part 3)

A Wolfram livecoding session

Anton Antonov
January 2020

Session overview

Review: last session’s example.

Review: the motivational example -- full blown LSA workflow.

Linear vector space representation:

LSA’s most fundamental operation,

matrix with named rows and columns.

Pareto Principle adherence

for a document,

for a document collection, and

(in general.)

Data

Out[]=

Dataset name	Collection size	Description	Why use it?	Source
"Hamlet" (play parts)	222 documents	The different parts in Shakespeare's play.	Small enough to digest.	GitHub
US State of Union speeches	252 documents	US presidents speeches.	Good English similar document length.	GitHub
Airbnb reviews	~525,000 documents	Airbnb reviews by customers.	A lot of relatively tight descriptions.	http://insideairbnb.com/get-the-data.html
Raku documentation	399 documents	Technical documentation of Raku (Perl 6).	Very instructive and easy to explain.	https://github.com/Raku/doc
Movie reviews	~10,000 documents	Movie reviews (with labels "positive"/"negative").	Good text data available in WL.	ExampleData[{"MachineLearning", "MovieReview"}]
"The Idiot" (chapters)	50 documents	Chapters of the novel by Dostoevsky in both English and Russian.	Show translation possibilities.	GitHub
Data science class questions	99 documents	Student questions from Data Science class teaching.	Eat your own dog food.	GitHub

Last session’s example --“Eat your own dog food”


Full LSA workflow (over Raku documentation)

Raku?

Formerly known as “Raku Perl 6”.

In[]:=

WebImage["https://www.raku.org"]

Out[]=

Where from?

In[]:=

WebImage["https://github.com/Raku/doc"]

Out[]=

Natural language commands

In[]:=

ToLSAMonWLCommand["create with aDocuments;make the document term matrix;show data summary;apply the LSI functions IDF, TermFrequency, Cosine;extract 36 topics using the method ICA and 12 max steps;show the topics table with 9 table columns;show thesaurus table of regex, array, chars, role, grammar;",False]

Out[]=

Hold[LSAMonUnit[aDocuments]⟹LSAMonMakeDocumentTermMatrix[]⟹LSAMonEchoDocumentTermMatrixStatistics[]⟹LSAMonApplyTermWeightFunctions[GlobalWeightFunctionIDF,LocalWeightFunctionNone,NormalizerFunctionCosine]⟹LSAMonExtractTopics[NumberOfTopics36,MethodICA,MaxSteps12]⟹LSAMonEchoTopicsTable[NumberOfTableColumns9]⟹LSAMonEchoStatisticalThesaurus[Words{regex,array,chars,role,grammar}]]

In[]:=

lsaNCRakuDoc=ToLSAMonWLCommand["create with aDocuments;make the document term matrix;show data summary;apply the LSI functions IDF, TermFrequency, Cosine;extract 36 topics using the method ICA and 12 max steps;show the topics table with 9 table columns;show thesaurus table of regex, array, chars, role, grammar;"];

Context value "documentTermMatrix":

Dimensions:

{371,24353}

Density:

0.00976839

Number ofdocuments per termsummary



1 # documents

1st Qu	1
Median	1
Min	1
3rd Qu	2
Mean	3.62407
Max	371



topics table:

1.000	numeric
0.924	keys
0.920	returning
0.907	native
0.840	method
0.837	elements
-0.815	handle
0.799	phaser
0.774	bind
0.734	output
0.725	error
-0.717	file

1.000	channel
0.994	proc
-0.958	phaser
0.808	syntax
-0.786	typecheck
-0.757	promise
0.748	package
0.744	placeholder
0.742	native
0.715	parameter
0.695	async
0.692	spec

1.000	comment
0.608	metamodel
-0.570	parameter
0.496	range
0.420	block
0.409	placeholder
0.402	regex
0.373	seq
0.343	consumed
-0.311	bind
-0.310	control
0.301	map

1.000	role
-0.913	token
0.818	control
-0.794	promise
0.766	scope
0.757	placeholder
0.748	bind
0.743	related
-0.726	proc
-0.709	attribute
0.689	comment
-0.674	failed

1.000	private
0.968	bind
0.910	block
0.872	repository
0.851	package
-0.732	parameter
0.697	regex
0.686	comment
-0.665	spec
0.596	say
0.537	promise
0.536	failed

1.000	comment
0.981	block
-0.794	date
-0.704	infix
-0.651	datetime
0.597	native
-0.549	proc
0.535	supply
0.533	control
-0.517	phaser
0.450	token
-0.419	async

1.000	syntax
-0.762	phaser
-0.706	enum
0.558	attribute
0.516	promise
-0.467	channel
-0.390	block
0.355	placeholder
-0.340	related
0.331	variable
0.329	array
0.305	signature

1.000	metamodel
0.614	channel
0.500	parameter
0.497	block
0.459	supply
-0.375	infix
0.374	bind
0.355	phaser
0.345	placeholder
0.342	promise
-0.321	codepoint
0.312	tap

1.000	native
0.934	term
-0.905	date
0.841	infix
-0.779	bind
0.705	block
0.690	failed
0.668	raku
-0.652	datetime
0.569	scope
-0.514	enum
-0.468	repository

1.000	token
-0.759	block
-0.727	repository
-0.685	supply
-0.634	placeholder
0.615	spec
0.599	bind
0.595	grammar
0.529	raku
-0.512	parameter
-0.508	role
-0.485	tap

1.000	channel
-0.910	control
0.769	placeholder
-0.577	token
-0.518	l<control
-0.506	supply
-0.501	bind
-0.501	exception\|/language/exceptions#control_exceptions>
0.487	=item
-0.427	exception>
0.401	consumed
-0.400	binding

1.000	private
0.993	scope
0.803	package
-0.794	channel
0.767	declaration
-0.706	range
-0.676	role
-0.675	raku
-0.659	related
-0.603	syntax
0.590	multi
0.587	metamodel

1.000	placeholder
-0.783	syntax
0.776	control
0.669	channel
-0.616	range
-0.569	supply
0.503	bind
-0.491	spec
0.485	date
0.484	block
0.482	consumed
0.448	=item

1.000	regex
0.926	native
0.732	=item
0.700	range
0.575	adverb
0.532	metamodel
-0.525	enum
-0.504	proc
-0.463	numeric
-0.457	path
-0.453	promise
-0.451	attribute

1.000	typecheck
0.952	promise
0.794	channel
-0.657	metamodel
0.528	native
-0.500	parameter
-0.486	raku
0.471	range
0.454	phaser
0.451	assignment
-0.437	=item
0.410	handle

1.000	regex
-0.825	metamodel
0.729	scope
-0.726	range
0.681	syntax
0.676	channel
0.623	declaration
-0.550	package
0.518	match
0.515	adverb
-0.514	proc
-0.441	repository

1.000	attribute
-0.897	metamodel
-0.682	syntax
0.578	=item
-0.403	private
0.353	parameter
-0.351	block
0.350	placeholder
0.331	enum
-0.328	date
0.321	package
-0.280	statement

1.000	proc
-0.767	phaser
0.736	async
0.697	repository
-0.626	handle
0.614	regex
0.607	failed
-0.596	codepoint
-0.579	private
0.564	numeric
-0.563	block
-0.563	$fh

1.000	control
0.839	consumed
-0.832	package
-0.824	=item
0.794	regex
0.791	repository
0.784	seq
-0.751	private
0.743	proc
0.726	path
0.681	message
-0.658	raku

1.000	spec
0.953	placeholder
-0.518	date
-0.467	consumed
-0.425	datetime
-0.413	parameter
-0.375	comment
0.344	codepoint
-0.336	seq
-0.331	role
-0.306	native
-0.278	proc

1.000	package
-0.829	bind
0.556	scope
0.553	date
0.535	control
-0.486	term
0.450	phaser
0.442	token
-0.439	infix
-0.380	slice
0.366	declaration
0.365	message

1.000	promise
0.900	comment
0.883	enum
0.775	spec
-0.730	private
0.705	parameter
-0.695	placeholder
-0.670	term
-0.662	package
-0.553	bind
0.545	=item
0.532	multi

1.000	metamodel
-0.833	parameter
-0.754	channel
-0.624	infix
-0.610	signature
0.592	bind
-0.578	term
0.514	native
0.502	handle
0.480	enum
0.459	attribute
0.458	package

1.000	supply
-0.753	block
0.673	channel
0.619	tap
-0.594	comment
-0.564	proc
-0.502	typecheck
-0.397	async
0.387	repository
0.372	date
0.358	native
0.342	promise

1.000	promise
0.999	enum
-0.907	path
-0.874	placeholder
0.707	channel
-0.689	method
0.654	infix
-0.628	returning
-0.605	role
-0.579	date
-0.530	token
-0.530	private

1.000	spec
0.939	promise
0.789	role
-0.698	token
-0.691	numeric
0.646	datetime
0.638	date
-0.620	channel
0.591	failed
-0.581	range
-0.550	attribute
0.492	native

1.000	typecheck
-0.814	promise
0.477	failed
0.475	enum
0.436	check
0.392	infix
0.374	assignment
-0.359	phaser
0.345	token
-0.327	scope
0.322	placeholder
-0.281	declaration

1.000	package
-0.992	metamodel
-0.921	range
-0.905	phaser
0.837	syntax
-0.792	bind
-0.753	infix
0.680	repository
0.655	typecheck
-0.635	multi
-0.627	numeric
0.612	keys

1.000	role
-0.512	date
-0.407	datetime
-0.347	native
-0.346	parameter
0.342	private
0.342	syntax
0.327	token
0.324	enum
-0.320	bind
-0.307	spec
0.307	attribute

1.000	spec
0.956	private
0.794	package
-0.741	repository
0.678	promise
0.637	attribute
0.636	regex
-0.615	enum
-0.600	metamodel
0.548	supply
-0.544	placeholder
0.523	consumed

1.000	promise
-0.716	proc
-0.548	async
0.535	regex
0.452	placeholder
-0.441	spec
-0.419	=item
0.366	infix
-0.358	supply
-0.320	phaser
0.312	metamodel
0.306	adverb

1.000	=item
0.924	syntax
-0.880	proc
0.712	phaser
0.674	repository
-0.641	async
0.630	spec
0.623	attribute
-0.560	scope
0.525	package
0.509	control
-0.498	placeholder

1.000	channel
0.821	comment
-0.723	token
-0.712	spec
0.608	metamodel
0.606	failed
-0.499	numeric
-0.497	placeholder
0.492	range
-0.480	consumed
0.467	control
0.467	private

1.000	repository
0.970	bind
-0.930	say
0.817	scope
0.737	distribution
0.736	term
0.702	spec
0.668	declaration
-0.623	output
-0.601	supply
0.565	infix
-0.535	placeholder

1.000	attribute
-0.558	promise
0.436	block
-0.421	=item
0.390	native
-0.363	token
-0.361	consumed
-0.348	parameter
-0.346	seq
-0.344	placeholder
0.315	submethod
0.315	required

1.000	channel
0.879	role
-0.634	syntax
0.558	attribute
0.558	package
-0.544	=item
-0.499	proc
0.452	metamodel
-0.445	enum
-0.434	raku
-0.410	placeholder
0.409	spec

statistical thesaurus:

term	statistical thesaurus entries
array	{array,index,list,arrays,@foo,element,c<array>,container,lists,lazy,push,containers}
chars	{chars,comb,repeated,contexts,l<iterable\|/type/iterable>,including,omitted,l<callable\|/type/callable>,lazily,applies,behaves,effects}
grammar	{grammar,parse,rule,top,actions,token,action,grammars,regexes,rest,args,whitespace}
regex	{regex,adverb,match,matches,abc,whitespace,pattern,character,construct,characters,quoting,describes}
role	{role,errors,related,common,stringy,additional,initialization,provide,compiler,public,act,roles}

Vector space model representation

The central object a sparse matrix with named rows and columns SSparseMatrix.

Text data -- Shakespeare’s sonnets

Text data - big data questions

In[]:=

Length[lsQuestions]

Out[]=

In[]:=

RandomSample[lsQuestions,6]

Out[]=

{ 4. How often do you generate report and data for your facility?, 3. Do you think that there would come a day that R will be replaced by other programs or languages?, 7. How much emphasis do you place on coding techniques like scripting, especially to help process huge amounts of data within a reasonable time frame? Are there other techniques you use to help expedite data processing?,My five questions for Anton:, Q3: What is the very first step you take when cleaning an enormous data set., 5. Do you think R will contribute to the future of AI with its machine learning capabilities?}

In[]:=

SeedRandom[94]lsTextParts=RandomSample[lsQuestions,20]

Out[]=

{,Questions: 1. Could you perhaps use another example to explain how you determined the most important variable?, 5. Are there certain journals you read to stay current on data-driven science?, Q1: What types of data would you say are the most challenging to collect, manage, and clean? Why?, 3. In one of our other classes we are learning about the ETL process and OMOP standards, when you’ve cleaned data in the past have you used the ETL process? If so, How was it done through R?, 2. Where do you see data scientist in healthcare in the future?, 1. What part of the data scientists’ daily work is the most challenging?, 5. Do you think R will contribute to the future of AI with its machine learning capabilities?, 4. What is a data scientist final destination? (from your point of view) or What do you see data scientists in 2040?, 2. How to determine which statistical program to utilize for a project to present the data?, 8. Do you think we will ever realize the promised potential of artificial intelligence (AI), and if not, what is keeping us from doing so?,, 2. You mentioned multiple libraries for data, which ones do you use the most? And have better interface with R?, 5. What was your toughest project and how long did it take to accomplish?, 2. How long do you think it will take to be 90% proficient in R programming language? Assume: a person works 7 hours a day in an R environment., 4. Do data analysts sometimes perform the duties of a data scientist and vice versa?, 1. I’ve read that R has a steep learning curve for beginning programmers and coders, what kind of advice would you have for someone who is new and starting off with R and computer programming altogether to overcome this learning curve? , a. They make prediction?,Questions:,}

Matrix representation

In[]:=

lsaObj=LSAMonUnit[lsTextParts]⟹LSAMonMakeDocumentTermMatrix[{},Automatic];

In[]:=

smat=lsaObj⟹LSAMonTakeDocumentTermMatrix

Out[]=

SparseArray

Specified elements: 121

Dimensions: {20,99}



In[]:=

MatrixForm[smat]

Out[]//MatrixForm=

In[]:=

MatrixPlot[smat]

Out[]=

In[]:=

RowSumsAssociation[smat]

Out[]=

id.0010,id.0027,id.0038,id.0047,id.00513,id.0064,id.0075,id.0086,id.0099,id.0107,id.0117,id.0120,id.0138,id.0144,id.01512,id.0169,id.01718,id.0182,id.0191,id.0200

In[]:=

ReverseSort@ColumnSumsAssociation[smat]

Out[]=

data11,learning4,think3,scientist3,use2,read2,questions2,project2,programming2,process2,long2,future2,etl2,curve2,challenging2,you’ve1,works1,work1,view1,vice1,versa1,variable1,utilize1,used1,types1,toughest1,steep1,stay1,statistical1,starting1,standards1,sometimes1,scientists’1,scientists1,science1,say1,realize1,promised1,programmers1,program1,proficient1,present1,prediction1,potential1,point1,person1,perform1,past1,overcome1,ones1,omop1,new1,multiple1,mentioned1,manage1,make1,machine1,libraries1,language1,kind1,keeping1,journals1,i’ve1,interface1,intelligence1,important1,hours1,healthcare1,final1,explain1,example1,environment1,duties1,driven1,determined1,determine1,destination1,day1,daily1,current1,contribute1,computer1,collect1,coders1,cleaned1,clean1,classes1,certain1,capabilities1,better1,beginning1,assume1,artificial1,analysts1,altogether1,advice1,accomplish1,90%1,20401

In[]:=

smat//Head

Out[]=

SSparseMatrix

In[]:=

SparseArray[smat]

Out[]=

SparseArray

Specified elements: 121

Dimensions: {20,99}



Further operations for the sparse matrix with named rows and columns

Arithmetic

In[]:=

MatrixPlot[10*smat]MatrixForm[10*smat]

Out[]=

Out[]//MatrixForm=

In[]:=

MatrixPlot[10+smat]MatrixForm[10+smat]

Out[]=

Out[]//MatrixForm=

Transposing

In[]:=

MatrixPlot[Transpose[smat]]

Out[]=

Similarity matrix

In[]:=

MatrixPlot[smat.Transpose[smat]]MatrixForm[smat.Transpose[smat]]

SSparseMatrix

::dnsame

:The dimension names {1,1} are the same; using {1, 2} instead.

Out[]=

SSparseMatrix

::dnsame

:The dimension names {1,1} are the same; using {1, 2} instead.

Out[]//MatrixForm=

	id.001	id.002	id.003	id.004	id.005	id.006	id.007	id.008	id.009	id.010	id.011	id.012	id.013	id.014	id.015	id.016	id.017	id.018	id.019	id.020
id.001	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.002	0	7	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0
id.003	0	0	8	1	1	1	1	0	2	1	0	0	1	0	0	2	1	0	0	0
id.004	0	0	1	7	1	1	2	0	2	1	0	0	1	0	0	2	0	0	0	0
id.005	0	0	1	1	17	1	1	1	2	1	0	0	1	0	0	2	2	0	0	0
id.006	0	0	1	1	1	4	1	1	3	1	0	0	1	0	0	3	0	0	0	0
id.007	0	0	1	2	1	1	5	0	2	1	0	0	1	0	0	2	0	0	0	0
id.008	0	0	0	0	1	1	0	6	0	0	1	0	0	0	1	0	2	0	0	0
id.009	0	0	2	2	2	3	2	0	11	2	0	0	2	0	0	5	0	0	0	0
id.010	0	0	1	1	1	1	1	0	2	7	0	0	1	1	0	2	0	0	0	0
id.011	0	0	0	0	0	0	0	1	0	0	7	0	0	0	1	0	0	0	0	0
id.012	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
id.013	0	1	1	1	1	1	1	0	2	1	0	0	8	0	0	2	0	0	0	0
id.014	0	0	0	0	0	0	0	0	0	1	0	0	0	4	1	0	0	0	0	0
id.015	0	0	0	0	0	0	0	1	0	0	1	0	0	1	12	0	1	0	0	0
id.016	0	0	2	2	2	3	2	0	5	2	0	0	2	0	0	11	0	0	0	0
id.017	0	0	1	0	2	0	0	2	0	0	0	0	0	0	1	0	22	0	0	0
id.018	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	0
id.019	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0
id.020	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

In[]:=

(*MatrixPlot[Transpose[smat].smat]MatrixForm[Transpose[smat].smat]*)

Matrix-vector multiplication -- most similar documents

In[]:=

Normal[SparseArray[smat〚{5},All〛]]

Out[]=

{{0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1}}

In[]:=

MatrixForm[smat.Transpose[smat〚{5},All〛]]

SSparseMatrix

::dnsame

:The dimension names {1,1} are the same; using {1, 2} instead.

Out[]//MatrixForm=

	id.005
id.001	0
id.002	0
id.003	1
id.004	1
id.005	17
id.006	1
id.007	1
id.008	1
id.009	2
id.010	1
id.011	0
id.012	0
id.013	1
id.014	0
id.015	0
id.016	2
id.017	2
id.018	0
id.019	0
id.020	0

In[]:=

ReverseSort@RowSumsAssociation[smat.Transpose[smat〚{5},All〛]]

SSparseMatrix

::dnsame

:The dimension names {1,1} are the same; using {1, 2} instead.

Out[]=

id.00517,id.0172,id.0162,id.0092,id.0131,id.0101,id.0081,id.0071,id.0061,id.0041,id.0031,id.0200,id.0190,id.0180,id.0150,id.0140,id.0120,id.0110,id.0020,id.0010

In[]:=

(lsaObj⟹LSAMonTakeDocuments)["id.005"]

Out[]=

3. In one of our other classes we are learning about the ETL process and OMOP standards, when you’ve cleaned data in the past have you used the ETL process? If so, How was it done through R?

In[]:=

(lsaObj⟹LSAMonTakeDocuments)["id.017"]

Out[]=

1. I’ve read that R has a steep learning curve for beginning programmers and coders, what kind of advice would you have for someone who is new and starting off with R and computer programming altogether to overcome this learning curve?

In[]:=

(lsaObj⟹LSAMonTakeDocuments)["id.013"]

Out[]=

2. You mentioned multiple libraries for data, which ones do you use the most? And have better interface with R?

Additional functions

In[]:=

MatrixForm[Clip[smat,{0,1}]]

Out[]//MatrixForm=

Pareto Principle adherence

See the resource function ResourceFunction["ParetoPrinciplePlot"] :

In[]:=

ResourceFunction["ParetoPrinciplePlot"][RandomReal[{0,1},1000]~Join~RandomReal[{0,0.1},1000]]

In Hamlet (single document)

In[]:=

text=ExampleData[{"Text","Hamlet"}];

In[]:=

text

In[]:=

words=Select[StringSplit[ToLowerCase[text],{" ",",",".",";","?","!","...","-"}],StringLength[#]>0&];Length[words]

Out[]=

32318

In[]:=

sentences=Select[StringSplit[ToLowerCase[text],{".",";","?","!","..."}],StringLength[#]>0&];Length[sentences]

Out[]=

4175

In[]:=

ColumnForm[RandomSample[sentences,2]]

Out[]=

o gertrude, come away

what ceremony else

In[]:=

LSAMonUnit[sentences]⟹LSAMonMakeDocumentTermMatrix[]⟹LSAMonEchoDocumentTermMatrixStatistics["ParetoPrinciplePlots"True]⟹LSAMonExtractTopics[12,Method"NNMF",MaxSteps12]⟹LSAMonEchoTopicsTable["NumberOfTableColumns"6];

Context value "documentTermMatrix":

Dimensions:

{4175,4376}

Density:

0.000780851

Number ofdocuments per termsummary



1 # documents

1st Qu	1
Median	1
Min	1
3rd Qu	2
Mean	3.26005
Max	358



topics table:

1.000	thou
0.863	earth
0.749	art
0.701	hast
0.432	grave
0.293	hamlet
0.265	thine
0.210	heaven
0.136	thee
0.121	use
0.102	long
0.100	good

1.000	speak
0.056	thee
0.049	nay
0.030	horatio
0.028	fellow
0.026	stay
0.026	set
0.022	let
0.022	believe
0.021	sword
0.019	man
0.018	gertrude

1.000	fear
0.059	marcellus
0.055	love
0.046	dear
0.044	ophelia
0.043	hand
0.031	goes
0.028	like
0.027	watch
0.026	free
0.026	great
0.026	doth

1.000	[exeunt
0.135	rosencrantz
0.121	guildenstern
0.072	hamlet
0.039	players
0.034	polonius
0.023	horatio
0.017	ghost
0.015	guil
0.010	queen
0.009	marcellus
0.008	dead

1.000	[exit
0.097	polonius
0.056	ghost
0.056	horatio
0.017	queen
0.017	lord
0.013	second
0.010	clown
0.001	offence
0.000	mean
0.000	hath
0.000	friends

1.000	[enter
0.239	hamlet
0.236	polonius
0.202	rosencrantz
0.194	guildenstern
0.160	horatio
0.122	queen
0.087	ghost
0.075	marcellus
0.063	laertes
0.058	ophelia
0.023	gentleman

1.000	hor
0.003	hamlet
0.003	lord
0.000	laertes
0.000	body
0.000	answer
9.292× -6 10	dead
7.017× -6 10	denmark
6.888× -6 10	better
6.634× -6 10	away
5.587× -6 10	tell
5.280× -6 10	gertrude

1.000	ros
0.017	guil
0.005	[to
0.005	friends
0.004	say
0.003	good
0.002	matter
0.001	mother
0.001	[exeunt
0.000	[enter
0.000	leave
0.000	laertes

1.000	oph
0.006	nay
0.006	ophelia
0.000	walk
0.000	enter
0.000	eye
0.000	given
3.697× -6 10	[exeunt
3.273× -6 10	believe
1.918× -6 10	follow
1.746× -6 10	fear
1.589× -6 10	earth

1.000	king
0.011	laertes
0.009	queen
0.009	father
0.009	like
0.006	body
0.005	shall
0.005	kill
0.005	nay
0.004	sir
0.004	comes
0.004	lord

1.000	thy
0.135	father
0.119	thee
0.116	thou
0.089	hamlet
0.075	make
0.068	thine
0.065	love
0.061	soul
0.059	dead
0.057	spirit
0.052	let

1.000	mother
0.744	follow
0.161	let
0.146	thee
0.074	matter
0.068	gertrude
0.066	good
0.065	night
0.065	father
0.059	queen
0.059	come
0.048	lord

In[]:=

ResourceFunction["ParetoPrinciplePlot"][Tally[words][[All,2]],ImageSizeLarge]

Out[]=

In[]:=

ResourceFunction["ParetoPrinciplePlot"][{Tally[words][[All,2]],Tally[DeleteStopwords[words]][[All,2]]},ImageSizeLarge]

Out[]=

In the questions

In[]:=

LSAMonUnit[lsQuestions]⟹LSAMonMakeDocumentTermMatrix[{},Automatic]⟹LSAMonEchoDocumentTermMatrixStatistics["ParetoPrinciplePlots"True];

Context value "documentTermMatrix":

Dimensions:

{99,320}

Density:

0.0179924

Number ofdocuments per termsummary



1 # documents

1st Qu	1
Median	1
Min	1
Mean	1.78125
3rd Qu	2
Max	33



In[]:=

LSAMonUnit[aDocuments]⟹LSAMonMakeDocumentTermMatrix[{},Automatic]⟹LSAMonEchoDocumentTermMatrixStatistics["ParetoPrinciplePlots"True];

Context value "documentTermMatrix":

Dimensions:

{371,24353}

Density:

0.00976839

Number ofdocuments per termsummary



1 # documents

1st Qu	1
Median	1
Min	1
3rd Qu	2
Mean	3.62407
Max	371



UN Human Rights in different languages (single document)

In[]:=

GetWords[text_String]:=Select[StringSplit[ToLowerCase[text],{" ",",",".",";","?","!","...","-"}],StringLength[#]>0&];textIDs=Select[ExampleData["Text"],StringMatchQ[#[[2]],"UNHuman"~~__]&];textNames=StringTake[#[[2]],{StringLength["UNHumanRights"]+1,-1}]&/@textIDs;textData=MapThread[Tooltip[Tally[GetWords[ExampleData[#1]]][[All,2]],#2]&,{textIDs,textNames}];textK=0;textData=Map[If[MemberQ[{"English","Hawaiian","Japanese","Korean","Latin","Maori","Russian"},#[[2]]],Callout[#,#[[2]],{100+35*(textK++),Below}],#]&,textData];ResourceFunction["ParetoPrinciplePlot"][textData,"ParetoGridLines"{Automatic,Automatic},PlotLegendstextNames,JoinedTrue,ImageSize1000,PlotLabelStyle["Pareto principle adherence for word tallies of\n\"UN Human Rights\" in different languages\n",Bold,Purple,FontSize14]]

Out[]=

	Chinese
	Danish
	Dutch
	English
	Esperanto
	French
	German
	Greek
	Hawaiian
	Indonesian

	Irish
	Japanese
	Korean
	Latin
	Maori
	Russian
	ScottishGaelic
	Spanish
	Swahili
	Swedish

In[]:=

Length[StringSplit[ExampleData[{"Text","UNHumanRightsJapanese"}]," "]]

Out[]=

164

In[]:=

Length[StringSplit[ExampleData[{"Text","UNHumanRightsEnglish"}]," "]]

Out[]=

1772

In[]:=

Length[StringSplit[ExampleData[{"Text","UNHumanRightsMaori"}]," "]]

Out[]=

3227

In[]:=

ExampleData[{"Text","UNHumanRightsMaori"}]

Out[]=

No te mea na te whakanoa a na te whakahawea ki nga mana o te tangata i tupu ai nga mahi whakarihariha i pouri ai te ngakau tangata, a ko te kohaetanga o tetahi ao hou e mahorahora ai te tangata ki te korero ki te whakapono, ki te noho noa i runga i te rangimarie a i te ora, kua panuitia hei taumata mo te koingotanga o te ngakau o te mano tini o te tangata. No te mea ki te kore te tangata ae akina kia tae ki te tino hemanawatanga kia hapai hoki i te pakanga hei turaki i te mana tukino hei pehi hoki i te iwi, he mea tika rawa kia tiakina nga mana o te tangata i raro i tenga ritenga o te ture. No te mea he mea tika rawa kia hapainga nga ritenga e tupu ai nga whakaaro whakawhanaunga i waenganui i nga iwi o te ao. No te mea ko nga iwi o te Kotahitanga kua whakaatu ki roto i te Kawenata i te tino whakapumautanga o to ratou whakapono mo te kaupapa o nga mana motuhake o te tangata, mo te ihi me te wana o te tinana tangata, a mo nga tika tauriterite o te tane me te wahine, a kua whakarite tikanga hoki kia hapainga nga ritenga toko i te ora me te whakapiki i nga ahuatanga o te oranga i tenei ao i roto i te rangatiratanga whanui. No te mea ko nga iwi o roto i tenei Kotahitanga kua oati i runga i te whakaaro kotahi kotaki ki te hapai i te ritenga whanui o te whakanui me te pupuri i nga tika o te tangata me nga kaupapa tuturu o nga rangatiratanga o te ora i tenei ao. No te mea e tutaki ai tenei oati he mea nui rawa kia matou te mano tini o te tangata ki enei tikanga rangatira. No reira inaianei: ko te Huihuinga Topu o Te Kotahitanga o Nga Iwi o Te Ao. Ka Paanui I Tenei Whakapuakitanga Whanui O Nga Mana O Te Tangata. Ka meinga nei hei tauira whanui mo nga tikanga hei whainga kia tutaki i te katoa o nga iwi i runga i te whai a tena tangata a tena tangata me nga ropu katoa o roto i nga whakahaere me te pupuri tonu i nga wa katoa i roto i te hinengaro i tenei Whakapuakitanga, me whakapau i te kaha i runga i te ritenga tohutohu me te ako ki te hapai i nga tikanga rangatira: a ma runga hoki i nga whakaritenga e ahu whakamua ana a ia iwi puta noa te ao, kia mau ai te tuturutanga ki te katoa, o te pono o te pupuri me te whakarite i enei ritenga i waenganui o nga iwi o roto i te kotahitanga, hui tahi atu ki nga iwi o nga whenua kei raro i o ratou mana whakahaere. Rarangi 1 Ko te katoa o nga tangata i te whanaungatanga mai e watea ana i nga here katoa; e tauriterite ana hoki nga mana me nga tika. E whakawhiwhia ana hoki ki a ratou te ngakau whai whakaaro me te hinengaro mohio ki te tika me te he, a e tika ana kia meinga te mahi a tetahi ki tetahi me ma roto atu i te wairua o te noho tahi, ano he teina he tuakana i ringa i te whakaaro kotahi. Rarangi 2 E whai mana ana ia tangata kia whiwhi ki te katoa o nga rangatiratanga me nga huarahi whanui o te ao e whakakaupapatia nei i roto i tenei Whakapuakitanga, kaua e araitia ahakoa pewhea, ara i runga i enei ahuatanga e whai ake nei, i te mea he iwi ke, i te ahua kua kiri ke, i te tanetanga i te wahinetanga, i te reo, i te whakapono, i te awhina ropu whakaara ture i tetahi atu kaupapa whakaaro ranei, ahakoa no roto mai i te iwi whanui no tetahi ropu ranei, no te kaupapa pupuri taonga, no te whanaungatanga mai, no tetahi tunga whai-tikanga ranei He apiti atu ki enei, kaua e meinga hei ritenga wehewehe te mea i whakakaupapatia na runga i nga whakahaere ture, i nga mana whanui ranei o te ao kua whakawhiwhia ki tetahi whenua ki tetahi wahanga whenua ranei no reira nei tetahi tangata, ahakoa taua wahanga whenua he whai mana motuhake, kei raro ranei i te Kaitiakitanga, he takiwa whenua ranei kahore nei ona Mana Kawanatanga Motuhake, kei raro ranei i tetahi atu ritenga whakawhaiti i tona mana motuhake. Rarangi 3 Ko ia tangata e whai take ana ki te mauri ora, me watea i nga tikanga tere, me maru hoki ia i raro i te mana o te ture. Rarangi 4 Kaua tetahi tangata e noho pononga a ko nga tikanga whakapononga i te tangata me takahi rawa atu. Rarangi 5 Kaua tetahi tangata e whakamamae noatia e tukua ranei ki nga tikanga whakaiti me nga whiu kahore nei i tika mo te tangata. Rarangi 6 Ko ia tangata e whai mana ana kia mohiotia ki nga wahi katoa he tangata ano ia i te aroaro o te ture. Rarangi 7 E tauriterite ana te katoa ki te aroaro o te ture a e tika ana kia tiakina e te ture, kaua he rereketanga mo tetahi i tetahi. E tika ana te katoa kia rite te tiakina kei kapea e tetahi tikanga e takahi ana i tenei whakapuakitanga, a e tetahi tikanga ranei e whakatara ana kia pera. Rarangi 8 Ko ia tangata e whai take ana kia whiwhi ki tetahi rongoa totika o te ture i na roto mai i nga Kooti Whakawa whai mana mo nga mahi e takahi ana i te tino ritenga o nga mana kua whakaputaina ki a ia e te kaupapa e te ture ranei. Rarangi 9 Kaua tetahi tangata e hopukia noatia e te ringa o te ture e puritia noatia ranei i roto i tetahi whare herehere e peia noatia ranei ki tetahi whenua ke. Rarangi 10 Ko ia tangata e tika ana kia whakatuturutia ki a ia tetahi whakawa tika ki te aroaro o te katoa e tetahi runanga wehekore whakahoahoa ranei, mo runga i te whakataunga i ona tika me nga tikanga hei whakarite mana tae atu hoki ki nga whakapae mona tera kua hara kino ia i raro i te ture. Rarangi 11 1. Ko ia tangata kua whapaea mo tetahi hara e ahei nei whiu e tika ana kia ki ia kahore ia i hara tae noa ki te wa e whakataua ai ae i hara ia e ai ki ta te ture i roto i tetahi whakawatanga i te aroaro o te katoa a i reira i whiwhi ia i te katoa o nga huarahi karo, tautoko hoki i a ia. 2. Kaua tetahi tangata e kiia kua hara mo runga mo tetahi hara e ahei nei kia whiua na runga na tetahi mahi i mahia, na tetahi mea ranei kihai i mahia e ia kaore nei e tika ana kia meinga he hara i raro i te ture o tetahi whenua i te ture ranei o te ao whanui i te wa o taua hara. Kaua hoki tetahi whiu e utaina atu ki te whiu e meinga ana mo taua te hara i taua wa. Rarangi 12 Kaua tetahi tangata e whakararuraru pokonoatia, tana whanau, tona kainga, ana pukapuka ranei, e takahi ranei i tona ingoa nui me tona ingoa pai. Ko ia tangata e ahei ana ma te ture e tiaki kei pa enei ahua whakararu me enei tukinotanga. Rarangi 13 1. Ko ia tangata e whai-tika mo ana haere katoa me ona wahi noho tuturu ai i roto i nga rohe o tona whenua. 2. Ko ia tangata e whai-tika kia whakarere i tetahi whenua, ahakoa ko tona whenua ake, a me te hoki ano ki tena whenua tupu. Rarangi 14 1. Ko ia tangata e whai-tika ana kia rapu kia whiwhi hoki ki tetahi oranga ngakau ki etahi atu whenua ina whanatu ia kia pahemo i nga tukino mona. 2. Ko tenei tika e kore e ahei kia inoitia mo runga i te take whakawhiu i tupu pono ake na etahi hara kahore nei no nga ropu hanga ture, a na roto mai ranei i etahi mahi e peka ke ana i te ahuatanga me nga kaupapa tuturu o te Kotahitanga o nga Mana Nunui o te Ao. Rarangi 15 Ko ia tangata e whai tika ana ki tetahi iwi tuturu mona. Kaua tetahi tangata e meinga kia takirihia atu e te ringa kaha i tona iwi tuturu, a kaua hoki e araia ina mea ia ki tetahi atu iwi hei iwi tuturu mona. Rarangi 16 1. Ko nga tane me nga wahine kua eke nei nga tau ki te pakehatanga, e whai-tika ana kia moe tane, wahine hoki, a kia whakatupu uri ki te ao, a kaua e araia tenei tika na runga ï te mea he momo tangata ke atu, he iwi ke, he whakapono ke ranei. E whai mana, ana ratou kia tauriterite te tika i runga i te marena i te wa e noho ana he tane he wahine a tae noatia te wa e wehe tuturu ai. 2. Ko te maramatanga me meinga i urutomotia i runga i te whakaaetanga watea, tuturu hoki a te tane me te wahine e marenatia ana. 3. Ko te whanau te hunga tuturu pupuri pono hoki i te kaupapa tika o te iwi, a e tino tika ana kia tiakina e te iwi me te Motu katoa. Rarangi 17 1. Ko ia tangata e whai-tika ana kia whiwhi ki ona taonga ake, kia whiwhi tahi ranei ratou ko etahi atu. 2. Kaua tetahi tangata e murua e te ringa kaha i ona taonga ake. Rarangi 18 1. Ko ia tangata e whai-tika ana ki ona ake whakaaro, o tona hinengaro me tona nei whakapono; e tapiritia ana hoki ki a ia te mana kia watea ia ki te whakauru atu ki tetahi atu whakapono, hahi ranei, a me watea hoki, ki a ia anake ki a ia ranei me etahi atu ki te aroaro o te katoa, ki tetahi wahi wehi ranei, te mana ki te whakarite i tona whakapono, hahi ranei, i runga i te ritenga ako atu i etahi, i whakahaere ranei i ona ritenga, i te karakia me te whakatutuki i ona ahuatanga. Rarangi 19 Ko ia tangata e whai-tika ana ki tona whakaaro i kite ai, a ki te whakapuaki hoki i ona whakaaro; ko tenei tika e tapiritia mai ana te ahei ona ki te pupuri i tana i whakaaro ai, a kaua hoki a ia e whakararurarutia a ka ahei hoki a ia ki te rapu, ki te tango mai me te tuku atu i nga whakamarama me nga rapunga whakaaro o te hinengaro, ahakoa pewhea te huarahi, i nawhea atu ranei i runga i nga rohe whakatakoto a te tangata. Rarangi 20 1. Ko ia tangata e whai-tika ana kia watea ia ki te whakatu hui, ropu ranei, i runga i te ritenga pupuri i te maungarongo. 2. Kaua tetahi tangata e akina kia uru noa ki tetahi ropu. Rarangi 21 1. Ko ia te tangata e whai-tika ana kia whai wahi ki nga whakahaere kawanatanga o tona whenua, a ia tonu a ma roto ata ranei i nga mangai i ata waitohutia. 2. Ko ia tangata e whai-tika ana kia tauriterite te huarahi atu mo te uru ki nga mahi mo te katoa i tona nei whenua. 3. Ko ta te iwi i whakatau ai me meinga hei kaupapa mo te mana o te kawanatanga; ko tenei whakatau me meinga kia whakapuakina ia wa ia wa i runga i te pooti tika he mea atu tuku ki te katoa i runga i te mana pooti tauriterite o tetahi tangata, a me meinga hoki na te pooti puku a na tetahi tikanga ranei i rite atu ki tera te watea o te tangata ki te pooti ki tana i whakaaro ai. Rarangi 22 Ko ia tangata, i te mea no roto ia i te iwi, e whakatika ana kia whiwhi ki nga tikanga toko i te ora mo te iwi, a e tika ana kia whakaritea atu ki a ia, i na roto atu i nga whakahaere whanui a tona iwi, i na roto mai ranei i nga whakakotahitanga o nga mahi a nga iwi o te ao a kia rite hoki ki te kaha o nga whakahaere a-ropu o roto ia iwi ia iwi te oranga tinana te noho i roto i te iwi., me nga tika ki nga whakahaere hapai i te mauri o te tangata, e tika nei kaua e hapa te tangata i enei mea hei pupuri i tona ihi me te whakawatea hoki i te tupu pakari o te tu-rangatira o te tangata. Rarangi 23 1. Ko ia tangata e whai-tika ana kia whiwhi ki te mahi, kia watea hoki ki te whawha ki tana mahi i hiahia ai, kia tika kia pai hoki nga ahuatanga o tana mahi a kia tiakina hoki kei noho kare mahi ia. 2. Ko ia tangata, ahakoa pewhea, e tika ana kia whiwhi ki te utu taurite mo nga mahi i tauriterite nga ahuatanga. 3. Ko ia tangata e mahi ana e whai-tika ana kia utua ki te utu tika pai hoki, kia ahei ai te whiwhi ona me tana whanau ki te oranga e tika nei hei whakaara i te ua o te tangata i te ao nei; a me tapiri atu ki enei, ina kitea e tika ana kia pera, etahi atu awhina o roto i nga whakahaere toko i te ora mo te iwi. 4. Ko ia tangata e ahei ana ki te waihanga ki te whakauru ranei ki tetahi ropu kaimahi hei tiaki i nga ahuatanga katoa e pa ana ki nga huarahi mahi mona. Rarangi 24 Ko ia tangata e whai-tika ana kia okioki, noho noaiho ranei i ona wa ano; ka tapiritia ki tenei, me meinga ko ona haora mahi me whakawhaiti ano ki nga haora e tika ana, me whiwhi hoki ia wa ia wa ki nga ra kore mahi i runga i te ritenga haere tonu o te utu mona i ana ra. Rarangi 25 1. Ko ia tangata e whai-tika ana kia whiwhi ki te oranga e hangai ana ki te oranga totika mo tona tinana me ona ahuatanga katoa, ona ake me tana whanau; i te kai. i tekakahu, i te whare, i te rongoa me nga whakaora i nga mauiui o te tinana, a tae atu ki nga huarahi toko i te ora mo te iwi e tika ana, a me whiwhi hoki ai i te tika, me te manaakitanga tuturu mona ina tupono mai nga wa kore mahi, nga mauiui o te tinana, nga wharanga, te pouarutanga, te kaumatuatanga, a te kore ranei e whiwhi i te oranga mona i nga runga mai i etahi ahuatanga kaore nei e taea e ia te pewhea ake. 2. Ko nga wahine whanau me te hunga tamariki e tika ana kia ata whakaarohia kia manaakitia hoki. Ko te katoa o nga tamariki, ahakoa i whanau mai i te hunga i te marenatia kahore ranei i marenatia, me meinga kia rite tahi te whiwhi ki nga awhina o nga ritenga toko i te ora mo te iwi. Rarangi 26 1. Ko ia tangata e tika ana kia watea ki a ia nga huarahi o te matauranga. Ko nga huarahi o nga akoranga me noho kore utu, otira i nga timatanga atu me ona kaupapa tuturu. Ko nga timatanga atu o nga huarahi akoranga me meinga kia utaina ki te katoa, kaua ma te hiahia noa o te tangata. Ko nga akoranga mo nga mahi-a-ringa me nga mahi-a-hinengaro me meinga kia whanui te horo ki te katoa, a ko nga akoranga ki nga taumata ikeike o te matauranga me rite tahi te watea ki te katoa i na runga ano ra i te kitea o te totika o te tangata. 2. Ko nga huarahi o te akoranga me whakaanga atu ki nga wahi e puta topu mai ai nga hua totika o te tangata, kia meinga ai hoki hei kaupapa whakapakari i nga mana o te tangata me nga tino kaupapa o te rangatiratanga o tona oranga i tenei ao. Me meinga i tenei ritenga kia ata matatau tetahi ki tetahi, kia watea i te ngakau wene, kia noho hoki i runga i te whakahoahoa tetahi iwi ki tetahi iwi, tetahi momo tangata me tetahi momo tangata me tetahi ropu whakapono ki tetahi atu ropu whakapono, a me meinga hoki kia hapai i nga whakahaere a te Kotahitanga o nga iwi Nunui o te Ao he mea a mau ai te maungarongo ki te mata o te whenua. 3. Kei nga matua te mana tuatahi ki te tohu i te ahua o nga akoranga hei tuku ki a ratou tamariki. Rarangi 27 1. Ko ia tangata e whai-tika ana kia watea tona huarahi mo te whakauru atu ki nga whakahaere hapai i te hinengaro tangata i roto i te iwi, kia whiwhi hoki ki nga oranga ngakau o roto i nga mahi ataahua, a kia pa tahi atu ki nga whakahaere hohonu o te matauranga e ahu whakamua atu ana, me ona hua katoa. 2. Ko ia tangata e whai-tika ana kia ata tiakina kia puta ano ki a ia nga hua o nga mea oranga ngakau, oranga tinana ranei, i na roto mai nei i nga mahi o te hohohutanga o te matauranga, i nga pukapuka whai hua i tuhia, i nga mahi ataahua ranei nana ake nai i whakapuawai ki te ao. Rarangi 28 Ko ia tangata e whai-tika ana ki nga ritenga o te noho pai o te iwi me te ao katoa, ma reira nei e tino tuturu ai nga tika me nga rangatiratanga kua whakararangitia nei ki roto i tenei Whakapuakitanga. Rarangi 29 1. Ko ia tangata me mahi i ana mahi mo te iwi, ma reira anake nei e watea ai, e tutuki tuturu ai hoki te waihangatanga o tona hinengaro. 2. I te wa i whakarite ana ia i ona tika me ona rangatiratanga, ko ia tangata me meinga ko nga ara anake mona me ma roto i nga whakatau a te ture, me motuhake hoki aua arai hei mea a mau ai e mohiotia ai e whakanuia ai hoki nga tika me nga rangatiratanga e etahi atu tangata, a hei mea hoki e tutuki ai nga ritenga pono o te noho kore-hara, o te noho i te rangimarie o te katoa me te painga whanui ki nga iwi e noho ana i raro i te mana kawanatanga o te katoa o nga iwi o tetahi whenua. 3. Ko enei tika me enei rangatiratanga kaua rawa e meinga kia whakahaerea i runga i etahi tikanga me etahi atu kaupapa e peka ke ana i a te Kotahitanga o nga Iwi Nunui o te Ao i mea ai. Rarangi 30 Kahore rawa i roto i tenei Whakapuakitanga tetahi mea e ahei ana kia whakamoaritia tera kei tetahi Mana Kawanatanga, kei tetahi ropu, kei tetahi tangata ranei tetahi mana ki te whakahaere i tetahi ritenga, ki te mahi ranei i tetahi mahi e anga atu ana hei tikanga turaki i tetahi o nga mano me nga rangatiratanga e mau ake nei.

Cite this as: Anton Antonov, "Simplified Machine-Learning Workflow #8" from the Notebook Archive (2020), https://notebookarchive.org/2020-09-55st5h2

Latent Semantic Analysis (Part 3)

Session overview

Data

Last session’s example --“Eat your own dog food”

Full LSA workflow (over Raku documentation)

Raku?

Where from?

Natural language commands

Vector space model representation

Text data -- Shakespeare’s sonnets

Text data - big data questions

Matrix representation

Further operations for the sparse matrix with named rows and columns

Arithmetic

Transposing

Similarity matrix

Matrix-vector multiplication -- most similar documents

Additional functions

Pareto Principle adherence

In Hamlet (single document)

In the questions

UN Human Rights in different languages (single document)

Last session’s example --“Eat your own dog food”
