Educational Materials

Simplified Machine-Learning Workflow #3

Anton Antonov

Author

Anton Antonov

Title

Simplified Machine-Learning Workflow #3

Description

Semantic Analysis (Part 4)

Latent Semantic Analysis (Part 4)

A Wolfram livecoding session

Anton Antonov
January 2020

Session overview

Review: last session’s matrix object.

With named rows and columns.

Queries representation.

Representing rstudio-conf-2019 abstracts in the vector space of WTC-2019 abstracts.

Making a search engine for:

□ Raku’s documentation, or

■ WTC-2019 abstracts.

Dimension reduction over an image collection.

Topics over random mandalas.

Representation of unseen mandala images.

Data

Course data

In[]:=

ResourceFunction["ImportCSVToDataset"]["~/MathFiles/Presentations/Live-coding sessions Latent Semantic Analysis Worflows 2019-2020/Data-breakdown.csv"]

Out[]=

Dataset name	Collection size	Description	Why use it?	Source
"Hamlet" (play parts)	222 documents	The different parts in Shakespeare's play.	Small enough to digest.	GitHub
US State of Union speeches	252 documents	US presidents speeches.	Good English similar document length.	GitHub
Airbnb reviews	~525,000 documents	Airbnb reviews by customers.	A lot of relatively tight descriptions.	http://insideairbnb.com/get-the-data.html
Raku documentation	399 documents	Technical documentation of Raku (Perl 6).	Very instructive and easy to explain.	https://github.com/Raku/doc
Movie reviews	~10,000 documents	Movie reviews (with labels "positive"/"negative").	Good text data available in WL.	ExampleData[{"MachineLearning", "MovieReview"}]
"The Idiot" (chapters)	50 documents	Chapters of the novel by Dostoevsky in both English and Russian.	Show translation possibilities.	GitHub
Data science class questions	99 documents	Student questions from Data Science class teaching.	Eat your own dog food.	GitHub
rstudio::conf 2019 abstracts	61 records	Abstracts of the rstudio::conf 2019 conference	For comparison representation.	GitHub
WTC-2019 abstracts	182 records	Abstracts of the Wolfram Technology Conference 2019	Eat your own dog food.	GitHub

In[]:=

WebImage["https://github.com/antononcube/SimplifiedMachineLearningWorkflows-book/tree/master/Data"]

Out[]=

Read WTC-2019 abstracts

In[]:=

aWTCRecords=Import["https://raw.githubusercontent.com/antononcube/SimplifiedMachineLearningWorkflows-book/master/Data/Wolfram-Technology-Conference-2019-abstracts.json"];aWTCRecords=Association[Map[#〚1〛Association[#〚2〛]&,aWTCRecords]];Length[aWTCRecords]

Out[]=

182

Read rstudio-conf-2019 abstracts

In[]:=

aRSCRecords=Import["https://raw.githubusercontent.com/antononcube/SimplifiedMachineLearningWorkflows-book/master/Data/RStudio-conf-2019-abstracts.json"];aRSCRecords=Association[Map[#〚1〛Association[#〚2〛]&,aRSCRecords]];Length[aRSCRecords]

Out[]=

Image collection

Using WFR RandomMandala.

In[]:=

SeedRandom[16];ResourceFunction["RandomMandala"][]

Out[]=

In[]:=

WebImage["https://resources.wolframcloud.com/FunctionRepository/resources/RandomMandala"]

Out[]=

Review

In[]:=

aWTCDesriptions=Map[#description&,aWTCRecords];

In[]:=

aWTCDesriptions//Length

Out[]=

182

In[]:=

lsaObjWTC=LSAMonUnit[aWTCDesriptions]⟹LSAMonMakeDocumentTermMatrix[{},Automatic]⟹LSAMonApplyTermWeightFunctions["IDF","TermFrequency","Cosine"]⟹LSAMonExtractTopics["NumberOfTopics"24,"MinNumberOfDocumentsPerTerm"2,Method"NNMF",MaxSteps20]⟹LSAMonEchoTopicsTable["NumberOfTableColumns"8];

topics table:

1.000	cloud
0.991	web
0.952	applications
0.589	system
0.482	python
0.451	integrate
0.425	private
0.412	build
0.402	developers
0.366	run
0.339	basics
0.327	existing

1.000	past
1.000	alumni
0.972	summer
0.875	student
0.683	students
0.567	present
0.500	positions/programs
0.500	interns
0.500	camp
0.500	ambassadors
0.465	opportunity
0.464	hear

1.000	blockchain
0.290	ledger
0.241	distributed
0.205	new
0.200	introductory
0.186	discover
0.185	oracle
0.183	services
0.174	projects
0.173	course
0.173	complete
0.173	interaction

1.000	educators
0.610	classroom
0.571	sylva
0.565	platform
0.549	questions
0.500	learning
0.469	grading
0.461	face
0.459	using
0.459	environment
0.451	assessments
0.437	auto

1.000	store
1.000	snacks
1.000	fellow
1.000	demonstrations
0.990	visit
0.949	beverages
0.872	attendees
0.869	enjoy
0.680	join
0.647	network
0.163	wolfram
0.043	projects

1.000	speech
0.488	analysis
0.415	video
0.322	audio
0.301	briefly
0.300	additions
0.294	understanding
0.271	built
0.235	processing
0.219	talk
0.218	networks
0.214	objects

1.000	round
1.000	lunch
0.880	table
0.794	networking
0.688	discussion
0.675	start
0.596	join
0.015	project
0.014	programming
0.013	code
0.012	educational
0.010	sessions

1.000	keynote
0.939	stephen
0.717	address
0.690	opening
0.188	wolfram’s
0.188	welcome
0.180	please
0.179	beverages
0.163	conference
0.123	join
0.114	wolfram
0.053	databases

1.000	neural
0.505	net
0.463	nets
0.341	learning
0.284	art
0.278	machine
0.272	tasks
0.271	scratch
0.245	applications
0.231	repository
0.228	train
0.216	state

1.000	developments
0.973	functions
0.688	special
0.665	calculus
0.659	documentation
0.506	related
0.500	heun
0.464	area
0.371	classroom
0.368	include
0.363	mathematical
0.353	new

1.000	asymptotic
0.962	art
0.665	compute
0.654	advances
0.618	gain
0.608	discover
0.589	theory
0.562	key
0.553	new
0.551	practices
0.522	latest
0.498	algorithms

1.000	cloud
0.839	mobile
0.724	introduction
0.720	apps
0.670	setting
0.597	player
0.596	cases
0.591	educational
0.580	wolfram\|alpha
0.574	learn
0.388	notebooks
0.280	use

1.000	financial
0.636	data
0.422	sources
0.375	analysis
0.339	finance
0.337	performance
0.286	access
0.259	knowledge
0.240	market
0.232	apis
0.232	providers
0.231	entityvalue

1.000	wolfram\|alpha
0.841	edition
0.651	notebook
0.386	hands
0.330	kelvin
0.286	book
0.234	combines
0.229	perfect
0.220	program
0.219	unified
0.214	start
0.209	best

1.000	toolkit
0.704	process
0.692	images
0.630	image
0.559	counting
0.539	processes
0.508	decision
0.500	person
0.478	processing
0.463	powerful
0.458	input
0.406	objects

1.000	clear=
1.000	<br
0.676	none
0.481	convex
0.433	solvers
0.431	optimization
0.278	30pm
0.276	00pm
0.236	integer
0.230	mixed
0.217	mathematica
0.154	educational

1.000	graph
0.954	peek
0.954	familiarize
0.887	sneak
0.835	functionalities
0.792	powerful
0.792	versions
0.756	access
0.754	geometric
0.751	provides
0.689	computer
0.674	easy

1.000	package
0.853	game
0.745	pap
0.681	planning
0.336	2015
0.333	analytics
0.319	development
0.317	constants
0.280	briefly
0.279	engine
0.278	analyze
0.275	video

1.000	numerics
0.909	discussed
0.802	mathematica
0.777	evaluation
0.747	various
0.733	functions
0.693	standard
0.614	important
0.603	advanced
0.592	new
0.588	properties
0.573	certain

1.000	finding
0.986	time
0.864	series
0.475	algorithms
0.463	reporting
0.406	anomaly
0.399	tool
0.381	data
0.359	topic
0.354	actual
0.348	presentation
0.332	look

1.000	models
0.489	physical
0.391	model
0.345	operations
0.323	transfer
0.319	proposed
0.313	fluid
0.312	based
0.307	image
0.307	various
0.304	nonlinear
0.300	extract

1.000	librarylink
1.000	c++
0.488	degree
0.419	set
0.404	experiences
0.395	good
0.391	started
0.353	workflows
0.345	university
0.331	project
0.321	familiarity
0.311	framework

1.000	spatial
0.917	point
0.354	patterns
0.292	modeling
0.257	data
0.241	sightings
0.241	refers
0.241	earthquakes
0.241	deposits
0.241	crime
0.241	animal
0.224	gold

1.000	computational
0.943	thinking
0.716	humanities
0.459	course
0.447	curriculum
0.378	designed
0.298	culture
0.296	experiment
0.274	sessions
0.267	design
0.255	topics
0.240	discuss

Sparse matrix with named rows and columns

In[]:=

smat=lsaObjWTC⟹LSAMonTakeDocumentTermMatrix

Out[]=

SparseArray

Specified elements: 7662

Dimensions: {182,3442}



In[]:=

MatrixForm[smat〚1;;12,1000;;1050〛]

Out[]//MatrixForm=

	edit	edition	education	educational	educator	educators	educause	edx	eeg	effect	effective	effectively	effect”	efficacy	efficiency	efficient	efficiently	effort	efforts	ehr”	eigenvalue	eigenvalues	electric	electrical	electrodynamics	electroencephalogram	electromagnetic	electromagnetism	electronic	electronics	electrons	elegance	element	elementary	elements	elliptic	elusive	email	embed	embeddedsql”	embedding	embodied	emerge	emergence	emergency	emergent	emerging	emily	emissions	emitted	emitting
Karl.Isensee.Oct.28.9:00.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Jayanta.Phadikar.Oct.28.9:00.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Tuseeta.Banerjee.Oct.28.1:00.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Jan.Poeschko.and.Joel.Klein.Oct.28.1:00.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Andrew.Steinacher.Sylvia.Haas.Keiko.Hirayama..Jason.Biggs.and.Luke.Titus.Oct.28.2:00.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Oct.28.3:00.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Oct.28.4:30.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Stephen.Wolfram.Oct.28.6:00.PM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Oct.29.7:30.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Russell.Foltz-Smith.Oct.29.8:30.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0
Gerli.Jõgeva.Oct.29.8:30.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Claudio.Parazzoli.Oct.29.8:30.AM	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

In[]:=

MatrixPlot[smat]

Out[]=

In[]:=

RowSumsAssociation[smat]

In[]:=

TakeLargest[ColumnSumsAssociation[smat],12]

Out[]=

wolfram183,language105,data100,talk69,new62,mathematica60,functions57,using53,use47,analysis46,learning40,none38

In[]:=

TakeLargest[ColumnSumsAssociation[lsaObjWTC⟹LSAMonTakeWeightedDocumentTermMatrix],12]

Out[]=

none15.1707,data4.21456,wolfram4.16885,new3.70651,language3.48292,functions3.42412,mathematica2.85901,talk2.8295,enjoy2.79883,networking2.71834,join2.65108,using2.53217

Queries using matrix-vector multiplications

In[]:=

ind=ColumnNamesAssociation[smat]["anomalies"]

Out[]=

171

In[]:=

svec=SparseArray[ConstantArray[0,ColumnsCount[smat]]];svec〚ind〛=1;

In[]:=

TakeLargest[RowSumsAssociation[smat.Transpose[{svec}]],3]

Out[]=

Anton.Antonov.Oct.29.1:30.PM3,Tuseeta.Banerjee.Oct.28.1:00.PM0,Jayanta.Phadikar.Oct.28.9:00.AM0

In[]:=

aWTCDesriptions[Keys[%]〚1〛]

Out[]=

In this presentation we show, explain and compare methods for finding anomalies, breaks and outliers in time series. We are interested in finding anomalies in both a single time series and a collection of time series. We (mostly) employ nonparametric methods. First, we look at some motivational examples from well-known datasets. Then we look into definitions of anomalies and definitions for measuring the success of time series anomaly detection. For a single time series, we apply both Wolfram Language built-in algorithms and additional specialized algorithms. We discuss in more detail algorithms based on <i>k</i> -nearest neighbors (KNN), dimension reduction, linear regression, quantile regression and prefix trees. For collections of time series, we discuss: transformations into uniform representations, simple outlier finding based on variable distributions, anomalous trend finding, anomaly finding with KNN and other related algorithms. We are going to discuss how anomaly finding helps in producing faithful simulations of multivariable datasets. Concrete, real-life time series are used in the examples.

In[]:=

MatrixPlot[SparseArray[lsaObjWTC⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeW]]

Out[]=

In[]:=

lsAnomalInds=FindAnomalies[SparseArray[lsaObjWTC⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeW],"AnomalyPositions"]

Out[]=

{4,32,33,45,73,93,105,109,119,134,140,174,180}

In[]:=

aWTCDesriptions〚lsAnomalInds〛

Queries representation

Let us see how rstudio-conf-2019 abstracts get represented into the space of WTC-2019 abstracts.

Focus texts (rstudio-conf-2019)

In[]:=

txt1=aRSCRecords〚12〛["Abstract"]

Out[]=

RStudio Package Manager is the newest professional product that helps teams, departments, and entire enterprises organize and centralize package management. If you’ve ever struggled with IT to get access to a new (any?) R package, reproduce an old result, or share your code with others, RStudio Package Manager can help! We’ll introduce the new product, discuss how R repositories can be used to solve problems and take a sneak peek at what is coming in 2019.

In[]:=

txt2=aRSCRecords〚56〛["Abstract"]

Out[]=

My talk will discuss how R, the tidyverse, and the community around R helped me to learn to code and create my first R package. My positive experiences with the resources for learning R and the community itself led me to create a blog detailing my experiences with R as a way to pass along the knowledge that I gained. The next step was to develop my first package. The debkeepr package integrates non-decimal monetary systems of pounds, shillings, and pence into R, making it possible to accurately analyze and visualize historical account books. It is my hope that debkeepr can help bring to light crucial and interesting social interactions that are buried in economic manuscripts, making these stories accessible to a wider audience.

Terms representation

In[]:=

matTerms1=lsaObjWTC⟹LSAMonRepresentByTerms[ToLowerCase@txt1]⟹LSAMonEchoFunctionValue[MatrixForm[#1〚All,Keys[Select[ColumnSumsAssociation[#1],#1>0&]]〛]&]⟹LSAMonTakeValue

	2019	access	code	coming	departments	discuss	entire	help	helps	introduce	management	new	package	peek	problems	product	result	share	sneak	solve	used	we’ll
1	0.168908	0.153097	0.120526	0.211964	0.211964	0.106177	0.168908	0.141288	0.192912	0.136337	0.168908	0.142391	0.58729	0.192912	0.120526	0.358787	0.192912	0.168908	0.179393	0.160341	0.0932808	0.146823

Out[]=

SparseArray

Specified elements: 22

Dimensions: {1,3442}



In[]:=

matTerms2=lsaObjWTC⟹LSAMonRepresentByTerms[ToLowerCase@txt2]⟹LSAMonEchoFunctionValue[MatrixForm[#1〚All,Keys[Select[ColumnSumsAssociation[#1],#1>0&]]〛]&]⟹LSAMonTakeValue

	accessible	account	analyze	audience	blog	bring	code	community	create	crucial	detailing	develop	discuss	economic	experiences	gained	help	helped	interactions	interesting	knowledge	learn	learning	led	light	making	non	package	possible	resources	social	step	stories	systems	talk	way	wider
1	0.153681	0.153681	0.116965	0.127734	0.168859	0.142912	0.0960161	0.307362	0.192032	0.134559	0.168859	0.127734	0.0845845	0.142912	0.307362	0.168859	0.112556	0.168859	0.153681	0.121963	0.112556	0.0987903	0.0700689	0.168859	0.153681	0.269118	0.142912	0.350894	0.116965	0.142912	0.121963	0.134559	0.168859	0.0910175	0.0434591	0.101787	0.168859

Out[]=

SparseArray

Specified elements: 37

Dimensions: {1,3442}



In[]:=

Norm@*SparseArray/@{matTerms1,matTerms2}

Out[]=

{1.,1.}

In[]:=

MatrixForm[matTerms1.Transpose[matTerms2]]

Out[]//MatrixForm=



	1
1	0.242533



Topics representation

In[]:=

matTopics1=lsaObjWTC⟹LSAMonRepresentByTopics[ToLowerCase@txt1]⟹LSAMonEchoFunctionValue[MatrixForm[#1〚All,Keys[Select[ColumnSumsAssociation[#1],#1>0&]]〛]&]⟹LSAMonTakeValue;matTopics1=WeightTermsOfSSparseMatrix[matTopics1,"None","None","Cosine"]

cloud-web-applications

graph-peek-familiarize

developments-functions-special

package-game-pap

blockchain-ledger-distributed

asymptotic-art-compute

cloud-mobile-introduction

finding-time-series

financial-data-sources

models-physical-model

wolfram|alpha-edition-notebook

clear=-<br-none

0.00730076

0.136995

0.0100969

0.255452

0.00428355

0.00552975

0.0118622

0.0289472

0.00411564

0.00318786

0.0323035

0.0131236

Out[]=

SparseArray

Specified elements: 24

Dimensions: {1,24}



In[]:=

TakeLargest[ColumnSumsAssociation[matTopics1],UpTo[6]]

Out[]=

package-game-pap0.853837,graph-peek-familiarize0.457898,wolfram|alpha-edition-notebook0.107973,finding-time-series0.0967548,clear=-<br-none0.0438652,cloud-mobile-introduction0.0396488

In[]:=

matTopics2=lsaObjWTC⟹LSAMonRepresentByTopics[ToLowerCase@txt2]⟹LSAMonEchoFunctionValue[MatrixForm[#1〚All,Keys[Select[ColumnSumsAssociation[#1],#1>0&]]〛]&]⟹LSAMonTakeValue;matTopics2=WeightTermsOfSSparseMatrix[matTopics2,"None","None","Cosine"]

neural-net-nets

developments-functions-special

package-game-pap

blockchain-ledger-distributed

asymptotic-art-compute

educators-classroom-sylva

cloud-mobile-introduction

finding-time-series

models-physical-model

librarylink-c++-degree

spatial-point-patterns

computational-thinking-humanities

0.00501161

0.0116588

0.168331

0.00462197

0.00521703

0.0191225

0.0304247

0.00702409

0.0191461

0.0453552

0.0000467977

0.0524092

Out[]=

SparseArray

Specified elements: 24

Dimensions: {1,24}



In[]:=

TakeLargest[ColumnSumsAssociation[matTopics2],UpTo[6]]

Out[]=

package-game-pap0.887771,computational-thinking-humanities0.276405,librarylink-c++-degree0.239202,cloud-mobile-introduction0.160459,models-physical-model0.100976,educators-classroom-sylva0.100851

In[]:=

Norm@*SparseArray/@{matTopics1,matTopics2}

Out[]=

{1.,1.}

In[]:=

MatrixForm[matTopics1.Transpose[matTopics2]]

Out[]//MatrixForm=



	1
1	0.747378



Search engine

Recommender object

In[]:=

matW=lsaObjWTC⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeW;

In[]:=

smrObjWTC=SMRMonUnit[]⟹SMRMonCreate[<|"Terms"lsaObjWTC⟹LSAMonTakeDocumentTermMatrix,"Topics"matW|>]⟹SMRMonApplyNormalizationFunction["Cosine"]⟹SMRMonEchoFunctionContext[#matrices&];

TermsSparseArray

Specified elements: 7662

Dimensions: {182,3442}

,TopicsSparseArray

Specified elements: 1083

Dimensions: {182,24}



Recommendations by terms

In[]:=

prof=Select[ColumnSumsAssociation[matTerms1],#>0&]

Out[]=

20190.168908,access0.153097,code0.120526,coming0.211964,departments0.211964,discuss0.106177,entire0.168908,help0.141288,helps0.192912,introduce0.136337,management0.168908,new0.142391,package0.58729,peek0.192912,problems0.120526,product0.358787,result0.192912,share0.168908,sneak0.179393,solve0.160341,used0.0932808,we’ll0.146823

In[]:=

smrObjWTC⟹SMRMonRecommendByProfile[prof,12]⟹SMRMonJoinAcross[aWTCDesriptions]⟹SMRMonEchoValue⟹SMRMonProveByMetadata[prof,{"Claudio.Parazzoli.Oct.29.8:30.AM"}]⟹SMRMonEchoValue;

value:

Score	Index	Item	Value
1.0	60	Andrea.Griffin.Oct.29.3:30.PM	This talk will introduce a publicly available course detailing a Wolfram Languag ⋱
0.81907	12	Claudio.Parazzoli.Oct.29.8:30.AM	The rapid evolution of quantum technology has not been met by an equivalent matu ⋱
0.702949	51	Eric.Jacopin.Oct.29.2:30.PM	I describe the (Game AI) Planning Analytics Package (PAP), which I have been dev ⋱
0.416429	105	Jaebum.Jung.Oct.30.1:30.PM	The Wolfram Language provides easy access to powerful graph functionalities. Fam ⋱
0.394496	109	Charles.Pooh.Oct.30.2:00.PM	The Wolfram Language provides easy access to powerful computational geometric fu ⋱
0.390659	178	Peter.Jacobse.PhD.Oct.31.4:30.PM	Scanning tunneling and atomic force microscopy are methods that have the power t ⋱
0.385013	2	Jayanta.Phadikar.Oct.28.9:00.AM	Learn the skills, knowledge and tools to develop your own project with the Wolfr ⋱
0.34177	37	Akira.Toma.and.Piero.Sanchez.Oct.29.12:30.PM	Discover the new additions to the blockchain functionalities in the Wolfram Lang ⋱
0.297823	49	Yuzhu.Lu.Oct.29.2:00.PM	This talk will present some new features and optimizations of Mathematica 2D and ⋱
0.286957	95	Adam.Strzebonski.Oct.30.11:30.AM	: I will present new Mathematica functionality for computing properties of math ⋱
0.281736	45	Jan.Poeschko.Oct.29.1:30.PM	After a general introduction to the Wolfram Cloud, we’ll talk about what’s new i ⋱
0.268558	177	Brenton.Bostick.Oct.31.4:30.PM	A new linting tool for finding bugs in the Wolfram Language is presented, as wel ⋱

value:Claudio.Parazzoli.Oct.29.8:30.AMpackage1.,solve0.273018,we’ll0.25,help0.240576,problems0.205225,used0.158833

In[]:=

txt1

Out[]=

Recommendations by topics

In[]:=

prof

Out[]=

In[]:=

prof2=Select[ColumnSumsAssociation[matTopics1],#>0&]

Out[]=

cloud-web-applications0.0244025,graph-peek-familiarize0.457898,developments-functions-special0.0337485,package-game-pap0.853837,blockchain-ledger-distributed0.0143176,asymptotic-art-compute0.0184829,cloud-mobile-introduction0.0396488,finding-time-series0.0967548,financial-data-sources0.0137563,models-physical-model0.0106553,wolfram|alpha-edition-notebook0.107973,clear=-<br-none0.0438652

In[]:=

smrObjWTC⟹SMRMonRecommendByProfile[prof2,12]⟹SMRMonJoinAcross[aWTCDesriptions]⟹SMRMonEchoValue⟹SMRMonProveByMetadata[prof2,"Eric.Jacopin.Oct.29.2:30.PM"]⟹SMRMonEchoValue⟹SMRMonProveByMetadata[prof2,"Jayanta.Phadikar.Oct.28.9:00.AM"]⟹SMRMonEchoValue;

value:

Score	Index	Item	Value
1.0	51	Eric.Jacopin.Oct.29.2:30.PM	I describe the (Game AI) Planning Analytics Package (PAP), which I have been dev ⋱
0.947004	2	Jayanta.Phadikar.Oct.28.9:00.AM	Learn the skills, knowledge and tools to develop your own project with the Wolfr ⋱
0.945975	114	Jane.Shen-Gunther.MD.Oct.30.2:30.PM	HPV is now recognized as the carcinogen of almost all invasive cervical cancers ⋱
0.933925	60	Andrea.Griffin.Oct.29.3:30.PM	This talk will introduce a publicly available course detailing a Wolfram Languag ⋱
0.926179	14	Michael.Ulrey.Oct.29.9:00.AM	This presentation is an update of last year’s presentation at this conference, c ⋱
0.922158	178	Peter.Jacobse.PhD.Oct.31.4:30.PM	Scanning tunneling and atomic force microscopy are methods that have the power t ⋱
0.90349	12	Claudio.Parazzoli.Oct.29.8:30.AM	The rapid evolution of quantum technology has not been met by an equivalent matu ⋱
0.854769	75	Dan.McDonald.PhD.Ian.Ford.Lin.Cong.and.Peter.Barendse..Ph.D.Oct.30.9:00.AM	We present updates to the automated geometric functionality of the Wolfram Langu ⋱
0.739902	181	Jay.Alan.Jackson.Oct.31.5:00.PM	This presentation will show how a combination of Mathematica, UnityLink and the ⋱
0.707896	58	George.Woodrow.III.Oct.29.3:30.PM	Fractal landscapes were first demonstrated in 1980 by Loren Carpenter, who worke ⋱
0.707081	66	Diego.Zviovich.and.Pedro.Fonseca.Oct.29.5:00.PM	Escape rooms have gained a lot of popularity in these last 10 years. Escape boxe ⋱
0.682351	49	Yuzhu.Lu.Oct.29.2:00.PM	This talk will present some new features and optimizations of Mathematica 2D and ⋱

value:Eric.Jacopin.Oct.29.2:30.PMpackage-game-pap1.,graph-peek-familiarize0.536283,finding-time-series0.113318,blockchain-ledger-distributed0.0167685

value:Jayanta.Phadikar.Oct.28.9:00.AMpackage-game-pap1.,cloud-web-applications0.0285797,asymptotic-art-compute0.0216469,models-physical-model0.0124793

In[]:=

txt1

Out[]=

In[]:=

aWTCDesriptions["Jayanta.Phadikar.Oct.28.9:00.AM"]

Out[]=

Learn the skills, knowledge and tools to develop your own project with the Wolfram Language. Start with fundamental programming concepts, including iterations, nesting, pattern matching and function definitions. Then move on to more advanced techniques for package development and deployment. Along the way learn tips and tricks for good programming practices, debugging and writing efficient code in the Wolfram Language.

Random mandala images topics

Image collection

In[]:=

SeedRandom[12];ResourceFunction["RandomMandala"]["RotationalSymmetryOrder"6,"ConnectingFunction"FilledCurve@*BezierCurve]

Out[]=

In[]:=

SeedRandom[12];ResourceFunction["RandomMandala"]["RotationalSymmetryOrder"3,"ConnectingFunction"FilledCurve@*BezierCurve]

Out[]=

In[]:=

k=200;SeedRandom[14];imgs=MapThread[(ResourceFunction["RandomMandala"]["RotationalSymmetryOrder"6,"ConnectingFunction"FilledCurve@*BezierCurve])&,{RandomChoice[{18,17,4243,1,113,4818},k],RandomChoice[{6,3},k]}];

In[]:=

Magnify[Multicolumn[imgs,20],0.4]

Out[]=

LSA preparation

In[]:=

AbsoluteTiming[imgs2=ImageResize[#,{100,100}]&/@imgs;]

Out[]=

{13.74,Null}

LSA extract topics

In[]:=

RandomSample[imgs2,6]

Out[]=





In[]:=

imgVecs=Flatten[ImageData[Binarize@ColorNegate@ColorConvert[#,"Grayscale"]]]&/@imgs2;

In[]:=

MatrixPlot[imgVecs,MaxPlotPoints800]

Out[]=

In[]:=

Magnify[ColumnForm[ListLinePlot[#,PlotRangeAll,AspectRatio1/6,ImageSizeLarge]&/@RandomSample[imgVecs,4]],0.7]

In[]:=

AbsoluteTiming[lsaObj=LSAMonUnit[]⟹LSAMonSetDocumentTermMatrix[SparseArray[imgVecs]]⟹LSAMonApplyTermWeightFunctions["None","None","Cosine"]⟹LSAMonExtractTopics[40,Method"NNMF","MaxSteps"12,"MinNumberOfDocumentsPerTerm"0]⟹LSAMonNormalizeMatrixProduct[NormalizedLeft];]

Out[]=

{59.7539,Null}

In[]:=

ListPlot[Norm/@SparseArray[lsaObj⟹LSAMonTakeH],FillingAxis,PlotRangeAll,PlotTheme"Scientific"]

Out[]=

In[]:=

lsaObj⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonEchoFunctionContext[ImageAdjust[Image[Partition[#,ImageDimensions[imgs2〚1〛]〚1〛]]]&/@SparseArray[#H]&];





Basis extraction

In[]:=

H=SparseArray[lsaObj⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeH]

Out[]=

SparseArray



Specified elements: 168260

Dimensions: {40,10000}

Data not saved. Save now



In[]:=

lsBasisImgs=ImageAdjust[Image[Partition[#,ImageDimensions[imgs2〚1〛]〚1〛]]]&/@SparseArray[H]

Out[]=





In[]:=

Binarize/@lsBasisImgs

Out[]=





In[]:=

Colorize[#,ColorFunction"Rainbow"]&/@RandomSample[lsBasisImgs,4]

Out[]=





In[]:=

ColorNegate/@Table[ImageAdjust[Blend[#&/@RandomSample[lsBasisImgs,4]]],12]

Out[]=





Approximation

Pick a known mandala image:

In[]:=

ind=RandomChoice[Range[Length[imgs2]]]imgTest=imgs2〚ind〛

Out[]=

191

Out[]=

Or generate a new random mandala:

In[]:=

testMandala=ResourceFunction["RandomMandala"]["ConnectingFunction"FilledCurve@*BezierCurve];imgTest=ImageResize[Image[testMandala],{100,100}]

Out[]=

In[]:=

imgTestVec=Flatten[ImageData[ColorNegate[ColorConvert[ImageResize[imgTest,ImageDimensions[imgs2〚1〛]],"Grayscale"]]]];Length[imgTestVec]

Out[]=

10000

In[]:=

matTest=ToSSparseMatrix[SparseArray[{imgTestVec}],"RowNames"{"TestImage"},"ColumnNames"Map[ToString,Range[Length[imgTestVec]]]];

In[]:=

matReprsentation=lsaObj⟹LSAMonRepresentByTopics[matTest]⟹LSAMonTakeValue;

In[]:=

lsCoeff=Normal@SparseArray[matReprsentation〚1,All〛];ListPlot[lsCoeff,FillingAxis,PlotRangeAll]

Out[]=

In[]:=

H=SparseArray[lsaObj⟹LSAMonNormalizeMatrixProduct[NormalizedRight]⟹LSAMonTakeH];

In[]:=

vecReprsentation=lsCoeff.H;

In[]:=

ImageAdjust@ColorNegate@Image[Clip[Partition[vecReprsentation,ImageDimensions[imgs2〚1〛]〚1〛],{0,1}]]

Out[]=

Blending of basis images

In[]:=

AbsoluteTiming[directBlendingImages=Table[RemoveBackground@ImageAdjust[Blend[Colorize[#,ColorFunctionRandomChoice[{"IslandColors","FruitPunchColors","AvocadoColors","Rainbow"}]]&/@RandomChoice[lsBasisImgs,4],RandomReal[1,4]]],12];]ImageCollage[directBlendingImages,BackgroundWhite,ImagePadding3,ImageSize600]

Out[]=

{0.678834,Null}

Out[]=

Cite this as: Anton Antonov, "Simplified Machine-Learning Workflow #3" from the Notebook Archive (2020), https://notebookarchive.org/2020-09-55sv02t