Accessing the World with the Wolfram Language: External Identifiers and Wikidata

Wikidata



Out[]=



73.48



Where is this data coming from? Who provides it? I’ll start at the beginning with a review of the history of Wikipedia, which led to the creation of a new sister project: Wikidata. Then I’ll demonstrate how the Wolfram Language facilitates access to this large and diverse data repository.

Quick History of Wikimedia Wikipedia—the free encyclopedia that anyone can edit—was created 20 years ago. Some advantages over traditional encyclopedias (think: heavy paper books) include hyperlinks within the text to other articles and links to the same article in a different language, images, infoboxes and so on.

Around that time, the semantic web was already in the works, with the first recommendation standardizing its data model, the Resource Description Framework (RDF), being published in 2004. The semantic web aims to apply the successful concept of hyperlinks to (machine-readable) data, creating so-called “linked data.” By standardizing the underlying data model, the data can be moved freely between systems. By linking the data—achieved by representing concepts as URLs (more technically, IRIs)—datasets can be combined easily.

About 10 years ago, Wikipedia reached the status of the “go-to” resource for anyone wishing to learn about a subject and to find references for further research. Articles and links between those articles are available in hundreds of languages. However, keeping the information available in infoboxes, which was essentially repeated across language editions, in sync was tedious. Even renaming a single article meant updating potentially hundreds of links from other language editions. The valuable data present in the articles was not readily machine readable, making queries like “give me all neurotransmitters encoded by such-and-such a protein” hard to answer.

Wikimedia, the foundation behind Wikipedia, presented a new sister project about seven years ago: Wikidata, a multilingual website to host “items”—that is, identifiers starting with the letter Q, followed by an integer.

Wikidata Development The first item ever created was Q1, representing the universe. Shortly after followed Q2, representing Earth, and so on. In the first development phase of Wikidata, an item was created for each existing Wikipedia page, in the order of some popularity measure.

Each item contains links to Wikipedia pages about the concept identified by the item, one per language. This facilitated managing the language links because changing the title of an article now required only one change in a central location. An item also contains for each language a label, a short description and alternative labels.

The second phase introduced the concept of properties: identifiers starting with the letter P followed by an integer. With properties, one can make statements, like (Q2, P31, Q3504248). For our English-speaking readers, this can be presented as (“Earth”, “instance of”, “inner planet”) using the English labels attached to the respective items and properties.

So now we have a website that stores machine-readable, multilingual data. It provides interfaces (APIs) that allow retrieving that data, one item at a time. While that is immensely useful on its own (and a clear improvement over having to extract data from a Wikipedia page), the real power becomes apparent with the Wikidata query service: it is an RDF (think: semantic web) database containing all the information about all the items.

SPARQL is the query language for RDF, just like SQL is the query language for relational databases. SPARQL can be used to query the Wikidata query service. In a previous blog, I explained the basics of SPARQL as well as the Wolfram Language’s symbolic representation of SPARQL, which facilitates writing programs that construct queries on the fly.

But you don’t have to read that (somewhat technical) blog post, nor do you have to learn SPARQL. Version 12.1 uses that same technology to build a function that is very easy to use (think: as easy as using

Entity

), from simple data retrieval and presentation to queries involving conditions.

Basics Before giving a more theoretical explanation, let’s start with some examples.

WikidataData

is the function to access data stored in Wikidata. For example, here is the mass of the Moon, shown previously:

In[]:=

WikidataData

Wikidata

Wikidata



Out[]=



73.48



The items to feed into WikidataData can be discovered using

WikidataSearch

In[]:=

WikidataSearch["moon"]

Out[]=



Wikidata

Wikidata

Wikidata

Wikidata

Wikidata

Wikidata

Wikidata

Christopher Robin Milne



Similarly, you can specify to look for properties:

In[]:=

WikidataSearch["Property""mass"]

Out[]=



Wikidata

Wikidata

MassBank accession ID

Wikidata

Wikidata

Wikidata

Wikidata

Wikidata



External Identifiers Now what are those bluish boxes? Let’s take a step back and look at entities first.

Identifier Systems

The following expression stands for the concept “Moon”:

In[]:=

Entity["PlanetaryMoon","Moon"]

Out[]=

Moon

When designing the entity framework, we could have chosen to represent the concept of the Moon simply as Entity[1234...]—that is, an arbitrary number (or string) that identifies the concept within the Wolfram ontology. However, we chose a two-component identifier consisting of a type (an “entity type”) and a name (a “canonical name”) with the requirement that a name is unique for a given type.

For the purpose of identifying a concept, the structure of an identifier does not matter. Advantages of partitioning the space of identifiers by type include, for instance, being able to list the properties applicable to a given entity.

The Wolfram ontology is a particular identifier system. Examples of other identifier systems include ISBN, DOI, ISO639-2 code (“language code”), LoC control number, MusicBrainz artist ID and so on. It is up to each identifier system to standardize the format of the identifier (from now on, ID), define their referent (that to which an ID refers) and mint new IDs, or nominate organizations to do so and to provide lookup services.

An ID alone, say 123, is meaningless, because a number of identifier systems use integers as IDs (OSM relation ID, PubChem SID and CID, Entrez Gene ID, etc.) and therefore refer to completely unrelated concepts.

The new symbol to represent an ID within an identifier system is

ExternalIdentifier

Represent the ID Q1 within the Wikidata identifier system:

In[]:=

ExternalIdentifier["WikidataID","Q1"]

Out[]=

Wikidata

Data Providers

Entities have a compact syntax for retrieving data:

In[]:=

Moon	PLANETARY MOON



mass



Out[]=

7.3459×

... which is a short form for:

In[]:=

EntityValue

Moon	PLANETARY MOON

mass

===%

Out[]=

True

This raises the question of why the following is not supported:

In[]:=

Wikidata



Wikidata



Out[]=

Wikidata



Wikidata



This is because the organization that defines an identifier system does not need to be the same as the one that hosts the data. In the case of Entity, Wolfram is responsible for both. But for, say, an ISBN, there is no canonical organization responsible for recording all the ISBNs that have been issued. That implies that for working with external identifiers, selecting a service (“choosing a lookup function”) is an explicit step.

In the case of ISBN, we got lucky because Wikidata does store ISBNs corresponding to Wikidata IDs. This mapping is used to make the following possible:

In[]:=

WikidataData

ISBN-10

1-57955-008-8

Wikidata

author



Out[]=



Wikidata

Stephen Wolfram



In fact, of the around eight thousand properties (and growing) that are available in Wikidata, about half are of type “external ID”—that is, mappings into other identifier systems.

Metadata

As there is no canonical service associated with certain identifier systems, one cannot rely on illustrative labels to become available when needed. But one might still want to present some human-readable information (instead of just the ID) in certain situations.

ExternalIdentifier allows embedding arbitrary metadata:

In[]:=

ExternalIdentifier["type","abc",<|"notes""some notes"|>]

Out[]=

type

abc

This metadata does not change the referent. The most important piece of metadata is the "Label", which is used for display:

In[]:=

ExternalIdentifier["WikidataID","Q2",<|"Label""Earth"|>]

Out[]=

Wikidata

URLs

For certain identifier types, a URL can be constructed. For such types, you can click inside the blue box (on the ID or label) to go to the website that describes the referent. The URL can be accessed like this:

In[]:=

Wikidata

["URL"]

Out[]=

URL

http://www.wikidata.org/entity/Q2



There is also a special kind of URL, the "ConceptURI", which identifies the concept within the semantic web (the same in this case):

In[]:=

Wikidata

["ConceptURI"]

Out[]=

URL

http://www.wikidata.org/entity/Q2



Such concept URIs are relevant when querying SPARQL endpoints.

Datatypes The Wolfram Language allows for the representation of a variety of basic as well as complex “values.” Complex values are presented to the user in an easily recognizable style. A large set of functions for plotting, creating maps and timelines, arithmetic operations, sorting and so on readily supports those datatypes.

Wikidata also supports a variety of datatypes. Those are translated to corresponding Wolfram Language expressions by WikidataData, taking into account precision of numbers, precision and units of quantities, precision and calendar type of dates and coordinate systems (“planets”) for geographic positions.

Strings

An example of a basic value is a string:

In[]:=

WikidataData

Wikidata

Plochingen

Wikidata

postal code



Out[]=

{73207}

“Demonym” is an example of a so-called monolingual text property. Its value depends on the setting for the language options:

In[]:=

WikidataData

Wikidata

Wikidata

demonym



Out[]=

{Terran,Earthling}

In[]:=

WikidataData

Wikidata

measured physical quantity

Wikidata

demonym

,Language"German"

Out[]=

{Erdling}

URLs

Retrieve a URL of an image representing an item:

In[]:=

WikidataData

Wikidata

kangaroo

,"ImageURL"

Out[]=

URL

http://commons.wikimedia.org/wiki/Special:FilePath/Kangaroo%20and%20joey03.jpg



As a small convenience, the image can be requested (saving you one

Import

call):

In[]:=

WikidataData

Wikidata

kangaroo

,"Image"

Out[]=





Geography

Geographic positions are represented as

GeoPosition

In[]:=

WikidataData

Wikidata

Berlin

,"GeoPosition"

Out[]=

{GeoPosition[{52.5167,13.3833}]}

This is for use in any geo-plotting or computation function:

In[]:=

GeoListPlot[%]

Out[]=

The automatic inclusion of the the coordinate system in the position allows, for instance,

GeoListPlot

to choose the right background.

In[]:=

WikidataData

Wikidata

Korolev

,"GeoPosition"

Out[]=

GeoPosition{72.77,164.58},

Mars



Here’s an example of a background chosen for Mars:

In[]:=

GeoListPlot[%]

Out[]=

Numbers and Quantities

Numbers can be considered quantities of dimension one (historically: “dimensionless quantities”). This is supported by the following identity:

In[]:=

Quantity[1,"PureUnities"]===1

Out[]=

True

Wikidata is following that system by having a single datatype to represent both with the quantity datatype. When users enter a value of a property with datatype quantity, the user interface optionally allows entering a unit. The units that can be entered in that unit field are Wikidata items. If no unit is entered, it defaults to Q199, the item representing the number 1. Being a unit, the number 1 has a unit symbol (which is typically omitted when writing down a value):

In[]:=

WikidataData

Wikidata

unit symbol



Out[]=

{1}

It also has a conversion to other (coherent) SI units (1, trivially), and a statement about which quantities it measures:

In[]:=

WikidataData

Wikidata

,

Wikidata

conversion to SI unit

Wikidata



Out[]=

{1},

Wikidata

dimensionless quantity



Let’s find some (direct) subclasses of the class of quantities of dimension one:

In[]:=

WikidataDataEntityClassAll,"SubclassOf"

Wikidata

dimensionless quantity

,

Wikidata

quantity symbol (LaTeX)

,"Association"//Short

Out[]=



Wikidata

inductive leakage factor

{σ},

Wikidata

quality factor

{Q},

Wikidata

loss factor

{d},

Wikidata

luminous efficiency

{V},125,

Wikidata

kinetic friction factor

{f,μ},

Wikidata

static friction coefficient

{

Wikidata

isentropic exponent

{ϰ},

Wikidata

inductive coupling factor

{k}

But I’m digressing. The takeaway is that Wikidata contains its own quantities and units ontology, which is necessary and of fundamental importance for representing any quantity-valued statement in Wikidata.

Now an example:

In[]:=

WikidataData

Wikidata

Eiffel Tower

Wikidata

Wolfram Language unit code



Out[]=

{

10000.

}

When hovering over the result, a tooltip indicates “unit: metric tons.”

There are quite a few “tons” out there. The linguistic interface allows selecting among the possible interpretations of “ton” known to the Wolfram Language:

Wikidata also knows a few “tons”:

In[]:=

WikidataSearch["ton"]

Out[]=



Wikidata

Wikidata

Wikidata

Wikidata

Wikidata

Wikidata

Wikidata



This example illustrates that there is too much ambiguity in commonly used unit names to rely just on the name to unambiguously identify a unit. To solve the ambiguity issue, Wikidata links its unit items to other unit ontologies:

In[]:=

WikidataData

Wikidata

tonne

,

Wikidata

QUDT unit ID

Wikidata

UCUM code

Wikidata

wurvoc.org measure ID

Wikidata



Out[]=



Wikidata QUDT unit ID

TON_M

,

Wikidata UCUM code

,

Wikidata wurvoc.org measure ID

tonne

,

Wikidata Wolfram Language unit code

"MetricTons"



This allows unambiguous translation of RDF (semantic web) data, which uses any of those unit ontologies to represent quantities to be translated to any other—including to and from the Wolfram Language.

Precision

The speed of light is a physical constant used to define the unit “meter.” It is exact when expressed in meters per second:

In[]:=

WikidataData

Wikidata

speed of light

Wikidata

numeric value



Out[]=

{

299792458

m/s

}

In[]:=

Precision[First[%]]

Out[]=

∞

The gravitational constant, on the other hand, is uncertain:

In[]:=

WikidataData

Wikidata

gravitational constant

Wikidata

numeric value



Out[]=



6.674×

-11

/(kg

)



The uncertainty is contained in the previous expression (its

FullForm

) and it can be made “visible” by applying

Around

In[]:=

MapAt[Around,First[%],1]

Out[]=

((

6.67430

0.00015

)×

-11

)

/(kg

)

(Nice formatting is not the only feature of Around.)

Dates

For dates not too far in the past, the (proleptic) Gregorian calendar is typically used:

In[]:=

WikidataData

Wikidata

Immanuel Kant

Wikidata

date of birth



Out[]=



Day: Sat 22 Apr 1724



Here is an example of a date given in the Julian calendar:

In[]:=

WikidataData

Wikidata

Plato

Wikidata

date of birth



Out[]=



Day: Sun 2 May -426(Juliancalendar)



The ability to enter dates in different calendar systems allows faithful representation of values given in (primary) sources. It leaves the task of converting between calendars to the application consuming such data:

In[]:=

CalendarConvert[%,"Gregorian"]

Out[]=



Day: Sun 27 Apr -426



The inception of the city Berlin is only known to year precision:

In[]:=

WikidataData

Wikidata

Berlin

Wikidata

inception



Out[]=



Year: 1237(Juliancalendar)



Result Forms If you know entities, then you’ll probably have discovered the possibility to “shape” the result. When requesting multiple values at once, the default result is a list, or more generally, an array:

In[]:=

Albert Einstein

PERSON

[{"BirthDate","DeathDate"}]

Out[]=



Day: Fri 14 Mar 1879

Day: Mon 18 Apr 1955



The position of the value in the result is the same as the position of the corresponding property in the input. To see the properties and values next to each other, one can request an

Association

or a

Dataset

instead:

In[]:=

Albert Einstein

PERSON

[{"BirthDate","DeathDate"},"Association"]

Out[]=

BirthDate

Day: Fri 14 Mar 1879

,DeathDate

Day: Mon 18 Apr 1955



WikidataData supports the same result forms:

In[]:=

WikidataData

Wikidata

Albert Einstein

,{"BirthDate","DeathDate"},"Association"

Out[]=

BirthDate

Day: Fri 14 Mar 1879

,DeathDate

Day: Mon 18 Apr 1955



Omitting the list of properties produces all available data:

In[]:=

WikidataData

Wikidata

Albert Einstein

,"Dataset"

Out[]=

Wikidata

Commons category

{Albert Einstein}

Wikidata

place of birth



Wikidata

Ulm



Wikidata

image

URL

http://commons.wikimedia.org/wiki/Special:FilePath/Albert%20Einstein%20Head.jpg



Wikidata

place of death



Wikidata

Princeton



Wikidata

sex or gender



Wikidata

male



Wikidata

father



Wikidata

Hermann Einstein



Wikidata

pronunciation audio

URL

http://commons.wikimedia.org/wiki/Special:FilePath/De-Albert%20Einstein-Pronuciation.ogg



Wikidata

permanent duplicated item



Wikidata

Q21200226

Wikidata

Q25696257



Wikidata

exact match

URL

http://data.nobelprize.org/resource/laureate/26



Wikidata

sibling



Wikidata

Maja Einstein



Wikidata

academic degree



Wikidata

Doctor of Philosophy in Physics



Wikidata

country of citizenship

{

…

}

Wikidata

date of death



Mon 18 Apr 1955



Wikidata

residence

{

…

}

Wikidata

date of birth



Fri 14 Mar 1879



Wikidata

native language



Wikidata

German



Wikidata

child



Wikidata

Eduard Einstein

Wikidata

Lieserl Einstein

Wikidata

Hans Albert Einstein



Wikidata

educated at

{

…

}

Wikidata

member of

{

…

}

Wikidata

archives at

{

…

}

rows 1–20 of

260

Entities Entities can be discovered easily with the linguistic assistant, thereby facilitating access to Wikidata:

In[]:=

WikidataData

,"Dataset"

Out[]=

›

http://commons.wikimedia.org/wiki/Special:FilePath...

One can also go in the other direction. Given an item, request the corresponding entity:

In[]:=

WikidataData

Wikidata