Essays, Posts & Presentations

How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine

Alec Shedelbower

Author

Alec Shedelbower

Title

How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine

Description

From concept to final project, how a Mathematica developer quickly built a digital piano using Unity and the Wolfram Language. Generating music using the SoundNote function, designing the mechanics of a piano, combining pieces into a functional program.

How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine

June 6, 2019

Alec Shedelbower, Kernel Developer, Algorithms R&D

You know what’s harder than learning the piano? Learning the piano without a piano, and without any knowledge of music theory. For me, acquiring a real piano was out of the question; I had neither the funds nor space in my small college apartment. So naturally, it looked like I would have to build one myself—digitally, of course. And luckily, I had Mathematica, Unity and a few hours to spare. Because working in Unity is incredibly quick and efficient with the Wolfram Language and UnityLink, I’ve created a playable section of piano, and even learned a bit of music theory in the process.

First, I determined that building the piano requires the following:

• Audio for each musical note

• Geometry for the piano keys

• A portable, interactive, real-time-rendering audio and 3D-physics engine

The first two can be accomplished trivially in the Wolfram Language. As for the last one, I opted to use the newly introduced UnityLink—a powerful link between the Wolfram Language and the real-time development platform Unity. Using UnityLink, it’s now possible to combine the advantages of the Wolfram Language’s impressive simulations with regards to rendering, audio and physics, with Unity’s efficient packaging of all three into standalone applications for web, desktop, mobile and console platforms.

What Is a Piano?

Before I dive into the code, let’s explore some of the background on the piano and the musical notes it plays. Understanding the theory behind the physical piano will help us to better recreate it digitally in Unity.

The piano traces its origins back to early 18th-century Italy, where it was invented by Bartolomeo Cristofori. Since then, it’s undergone many design changes, eventually resulting in a (mostly) standardized key configuration.
The modern piano has a total of 88 keys, 52 of which are white and are used to play the natural notes (A, B, C, D, E, F and G). The remaining 36 keys are black and are used to play the accidentals (A♯/B♭, C♯/D♭, D♯/E♭, F♯/G♭ and G♯/A♭). The ♯ and ♭ symbols stand for sharp and flat, respectively. Here you can see all 88 keys with their corresponding notes labeled:

The notes can be further divided into octaves, each of which contains 12 keys. Two keys with the same note but in different octaves will have different pitches. The octaves of a piano are color-coded in this diagram:

A piano contains seven full octaves, with four extra keys on the ends. These extra keys allow the scales of A minor and C major to be played in all seven octaves.

In this blog post, and for simplicity’s sake, I'll focus on a single musical scale (ordered list of notes), but you can apply this method to create the entire piano. Let’s use one of the most common scales—the C major scale. This scale contains only the natural notes in the order C, D, E, F, G, A and B. Any C note can be chosen as the start of the scale. Here, I'm going to use the C note in the fourth octave (also known as C4 or middle C):

If you take a closer look, you can see that this subsection of our piano contains all seven natural notes and all five accidentals. Note that I also included the C key from the next octave (C5) in the scale, as this helps “round off” the scale:

Making Music

Whew! With the background out of the way, I can finally get to the code. To get the sounds of the piano keys I use the symbol

SoundNote

, which can generate any note from a large collection of instruments. For a single note, you simply give it the note name, duration and instrument. When wrapped in

Audio

, it creates an audio object that can be played directly in a notebook:

In[]:=

Audio[SoundNote["C",3,"Piano"]]

Out[]=

00:00

00:03

Data in Notebook

To get a note in a specific octave, you simply concatenate the octave number to the end of the note name. For instance, I can get all the natural notes in the fourth octave using the code shown here:

In[]:=

naturalNotes={"C","D","E","F","G","A","B"};Table[Audio[SoundNote[note<>"4"]],{note,naturalNotes}]//AudioJoin

Out[]=

00:00

00:07

Data in Notebook

Generating Geometry

The exact shape and dimensions of piano keys vary by manufacturer. I opted to keep things simple by approximating each key as a prism. The advantage of using prisms is that I only need to specify the base polygon and extrude upward. However, ensuring no keys overlap requires five base polygon variations.

First, I define the dimensions of the keys in inches:

In[]:=

whiteWidth=7/8;whiteLength=6;blackWidth=15/32;blackLength=4;height=7/8;gap=1/16;

Next, I define all of the coordinates that will be used to construct the base polygons:

In[]:=

coordinates={{0,0},{whiteWidth,0},{whiteWidth,whiteLength},{0,whiteLength},{blackWidth/2,whiteLength},{blackWidth/2,whiteLength-blackLength},{0,whiteLength-blackLength},{whiteWidth,whiteLength-blackLength},{(whiteWidth-blackWidth)+(blackWidth/2),whiteLength-blackLength},{(whiteWidth-blackWidth)+(blackWidth/2),whiteLength},{whiteWidth+blackWidth/2,whiteLength-blackLength},{whiteWidth+blackWidth/2,whiteLength}};

Now it’s only a matter of picking the correct coordinates for each polygon:

In[]:=

coordIndices={{1,2,3,4},{1,2,3,5,6,7},{1,2,8,9,10,4},{1,2,8,9,10,5,6,7},{9,11,12,10}};polygons=Table[Polygon[coordinates[[indices]]],{indices,coordIndices}];RowGraphics

,#&/@polygons,Spacer[20]

Out[]=

All that’s left is to convert the base polygons into 3D prisms. This can be done easily using

RegionProduct

to multiply the polygons by a line segment with a given height:

In[]:=

line=BoundaryMeshRegion[{{0},{height}},Point[{{1},{2}}]];regions=Table[RegionBoundary[RegionProduct[BoundaryMeshRegion[polygon],line]],{polygon,polygons}];Row[regions]

Out[]=

Preparing the Project

Now that I have the audio and geometry, it’s time to combine them in Unity to make a working piano. As I mentioned previously, this is made possible with UnityLink.

With Unity installed, loading UnityLink is as simple as a single function call:

In[]:=

Needs["UnityLink`"]

I start by opening a new Unity project, which I’ve named "MyPiano":

In[]:=

UnityOpen["MyPiano"]

With the project open, I can now send and receive data from Unity. I will eventually want to create my piano in a Scene—a 3D environment that can act as a menu, game level or any other distinct part of a Unity application. But before I create my Scene, I have to first transfer the audio and geometry content I created earlier to Unity. Once it has been added, I will be free to use it in my Scene.

While not required, it's good practice to keep your Unity project organized with subdirectories in the project's Assets directory. The Assets directory contains all of the assets used in the project (textures, audio clips, meshes, etc.). In the line shown here, I create five directories in the Assets directory using CreateUnityAssetDirectory:

In[]:=

CreateUnityAssetDirectory[{"Meshes","Audio","Materials","Scenes","Scripts"}];

Next, I want to create the script that will enable user interaction on the piano. I wrote this script in Unity’s main scripting language C# (pronounced “C-sharp”). This script will be attached to each key of my piano. Whenever the user clicks a key or presses its corresponding button, it plays the audio of that key’s note and physically moves the key up and down. Here’s a portion of this script:

Out[]=

using UnityEngine;

public class PianoKey : MonoBehaviour

{

// Settings

public string key;

public AudioClip clip;

public float pressDuration = 0.1f;

public float unpressDuration = 0.2f;

// State

private enum State { Pressing, Unpressing, Idle }

...

}

To import the script into Unity, I manually export it to my project’s Assets directory:

In[]:=

filepath=FileNameJoin[{$UnityAssetsDirectory,"Scripts","PianoKey.cs"}];Export[filepath,pianoKeyScriptText,"String"]

Out[]=

/Users/alecs/MyPiano/Assets/Scripts/PianoKey.cs

I then use UnityTokenExecute to refresh the assets in my project, which causes the script to compile (Unity scripts need to be compiled before they can be used in a Scene):

In[]:=

UnityTokenExecute[{"Assets","Refresh"}]

Now I go about transferring the audio. I do this by passing the Audio of each note to the function CreateUnityAudioClip, which automatically converts it to Unity’s AudioClip object and stores it in the Assets directory. These AudioClip objects are represented as UnityAudioClip expressions in the Wolfram Language:

In[]:=

notes={"C4","D4","E4","F4","G4","A4","B4","C#4","D#4","F#4","G#4","A#4","C5"};clips=Association[Table[audio=Audio[SoundNote[note,2.5,"Piano"]];noteCreateUnityAudioClip[File["Audio/note_"<>note],audio],{note,notes}]];clips//Short

Out[]//Short=

C4UnityAudioClip

AudioClip

: note_C4

length: 2.53

,11,C5UnityAudioClip

AudioClip

: note_C5

length: 2.53



Next, I transfer the geometry of my piano keys. This time, however, I use CreateUnityMesh to automatically convert my MeshRegions to Unity’s Mesh objects, represented as UnityMesh expressions in the Wolfram Language:

In[]:=

meshes=Table[CreateUnityMesh[File["Meshes/mesh_"<>ToString[i]],regions[[i]]],{i,Length[regions]}];meshes//Short

UnityMesh

Mesh

: mesh_1

vertices: 24

,3,UnityMesh

Mesh

: mesh_5

vertices: 24



The last assets I create are the materials for my piano keys. I simply create a white and a black material using CreateUnityMaterial:

In[]:=

whiteMaterial=CreateUnityMaterial[File["Materials/White"],White]blackMaterial=CreateUnityMaterial[File["Materials/Black"],Black]

Out[]=

UnityMaterial

Material

: White

shader: Standard



Out[]=

UnityMaterial

Material

: Black

shader: Standard



Setting the Scene

With all of the Assets transferred, I can finally make the Scene for my piano. I start by creating a new default Scene:

In[]:=

CreateUnityScene[File["Scenes/Piano"]]

Out[]=

UnityScene

Scene

: Piano

path: Assets/Scenes/Piano.unity



If you’re new to Unity, here’s a brief description of Scenes. Scenes contain Game Objects, which in turn act as containers for Components. You can think of the Scene as an environment, Game Objects as the things in that environment and Components as the behaviors of those things.

For example, my default Scene already contains two game objects, Main Camera and Directional Light:

The Main Camera game object contains a Camera component, while the Directional Light game object contains a Light component:

In my piano Scene, I’m going to make a Game Object for each key. I’ll then attach the script component I created earlier to each of these game objects, so they make sound and move when the user interacts with them.

I could just add each key one at a time; however, that would prove to be tedious and difficult to extend in the future. Instead, I define the information about each white key and each black key in two lists. I can then iterate over these lists to create each key automatically. For each key, I specify the computer keyboard key it corresponds to, the musical note it should play and the index of the mesh it should use. Note that the mesh index for black keys is implicitly assumed to be 5:

In[]:=

whiteKeys="Keycode""q","Note""C4","Mesh"3,

;blackKeys=<|"Keycode""2","Note""C#4"|>,

;

To keep my Scene organized, I also group all of my keys under a parent game object named "Piano Scale":

In[]:=

parent=CreateUnityTransform["Piano Scale"]

Out[]=

UnityTransform

Transform

: Piano Scale

position: {0.,0.,0.}



I iterate over all the white keys first:

In[]:=

Do[key=whiteKeys[[i]];name="Key "<>key["Note"]<>" (White)";go=CreateUnityGameObject[name,meshes[[key["Mesh"]]]];go[["Transform","Position"]]={(i-1)*(whiteWidth+gap),0,0};go[["Transform","Parent"]]=parent;script=CreateUnityComponent[go,"PianoKey"];script[["Key"]]=key["Keycode"];script[["Clip"]]=clips[key["Note"]];,{i,Length[whiteKeys]}]

This is followed by the black keys:

In[]:=

Do(*Codenearlyidenticaltotheloopoverwhitekeys*)

CompoundExpression[

]

,{i,Length[blackKeys]}

For each key, I create a Game Object with the appropriate mesh using CreateUnityGameObject. After setting the position of this Game Object, I attach the custom script I created earlier by passing the Game Object and script name to CreateUnityComponent. I finish by specifying the keycode and audio clip for that key.

And just like that, I have a working (partial) piano. However, it doesn’t look as good as it could. To remedy this, I first set the material of all the keys to the white or black materials I created earlier:

In[]:=

whiteMeshRenderers=FindUnityComponent["*White*","MeshRenderer",All];Do[wmr[["SharedMaterial"]]=whiteMaterial,{wmr,whiteMeshRenderers}]

In[]:=

blackMeshRenderers=FindUnityComponent["*Black*","MeshRenderer",All];Do[bmr[["SharedMaterial"]]=blackMaterial,{bmr,blackMeshRenderers}]

Next, I adjust the properties of the camera for a better view of the piano keys:

In[]:=

camera=FindUnityComponent[All,"Camera"];camera[["Transform","Position"]]={(whiteWidth+gap)*4,6.75`,-3.5`};camera[["Transform","EulerAngles"]]={45.`,0.`,0.`};camera[["ClearFlags"]]="SolidColor";camera[["BackgroundColor"]]=GrayLevel[0.15];

Finally, I adjust the lighting to better highlight the piano:

In[]:=

light=FindUnityComponent["Directional Light","Light"];light[["Transform","EulerAngles"]]={90,0,0};

And with this, we get the final result:

Now that looks better! Before moving on, I also want to save all the changes I just made to my Scene by calling SaveUnityScene:

In[]:=

SaveUnityScene[]

Out[]=

True

Playing the Piano

To test the piano in the Unity editor, I can use UnityPlay and UnityStop to switch between the Play and Edit modes. When I’m satisfied with the results, I can build the project to a standalone application using UnityBuild.

The following command will automatically build the project to a file in my project directory for my current platform (macOS):

In[]:=

UnityBuild[]

Out[]=

Success

✓

ErrorCount: 0

WarningCount: 1



With the build successful, I can immediately open and play my piano application:

In[]:=

SystemOpen[%["Application"]]

One of the advantages of working in Unity is its ability to build to numerous platforms without having to change your code. If you can play a game on a platform, odds are that Unity can build to it.
It can even be built to run in a web browser. Go ahead and try it!

Your Turn!

This small section of the piano can easily be extended to a full piano keyboard. With more than 160 styles and percussions available in SoundNote, you could also build other instruments or even combine them into a single synthesizer.

To start working with UnityLink in the Wolfram Language, visit the online documentation page or try out one of the sample projects. There’s so much you can do with the built-in interface, and I look forward to seeing what projects you come up with on Wolfram Community!

Cite this as: Alec Shedelbower, "How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine" from the Notebook Archive (2019), https://notebookarchive.org/2019-06-2tqvszr