Kemi Ukadike: Multimodal Thumb Piano

(image: Thumb Piano app)

Having been in a situation where I lost control of my fine motor skills, including use of my fingers, I wanted to create a musical instrument that can be played by those with weaker upper body strength. The way it works now is that when the mouse is rolled over each key, it emits the corresponding sound.

I have also included facial gesture as a mode of input. Therefore, when a user looks at the thumb piano they can play it with their face.

Mouse Over Version

Facial Gesture version (uses the web camera)

The process:

The device was made using p5.js, a tool for creative coding. I recorded the actual notes from playing the tines of the thumb piano, and mapped them to each key.

I also used the ML5 gesture recognition model, and mapped it to the thumb piano keys.

User Paths

User 1

Parent or Caregiver that wants to teach African instruments to a child.
Willing to use a kit or instructable
Willing to download the app for the sound synthesis connected to this device

User 2

A music lover that is into experimentation
Is able to tinker with low-energy electronics
looking for cultural references in music
Wants to connect to African Culture

User 3

An occupational therapist looking for tools to help with dexterity
Is willing to make the kit with a patient
Believes in music therapy
Buys into the whole instructables idea

User 4

Someone wanting to use the app only
Wants to also mess around with gesture-controlled devices
The device and the app instrument are the same object

User 5

A person with limited mobility
Looking to play music but does not have the muscle strength (I say this from personal experience)
Enjoys music therapy
Willing to connect to African culture

User Mapping

Idea	Metaphor	Model	Display
Play Music without the use of hands for arm mobility-challenged people.	Moving to music	Use ML5's gesture control model, with p5.js piano keys example	Kalimba app interface
Error	Scenario	Task	Control
Some people have weak arm / hand muscles and can not play a traditional thumb piano.	The app is used for therapeutic purposes, to engage the senses. perhaps in occupational therapy.	Download Kalimba app Read instructions for facial movements to keys Play African songs Play folk tunes	A face tracking app to play the sounds of a traditional thumb-piano.