Kemi Ukadike: Multimodal Thumb Piano

thumb piano app

(image: Thumb Piano app)

Having been in a situation where I lost control of my fine motor skills, including use of my fingers, I wanted to create a musical instrument that can be played by those with weaker upper body strength. The way it works now is that when the mouse is rolled over each key, it emits the corresponding sound.

I have also included facial gesture as a mode of input. Therefore, when a user looks at the thumb piano they can play it with their face.


Mouse Over Version

Facial Gesture version (uses the web camera)




The process:

The device was made using p5.js, a tool for creative coding. I recorded the actual notes from playing the tines of the thumb piano, and mapped them to each key.

I also used the ML5 gesture recognition model, and mapped it to the thumb piano keys.

User Paths

User 1

  • Parent or Caregiver that wants to teach African instruments to a child.
  • Willing to use a kit or instructable
  • Willing to download the app for the sound synthesis connected to this device

User 2

  • A music lover that is into experimentation
  • Is able to tinker with low-energy electronics
  • looking for cultural references in music
  • Wants to connect to African Culture

User 3

  • An occupational therapist looking for tools to help with dexterity
  • Is willing to make the kit with a patient
  • Believes in music therapy
  • Buys into the whole instructables idea

User 4

  • Someone wanting to use the app only
  • Wants to also mess around with gesture-controlled devices
  • The device and the app instrument are the same object

User 5

  • A person with limited mobility
  • Looking to play music but does not have the muscle strength (I say this from personal experience)
  • Enjoys music therapy
  • Willing to connect to African culture

User Mapping





nose key board

Play Music without the use of hands for arm mobility-challenged people.

dancing figure

Moving to music

machine learning graph

Use ML5's gesture control model, with p5.js piano keys example

kalimba app

Kalimba app interface





user injury

Some people have weak arm / hand muscles and can not play a traditional thumb piano.

rehabilitation setting

The app is used for therapeutic purposes, to engage the senses. perhaps in occupational therapy.

  • Download Kalimba app
  • Read instructions for facial movements to keys
  • Play African songs
  • Play folk tunes

face tracking app

A face tracking app to play the sounds of a traditional thumb-piano.


Above: Verplank's IxD framework



7 axis dimension space diagram for the kalimba app
7-axis dimension space diagram