Kemi Ukadike: Multimodal Thumb Piano

thumb piano app

(image: Thumb Piano app)

Having been in a situation where I lost control of my fine motor skills, including use of my fingers, I wanted to create a musical instrument that can be played by those with weaker upper body strength. The way it works now is that when the mouse is rolled over each key, it emits the corresponding sound.

I have also included facial gesture as a mode of input. Therefore, when a user looks at the thumb piano they can play it with their face.

 

Mouse Over Version

Facial Gesture version (uses the web camera)

 

 

 

The process:

The device was made using p5.js, a tool for creative coding. I recorded the actual notes from playing the tines of the thumb piano, and mapped them to each key.

I also used the ML5 gesture recognition model, and mapped it to the thumb piano keys.

User Paths

User 1

  • Parent or Caregiver that wants to teach African instruments to a child.
  • Willing to use a kit or instructable
  • Willing to download the app for the sound synthesis connected to this device

User 2

  • A music lover that is into experimentation
  • Is able to tinker with low-energy electronics
  • looking for cultural references in music
  • Wants to connect to African Culture

User 3

  • An occupational therapist looking for tools to help with dexterity
  • Is willing to make the kit with a patient
  • Believes in music therapy
  • Buys into the whole instructables idea

User 4

  • Someone wanting to use the app only
  • Wants to also mess around with gesture-controlled devices
  • The device and the app instrument are the same object

User 5

  • A person with limited mobility
  • Looking to play music but does not have the muscle strength (I say this from personal experience)
  • Enjoys music therapy
  • Willing to connect to African culture

User Mapping

Idea

Metaphor

Model

Display

nose key board

Play Music without the use of hands for arm mobility-challenged people.

dancing figure

Moving to music

machine learning graph

Use ML5's gesture control model, with p5.js piano keys example

kalimba app

Kalimba app interface

Error

Scenario

Task

Control

user injury

Some people have weak arm / hand muscles and can not play a traditional thumb piano.

rehabilitation setting

The app is used for therapeutic purposes, to engage the senses. perhaps in occupational therapy.

  • Download Kalimba app
  • Read instructions for facial movements to keys
  • Play African songs
  • Play folk tunes

face tracking app

A face tracking app to play the sounds of a traditional thumb-piano.

 

Above: Verplank's IxD framework

 

 

7 axis dimension space diagram for the kalimba app
7-axis dimension space diagram

 

 

 

Tutorials

Building REST API in Nodejs / MongoDB /Passport /JWT

building REST API with Nodejs / MongoDB /Passport /JWT

 

if your don’t know Node and JS before Take this Wes Bos Course that helps your jump start

Why do people fall in love with Python?

Most of the people think the word “python” is a snake but it is the most popular programming language.

Top Tutorials to Learn WordPress Website Creation For Beginners

WordPress is the perfect tool for creating a fully customized website for any purpose.