Discover traditional Norwegian music by drawing gestures with your iPhone or iPad.
First draw a gesture by moving your iPhone or iPad up and down, to indicate ascending and descending melody. A simple sound is played while you move your device, its pitch gets higher when you move up, and gets lower when you move down. Once you have terminated your gesture, the piece of music that contains the melodic gesture closest to your gesture is presented. The title and genre of that piece of music is indicated, as well as the performer and the district in Norway where it comes from. A melodic curve of the specific melodic gesture from the song is displayed below your own gesture. The matching between your gesture and the melody is based on a temporal alignment of their contours: the ascending parts of your gesture is aligned with the ascending parts of the melody, and same for the descending parts. Some parts may be skipped, and are shown in red in the curves. When the music is played, a cursor moves throughout both curves so that you can follow the music while looking at the gestures.
SoundTracer uses Apples ARKit technology, solely in order to precisely track the location of your iPhone or iPad. For that reason, there is no actual Augmented Reality, and the camera is solely used to improve the precision of the tracking, without displaying the actual frames of the camera.
For the moment, the catalogue of music is composed of around 50 pieces, and only the initial part of the melody is considered. The catalogue includes a dozen tunes played on the traditional Norwegian Hardanger fiddle, and songs sung a cappella. SoundTracer will be progressively improved with more music, more refined methods to compare your gestures with the music, and much more.
SoundTracer is an innovation project led by Alexander Refsum Jensenius, supported by the University of Oslo and developed in collaboration with the National Library of Norway. More information here: https://www.hf.uio.no/imv/english/research/projects/soundtracer/