Software
Below you can find links to the open source software I developed during my research. It is always nice to hear how this software is used, don’t hesitate to drop me a line. Bug reports are welcomed as well.
TarsosDSP: a Java Library for Audio Processing
TarsosDSP is a Java library for audio processing. Its aim is to provide an easy-to-use interface to practical music processing algorithms implemented, as simply as possible, in pure Java and without any other external dependencies. TarsosDSP features an implementation of a percussion onset detector and a number of pitch detection algorithms: YIN, the Mcleod Pitch method and a “Dynamic Wavelet Algorithm Pitch Tracking” algorithm. Also included is a Goertzel DTMF decoding algorithm, a time stretch algorithm (WSOLA), resampling, filters, simple synthesis, some audio effects, and a pitch shifting algorithm.
To show the capabilities of the library, TarsosDSP example applications are available.
Links:
Feature extraction with TarsosDSP: Constant-Q, beat-detection, onset-detection, pitch estimation
Panako: an Acoustic Fingerprinting System
Panako is an extendable acoustic fingerprinting framework. The aim of acoustic fingerprinting is to find small audio fragments in large audio databases. Panako contains several acoustic fingerprinting algorithms to make comparison between them easy. The main Panako algorithm uses key points in a Constant-Q spectrogram as a fingerprint to allow pitch-shifting, time-stretching and speed modification. The aim of Panako is to serve as a platform for research on Acoustic Fingerprinting Systems.
Links:
Panako fingerprint with modifications
Olaf: Overly Lightweight Acoustic Fingerprinting
Olaf is a portable, landmark-based, acoustic fingerprinting system released as open source software. Olaf runs on embedded platforms, traditional computers and in the browser. Olaf is able to extract fingerprints from an audio stream, and either store those fingerprints in a database, or find a match between extracted fingerprints and stored fingerprints. It implements an algorithm similar to the one described in a classic ISMIR paper and has similar retrieval performance. It facilitates the many use cases acoustic fingerprinting has to offer such as duplicate detection, meta-data coupling, and synchronization.
Links:
Olaf in the browser
MIDI and OSC Tools: mot
mot consists of several MIDI and OSC command line tools. These are mainly of interest to debug and check OSC messages and MIDI devices. The tools are written in Rust and provide a collection of sub-applications for various MIDI and OSC tasks.
Applications:
- midi_echo - prints MIDI messages coming from a connected MIDI device
- osc_echo - prints OSC messages arriving at a certain UDP port
- osc_send - sends OSC messages from STDIN to a certain host
- midi_to_osc - a MIDI to OSC bridge which sends MIDI messages to an OSC target
- osc_to_midi - receives OSC messages and sends them to a MIDI device
- midi_processor - processes incoming MIDI messages using Lua scripts with transformations
- midi_roundtrip_latency - measure MIDI round-trip latency
Links:
mot command line tools for MIDI and OSC
Web Applications and Libraries
Thanks to WebAssembly it is possible to repurpose software for use on the web, even if it was originally designed with other use in mind. I have developed some practical tools/proof of concepts by repackaging and extending existing software for use on the web. The main advantage is the ease of use. Software is accessible to anyone with a browser, without having to install any additional software. Another advantage should be longevity: web standards have a tendency to be supported for a long time.
Projects:
- SyncSink.wasm - Synchronize media files by audio-to-audio alignment
- ffmpeg.audio.wasm - An audio focused ffmpeg build for the web
- ltc.wasm - SMPTE decoding in the browser
- pffft.wasm - An FFT library for the web
- Emotopa - Patterns in Pitch Organization
- Gabber - Visualizing constant-Q transform
- Olaf - Acoustic fingerprinting in the browser
JGaborator: High Resolution Spectral Transforms
This library calculates fine-grained constant-Q spectral representations of audio signals quickly from Java. The spectral transform can be visualized or further processed in a (Music Information Retrieval) processing chain.
The calculation of a Gabor transform is done by a C library named Gaborator. A Java native interface (JNI) bridge to the C Gaborator is provided here. A combination of Gaborator and a fast FFT library (such as pfft) allows fine grained constant-Q transforms at a rate of about 200 times real-time on moderate hardware.
Links:
Fine grained spectral transforms in Java
SyncSink: Video and Audio Synchronization Tool
SyncSink is able to synchronize video and audio recordings of the same event. As long as some audio is shared between the multimedia files a reliable synchronization solution will be proposed. SyncSink is ideal to synchronize video recordings of the same event by multiple cameras or to align a high definition audio recording with a video recording (with less qualitative audio). SyncSink is also used to facilitate synchronization of multimodal research data e.g. to research the interaction between movement and music.
Links:
SyncSink program with multimedia files
Tarsos: Software for Pitch Analysis
Tarsos is a software tool to analyze and experiment with pitch organization in all kinds of musics. Most of the analysis is done using pitch histograms and octave reduced pitch class histograms. Tarsos has an intuitive user interface and contains a couple of command line programs to analyze large sets of music.
To run Tarsos you need a recent Java runtime on your machine.
Links:
Pitch analysis with Tarsos
TarsosLSH: Locality Sensitive Hashing in Java
TarsosLSH is a Java library implementing Locality-sensitive Hashing (LSH), a practical nearest neighbor search algorithm for multidimensional vectors that operates in sublinear time. It supports several LSH families: the Euclidean hash family (L2), city block hash family (L1) and cosine hash family. The library tries to hit the sweet spot between being capable enough to get real tasks done, and compact enough to serve as a demonstration on how LSH works.
Links:
General vs. locality sensitive hashing (from cybertron.cg.tu-berlin.de)
TeensyDAQ: Analog Signal Visualization
TeensyDAQ is a Java application to quickly visualize and record analog signals with a Teensy micro-controller and some custom software. It is mainly useful to quickly get an idea of how an analog sensor reacts to different stimuli.
Features: - Visualize up to five analog signals simultaneously in real-time - Capture analog input signals with sampling rates up to 8000Hz - Record analog input to a CSV-file and visualize previously recorded CSV-files via drag-and-drop - Works on Linux, Mac OS X and Windows - Navigate back in time and zoom, pan and drag during capture sessions - Listen to your input signal (especially practical with analog microphone input)
Links:
TeensyDAQ user interface
AMPEL: Augmented Movement Platform
The idea behind AMPEL (by Lousin Moumdjian, Marc Leman, Peter Feys) is to combine both motor and cognitive rehabilitation in a single combined ‘embodied learning’ paradigm. To this end a device was developed which combines a short term memory task with a movement task: in this case walking over interactive tiles.
AMPEL is built together with a number of collaborators for the concept, the hardware design and implementation. I was responsible for the development of the embedded software in the tiles and for development of the GUI application for researchers.
Links:
Schematic representation of the AMPEL platform
Pidato: Vibrato on a Digital Piano
The Pidato experiment demonstrates a rather straightforward method to handle vibrato on a digital piano. It ‘solves’ the age-old problem on what to do with the enigmatic “vibrato” instructions on some piano solo scores of Franz Liszt.
The way it works is by translating movement (accelerometer data) to MIDI messages. Pidato consists of a hardware and software part. The hardware consists of an Arduino, MIDI-ports and a three axis accelerometer. The software should know when a vibrato like movement is made and how to translate such movement to MIDI messages. The software therefore contains a periodicity estimator and frequency detector to detect how periodic a movement is and how fast the movement is repeated. This was done with the YIN algorithm (more commonly used in audio signal analysis).
Links:
Wire schema of the Pidato implementation