Panako – Acoustic Fingerprinting

Panako is an acoustic fingerprinting system. The system is able to extract fingerprints from an audio stream, and either store those fingerprints in a database, or find a match between the extracted fingerprints and stored fingerprints. Several acoustic fingerprinting algorithms are implemented within Panako. The main algorithm, the Panako algorithm, has the feature that audio queries can be identified reliably and quickly even if they has been sped up, time stretched or pitch shifted with respect to the reference audio. The main focus of Panako is to serve as a demonstration of the Panako algorithm. Other acoustic fingerprinting schemes are implemented to serve as a baseline for comparison. More information can be found in the article about Panako.

Please be aware of the patents US7627477 B2 and US6990453 and perhaps others. They describe techniques used in some algorithms implemented within Panako. The patents limit the use of some algorithms under various conditions and for several regions. Please make sure to consult your intellectual property rights specialist if you are in doubt about these restrictions. If these restrictions apply, respect the patent holders rights. The first aim of Panako is to serve as a research platform to experiment with and improve fingerprinting algorithms.

This document covers installation, usage and configuration of Panako.

The Panako source code is licensed under the GNU Affero General Public License.

Overview

1. Why Panako?
2. Getting Started
3. Usage
   1. Store Fingerprints
   2. Query for Matches
   3. Monitor Stream for Matches
   4. Print Storage Statistics
   5. Print Configuration
4. Further reading
5. Panako and docker
6. Reproduce the ISMIR Paper Results
7. Credits
8. Changelog

Why Panako?

Content based music search algorithms make it possible to identify a small audio fragment in a digital music archive with potentially millions of songs. Current search algorithms are able to respond quickly and reliably on an audio query, even if there is noise or other distortions present. During the last decades they have been used successfully as digital music archive management tools, music identification services for smartphones or for digital rights management.

Most algorithms, as they are described in the literature, do not allow substantial changes in replay speed. The speed of the audio query needs to be the same as the reference audio for the current algorithms to work. This poses a problem, since changes in replay speed do occur commonly, they are either introduced by accident during an analog to digital conversion or are introduced deliberately.

Analogue physical media such as wax cylinders, wire recordings, magnetic tapes and grammophone records can be digitized at an incorrect or varying playback speed. Even when calibrated mechanical devices are used in a digitization process, the media could already have been recorded at an undesirable speed. To identify duplicates in a digitized archive, a music search algorithm should compensate for changes in replay speed

Next to accidental speed changes, deliberate speed manipulations are sometimes introduced during radio broadcasts: occasionally songs are played a bit faster to fit into a timeslot. During a DJ-set speed changes are almost always present. To correctly identify audio in these cases as well, a music search algorithm robust against pitch shifting, time stretching and speed changes is desired.

The Panako algorithm allows such changes while maintaining other desired features as scalability, robustness and reliability. Next to the panako algorithm there is also an implementation of the algorithm described in An Industrial-Strength Audio Search Algorithm (internally identified as Olaf). Also the algorithm in A Robust Audio Fingerprinter Based on Pitch Class Histograms – Applications for Ethnic Music Archives is available. To make comparisons between fingerprinting systems easy, researchers are kindly invited to contribute algorithms to the Panako project.

Alternative open source music identification systems are audfprint, NeuralFP and echoprint. Alternative systems with similar features are described in US7627477 B2 and in Quad-based Audio Fingerprinting Robust To Time And Frequency Scaling by Reinhard Sonnleitner and Gerhard Widmer.

Getting started

Panako is organized as follows:

• src contains the Java source files.
• lib contains the dependencies (JAR-files).
• build contains an ant build file to compile Panako and the documentation. Use ant release to build the source and compile Panako.
• scripts contains helper scripts for evaluation and to analyse results

To compile Panako, the JDK 11 or later is required. See the installation instructions on the Java website, for installation on your operating system.

Panako uses the Gradle build system. Install it on your system.
Once Gradle and the other components are installed correctly the following commands should get you started:

git clone --depth 1 https://github.com/JorenSix/Panako.git
cd Panako
./gradlew shadowJar #Builds the core Panako library
./gradlew install #Installs Panako in the ~/.panako directory
./gradlew javadoc #Creates the documentation in doc
sudo cp resources/defaults/panako /usr/local/bin #copies the panako startup script to your path

The last command copies the startup script in resources/defaults/panako to a directory in your path. The script allows for easy access to Panako’s functionality. If this does not work for you, you can still call Panako using java -jar ~/.panako/panako.jar [..args].

Panako decodes audio using by calling ffmpeg, an utility for decoding and resampling audio.
The ffmpeg command needs to on the systems path. On a Debian like system:

apt-get install ffmpeg

Test Panako. You might need a new shell to use panako. Note that you also need SoX and wget on your system:

#brew install sox wget
#apt-get install sox wget
panako -v #prints version
panako stats #db info
cd resources/scripts/create_test_dataset
ruby create_dataset.rb #make sure sox
panako store dataset/*
panako query query/*

Panako Usage

Panako provides a command line interface, it can be called using panako subapplication [argument...]. For each task Panako has to perform, there is a subapplication. There are subapplications to store fingerprints, query audio fragments, monitor audio streams, and so on. Each subapplication has its own list of arguments, options, and output format that define the behavior of the subapplication. The screen capture below shows a typical interaction with Panako.

To save some keystrokes the name of the subapplication can be shortened using a unique prefix. For example panako m file.mp3 is expanded to panako monitor file.mp3. Since both stats and store are valid subapplications the store call can be shortened to panako sto *.mp3, panako s *.mp3 gives an invalid application message. A trie is used to find a unique prefix.

What follows is a list of those subapplications, their arguments, and respective goal.

Store Fingerprints – panako store

The store instruction extracts fingerprints from audio tracks and stores those in the datastorage. The command expects a list of audio files, video files or a text file with a list of file names.

#Audio is converted automatically
panako store audio.mp3 audio.ogg audio.flac audio.wav 

#The first audio stream of video containers formats are used.
panako store audio.mpc audio.mp4 audio.acc audio.avi audio.ogv audio.wma 

#Glob characters can be practical
panako store */*.mp3

#A text file with audio files can be used as well
#The following searches for all mp3's (recursively) and
#stores them in a text file
find . -name '*.mp3' > list.txt
#Iterate the list
panako store list.txt

Remove fingerprints – panako delete

This application removes fingerprints from the index. It essentially reverses the store operation. The operation can be checked with panako stats

panako delete test.mp3

The default key-value-store is backed by some kind of B-tree structure. Removing many elements from such structure might leave the tree in an unbalanced state, which results in worse performance. I am not sure about the performance implications of deletion for LMDB but it might be of interest to either rebuild the index or avoid deletion as much as possible.

Query for Matches – panako query

The query command extracts fingerprints from an audio frament and tries to match the fingerprints with the database.

panako query short_audio.mp3

Monitor Stream for Matches – panako monitor

The query command extracts fingerprints from a short audio frament and tries to match the fingerprints with the database. The incoming audio is, by default, chopped in parts of 25s with an overlap of 5s. So every 25-5=20s the database is queried and a result is printed to the command line.

panako monitor [short_audio.mp3]

If no audio file is given, the default microphone is used as input.

Print Storage Statistics – panako stats

The stats command prints statistics about the stored fingerprints and the number of audio fragments. If an integer argument is given it keeps printing the stats every x seconds.

panako stats # print stats once
panako stats 20 # print stats every 20s 

Print Configuration – panako config

The config subapplication prints the configuration currently in use.

panako config

To override configuration values there are two options. The first option is to create a configuration file, by default at the following location: ~./panako/config.properties. The configuration file is a properties text file. An commented configuration file should be included in the doc directory at doc/config.properties.

The second option to override configuration values is by adding them to the arguments of the command line call as follows:

panako subapplication CONFIG_KEY=value

For example, if you do not want to check for duplicate files while building a fingerprint database the following can be done:

panako store file.mp3 CHECK_FOR_DUPLICATES=FALSE

The configuration values provided as a command line argument have priority over the ones in the configuration file. If there is no value configured a default is used automatically. To find out which configuration options are available and their respective functions, consult the documented example configuration file doc/config.properties..

Resolve an identifier for a filename – panako resolve

This application simply returns the identifier that is used internally for a filename. The following call returns for example 54657653:

panako resolve test.mp3

The internal identifiers are currently defined using integers.

Prints how similar two audio files are – panako same

This application checks the similarity of two files. The percentage returned indicates the percentage of seconds for which fingerprints match between the first and second file. So 100% that matches are found in every second. A result of 30% still means that much of the audio matches.

panako same first.mp3 second.mp3

Note that this operation is performed in memory. Nothing changes on disk.

Panako and Docker

Panako can also be ran in a containerized environment. A Dockerfile is provided which should both work on x86_64 as aarch64. To build the container and run commands the following should get you started. Note that the database with fingerprints is located on the host at ~/.panako/docker_dbs:

cd resources/scripts/
docker build -t panako:2.1 .
mkdir -p $HOME/.panako/docker_dbs
wget "https://filesamples.com/samples/audio/mp3/sample3.mp3"
docker run -v $HOME/.panako/docker_dbs:/root/.panako/dbs panako:2.1 panako store sample3.mp3
docker run -v $HOME/.panako/docker_dbs:/root/.panako/dbs panako:2.1 panako stats

Further Reading

Some relevant reading material about acoustic fingerprinting. The order gives an idea of relevance to the Panako project.

1. Six, Joren and Leman, Marc Panako – A Scalable Acoustic Fingerprinting System Handling Time-Scale and Pitch Modification (2014)
2. Wang, Avery L. An Industrial-Strength Audio Search Algorithm (2003)
3. Cano, Pedro and Batlle, Eloi and Kalker, Ton and Haitsma, Jaap A Review of Audio Fingerprinting (2005)
4. Six, Joren and Cornelis, Olmo A Robust Audio Fingerprinter Based on Pitch Class Histograms – Applications for Ethnic Music Archives (2012)
5. Arzt, Andreas and Bock, Sebastian and Widmer, Gerhard Fast Identification of Piece and Score Position via Symbolic Fingerprinting (2012)
6. Fenet, Sebastien and Richard, Gael and Grenier, Yves A Scalable Audio Fingerprint Method with Robustness to Pitch-Shifting (2011)
7. Ellis, Dan and Whitman, Brian and Porter, Alastair Echoprint – An Open Music Identification Service (2011)
8. Sonnleitner, Reinhard and Widmer, Gerhard Quad-based Audio Fingerprinting Robust To Time And Frequency Scaling (2014)
9. Sonnleitner, Reinhard and Widmer, Gerhard Robust Quad-Based Audio Fingerprinting (2015)
10. S. Chang et al., “Neural Audio Fingerprint for High-Specific Audio Retrieval Based on Contrastive Learning” (2021) – ICASSP 2021.

Credits

The Panako software was developed at IPEM, Ghent University by Joren Six.

Some parts of Panako were inspired by the Robust Landmark-Based Audio Fingerprinting Matlab implementation by Dan Ellis.

If you use Panako for research purposes, please cite the following works:

@inproceedings{six2014panako,
  author      = {Joren Six and Marc Leman},
  title       = {{Panako - A Scalable Acoustic Fingerprinting System Handling Time-Scale and Pitch Modification}},
  booktitle   = {{Proceedings of the 15th ISMIR Conference (ISMIR 2014)}}, 
  year        =  2014
}

@inproceedings{six2021panako,
  title={Panako 2.0-Updates for an acoustic fingerprinting system},
  author={Six, Joren},
  booktitle={Demo / late-breaking abstracts of 22st International Society for Music Information Retrieval Conference, ISMIR},
  year={2021}
}

Reproduce the ISMIR Paper Results

The directory scripts/reproducibility contains scripts to reproduce the result found in the
Panako ISMIR 2014 paper. The scripts follow this procedure:

1. The Jamendo dataset is downloaded.
2. The fingerprints are extracted and stored for each file in the data set.o download an openly available dataset
3. Query files are created from the Jamendo data set.
4. Panako is queried for each query file, results are logged
5. Results are parsed.

To run the evaluation scripts, a UNIX like system with following utilities is required:

• ffmpeg: an utility to convert media files
• wget: an utility to download files
• SoX, inlcuding support for MP3 and GSM encoding.
• bash: the bash shell
• Ruby: The ruby programming language runtime environment.
• Unzip: to unzip the metadata file.

Evidently the panako software should also be installed. Please see above to install the Panako.

If all requirements are met, running the test is done using bash run_experiment.bash

Changelog

Improvements to SyncSink. The cross correlation now behaves well also in edge cases. The Sync code has been simplified to allow maintenance. Included unit tests. Updated to TarsosDSP version 2.4.

This release adds a simplified version of chromaprint and an implementation of ‘A Highly Robust Audio Fingerprinting System’ by Haitsma and Kalker

A major overhaul of Panako. The main aim of this release is to ensure the longevity and maintainability of Panako. The featureset has been reduced to focus on core functionality. The version bump is also explained by the use of lambdas and the need for a newer JRE (8+). A list of changes:

• The number of dependencies has been drastically cut by removing support for multiple key-value stores.
• The key-value store has been changed to a faster and simpler system (from MapDB to LMDB).
• The SyncSink functionality has been moved to another project (with Panako as dependency).
• The main algorithms have been replaced with simpler and better working versions:
  • Olaf is a new implementation of the classic Shazam algorithm.
  • The algoritm described in the Panako paper was also replaced. The core ideas are still the same. The main change is the use of a Gabor transform to go from time domain to the spectral domain (previously a constant-q transform was used). The gabor transform is implemented by JGaborator which in turn relies on The Gaborator C++ library via JNI.
• Folder structure has been simplified.
• The UI, which was mainly used for debugging, has been removed.
• A new set of helper scripts are added in the scripts directory. They help with evaluation, parsing results, checking results, building panako, creating documentation,…
• Changed the default panako location to ~/.panako, so users can install and use panako more easily (withouth need for sudo rights)

Changed the build system to Gradle, mainly to make upgrading dependencies more straightforward. The folder structure has been changed accordingly. The default JDK target has been changed to Java SE 11 (LTS). Panako now also supports Apple M1 targets.