February 9, 2015

Mixing Music with Machine Learning

Mixing Music with Machine Learning
An Interview with Dr. Brian McFee.

Dr. Brian McFee, who is working as a Data Science Fellow at Center for Data Science at New York University, was recently awarded for two of his works submitted in ISMIR 2014 conference, Taipei, Taiwan. The annual conference of the International Society for Music Information System Retrieval (ISMIR) is the world’s leading interdisciplinary forum on accessing, analyzing, and organizing many types of digital music.

Brian received both the Best Presenter awards – for Best oral presentation and Best Poster Presentation, for two different submissions: –
1.) Best Oral Presentation: “Analyzing Song Structure with Spectral Clustering”, by Brian McFee and Dan Ellis (Professor at Columbia University).
2.) Best Poster Presentations: “MIR_EVAL: A Transparent Implementation of Common MIR Metrics”, Colin Raffel (from Columbia University), Brian McFee, Eric J. Humphrey (PhD candidate at NYU), Justin Salamon (post-Doctoral Researcher at NYU), Oriol Nieto (PhD candidate at NYU), Dawen Liang (PhD candidate at Columbia), Daniel P. W. Ellis (Professor at Columbia University).

The Interview:
Question: Which field is ISMIR concerned with and what is the strength of the gathering at the conference?
Brian: The conference attracts people involved in different fields of research, including digital libraries, signal processing, machine learning, and musicologists. The conference had a gathering of around 300 people.

Q: What is the paper that was awarded the Best Oral Presentation Award about?
B: The paper which was awarded the best oral presentation award was “Analyzing Song Structure with Spectral Clustering”. It is about analyzing the structure of the music, primarily to find the repetitions within the music structure. Its inspiration came from the Infinite Jukebox, which came up at Music Hackday at MIT in 2012. The infinite jukebox plays a song and jumps among different sections based on detected repetitions of structural components. This analysis aims to model the infinite jukebox in order to discover the latent structure of music.

Q: Tell us something about your research that earned the Best Poster Award?
B: This paper was titled “MIR_EVAL: A Transparent Implementation of Common MIR Metrics”, which describes a coordinated effort led by first author Colin Raffel, and involving researchers at NYU and Columbia University. It’s about an open source software package that we developed to provide a standardized implementation of commonly used evaluation metrics for Music Information Retrieval (MIR) algorithms. MIR algorithms are basically aimed to produce semantic information from audio signals or symbolic representations. To evaluate a system’s effectiveness in replicating the human annotations, certain metrics have been developed and accepted as standards over time. To avoid any discrepancies and incorrectness in analysis due to differences in implementations, having a standardized software package which implements the common metrics to evaluate MIR systems is quite helpful.

Q: What is your field of work?
B: My work is on machine learning in general, but motivated by applications in music analysis and recommendation systems. I tend to work on algorithms which integrate heterogeneous features, such as tags, acoustic features, and lyrics. Lately, I have been specifically working on structural analysis of music, and making new interfaces to expose and visualize structure.

Q: Have you worked on something like Shazam?
B: Not really. Shazam is primarily about fingerprinting, that is, finding an exact match for a query recording. My research interests skew more toward finding clusters and collections of songs which are similar, but not identical, to a query.

Q: So do you have any help in your work, since it involves understanding of both musical and technological aspects?
B: Yes. I am working with people in both the Center for Data Science (CDS) and Music and Audio Research Laboratory (MARL) at NYU. I am collaborating with both faculty and graduate students on a variety of projects, and generally looking forward to exploring the intersection of general data analysis tools and musical domain knowledge.

If you would like to discuss more about any of the topics discussed above or about Research on Music Analysis with Dr. McFee, please contact him at brian.mcfee@nyu.edu.
Story by Rishabh Jain, master’s degree candidate and writer for the Center for Data Science.

Download release