mariecreel/automatic_music_events

NSF REU: Computational Methods for Music, Media, and Minds at the University of Rochester. Faculty Mentors: Prof. Zhiyao Duan (ECE), Prof. Matthew Brown (Music) Collaborators: Christodoulos Benetatos, Christopher Winders, Ph. D.

#Automatic Rendering of Augmented Events in Immersive Concerts

Mentor(s): Zhiyao Duan (Electrical and Computer Engineering); Matthew Brown (Music Theory, Eastman School of Music)

In immersive concerts, the audience’s music listening experience is often augmented with texts, images, lighting and sound effects, and other materials. Manual synchronization of these materials with the music performance in real time becomes more and more challenging as their number increases. In this project, we will design an automatic system that is able to follow the performance and control pre-coded augmented events in real time. This allows immersive concert experiences to scale with the complexity of the texts, images, lighting and sound effects. We will work with TableTopOpera at the Eastman School of Music on implementing and refining this system.

Funding for this research is provided by the National Science Foundation, award no.1659250.

automatic_music_events's Issues

normalization of musicxml chromagram makes monophonic onset too quiet in comparison to polyphonic sections

Onset during polyphonic sections is artificially "louder" than monophonic due to normalization going over entire chromagram once filled. Probably need to normalize frame by frame rather than waiting until end of processing.

OnlineDTW thread

create thread to carry out online dtw comparison btwn two chromagrams

musicxml files with multiple parts not being processed properly

not all information from musicxml files with multiple parts is being processed properly. some information is missing from the final chromagram, i believe that the information is not being accessed properly (maybe stored deeper than monophonic information) and so the chromagrams are being left empty at parts where data is not reached.