HARVEY MUDD COLLEGE

Multimodal Interaction for Musical Improvisation:
From Algorithms to Interactive Systems

Abstract

Mimi is a multi-modal interactive musical improvisation system that explores the impact of visual feedback in performer-machine interaction. The Mimi system enables the performer to experiment with a unique blend of improvisation-like on-the-fly invention, composition-like planning and choreography, and expressive performance. Mimi’s improvisations are created through a data structure called factor oracle. Mimi’s visual interface gives the performer and the audience instantaneous and continuous information on the state of the oracle, its recombination strategy, the music to come, and that recently played. The performer controls when the system starts, stops, and learns, the playback volume, and the recombination rate. Mimi is not only an effective improvisation partner, it also provides a platform through which to interrogate the mental models necessary for successful improvisation.

I will describe the factor oracle data structure and related algorithms that together power Mimi’s improvisation engine. I will then highlight the challenges that arise when implementing these algorithms in the context of a complex interactive system, and describe the solutions implemented in Mimi.

The Mimi project is joint work with Elaine Chew, Isaac Schankler and Dennis Thurmond.

Mimi videos and publications are available on the project website.

Biography

Alexandre R.J. François is a Visiting Associate Professor of Computer Science at Harvey Mudd College.

From 2004 to 2010, he was a Research Assistant Professor of Computer Science in the USC Viterbi School of Engineering at the University of Southern California. In 2008-2009, he was a Visiting Assistant Professor in the Department of Computer Science at Tufts University. From 2001 to 2004 he was a Research Associate with the Integrated Media Systems Center and with the Institute for Robotics and Intelligent Systems, both at USC.

François’ research focuses on the modeling and design of interactive (software) systems, as an enabling step towards the understanding of perception and cognition. He is creator of the Software Architecture for Immersipresence (SAI), a general formalism for the design, analysis and implementation of complex software systems. His open source Modular Flow Scheduling Middleware (MFSM; mfsm.sourceforge.net) provides a cross- platform, multi-threaded implementation of SAI’s abstractions.

His interdisciplinary research projects explore interactions within and across music, vision, visualization and video games. He was a 2007-2008 Fellow of the Radcliffe Institute for Advanced Study at Harvard University, where he co-lead a music research cluster on Analytical Listening Through Interactive Visualization.

Leveraging the SAI/MFSM framework, his experimental courses in software development, graduate and undergraduate, pool the efforts of the entire class on a single, ambitious collaborative project. One such project, an interactive game titled An Ant’s Life collectively designed and prototyped by 13 students at Tufts University, was a finalist in the SIGGRAPH 2009 Research Challenge competition.

François received the Diplôme d’Ingénieur from the Institut National Agronomique Paris-Grignon (France) in 1993, the Diplôme d’Etudes Approfondies (M.S.) from the University Paris IX - Dauphine (France) in 1994, and the M.S. and Ph.D. degrees in Computer Science from USC in 1997 and 2000 respectively.