Kooker in School

A Research Topic Overview 

The last decade has seen a flourishing of new human-computer interaction methods, particularly in the arena of computer music. New technologies are allowing “non-musical” people to enjoy the experience of creating music.

Many of these techniques emphasize collaboration. Installations like the Reactable encourage multiple players to gather around a table, placing and moving blocks to interact and create sound. Devices like the iPhone, with wireless connectivity, are allowing players to collaborate remotely, or at least from opposite sides of the room. (See MoPho, iDiMP, and Smule’s iPhone Ocarina app for examples.) Much of the joy of playing these new instruments comes from discovering them with other people.

This discovery process is currently being fueled in large part by the advent of multitouch technologies. There are a few different ways to produce a multitouch interface; for example, the iPhone uses capacitive technology, while the Reactable uses an IR camera mounted underneath a glass surface and image processing software to recognize the objects (and fingers) on the table. In either case, it is clear that modern computer music instruments need multitouch input devices, or need a clear reason for not having them.

An area with lots of room for growth, however, is portability. Most new instruments and interfaces are hard-wired to computers running Processing or Pd. Projects like iDiMP and Ocarina put both the interface and the musical synthesis engine inside the same device (in this case, the iPhone). Junguk Cho’s work in FPGA image processing could help in this effort to make portable, versatile electronic instruments.

These noteworthy developments in electronic musical instrument interfaces make the present an exciting time to work in this field. Below, I list important projects and people, both in academia and in industry.

Links

• Reactable: The main inspiration for the project.
• ReacTIVision: The open source software project based on the Reactable project. The Reactable team took the software responsible for processing the camera feed and producing OSC messages, forked it, and open sourced it.      
  

  Diagram showing ReacTIVision within the Reactable project

• iDiMP: A previous class project. In CSE 237B, my partner and I created a real-time, networked music performance application for iPhone and iPod touch. It utilized the multitouch input capabilities of the devices, and also pushed the limits of the devices’ networking and processing abilities.
• Craft: A successful project using an ATmega88 microcontroller to generate real-time audio and video. Maybe this could be used as a video source for heavily stylized (i.e. low-fi) UI.
• MoPhO: Ge Wang at Stanford’s CCRMA directs the Mobile Phone Orchestra, a repertoire-based ensemble using iPhones as the primary musical instrument.
• SLOrk: Stanford Laptop Orchestra. Featured on Apple’s website.
• Ocarina: Smule’s simple music app for iPhone with impressive discovery features.
• Processing: Started as a project at the MIT Media Lab. A programming language and IDE that takes on an “electronic sketchbook” model. Many Reactable-ish projects are written in Processing.
• Wiring: “Processing for Microcontrollers.” Alpha-quality, probably not useful for this project, but interesting nonetheless.
• Chuck: “Strongly-timed, Concurrent, and On-the-fly Audio Programming Language.” An ensemble of electronic musicians sit at their laptops and write code in real time, producing music.
• CRCA@UCSD: Center for Research in Computing and the Arts. Working on new live performance techniques for computer music, among other things. Well known for Pd (pure data) graphical data flow programming environment.
• Sound Blocks: Masters thesis for an MIT Media Lab student. (Mind, Music, and Machine Group)
• Junguk Cho: Postdoc working with Ryan Kastner. His Motion/Vision Integrated Embedded System might replace the ReacTIVision software.
• Arduino Playground: Lots of examples of people building interfaces for computer music out of Arduino devices. One user says: “With the use of open design tools like Arduino and Pure Data we are able to hack almost everything and turn it into a musical instrument.”