The Developmental History of MIDIART^tm Live.

Software that Visualizes MIDI Based Music.

by Tony Fragiacomo

The Initial Idea:

In the mid 1980’s I became an avid MIDI enthusiast and began following the evolution of this interesting technology. During this time, a great deal of new MIDI music software was being developed for PC’s. Some of this software was even capable of playing MIDI music while simultaneously rendering the standard musical notation in real time. When I first witnessed this type of software, I began wondering if it would also be possible to develop software capable of transforming each sonic event contained within a MIDI data stream into more artistic and dynamic two-dimensional visual objects, using various computer software algorithms. This approach would be very different than simply transforming the music into standard musical notation, since musical notation is composed of static or fixed visual objects of a very precise nature. My idea was to transform MIDI music into dynamic animated visual objects, all in real time.

Four years ago, I began to learn about the features and capabilities of the Microsoft Windows operating system. It was at this time I realized the software I had envisioned could be developed utilizing the multimedia extensions provided by the Windows 3.1 system.

Using the multimedia extensions, I believed it would be possible to create various software algorithms that would be notified whenever an individual musical note (MIDI event) was being played in real time. During the period of the note being played, each algorithm could render a dynamic animated visual object on the computer screen. Each software algorithm would be free to render a visual object in a unique way. Then, a different algorithm could be assigned to render visual objects for each musical instrument being played within the MIDI based musical composition.

The Idea Comes to Life:

In late 1993, I began work to prove whether it would be possible to create the software I had envisioned. I decided to develop the software using a combination of the C and C++ computer languages. I knew the user interface could be developed in C++, but lower level algorithms would best be developed using the more efficient C language.

I started by locating a public domain software algorithm that was capable of parsing the MIDI messages contained within a standard MIDI data file. I enhanced this public domain software in order to create a list of MIDI messages that would be sent to both the MIDI sound generator, as well as the visual object generating algorithms I planned to develop. I packaged all the software that dealt with the processing of the MIDI data, including the transmission of the data to the sound generator device driver and the timing of the transmitted data, into a Windows DLL or Dynamic Link Library. This package was developed fairly quickly.

Next, I decided to create a algorithm for rendering some simple visual animated objects. I started by developing an algorithm that placed a solid colored circle inside the display window whenever a MIDI note was being played. The color of the solid circle was based on the MIDI channel, and the size of the circle was based on the MIDI note velocity. The size of the circle would quickly shrink over time, and would only be visible between the MIDI note on and note off events. Thus, all the visual objects being rendered would only be seen while the individual notes were sounding.

While experimenting with this simple algorithm, I realized that it would be more interesting to manually control the various parameters of the algorithm while the music was playing. I assigned function keys on the computer keyboard to control various parameters of this algorithm. For example, two function keys would control the average size of the colored circles being rendered, and two more function keys would control the speed at which the circles would shrink or expand over time. Once this algorithm was developed, I began seeing for the first time that some incredibly interesting visual images could be generated in real time from the MIDI based music. I knew then that my work was just beginning.

The First Product:

During the time I was experimenting with this simple visual algorithm, I decided to determine if anyone else had developed multimedia software that could transform MIDI music directly into interesting and artistic visual images. I spent some time searching the Internet, as well as the Compuserve on-line service, in order to try and find software products similar to the one I had envisioned. I found that there had been some attempts to produce computer graphic images controlled by a digitally sampled audio stream. This type of software normally breaks down the audio stream into frequency bands or components, and then tries to manipulate psychedelic computer graphic images based on the amplitude of the frequency components contained within the audio stream. I realized this method did not do a good job of synchronizing the visual images to music contained in the audio stream. This is because the frequency components of various musical instruments overlap in time, and there is currently no method of recognizing or separating out the individual musical notes being played by the musical instruments within the digitally sampled audio stream. With a MIDI audio stream, the individual musical notes played by each musical instrument are available as data elements that can be directly mapped to visual events over time. But none of the software I had seen attempted to create visual images from the MIDI data directly.

With this knowledge in mind, I decided to further develop my idea, and in the process, create a software product I could market via the shareware distribution method. Using this distribution method, I hoped to recoup some of the cost of developing the software.

I decided to call my first product MIDIART, which would emphasize the artistic nature of the software, along with the fact that it utilized MIDI in a whole new way. I also decided that the first version of the product would simply generate visual images from pre-recorded standard MIDI files only. Later, I would set out to develop another product which would be capable of generating visual images from MIDI data being played live by on stage musicians.

MIDIART Version One:

After deciding to develop my first shareware software product, I set to work defining the user interface for this new product. I wanted the user interface to be simple yet flexible, in order to allow the user to become a visual artist, capable of creating unique visual presentations using MIDI. I decided to make the software respond to single key presses from the computer keyboard. Each key press would then change some aspect of the music or the visual images being rendered on the screen. I developed key press commands to start, stop, and rewind the music, as well as commands to select individual MIDI channels, in order to control volume levels and instrument selection on each channel. In order to control the parameters of the software algorithms creating the visual images, I decided to utilize the computer keyboard function keys.

Another decision I made early on, was to call the my visual software algorithms Image Generators, and to group these algorithms together in separate DLL packages. This would give me the ability of adding new Image Generator DLL packages without modifying the other components of the software. (The user interface and the MIDI processing DLL mentioned above.) The job of each Image Generator would be to take incoming MIDI events, along with function key press events from the user, and use this information to create visual objects and control the animation of these visual objects.

After creating a standard Image Generator API, (Application Programming Interface) I set out to repackage the simple algorithm I described above into the very first Image Generator. I then gave this Image Generator a unique name, and made it accessible from the user interface of the program. It was now time to begin developing more Image Generators.

The Evolution of Image Generators:

The first several Image Generators I developed were very simple, each one rendering and animating two-dimensional visual objects on the screen while each individual musical note was actively sounding. I allowed the color of these visual objects to be assigned based on the MIDI channel, but the visual objects were of the same form regardless of the MIDI channel. Later on, I developed a special Image Generator called the Mixer, that would allow each of the simple Image Generators to be assigned to a specific MIDI channel. Thus, the visual objects rendered could be different for each MIDI channel.

As time went on, I continued to develop more complex Image Generators, and experimented with different algorithms and techniques. One technique involved creating a static image, and then cycling the colors assigned to this static image. The color cycling was triggered by each musical note, and the amount of color cycling could be controlled by the user. Another technique involved plotting a series of simple lines whose path was determined by trigonometric functions. Each musical note would trigger a new series of lines, which would result in complex standing wave patterns. Yet another technique involved generating points using a fractal Iterated Function System algorithm. The individual musical notes would then determine the color of the points plotted within the complex fractal image. With this Image Generator, the fractal function could be modified by the user by manipulating numbers contained in a data file. Thus, different fractal image shapes could be defined.

To give the user even more control over the images being rendered by the music, I also decided to develop the Bitmaps Image Generator. This Image Generator allows the user to assign a sequence of bitmapped frames or images to each MIDI channel. The Image Generator then cycles through these bitmapped frames very quickly during the time that each MIDI note is sounding. This technique creates animation’s that are definable by the user. The function keys assigned to control this Image Generator are used to position the bitmapped images on the computer screen, as well as control the speed of animation.

Adding Performance Features:

As I myself became a user of the product I was developing, I quickly realized that the software could be improved through the addition of a few performance control features. The first feature I added was the capability of recording each function key press made during the performance of a MIDI file. After recording these key press events, which affect the visual images being generated, the software is then capable of re-triggering these key press events in the same sequence and at the same periods in time, whenever the same MIDI file is replayed by the user. This feature gives the user the ability to create interesting visual transformations that are made in sync with the music, and record these transformations such that the same transformations are reproduced whenever the MIDI file is performed a second time.

Another performance feature I added was the capability of storing the Image Generator visual parameters to computer disk. I stored the visual parameters in a separate disk file associated with the selected MIDI file. Thus, whenever the MIDI file is re-selected for playback, the Image Generator visual parameters are restored. Later on, I allowed for multiple visual parameter files. When multiple visual parameter files are stored, the user is allowed to select which parameters to use during the performance of the MIDI file. This selection can be made in real time during playback.

MIDIART Live:

After developing the MIDIART product and releasing it as shareware on the Internet and other on line services, a number of users began sending me their feedback and ideas concerning the product. The most common request I was receiving was to add the ability of visualizing MIDI data being generated from a live source such as a MIDI keyboard. In 1995 I began work on just such a capability, and decided to name the new product MIDIART Live. I wanted MIDIART Live to be able to visualize live MIDI data along with MIDI data contained in a standard MIDI file. I decided early on to mix these two sources of MIDI data. It took only a week or two to add this new enhancement.

Next, I realized it would be nice to give the musician performing with MIDIART Live the ability to control the visual images directly from a MIDI keyboard or any external MIDI controller device. I decided to add a feature where by any incoming MIDI message could be used to trigger a MIDIART software command key. This feature would have three main advantages. First, a musician could control the MIDIART visual Image Generator parameters via any MIDI control device, without having to be directly located next to the computer keyboard. Second, it would allow the MIDIART software to be controlled by none keyboard controllers, such as pitch and modulator wheels, slider bars, toggle switches, and key aftertouch controls. Third, a single MIDI message could be programmed to trigger multiple MIDIART command keys. All of these features all now incorporated into the MIDI external control component of the MIDIART Live software.

The MIDIWorks Player:

After developing MIDIART Live, another MIDIART user gave me a new idea. The user wanted to know if it would be possible to package a MIDIART performance in such a way that it could be easily distributed from his World Wide Web page on the Internet. I thought this was such a good idea that I set out develop another product based on software components I had already developed.

My idea was to create a MIDIART player program similar to other player packages on the Internet, such as MPEG video players and the Real Audio player. I could have developed this player as a Netscape plug in component, but I decided early on that I did not want to tie into any specific Internet browser technology, and I also wanted the MIDIART player to function properly outside of any Internet browser. Instead, I decided to package a MIDI file along with various MIDIART performance data into a single small Windows executable file. When this small file is executed by a user, the data contained within it is unpacked and placed in a temporary location on the computer disk. Once the data is stored on disk, the player program is then envoked automatically in order to recreate the MIDIART performance. I decided to call this new player program the MIDIWorks Player, and the small executable files containing the data, MIDIWork files. MIDIWork files are created using the MIDIART Live software. The ability to create MIDIWork files was one of the later enhancements I made to the MIDIART Live software product.

Today, anyone can download the MIDIWorks Player software from my World Wide Web page located at www.midiworks.com, and after installing this software, can play MIDIWorks files by simply selecting them directly from the same Web page. In order for someone to develop their own MIDIWork files, they need to download, install, and learn to use the MIDIART Live software.

MIDIWork files are really an extension of MIDI files. MIDIWork files embed standard MIDI files along with MIDIART performance data. In the future, these files could be freely distributed to others just like MIDI files are now distributed widely on the Internet.

Future Possibilities:

In order for MIDIART to keep pace with the rapid advances in computer hardware and software technology, MIDIART must evolve to incorporate some of these new technologies. One obvious evolution would be to incorporate 3D software algorithms in order to generate real time 3D visual images. MIDIART could incorporate its own 3D rendering software algorithms, or it could utilize 3D components of existing technologies such a VRML. (Virtual Reality Modeling Language.) I am now in the process of working with a VRML developer in order to experiment with connecting real time MIDI events to VRML 3D scenes.

Another future evolution of MIDIART would be to create visual objects in the physical world, as opposed to the virtual world on the computer screen. My idea would be to use real time MIDI events to create and control 3D sculptures created by other artists. This would involve interfacing the computer to a variety of physical sculptural creations. The possibilities of creating new forms of visual art with MIDI seem to be endless, and of course are only limited by the authors imagination.

About the Author:

Tony Fragiacomo studied music, physics, and computer science at Saint Olaf College in Northfield Minnesota. In 1983 he received his BA degree in physics. After college, he began a career in computer software engineering. In the mid-eighties he became a avid MIDI enthusiast and began learning how to develop MIDI software using the C programming language. In 1993, he realized that his rather bizarre idea of converting music directly into visual images could be accomplished by utilizing the multimedia extensions of the Windows 3.1 operating system. By the end of 1993, MIDIART was born.

Today the author lives in Minneapolis Minnesota, and earns a living as a contract software engineer specializing in real time control and data acquisition applications. He has been developing software for both UNIX work station platforms and PC platforms.