I need to get my hands on this:

The Audiovisual Environment Suite (AVES) is a set of five interactive systems which allow people to create and perform abstract animation and synthetic sound in real time. Each environment is an experimental attempt to design an interface which is supple and easy to learn, yet can also yield interesting, infinitely variable and personally expressive performances in both the visual and aural domains. Ideally, these systems permit their interactants to engage in a flow state of pure experience.

The AVES systems are built around the metaphor of an inexhaustible and dynamic audiovisual “substance,” which is freely deposited and controlled by the user’s gestures. Each instrument situates this substance in a context whose free-form structure inherits from the visual language of abstract painting and animation. The use of low-level synthesis techniques permits the sound and image to be tightly linked, commensurately malleable, and deeply plastic.

(via Creative Generalist)Very nearly what I’ve imagined here.

Here’s something similar which when I first saw the link, thought was the same project:

PANSE is an acronym and stands for Public Access Network Sound Engine. It’s a streaming audio program with a built-in tcp server. It’s meant to be an open platform for experimental interactive audio-visual netart and is open to all. So-called “modules” (clients) can be created using Flash, Java, Perl or whatever else you can think of. Messages can be sent to it to control the highly flexible audio that is set up as two 16 step sequencers, a monophonic synthesizer and an effects generator. But it also streams out numerical data about the audio being played. This data can be used to control visual representations. It’s very interesting to see how the design of an interface effects the way people interact with such a project. As with my previous projects, PANSE is multi-user based, so if more than one person is interacting with it at the same time, they will see and hear what the others are doing. This is why I prefer to call them modules rather than clients. It’s like a modular synthesizer where seperate units control seperate aspects of what’s going on. In PANSE, not all of the interfaces allow control over all parameters. In fact, currently there is only one interface that allows control over all of the different parameters.

(via Anne)