Part 1 Practice

1 Introduction
2 Muscle Machine
3 Technology
4 Concept
5 Conclusions
Part 2 Theory
6A Robo-Sound
Performance
6B Robo-Sound Performance
7 Neural Networks
8 Artificial Intelligence
9A Interactivity
9B Interactivity
10 Mapping
11 Collaborations
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Stan Wijnans

MuDanx
Home
Curriculum Vitae
Projects
Academics
Masters PhD Publications
Contact
Webdesign by
Stan Wijnans
 

10 Mapping gestures to sound in interactive performances

Five-fold path Connectivity, immersion, interaction, transformation, emergence (Roy Ascott 1996)

                                   File written by Adobe Photoshop® 4.0
                           A digital Musical Instrument representation (Wanderley 2001:16)Primary feedback

To describe the data flow in the diagram above, we firstly identify the input gesture from the outside world as a movement parameter that is being tracked by a gestural controller. Secondly it is important to be aware of existing feedback available to the performer. encompasses visual, auditory and tactile - kinetic feedback and secondary feedback relates to the sound produced by the instrument (computer) (Wanderley 2001:18). The gestural controller sensor converts the movement energy from the performer into electricity to be interpreted by an interface connected to a computer. These computer data then require a process of mapping. Mapping refers to the liaison or correspondence between control parameters (derived from performer actions) and sound synthesis parameters (Hunt, Wanderley, Kirk 2000:1). As we can read above a definition of the connection between the movement parameters in the performance and the parameters for the sound composition by the computer is an important issue. Joel Ryan notes below how the data can be treated:

Shaping Response Controller data can be shifted or inverted [addition], compressed
and expanded [multiplication], limited, segmented or quantized [thresholding]. Methods
which keep track of the history of a signal allow measurement of rates of change,
smoothing and other types of filtering to amplify specific features in the signal or to add
delays and hysteresis in the response [differencing, integration, convolution]. The rates
of data transmitted can be reduced and expanded [decimation and interpolation]. Linear
and nonlinear transformations allow the shaping or distortion of the signals to any desired
response [functional and arbitrary mappings] (Ryan 1992:5).

The next step is to create an interdisciplinary work that succeeds from both the choreographic and compositional perspectives. I mentioned before, when I described the choice making process of the sounds for Stelarc’s ‘Anatomical Exoskeleton’ (see p10), that creating a believable relationship between the gestural movements triggering the sensors and the composed sound in interactivity can be crucial. We can read below how this ‘believability’ can be created:

• An emergent integrity arises when the relationship between these two systems is ‘believable’.
• Believability depends upon gestural coherence.
• Gestural coherence is achieved through a system of mapping that mediates the two parallel structural systems (musical and choreographic).
• Musical structure emerges from dance gesture through a schema that provides for a mixture of the definable gesture-to-synthesis parameter mappings.
(Weiß et all 2000:2)

As Joel Ryan (1992) describes that this ‘mediating’ can be achieved by using the physicality between gestures and sound by a ‘retraining’ period in which dancers can create a “gesture of sound”:

… Gestures can be expanded or elaborated at any level of interpretation from the sensor data, to
'note events', to the samples of sound. Since there is no inherent stylistic concreteness the
composer is free to take any gesture and transform by whatever seems appropriate, from
one kind to another [e.g. a simple touch triggering a complex melodic form or a point to a
curve]; from the habits of one instruments to another [finger patterning to the inner
movement of synthetic harmonics]; one time transformed to another [stiffness to
gracefulness]; one tactile texture transformed to another [smooth to rough]; one level of
abstraction to another [shape to symbol] (Joel Ryan 1992:5).

 I conclude this chapter noting : “An ideal system designed for real-time sound synthesis [and interactive performance] should therefore be able to provide an environment where a performer may experiment with different gestural controllers and select the appropriate mapping strategies for the chosen controller and specific sound synthesis method [sound composition]” (Wanderley, Schnell & Rovan 2000:1). We have looked at the sound relationship with robotic machines, neural networks, artificial intelligence, interactivity, interactive improvisation and mapping. Before concluding this writing, I will consider some issues about human collaborations in art - technology projects, a subject that arose frequently during this MA research.

 

Hunt, A., Wanderley, M. & Kirk,R. (2000) 'Towards a model for instrumental mapping in expert musical interaction'. Proc. of the 2000 International Computer Music Conference. San Francisco, CA: International Computer Music Association, pp. 209--211. (2000.

Ryan, J. (1992) 'Some remarks on musical instrument design at STEIM'. Contemporary Music Review (6:1).

Wanderley, M. (2001) Performer-Instrument Interaction: Applications to Gestural Control of Sound Synthesis. PhD. thesis. France, Paris: University Paris.

Weiß, F. & Wechsler, R. (2000) 'The multi-dimensional mapping of movement-to-sound within an integrated Eyecon and Max Environment'. 10th Symposium International des Arts Électroniques, Paris.