Our original interface, in all human endeavor, is the moving body and its senses. All extensions we may create digitally are and must be based in our primary sensuous being. Thus, how we ARE and how we move have everything to do with what sort of interfaces we will make.
Demonstration of Performance Environment
We are a collaborative team that creates computer-mediated interactive music and dance performances, which utilize motion tracking systems to create bodily immersive music-making environments. Within this environment we improvise music and visual art through full-bodied dance movement. Our performance aesthetic necessitates a computer interface within which one physically dwells. This spatial/sonic/visual environment is an extraordinary interface in which the computer mediates the relationship between a persons movement and the resulting music. This makes for a rather magical theatrical experience, and allows us to achieve a unique integration of music and dance, where the music is sounded as a direct result of the movement, and movement aesthetic is conditioned by the act of being the music. Additionally, video effects are responsive to movement, which further influences and inspires movement choices and qualities. Thus, our interface becomes a medium through which we can unify the sound, movement, and visual arts.
Integrations: Art, Man, Machines and Environments
Integration of Artistic Mediums
Historically, in traditional dance performances, the choreography, music and scenic elements occur simultaneously, yet remain fundamentally separable. Within our interface, these are simultaneously created of and by each other as a true whole. And while we initially set out to integrate music and dance, what we unexpectedly found while creating and playing the instrument we call Leonardos Chimes,is that this interactive environment is also integrative for the individual who plays it. We discovered an extraordinary self-movement feedback experience that fosters holistic integration of mental and physical faculties, developing kinesthetic, spatial, and musical aptitudes.
Integration of the Person in the Environment
One of the most curious effects we experience is that moving in this environment has the effect of making one feel more whole, more unified in thought and action. We are intrigued with the notion of making systems that improve our integration as human beings. If the interaction between man and machine brings about the betterment of the man, is the machine even more wholly part of our living humanity?
People are used to interfacing with the environment by taking in information with the eyes and ears, and effecting change with the face and hands [see Figure 1]. Notice in fig. 1 how prominent hands and face are in proportion of cortex motor units. The hand is itself larger than the entire remainder of the lower body. One may experience a radical shift when beginning to interface with the world with a more balanced and whole use of the bodymind.1 This is the special domain of the dancer, and is why, typically, computers and dancers may not tend to mix. A large-motor-movement interface demands integrative development of the bodily aspect of our mind. New perceptual doors open with such a shift in perspective.
Figure 1: A.C. Guyton: Features are distorted in relative size to represent proportion of cortex motor units dedicated to each body part. Hands and face are most prominently represented in cortex.
A common idea in the debate about the benefits and costs of computers in human life is that the use of computers is likely to cause bodily atrophy, or a split in the relationship between a persons intellectual and physical sides. Imagine the stereotypical computer geek huddling over his terminal, or the child spending hours collapsed on the couch playing video games, eyes pulled inward and down toward the TV and hands tightly gripping the controller. Our interface more ideally addresses three issues in psychophysical well-being. First it discourages isolation of the mental from the physical, by requiring full-bodied movement to interface with the computer system. Second, it encourages full range physical extension outward into space instead of only forward, narrowly focused contraction. And third, movement occurs within a multi-sensory, interactively responsive environment, which provides feedback to the player to help integrate thought and action as a whole.
Feedback Loops: A Mechanism for Synaesthetic-Sensory Integration
This instrument creates and amplifies a synaesthetic-sensory interactive response field. Because of the spatial arrangement of the instruments active areas, the player is encouraged to move with full range of motion to activate sonic and visual responses. When the player hears and sees the results of his movement, he experiences a feedback for the shape, location, timing and quality of the movement. This feedback tunes the players spatial, proprioceptive, and kinesthetic awareness. Furthermore, the player is increasingly inspired to dance to the musicdance to his musicwhich generates yet more sound, inspiring the player to move in response. Thus, the experience for the mover is a feedback loop in which his movement makes music that inspires him to move in ever-more creative ways to make more sound, and so on
Skilled, full range motion, that is, dancing, constitutes the instruments virtuosic technique. Once basic bodymind understanding of the cause/effect relationship with the response field is established, the feedback loop encourages the player to become more skilled in movement, spatially, temporally and qualitatively.
If making music is the goal, everything we do sonically in Leonardos Chimes could be done with fingertips on laptops. So why bother to make it take up so much space? The reason is the importance of involving more of the human body. Most modern technology is miniaturized, from an assumption it is preferable to minimize movement activities. We made the computer take up an enormous amount of space to require more movement. Involving more of the bodymind in the experience leads to a more enveloping feedback response and imparts an organic quality to the music that seems lacking in much contemporary computer music.
A More Integrative Interface
Through our artistic and aesthetic explorations we have come to realize that this interface has implications beyond the theatrical, into the realms of physical transcendence, integrative therapies, and the experience and contemplation of both the real and the philosophical relationship of man to machine.
Innerspace and Interface
The feedback loops are a subjective experience that constitute the innerspace of this interface. When we began to feel this sensory feedback, we noticed that previous intellectual distinctions between movement/sound, self/environment, man/computer began to blur in our experience, replaced by one extraordinary, melded sensation. Thus, one might argue that this interface not only connects previously disparate parts, but also expands the innerspace into the outerspace, eliminating separations between person, computer, and the surrounding environment.
In contrast to present definitions of interface which suggest the connecting or communicating between separate things,2 we are beginning to prefer the use of the word medium, because it suggests communion rather than mere connectivity.3 Using this medium elicits a sensation that ones innerspace is integrated with the outerspace. This sensation has a profound effect upon the very notion of the computer and its role in the human experience.
The Man in the Machine and the Machine in the Man
The integrative nature of this experience begins to blur philosophical and lived distinctions between man and machine. By giving the computer sight via cameras, and thought via complex parameters for responses, the player begins to feel as though he is relating with an entity, as opposed to using a tool. There develops a sensation of moving in a liminal space, across an increasingly porous membrane of mind and machine, somewhere in between the muscles and the microchips. One begins to feel like they are simultaneously in the machine, and that the machine is in them. It is a dancean exchange of sensation and mutual response.
Other Applications of Such Environments
We invented Leonardos Chimes as an instrument for performance art. It is a medium in which ones imagination is allowed to stretch and experience sensory immersion and interplay in a super-elastic, magic plastic world of motion, sight and sound. As a work of theatre, we made the clear decision that this environment should have some of the normal cause/effect relationship of real world environments. Otherwise, it is too illogical and confusing. But we also chose ways of bending the rules along certain parameters to make it more intriguing. This sensation of stretching the constraints of physical reality has an uncanny effect on the mind, allowing it to suspend old assumptions and seek new pathways, ideas and solutions. Playing this instrument has been invigorating to us, not only for our art, but also for our imaginative capacity in general.
We believe this instruments capacity to inspire creativity could be extended beyond its original artistic intention, because it encourages ones imagination to play outside the normal boundaries of cause and effect. Creativity is the ability to suspend habitual assumptions about how things work to reveal new options, allowing previously unrelated ideas to interact in novel ways. A persons neuro-motor system has habits about how to think about the world. These well-worn pathways are conditioned by the physical world, where cause and effect is relatively predictable. In a computer-mediated environment, some of the worlds usual rules can be bent or broken (e.g. the ability to pass your hand through the virtual drum one is beating or hearing a beautiful sound that is improbably emanating from your body in motion). This allows the mind of the player or audience member to stretch into new domains of possibility.
Since the brain is working outside its normal frame, one feels the sensation of possibility as a generalized perspective, applicable to other areas where imagination, creativity and problem solving are useful. Such environments may have profound effect on the neurological level, increasing flexibility, both physically and mentally; the mindbody is the real super-elastic, magic plastic.
Integrative Body Therapies/Physical Education & Rehabilitation
This system offers many possible educational and re-educational applications for developing basic sensory-motor skills, and use in integrative body therapies. In many approaches to rehabilitation and physical education, one of the central issues is cultivating sensory awareness and perceptual accuracy. In particular, proprioceptive and kinesthetic sensitivities must be developed for proper self-perception in time/space movement skills. Because systems like our interface provide such extraordinary feedback about location and timing of movement to the player, it gradually increases awareness of the body in relationship to space and time, fundamental to all movement skill. We suggest that this sort of system could be valuable to the improvement of peoples physical coordination. This nicely counters traditional notions of computers contributing to an increasing dissociation of our minds from our bodies. This computer interface actually facilitates integration of ones faculties rather than isolating them. By giving the person a higher order end goal, that is, creating beautiful sound, all lower level faculties are organized into whole thought/actions.
Topics for Discussion/Questions/Further Implications
The experience of playing this instrument not only allowed us to break previous assumptions about the parts of a dance performance (dance, music, visual scenery) and make unique, intriguing art, but also opened many wider questions. Following are some additional notes and philosophical musings on ideas about interface, systems, and integration (of machines, of man and of both) toward new, larger wholes.
What is interface?
Interface is defined as a point where two systems meet and interact,4 and a device or program enabling a user to communicate with a computer. But both of these definitions maintain the dualistic sense of fundamentally separate things merely interacting.
After playing the instrument we began to experience sensations that challenge dualistic descriptions of the separateness of things and opened philosophical sounding questions such as: Where am I? Am I only here, where my body is, or am I also there, in the computer? Where is the computer? Is it only there, on the table, or is it also here, inside of me in my mind, my body, my action and response? What is the computer? Is it only a thing I use, or is it a more integral extension of myself?
Is the word interface the ideal term for this or any system? Why not medium?
Medium is defined as: 1. An intervening substance through which impressions are conveyed to the senses or a force acts on objects at a distance. 2. The substance in which an organism lives or is cultured. 3. The material used by artist, composer, writer. 4. Storage method for digital data.5
Ours is a spatial, sonic, photonic, kinesthetic/neurological mediuma synaesthetic medium through which a person kinesthetically experiences Self. This instrument is a unique medium for us to relate to each other through space, time, touch, sound, and sight.
Why does playing this instrument feel unifying, or integrating for the player?
When playing this instrument, I dont feel the usual physical boundaries, compared with, say, striking a drum. Tactility might typically give one a sensation of separateness from the thing being touched. But because I cant feel a surface to touch, I begin to sense the space proprioceptively; place becomes an internal sensationthis brings the outer to the innerspace (arguably this is always true, even with tactile touch, but perhaps becomes more curious or obvious a sensation when it happens in the absence of tactile feedback). Thus, the only way to feel a location or time in this space is to feel me more accuratelyan idea which we call self-referential space.
Self-referential space and time
Time and space are fundamentally two ways of describing the experience of measuring movement. The original reference point for any measurement of space or time is the human bodys own dimensions and the kinesthetic/proprioceptive meaning of here vs. there and the duration of change between two perspectives. Without movement (or perception of change), there would be no experienced time or space.
Playing the instrument tunes kinesthesia and time/space skill. Time and space are bodily sensations first, and intellectual ideas second. So one implication of playing this instrument is that it tunes this original reference for space and timethe Selfand therefore is an intriguing tool for improving ones accuracy of time/space movement skills.
What does extension and large movement have to do with the integrative experience?
Higher-Order Integrative Tasks
Movement control skills involved in playing this instrument are not done merely for some arbitrary objective of physical control (an artistic flaw to which dance techniques occasionally fall prey). Rather, such control is also the necessary means to play music and make video art, which becomes a higher-order organizing objective for ones movement skills. Ones bodymind not only manages balance, but is also engaged in evaluation and execution of a spontaneous musical experience, integrating areas of thought and action into one problem-solving activity. Perhaps because the task of balancing on one foot and stretching in two directions is not the end goal, but merely part of an overall solution, balance and control become easier because they are employed toward higher order goals. The basketball player leaps very high indeed, but not because he is focused on the leap; his larger objective (to put the ball through the hoop) naturally organizes all necessary lower order skills to achieve a higher end.
Although we have not yet experimented with different spatial arrangements of active areas, it occurs to us that this system offers flexibility of configuration, which would allow us to design it to encourage movement for a variety of shape, line, or physical control goals.
What exactly do we mean when we say that playing this instrument integrates the man, and the man in the machine?
Innerspace experience affects the Interface, which in turn affects the Innerspace, which affects the Interface they both are mutually influential aspects of a whole.
There is a phenomenal6 sensation when I am in this environment that is as if my mind is no longer only in my head, but also extends into my body, and then spreading out, surrounding me in space. It is as if my brain is a pool in which I am immersed, and yet at the same time, the computer-mediated field is also immersed inside me, via my senses and response/actions to and in that field.
Arthur Koestler coined the term holons to refer to that which is whole in one context and yet is also only part of a wider whole in a larger frame or system (Wilber, 36). Ken Wilber says, In any developmental sequence, what is whole at one stage becomes merely a part of a larger whole at the next stage. A [whole] letter is part of a whole word, which is part of a whole sentence, which is part of a whole paragraph, and so on. He speaks of expanding links[and] an increase in unity and wider identities as a way to illuminate the meaning of Koestlers concept of holarchies (Wilber, 36). Holarchies are like hierarchies, but holistically expanding. When a holarchy expands to higher levels, it extends (transcends) AND includes all previous levels, analogized by a set of nested Russian dolls. In the Human Being, the holarchy expands from elements to compounds, to proteins, to organelles, to cells, to organs, to organ systems, to brain, to mind, and so on. In the computer, it unfolds from elements to compounds, to materials, to chips and circuits, to hard drives, to operating systems, and so on. Interface is just a way of trying to describe a fuzzy zone where the holarchies of computer and person overlap and begin to merge into still larger holarchies, now with the machine and the man operating as new, larger holon (whole, but still only part of yet larger systems, such as the internet, which is whole, but still only part of a wider community of people, and so on).
So one more time what is interface?
When interface is properly functioning, it does not merely connect entities at a boundary; it eliminates boundaries/separations between entities and enfolds entities into one another. The purpose of a good interface is to eliminate separations and make a seamless, unified sensation of thought transforming into action/response unifying us with the world. In this sense, ones mind is extended by body, which is further extended into and through computers to locations in space (and virtual spaces) of an ever-extending meaning of world.
Interface is really more of a concept than a thing. So our ideas about interfacethe way we describe and think about itdefine its capabilities and its limitations. If we only think of interface as a way of connecting separate things, then it implies that the things (person, machine, environment) are still fundamentally separate, but connected. But perhaps the real goal of interface is to unify separate things into new, whole, integral systems.
Guyton, A.C. 1971. Basic Human Physiology: Normal Function and Mechanisms of Disease. Philadelphia: W.B. Saunders Co.
Husserl, Edmund. 1960. Cartesian Meditations: An Introduction to Phenomenology. Translated by Dorian Cairns. The Hague: Martinus Nijhoff Publishers.
Koestler, Arthur. 1976. The Ghost in the Machine. New York: Random House.
Merleau-Ponty, Maurice. 1962. Phenomenology of Perception. Translated by Colin Smith. London: Routledge.
Wilber, Ken. 2001. The Eye of Spirit: An Integral Vision for a World Gone Slightly Mad. Boston & London: Shambhala.
Oxford American Dictionary, as implemented in the dashboard widget of the Apple Macintosh OSX interface.
1 Bodymind is one term among several variations that attempts to correct dualistic notions that the body and mind are fundamentally separate but connected. It is an attempt to speak more properly of the whole, integrated nature of a person, mental and physical aspects functioning as a single continuum.
3 Definition: Medium is 1. An intervening substance through which impressions are conveyed to the senses or a force acts on objects at a distance. 2. The substance in which an organism lives or is cultured. 3. The material used by artist, composer, writer. 4. Storage method for digital data.
(All definitions are from the Oxford American Dictionary, as implemented in the Dashboard widget in the Macintosh OSX interface.)
6 I use the term phenomenal here in reference to phenomenology, a philosophical investigation of the subjectivity of lived experience, proposed by Edmund Husserl and Maurice Merleau-Ponty. A central intent of their philosophy was to attempt to create a rigorous science but one that offers an account of space, time and the world as we live themto give a direct description of our experience as it is, without taking account of its psychological origin and the causal explanations which the scientist, the historian or the sociologist may be able to provide. (Merleau-Ponty, vii).