Movement Sonification and Its Application in Pedagogical and Artistic Contexts
How can technology support a creative and embodied studio-based practice? How can somatic practices and epistemologies be shifted and enhanced through technological experimentation? How do sensitivities to and perceptions of sound shift through technological engagement, and how can we describe and learn from those discoveries?
In this Oral Introduction to the practice-research project Movement Sonification and Its Application in Pedagogical and Artistic Contexts, co-developed with technologist Simon East between 2014 and 2019, Maria Kapsali talks us through her work at the intersection of sound studies, movement studies and technology. We are invited to think about movement sonification and its potential for learning, for creative expression, and for sense-making across different domains of cultural and artistic practice, and learn about Sonolope, the technology developed as part of the project, and the multiple possible areas of research it opens up.
Maria Kapsali is Lecturer in Physical Performance at the University of Leeds. Her research focuses on performer training and embodied and somatic practices; her most recent volume is Performer Training and Technology: Preparing Our Selves (Routledge, 2021).
[00:02:41] WHAT IS THE PROJECT’S TITLE?
The title of this project is Movement Sonification and its Application in Pedagogical and Artistic Contexts. This is a practice-research project that developed over a period of five years between 2014 and 2019 in collaboration with technologist Simon East.
[00:03:05] HOW DID YOU COME TO MAKING THIS PROJECT? / WHERE DOES IT SIT IN RELATION TO YOUR PREVIOUS WORK AND INTERESTS?
The project emerged out of a scheme at the University of Leeds, that is called Creative Labs, and that was back in 2014. As part of the scheme, an academic is partnered up with an industry partner across a theme or a shared interest, and I was partnered up with technologist Simon East who specialises in the development of applications for mobile phones. The good thing about this project is that it doesn’t expect any kind of final product at the end of it, it’s just about for the academic and the external partner to meet together and just see what kind of ground they share or what kind of ideas they would like to take forward. So Simon and I considered a number of things that relate to our respective areas. I was thinking a lot about performer training and performance-making, and what might be the use or the function of digital technologies within that and Simon was also thinking about the creative development and application of the mobile phone. So between us we kind of talked about – I think, initially, we had about four or five ideas, and one of them was about movement sonification. And this one kind of stuck with us, it kind of felt the more productive one, or the more enjoyable one. So gradually, we took it further and progressively, we developed ‘Sonolope’, and this is – if you want – the name we gave to a particular prototype for developing a movement sonification system. Now, what is specific about Sonolope is that it uses the sensors that are embedded in the mobile phone in order to pick up data from the user’s body, specifically data related to the user’s axis and position in the space. And then these data are translated into sound, so when the user is moving, sound is being created. And what the system does is that it creates the impression that it’s the actual movement that generates the sound.
With Sonolope, Simon and I – as well as a number of other researchers and artists and professionals that came on the project – we began to explore how movement sonification could operate. First of all, from a technological point of view: so, how could we get that technology to work in a way that was supporting the actual creative and embodied process in the studio? But we also started thinking about what might be the creative, the artistic, and the pedagogical application of this within a performance-related context. And we did this through a series of workshops, participation in public events, in festivals, installations, conference presentations over a period of five years.
Where does the project sit in relation to my existing research interests and expertise? Well, up till that point my research had been situated in the field of performer training, and I was also – and still am – a practitioner of various movement disciplines and somatics. So I think the first thing I identified in movement sonification was that it enriched and enhanced movement experience in ways that I hadn’t encountered before. And I also found that it allowed me to revisit specific aspects of my training and look at them in new ways: for example, it opened up a whole new way of thinking about the generation of imagery for example, or a whole new way of experiencing space. The interesting thing is, you know, that the project is interdisciplinary and it involves movement, it involves sound, it involves digital technology, it involves a digital device. And what is interesting is that I didn’t have any particular expertise in sound, I don’t play a musical instrument. I think I was coming to sound with a kind of knowledge and habits that a female, fully abled, white body would bring to this – no particular expertise or knowledge, if you want. And what I found interesting was that what also developed through the project was my sensitivity to sound and my ability to listen. I started engaging with various bibliographies, for example, with ideas and practices around deep listening. But what was interesting was that after a few years, even when I now move to music or sound without movement sonification, just when it plays from a CD player, is that my experience now with sound and music has completely shifted, and I feel the sound and the music very much [more] into my body than I did before the project. So I think what is interesting is that it shifted sensitivities and perception within the body, what I would pick up in a studio or in a situation, what I would respond to. And I think this is the area of my previous practice that responded to and fed further. My interest has always been – as I’ve said before, my research is situated within the field of performer training – and my interest has always been in movement disciplines and somatics, and their creative potential, but also the way they structure or inform human embodiment, and what they reveal about the way human embodiment and subjectivity is understood today. So working with movement sonification made two things possible at once: on one hand, working with sound and movement allowed us to combine these two things in new ways, but also we were able to use this hybrid experience between the sonic and the kinaesthetic to understand how the body is understood and is talked about in different fields and within different contexts.
[00:11:06] WHAT IS THE PROJECT ABOUT? WHAT IS THE MAIN OVERARCHING THESIS OR ARGUMENT?
Looking back, I would say that what the project is about is: about exploring the potential of movement sonification for learning, for creative expression, and for sense-making across different domains of cultural and artistic practice. It is also about interrogating the way mobile devices can be used in new ways, or ways that fall outside of established agendas and marketing imperatives. So to give you an example: I said before that Sonolope is working with the sensors that are embedded in the mobile phone – one of those sensors is the gyroscope. And what the gyroscope does is that it knows which way your phone is facing, so when you want to look at a picture for example, or read the newspaper, the text is facing the same way your face is, so you’re not looking at the picture upside down. Now, this is related to a very particular practice of seeing and reading. And in some ways, it is very ocular-centric. It also imposes if you want, or assumes, a particular kinetic relationship between the phone and the body of the user, the phone and the user’s relationship to space. So in order to read the newspaper, I have to have the phone in my hand, I have to be facing the phone, and so on and so forth.
Now, with movement sonification we’re using an existing functionality of the phone for very, very different purposes and in very different ways – with movement sonification, the phone can be attached on any part of the body. It’s not about looking at the phone, although we are using the sensor that is very often related to acts of looking. So I was very interested in the way movement sonification could complicate or extend further the way a very, very particular, and very commonly used digital device can be extended further. So the overarching thesis of the project – I think it’s about the combination between the somatic and the sonic, the kinaesthetic and the sonic, that can be engendered in practices of movement sonification. And I suppose, the argument is that these practices can offer rich, deeply-satisfying embodied experiences, which might have creative and theoretical potential, and can also serve as ways for establishing or exploring new ways for working with digital technology.
[00:14:25] WHAT DOES THE PROJECT CONTRIBUTE TO THE EXISTING BODY OF KNOWLEDGE IN ITS FIELD?
Sonification is an emerging field of practice and research – movement sonification can be seen as a subcategory within it. There are also the kind of cognate explorations that may be termed embodied generative music or gesture-controlled music. I think it’s an emerging field and an interdisciplinary field of practice and research that is growing quite quickly. So to my mind, there are three key areas that the project can contribute to.
It contributes to the emerging field of sonification and foregrounds the somatic dimension of movement sonification. So quite a lot of projects and other types of instruments that have been developed operate or if you want, focus on the use of hands and arms, whereas with Sonolope the idea was that the whole body can be engaged in the practice. In this way, we can think about movement sonification as a training tool for movement, for developing movement awareness, just because the generation of sound may take movement in new areas and new directions that the user would not necessarily have gone into. It can also be used as a training tool for training performers in intermedial practice. So one of the things about intermedial work is that the performer operates within a space that has a lot of stimuli – the performer may be working with text, may be working with light and sound and all kinds of cues and technologies that may be operating autonomously. And I think what movement sonification does is that it can develop this kind of multi-focal attention and ability to be responsive to non-human partners, such as a sensor that operates in response to someone’s movement or speaking. I think it has a lot of potential to be used as a dramaturgical tool, because the user’s movement can generate sound but it can also generate pre-recorded text, and this opens up possibilities about how a particular text is presented to an audience, or even how a character becomes constructed. And then, finally, it can be used as an interpretive tool in galleries and museums, and this is the last aspect of the project that was explored. And the way it does that is by offering non-visual ways for visitors to engage with artwork. And you can imagine, this can be very important for people that are blind or are partially sighted, and they would like to engage with the so-called visual arts. So movement sonification offers a mode of engagement that brings in different sensory modalities. It brings in sonic and kinaesthetic, and it doesn’t solely rely on the visual.
The second area where I think movement sonification can contribute is this wider area of the body or theorisations of embodiment. In my experience, I found that working with movement sonification can offer insights around the experiences and theorisations of embodiment, but also it brings attention – or at least, it has brought my attention – to how central vision has been in conceptualisations of the body within 20th-century philosophy and theory, and somatic practice as well. So I think what it can offer is: it can offer the opportunity to displace the centrality of the visual and open ways for foregrounding the kinaesthetic and the sonic. There is a very evocative question by Jean-Luc Nancy I would like to read out to you. In a little book called Listening he says: ‘What does it mean for a being to be immersed entirely in listening, formed by listening or in listening, listening with all his [sic] being?’ [2007: 4] But we can say ‘with her being’ as well. My sense is that movement sonification can offer a unique answer to this question, and say that it can allow a unique intersubjective, playful, experiential way to engage with this question, and offer answers to it.
And finally, the third area where I think this project can contribute, is this wider field of mobile studies, which is – again, it’s very interdisciplinary, and we see various fields engaging with the mobile phone. So, I think this is another area.
[00:20:50] WHAT IS THE STRUCTURE OF THE PROJECT AND WHY?
As I said, the project evolved out of a series of interweavings and reiterations. The interweaving was between theory and practice, and the reiterations were the development of the practice out of a series of encounters. One pattern – which I think was kind of consistent across the five years of the project – was that we would organise closed workshops, where we would invite an artist or a company, and we would just play together for a few sessions and then there would be a bigger opening. So, we would go to a festival and we would engage with the general public, or we would take place – we would participate in a particular event. It was, I think, that kind of constant movement between more focused explorations and then interactions with the general public, and very different kinds of audiences. We worked, for example, for one week at the National Media Museum in Bradford, and that was exclusively, or at least predominantly, for schoolchildren. So within a week, we saw over 500 children. The event that we did at the Leeds Art Gallery in 2018 was as part of the Being Human Festival and there, we engaged with the general public – whoever happened to walk in the gallery that day – but also, specifically with Wilberforce Trust that works with blind and partially sighted people. I think the structure kind of developed in this fractal way, and it was very much responsive to opportunities that arose, to which the project responded.
Now, the written material that has developed out of the project engages with two key aspects. So the article that appeared in 2017 discusses the way movement sonification can contribute to somatic practice and theorisations of embodiment. And the second article that is due to come out in December 2020 looks at that specific project in Leeds Art Gallery as part of the Being Human Festival, and examines the way movement sonification can offer a multisensory engagement with sculpture for audiences – for sighted, partially sighted, and non-sighted audiences. And one of the arguments of the article is that through that multisensory encounter, we can rethink what the artwork is and how we can know it.
[00:24:20] HOW ENJOYABLE/DIFFICULT WAS IT TO MAKE THE PROJECT?
[Laughs] Practice-research projects come with all sorts of joys and frustrations! I think one of the things about practice-research is that it takes so much time to organise, so before you even get to the point of doing it, all kinds of things – from booking a space, risk assessment, ethical reviews – all kinds of things need to be put in place. And then, of course, when it does happen, it’s a real opportunity for knowing in different ways, being playful and I would say, developing a very different relationship to knowledge and collaboration than what is available through desk-based research. I think what was enjoyable about all of this was that it was so deeply collaborative, and it relies on the cooperation of all kinds of partners. So, anything from being able to get people in the space to having wifi and having the technology behaving. And there were cases when none of these things could be taken for granted!
[00:25:50] WHAT IS YOUR FAVOURITE PASSAGE, CHAPTER OR IDEA FROM THIS PROJECT?
The other thing that was very rewarding about this project was the occasions when we managed to make Sonolope available to audiences that would not have encountered [it] otherwise. And I think my favourite aspect of the whole project is the expression on people’s faces when they realise that it’s their movement that creates the sound. It’s that very first moment, and I’ve seen it on the face of so many people over these years. It’s a combination – there is joy, there is disbelief, there is bafflement in there, there is playfulness, and it’s all together in the very first moment when we say: ‘Yeah, that’s you, you are making this sound!’
Transcription by Kalina Petrova
[00:00:15 to 00:02:37] Montage of video material about the use of Sonolope generated and selected by Maria Kapsali and edited by Juan Salazar
Kapsali, M. (2020) ‘Sonic Bodies: Movement Sonification and the Hybridisation of Aesthetic Experience’, Performance Research, 25 (4), https://doi.org/10.1080/13528165.2020.1842029.
Kapsali, M. (2017) ‘Making the Body All Ears: Sonolope and a Somatic Practice of Really Listening’, International Journal of Performance Arts and Digital Media, 13:1, 39-54, https://doi.org/10.1080/14794713.2017.1300446.
Nancy, Jean-Luc (2007) Listening, New York: Fordham.
International Metabody Forum, University of Brunel (2016)
Sonic Bodies in Being Human Festival (2018)
Rehearsals for House by A Quiet Word (2018)