cecilia dong

I’m a 3rd year Computer Science student from Beijing, China. As a person with Synesthesia, I’m particularly interested in how different sensations interact and how to share this experience with other people. My project is an interdisciplinary topic which involves art, psychology, philosophy, AI, statistics and cognitive science. Using technology as a tool, I’ll explore how we can visualize the sound we hear, or even visualize the feelings that we generate when hearing it, and then build these up in Virtual Reality. The process of conducting this project is also a journey of exploring what it is all about to be human in a technology driven world.

Sensorial Visualization in Virtual Reality

The aim of this project is to create an immersive experience of synaesthesia that bridges the gap between people’s visual and auditory sensation. With the ability to ‘see the shape’ and ‘feel the texture’ of the music while listening, I would like to share this amazing experience with other people.

In order to share my synaesthesia experience to others, I decided to build a VR music visualizer. The Software and devices used for this project are: (1)Unity3D: for VR design(2)VisualStudio: for editing C# Script that process music frequency data and (3)Oculus: the VR device.

I got started by choosing a piece of music that I like the most. The sound file I used for this project is called Algorithms. It is a piece of modern music without lyric, and as I listen to it I could see hundreds of transparent round particles floating around along with the music. The texture is smooth, light and slippery on the surface. As I listen to it, I feel relaxed and truly touched, and I believe this is the perfect music for my project.

Since the project involved processing music data and manipulating how the particles change, I needed to write my own scripts for building up the music visualizer. There are 4 scripts written in C#, which help capture the sound frequency, building up the space for the flow and movement of particles, manage the speed and the movement setting of the particles flow, and control the change of colours. An additional imported script is needed to provide the required library.

As I finished building up the music visualizer, I realized that I needed to add some design to truly represent my own synaesthesia experience! I used transparent particles from the Unity asset store to ‘visualize my mind’ and aligned different colours to different frequencies of the music, so that the colors would change along with the music.

Here is a screen shot of the visualizer playing out as a 2D video. If viewed in VR, it feels like you are in a planetarium for the mind. These particles float and change around you and they are so real that you feel you can catch them.

I’m truly grateful to have been given this opportunity to explore the beauty of the human mind and I’m looking forward to a further development of this project. For the next step, I will be working on collecting sensorial data from people and visualize them in VR in real time, which create an experience that’s similar to synaesthesia.