December 5, 2016

Invention and innovation: a symphony of sound and color

Michelle ‘Misha’ Oraa Ali ’17

For many of my early years, I thought everyone experienced the world as I did. When I encounter certain stimulus, such as hearing the musical note A, it involuntarily elicits another seemingly unrelated sensation, such as seeing the color red. It wasn’t until I read about synesthesia in V. S. Ramachandran and Sandra Blakeslee’s book The Phantom of the Brain that I had a serious “Aha!” moment.

Explaining this neurological phenomenon to non-synesthetes—97 percent or so of the world’s population—has never been easy. But my journey at Mount Holyoke has led me to take a new crack at it. And in doing so, I see possibilities for my creation to help others experience music in new ways.

Inspiration

In fall 2015, I took an advanced seminar that was cross-listed between the biology and music departments. In Art, Music, and the Brain, we studied the neurobiology of visual and auditory perception, frequenting the College Art Museum and Pratt Hall, the music building. We learned to play the violin, discussed the science behind color vision, and questioned whether the perception of dissonance is universal.

Blog_Ali_Kandinsky.png

For an independent class project, I created an animation on Wassily Kandinsky. I was drawn to this modern artist because he is a fellow music-to-color synesthete. On the heels of my project, I decided to build a device that simulates the experience of synesthesia. Not for credit, not for a class, not for a thesis. Just for the love of creating. I call the machine MuSyC: Music-Synesthesia-Color. It takes audio input and converts it into visual feedback in real time.

My vision for the device is twofold. It may be an accessibility tool for those who are deaf or hard of hearing, enabling them to experience music visually in combination with tactile vibrations and feedback. It may also be an educational/pedagogical tool, and tuned to change color in response to closely related notes.

Blog_Ali_MuSyC_poster3.jpg

The first prototype of MuSyC

From dream to reality

With mentorship from the College’s Makerspace, I secured a Five College Digital Humanities micro-grant. Makerspace is an amazing resource on campus. Its staff and workshops encourage students from all disciplines to create and collaborate on projects. And the Five College Digital Humanities Program, which funds projects in support of student scholarship, is one of the Five Colleges’ best-kept secrets. With its funding, I purchased materials I needed for my machine.

Along with Cassiel Moroney McCune, then a first-year student at Mount Holyoke, and Kyoko Sano of Hampshire College, I created the first prototype at Hampshire’s February 2016 hackathon. I arrived with a brain full of ideas and a box full of hardware—and magically found students interested in music and coding.

Blog_Ali_MuSyC_1.jpgAn inside look at the first prototype

The prototype featured a row of LED lights, plexiglass, and a little mic to capture ambient sounds. Together we developed an algorithm to make the machine work as I envisioned: As the audio stimulus changes in pitch/frequency, there is a corresponding change in color. And as the audio stimulus changes in volume/amplitude, the number of activated LED lights increases or decreases.

Blog_Ali_MuSyC_poster1.jpgThe second prototype of MuSyC

The judges at Hamp Hack saw what I see: my machine is at the intersection of performance art, digital technology, fabrications, the physics of music, neuroscience, and applications for pedagogical practice. We won the Most Interdisciplinary Project award. The next prototype featured improved circuitry, a ring of lights, and a 3-D printed case with an independent battery supply. (Thanks, Luke Jaeger of Makerspace!)

undefined

Watch the second prototype of MuSyC—in action

MuSyC on the road

In August 2016, I took part in the conference of the International Association of Empirical Aesthetics in Vienna, Austria. MHC backs students’ ideas with the resources to make them happen—my travel was financed by the Harap Scholarship Fund, the Margaret C. Gould Scholarship, and the President and Dean of College funds. I presented twice: to introduce MuSyC and to share a poster on facial disfigurement research I’d conducted during a summer at the University of Pennsylvania. (Tip: if you ever read about a professor doing cool work, reach out and ask if there is space in the lab for you. That worked for me!) I shared a platform with cognitive neuroscientists, art historians, and graphic designers. I was proud to win the Best Poster prize and the Robert Francès Award for most outstanding student research contribution.

In November 2016, I presented my device at the Tech Expo at MIT Media Lab’s Hacking Arts Conference. Then, armed with fresh suggestions for improvement, I went to the event’s 24-hour Hackathon. I teamed up with students from MIT, Rutgers University, and Washington University in St. Louis who had backgrounds in electrical engineering, music, and design. Together, we rebuilt the code from scratch and made it more dynamic and sensitive to acoustic frequency—and MuSyC.Mi was born! Our team was one of the Hackathon’s 12 finalists. Thank you, Stella Yang, Grant Falkenburg, Arushee Agrawal, and Aisvarya Chandrasekar!

Freedom and future

Seeking out interdisciplinary opportunities is an extension of my liberal arts education at Mount Holyoke. I feel equipped to approach problems from different perspectives and to take ownership of my learning. The Mount Holyoke community is not one to pigeonhole people into unmovable categories. I refuse to define myself as solely a scientist or an artist or a feminist. I know I can be all three. For someone who doesn’t want life to just fall into her lap, being part of this community is the best way to chart the future of my choosing.

Ready to make magic?  Apply Now Button

Michelle ‘Misha’ Oraa Ali ’17, a Filipino-Indian international student, majors in neuroscience and behavior. She’s also pursuing a self-designed minor in graphic narrative and visual storytelling. She is a liaison for the Program in Neuroscience and Behavior and works at Archives and Special Collections, where she catalogues the College’s extensive zine collection. Since her sophomore year, Ali has done cognitive neuroscience and psycholinguistics research with Assistant Professor Mara Breen on implicit prosody and rhythm, and has looked at the effects of steroid hormone secretion on metamorphosis in fruit flies with Professor Craig Woodard. At the start of her senior year, Ali received a Five College Digital Humanities student fellowship. In the future, Ali hopes to pursue scientific research and create projects at the intersection of art and neuroscience.
Comments