Synaesthesia in Art and Science

Synaesthesia: a condition in which someone experiences things through their senses in an unusual way, for example by experiencing a colour as a sound, or a number as a position in space.

Umut Eldem’s research and works focus on exploring synaesthesia as an aesthetic of artistic expression. Using sounds, colours, and other senses as metaphors for each other his works aim to create synaesthetic worlds and experiences, drawing on the experiences of synaesthetic people. His experiments are being used to explore the aesthetic effects of combining the senses, and to create tools for synaesthetic audio-visualisation and musical performance. Umut Eldem has also given classes and workshops on audiovisual history, technique, and practice.


Synaesthetic Works


Brief Gaze of the Infinite Maze (2021)

“Brief Gaze of the Infinite Maze” is an interactive web-based installation in which the viewer can explore an almost-infinite synaesthetic audiovisual space. Fractals are infinitely complex, never ending visual patterns. In this installation, each frame is turned into a soundscape in real time using an algorithm analyzing the perception of colors and shapes on the screen. The infinitely complex patterns turn into an ever-changing world of sound: Exploring the fractals creates complementary sonic events reflecting the viewer’s unique journey in this audiovisual world. This installation has been created with the support of Screen Dive.

Click here to go to the interactive installation.


    Shallow Steps (Installation) (2021)

    Shallow Steps’ is an installation performance that explores the synaesthetic space between sound, vision, audience, and infinity. The technique and software for the translation of the visuals to sound is created through the composer’s research on ‘Cognitive Audiovisual Transformation’- an approach to synaesthetic art that prioritizes cognitive elements instead of mathematical correspondences. The self-repeating structure of fractals, especially mathematical fractals such as the Mandelbrot set, is the starting point of the work. Applying certain mathematical formulae on a visual plane creates a specific, self-repeating shape. Zooming into this shape creates more intricate patterns. Theoretically these patterns continue until infinity, ever-changing and yet always unique. In this work it is this pattern that is transformed into sound and given an audiovisual form, as an automata entity.

    Presented at the AEC Annual Congress in Royal Conservaroire of Antwerp, Belgium.


      The Road Endless (2021)

      “The Road Endless” is a collection of three audiovisual pieces written for the harpsichord and live electronics. All the pieces use audio visualisation techniques developed by composer Umut Eldem as a part of his doctoral research into synaesthesia. The starting point of the visual design is the synaesthetic experience of the performer, whose associations between colors and notes turn into synaesthetic visions on the screen. The first piece, “As Above”, references the old Visual Music aesthetics of the 1930s. “The Hit” is a dialogue between the computer generated sounds and the harpsichord. The final piece, “B of B”, is composed for the ‘Music for the Bees’ project and puts colourful bees on the screen dancing to the music. All the pieces tell an abstract story between the harpsichord, the visuals, and the audience.

      • Performed by: Liselotte Sels


      Glimmer (2019)

      “Glimmer” is based on the act of breathing, and the intricacies it reveals as one focuses on the different aspects of this natural reflex. The performance focuses on the intuitive presentation of multi-sensory art. While the work is oriented around music, a live audio visualization system, created as a part of the composer’s research, accentuates musical elements (pitch, direction, timbre, form) and transforms them into a visual component.

      • Performed by: Dries Meerts


      Hearing Glass (2019)

      “Hearing Glass” uses a computer camera to give passers-by a synaesthetic experience. The camera uses a live feed to create a soundscape that comes out of either speakers or headphones, depending on the location and the likelihood of disturbance. A transformed live feed from the camera is shown on the computer screen. The audiovisual transformation from picture to soundscape is based on Eldem’s research on synaesthesia, the experience of multiple senses at once, as the colours and shapes from the pictures are automatically transcribed in accordance with synaesthetic theory.


      Prelude (2018)

      The text of “Prelude” is based on the ancient creation myths from all throughout the world, a story told by many viewpoints across all continents, with the music, the soundscape, and the colour lights depicting the creation in tandem.

      • Performed by: Atilla Filiz

      Color FM (2017)

      “Color FM” is part of Eldem’s artworks centered around synaesthesia- the experience of music and colors together. He hid different music boxes, each with their own matching color and original music, in Mirador during the Kolla Festival. The festival-goers could seek them like a treasure hunt, or encounter into them if they went exploring on their own. Each box was placed according to the color and the associated feelings: a dangerous place for red, a place deep in nature for green, etc.



        Presentations

        • Screen Dive Live Discourse
          – ZER0|1NE II Festival 2021

        • Navigating Cross-modal Spaces in Visual Music
          UPDag #3 2021

        • Techniques and Aesthetics in Cross-modal Audiovisual Translation
          CREATIE Sessions 2021

        • Live Synaesthetic Visualisation”? Considerations in Artistically Visualised Sound
          Expanded Animation 2020 – Synaesthetic Syntax

        • The Sound/Colour Relationship in Artistic Practice: Visualising Synaesthetic Principles
          European Platform for Artistic Research in Music (EPARM) Conference 2019

        • Consolidating The Artistic And The Scientific
          Methods in Artistic Research 2019

        • Beyond The Pitch – Applying Synaesthetic Principles to The Artistic Practice
          IASAS Moscow Synaesthesia Conference 2019

        • Between The Senses: Sound, Colour, and Synaesthesia
          Newline 2019

        • Between The Senses: What Our Brains Reveal About Our Connection Between Sound and Colour
          RE:SOUND – Media Art History Conference 2019