Multisensory Experience of a Poem

This is a story of how Rebecca Xu (Syracuse University) and Sean Zhai (that's me) progressively transformed a poem into multisensory experiences using ASMR, sound, vocal, visual texture, and abstract patterns.

The Beginning

It started when Rebecca showed me the book The Illumination Project, gifted by Emily Vey Duke at her reading at the Light Work studio. Rebecca suggested creating an animated visual interpretation of the poems.

The book contains illustrations by Shary Boyle and poems by Vey Duke, intertwined and inspired by each other. Rebecca and I each picked three favorites from the book. Among the choices, there was one poem in common, which was Bloodie Write an Anthem.

Artist Intention

Bloodie is the name of a young girl who travels across fantastical lands populated by all manner of creatures. The poem we chose situated Bloodie in the time of awakening, subtle and fragile; it almost reminded me of Proust’s madeleine moment which would lead to her self-discovery.

Poetry is a language art; we would present the poem in its original form first, then going through a process of decomposition and reassembling based on linguistic analysis. Our goal is to explore the multisensory methods of creating an enriched expression, not to dictate a specific rendition but to preserve the ambiguity and induce a heightened experience.

In his series of lectures titled Unanswered Questions at Harvard University, Leonard Bernstein drew the comparison in parallel for the creative process of making music and the poetry. Bernstein stated that a line of the poem could be constructed from a more descriptive deep structure using transformational grammar. We attempted various transforms in this piece, such as text to vocal music, alphabets to geometrical forms, and timbre to visual textures.

Structure

The structure of this narrative poem is in ABA form. Section A describes Bloodie’s behavior and Section B is the lyrics of her singing. In audio-visual development, we treat the two sections differently.

In section A, the poem is heard in the actual words; visually, moving drawing lines are rendered in slow-growing or disappearing motion.

In section B, poem reading is replaced by lyricless female vocal; the graphic elements develop more vigorously to support the singing voice.

ASMR & Vocal

ASMR stands for Autonomous Sensory Meridian Response, which is a sensation caused by gentle physical stimulate, which could be triggered by sound over headphones. The intimate atmosphere of ASMR naturally fits this poem. Regan Henley contributed the ASMR sound of crushing leaves, sand rubbing, and reading of the poem.

The soprano vocal is performed by Luxin Zhang.

Visual Study

You probably know that many songwriting starts with the hook, which is a short passage or phrase that is catchy and impressive; the beginning section can come at a much later time. The reason we do visual study is similar to the songwriters, but we do it in visual forms. We need to find what inspires us before we can develop the piece.

Paul Klee famous said: “A drawing is simply a line going for a walk. ” Hand drawing is arguably the most intuitive way to make an idea visible. It actually helps me to think, and I would be a considerable amount of drawings to find what feels right. These drawings seemed to happen in a state between the conscious and subconscious.

At this early stage, I like to produce some images on the computer. This is similar to a musician tries a riff to get a feel, some programming ideas will be refined and incorporated into the final project. The images are created by drawing lines using programming.

A computer sees poems as a stream of letters. A new line or space is used to divide the stream into groups of words; a word in English has syllables, and each syllable can be further broken down to letters of vowels and consonants.

Reading a poem is equivalent to feed the stream to a program as input. When the program is a graphic engine, we can expect it to turn the poem into visual images. We can use a generative algorithm to build a graphic engine.

We can start by mapping the letters into graphics using the following rules:

• a vowel maps to a closed curve
• a consonant maps to an open curve

Imagine that once we have the graphic engine, feeding the poem into one end will produce visual images on the other end.

Graphic Engine

This is actually fun! There are many creative coding tools available, we used Processing for this project.

Perhaps we were all once be amazed by a kaleidoscope, it is more exciting with generative programming, as tuning the parameters often produce dramatically different results.

The Work

We went through 9 iterations to find the rendering we both like. The work is completed in one month of time. It is best viewed in a dark space with projections while wearing headphones.

The work exhibited in several festivals and conferences, including ISEA 2019 (International Symposium on Electronic Art).

My own experience suggests that language allows me the feeling that it can go only as far as my consciousness will take it, even though I know the opposite is true, that I go where language leads. And it leads me again and again to the sense that it is holding something back, that it contains more than I can possibly grasp, that mysteries exist, and are encountered most seductively in poems. I even feel at times that poems are the protective shell of the seductiveness of language.

— Mark Strand (American Poet)

Epilog

The creative processes are often very different, and how the artists work mostly remained a mystery. I hope more people could share their creative processes.

Cheers.