Year
2025-Present
Recognition
Flagship project (1 of 6)
Interactive audio-haptic perception for blind and low-vision users
This research is funded through the AI Institute for Artificial and Natural Intelligence (ARNI), a $20 million multidisciplinary initiative established by the National Science Foundation (NSF).
PI: Kathleen McKeown, Columbia
Co-PI: Tatiana Aloi Emmanouil, CUNY; Nikolaus Kriegeskorte, Columbia; Brian Smith, Columbia; Carl Vondrick, Columbia
Abstract: A picture is worth a thousand words, but only if the viewer can experience it. Our project aims to help blind and low-vision (BLV) people experience works of art, including their sensory and semantic dimensions, their emotional valence, the artist’s intentions, and other contextual information. Our project brings together researchers from vision, language, and interaction design, all informed by cognitive science, to develop AI Art Immersion for the visually impaired. Our work is inspired by the human visual system, which gives us an instant sense of our surroundings and lets us explore a complex scene interactively with eye movements and internal shifts of attention, enabling us to rapidly grasp the layout, the objects and characters present, and how the elements relate to each other.
Partnerships:
Helen Keller Services for the Blind
National Gallery of Art
New York Hall of Science
This year:
Exploring how the system can best provide interactive exploration of images using haptics, speech, and sound
Project Updates:
Nominated to present a poster at the 2026 ARNI Site Visit
