The Animal Whisperer at the STEAM Challenge

The STEAM Challenge 2026, under the motto “Innovation under Pressure”, will bring innovative ideas and entertaining performances to Spirgarten Zurich on March 3, 2026. Six teams and individual participants will present their projects live before an audience and a jury, competing against one another. The evening will be hosted by Sara Taubman-Hildebrand, while comedians Gülsha Adilji, Reena Krishnaraja, and Zukkihund will provide additional entertainment with humorous interludes. One of the featured projects is the Animal Whisperer, initiated by Prof. Dr. Oliver Bendel. His former student Nick Zbinden (second from left in the photo, together with Ilyena Hirskyj-Douglas and Oliver Bendel) implemented the Cow Whisperer, the Horse Whisperer, and the Dog Whisperer on his behalf. Using Retrieval-Augmented Generation (RAG), a Multimodal Large Language Model (MLLM) was endowed with the required capabilities. First, the body language of cows, horses, or dogs, as well as the overall situation, is analyzed and evaluated. The system then provides recommendations for how humans should behave when interacting with the animals. At Animal-Computer Interaction 2024 in Glasgow, Oliver Bendel and Nick Zbinden were already awarded a prize for their paper “The Animal Whisperer Project: A GenAI App for Decoding Animal Body Language and Behavior”. Further information is available at innovation.zuerich/#steam-challenge-innovation-under-pressure-2.