Wednesday, December 6, 2023
Researchers at the University of Nottingham have developed a musical robot that can help disabled musicians improvise and access instruments in ways that were previously impossible.
Jess+ was developed as part of the DigiScore project, the first large-scale collaboration between the University’s music department, Orchestras Live and Sinfonia Viva. The AI robot helps musicians with disabilities collaborate as part of mixed ensembles and improvise in the creation of live scores where dexterity is a barrier.
The robot was created by Craig Vere, professor of music and computer science at the University of Nottingham, and was used in a series of workshops with physically disabled musicians Jess Fisher, Claire Barbra and Deirdre Bensick of Sinfonia Viva.
During the workshop, the robot moved and made marks on paper in response to real-time sounds produced by live musicians. The musicians interpreted these movements and marks as notation material and responded by producing sounds like a graphic score. The central research question of the project was: “How can we build an embodied AI system that facilitates the co-creation of improvisational ensembles by disabled and able-bodied musicians, such that the expressiveness of all musicians is amplified and connected in a co-creative system?”
As DigiScore’s Principal Investigator, Professor Vear has utilised his expertise in the fields of music, digital performance, creative technology, AI, creativity, gaming mixed reality and robotics as the focus of the Jess+ project.
It’s been an honor to work with incredible musicians to develop an AI that will benefit and change people’s lives. At a time when creative AI is perceived as taking something away from musicians, we’ve developed an AI that extends, amplifies and transforms musicality, creativity and inclusion. We believe this approach can be used in the future to empower people who face barriers to making music and expressing themselves in supportive, inclusive and uplifting ways,” said Craig Via, professor of music and computer science.
Musicians with disabilities face significant barriers to music-making. One of these barriers is that while non-disabled people can make music in many different ways, disabled musicians can’t, requiring new instruments, new creative processes, and new hierarchies of “success.”
Another issue is how to get a mixed ensemble of disabled and non-disabled musicians to improvise together in a way where dexterity is not a limiting factor. Yet another barrier is that disabled musicians may be excluded from the creative process of having sheet music written for them. All these obstacles can leave creative potential untapped and set up hierarchical measures of creative engagement, limiting the potential contributions of disabled musicians.
In addressing these questions, the research team, along with project partners Sinfonia Viva, a UK-based orchestra and education organisation, and Orchestras Live, a national producer that brings moving orchestral experiences to communities across England, were able to use AI and Professor Vear’s digital score concept to create a new way of making music.
Findings from this project revealed that the musicians developed unexpected and unique relationships with Jess+: they each found it to be a great listener with its own creative “voice,” and they each felt connected to the system.
It was a non-judgmental, accepting attitude that participants said fostered a new expressive freedom and confidence in taking musical risks when improvising.
Image courtesy of BBC R&D. Musicians from left: Jess Fisher, Claire Barbra, and Deidre Bensick
For Jess, a musician, the system has allowed her to express emotions that she is unable to express with her current digital setup. She repeatedly referred to the system as her “friend” and “storyteller,” and said she felt augmented by it, allowing her to express her own emotions through the music.
“I wanted to explore that part of myself and express the feelings I have inside of me outwardly through it,” Jess said.
A recent performance by the Jess+ team took place at BBC Studios in London as part of the Bridging Responsible AI Divides (BRAID) launch event. BRAID is a three-year national research programme in partnership with the BBC and the Ada Lovelace Institute, bringing together UK-based researchers from the arts and humanities sectors to advance responsible AI innovation.
You can watch a video of the BBC performance here.
Story Credit
For further information please contact Professor Craig Vear in the School of Music, University of Nottingham ([email protected]).
Notes to editors:
About the University of Nottingham
Ranked 32nd in Europe and 16th in the UK in the QS World University Rankings: Europe 2024, the University of Nottingham is a founding member of the Russell Group of research-intensive universities. Studying at the University of Nottingham is a life-changing experience and we pride ourselves on unleashing the potential of our students. With a pioneering spirit embodied in the vision of our founder Sir Jesse Boot, we have pioneered establishing campuses in China and Malaysia as part of a global network of education, research and industrial engagement.
The University of Nottingham has been named Sports University of the Year in The Times/Sunday Times Best University Guide 2024, the third accolade since 2018, and has also been named in the Daily Mail’s 2024 University Guide.
The university is one of the best in the UK for research strength and is ranked 7th in the UK for research power according to REF 2021. Home to discoveries such as MRI and ibuprofen, our innovations are transforming people’s lives and addressing global issues such as sustainable food supplies, ending modern slavery, developing greener transport and reducing dependence on fossil fuels.
The University is a leading employer and industry partner both locally and globally, with our graduates the second most sought after by the UK’s top employers, according to High Fliers Research’s 2022 Graduate Market Report.
We are partnering with Nottingham Trent University to lead the Nottingham University Initiative, a pioneering initiative that brings together the city’s two world-class universities to improve levels of prosperity, opportunity, sustainability, health and wellbeing for residents of the city and region we are proud to call home.
In other news…