Ars Electronica Archive
Prix
Prix Map
Pictures
Videos
Publications
Special Projects
Women in Media Art
S+T+ARTS
STARTS is an initiative of the European Commission to foster alliances of technology and artistic practice. As part of this initiative, the STARTS residencies foster pioneering collaborations and the STARTS Prize honors the most inspired results in the field of creativity and innovation at the crossings of science and technology with the arts. STARTS is funded through various schemes provided by the European Commission.
MetaPhase: a contrapuntal dialogue between a pianist and her avatar in the metaverse
Giusy Caruso,
LWT3
MetaPhase is the artistic output of a cross-sector encounter between avant-garde pianist and researcher Dr. Giusy Caruso (BE) and the innovative start-up LWT3 (IT). This joint research was driven to explore the creative potential of data processing, human-machine interaction, and biotechnological applications in a XR performance to enhance performers’ expressiveness and audience fruition. The core is a wearable easy-to-use prototype developed by LWT3 able to acquire bio-signals (surface electromyography sEMG). The biometric metadata collection is shared via the LWT3 data platform accessible to performers/scholars/citizens.
This first experimentation of motion tracking and VR technology on stage engages both performers and audience in a groundbreaking live music co-creation which showcases the virtual METAPHASE process: the biological cellular splitting that here refers to the pianist duplication playing with her virtual agent in a meta-ecosystem.
The pianist performs by wearing a suit with light-reflecting markers and biosensor to allow the system to track her gestures and the related muscles effort in real-time. Then, the pianist wears the Oculus Rift to enter in the meta-performance-scene developed by LWT3 with Unity platform and performs the counterpoint part of Steve Reich’s Piano Phase for two pianos by interacting with another avatar-pianist playing the first part of the piece. This second virtual human is animated by the real pianist’s expressive motion previously recorded together with the audio track on a Yamaha Disklavier piano. LWT3's wearable interfaces acquire bio-signals such as sEMG from the real pianist to modify visuals, video effects, and audio parameters.
Spectators are involved in an immersive XR performance by selecting viewpoints, viewing data in real time or in postproduction together with the pianist. This interdisciplinary collaboration between science-technology-art wants to expand the possibilities of a futuristic and phygital techno-aesthetics.
Links: https://lwt3.com/project/giusycaruso/
This first experimentation of motion tracking and VR technology on stage engages both performers and audience in a groundbreaking live music co-creation which showcases the virtual METAPHASE process: the biological cellular splitting that here refers to the pianist duplication playing with her virtual agent in a meta-ecosystem.
The pianist performs by wearing a suit with light-reflecting markers and biosensor to allow the system to track her gestures and the related muscles effort in real-time. Then, the pianist wears the Oculus Rift to enter in the meta-performance-scene developed by LWT3 with Unity platform and performs the counterpoint part of Steve Reich’s Piano Phase for two pianos by interacting with another avatar-pianist playing the first part of the piece. This second virtual human is animated by the real pianist’s expressive motion previously recorded together with the audio track on a Yamaha Disklavier piano. LWT3's wearable interfaces acquire bio-signals such as sEMG from the real pianist to modify visuals, video effects, and audio parameters.
Spectators are involved in an immersive XR performance by selecting viewpoints, viewing data in real time or in postproduction together with the pianist. This interdisciplinary collaboration between science-technology-art wants to expand the possibilities of a futuristic and phygital techno-aesthetics.
Links: https://lwt3.com/project/giusycaruso/
Concept and pianist (Royal Conservatoire Antwerp, IPEM – University Ghent): Dr. Giusy Caruso
LWT3 Team:
Hardware and data engineer: Dr. Paolo Belluco
Designer and VR director: Samuele Polistina
VR developer: Andrea Randone
Fashion and product designer: Luigi Sossi
With support from: Royal Conservatory of Antwerp, IPEM – University Ghent, Yamaha Music Europe
LWT3 Team:
Hardware and data engineer: Dr. Paolo Belluco
Designer and VR director: Samuele Polistina
VR developer: Andrea Randone
Fashion and product designer: Luigi Sossi
With support from: Royal Conservatory of Antwerp, IPEM – University Ghent, Yamaha Music Europe
Dr. Giusy Caruso (BE) is a concert pianist and postdoctoral researcher focused on studying human-machine interaction (HMI) for the creation of futuristic multimedia formats and music performance analysis. Chariwoman of the CREATIE research group at the Royal Conservatoire Antwerp and affiliated researcher at IPEM-Ghent University, she is Official Music Advisor of LWT3 Society Milan.
LWT3 (IT) is an innovative company founded by Dr. Paolo Belluco (sensors, biosignals and machine learning) and designer Samuele Polistina (UX/UI, product design). With expertise in data analysis, visualization, IoT infrastructure development, and human-machine interaction solutions, LWT3 works in their Data Driven Performance Lab in Milano across three pillars: Industries workers, Sport Science, and Music and Performing Arts.
LWT3 (IT) is an innovative company founded by Dr. Paolo Belluco (sensors, biosignals and machine learning) and designer Samuele Polistina (UX/UI, product design). With expertise in data analysis, visualization, IoT infrastructure development, and human-machine interaction solutions, LWT3 works in their Data Driven Performance Lab in Milano across three pillars: Industries workers, Sport Science, and Music and Performing Arts.