Ars Electronica Archive
Prix
Prix Map
Pictures
Videos
Publications
Special Projects
Women in Media Art
Prix
The Prix Ars Electronica Archive is a collection enabling search and viewing of all the submissions since 1987. The award-winning projects are documented with catalogue texts and audio-visual media. All other submissions can be searched by title/artist and displayed with year, category in list form. Please cite the credits (artwork name, artist and photographer) and only use the materials if your article is related to Ars Electronica.
Squarepusher / Terminal Slam
Daito Manabe,
Kenichiro Shimizu
Original: Squarepusher / Terminal Slam (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-2.jpg) | 1920 * 1920px | 2.1 MB
Original: Squarepusher / Terminal Slam (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-1.jpg) | 1920 * 1920px | 2.1 MB
Original: Squarepusher / Terminal Slam (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-8.jpg) | 1920 * 1920px | 2.2 MB
Original: Squarepusher / Terminal Slam (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-4.jpg) | 1920 * 1920px | 2.5 MB
Original: Squarepusher / Terminal Slam (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-5.jpg) | 1920 * 1920px | 2.5 MB
Original: SQUAREPUSHER_TERMINAL_SLAM_VIDEO-9.jpg (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-9.jpg) | 1920 * 1920px | 2.5 MB
Original: SQUAREPUSHER_TERMINAL_SLAM_VIDEO-13.png (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-13.png) | 1920 * 1920px | 4.3 MB
Original: SQUAREPUSHER_TERMINAL_SLAM_VIDEO-11.jpg (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-11.jpg) | 1920 * 1920px | 2.8 MB
Original: SQUAREPUSHER_TERMINAL_SLAM_VIDEO-20.png (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-20.png) | 1920 * 1920px | 4.7 MB
Original: SQUAREPUSHER_TERMINAL_SLAM_VIDEO-19.png (SQUAREPUSHER_TERMINAL_SLAM_VIDEO-19.png) | 1920 * 1920px | 3.9 MB
Terminal Slam is a subversive music video where all the ads around the city are deleted or replaced with the ones related to the artist Squarepusher. The inspiration came from Daito Manabe's thought that the scenery in the city will become rewritable with AR/MR glasses in the near future, as seen in the video. The three main techniques used to create the video utilizing machine learning technologies are:
- Object Detection (to recognize the position of objects in the image)
- Semantic Segmentation (to classify the segment of the image into categories based on the meaning of each pixel by labeling every pixel)
- Image Inpaint (image restoration technique)
Also, masks are generated from those data to add optical camouflage effects in post-processing such as glitch effects only on the areas where humans are. Since such detection to extract only ads is not yet possible with current machine learning technologies, some operations had to be done by hand. But it is expected that these operations also can be automated in the near future.
We first started from organizing video and sequential image data and processed them prior to production using python and OpenCV, a library for computer graphics. Next we proceeded to object detection using YOLO and semantic segmentation using DeepLab and YOLACT to be added to offline editing of the video. This process was all done by TensorFlow and PyTorch. This provides the data of where the people or objects exist in each frame of video. However, billboards and advertisements are still difficult to detect automatically with machine learning approaches, so they are extracted manually. These extracted advertisements are adjusted visually so that their visual styles match with other footage processed with object detection and semantic segmentation. Python and OpenCV were adopted for this post-production. Based on object detection and semantic segmentation, mask data were generated and inpainting techniques were applied to those masks. Inpainting is a technique that allows you to "paint" only a specific area of a picture or video. This is done with Python and TensorFlow. These materials are exported with sound to be audio- reactive using openFrameworks (C++) and veneer. Finally the source video, midi data, and footage added with glitch effect are combined for final rendering. This is done with openFrameworks and FFmpeg.
Links: https://research.rhizomatiks.com/s/works/squarepusher/
- Object Detection (to recognize the position of objects in the image)
- Semantic Segmentation (to classify the segment of the image into categories based on the meaning of each pixel by labeling every pixel)
- Image Inpaint (image restoration technique)
Also, masks are generated from those data to add optical camouflage effects in post-processing such as glitch effects only on the areas where humans are. Since such detection to extract only ads is not yet possible with current machine learning technologies, some operations had to be done by hand. But it is expected that these operations also can be automated in the near future.
We first started from organizing video and sequential image data and processed them prior to production using python and OpenCV, a library for computer graphics. Next we proceeded to object detection using YOLO and semantic segmentation using DeepLab and YOLACT to be added to offline editing of the video. This process was all done by TensorFlow and PyTorch. This provides the data of where the people or objects exist in each frame of video. However, billboards and advertisements are still difficult to detect automatically with machine learning approaches, so they are extracted manually. These extracted advertisements are adjusted visually so that their visual styles match with other footage processed with object detection and semantic segmentation. Python and OpenCV were adopted for this post-production. Based on object detection and semantic segmentation, mask data were generated and inpainting techniques were applied to those masks. Inpainting is a technique that allows you to "paint" only a specific area of a picture or video. This is done with Python and TensorFlow. These materials are exported with sound to be audio- reactive using openFrameworks (C++) and veneer. Finally the source video, midi data, and footage added with glitch effect are combined for final rendering. This is done with openFrameworks and FFmpeg.
Links: https://research.rhizomatiks.com/s/works/squarepusher/
Director, glitch effects and interaction designer: Daito Manabe (Rhizomatiks)
Film and editing director: Kenichiro Shimizu (PELE)
Machine learning engineer: Yuta Asai (Rhizomatiks)
Video export tool developer: 2bit
Effects artist: Aya Takamatsu (Rhizomatiks)
Ad graphic designer: Kaori Fujii (Rhizomatiks)
CG director: Junichi Ebe (Freelance）
Effects Supervisors: Kenta Katsuno (+Ring), Takeshi Ozaki (+Ring)
Effects artists: Mikita Arai (Freelance), Masaki Takahashi (Freelance）
Digital artists: Yuki Hirakawa (+Ring), Yu Onishi (+Ring), Kenta Hasegawa (+Ring), Ayaka Yamaguchi (+Ring), Takeya Kamimura (+Ring), Ryuichi Ono (Freelance）
CG producer: Toshihiko Sakata (+Ring)
VFX artist:Y oshinobu Okino (Nomad）
Colorlist: Felipe Szulc (Nomad)
Cinematographer: Kazuki Takano (TRIVAL)
Cinematographer / Operator: Takuya Higa (Cyanworksandfilms)
MIDI data designer: Kyoko Koyama
Cast: SARA (ELEVENPLAY)
Film Producer: Chikako Nagai (PELE)
Producer: Takao Inoue (Rhizomatiks)
Film and editing director: Kenichiro Shimizu (PELE)
Machine learning engineer: Yuta Asai (Rhizomatiks)
Video export tool developer: 2bit
Effects artist: Aya Takamatsu (Rhizomatiks)
Ad graphic designer: Kaori Fujii (Rhizomatiks)
CG director: Junichi Ebe (Freelance）
Effects Supervisors: Kenta Katsuno (+Ring), Takeshi Ozaki (+Ring)
Effects artists: Mikita Arai (Freelance), Masaki Takahashi (Freelance）
Digital artists: Yuki Hirakawa (+Ring), Yu Onishi (+Ring), Kenta Hasegawa (+Ring), Ayaka Yamaguchi (+Ring), Takeya Kamimura (+Ring), Ryuichi Ono (Freelance）
CG producer: Toshihiko Sakata (+Ring)
VFX artist:Y oshinobu Okino (Nomad）
Colorlist: Felipe Szulc (Nomad)
Cinematographer: Kazuki Takano (TRIVAL)
Cinematographer / Operator: Takuya Higa (Cyanworksandfilms)
MIDI data designer: Kyoko Koyama
Cast: SARA (ELEVENPLAY)
Film Producer: Chikako Nagai (PELE)
Producer: Takao Inoue (Rhizomatiks)
Daito Manabe(JP) is a Tokyo-based artist, interaction designer, programmer, and DJ. He launched Rhizomatiks in 2006. Co-director of Rhizomatiks Research since 2015 and professor at Keio University SFC. Manabe's work in design, art, and entertainment takes a new approach to everyday materials and phenomenon. His practice explores the potentialities inherent to the human body, data, programming, computers, and other phenomena, thus probing the interrelationships and boundaries delineating analog and digital, real and virtual.
Kenichiro Shimizu (JP) is a video director and a VFX artist who uses CG, VFX, AR, and VR effectively for high quality visual effects, e.g. in adverts and film. Awards include the ACC Grand Prix / Minister of Internal Affairs and Communications Award and the Journalist Award. In 2018, he launched his own company Pele Co., Ltd., working with video installation for events or live shows and vision production.
Kenichiro Shimizu (JP) is a video director and a VFX artist who uses CG, VFX, AR, and VR effectively for high quality visual effects, e.g. in adverts and film. Awards include the ACC Grand Prix / Minister of Internal Affairs and Communications Award and the Journalist Award. In 2018, he launched his own company Pele Co., Ltd., working with video installation for events or live shows and vision production.