Soft Architecture: WYSIWYG - As an extension of the research work conducted with the Topological Media Lab (TML), Sha Xin Wei and his team are creating textile objects such as wall hangings, blankets, scarves, and jewelry that create sound as they are approached or manipulated. These sonic blankets can be used for improvised play.

A phonetic pun on the old acronym for What You See is What You Get from the era of the Graphical User Interface, WYSIWYG (for wearable, sonic instrument, with gesture) draws on music technology, dance, children’s group games, textile arts, and fashion. Created first and foremost to sustain social play for people of all ages, WYSIWYG allows players to express themselves whether enjoying time in a park, dancing at a club, passing the time during a long car trip, or just playing at home.

The research is being carried out in collaboration with Marcelo Wanderley, an associate professor at the McGill University Schulich School of Music in Montreal, and draws on Wanderley’s research into the gestural control of sound synthesis and new interfaces for musical expression. The custom-designed digital instruments embedded in the cloth sample movement to transform ambient body movement and freehand gestures into new sounds or “voices” associated with a player or transmitted to other players in the vicinity. These devices can also be embedded into furnishings or other types of objects. In addition, they can store and re-synthesize sounds by nuancing them using data transmitted by nearby sensors. The research project therefore targets the creation of a series of devices – some made from soft material – that will react in different ways to proximity and contact, movements, noise characteristics, and the progress of the game itself. The sonic behavior of the devices are designed in the spirit of games such as hide-and-seek and blind-man’s buff and also work well with a variable number of players in both ad hoc and rehearsed situations.

When the project was launched in November 2006, the WYSIWYG team experimented with a prototype ”blanket” able to sense how it is handled. During the presentation, eight people manipulated this photo-sensitive blanket to produce a spatial sonic landscape. In July 2007, dancers performed a semi-choreographed movement improvisation around a 20’ suspended “tapestry”and a 6’ “tablecloth” woven with conductive thread on a Jacquard loom by Joey Berzowska’s XS Labs.

With WYSIWYG, Sha and his team intend to develop other architectural-scale sensate cloths that function as agents co-performing with dancers and as image-bearing, kinetic surfaces in other performance contexts.

The WYSIWYG team:

Prof. Sha Xin Wei — PI
Prof. Marcelo Wanderley — PI
David Gaultier — mechatronics & feature extraction programming
Freida Abtan — sound instrument creator (software)
David Birnbaum — sound instrument creator
Elliot Sinyor — sound instrument creator
Harry Smoak — assistant project technical coordinator
Doug van Nort — gesture/sound feature extraction, mapping
Rodolphe Koehly — physical materials advisor