*click to play* A haunted computer – I found it on the street in front of my house last year and had no idea what to do with it, since it was too old to install anything remotely useful. I let the weather data take over and disrupt its ecosystem. The data controls mouse and keyboard, opening and closing programs, going online, moving windows, making screenshots and saving/deleting files folders.

Weather data playing various genres of computer games. What appears to be entirely chaotic at first slowly shows certain patterns of movement and actions which change over time.

Some quick experiments with different scales of data translated into bone rotations and resulting in a vocabulary of moves and gestures. The tricky part is to keep the rotations within more or less realistic constraints.

No bone constraints

WEATHER DATA EXPERIMENTS

Several experiments I did with weather data in the first semester. In all of them, I am using the weather data to hijack computer’s mouse and keyboard. The data comes from KMNI data platform, mostly high-resolution maps of ground-based observations for the Netherlands, which includes temperatures and wind data from different sensors distributed across the country. I am animating the data in different ways and playing around with its scale and general control settings.

VOICE REGISTERS

This project initially began outside of Sandberg and in my studio, though now I am noticing more and more that it's actually in dialogue with my research, as well as with some of the classes. I’ve picked it up again over the last semester, and planning to continue integrating its bits and pieces more into my work. The project revolves around the notion of typographic voice, and the relationship between text and sound, as well as speaking, reading and writing. I am using audio recordings as design instruments, injecting individual expression and speech patterns back into digital text, patching together different tools and software for voice/sound analysis, phonetic transcription and more, and using the results to control the CSS of a text, letter by letter and word by word.

Here, a voice recording is used to control y-position per word and per letter (pitch), white space (pause duration), letter width (spoken duration), letter weight (volume) and rotation (intensity).

A voice-turned-type record of a discussion in Clara's class – in the recording, I am presenting a screenshot dump I've accumulated throughout my research. Clara sent us sound recordings and transcripts of each session, so I am planning to do more with those. Unlike some other recordings I've worked with, which are primarily readings of different kinds, these are unrehearsed, unscripted conversations, so I'd be interested in translating the spontaneity through text. In class we've been talking about the role and the agency of a secretary – something that comes up in my own research a lot, as since recently I've been obsessing over court stenography and voice capture.

Gathering input for my sonic diary in Flavia's class, I asked Sara to vocally interpret some of the texts. The prompt was to respond to typographic visualisations, without listening to the original audio. I imagine I could do more with this – perhaps visualising Sara's singing and asking someone else to respond, and again, turning it into something like the chinese whispers game.

A record of oral arguments in Happy the Elephant's court hearing – a case filed on her behalf by the organisation called Non Human Rights project. This case was part of my legal personhood research (I'll talk about it in the presentation). Not sure what I'm going to do with this yet, but I am circling around the themes of voice, performance in courtrooms and legal identities, as well as court stenography, involved equipment and the figure of stenographer.

Sara's vocal response to a part of the transcript

Some process visualisations – pitch in recording controls the y-position of letters and words.

Poetry is interesting to visualise in general because of pace and repetitions, as well as every speaker's individual interpretation of a poem's rhythm, which you can then trace back in the text.

Sara's vocal interpretation of the poem, originally read by two different speakers.

Trying to distinguish between different speakers – this part of the research is quite tedious and technical. Apparently there are many ways to do that through voice analysis, some of which are infinitely complex and some quite straight forward. So far, the most reliable way is to break the text into phrases and calculate an average pitch per phrase.

Some excerpts from Winnie-the-Pooh read by different people – as with poetry, reading children's literature involves different kinds of vocal expression and voice acting than reading prose or official statements.

Getting accurate timestamps per word and per letter from an audiorecording is very tricky – for that, I am using phonetic transcription, which breaks up each word into phonemes rather than letters. Phonetic transcription is also used in courtrooms – trained stenographers use special phonetic keyboards, transcribing at a rate of 200-300 words per minute.

Some silly little experiments with video and facial expression. The color of letters changes depending on fluctuating emotions.

I'm currently developing a quick and dirty technique to interpolate between fonts and distort them. These are just some rough first sketches, but the idea is to control the interpolation with voice, generating new letterforms throughout the text and with each new speaker.

We started by reading an excerpt from Carrier Bag Theory of Fiction while eating porridge mixed with various toppings. A bowl with oats became our first example of a container and a thematic thread throughout the workshop (oats also make an appearance in LeGuin's essay).

We used a miro board as a collective space for describing, visualising and filling the containers with content.

Afterwards, we prompted the class to collectively curate the container map by grouping, rearranging and nesting their containers. In this round, Frankie was born.

Before bringing the workshop to others, we did a test-run ourselves. We all wrote short descriptions of our containers and their features, and I translated them into a simple modular backend+frontend structure. Each container can be filled with specific types of content and is re-usable. We start with a served bowl of porridge, since that became the overarching theme of our workshop.

An additional digital booklet with instructions on how to build a backend and frontend container with its own set of rules. Since we didn't have enough time to go through all this during the workshop (and we didn't want to turn it into a coding class), we made the booklet donwloadable for those interested in more practical guidelines.

A schematic summary of backend and frontend translation, printed and handed out during the workshop.

A LEAF A GOURD A SHELL A NET A BAG A SLING A SACK A BOTTLE A POT A BOX A CONTAINER ( FILLED WITH PORRIDGE )

Workshop held together with Cassandre, Tine & Chiara for Anja's class. We borrowed the title from Ursula LeGuin’s essay “A Carrier Bag Theory of Fiction”, in which she speaks of containers as narrative devices in fiction. Taking that as a point of departure, we decided to apply this lens to thinking about the department's upcoming resource-sharing platform (also known as digital library, also known as department garden). Containers, and spatial metaphors in general, are essential in building, designing and using digital environments, so we decided to prompt the class to collectively draft a system of interconnected containers which carry different stories and meanings. Later, these container can be translated into coded structures. No matter how poetic, their properties can be used in backend and frontend development.

De Amsterdamse Studentenprotesten – reconstruction of student protests

A studio project done in collaboration with VPRO medialab – an interactive 3D reconstruction of the Amsterdam student encampments that took place earlier in 2024. Working with an immense archive of eyewitness footage allowed us to recreate the events of these days spatially and chronologically, in an attempt to offer a more comprehensive impression of these days and potential ground to counter the prevailing media narrative, where the protesters are oftentimes antagonised while the police violence is left unquestioned. The project started with a workshop from Forensic Architecture, where they shared some techniques and methods they use in their investigative work, many of which guided the project later on. My part in the project was 3D modelling, design, locating and tracking the footage and developing the overall interaction logic (it was intense).

You can visit it here (desktop only): https://www.vpro.nl/studentenprotesten.html