To Position, Deposition, and Reposition: The Division of Meaning and Power

This was by far the longest paper I have ever written. I found the process very challenging, but also rewarding.

And I can now officially say my third semester at ITP is over! Yahoo! Happy holidays!

Anyway, once I got the right XBees, they initially would not work very well. As it turned out, one of my pins had a COLD SOLDER. What a pain! Even after all that, I was getting communication errors with XCTU. I final figured out how to reset the XBee by pulling it out from the USB explorer, putting it back in, then hitting the “Restore” command in XCTU. After that, things went smoothly and I was chatting away like hyped-up valley girl.

Truth.

The first problem I ran into was that the router XBee COMPLETELY stopped communicating with my computer through the USB explorer. It took me a while to figure out how to reset the XBee. After that I got the router and coordinator talking to each other, but I was not getting a range of values on the serial data being received. For some reason, I would only get 518 as the value of the light sensor. Eventually I realized that there was a cold solder on pin 20 of the router XBee breakout board. Once I re-soldered the pin, everything worked like a charm!

I iterated over several ideas before I came to Music Wall. Initially what I wanted to create was an idea that I have had for some time for a musical instrument called “Beat Burger Helper” that would be sort of an assistant to beat-boxers. The premise is that when the performer makes sounds into the Beat Burger, the Beat Burger would process those sounds and play a sample of the sound that the user was attempting to emulate. For instance making the sound of a kick drum with your mouth, would result in a sample of an actual kick drum being played. Well, although this was a fun idea, it turned out to be too complicated to accomplish in the allotted time frame of one week.

The idea of the Music Wall came to me as an evolution of a previous idea of creating music by scanning the audience in the performance space. After an long night of coding what I came up with my initial prototype for “Music Wall.” This is a piece of software that uses computer vision to scan any surface to find objects (blob detection) and turn these objects into discrete musical notes. Essentially, it functions as a sequencer. In the video demo below, I am simply using pieces of grey tape on a white wall to compose a melody. The height of the piece of tape determines the pitch of the note. The horizontal position determines the length of the note, and its position in time, and the size of the determine the volume (larger objects are louder, smaller ones are quieter). At the end of the demo I give a shot of the Processing sketch in action.

Sorry for the poor quality and focus on the video. Thanks to my sister Anna for handling the camera (my Droid Incrapable).

• Creating a feedback loop between the music, and the response that it elicits in the performer experiencing the music (Cage and Barishnikov)
• Nature of music; why do people like music to begin with; what does it mean?
• Emotional relationship between man and technology/computer/machine
  • How does technology extend/empower man?

Process

• Heartbeat creates beat/rhythm for the music performance. Heartbeat is synthetically augmented over time
• Facial expression analysis triggers audio that is analogous with that expression
• In the performance setting, the body of the performer is made to seem small, insignificant, in comparison to his technology/ computer extension/ representation (Wizard of Oz)
• Visual effects modify the video signal in response to the emotions/physiological states of the performer or the music being generated

Technology

• ofxFaceTracker is an open source face tracking library for Open Frameworks
• FaceOSC is a library ported by Kyle McDonald over to OpenFrameworks, it is capable of capturing detailed facial analysis data, and sending it as midi out.

Feedback

• How to tell a story during the performance? What is the experience of the performer?
  • is he interacting with some media, playing a video game, watching a movie, talking with friends? What is the arc of the story?
• Considering adding more layers of sound sampled from the body;
  • Galvanic Skin Response
  • EEG brain waves
  • sound of breath?

• How to make it more musical? (get more accurate control over the sound; discrete notes, chords, etc)
• How to use head rotation to further augment the sound (add a sense of distance/directness)

• Does the aesthetic work? (sound and visual)

• causes disassociation with the outside world, disruption of natural sleep cycle
• depletion of vitamin D, Seasonal Affective Disorder
• “A growing body of evidence suggests it can raise your risk of cancer, increase susceptibility to heart attack, diabetes and other disorders, and at least partly account for the region’s sky-high rates of multiple sclerosis.” – Seattle Times
• it killed Mozart!!! (Requiem Mass in D-minor) [The Guardian]

I then propose my solution: the Real Lighting Network System. A light sensor that senses the light outdoors, and adjusts the color-temperature and brightness of the light indoors to reflect the light outside.

I conclude by going into some more details of how this could be achieved using modern lighting technology, and other ideas for how the system can be used:

• contemporary lighting options are able to closely simulate the temperature of daylight
• by sharing sensor data, users can simulate lighting from any part of the world within their own homes

1: One set is made up of an Xbee, Arduino, button, and two LEDS.

2: The other is an Xbee, Arduino, buzzer, and two LEDS.

When the user first pushes the button on component #1, a message is sent from #1 Xbee to the Xbee on #2. When this message is received, #2 responds by flashing the red LED, and sends a confirmation message back to #1, which responds by flashing its red LED as well. After the user pushes the button 10 times, #2 responds by buzzing, and flashing the green light, and also sending a different confirmation message back to #1. In response, #1 flashes its green LED as well.

And here is a goofy video of it in action at midnight on Friday night:

Emily posted about this project as well.

Illumination is an interactive art installation created by myself in collaboration with Yonatan Ben-Simhon. It uses light to discover and reveal poetry from within printed texts.

What happens
The basic set-up of Illumination is a table in a dark space, lit from above by a spot-light. Visitors can take any piece of paper with printed text on it (newspaper or magazine articles, pages from a book, etc) and place it on the table. Upon doing so, the light shining on the table beings to flicker, and the text on the page begins to fluctuate and shake. After a few seconds, the light goes out. Slowly, individual words of the text begin to light up. One by one, words light up and construct poetry. Take away the text and place a new text on the table, and a new poem is revealed within the text. The unique language and tone of each text reveals a unique poem.

How it was done
From a technical perspective, Illumination uses a camera, projector, and computer that work in synch to create this magical experience. The camera and projector are suspended above the table.  When a visitor places a text on the table surface, the camera above the table takes a high-resolution photo of the text which is uploaded to a computer nearby. A custom Python script running on the machine saves the JPEG image received from the camera as a TIFF. The script then uses the open-source Optical Character Recognition (OCR) software Tesseract to generate an HOCR.html output file that consists of both the text of the image that was processed and the positions and locations of every words that is located.

A poetry-generation Python script trains a Markov model on hundreds of works of poetry and fiction, creating a statistical model of word sequences that appear in poetic and non-fiction texts. In addition, the texts are all tagged for parts of speech, and the Markov model is then seeded with the tagged texts as well. Then, the word and part-of-speech sequences that are generated by the Markov model are searched for in the OCR text, generating a poem of several lines long. This sequence of lines and words is sent as a multi-dimensional array of word indexes to an implementation of Processing.py that then displays the box locations of the words, animating them as they appear and fade out over time.

More Sample Illumination Poetry

As a case study of the entire process outlines above, here is a sample photo of text that was taken:

Here is the HOCR output.html file generated by Tesseact: output.html (right-click to view source to see the box data)

And here is some sample poetry generated by Illumination:

will at any sight
a will of town instincts
the same, as well
will of his actions
will and sight policy
and sight into instincts

will like a desert endgame well
and to have it cornered.
drawn profiles and instincts have
it is a will fight well
a will to turn large
to turn the instincts from
a will of some benefit,

The Code

"""
illuminate.py

by Jack Kalish and Yonatan Ben-Simhon
NYU ITP 2011
"""

import processing.video.Capture as Capture
import commands, subprocess, math, re, os, shutil, glob, random
from BeautifulSoup import BeautifulSoup
#from markov import MarkovGenerator
import imageadjuster.ImageAdjuster as ImageAdjuster;
from threading import Thread
from MarkovGenerator import MarkovGenerator
#from FakeMarkovGenerator import MarkovGenerator

#import MarkOutPoet
#markOutPoet = MarkOutPoet()
#markOutPoet.quickPoemForJack()

imgPath = 'captures/Capture_00001.JPG'

class Illuminate(object):
def setup(self):
#instantiate class vars
self.fadeAlpha = 2
self.lightColor = [255,244,208]
#self.lightColor = 0xFFEF91
self.calibrate = False
self.speed = 250
self.showImg = 1
self.wordMargin = 3
self.boxList = []
self.currentBoxNum = [0]
self.lastTime = [0]
self.mt = []
self.words = []
self.loading = 0
self.thread = None
self.generator = None
self.lightOn = True
self.generator = None
self.showTime = 1000
# calibrate
f = open('config.txt')
str = f.read()
print "str: "+str
calib = str.split(',')
print calib
try:
self.imgPos = [int(calib[0]),int(calib[1])]
except:
#set default calib value
calib = [507,324,535,297]
self.imgPos = [int(calib[0]),int(calib[1])]
self.imgSize = [int(calib[2]),int(calib[3])]
self.saveCalib();
#set screen
#frame.setLocation(1280,0);
size(1600, 900)
#myCapture = Capture(this, width, height, 30)
fill(255, 5000)
noStroke()
self.adjust = ImageAdjuster(this)
self.img = loadImage(imgPath)

self.line = []
self.lines = []
self.lineCnt = 0
self.wordCnt = 0

def draw(self):
#print 'drawing'
#clearCaptures()

#self.showLoader()
if self.lightOn:
self.light()

if self.loading == 1:
if self.thread and self.thread.isAlive():
self.showLoader()
else:
self.loading = 0
if self.onThreadComplete != None:
self.onThreadComplete()

#print "self.lines: ",
#print self.lines

def setup(): illuminate.setup()
def draw(): illuminate.draw()
#def mousePressed(): illuminate.mousePressed()
def keyPressed(): illuminate.keyPressed()

The basic set-up is this: Next to a table sits a stack of newspaper and magazine articles. The visitor places one of the articles on the table. A camera above the table reads the text and processes it. Then, everything goes dark, and magically the text is lit up by an overhead projector. Individual words and phrases are lit up, creating poetry out of the original text.

We have already been working on this project for several weeks. Last week, we created a quick test of the installation set-up. Using a chain of tagged parts of speech, we then searched for the same parts of speech in the following sample article:

The source text that we used for the POS tagging was the line:

“And now for something completely different.”

The resulting POS tags were: CC, RB, IN, NN, RB, JJ

Sample Text: yet  never  with  passage  about
(created off: And now for something completely different)

word = yet, pos = CC, position = 70
word = never, prevPos = RB, pos = RB, position = 475
word = with, prevPos = IN, pos = IN, position = 484
word = passage, prevPos = NN, pos = NN, position = 518
word = about, prevPos = RB, pos = RB, position = 551