-
Recent Posts
Recent Comments
Archives
Categories
Meta
DRIVE BY C4D
Posted in Uncategorized
Leave a comment
To Position, Deposition, and Reposition: The Division of Meaning and Power
Here is a PDF of my final paper for Politics of the Gaze, a media theory class at NYU taught by Allen Feldman that I was taking this semester:
To Position, Deposition, and Reposition: The Division of Meaning and Power
This was by far the longest paper I have ever written. I found the process very challenging, but also rewarding.
And I can now officially say my third semester at ITP is over! Yahoo! Happy holidays!
XBee Basic Chat
I initially had some trouble getting my XBees up and running. The problem started with me accidentally ordering the series 1 XBee instead of the series 2, TWICE. Once from Amazon and then again from Sparkfun because the descriptions were confusing and poorly labeled.
Anyway, once I got the right XBees, they initially would not work very well. As it turned out, one of my pins had a COLD SOLDER. What a pain! Even after all that, I was getting communication errors with XCTU. I final figured out how to reset the XBee by pulling it out from the USB explorer, putting it back in, then hitting the “Restore” command in XCTU. After that, things went smoothly and I was chatting away like hyped-up valley girl.
Truth.
Posted in Sensitive Buildings
Leave a comment
Wireless Light Sensor
Getting the Romantic Light Sensor from chapter 3 of Building Wireless Sensor Networks took be an exorbitantly long time. But here it it!
The first problem I ran into was that the router XBee COMPLETELY stopped communicating with my computer through the USB explorer. It took me a while to figure out how to reset the XBee. After that I got the router and coordinator talking to each other, but I was not getting a range of values on the serial data being received. For some reason, I would only get 518 as the value of the light sensor. Eventually I realized that there was a cold solder on pin 20 of the router XBee breakout board. Once I re-soldered the pin, everything worked like a charm!
Posted in Uncategorized
Leave a comment
NIME Music Wall prototype
This semester at ITP I am really excited to be taking NIME: New Interfaces for Musical Expression. On the first day of class, out professor, Greg Shakar, asked for us to create a musical instrument in one week, and prepare a brief performance for the class.
I iterated over several ideas before I came to Music Wall. Initially what I wanted to create was an idea that I have had for some time for a musical instrument called “Beat Burger Helper” that would be sort of an assistant to beat-boxers. The premise is that when the performer makes sounds into the Beat Burger, the Beat Burger would process those sounds and play a sample of the sound that the user was attempting to emulate. For instance making the sound of a kick drum with your mouth, would result in a sample of an actual kick drum being played. Well, although this was a fun idea, it turned out to be too complicated to accomplish in the allotted time frame of one week.
The idea of the Music Wall came to me as an evolution of a previous idea of creating music by scanning the audience in the performance space. After an long night of coding what I came up with my initial prototype for “Music Wall.” This is a piece of software that uses computer vision to scan any surface to find objects (blob detection) and turn these objects into discrete musical notes. Essentially, it functions as a sequencer. In the video demo below, I am simply using pieces of grey tape on a white wall to compose a melody. The height of the piece of tape determines the pitch of the note. The horizontal position determines the length of the note, and its position in time, and the size of the determine the volume (larger objects are louder, smaller ones are quieter). At the end of the demo I give a shot of the Processing sketch in action.
Sorry for the poor quality and focus on the video. Thanks to my sister Anna for handling the camera (my Droid Incrapable).
Posted in Uncategorized
Leave a comment
NIME Final Concept – Emotion Machine
Concept
- Creating a feedback loop between the music, and the response that it elicits in the performer experiencing the music (Cage and Barishnikov)
- Nature of music; why do people like music to begin with; what does it mean?
- Emotional relationship between man and technology/computer/machine
- How does technology extend/empower man?
Process
- Heartbeat creates beat/rhythm for the music performance. Heartbeat is synthetically augmented over time
- Facial expression analysis triggers audio that is analogous with that expression
- In the performance setting, the body of the performer is made to seem small, insignificant, in comparison to his technology/ computer extension/ representation (Wizard of Oz)
- Visual effects modify the video signal in response to the emotions/physiological states of the performer or the music being generated
Technology
- ofxFaceTracker is an open source face tracking library for Open Frameworks
- FaceOSC is a library ported by Kyle McDonald over to OpenFrameworks, it is capable of capturing detailed facial analysis data, and sending it as midi out.
Feedback
- How to tell a story during the performance? What is the experience of the performer?
- is he interacting with some media, playing a video game, watching a movie, talking with friends? What is the arc of the story?
- Considering adding more layers of sound sampled from the body;
- Galvanic Skin Response
- EEG brain waves
- sound of breath?
- How to make it more musical? (get more accurate control over the sound; discrete notes, chords, etc)
- How to use head rotation to further augment the sound (add a sense of distance/directness)
- Does the aesthetic work? (sound and visual)
Posted in NIME
Leave a comment
Imagining Sensitive Buildings – Real Lighting Network System
Below are the slides from my presentation of the concept for the Real Lighting Network System. This was created for the Imagining Sensitive Buildings assignment for the class Sensitive Buildings at ITP. I started off the presentation by talking about various apartments that I lived in in Brooklyn that don’t have adequate natural lighting, and how this is a common problem in urban dwelling environments. I then go on to talk about the various problems and health hazards associated with not getting enough exposure to sunlight:
- causes disassociation with the outside world, disruption of natural sleep cycle
- depletion of vitamin D, Seasonal Affective Disorder
- “A growing body of evidence suggests it can raise your risk of cancer, increase susceptibility to heart attack, diabetes and other disorders, and at least partly account for the region’s sky-high rates of multiple sclerosis.” – Seattle Times
- it killed Mozart!!! (Requiem Mass in D-minor) [The Guardian]
I then propose my solution: the Real Lighting Network System. A light sensor that senses the light outdoors, and adjusts the color-temperature and brightness of the light indoors to reflect the light outside.
I conclude by going into some more details of how this could be achieved using modern lighting technology, and other ideas for how the system can be used:
- contemporary lighting options are able to closely simulate the temperature of daylight
- by sharing sensor data, users can simulate lighting from any part of the world within their own homes
Links to sources:
Posted in Uncategorized
Leave a comment
Nap Feedback Buzzer
For the second assignment for Sensitive Buildings, I worked with Emily Webster. We made a “Nap Feedback Buzzer.” It is composed of two sets of components.
1: One set is made up of an Xbee, Arduino, button, and two LEDS.
2: The other is an Xbee, Arduino, buzzer, and two LEDS.
When the user first pushes the button on component #1, a message is sent from #1 Xbee to the Xbee on #2. When this message is received, #2 responds by flashing the red LED, and sends a confirmation message back to #1, which responds by flashing its red LED as well. After the user pushes the button 10 times, #2 responds by buzzing, and flashing the green light, and also sending a different confirmation message back to #1. In response, #1 flashes its green LED as well.
And here is a goofy video of it in action at midnight on Friday night:
Posted in Sensitive Buildings, Uncategorized
Leave a comment
Illumination: real-time generative blackout poetry
Illumination is an interactive art installation created by myself in collaboration with Yonatan Ben-Simhon. It uses light to discover and reveal poetry from within printed texts.
What happens
The basic set-up of Illumination is a table in a dark space, lit from above by a spot-light. Visitors can take any piece of paper with printed text on it (newspaper or magazine articles, pages from a book, etc) and place it on the table. Upon doing so, the light shining on the table beings to flicker, and the text on the page begins to fluctuate and shake. After a few seconds, the light goes out. Slowly, individual words of the text begin to light up. One by one, words light up and construct poetry. Take away the text and place a new text on the table, and a new poem is revealed within the text. The unique language and tone of each text reveals a unique poem.
How it was done
From a technical perspective, Illumination uses a camera, projector, and computer that work in synch to create this magical experience. The camera and projector are suspended above the table. When a visitor places a text on the table surface, the camera above the table takes a high-resolution photo of the text which is uploaded to a computer nearby. A custom Python script running on the machine saves the JPEG image received from the camera as a TIFF. The script then uses the open-source Optical Character Recognition (OCR) software Tesseract to generate an HOCR.html output file that consists of both the text of the image that was processed and the positions and locations of every words that is located.
A poetry-generation Python script trains a Markov model on hundreds of works of poetry and fiction, creating a statistical model of word sequences that appear in poetic and non-fiction texts. In addition, the texts are all tagged for parts of speech, and the Markov model is then seeded with the tagged texts as well. Then, the word and part-of-speech sequences that are generated by the Markov model are searched for in the OCR text, generating a poem of several lines long. This sequence of lines and words is sent as a multi-dimensional array of word indexes to an implementation of Processing.py that then displays the box locations of the words, animating them as they appear and fade out over time.
More Sample Illumination Poetry
As a case study of the entire process outlines above, here is a sample photo of text that was taken:
Here is the HOCR output.html file generated by Tesseact: output.html (right-click to view source to see the box data)
And here is some sample poetry generated by Illumination:
will at any sight
a will of town instincts
the same, as well
will of his actions
will and sight policy
and sight into instincts
will like a desert endgame well
and to have it cornered.
drawn profiles and instincts have
it is a will fight well
a will to turn large
to turn the instincts from
a will of some benefit,
The Code
"""
illuminate.py
by Jack Kalish and Yonatan Ben-Simhon
NYU ITP 2011
"""
import processing.video.Capture as Capture
import commands, subprocess, math, re, os, shutil, glob, random
from BeautifulSoup import BeautifulSoup
#from markov import MarkovGenerator
import imageadjuster.ImageAdjuster as ImageAdjuster;
from threading import Thread
from MarkovGenerator import MarkovGenerator
#from FakeMarkovGenerator import MarkovGenerator
#import MarkOutPoet
#markOutPoet = MarkOutPoet()
#markOutPoet.quickPoemForJack()
imgPath = 'captures/Capture_00001.JPG'
class Illuminate(object):
def setup(self):
#instantiate class vars
self.fadeAlpha = 2
self.lightColor = [255,244,208]
#self.lightColor = 0xFFEF91
self.calibrate = False
self.speed = 250
self.showImg = 1
self.wordMargin = 3
self.boxList = []
self.currentBoxNum = [0]
self.lastTime = [0]
self.mt = []
self.words = []
self.loading = 0
self.thread = None
self.generator = None
self.lightOn = True
self.generator = None
self.showTime = 1000
# calibrate
f = open('config.txt')
str = f.read()
print "str: "+str
calib = str.split(',')
print calib
try:
self.imgPos = [int(calib[0]),int(calib[1])]
except:
#set default calib value
calib = [507,324,535,297]
self.imgPos = [int(calib[0]),int(calib[1])]
self.imgSize = [int(calib[2]),int(calib[3])]
self.saveCalib();
#set screen
#frame.setLocation(1280,0);
size(1600, 900)
#myCapture = Capture(this, width, height, 30)
fill(255, 5000)
noStroke()
self.adjust = ImageAdjuster(this)
self.img = loadImage(imgPath)
self.line = []
self.lines = []
self.lineCnt = 0
self.wordCnt = 0
def draw(self):
#print 'drawing'
#clearCaptures()
#self.showLoader()
if self.lightOn:
self.light()
if self.loading == 1:
if self.thread and self.thread.isAlive():
self.showLoader()
else:
self.loading = 0
if self.onThreadComplete != None:
self.onThreadComplete()
#print "self.lines: ",
#print self.lines
if self.calibrate:
image(self.img, self.imgPos[0], self.imgPos[1], self.imgSize[0], self.imgSize[1])
if len(self.lines)>0:
#background(0)
if self.lineCnt < len(self.lines)-1: '''print "len(self.line): ", print len(self.line) print "len(self.words): ", print len(self.words) print "self.wordCnt: ", print self.wordCnt''' if millis() - self.lastTime[0] > self.showTime:
if self.wordCnt >= len(self.line):
#background(0)
#go to next line
self.wordCnt = 0
self.lineCnt += 1
self.line = self.lines[self.lineCnt]
self.showTime = 2000
self.fadeAlpha = 10
#print "new line length:"
#print len(self.line)
#print " ".join(self.line)
#print len(self.mt[self.currentBoxNum[0]])*100
else:
#background(0,10)
#lightRandomWord()
#print "self.wordCnt: ",
#print self.wordCnt
#print "line[self.wordCnt] ",
#print self.line[self.wordCnt]
self.lightWord(self.line[self.wordCnt])
#self.lightNextMt()
self.showTime = math.sqrt(self.getCurrentWordLength())*self.speed
self.wordCnt += 1
self.fadeAlpha = 2
self.lastTime[0] = millis()
fill(0,0,0,self.fadeAlpha)
rect(0,0,width,height)
else:
#we gotta end it here, no more lines!
self.end()
def light(self):
background(self.lightColor[0],self.lightColor[1],self.lightColor[2])
def end(self):
print "the poem is over"
self.runThreadOn(self.makeNewPoem)
def makeNewPoem(self):
print "thinking of a new poem...hmmm"
#make a new poem between 3 and 14 lines long
self.lineCnt = 0
self.wordCnt = 0
numLines = round(random.random()*11)+3
print "ok, it will be ",
print numLines,
print "lines long"
lines = self.generator.generateFromText(numLines)
self.lines = lines
self.line = lines[0]
self.showTime = 5000
def getCurrentWordLength(self):
return len(self.words[self.line[self.wordCnt]])
def runThreadOn(self, method, callback=None):
if self.thread and self.thread.isAlive():
print "thread already running"
return
self.onThreadComplete = callback
self.thread = Thread(target=method)
self.thread.start()
self.loading = 1
self.calibrate = False
def runThread(self):
if self.thread and self.thread.isAlive():
print "thread already running"
return
self.thread = Thread(target=self.run)
self.thread.start()
self.loading = 1
self.calibrate = False
def showLoader(self):
#print 'loading'
#text("processing...", width/2, height/2)
#jiggle the image?
randomness = 3.0
#print "random: "
#print random(1)
background(self.lightColor[0], self.lightColor[1], self.lightColor[2])
image(self.img, self.getWiggle(self.imgPos[0], randomness), self.getWiggle(self.imgPos[1], randomness), self.getWiggle(self.imgSize[0], randomness), self.getWiggle(self.imgSize[1], randomness))
#background(self.lightColor,random.random()*255)
fill(0, random.random()*50)
rect(0,0,width,height)
#image(self.img, self.imgPos[0], self.imgPos[1], self.imgSize[0], self.imgSize[1])
#image(self.img, self.imgPos[0]*random(randomness*-1, randomness), self.imgPos[1]*random(randomness*-1, randomness), self.imgSize[0]*random(randomness*-1, randomness), self.imgSize[1]*random(randomness*-1, randomness))
def getWiggle(self, val, r):
#print 'input val:',
#print val
val += (random.random()*r*2) - r
#print 'output val',
#print val
return val
def run(self):
#clear screen
background(0)
self.showImg = 0
#wait for a new image to become available
files = glob.glob('captures/*')
while len(files) < 1:
while files[0] != 'captures/capture_1.JPG':
files = glob.glob('captures/*')
print "waiting for an image"
#colorMode(HSB, 100);
#tint(0,0)
#adjust contrast
#img.adjust.contrast(g, 2)
self.capture() #convert to tiff
#delete the original jpeg
#commands.getstatusoutput('rm -f captures/capture_1.JPG')
self.performOCR()
self.words = self.parseWords()
#TRAIN THE MARKOV GENERATOR
self.generator = MarkovGenerator(n=3, max=30, min=4, words=self.words)
def onOCRComplete(self):
print "finished reading!"
self.lightOn = False
self.runThreadOn(self.makeNewPoem)
#now get generate the poetry from the text...
#doMarkov()
def performOCR(self):
print "reading text..."
subprocess.call(['tesseract',r'captures/capture.tif', 'output', '-l', 'eng' ,'+hocr.txt'])
#commands.getstatusoutput('tesseract capture.tif output -l eng +ocr/hocr.txt')
#commands.getstatusoutput('tesseract captures/capture.tif output -l eng +hocr.txt')
#commands.getstatusoutput('tesseract ocr/article.tif output -l eng +ocr/hocr.txt')
def lightRandomWord(self):
self.lightWord(random.randint(0,len(boxList)))
def lightWord(self,id):
#print "light word: ",
print self.words[id],
self.lightBox(self.boxList[id])
def lightNextBox(self):
#background(0)
#print 'light next word:'
#print boxList[currentBoxNum[0]]
self.lightBox(self.boxList[self.currentBoxNum[0]])
self.currentBoxNum[0] += 1
def lightBox(self,r):
xScale = self.imgSize[0]/float(2272)
yScale = self.imgSize[1]/float(1704)
#print r
#fill(255,10)
fill(self.lightColor[0],self.lightColor[1],self.lightColor[2])
rect((float(r[0]))*xScale+self.imgPos[0]-(self.wordMargin), (float(r[1]))*yScale+self.imgPos[1]-(self.wordMargin), float(r[2])*xScale+(self.wordMargin*2), float(r[3])*yScale+(self.wordMargin*2))
#delay(1000)
def moveImage(self,x,y):
self.imgPos[0] += x
self.imgPos[1] += y
self.saveCalib()
def scale(self,x,y):
self.imgSize[0] += x
self.imgSize[1] += y
self.saveCalib()
def saveCalib(self):
print "save calib"
f = open('config.txt', 'r+')
f.truncate()
calib = str(self.imgPos[0])+","+str(self.imgPos[1])+","+str(self.imgSize[0])+","+str(self.imgSize[1])
print "calib: "+calib
f.write(calib)
f.close()
def clearCaptures(self):
print "clear captures dir"
files = glob.glob('captures/*')
print "files:"
print files
for f in files:
print f
os.remove(f)
#if f!= 'captures/capture_1.JPG':
#os.remove(f)
#remove all images except for the first
'''for files in os.walk('/captures'):
print "file:"
print f
if f!= 'capture_1.JPG':
os.unlink(os.path.join(root, f))'''
def capture(self):
print "save tif"
self.img.save("captures/capture.tif");
def parseWords(self):
soup = BeautifulSoup(open('output.html'))
print "parsing words..."
boxes = soup.findAll('span', { "class" : "ocr_word" })
cnt = 0
words = []
for word in boxes:
#generate arrary of only words
w = word.contents[0].contents[0]
words.append(w)
title = word['title']
#print "title: " + title
r = title.split(' ')
r.pop(0)
#generate another array of box data
#convert array to processing rect object coords (x,y,W,H)
r[2] = float(r[2]) - float(r[0])
r[3] = float(r[3]) - float(r[1])
self.boxList.append([r[0], r[1], r[2], r[3]])
cnt += 1
print "words:",
print words
return words
#print "words array: "
#print words
def keyPressed(self):
print "keypressed: "
print key
if key == CODED:
if keyCode == UP:
self.moveImage(0,-1)
elif keyCode == DOWN:
self.moveImage(0,1)
elif keyCode == LEFT:
self.moveImage(-1,0)
elif keyCode == RIGHT:
self.moveImage(1,0)
elif key==61:
self.scale(1,0);
elif key==45:
self.scale(-1,0);
elif key==93:
self.scale(0,1);
elif key==91:
self.scale(0,-1);
elif key==99:
#c - turn calibration on
background(0)
self.calibrate = not self.calibrate
elif key==32:
#spacebar - run
self.runThreadOn(self.run, self.onOCRComplete)
illuminate = Illuminate()
def setup(): illuminate.setup()
def draw(): illuminate.draw()
#def mousePressed(): illuminate.mousePressed()
def keyPressed(): illuminate.keyPressed()
Final Project Proposal – Light-up Poetry
For the final project I am collaborating with Yonatan Ben-Simhon. We are creating an interactive art installation inspired by Austin Kleon’s Newspaper Blackout Poems project.
The basic set-up is this: Next to a table sits a stack of newspaper and magazine articles. The visitor places one of the articles on the table. A camera above the table reads the text and processes it. Then, everything goes dark, and magically the text is lit up by an overhead projector. Individual words and phrases are lit up, creating poetry out of the original text.
We have already been working on this project for several weeks. Last week, we created a quick test of the installation set-up. Using a chain of tagged parts of speech, we then searched for the same parts of speech in the following sample article:
The source text that we used for the POS tagging was the line:
“And now for something completely different.”
The resulting POS tags were: CC, RB, IN, NN, RB, JJ
Sample Text: yet never with passage about
(created off: And now for something completely different)
word = yet, pos = CC, position = 70
word = never, prevPos = RB, pos = RB, position = 475
word = with, prevPos = IN, pos = IN, position = 484
word = passage, prevPos = NN, pos = NN, position = 518
word = about, prevPos = RB, pos = RB, position = 551



