9a – Video and Pixels

Key concepts: working with images and video as pixels!

Background concepts:
for loops, arrays, the PImage datatype, using the video library
Background video tutorials (if you need to backup on these concepts):
Arrays:
http://icm.shiffman.net/9.0/
http://icm.shiffman.net/9.1/
for loops:
http://icm.shiffman.net/6.3/
  variable scope:
http://icm.shiffman.net/6.4/
A loop inside draw
http://icm.shiffman.net/6.5/

PIXELS!

PART 1 – Images
Review the images section of this site (or chapter or section 15.5 on page 262 of the orange Shiffman book) and consider the PImage data type in the Processing reference.

Processing has two more image functions to access pixels on your screen.
• loadPixels( )—This function is called before you access the pixel array, saying “load the pixels, I would like to speak with them!”
• updatePixels( )—This function is called after you finish with the pixel array, saying “Go ahead and update the pixels, I’m all done!”

These functions allow us to set and talk to each pixel of the screen by loading them into an array called:

pixels[i];

Where i is the position along the array. It is like we load the color values of each pixel of the screen into a big long list.

Screen Shot 2015-03-24 at 8.22.29 AM

We an then talk to individual pixels by calculating where they are along this list:

Screen Shot 2015-03-24 at 8.22.43 AM

For example we can set pixel color according to their 2D location:

size(200,200);
loadPixels();
// Loop through every pixel column
for (int x = 0; x < width; x++) {
  // Loop through every pixel row
  for (int y = 0; y < height; y++){
   // Use the formula to find the 1D location
   int loc = x + y * width;
   if (x % 2 == 0){ // If we are an even column 
     pixels[loc] = color(255);
   }else { // If we are an odd column 
     pixels[loc] = color(0);
   } 
 }
updatePixels();
}

Shiffman explains again here.

Make sure you understand the process of ‘loading pixels’ and ‘setting pixels’.

EXERCISE:
Screen Shot 2015-03-24 at 8.28.09 AM

VIDEO
Video files:
lights2
frisbee golf pro

Processing can load and manipulate video like it can with PImages. To do so you need to use either the video or the capture class. The references for these classes are here:  https://processing.org/reference/libraries/video/

import processing.video.*;
Movie movie;
 
void setup() {
  size(640, 360);
  background(0);
  movie = new Movie(this, "mom.mov");
  movie.play(); // play until the end
  // movie.loop(); // does the same thing as play but loops the video
}
 
void draw() {
  if (movie.available() == true) {
    movie.read();
  }
  image(movie, 0, 0, width, height);
}
import processing.video.*;
Movie movie;
boolean playPause = true; 
void setup() {
  size(640, 520);
  background(0);
  movie = new Movie(this, "lights2.mov");
}
 
void draw() {
  if (playPause == true) {
    movie.loop(); // play until the end
  }
  else {
    movie.pause(); // play until the end
  }
  if (movie.available() == true) {
    movie.read();
  }
  image(movie, 0, 0, movie.width, movie.height);
}
 
void mouseReleased() {
  playPause = !playPause;
}

Exercise:
Create a video player with 4 buttons,

  • pause
  • jump to a point
  • double speed
  • swap video

You can do this by looking at video examples, as well as the video library API.

You’ll notice that when we draw the video on to the screen we’re actually using image(). Yup – same code that we use for PImages! This means that each frame of video is being drawn one at a time, which means we can mess with the pixel array, just like we did with images.

import processing.video.*;
Movie movie;
boolean flip;
PImage destination; 
void setup() {
  size(640, 360);
  background(0);
  movie = new Movie(this, "lights2.mov");
  destination = createImage(640, 360, RGB);
  movie.loop();
}
 
void draw() {
  if (movie.available() == true) {
    movie.read();
  }
  movie.loadPixels(); //load the frame into an array
  destination.loadPixels(); // load destination into an array
  for (int i = 0; i &lt; movie.width; i++) { // For each pixel in the video frame...
    for (int j = 0; j &lt; movie.height; j++) { // For each pixel in the video frame...
      color currColor = movie.get(i, j); // get current color
      color newcolor = color (green(currColor), blue(currColor), red(currColor));  // swap colors
      if (flip == true) { 
        destination.set(movie.width - i, j, newcolor); // put the new color into destination
      }
      else {
        destination.set(i, j, currColor); // put the old colors into destination
      }
    }
  }
  destination.updatePixels(); 
  image(destination, 0, 0); //put destination array on the screen
}
void mouseReleased() {
  flip = !flip;
}

You can “glitch” video as well.

import processing.video.*;
Movie movie;
PImage destination; 
int glitchStart, glitchEnd, glitchShift; 
void setup() {
  // size( 1920,  1080); 
  size(640, 360);
  background(0);
  movie = new Movie(this, "mom.mov");
  movie.loop(); // does the same thing as play but loops the video 
  destination = createImage(640, 360, RGB);
}
 
void draw() {
  if (movie.available() == true) {
    movie.read();
  }
  movie.loadPixels(); //load the frame into an array
  if (random(100) &gt; 70) {
    glitchStart = int(random(0, movie.height - 100)); 
    glitchEnd = glitchStart + int(random(20, 300)); 
    glitchShift = int(random(-50, 50));
  }
  for (int i = 0; i &lt; movie.width; i++) { // For each pixel in the video frame...
    for (int j = 0; j &lt; movie.height; j++) { // For each pixel in the video frame...
      color currColor = movie.get(i, j); // get current color
      color newcolor = color (green(currColor + glitchShift), blue(currColor- glitchShift), red(currColor + glitchShift));  // swap colors
      if (j &gt; glitchStart &amp;&amp;  j &lt; glitchEnd) {
        destination.set(i+ glitchShift, j, newcolor); // put the new color into destination
      }
      else {
        destination.set(i, j, currColor); // put the old colors into destination
      }
    }
  }
  if (mousePressed == true) {
    for (int i = 0; i &lt; movie.width; i++) { // For each pixel in the video frame...
      for (int j = 0; j &lt; movie.height; j++) { // For each pixel in the video frame...
        color newcolor = movie.get(mouseX, j); // get current color
        destination.set(i, j, newcolor); // put the new color into destination
      }
    }
  }
  destination.updatePixels(); 
  image(destination, 0, 0); //put destination array on the screen
}

Live Video!

Live video uses the same library as above
Go under File – Examples – Video – Capture – GettingStartedCapture to find a good example.

You’ll notice that the program sends a list of the supported cameras to the message window. You may want to change the camera resolution by picking a new number in the line
cam = new Capture(this, cameras[0]);
for me, a camera[3] was a better fit, but it may be different on your computers. While you can use a camera to capture an image, what if we think of it as a “light sensor” instead?
This sketch compares the the pixels that have changed from the last time the screen refreshed. It compares the number of pixels that have changed on the right side vs the left side. One nice trick, this script uses dist to get the “distance” between the two colors. Awesome hack.

import processing.video.*;
Capture cam;
PImage destination; 
int ballX; 
 
void setup() {
  size(640, 360);
  String[] cameras = Capture.list();
  for (int i = 0; i &lt; cameras.length; i++) {
    println(cameras[i]);
  }
  destination = createImage(640, 360, RGB);
  cam = new Capture(this, cameras[3]);
  cam.start();
  ballX = width /2;
}
 
void draw() {
  if (cam.available() == true) {
    cam.read();
  }
  cam.loadPixels();
  int left = 0; 
  int right = 0; 
  for (int i = 0; i &lt; cam.width; i++) { 
    for (int j = 0; j &lt; cam.height; j++) {
      color currColor = cam.get(i, j);
      color oldColor = destination.get(i, j);   
      if (dist(red(currColor), green(currColor), blue(currColor), red(oldColor), green(oldColor), blue(oldColor)) &gt; 100) {
        // This line above adds a buffer so it's not so sensitive. 
        if (i &lt; cam.width/2) {
          left ++;
        }
        else {
          right ++;
        }
      }
      destination.set(i, j, currColor); // put the old colors into destination
    }
  }
  destination.updatePixels(); 
  image(destination, 0, 0); //put destination array on the screen
  if (left &gt; right) {
    ballX -= 10;
  }
  else if (left &lt; right) {
    ballX +=  10;
  }
  ballX = constrain(ballX, 0, width); 
  color X = get(ballX, height/2); 
  fill(X); 
  ellipse( ballX, height/2, 80, 80);
}