Physical Computing Final

Resources

p5 Web Serial Documentation GitHub Guide

Lab: Serial Input to p5.js Using the p5.webserial Library – ITP Physical Computing

Adafruit Learning System Adafruit I2S MEMS Microphone Breakout

Paper Prototypes Planning: Unsupported Embed

Phase 1: Testing microphone sensor

We considered airflow sensors, Adafruit Industries Adafruit Sensirion SHTC3 Temperature & Humidity Sensor and ended up running a first test and wiring for Adafruit I2S MEMS Microphone Breakout - SPH0645 with Arduino Nano 33 IoT:

• LRCLK/WS connects to A2 pin on Nano
• BCLK connects to A3 pin on Nano
• Data/SD connects to pin 4 on Nano

Arduino Example Code 1

/*
 This example reads audio data from an Invensense's ICS43432 I2S microphone
 breakout board, and prints out the samples to the Serial console. The
 Serial Plotter built into the Arduino IDE can be used to plot the audio
 data (Tools -> Serial Plotter)

 Circuit:
 * Arduino/Genuino Zero, MKR family and Nano 33 IoT
 * ICS43432:
   * GND connected GND DONE
   * 3.3V connected to 3.3V (Zero, Nano) or VCC (MKR) DONE
   * (LRCLK or) WS connected to pin 0 (Zero) or 3 (MKR) or A2 (Nano) 
   * CLK connected to pin 1 (Zero) or 2 (MKR) or A3 (Nano)
   * SD connected to pin 9 (Zero) or A6 (MKR) or 4 (Nano)

 created 17 November 2016
 by Sandeep Mistry
 */

#include <I2S.h>

void setup() {
  // Open serial communications and wait for port to open:
  // A baud rate of 115200 is used instead of 9600 for a faster data rate
  // on non-native USB ports
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // start I2S at 8 kHz with 32-bits per sample
  if (!I2S.begin(I2S_PHILIPS_MODE, 8000, 32)) {
    Serial.println("Failed to initialize I2S!");
    while (1); // do nothing
  }
}

void loop() {
  // read a sample
  int sample = I2S.read();

  if (sample) {
    // if it's non-zero print value to serial
    Serial.println(sample);
  }
}

Arduino Code 2 Working

#include <I2S.h>

void setup() {
  Serial.begin(115200);
  while (!Serial) {
    ; // Wait for serial port to connect. Needed for native USB only
  }

  // Start I2S at 8 kHz with 32-bits per sample
  if (!I2S.begin(I2S_PHILIPS_MODE, 8000, 32)) {
    Serial.println("Failed to initialize I2S!");
    while (1); // do nothing
  }
}

void loop() {
  // Read a sample from the I2S microphone
  int sample = I2S.read();

  // Only send significant audio levels
  if (sample > 100) { // Threshold to filter out background noise
    Serial.println(abs(sample) / 100); // Scale down data for p5.js
  }
}

p5 Web Serial Trial Code

experiment_arduino_microphone by ineslucas -p5.js Web Editor

Delicate gold copy copy copy by ranjanir -p5.js Web Editor / / Working Code, but not the microphone ended up not being the interaction we were looking for, as you had to breath into it at an angle.

Phase 2: Setting on a concept & planning out layers

Phase 3: Building a Velostat circular mat

Instructables O-mat → Our classmate Nasif was incredibly helpful in the testing phase, and we ended up verifying it works, the structure and subsequently ordering the needed quantities. Velostat purchased from Less EMF Paints Archives.

The original Processing Code for a square grid is as follows below. We figured out we don’t necessarily need to use Processing as that would mean learning Java.

Through the p5 prototyping below, we realised we need around 90 sections (30 below divided by 3, creating 90).

Problem: getting enough pins to connect around 90 sections

• Can we use the I2C bus and an I2C multiplexer for analog signal from copper + velostat? Would the output of I2C communication be helpful in retrieving the range output from the pressure applied on the Velostat?
• It seems like the Arduino Nano only takes 3 multiplexers (3 * 5 wires needed and a max of 19 Arduino pins). For the Sparkfun multiplexer sparkfun SparkFun Analog/Digital MUX Breakout - CD74HC4067, that would mean 3*16 = 48 regions of sensing, but it’s looking like we need something closer to 90. It’s possible I’m being shortseighted. Would using an array mean we need more connections or less? More actually.

Solution is basically between using an Arduino Mega with 54 pins (16 with PWM) and the Spark Fun or using more I2C multiplexers? [ Question for Prof ]

How to use TX / RX to connect 2 Arduinos: Instructables I2C Between Arduinos

Class feedback session: Solution

• We’re building a capacitive sensor. // custom pressure sensor.
• When Serial is open, it can’t be open to p5 and TouchDesigner at the same time. There however can be two way serial. There’s also other ways to communicate besides serial - like Midi or OSC.
• Main issue is that 90 is a lot of sections and a lot of code.
• Actually we can use 2 sections ⭕️, making it 32 total or fewer sections in the palm and more in the fingertips = 15 + 30.
• Need to verify this information but the Arduino Mega has no wifi.
• I can chain together the Analog/Digital MUX, which means that the amount of Nano pins wouldn’t be a constraint.
• Sparkfun Analog to digital MUX Multiplexer has a different function than the I2C multiplexer. An I2C multiplexer is for connecting I2C devices that have the same address, so that way you don’t need to change the addresses, you can target the multiplexer instead.

Fabrication

Tools used

• Table Saw to cut up MDF
• Laser Cutter
• Heater for cables

Resources for creating circuit

Multiplexer

www.sparkfun.com Datasheet (not schematic)

Operating Conditions: Supply Voltage Range, Vcc → HC Types 2V to 6V

→ When do we know if we’ve reached the maximum?

🍎 This may be a problem… “Only one of these is selected at any time, based on the control pins.”

How much resistance do resistors need?

To determine the resistance value for your resistor, we need to analyze the circuit and the voltage behavior of the CD74HC4067 in conjunction with the Arduino Nano IoT 33.

Voltage Output on the Channel Pins

The output voltage from any channel pin (e.g., C0) of the CD74HC4067 will depend on the input voltage supplied to the COM (SIG) pin and the control logic state. The multiplexer simply connects the selected channel (C0–C15) to the COM pin with minimal resistance.

• Supply Voltage (VCC):

The CD74HC4067 works with a power supply voltage of 2–6V. If you're powering the board from the Arduino's 3.3V pin, the output voltage on any channel pin will be approximately 3.3V when HIGH.

• Output Impedance:

The ON resistance of the multiplexer (when a channel is active) is around 70–100Ω (depending on VCC). This slight resistance doesn't significantly affect most circuits.

Resistor Calculation for Your Homemade Sensor

For your sensor connected to C0, the resistor to ground is likely acting as a pull-down resistor or forming a voltage divider to measure an analog signal. The resistance value depends on:

1. Desired Voltage Range Across the Sensor:

The voltage across the sensor should fall within the Arduino Nano IoT 33's ADC input range (0–3.3V). The ADC has a 12-bit resolution, so you'll get finer measurements if the voltage variation across the sensor spans as much of this range as possible.

2. Sensor Characteristics:

What is the expected resistance range of your homemade sensor? This is critical for determining the pull-down resistor.

Case 1: Sensor as a Variable Resistor

​

Case 2: Fixed Resistor for Digital Input

If your sensor output is binary (e.g., HIGH/LOW detection), use a pull-down resistor:

• Typical values: 10kΩ works well in most cases.

We have 27 pins left! 🍓

Writing code to read sensors

Moving to VS Code & a React app with p5

NANO 33 IoT — PlatformIO v6.1 documentation

Start project with Arduino MKR WiFi 1010 as a replacement for Nano 33 IoT

What we did:

• finished hooking up sensors
• Figured out the circuit
• Chained multiplexers
• Added p5 to a React application, started setting up the structure to receive and handle API data for top stories
• Added a way to real serial monitor on VS Code
• Wrote the code for chaining and reading 54 sensors
• Wrote the code to do first baseline readingsz

Feedback from Playtesting #1:

News showing is it bad? Or chaotic? Could show comments instead of news.

Youtube Trending Videos

Visuals:

• can use text for news - text rotating circles like in news channels

Will it be hard to get

Can use felt and

What if we visualize each news story as its own organism?

Play act or

Next week: bring an actual visuals!

Next steps - AI reference advice

Project building: prep for playtest 2

@December 2, 2024

Tasks aren’t building with the same code.

If I reset → unplug → build - monitor doesn’t complete successfully ❌

React can’t read if platformIO is reading.

Print out pics of what’s happening → For interaction.

multiplexee does the same as the for loop

variable resistor = we have an homemade w velostat

there’s a multiplexer library??!

Assumptions: do people understand the touch grass thing? Does the concept make sense.

Make sure to handle cleanup properly in your React component (closing the serial port when the component unmounts)

Debug

• I can list all the ports + select port from the terminal.
Copy

ls /dev/cu.*


• Only one thing can be connected at a time. It’s not broadcasted.
• Don’t get rid of if serial.
• I need I close serial monitor,
• Have p5 send a 0, so that it knows when it’s been pinged. Same as I was sending over
• Mouse press could be the condition and then it reads automatically. Make it user initiated. Don’t do a while loop.

p5.serialport and p5.webserial Compared – ITP Physical Computing

Physical Computing L4: p5.js Serial I/O by the Makeability Lab

p5xjs Creating Custom Geometry in WebGL

npm npm: @p5-wrapper/react

Basic usage of @p5-wrapper/react

import { ReactP5Wrapper } from "@p5-wrapper/react";

const P5sketch = () => {
  function sketch(p5) {
    p5.setup = () => {
      p5.createCanvas(700, 400);
      p5.background(200);
    };

    p5.draw = () => {
      p5.fill(181, 79, 111);
      p5.noStroke();
      p5.rect(200, 200, 100, 50);  // x, y, width, height
    };
  }

  return <ReactP5Wrapper sketch={sketch} />;
};

export default P5sketch;

As you work on this any microcontroller-to-computer application, you will be switching back and forth between the app that programs the microcontroller (in this case, the Arduino IDE) and the app that the microcontroller is communicating with (in this case, p5.js in the browser). You have to keep in mind that only one of these at a time can access a serial port.

That means that when you want to reprogram your Arduino from the Arduino IDE, you should to stop your sketch in the browser window to do so. Then, restart the browser sketch when you’re done reprogramming the Arduino. You don’t need to quit the Arduino IDE each time, because it knows to release the serial port when it’s not programming. However, you do need to close the Serial Monitor in the Arduino IDE when you are using WebSerial in the browser.

We changed the Arduino Code to continuously read, but C1 is still off the charts. Is it because electricity takes the path of least resistance?

Code for Ripples on p5 editor - check if flower or ripple output without p5.WebSerial library

let sensorValues = []; // Array to store sensor readings
let ripples = [];
let lotuses = [];
let caustics = [];
let serial; // WebSerial object
let port; // Port object
let reader; // Reader for serial data
let decoder = new TextDecoder(); // Decoder for incoming serial data

function setup() {
  createCanvas(800, 600);
  noSmooth();

  // Initialize caustics grid
  for (let i = 0; i < width; i += 20) {
    for (let j = 0; j < height; j += 20) {
      caustics.push({ x: i, y: j, offset: random(TWO_PI) });
    }
  }

  // Create a button to connect to the serial port
  let connectButton = createButton("Connect to Serial");
  connectButton.position(10, 10);
  connectButton.mousePressed(connectToSerial);
}

function draw() {
  background(0, 0, 150); // Deep blue water

  // Draw caustics
  drawCaustics();

  // Draw and update all ripples
  for (let i = ripples.length - 1; i >= 0; i--) {
    let ripple = ripples[i];
    drawRipple(ripple.x, ripple.y, ripple.size);
    ripple.size += 2;

    if (ripple.size > 100) {
      lotuses.push({ x: ripple.x, y: ripple.y, frame: 0 });
      ripples.splice(i, 1);
    }
  }

  // Draw and animate blooming lotuses
  for (let lotus of lotuses) {
    drawBloomingLotus(lotus.x, lotus.y, lotus.frame);
    if (lotus.frame < 30) lotus.frame++;
  }

  // Use sensor values to create ripples
  if (sensorValues.length > 0) {
    for (let r = 0; r < sensorValues.length; r++) {
      for (let c = 0; c < sensorValues[r].length; c++) {
        if (sensorValues[r][c] > 100) {
          let x = map(c, 0, sensorValues[r].length, 0, width);
          let y = map(r, 0, sensorValues.length, 0, height);
          ripples.push({ x, y, size: 10 });
        }
      }
    }
  }
}

async function connectToSerial() {
  // Request a port and open a connection
  try {
    port = await navigator.serial.requestPort();
    await port.open({ baudRate: 9600 });
    serial = port.readable.getReader();

    // Start reading data from the serial port
    readSerial();
  } catch (err) {
    console.error("Error connecting to serial:", err);
  }
}

async function readSerial() {
  while (port.readable) {
    try {
      const { value, done } = await serial.read();
      if (done) {
        serial.releaseLock();
        break;
      }
      // Decode the data and process it
      const data = decoder.decode(value);
      processSerialData(data.trim());
    } catch (err) {
      console.error("Error reading serial data:", err);
      break;
    }
  }
}

function processSerialData(data) {
  // Parse CSV data from Arduino
  let rows = data.split("\n");
  sensorValues = rows.map((row) =>
    row.split(",").map((val) => parseInt(val.trim()))
  );
}

function drawRipple(x, y, size) {
  noFill();
  stroke(255, 255, 255, 100);
  strokeWeight(2);
  ellipse(x, y, size, size);
}

function drawBloomingLotus(x, y, frame) {
  push();
  translate(x, y);

  let scaleAmount = map(frame, 0, 30, 0, 1);
  noStroke();

  fill(255, 182, 193);
  for (let angle = 0; angle < TWO_PI; angle += PI / 6) {
    let petalX = cos(angle) * 30 * scaleAmount;
    let petalY = sin(angle) * 30 * scaleAmount;
    ellipse(petalX, petalY, 20 * scaleAmount);
  }

  fill(255, 105, 180);
  ellipse(0, 0, 20 * scaleAmount);

  pop();
}

function drawCaustics() {
  noStroke();
  fill(255, 255, 255, 30);

  for (let caustic of caustics) {
    let wave = sin(frameCount * 0.05 + caustic.offset) * 5;
    ellipse(caustic.x + wave, caustic.y + wave, 15, 15);
  }
}

Possibility to improve cleanup on React code - not implemented.

For these 3 error messages:

• The RAF (RequestAnimationFrame) violation - this is just a performance warning and not related to our serial connection
• The message port error - this is a Chrome-specific message that's actually not an error, just a notification
• The Bluetooth service message - this is just informing us that AirPods were filtered out of the serial port selection

const P5SketchWrapper = () => {
  // ... existing state declarations ...

  // Add connection status tracking
  useEffect(() => {
    // Handle tab visibility changes
    const handleVisibilityChange = () => {
      if (document.hidden && isConnected) {
        console.log('Tab hidden, maintaining connection...');
      }
    };

    // Handle before unload
    const handleBeforeUnload = () => {
      if (isConnected) {
        console.log('Page closing, cleaning up serial connection...');
      }
    };

    document.addEventListener('visibilitychange', handleVisibilityChange);
    window.addEventListener('beforeunload', handleBeforeUnload);

    return () => {
      document.removeEventListener('visibilitychange', handleVisibilityChange);
      window.removeEventListener('beforeunload', handleBeforeUnload);
    };
  }, [isConnected]);

  const readSerial = async (reader, writer) => {
    try {
      while (true) {
        const { value, done } = await reader.read();
        if (done) {
          console.log('Serial device disconnected normally');
          break;
        }
        
        const text = new TextDecoder().decode(value);
        const values = text.trim().split(',').map(Number);

        if (values.length === 6) {
          setCircleSizes(values);
        }
      }
    } catch (err) {
      if (err.name !== 'NetworkError') {
        console.error('Serial read error:', err);
      }
    } finally {
      setIsConnected(false);
    }
  };

  // ... rest of your component code ...
};

Playtest #2 Feedback:

• Don’t use real news
• Overwhelming cloud of negative words
• AI generated Gibberish

If you put the installation on the floor people will step on it.

❤️ Grounding ❤️

When you come in, i should

Claspy mechanisms / things

Nasif likes when name contextualizes

Rubbery gas kit - for rubber Talk to phil 2 clasps against each other L brackets

Digital noise that dissipates with collective play

Demo present or video/pics to submit Have people interact Class can go to staging space

ICM Visual Questions:

• Effect of text melting / dissipating into a swirl → into a pond
• setup function in p5 is being fired repeatedly
• Looked into GitHub issues → could be because of States
• I’m also putting the variables in a cleanup function.
• using the updateWithProps p5 react wrapper function.

Embed GitHub

FastAPI in Containers - Docker - FastAPI

Solving issue of React re-rendering setup p5 function → switching from state to refs

Key changes

• Replaced all useState with useRef
• Added refs for visualization states (ripples, lotuses, caustics)
• Removed state updates that were causing re-renders
• Used p5Instance.current?.redraw() to trigger redraws only when needed
• Removed dependency array from cleanup useEffect since we're using refs
• Modified all state access to use .current property of refs

The visualization will now update only when new serial data arrives or when explicitly redrawn.

Oceanic whippet by ranjanir -p5.js Web Editor

WhatRuns - Chrome Web Store

p5xjs Setting Up Your Environment

npm npm: p5

Connecting all 36 columns x 3 rows = 108 home made FSRs

Fabrication

For the Optoma projector to work, we’d need to raise the structure by 15 inches~