đź’»

Computational Media: Final

Computational Media: Final

Codebase available on GitHub:

Embed GitHub

As someone who had never coded before up until 1.5 years ago, something that fundamentally changed my understanding of computer systems was understanding where libraries like p5.js sit in the broader landscape. Running p5.js outside the web editor, serving it on my own machine and experimenting with different libraries was extremely valuable to me. When you run p5.js in your own local development environment, you gain greater control over the project's structure and can integrate and grow it with modern web development workflows. You can use package managers, build tools, and combine it with other libraries or frameworks. Part of that work is using the Web Serial API instead of a p5 specific library.

Not only that, integrating serial data with React Three Fiber and Arduino opens up a lot of possibilities for creating interactive installations by connecting physical sensors to 3D web environments through serial communication. By establishing real-time data flow between Arduino sensors and a web application, I can start creating responsive installations where physical interactions trigger digital responses. However, this setup requires careful management of the serial connection. Only one application can access a serial port at a time, for example.

At time of presentation, my final project ended up being a bit of a research project on how to integrate serial data with React Three Fiber and Three.js.

Bridging Physical and Digital: Serial Data in React Three Fiber

The process involves three main components:

  • Arduino sending sensor data in CSV format - using PlatformIO to execute and build the project
  • Serial communication handling in React
  • Data visualization in Three.js scene - using React.js, React Three Fiber and Drew

Key Considerations for future implementation

Serial Port Management:

  • Only one application can access a serial port at a time
  • Close the Serial Monitor in Arduino IDE when using WebSerial in the browser
  • Handle proper cleanup when component unmounts

Data Synchronization:

  • Add small delays between row readings to account for analog-to-digital converter limitations
  • Use proper calibration to establish baseline values - currently taking the average of 50 readings, however this could use some work.

State Management:

  • Use refs instead of state for visualization updates to prevent unnecessary re-renders
  • Trigger redraws only when new serial data arrives

Performance Optimization:

  • Avoid console.log during serial communication as it can slow down the process
  • Consider implementing a call-and-response pattern between p5 and Arduino

Other common pitfalls

  • Reset timing: If you reset without unplugging, the monitor reads correctly, but if you reset and unplug, the monitor may not complete successfully
  • React can't read if PlatformIO is reading the serial port - close the monitor!
  • Handle cleanup properly in your React component by closing the serial port when the component unmounts

Code samples: Getting serial data into Three.JS

image

It actually worked! The 36 sensors were mapped to the scale of the Tree.js object and it would move depending where the user touches one of the 36 homemade force sensing resistor sensor! Later on, to p5.js only (not a React app!) we managed to plug all 108 and ultimately showed a decent result on the Winter Show. The visuals there are lacking - will continue the work.

Code for Three JS scene

/src/App.jsx

import './App.css';
import { Canvas } from '@react-three/fiber';
import * as THREE from 'three';
import Scene from './components/Scene.jsx';
import { useRef, useEffect, useState } from 'react';
// import P5sketch from './components/p5sketch.jsx';
// import CircleSectionsP5 from './components/CircleSectionsP5.jsx';
// import P5SketchWrapper from './components/P5SketchWrapper.jsx';

function App() {
  const cameraRotation = [0, 1.5, 0];
  const cameraPosition = [4.5, 3, 0];

  const portRef = useRef(null);
  const readerRef = useRef(null);
  const writerRef = useRef(null);
  const treeScalesRef = useRef(new Array(27).fill(1.5)); // Store 27 values (9 for each circle)
    // Turn 27 into a variable
  const [isConnected, setIsConnected] = useState(false); // connection state

  // POssible soluton 👾🟢
  const [treeScalesUpdate, setTreeScalesUpdate] = useState(0);


  // Connect to serial port
  const connectSerial = async () => {
    try {
      const newPort = await navigator.serial.requestPort();
      await newPort.open({ baudRate: 9600 });

      const newReader = newPort.readable.getReader();
      portRef.current = newPort;
      readerRef.current = newReader;
      setIsConnected(true);  // Update connection state

      readSerial(newReader);
    } catch (err) {
      console.error('Serial port error:', err);
      setIsConnected(false);
    }
  };

  // Read serial data
  const readSerial = async (reader) => {
    try {
      while (true) {
        const { value, done } = await reader.read();
        if (done) {
          console.log("Serial port closed by device");
          setIsConnected(false);
          break;
        }

        // Parse the CSV data
        const text = new TextDecoder().decode(value);
        const values = text.trim().split(',').map(Number);

        if (values.length === 27) {
          // TODO: 27 as a variable Nr of values sent by Serial Monitor
          treeScalesRef.current = values;

          // 🌸 Possible solution 👾🟢
          // Force update
          setTreeScalesUpdate(prev => prev + 1);
          // 🌸 Possible solution 👾🟢

          // console.log('Tree scales being passed to Scene:', treeScalesRef.current);
        }
      }
    } catch (err) {
      console.error('Error reading serial:', err);
      setIsConnected(false);
    } finally {
      reader.releaseLock();
      readerRef.current = null;
    }
  };

  // Cleanup on unmount
  useEffect(() => {
    return () => {
      if (readerRef.current) {
        readerRef.current.cancel();
      }
      if (portRef.current) {
        portRef.current.close();
      }
    };
  }, []);

  return (
    <>
      {/* <P5SketchWrapper/> */}
      {/* <P5sketch/>
      <CircleSectionsP5/> */}

      {/* Connect button outside Canvas */}
      <div style={{
        position: 'fixed',
        top: '20px',
        left: '20px',
        zIndex: 1000,
      }}>
        {!isConnected && (
          <button onClick={connectSerial}>
            Connect to Serial Port
          </button>
        )}
      </div>

      <Canvas
          shadows
          dpr={ 1 }
          style={{ width: '100vw', height: '100vh', backgroundColor: 'rgb(181, 79, 111)'}}
          // position: 'fixed', top: 0, right: 0,

          gl={ {
            antialias: true, // For smooth edges
            alpha: true, // for transparent background
            toneMapping: THREE.ACESFilmicToneMapping, // Default
          } }
          camera={{
            fov: 45,
            near: 0.1,
            far: 200,
            position: cameraPosition,
            rotation: cameraRotation,
          }}>
          <Scene treeScales={treeScalesRef.current} key={treeScalesUpdate}/>
        </Canvas>
    </>
  )
}

export default App

/src/components/Scene.jsx

/* eslint-disable react/prop-types */

// cross talk
// IIE sensors magazine
// when you do a read, add a 1 milisecond delay everytime you change a row,
// the arduino has to connect to the analog to digital converter
// There's only 1.
// before you add analog read of each row, add a delay.

// or look in

// high resistors = less noise but less 

import { useEffect, useState } from 'react';
import Tree from './Tree';
import { SoftShadows, OrbitControls, MapControls } from "@react-three/drei";

export default function Scene({ treeScales }) {
  // React's rendering cycle isn't automatically triggering updates when treeScalesRef.current changes.
  const [scales, setScales] = useState(treeScales);

  // Update scales when treeScales changes
  useEffect(() => {
    setScales(treeScales);
  }, [treeScales]);


  const numberOfTrees = 36;

  // Includes negative values....
  const mapSerialToScale = (value) => {
    // Ensure value is a number and has a default
      // 🌸 So, is this type check OR transforming it into a number in case it was a string?
    value = Number(value) || 0;

    // Map from 0-700 (sensor range) to 0-1 (scale range)
    const inputMin = 0;
    const inputMax = 700; // 🌸
    const scaleMin = 0.15;
    const scaleMax = 1.5;

    // Linear mapping
    const mappedValue = scaleMin + ((value - inputMin) * (scaleMax - scaleMin)) / (inputMax - inputMin);

    // Debug log to see the values
    // console.log('Input value:', value, 'Mapped value:', mappedValue);

    // Clamp between 0 and 1 but it was already clamped by the Arduino?
    return Math.max(scaleMin, Math.min(scaleMax, mappedValue));
  };

  const getTreeScale = (circleType, index) => {
    if (index < 9) {
      switch(circleType) {
        case 'first': return mapSerialToScale(treeScales[index]);
        case 'inner': return mapSerialToScale(treeScales[index + 9]);
        case 'outer': return mapSerialToScale(treeScales[index + 18]);
        default: return 1.5;
      }
    }
    return circleType === 'outer' ? 1.5 : 1.25;
  };

  // const getOuterTreeScale = (index) => {
  //   if (index < 9) {
  //     return 5;
  //   }
  //   return 1.5;
  // };

  const generateTrees = () => {
    const trees = [];
    const firstRadius = 9;
    const innerRadius = 14;
    const outerRadius = 20; // Radius of the circle

    // FIRST (INNERMOST) CIRCLE OF TREES
    for (let i = 0; i < numberOfTrees; i++) {
      const angle = (i * 2 * Math.PI) / numberOfTrees;
      // const x = firstRadius * Math.cos(angle);
      // const z = firstRadius * Math.sin(angle);
      // const y = -1.5;
      // same as prev.

      trees.push(
        <Tree
          key={`first-${i}`}
          position={[
            firstRadius * Math.cos(angle),
            -1.5,
            firstRadius * Math.sin(angle)
          ]}
          scale={getTreeScale('first', i)}
          // position={[x, y, z]}
          // scale={1.25}
        />
      );
    }

    // INNER CIRCLE OF TREES
    for (let i = 0; i < numberOfTrees; i++) {
      const angle = (i * 2 * Math.PI) / numberOfTrees;
      const x = innerRadius * Math.cos(angle);
      const z = innerRadius * Math.sin(angle);
      const y = -1.5;

      trees.push(
        <Tree
          key={`inner-${i}`}
          position={[
            innerRadius * Math.cos(angle),
            -1.5,
            innerRadius * Math.sin(angle)
          ]}
          scale={getTreeScale('inner', i)}

          // position={[x, y, z]}
          // scale={1.25}
        />
      );
    }

    // OUTER CIRCLE OF TREES
    for (let i = 0; i < numberOfTrees; i++) {
      // Calculate angle for each tree (in radians)
      const angle = (i * 2 * Math.PI) / numberOfTrees;

      // Calculate position using trigonometry
      // const x = outerRadius * Math.cos(angle);
      // const z = outerRadius * Math.sin(angle);
      // const y = -1.5; // Consistent height for all trees

      trees.push(
        <Tree
          key={i}
          // position={[x, y, z]}
          // scale={getOuterTreeScale(i)}
          position={[
            outerRadius * Math.cos(angle),
            -1.5,
            outerRadius * Math.sin(angle)
          ]}
          scale={getTreeScale('outer', i)}
        />
      );
    }
    return trees;
  }
  return <>
    {/* 3D SCENE CONTENT */}
    <SoftShadows size={ 80 } samples={ 20 } focus={ 0 } />
    <OrbitControls />
    <MapControls />

    {generateTrees()}

    <ambientLight intensity={1} />
    <directionalLight
      position={[ 1, 3, 1.8]}
      intensity={ 4 }
      castShadow
      shadow-mapSize={[1024 * 3, 1024 * 3]}
      shadow-camera-top={ 4 }
      shadow-camera-right={ 4 }
      shadow-camera-bottom={ -3 }
      shadow-camera-left={ -2 }
      shadow-camera-near={ 0.5 }
      shadow-camera-far={ 50 }
    />
  </>
}

Arduino code

//////////////// TEST READINGS FOR 3 ROWS AND 2 COLS ///////////////
//// but readMat is called on loop instead of triggered by a person

// âś… continuously sends data instead of reading only when triggered by serial input '0'
// âś… machine-readable CSV format instead of verbose human-readable output
// âś… Baseline subtraction: difference calculated during output VS difference calculated during reading
// âś… performs calibration silently
// đź‘ľ New version automatically sends data every 100ms in a CSV format, while the original only sends data when requested through serial input


/////// TESTING with 9 sensors and multiplexer ///////
#include <Arduino.h>

// Pin definitions
const int ROW_PINS[] = {A0, A1, A2};  // Analog input pins for rows
const int MUX_SIG = 8;    // Multiplexer signal pin
const int MUX_S3 = 9;     // Multiplexer control pin S3
const int MUX_S2 = 10;    // Multiplexer control pin S2
const int MUX_S1 = 11;    // Multiplexer control pin S1
const int MUX_S0 = 12;    // Multiplexer control pin S0

const int NUM_ROWS = 3;
const int NUM_COLS = 9;

//// đź‘ľ PROBABLY NEEDS CHANGES /////
// Arrays to store sensor values
int currentValues[3][9];    // Current readings
int baselineValues[3][9];   // Calibration baseline
const int NUM_CALIBRATION_SAMPLES = 20;

void calibrate();
void readMat();

void setup() {
  Serial.begin(9600);

  // Set pin modes
  for (int r = 0; r < NUM_ROWS; r++) {
    pinMode(ROW_PINS[r], INPUT); // Analog
  }

  // Setup multiplexer pins
  pinMode(MUX_SIG, OUTPUT);
  pinMode(MUX_S3, OUTPUT);
  pinMode(MUX_S2, OUTPUT);
  pinMode(MUX_S1, OUTPUT);
  pinMode(MUX_S0, OUTPUT);
  digitalWrite(MUX_SIG, LOW);  // Start with signal LOW

  delay(1000);  // Allow serial to initialize
  Serial.println("Starting up...");
  calibrate();
}

// 🍑 Helper function to set multiplexer channel
  // handle the 4-bit channel selection
void setMuxChannel(int channel) {
  digitalWrite(MUX_S0, channel & 0x01);
  digitalWrite(MUX_S1, (channel >> 1) & 0x01);
  digitalWrite(MUX_S2, (channel >> 2) & 0x01);
  digitalWrite(MUX_S3, (channel >> 3) & 0x01);
}

//// đź‘ľ LOOP FUNCTION UNCHANGED?
void loop() {
  readMat();

  // Send sensor readings in CSV format
  for (int r = 0; r < NUM_ROWS; r++) {
    for (int c = 0; c < NUM_COLS; c++) {
      Serial.print(currentValues[r][c]);
      if (c < NUM_COLS - 1) Serial.print(","); // Add comma between columns
    }
    if (r < NUM_ROWS - 1) Serial.print(","); // Add comma between rows
  }
  Serial.println();  // Newline after each reading

  delay(20); // ⚪️ Adjust delay as needed
  // from 100, to 20, to completely removing
}

void calibrate() {
  for (int r = 0; r < NUM_ROWS; r++) {
    for (int c = 0; c < NUM_COLS; c++) {
      baselineValues[r][c] = 0;
    }
  }

  for (int sample = 0; sample < NUM_CALIBRATION_SAMPLES; sample++) {
    for (int c = 0; c < NUM_COLS; c++) {
      setMuxChannel(c);
      digitalWrite(MUX_SIG, HIGH);
      delay(10);
      for (int r = 0; r < NUM_ROWS; r++) {
        baselineValues[r][c] += analogRead(ROW_PINS[r]);
      }
      digitalWrite(MUX_SIG, LOW);
    }
  }

  // Averaging code?
  for (int r = 0; r < NUM_ROWS; r++) {
    for (int c = 0; c < NUM_COLS; c++) {
      baselineValues[r][c] /= NUM_CALIBRATION_SAMPLES;
    }
  }
}

void readMat() {
  for (int c = 0; c < NUM_COLS; c++) {
    setMuxChannel(c);
    digitalWrite(MUX_SIG, HIGH);
    delay(10);
    for (int r = 0; r < NUM_ROWS; r++) {
      // Clamping the values to 0 so that there's no negatives:
      int diff = analogRead(ROW_PINS[r]) - baselineValues[r][c];
      currentValues[r][c] = (diff < 0) ? 0 : diff;
    }
    digitalWrite(MUX_SIG, LOW); // MUX_SIG instead of COL_PINS[c]
  }
}

Next steps:

  • Fetching news story data from API. In the end we switched to p5 for simplicity’s sake within the team (visualisation below with all 108 sensors mapped to p5.)
  • How to render text within WebGL? We had a major bug when implementing p5 and I suspect it’s because WebGL would have some trouble rendering p5.
image

Notes from first feedback session

  • There’s a p5 speech library, because Chrome has text to speech.
  • For the news, can use dummy data like Reductress or The Onion
  • Prof recommends figuring out API last because what’s important is the interaction.
  • Condition the interaction.
  • People may treat it as game or press on it with brute force.
  • You can start pre-loading right as the user hovers.
  • For a project with a lot of images, load them on demand, but don’t call methods on the images until they’ve been loaded.