As someone who had never coded before up until 1.5 years ago, something that fundamentally changed my understanding of computer systems was understanding where libraries like p5.js sit in the broader landscape. Running p5.js outside the web editor, serving it on my own machine and experimenting with different libraries was extremely valuable to me. When you run p5.js in your own local development environment, you gain greater control over the project's structure and can integrate and grow it with modern web development workflows. You can use package managers, build tools, and combine it with other libraries or frameworks. Part of that work is using the Web Serial API instead of a p5 specific library.
Not only that, integrating serial data with React Three Fiber and Arduino opens up a lot of possibilities for creating interactive installations by connecting physical sensors to 3D web environments through serial communication. By establishing real-time data flow between Arduino sensors and a web application, I can start creating responsive installations where physical interactions trigger digital responses. However, this setup requires careful management of the serial connection. Only one application can access a serial port at a time, for example.
At time of presentation, my final project ended up being a bit of a research project on how to integrate serial data with React Three Fiber and Three.js.
Bridging Physical and Digital: Serial Data in React Three Fiber
The process involves three main components:
Arduino sending sensor data in CSV format - using PlatformIO to execute and build the project
Serial communication handling in React
Data visualization in Three.js scene - using React.js, React Three Fiber and Drew
Key Considerations for future implementation
Serial Port Management:
Only one application can access a serial port at a time
Close the Serial Monitor in Arduino IDE when using WebSerial in the browser
Handle proper cleanup when component unmounts
Data Synchronization:
Add small delays between row readings to account for analog-to-digital converter limitations
Use proper calibration to establish baseline values - currently taking the average of 50 readings, however this could use some work.
State Management:
Use refs instead of state for visualization updates to prevent unnecessary re-renders
Trigger redraws only when new serial data arrives
Performance Optimization:
Avoid console.log during serial communication as it can slow down the process
Consider implementing a call-and-response pattern between p5 and Arduino
Other common pitfalls
Reset timing: If you reset without unplugging, the monitor reads correctly, but if you reset and unplug, the monitor may not complete successfully
React can't read if PlatformIO is reading the serial port - close the monitor!
Handle cleanup properly in your React component by closing the serial port when the component unmounts
Code samples: Getting serial data into Three.JS
It actually worked! The 36 sensors were mapped to the scale of the Tree.js object and it would move depending where the user touches one of the 36 homemade force sensing resistor sensor! Later on, to p5.js only (not a React app!) we managed to plug all 108 and ultimately showed a decent result on the Winter Show. The visuals there are lacking - will continue the work.
Code for Three JS scene
/src/App.jsx
/src/components/Scene.jsx
Arduino code
Next steps:
Fetching news story data from API. In the end we switched to p5 for simplicity’s sake within the team (visualisation below with all 108 sensors mapped to p5.)
How to render text within WebGL? We had a major bug when implementing p5 and I suspect it’s because WebGL would have some trouble rendering p5.
Notes from first feedback session
There’s a p5 speech library, because Chrome has text to speech.
For the news, can use dummy data like Reductress or The Onion
Prof recommends figuring out API last because what’s important is the interaction.
Condition the interaction.
People may treat it as game or press on it with brute force.
You can start pre-loading right as the user hovers.
For a project with a lot of images, load them on demand, but don’t call methods on the images until they’ve been loaded.
import './App.css';
import { Canvas } from '@react-three/fiber';
import * as THREE from 'three';
import Scene from './components/Scene.jsx';
import { useRef, useEffect, useState } from 'react';
// import P5sketch from './components/p5sketch.jsx';
// import CircleSectionsP5 from './components/CircleSectionsP5.jsx';
// import P5SketchWrapper from './components/P5SketchWrapper.jsx';
function App() {
const cameraRotation = [0, 1.5, 0];
const cameraPosition = [4.5, 3, 0];
const portRef = useRef(null);
const readerRef = useRef(null);
const writerRef = useRef(null);
const treeScalesRef = useRef(new Array(27).fill(1.5)); // Store 27 values (9 for each circle)
// Turn 27 into a variable
const [isConnected, setIsConnected] = useState(false); // connection state
// POssible soluton 👾🟢
const [treeScalesUpdate, setTreeScalesUpdate] = useState(0);
// Connect to serial port
const connectSerial = async () => {
try {
const newPort = await navigator.serial.requestPort();
await newPort.open({ baudRate: 9600 });
const newReader = newPort.readable.getReader();
portRef.current = newPort;
readerRef.current = newReader;
setIsConnected(true); // Update connection state
readSerial(newReader);
} catch (err) {
console.error('Serial port error:', err);
setIsConnected(false);
}
};
// Read serial data
const readSerial = async (reader) => {
try {
while (true) {
const { value, done } = await reader.read();
if (done) {
console.log("Serial port closed by device");
setIsConnected(false);
break;
}
// Parse the CSV data
const text = new TextDecoder().decode(value);
const values = text.trim().split(',').map(Number);
if (values.length === 27) {
// TODO: 27 as a variable Nr of values sent by Serial Monitor
treeScalesRef.current = values;
// 🌸 Possible solution 👾🟢
// Force update
setTreeScalesUpdate(prev => prev + 1);
// 🌸 Possible solution 👾🟢
// console.log('Tree scales being passed to Scene:', treeScalesRef.current);
}
}
} catch (err) {
console.error('Error reading serial:', err);
setIsConnected(false);
} finally {
reader.releaseLock();
readerRef.current = null;
}
};
// Cleanup on unmount
useEffect(() => {
return () => {
if (readerRef.current) {
readerRef.current.cancel();
}
if (portRef.current) {
portRef.current.close();
}
};
}, []);
return (
<>
{/* <P5SketchWrapper/> */}
{/* <P5sketch/>
<CircleSectionsP5/> */}
{/* Connect button outside Canvas */}
<div style={{
position: 'fixed',
top: '20px',
left: '20px',
zIndex: 1000,
}}>
{!isConnected && (
<button onClick={connectSerial}>
Connect to Serial Port
</button>
)}
</div>
<Canvas
shadows
dpr={ 1 }
style={{ width: '100vw', height: '100vh', backgroundColor: 'rgb(181, 79, 111)'}}
// position: 'fixed', top: 0, right: 0,
gl={ {
antialias: true, // For smooth edges
alpha: true, // for transparent background
toneMapping: THREE.ACESFilmicToneMapping, // Default
} }
camera={{
fov: 45,
near: 0.1,
far: 200,
position: cameraPosition,
rotation: cameraRotation,
}}>
<Scene treeScales={treeScalesRef.current} key={treeScalesUpdate}/>
</Canvas>
</>
)
}
export default App
/* eslint-disable react/prop-types */
// cross talk
// IIE sensors magazine
// when you do a read, add a 1 milisecond delay everytime you change a row,
// the arduino has to connect to the analog to digital converter
// There's only 1.
// before you add analog read of each row, add a delay.
// or look in
// high resistors = less noise but less
import { useEffect, useState } from 'react';
import Tree from './Tree';
import { SoftShadows, OrbitControls, MapControls } from "@react-three/drei";
export default function Scene({ treeScales }) {
// React's rendering cycle isn't automatically triggering updates when treeScalesRef.current changes.
const [scales, setScales] = useState(treeScales);
// Update scales when treeScales changes
useEffect(() => {
setScales(treeScales);
}, [treeScales]);
const numberOfTrees = 36;
// Includes negative values....
const mapSerialToScale = (value) => {
// Ensure value is a number and has a default
// 🌸 So, is this type check OR transforming it into a number in case it was a string?
value = Number(value) || 0;
// Map from 0-700 (sensor range) to 0-1 (scale range)
const inputMin = 0;
const inputMax = 700; // 🌸
const scaleMin = 0.15;
const scaleMax = 1.5;
// Linear mapping
const mappedValue = scaleMin + ((value - inputMin) * (scaleMax - scaleMin)) / (inputMax - inputMin);
// Debug log to see the values
// console.log('Input value:', value, 'Mapped value:', mappedValue);
// Clamp between 0 and 1 but it was already clamped by the Arduino?
return Math.max(scaleMin, Math.min(scaleMax, mappedValue));
};
const getTreeScale = (circleType, index) => {
if (index < 9) {
switch(circleType) {
case 'first': return mapSerialToScale(treeScales[index]);
case 'inner': return mapSerialToScale(treeScales[index + 9]);
case 'outer': return mapSerialToScale(treeScales[index + 18]);
default: return 1.5;
}
}
return circleType === 'outer' ? 1.5 : 1.25;
};
// const getOuterTreeScale = (index) => {
// if (index < 9) {
// return 5;
// }
// return 1.5;
// };
const generateTrees = () => {
const trees = [];
const firstRadius = 9;
const innerRadius = 14;
const outerRadius = 20; // Radius of the circle
// FIRST (INNERMOST) CIRCLE OF TREES
for (let i = 0; i < numberOfTrees; i++) {
const angle = (i * 2 * Math.PI) / numberOfTrees;
// const x = firstRadius * Math.cos(angle);
// const z = firstRadius * Math.sin(angle);
// const y = -1.5;
// same as prev.
trees.push(
<Tree
key={`first-${i}`}
position={[
firstRadius * Math.cos(angle),
-1.5,
firstRadius * Math.sin(angle)
]}
scale={getTreeScale('first', i)}
// position={[x, y, z]}
// scale={1.25}
/>
);
}
// INNER CIRCLE OF TREES
for (let i = 0; i < numberOfTrees; i++) {
const angle = (i * 2 * Math.PI) / numberOfTrees;
const x = innerRadius * Math.cos(angle);
const z = innerRadius * Math.sin(angle);
const y = -1.5;
trees.push(
<Tree
key={`inner-${i}`}
position={[
innerRadius * Math.cos(angle),
-1.5,
innerRadius * Math.sin(angle)
]}
scale={getTreeScale('inner', i)}
// position={[x, y, z]}
// scale={1.25}
/>
);
}
// OUTER CIRCLE OF TREES
for (let i = 0; i < numberOfTrees; i++) {
// Calculate angle for each tree (in radians)
const angle = (i * 2 * Math.PI) / numberOfTrees;
// Calculate position using trigonometry
// const x = outerRadius * Math.cos(angle);
// const z = outerRadius * Math.sin(angle);
// const y = -1.5; // Consistent height for all trees
trees.push(
<Tree
key={i}
// position={[x, y, z]}
// scale={getOuterTreeScale(i)}
position={[
outerRadius * Math.cos(angle),
-1.5,
outerRadius * Math.sin(angle)
]}
scale={getTreeScale('outer', i)}
/>
);
}
return trees;
}
return <>
{/* 3D SCENE CONTENT */}
<SoftShadows size={ 80 } samples={ 20 } focus={ 0 } />
<OrbitControls />
<MapControls />
{generateTrees()}
<ambientLight intensity={1} />
<directionalLight
position={[ 1, 3, 1.8]}
intensity={ 4 }
castShadow
shadow-mapSize={[1024 * 3, 1024 * 3]}
shadow-camera-top={ 4 }
shadow-camera-right={ 4 }
shadow-camera-bottom={ -3 }
shadow-camera-left={ -2 }
shadow-camera-near={ 0.5 }
shadow-camera-far={ 50 }
/>
</>
}
//////////////// TEST READINGS FOR 3 ROWS AND 2 COLS ///////////////
//// but readMat is called on loop instead of triggered by a person
// âś… continuously sends data instead of reading only when triggered by serial input '0'
// âś… machine-readable CSV format instead of verbose human-readable output
// âś… Baseline subtraction: difference calculated during output VS difference calculated during reading
// âś… performs calibration silently
// đź‘ľ New version automatically sends data every 100ms in a CSV format, while the original only sends data when requested through serial input
/////// TESTING with 9 sensors and multiplexer ///////
#include <Arduino.h>
// Pin definitions
const int ROW_PINS[] = {A0, A1, A2}; // Analog input pins for rows
const int MUX_SIG = 8; // Multiplexer signal pin
const int MUX_S3 = 9; // Multiplexer control pin S3
const int MUX_S2 = 10; // Multiplexer control pin S2
const int MUX_S1 = 11; // Multiplexer control pin S1
const int MUX_S0 = 12; // Multiplexer control pin S0
const int NUM_ROWS = 3;
const int NUM_COLS = 9;
//// đź‘ľ PROBABLY NEEDS CHANGES /////
// Arrays to store sensor values
int currentValues[3][9]; // Current readings
int baselineValues[3][9]; // Calibration baseline
const int NUM_CALIBRATION_SAMPLES = 20;
void calibrate();
void readMat();
void setup() {
Serial.begin(9600);
// Set pin modes
for (int r = 0; r < NUM_ROWS; r++) {
pinMode(ROW_PINS[r], INPUT); // Analog
}
// Setup multiplexer pins
pinMode(MUX_SIG, OUTPUT);
pinMode(MUX_S3, OUTPUT);
pinMode(MUX_S2, OUTPUT);
pinMode(MUX_S1, OUTPUT);
pinMode(MUX_S0, OUTPUT);
digitalWrite(MUX_SIG, LOW); // Start with signal LOW
delay(1000); // Allow serial to initialize
Serial.println("Starting up...");
calibrate();
}
// 🍑 Helper function to set multiplexer channel
// handle the 4-bit channel selection
void setMuxChannel(int channel) {
digitalWrite(MUX_S0, channel & 0x01);
digitalWrite(MUX_S1, (channel >> 1) & 0x01);
digitalWrite(MUX_S2, (channel >> 2) & 0x01);
digitalWrite(MUX_S3, (channel >> 3) & 0x01);
}
//// đź‘ľ LOOP FUNCTION UNCHANGED?
void loop() {
readMat();
// Send sensor readings in CSV format
for (int r = 0; r < NUM_ROWS; r++) {
for (int c = 0; c < NUM_COLS; c++) {
Serial.print(currentValues[r][c]);
if (c < NUM_COLS - 1) Serial.print(","); // Add comma between columns
}
if (r < NUM_ROWS - 1) Serial.print(","); // Add comma between rows
}
Serial.println(); // Newline after each reading
delay(20); // ⚪️ Adjust delay as needed
// from 100, to 20, to completely removing
}
void calibrate() {
for (int r = 0; r < NUM_ROWS; r++) {
for (int c = 0; c < NUM_COLS; c++) {
baselineValues[r][c] = 0;
}
}
for (int sample = 0; sample < NUM_CALIBRATION_SAMPLES; sample++) {
for (int c = 0; c < NUM_COLS; c++) {
setMuxChannel(c);
digitalWrite(MUX_SIG, HIGH);
delay(10);
for (int r = 0; r < NUM_ROWS; r++) {
baselineValues[r][c] += analogRead(ROW_PINS[r]);
}
digitalWrite(MUX_SIG, LOW);
}
}
// Averaging code?
for (int r = 0; r < NUM_ROWS; r++) {
for (int c = 0; c < NUM_COLS; c++) {
baselineValues[r][c] /= NUM_CALIBRATION_SAMPLES;
}
}
}
void readMat() {
for (int c = 0; c < NUM_COLS; c++) {
setMuxChannel(c);
digitalWrite(MUX_SIG, HIGH);
delay(10);
for (int r = 0; r < NUM_ROWS; r++) {
// Clamping the values to 0 so that there's no negatives:
int diff = analogRead(ROW_PINS[r]) - baselineValues[r][c];
currentValues[r][c] = (diff < 0) ? 0 : diff;
}
digitalWrite(MUX_SIG, LOW); // MUX_SIG instead of COL_PINS[c]
}
}
