Seeing without a camera
Building a real-time activity detector from WiFi signals: an ESP32, a signal-processing pipeline, a Random Forest, and a live dashboard with a 3D pose view.
When a person moves through a room, they change how WiFi signals bounce around it. No camera, wearable or microphone is involved. The router is already there; the person is the thing disturbing its signal.
This project turns that disturbance into something readable in real time: is anyone here, how much are they moving, are they breathing, did someone fall, and are they sitting or standing. This post covers how the system is built, from the serial port to the browser, and the signal-processing choices that made it behave.
What CSI is, in one paragraph
Channel State Information describes how a WiFi signal arrived at the receiver, subcarrier by subcarrier. A still room gives a steady pattern. A moving person reshapes the reflections, so the pattern changes. An ESP32 running CSI firmware streams these values over USB serial, and everything else is software that decides what the changes mean.
The data flow
Data flow
- baud autodetect (921600, 115200, …)
- line parser: CSV, JSON or array
- alternatives with the same interface: replay and simulated sources
- feeds a 60 s ring buffer
- writes labelled CSV while recording
One interface, three sources
The pipeline never knows where frames come from. One source reads the real board, one plays back a recorded session, and one generates synthetic frames. That means I can develop the dashboard with no hardware attached, replay a recorded session to debug a detector, and run the same code path in all three cases.
The recorder writes labelled CSV, and that is also the replay format. One file format serves as training data, test fixture and demo reel.
Threads and concurrency
Threads
| Thread | Role |
|---|---|
| Source thread | reads serial, file or simulator, parses, and pushes frames to a bounded queue. Drops the oldest frame when full. |
| Consumer thread | drains the queue into the pipeline ring buffer and the recorder. |
| uvicorn event loop | REST handlers and a WebSocket tick of about 15 Hz. Heavy metric math runs in the default executor so the loop never blocks. |
Two decisions matter most. First, the queue is bounded and drops the oldest frame when it fills: for a live display, fresh data beats complete data, and a slow consumer should not create an ever-growing lag. Second, switching sources stops the old thread and rebuilds the pipeline, detectors and classifier from scratch, so state from a recording can never leak into a live session.
Signal processing choices
This is where most of the real work is. Each choice is a small decision with a reason.
- Amplitude only. ESP32 CSI phase is dominated by sampling-time offset and packet-boundary jitter. Without multi-antenna phase sanitisation it adds noise rather than signal, so I use amplitude and ignore phase.
- EMA at ingest (α = 0.3). An exponential moving average keeps the per-frame cost proportional to the number of subcarriers.
- Hampel filter per window, not per frame. It is more expensive, so it runs once per analysis window instead of on every frame. Cheap work runs often; costly work runs rarely.
- Resample irregular timing. CSI arrives per received packet, so the samples are not evenly spaced. Before any FFT or Welch analysis, I resample onto a uniform grid by interpolation. Skipping this step quietly corrupts every frequency estimate.
- Motion energy. The mean over subcarriers of the temporal standard deviation over a 2 s window. It is the standard, robust statistic for "is the channel changing", and most detectors build on it.
Every tunable lives in one place, so changing a threshold never means hunting through the code.
From signal to meaning
The detectors read the same shared buffer in parallel:
- Presence uses calibration of the empty room and hysteresis, so it does not flicker around its threshold.
- Motion level reports how much the channel is changing.
- Breathing estimates breaths per minute from the power spectral density, which only works when the person is still.
- Fall is a heuristic on the motion pattern.
- Activity is a Random Forest over features from sliding windows, trained on labelled recordings, with the model hot-reloaded when a new one is saved so I can retrain without restarting the server.
Heuristics handle what has a clear physical signature (presence, breathing). A learned model handles what does not (sitting versus standing). Choosing the simple tool where it works kept the classifier small.
The dashboard
The web UI has no build step and is served as plain static files. A single WebSocket message carries three things:
- Heatmap rows: only the new columns since the last tick.
- Metrics: detector, activity and status values, sent about three times a second.
- Motion series: a 20 s motion-energy sparkline.
The waterfall heatmap scrolls by shifting the existing canvas left and painting only the new columns, so the cost per tick stays constant however long the session runs. Sending deltas rather than the full history is what keeps a 15 Hz display light on both the server and the browser.
The 3D pose view
The most fun part is the 3D pose view: a Three.js scene with a procedural skeleton avatar, bundled locally with no CDN. The dashboard passes every metrics message to the 3D scene, which maps it onto the avatar.
What drives the avatar
| Signal | Avatar response |
|---|---|
| activity label | pose target (standing, sitting, walking, waving, lying). Joints ease toward it each frame; walking adds a gait cycle scaled by motion level. |
| breathing_bpm | chest scale oscillation, visible when the person is still. |
| fall | snaps to lying and shows a pulsing red floor ring. |
| motion level | agitation, and the colour of the router-to-ESP32 link line. |
| label = empty | avatar ghosts to 10 % opacity. |
A bug that taught me something
With a trained model loaded, the avatar follows the classifier label directly and fades out only when the label is empty. It is deliberately not gated on the presence detector. The presence detector can briefly drop while someone sits very still, and gating on it would freeze the avatar a label behind. Presence only controls opacity when no model has been trained yet. When two signals disagree, decide which one owns each decision.
On the dashboard, activity colours live in one place, applied to the Activity tile and to each per-class probability bar. The avatar itself keeps fixed material colours, so colour means one thing in each place.
The honest limit
The avatar is a visualisation of the classifier's pose class. It is not measured joint positions. A single-antenna ESP32 cannot provide those, so I would be misleading anyone who read the skeleton as a pose estimate. Showing the label as an animated figure is a good way to make the output understandable, as long as the post and the UI say what it really is.
What I would do next
These are ideas, not shipped work:
- More than one room and one person. The model learns the room it was trained in, so a quick recalibration flow would make it usable elsewhere.
- More antennas, which is what makes phase usable and opens the door to real location and pose information.
- More activity classes, now that recording and retraining with hot-reload is cheap.
Takeaways
- Make the data source swappable. Real, recorded and simulated sources behind one interface made everything easier to test.
- Prefer fresh data over complete data in anything live: bounded queues that drop the oldest frame.
- Do cheap work often and expensive work rarely: EMA per frame, Hampel per window.
- Fix timing before frequency analysis. Uneven samples break FFTs silently.
- Be clear about what a visualisation is. A moving avatar says "classifier label", not "measured skeleton".
If you work on WiFi sensing, I would like to hear how you handle phase on single-antenna hardware, and what you do about moving between rooms.