From store camera footage to 3D shelf attention: computer-vision software in the browser

An internal R&D project that turns one in-store security camera recording into 3D stick figures, head-direction cones and shelf attention, then runs batch analysis and writes a report. It runs entirely in the browser, and the video never leaves the computer.

I want something similar

Client
Internal R&D project (no client)
Industry
Retail (supermarkets, drugstores, convenience stores)
Location
Taiwan
Year
2026
Status
Internal R&D, not deployed for a client
3D aisle model with stick figures and orange head-direction cones, each cone showing the estimated error range
  • p95 frame-to-frame position jitter: 25.2 cm → 7.4 cm
  • 172 automated tests and 10 end-to-end checks
  • Batch analysis at about 2.5× real time, byte-identical across runs

Inside the system

We built this prototype in Traditional Chinese, so the screens are in Chinese.

Store camera view with the detected pose skeletons and the floor calibration frame.
Looking at the shelf face from the aisle: cells swept by the error cones accumulate attention.
Suggested changes from the layout study report (sample data): which product moves from which shelf level to which.

What was getting in the way

  • A camera cannot see the eyes, only estimate where the head points, so finer side-to-side detail is physically not there
  • A wrong coordinate transform does not show in the 3D view; only the shelf attention quietly comes out wrong
  • Pose estimates jitter from frame to frame, so the stick figures jump around
  • Long recordings have to be analyzed quickly, and the results have to be reproducible

How the system solves it

  • Work out the error budget first, set the resolution at shelf unit × level, and draw the error range on screen
  • Executable assertions on every coordinate transform, instead of checking by eye
  • One Euro filters that smooth position, body shape and head direction separately, plus matching each person across frames
  • A batch mode with fixed sampling and step-by-step seeking, GPU acceleration, and statistics in 5-minute buckets

What is inside

In-browser pose estimation

MediaPipe finds each shopper’s body keypoints in the browser. Pose only, no face recognition, and the video is never uploaded.

Homography calibration

Put markers on the floor and click them in the camera image. An N-point least-squares solve maps the image to the floor, with the error at each point and a check at standing positions.

3D shelves and head direction

Parametric shelves are generated in code and the skeletons placed in the 3D scene to estimate which shelf unit and level each head points at. The error range is drawn as a cone.

Reproducible batch analysis

Long recordings are analyzed step by step at a fixed sampling rate, with statistics in 5-minute buckets. Two runs on the same video give identical results.

Staff exclusion and camera pre-check

Excludes staff by the color of their ID vests and by restocking hours. A 2-minute recording shows whether the camera angle and resolution are good enough.

Layout study report

Validates the planogram CSV, aggregates two periods, detects changes and lists suggested product swaps. Results are flagged automatically when the sample is too small or the camera does not qualify.

What changed

  • One Euro filtering cut p95 frame-to-frame jitter from 25.2 cm to 7.4 cm for standing position and from 27.9 cm to 7.6 cm for head position
  • All 172 automated tests and 10 end-to-end checks pass, including 27 that run batch analysis on real video
  • Batch analysis of long recordings runs at about 2.5× real time on an Apple M5, so 48 hours of footage takes about 19 hours

How the project came together

This is an internal R&D project that combines computer vision with a 3D web interface. The input is one in-store security camera (CCTV) recording. The output is how much attention each shelf unit and each shelf level receives, plus a report with layout suggestions. Everything runs in the browser, so the video never leaves the computer.

The pipeline has several stages. MediaPipe finds each shopper’s body keypoints in the browser. It reads pose only and does no face recognition. Four markers on the floor give a homography, which maps positions in the camera image to real floor coordinates in meters. The skeletons are then placed in a procedurally generated 3D shelf model, head direction is estimated, and attention is accumulated into each shelf cell through an error cone. Long recordings run in a batch mode that writes statistics in 5-minute buckets.

We only claim the precision we actually have. A camera sees where the head points, not where the eyes look, and the eyes can turn about ±20–25° without the head moving. So we only claim which shelf unit and which level. The orange cone on screen is the error range, and attention is drawn as grid cells rather than a smooth rainbow heatmap. When the sample is too small or the camera is not good enough, the report flags the results as unusable for decisions.

A few engineering details. A coordinate transform once had a left-right mirroring bug that was completely invisible on screen, so every transform now has executable assertions. One Euro filtering cut the p95 frame-to-frame position jitter from 25.2 cm to 7.4 cm (head position: 27.9 cm to 7.6 cm). Long recordings are analyzed step by step at a fixed sampling rate, and the results are deterministic: the same video gives byte-identical results. On an Apple M5 the analysis runs at about 2.5× real time, so 48 hours of footage takes about 19 hours.

The project has 172 automated tests and 10 end-to-end checks, and it went through two rounds of independent code review before delivery. The reviews caught problems that would have put wrong numbers in the report, such as coverage skewed toward peak hours and staff-exclusion hours counted twice.

For us, the project comes down to three things: mapping a camera to physical space, tracking bodies over time, and being explicit about error.

Built with

  • MediaPipe Pose (WASM)
  • three.js / WebGL
  • JavaScript (ES modules)
  • Vite
  • Playwright (headless batch analysis and end-to-end tests)
  • Node.js

Want a system like this?

Tell us how your process works today. We will scope a version for your team, with a requirements document, a detailed quote and a timeline.

I want something similar

We reply within 24 hours. Or email support@foreverwebs.com