Face tracker for the Reachy Mini, written in Odin with a thin C++ shim. The same source builds for x86_64 (dev box, USB webcam or daemon socket) and for aarch64 (the Pi inside the robot). The aarch64 build links statically and ships as a single binary.

No OpenCV. Detection is libfacedetection (INT8 CNN with NEON on the Pi, AVX2 on x86). JPEG is libjpeg-turbo. Resize is stb_image_resize2.h. Text is rendered from an embedded 8x8 bitmap font.

Three threads share frames through bounded, freshest-wins handoffs. The main loop never blocks on detection or JPEG encoding.

flowchart LR
  cam[camera<br/>gst IPC or rpicam] --> cap[capture]
  cap --> main[main loop<br/>resize, draw, motor send]
  main -- submit frame --> det[detect worker<br/>libfacedetection]
  det -- latest result --> main
  main -- offer frame --> enc[encoder thread<br/>libjpeg-turbo]
  enc --> http[MJPEG HTTP<br/>port 8080]
  main -- POST /api/move/set_target<br/>GET /api/state/full --> daemon[Reachy daemon<br/>port 8000]
  daemon --> motors[head + body motors]

The detect worker and the encoder both follow the same rule: the producer hands off a frame, and any unconsumed previous frame is dropped. Detection stays at most one frame behind capture even when the worker can't keep up.

sequenceDiagram
  autonumber
  participant Cam as camera fd
  participant Main as main loop
  participant Det as detect worker
  participant Trk as tracker
  participant Bot as Reachy daemon

  Cam->>Main: read_frame (BGR)
  Main->>Bot: GET /api/state/full
  Bot-->>Main: head_yaw, head_pitch, body_yaw
  Main->>Det: submit(small_frame, snapshot)
  Det-->>Main: latest faces + paired snapshot
  Main->>Trk: tracker_update(face, snapshot, dt)
  Trk-->>Main: cmd_head_yaw, cmd_head_pitch, cmd_body_yaw
  Main->>Bot: POST /api/move/set_target (20 Hz)
  Main->>Main: draw overlay, offer to encoder

The motor snapshot read in step 2 is bundled with the frame in step 3 and travels with the detection result. The tracker pairs the face position with the pose that was current when the frame was captured. Without that pairing, fast head movement causes the tracker to combine a stale pixel offset with a fresh motor pose, double-counting motion and oscillating.

The tracker reasons in the world frame. The daemon's IK takes target_head_pose.yaw as the head's world pose and computes head_joint = head_yaw - body_yaw internally. Steady state is body equal to the face position and the head joint at zero.

Two timescales:

• A fast head EMA (head_alpha=0.55) on the world-frame target, clamped to the body plus or minus 30 degrees so the joint stays in range while the body lags.
• A slow body EMA (body_alpha=0.08) toward a smoothed face position (target_alpha=0.10). The body only steps when the head is still twisted relative to it. That guard is what stops the body once the head joint is straight.

When no face has been seen for 5 seconds the head sweeps around the body in a small sinusoidal idle pattern.

Requirements on the host:

• odin, g++, cmake, git, curl
• aarch64-linux-gnu-g++ and aarch64-linux-gnu-gcc for the Pi build
• nasm is optional (enables SIMD in libjpeg-turbo)

The first build downloads libfacedetection and libjpeg-turbo into .deps/ and caches the built archives in libs/<arch>/. Subsequent builds reuse the cache.

./build.sh           # native x86_64  -> ./reachy_vision
./build.sh aarch64   # cross to Pi    -> ./reachy_vision.aarch64
./build.sh clean     # wipe .deps and libs

The aarch64 binary is statically linked against the cross toolchain's bundled glibc, libstdc++, libgomp and libgcc. It has no NEEDED entries and runs on the stock Reachy image without library version coordination.

Local, with the daemon already running on 127.0.0.1:8000:

./reachy_vision --source=gst
# preview: http://localhost:8080

Local, without the daemon, talking to a Pi camera directly:

./reachy_vision --source=rpicam

On the Pi, after ./deploy.sh:

ssh pollen@reachy-mini.local /home/pollen/reachy_vision/run.sh
# preview: http://reachy-mini.local:8080

run.sh passes --source=gst --robot --robot-host=127.0.0.1. The gst source spawns gst-launch-1.0 unixfdsrc socket-path=/tmp/reachymini_camera_socket ! videoconvert ! videoscale ! ... ! fdsink fd=1, which consumes the daemon's camera IPC socket as a stream of raw BGR frames.

--robot defaults to off, so motor commands are only logged. Pass --robot to actually POST to the daemon.

./build.sh aarch64
./deploy.sh           # copy + restart
./deploy.sh logs      # tail /tmp/reachy_vision.log
./deploy.sh status    # show pid + last log lines
./deploy.sh stop      # kill it
./deploy.sh sleep     # POST /api/daemon/stop?goto_sleep=true
./deploy.sh wake      # POST /api/daemon/start?wake_up=true
./deploy.sh help

Env vars (defaults shown):

• PI_HOST=pollen@reachy-mini.local
• PI_PASS=root (set empty to use ssh keys)
• PI_DIR=/home/pollen/reachy_vision

deploy.sh rsyncs the binary plus run.sh into PI_DIR, kills any running tracker, then launches the new one under setsid -f so the ssh channel closes cleanly.