Overview
Software that sits inline between a Roku and the TV and rewrites faces in the video in real time -; giving everyone on screen googly eyes or a permanent scowl. The actor keeps acting; their face refuses to cooperate.
How It Works
An Elgato HD60 S captures the HDMI stream (it enumerates as a DirectShow webcam on Windows, so the whole thing is just a video pipeline). Each frame runs through a YuNet ONNX face detector to find faces, then MediaPipe Face Mesh meshes each one into 468 landmarks. Selected landmarks are shoved toward a target expression -; scowl, surprise, disgust, or full “crazy” -; and the real video pixels are warped to match via per-face triangulation. Offsets are measured in units of inter-ocular distance in the face’s own coordinate frame, so the effect stays correct as heads tilt, turn and change size. Googly-eye compositing rides on top.
# live, focus the video window:
0 passthrough (compare) 1/2/3 scowl / surprise / disgust
space cycle expressions [ / ] effect intensity
f fullscreen (drag to the TV first) q quit
It handles up to six faces at once and runs from either a recorded file (early testing was on a Murder, She Wrote episode) or the live capture device.
Current Status
Working on both recorded video and live capture. The expression set, multi-face detection, intensity controls and googly overlay are all in; tuning the warp intensity and capture-device wrangling are the current fiddly bits.