Skip to main content

Video Streaming & MediaMTX Architecture

📌 What is this for? (Quick Summary for New Interns)​

In Simple Words:
When operating a high-speed vehicle remotely, you need to see what is in front of the vehicle in real time.
If video lags by 5 or 10 seconds (like standard YouTube or Twitch streams), it would be far too late to stop the pod in an emergency.

This subsystem solves that problem:

  • It takes live video from cameras on the vehicle and streams it to our web dashboards in under 300 milliseconds (a fraction of a second).
  • If a camera gets unplugged or broken, it automatically switches to a colorful test screen so the operator knows the connection is working, but the camera is disconnected.

1. How Video Reaches Your Browser​

sequenceDiagram
    participant Cam as Physical Camera (192.168.1.200)
    participant StreamApp as stream_service (app.py on Pod)
    participant MediaMTX as MediaMTX Server (cam.tutr.tech)
    participant Browser as React Dashboard (LiveCamPiP)

    Note over StreamApp: Background Check: Is camera plugged in?
    StreamApp->>Cam: Quick network check (port 554, 1.5s timeout)
    alt Camera Connected
        Cam-->>StreamApp: Camera responds OK
        Note over StreamApp: Stream physical camera video
    else Camera Offline / Unplugged
        StreamApp-->>StreamApp: No answer
        Note over StreamApp: Automatically show color test pattern
    end

    StreamApp->>MediaMTX: Send video to cloud (Port 8554)
    Note over MediaMTX: Converts video to WebRTC (browser-friendly)

    Browser->>MediaMTX: Connect to stream (https://cam.tutr.tech:8889/cam1/whep)
    MediaMTX-->>Browser: Connection established
    MediaMTX->>Browser: Video packets stream instantly
    Note over Browser: Plays smoothly in <video> player with < 300ms delay

2. Why We Don't Use Standard Video Settings​

Normal security cameras only send a full picture ("keyframe") once every 10 to 15 seconds to save internet bandwidth.
If you open a web browser to watch the stream, the browser has to sit there doing nothing, waiting up to 15 seconds for the next keyframe before it can show you anything. That causes spinning loading wheels and black screens.

To fix this, in stream_service/app.py, we configured FFmpeg with specific low-latency settings:

cmd = [
    'ffmpeg', '-re',
    ...,
    '-c:v', 'libx264',
    '-preset', 'ultrafast',
    '-tune', 'zerolatency',
    '-profile:v', 'baseline',      # Simple format that every laptop and tablet can play
    '-pix_fmt', 'yuv420p',
    '-g', '15',                   # Sends a full picture every 1 second (15 frames at 15 FPS)
    '-bf', '0',                   # Zero delay frames: displays immediately without buffering
    ...,
    '-f', 'rtsp',
    '-rtsp_transport', 'tcp',
    target_rtsp
]

The 3 Golden Rules We Followed:​

  1. -g 15 (1 Full Picture Every Second): When someone opens the dashboard, they see video within 1 second instead of waiting 15 seconds.
  2. -bf 0 (Zero Delay Frames): Disables predictive frames that require the computer to wait for future frames before showing the current one.
  3. -profile:v baseline: Ensures video plays on old laptops, iPhones, Androids, or control tablets without hardware decoding errors.

3. Automatic Test Screen If Camera Unplugs​

If someone trips over an Ethernet cable on the test track and the camera disconnects:

  1. The stream_service probes the camera IP address every few seconds.
  2. If the connection fails, it doesn't crash. It seamlessly switches to generating a colorful SMPTE test pattern with an embedded clock timestamp.
  3. As soon as the cable is plugged back in, it detects the camera and switches back to live video automatically without needing to restart any services.