Production Incidents & Proven Solutions
📌 What is this for? (Quick Summary for New Interns)​
In Simple Words:
When you build complex systems combining hardware, Linux single-board computers, cloud databases, and web browsers, bugs will happen!Instead of letting future engineers spend days rediscovering the same tricky problems, we documented every real bug we ran into and exactly how we solved it.
If something breaks, check this table first before pulling your hair out!
1. The Incident Cheat Sheet​
| Incident | Where? | What Went Wrong? | What Caused It? | How We Fixed It |
|---|---|---|---|---|
| INC-01 | Login / API | Continuous 401 Unauthorized on localhost. dashboard.tutr.tech worked, but local failed. | .env was pointing to an old staging user pool (bg6ZECDii) instead of production (IXNCZVwj4). | Updated .env and main.jsx to IXNCZVwj4 and client 5eup0104notrn3fn4415u3fq3b. |
| INC-02 | Camera Stream | Video player sat black with an infinite spinning wheel on Chrome. | Cameras only sent a full keyframe every 15s. Chrome WebRTC refused to play until a keyframe arrived. | Added -g 15 (1s keyframe interval) and -bf 0 (zero delay frames) in stream_service/app.py. |
| INC-03 | MediaMTX Video Server | MediaMTX service crashed after an update (exit-code 1). | The config setting rtspTransports was deprecated in MediaMTX v1.9.3+. | Removed rtspTransports and replaced it with protocols: [tcp] in /etc/mediamtx/mediamtx.yml. |
| INC-04 | Cloud Lambda | Backend crashed with 500 Server Error on every request. | DynamoDB table ActiveSessions didn't exist yet; Lambda crashed on unhandled exception. | Added try...catch around database calls to safely ignore missing tables during setup. |
| INC-05 | AWS Security Rules | AWS command to allow web port 80 failed to apply. | Rule description had an unescaped apostrophe (Let's Encrypt). | Removed the apostrophe (Lets Encrypt). |
| INC-06 | GitHub CI Tests | Trajectory test test_t2_6 failed on GitHub Actions but passed locally. | .gitignore had *.csv, which ignored telemetry.csv so the test runner didn't have the data file. | Whitelisted !emulator/telemetry.csv in .gitignore and committed the file directly. |
| INC-07 | Three.js Visualizer | Browser console filled with red warnings on every frame render. | THREE.Clock is deprecated in newer Three.js versions. | Switched timing calculation to THREE.Timer or delta tracking. |
| INC-08 | Multi-Tab Safety | Opening a second tab caused the pod to jump or cancel commands. | Two people or tabs were controlling the vehicle at the same time. | Created "The Bouncer" session rule that auto-logs out older tabs. |
2. In-Depth Case Studies​
Read these detailed write-ups if you need the full backstory on our biggest fixes: