Key Takeaways
- jev-drone is a MuJoCo simulation project: a quadrotor chases a ground rover through a five-station obstacle course using only its onboard camera. No scripted trajectories are involved.
- A small judgment model (TypeSafe Jev) is consulted at roughly 2.5–3Hz and is advisory only. 50Hz code always holds the safety veto.
- Ablation: with Jev switched off, the greedy baseline never clears station 2. All three runs stopped at x=17.7m because it cannot express a “fly over” maneuver.
- With Jev in the loop, one 65-second run covered 77.5m, held the target in view 82% of the time (baseline about 19%), and spent 9% of the run in the safety reflex (baseline 65–71%). Collisions were zero on both sides.
- The headline result is a single run, not a seed-matched mean. On an earlier, simpler course Jev showed no advantage, and the authors say so explicitly.
Putting a language-scale model inside a real-time drone control loop is not really a question of whether it can answer, but of how fast it answers and what happens when it is wrong. Inference latency is measured in hundreds of milliseconds; a quadrotor’s reflex is measured in milliseconds. jev-drone, an open-source MuJoCo project, resolves this with a clean split: the code decides what is out there, and the model decides what it means and what to do about it. The airframe is the Skydio X2 from MuJoCo Menagerie, with real geometry, four rotors and a geometric controller. The mission is led by a red ground rover: follow it, and hit nothing.
How jev-drone Splits the Drone Control Loop
Most teams pick one of two extremes. Hand-written rules are safe, but “the obstacle ahead is low enough to fly over” is very hard to write as a rule. Letting the model emit control setpoints directly is slow, and nobody dares ship it. jev-drone sits between them, and the split is the whole design.
The Five-Station Obstacle Course
Each station is built to break one specific assumption. The pillar slalom tests ordinary avoidance. A low crossbeam spans the entire corridor with no gap on either side, so the only answer is to climb over it. A rotating arm sweeps back and forth and has to be timed. A sliding gate leaves a roughly 3.2m gap that moves laterally; the door is too tall to climb, so the drone must thread the slot. Finally, a dense pillar cluster, where the authors recorded the drone genuinely losing the rover, flying to its estimated position and reacquiring it. They call that recovery segment “the difference between a demo and a system.”
Ablation: What the Baseline Cannot Do
The ablation is the most convincing part of the project. Switch Jev off and the system degrades to a greedy “turn toward the wider side” baseline. It does not crash, but it never clears station 2: all three runs stopped at x=17.7m, because the climb-over maneuver simply is not expressible in that policy. With Jev in the loop, a single 65-second run completed the 77.5m course, kept the target in view 82% of the time against roughly 19% for the baseline, and cut time stuck in the safety reflex from 65–71% down to 9%. Both sides recorded zero collisions. The authors are careful about the claim: the Jev column is one run, not a seed-matched average, and on an earlier simpler course a three-seed comparison showed no Jev advantage. The claim the repository actually supports is narrow — the baseline structurally lacks that maneuver, and the model supplies it.
One Job Per Frequency
According to the run.py file header and the module docstrings, the system is split into four layers, each running at its own rate.
| Layer | Rate | Module | Responsibility |
|---|---|---|---|
| Geometric controller | 500Hz | flight.Pilot | Velocity → attitude → motor thrust cascade, thrust-priority mixing, slew-limited commands |
| Guidance + hard safety reflex | 50Hz | run.Guidance | Always holds the safety veto; slows and slides toward open space when path_ahead is under 2.2m |
| Onboard perception | 15Hz | flight.Eye | Turns 96×72 depth and segmentation buffers into a one-page scene summary |
| Tactical judgment | ~2.5–3Hz | tactics.Tactician | Advisory only: maneuver, risk score, whether the target is truly lost |
The 500Hz layer uses quaternion attitude error rather than a geometric error, because the geometric formulation drives restoring torque to zero at 180° of inverted flight, while the quaternion version keeps torque maximal there. The 15Hz perception layer emits five forward distance sectors, obstacle top-edge height, clearance overhead / level / below, plus target bearing and range. It never cheats with true coordinates — what the drone sees is what a real camera could see. Only the top layer talks to the model, and it consumes JSON and returns typed answers with probabilities.
Perception Details Worth Copying
According to flight.py, the camera is mounted at the nose rather than the body center, so the airframe does not register as its own obstacle. Threat assessment uses only a narrow cone about 40° dead ahead, while the 110° wide lens is reserved for target tracking — being able to see something is not the same as being able to hit it, and a source comment notes that the middle third of the wide lens mostly measures corridor walls. Each sector takes the nearest distance and then applies a median filter over the last three frames; without it, single-pixel depth noise makes clearance readings jump around, and no decision layer can commit to a signal like that.
Installation and First Flight
Dependencies are light: mujoco 3.13+, numpy, typesafe-sdk, pillow, imageio and imageio-ffmpeg. The entry point is setup.sh, which creates a virtual environment, sparse-checkouts only the skydio_x2 airframe directory from MuJoCo Menagerie, and symlinks the asset paths. The judgment model goes through the TypeSafe API, so TYPESAFE_API_KEY must be present in .env. tactics.py is blunt about it: without that variable, Tactician initialization raises a RuntimeError, and the flight code will not pretend it can run.
./setup.sh
cp .env.example .env # get a key from console.typesafe.ai/keysThen load the environment and run. MUJOCO_GL picks the render backend: glfw on a desktop, egl on a headless server.
set -a && . ./.env && set +a
export MUJOCO_GL=glfw # use egl on headless servers
.venv/bin/python run.py --seconds 65 --seeds 1One easily missed detail lives in run.py: simulation time is forced to track wall clock, with a comment saying that otherwise “you are cheating against a model that takes hundreds of milliseconds per call.” The README likewise labels –fast as “unfair to Jev.” Add the TRACE=1 prefix to watch every judgment as it changes, printing maneuver, probability, risk, position and the number of blocked sectors.
Each run prints one line of JSON metrics: collisions, target_visible_pct, mean_standoff_m, distance_flown_m, crossed_barrier, crashed_at_s, plus Jev statistics for call count, fingerprint skips, token spend, and median and p90 latency. Two signals tell you the run succeeded: crossed_barrier is true and collisions is zero. If nothing runs, check that the key is loaded and the MUJOCO_GL backend is available. Everything is tunable: –seeds changes the random seeds, –seconds changes duration (source default 35 seconds, README example 65), and –hz and –budget lower the model call rate and budget ceiling so you can trial cheaply first.
Two More Workflows: Ablation and Replay
The first is the ablation control, and it is completely free. –no-jev disables the model; pair it with –fast and multiple seeds to reproduce where the baseline gets stuck. The second is video and replay. –video renders an annotated video and simultaneously writes per-frame pose, depth and judgments into a .tape.npy tape file. replay.py re-renders from the tape without running physics, calling the API or spending credits, so you can re-cut any time window. The hud.py panel draws the whole Jev layer: the onboard depth map, five sector clearance bars, three clearance levels, a probability bar per maneuver, and three gauges for risk, confidence and whether the target is genuinely lost, with call count and latency along the bottom.
.venv/bin/python run.py --no-jev --fast --seconds 65 --seeds 0 1 2 # free ablation baseline
.venv/bin/python run.py --seconds 65 --seeds 1 --video course.mp4 # video with telemetry
.venv/bin/python replay.py course.mp4.tape.npy out.mp4 --from 12 --to 30How a Single Judgment Call Happens
The code first decides whether the question is worth asking. According to tactics.py, a call is only issued when the nearest obstacle is inside 4m, at least one sector is blocked, or the target has been lost for more than a second. On an open corridor with the target in sight, it never asks. Before each hand-off it computes a coarse scene fingerprint, and if the situation has not materially changed it reuses the previous judgment. The fuse is also set in code: at most 160 calls per episode, at most 3Hz. Gated in practice, the two runs recorded in the README came to about 110 calls over 65 seconds and 80 calls over 80 seconds, roughly 1.2–1.7Hz — not a constant 2.5, and it should not be overstated.
The call never blocks flight. Tactician runs on its own worker thread, offer() pushes a scene into the queue without blocking, and read() returns the cached judgment at any time. The flight code only checks two things: whether the judgment came from Jev, and whether it is still inside the 1.5-second freshness window. Each call asks three typed questions at once: maneuver is one of six options (hold heading, veer left, veer right, climb, brake, reacquire), risk scores current danger, and target_truly_lost separates a genuinely lost target from one briefly hidden behind a pillar.
The adoption layer adds debounce and veto. Once a maneuver is committed it is not changed for about 1.4 seconds, which prevents flapping between options. When the model proposes climb, the code requires at least four blocked sectors and overhead clearance meaningfully greater than level clearance before allowing it. A risk score above 1.45 triggers unconditional deceleration first. At the bottom sits the 50Hz hard reflex, which looks only at the path directly ahead (the source factors out path_ahead_m; side walls do not trigger it) and below 2.2m cuts speed and slides toward the more open side. The README notes this boundary was taught by the sliding gate: braking on “nearest obstacle in the whole field of view” marks every narrow slot as permanently off-limits.
The most valuable lesson in the project is about state design. Jev initially never chose climb. Debugging showed it was not a model failure: the state contained only five horizontal sectors with no vertical information at all, so flying over an obstacle was not even inferable. Once obstacle top-edge height, whether the top edge is visible, and the drone’s own climb ceiling were added to the state, the probability of choosing climb rose to 0.93. The authors’ conclusion: the answer has to already exist in the state. That was a bug in the state design, not a model failure.
Reacquisition is worth copying too. Camera bearing goes stale the instant the target leaves the frame, so Guidance continuously converts bearing, range and its own pose into a world-frame target position and extrapolates with observed velocity. That velocity must be differenced over roughly a one-second baseline; a source comment notes that single-frame pixel noise at 0.07 seconds differences out to absurd speeds of tens of m/s. After a loss the drone flies to the estimated position and then layers a sweeping search on top. That is exactly how the pillar cluster got the rover back.
Limits and Costs
First, the nature of the thing: this is simulation, not hardware. The X2 airframe comes from MuJoCo Menagerie and is distributed under its own license; the project itself is MIT. The judgment model runs over a network API, so it needs a key, a connection and per-token billing, with a hard 160-call budget per episode set in code. The README’s tunnel experiment measured a median latency of about 0.118 seconds and offers a reaction budget: at 9m/s, perception costs 0.03 seconds and the model 0.118 seconds, while a 2m lateral displacement of the airframe takes 0.7 seconds. The model is only a fraction of the loop, and making the decision faster would not help much.
Keep the evidence in proportion. The full-course success is a single 65-second run, and the authors stress the variance and the absence of any advantage on simpler courses. Do not retell it as “the model made the drone better.” There is also a simulation limitation: moving obstacles are implemented by teleporting mocap bodies, and their speed is deliberately slowed, because a fast sweep could otherwise spawn an obstacle inside the airframe. Sensor noise, communication jitter and wind are all outside the scope of this work, and they are what you would have to solve to move this layering onto a real airframe.
How to Get Started
Suggested order: run the –no-jev –fast baseline first and watch it stop in front of the crossbeam; then configure the key and run once with the model in the loop, comparing the metric differences between the two policies; finally read the three questions, the scoring table and all thresholds at the top of tactics.py — the authors deliberately concentrated everything a human should review in that one file. Repository: github.com/RomanSlack/jev-drone.
FAQ
What is jev-drone?
jev-drone is an open-source MuJoCo simulation project in which a Skydio X2 quadrotor follows a ground rover through a five-station obstacle course using only its onboard camera. A small judgment model advises the control loop while ordinary code keeps the safety veto.
Does the model actually control the drone?
No. The model only advises on maneuver choice, risk level and whether the target is genuinely lost, at roughly 2.5–3Hz. A 50Hz guidance layer can veto any suggestion, and a 50Hz hard reflex cuts speed when an obstacle is inside 2.2m directly ahead.
Why does the baseline fail at station 2?
The greedy baseline can only turn toward the wider side, and a low crossbeam spans the entire corridor with no gap. All three baseline runs stopped at x=17.7m because “climb over it” is not expressible in that policy, which is precisely the gap the judgment model fills.
How much does the judgment model cost per run?
Calls are gated to at most 160 per episode and 3Hz, and scene fingerprints skip repeats. Recorded runs used about 110 calls over 65 seconds and 80 over 80 seconds, roughly 1.2–1.7Hz. Billing is per token through the TypeSafe API, so a key is required.
Can this be moved to a real drone?
Not directly. The project is simulation-only, moving obstacles are teleported mocap bodies with deliberately reduced speed, and the model needs a network round trip of about 118ms median. Sensor noise, communication jitter and wind would all need to be solved first.
About Aomway
Aomway has spent more than 15 years building video transmission, antenna and flight-control hardware for FPV and industrial UAV teams. Explore the full range of Aomway video transmitters, antennas and optical fiber systems for long-range and interference-heavy missions.
Have questions about this article? Feel free to contact us at [email protected] — we’re happy to help!
