Editor’s note: The challenge of high-speed autonomous flight through cluttered forests and dense indoor spaces finally has a solution. A team from Beijing Institute of Technology introduces FlowPilot, a lightweight World-Action Model (WAM) that simultaneously predicts the future environment and generates smooth, trackable trajectories. Inference runs in under 18 ms on an embedded Jetson Orin NX, and a real drone reaches up to 5.5 m/s while weaving through obstacles, outperforming both classical planners and end-to-end learning methods.
- Authors: Runqing Wang, Ding Yu, Pengyuan Min, Xinhong Zhang, Wei Xiao, Yu Hu, Jie Chen, Fu Zhang, Gang Wang
- Affiliations: National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology; Zhongguancun Institute; Shanghai Jiao Tong University; Department of Mechanical Engineering, The University of Hong Kong
- Paper title: FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation
- Paper link: arxiv.org/abs/2608.00635
Key Takeaways
- FlowPilot is a lightweight onboard dual-stream World-Action Model that predicts future depth and generates trajectories together.
- It uses a 7th-order Bernstein polynomial action representation, giving analytically smooth velocity, acceleration and jerk without a post-smoothing module.
- The full onboard perception-to-trajectory pipeline runs in under 18 ms on a Jetson Orin NX, matching a 100 Hz MPC controller.
- A three-layer data pyramid (massive simulation, high-fidelity rendering, real drone data) plus two-stage training bridges the sim-to-real gap.
- Real flights reach 5.5 m/s in outdoor forest and stay collision-free in dense obstacles, beating EGO-Planner and YOPO baselines.
Research Background: Both Main High-Speed Avoidance Routes Have Hard Flaws
Scenarios such as search and rescue, power-line inspection and urban last-mile delivery all require small drones to fly fast and autonomously through dense obstacle environments without a prior map. The core conflict is that “perceptual anticipation” and “trajectory planning” cannot be coupled in real time and efficiently. The two mainstream technical routes each have weaknesses.
Classical Mapping-and-Optimization (EGO-Planner / GCOPTER)
These methods first reconstruct a local 3D map, then optimize a trajectory.
- Strengths: clear geometric constraints, smooth output trajectories, controller-friendly.
- Fatal flaw: at high speed in dense obstacles, map reconstruction lags, perceptual error accumulates, planning falls behind and collisions are easy; the speed ceiling is low.
End-to-End Learned Navigation (YOPO / high-speed field-flying policies)
These output motion commands directly from depth images, skipping the mapping step.
- Strengths: low inference latency, tolerant of mild sensor noise.
- Fatal flaw: mostly “reactive policies” that decide only from the current frame and do not anticipate upcoming environmental change, so they lack foresight at high speed.
Adaptation Shortcomings of Existing World-Action Models (WAM)
Mainstream WAMs are mostly used for desktop robotic arms: they are huge and output actions at low frequency, entirely unsuited to a drone’s low onboard compute and 100 Hz replanning needs. Moreover, most learned methods output discrete waypoints whose numerical derivatives produce violent velocity and acceleration jitter that the flight controller struggles to track.
The core goal of this work: build a lightweight onboard World-Action Model that predicts future obstacle distribution while generating inherently smooth trajectories a controller can use directly, balancing real-time performance, look-ahead perception and tracking stability.
Four Core Contributions
- Onboard lightweight dual-stream World-Action Model FlowPilot: a novel Mixture-of-Transformers (MoT) dual-stream architecture where a video stream predicts future depth frames and an action stream generates flight trajectories, exchanging two-way information between “scene anticipation” and “trajectory generation” through shared attention. On the real drone only the action branch is kept, avoiding depth decoding and cutting compute heavily.
- Flight-controller-native action representation with a 7th-order Bernstein polynomial: instead of jittery discrete waypoints, the trajectory is described by a 7th-order Bernstein polynomial. The first three control points are hard-constrained by the drone’s current position, velocity and acceleration, so the network predicts only five free control points. The trajectory is inherently continuous, with closed-form analytical velocity, acceleration and jerk — no post-smoothing module — and the network output dimension is greatly compressed.
- Three-layer hierarchical data pyramid + two-stage training: the system fuses large-scale simulation, high-fidelity rendering and real-machine depth data, balancing sample volume with real sensor noise. Training proceeds in two steps: first learn environment dynamics, then align trajectory generation, greatly reducing the difficulty of training a world model.
- Complete onboard closed-loop real-world validation: on a Jetson Orin NX embedded board the whole perception-and-planning pipeline runs in under 18 ms; simulation supports up to 8 m/s flight; real indoor and forest scenes reach a peak of 5.5 m/s, with navigation success rates in dense obstacles clearly higher than classical baselines such as EGO-Planner and YOPO.
Full Method Breakdown
Problem Modeling: Reasoning in the Drone Body Frame
All model inputs and outputs are transformed into the drone’s own body coordinate frame, requiring no global map as reference.
Inputs: a monocular depth image, a body-frame state-to-goal vector, a cruise-speed command, and the previous planning cycle’s trajectory state vector containing pose, motion and goal information, for a total of 25 dimensions.
Output: future depth latent variables, and 5×3 Bernstein free-control-point latent variables.
Dual-Stream Mixture-of-Transformers (MoT) Architecture
The network has three main parts — a video expert stream, an action expert stream, and shared cross-modal attention — with two operating modes for training and deployment:
- Video expert stream (environment anticipation branch)
A frozen pretrained Wan2.2 VAE encodes the depth image, embedded with 3D rotary position encoding (RoPE-3D). During flow-matching iterations it predicts the latent-space denoising velocity field, achieving future depth-frame prediction within a 1.6 s horizon. - Action expert stream (trajectory generation branch)
The encoder embeds the state-goal, speed command, history trajectory and noisy Bernstein control-point tokens, and outputs the action-latent denoising velocity field, producing only the five free control points. - Shared cross-modal attention (the core of two-way coupling)
Within each MoT layer, the video and action streams have independent mappings and feed-forward networks but share one attention computation space. The action branch reads the predicted future obstacle distribution and actively steers around obstacles; the video branch uses the trajectory being generated to precisely predict environmental change along the flight path — genuinely “environment anticipation guiding the trajectory, and the trajectory constraining environment prediction.”
Deployment-specific action-centric inference: during training both streams are supervised together; during real flight the depth-decoding path is dropped entirely and only the action branch runs to output control points, removing redundant computation to guarantee real-time performance.
Core Highlight: 7th-Order Bernstein Polynomial Trajectory Representation
The drawback of traditional discrete waypoints: adjacent waypoints have no smoothness constraint, so derivatives produce violent acceleration oscillation and an extra smoothing module is required, adding latency. FlowPilot represents the complete trajectory within the 1.6 s planning window using a three-dimensional 7th-order Bernstein polynomial.
The polynomial has eight control points; the first three are hard-constrained by the drone’s real-time motion state so the trajectory start matches the current motion exactly. The network only needs to predict the remaining five free control points (just 15 dimensions), giving three advantages:
- Analytical differentiation yields velocity, acceleration and jerk directly, with no numerical-differentiation jitter — inherently smooth.
- The trajectory has no jumps and connects seamlessly with the drone’s current motion state.
- The very low output dimension shortens the attention sequence and greatly reduces inference time.
Three-Layer Depth Pyramid Dataset: Balancing Scale and Realism
To solve the sensor domain gap between simulation and the real drone, a hierarchical data system is built: quantity decreases from bottom to top while realism increases.
- Bottom layer (massive simulation): 16 h of IsaacLab high-speed parallel-simulation depth data covering a huge range of obstacles and flight conditions.
- Middle layer (high-fidelity simulation): 8 h of Flightmare photorealistic rendering data adding complex geometry and realistic textures.
- Top layer (real-machine ground truth): 2 h of real depth footage reproducing the native noise, range truncation and dropped frames of a RealSense D455 camera.
Flow-Matching Joint Loss: Enabling Multimodal Avoidance Trajectory Sampling
Linear-interpolation flow matching models the multimodal distribution of trajectories, solving the fatal problem of deterministic regression: multiple detour routes get averaged into a collision path. The two streams use independent noise schedules, and the total loss is a weighted sum of the trajectory loss and the depth-prediction loss: two independent Gaussian noises that let the network optimize both environment prediction and trajectory generation simultaneously.
Two-Stage Hierarchical Training to Reduce Difficulty
Stage 1: World-model pretraining (learn environment anticipation only)
Only the video-stream branch is activated; attention masks block action- and history-trajectory tokens. All three pyramid layers are used to learn the mapping from current depth to future scene, establishing a general environment-dynamics prior.
Stage 2: World-action alignment joint training (simultaneous dual-stream optimization)
All cross-modal attention is opened and both streams train together; trajectory supervision labels come from expert trajectories generated by the high-precision lidar planner SUPER, using only high-fidelity forest simulation samples. A soft history-trajectory conditioning constraint is added: the previous complete Bernstein control points are encoded as tokens into the action stream, and during training there is a 20% probability of randomly masking them with a blank embedding as a soft regularizer. This keeps continuous-replanning trajectories smooth in time without hard-binding to old trajectories and degrading the flight path.
Comprehensive Experimental Validation: Simulation, Embedded and Real-Flight Testing
Experiments span three areas: simulation comparison and ablation, onboard latency measurement, and real drone flight. All data comes from the original paper with no secondary processing.
Simulation Closed-Loop Comparison: Leading Across Dense High-Speed Scenarios
Baselines compared: (1) EGO-Planner (classical optimization planner); (2) YOPO (single-stage learned planner); (3) FlowPilot(action-only), an ablation that removes the video anticipation branch. Test conditions: command speeds 3–8 m/s, two obstacle sparsities (Sparsity=10 dense, Sparsity=15 sparse), with navigation success rate as the metric.
Result: as flight speed rises and obstacle density increases, the success rates of EGO-Planner, YOPO and the action-only variant fall off a cliff, while full FlowPilot keeps the highest success rate throughout thanks to future-environment anticipation, proving that “scene–trajectory two-way modeling” is the key to high-speed avoidance.
Key Ablation: Simultaneous Depth Anticipation Is Indispensable
A control group called “frozen depth stream” fixes the future depth latent throughout denoising iterations, blocking look-ahead environment information from reaching the trajectory branch. The gap is huge: 
freezing depth prediction makes the collision rate surge sixfold while flight efficiency also drops, showing that simultaneously anticipating the future depth is not a side task but a core look-ahead constraint of trajectory generation.
Embedded Onboard Latency Test: Whole Pipeline Under 18 ms
Hardware platform: Jetson Orin NX Super 16 GB, RealSense D455 depth camera, TensorRT quantized acceleration, a 1.6 s planning window and 3-step Euler denoising. The maximum time from perception to trajectory output is only 17.055 ms, perfectly matching a drone’s 100 Hz MPC tracking controller: 
the downstream OMMPC model predictive control averages only 2.384 ms, so the overall closed-loop control period stays stable and controllable.
Real Drone Flight: Indoor Zero-Shot plus Long-Range Outdoor Forest Weaving
Drone hardware: OddityRC 35Pro frame, T-Motor power system, PX4 flight controller, VINS-Fusion visual odometry, with no ground station and no offline prebuilt map throughout.
- Indoor zero-shot generalization: the model was trained only on forest-tree obstacles, yet tested in a brand-new dense artificial block environment — peaking at 3.8 m/s in open areas and holding 3.1 m/s collision-free in extremely narrow areas, proving cross-obstacle-type generalization.
- Outdoor hilly forest long-range flight: terrain height variation up to 4 m, trunk diameters 0.3–0.6 m, obstacle density matching the dense simulation scenario:
- 80 m route: 2.7 m/s in dense forest, 4.8 m/s in open terrain;
- 100 m long-range flight: steady 4 m/s in dense areas and a peak of 5.5 m/s in open areas, collision-free throughout.
Summary and Future Directions
Core Results Recapped
- The dual-stream World-Action Model FlowPilot brings flow-matching world modeling to onboard drone navigation, with shared attention connecting future-environment anticipation and smooth trajectory generation, solving the two pain points of insufficient look-ahead and trajectory jitter.
- The 7th-order Bernstein polynomial as a dedicated action representation naturally outputs a continuous reference trajectory with analytical motion derivatives, perfectly matching flight-controller tracking needs while greatly compressing network output dimension and reducing inference latency.
- A three-layer data pyramid plus two-stage training strategy effectively bridges the sim-to-real sensor gap and lowers the barrier to training World-Action Models.
- Full-chain real-flight closure: the embedded pipeline runs in under 18 ms, indoor and forest scenes reach up to 5.5 m/s agile avoidance, and overall performance surpasses traditional optimization algorithms and end-to-end learning baselines.
Current Limitations and Future Work
- Current limitations: support for monocular depth cameras and short-horizon planning only; insufficient robustness for thin lines, transparent materials and obstacles beyond depth range.
- Future plans:
- Introduce a memory module to extend long-horizon environment prediction;
- Fuse multimodal sensors such as lidar and IMU to improve perception robustness;
- Extend to multi-drone swarm cooperative autonomous navigation.
If you have any questions about this topic, feel free to contact us at [email protected]
FAQ
Q: What problem does FlowPilot solve that EGO-Planner cannot?
EGO-Planner rebuilds a local map and optimizes, so at high speed in dense obstacles the reconstruction lags and errors accumulate. FlowPilot anticipates the future environment directly, so it keeps a high success rate as speed and density rise.
Q: Why use a 7th-order Bernstein polynomial instead of discrete waypoints?
It gives inherently continuous trajectories with closed-form analytical velocity, acceleration and jerk — no numerical-differentiation jitter and no post-smoothing module. It also compresses the network output to just five free control points.
Q: How fast can it run onboard?
The full perception-to-trajectory pipeline takes under 18 ms (max measured 17.055 ms) on a Jetson Orin NX, matching a 100 Hz MPC controller; the downstream OMMPC averages about 2.384 ms.
Q: Can it generalize to obstacles it was never trained on?
Yes. Trained only on forest trees, it flew collision-free in a dense indoor block environment, demonstrating cross-obstacle-type generalization.
Q: Does Aomway build platforms for this kind of agile flight research?
Aomway supplies FPV and UAV hardware, including frames, power systems and video links, and follows advances in onboard autonomy for drones. If you need components for a high-speed autonomous platform, contact Aomway at [email protected].
Have questions about this article? Feel free to contact us at [email protected] — we’re happy to help!
