JitTrack: Peking University Onboard MOT Framework Against Viewpoint Jitter for Agile UAVs (2026)

✨ Key Insight: High-speed agile drone flight control maneuvers easily trigger violent frame jitter, causing large target offsets between consecutive frames, frequent ID switches, and track loss—a major industry pain point! A Peking University team introduces JitTrack, a motion-aware Transformer tracking framework. Three innovative modules natively adapt to drone 3-axis camera motion, and the team built a complete “perception-planning-flight-control” closed-loop onboard system. Validated through both offline datasets and real drone flights, it dramatically improves target identity locking under jitter scenarios. At Aomway, we closely follow such research because robust onboard tracking is critical for autonomous aerial systems.

• Authors: Yachun Shan, Feitian Zhang
• Affiliation: School of Advanced Manufacturing and Robotics, Peking University
• Paper Title: JitTrack: Onboard Multi-Object Tracking Against Viewpoint Jitter for Agile UAVs
• Paper Link: https://arxiv.org/abs/2608.10485

Research Background: The Critical Pain Point of Drone Tracking

Aerial tracking is fundamentally different from ordinary surveillance!

Multi-object tracking (MOT) is core technology for airspace security and autonomous follow-shooting, but fixed-camera algorithms suffer “culture shock” on drones: during agile flight, body attitude changes rapidly—pitch, yaw, and roll cause violent frame jitter, producing large target displacements and appearance mutations between consecutive frames, directly triggering ID switches and target loss. Combined with limited onboard compute, real-time and lightweight requirements are stringent.

As the data shows: during violent camera jitter intervals (frames 68–78, 165–181), ID switch counts surge simultaneously—viewpoint shaking severely disrupts inter-frame target association.

Both existing technical routes have critical flaws

1. Traditional camera motion compensation (CMC) approaches: require a separate motion estimation module, adding computation overhead; under irregular extreme drone maneuvers, motion estimation easily fails, with poor robustness.
2. Query-based Transformer trackers (TrackFormer/MOTR, etc.): these algorithms heavily rely on spatio-temporal consistency of inter-frame query features. Once the frame jitters violently, tracking queries passed from the previous frame become spatially misaligned with current targets, and matching performance drops off a cliff.

Engineering deployment gap: vision only, no full-aircraft closed loop

Most existing drone tracking algorithms only conduct offline video evaluation, rarely connecting the full chain of visual perception, trajectory planning, and flight control execution—making direct deployment for autonomous active follow-shooting impossible.

Traditional training strategies detached from real aerial scenes

Generic denoising training only adds random Gaussian noise to target boxes, without simulating the frame offsets caused by real drone 3-axis motion. Training samples mismatch real jitter scenarios, yielding weak generalization.

Three Core Innovations

1. New framework JitTrack: a pioneering motion-aware Query-based multi-object tracking architecture. No extra external camera motion compensation module needed—natively adapts to drone viewpoint jitter at the Transformer query layer, greatly improving target identity retention under complex maneuvers.
2. Three complementary optimization modules:

• Semantic Query Refinement (SQR): balances semantic weights between new-target detection queries and historical tracking queries, reducing missed detection of new targets;
• Motion-Aware Query Rectification (MAR): models global camera motion offsets, adaptively correcting misaligned tracking queries;
• Motion-Informed Denoising Training (MIDT): simulates drone pitch/yaw/roll perturbations to generate training noise, strengthening jitter robustness.

1. Complete onboard closed-loop engineering system: fuses RGB vision + LiDAR multi-sensor, connecting the full chain of perception, trajectory planning, and PX4 flight control; offline comparison on VisDrone2019-MOT and UAVDT (two authoritative aerial datasets) plus real quadrotor flight trials validate practical deployment value.

Core Research Method Breakdown

Semantic Query Refinement (SQR)

1. Existing problem

Freshly initialized detection queries have weak semantic features; historical tracking queries dominate self-attention computation, suppressing newly appearing targets in the frame and causing significant missed detections.

2. Implementation steps + formulas

1. Image feature injection into queries: fuse raw detection queries with current frame image encoded features via cross-attention, then pass through self-attention to obtain semantically enhanced queries.
2. Similarity dilution to eliminate redundancy: compute similarity weights between semantic queries and historical tracking queries.
3. Corrected query weights: suppress duplicate responses to existing targets while retaining sensitivity to new target recognition; dilution strength is a hyperparameter.

Motion-Aware Query Rectification (MAR)

1. Core pain point

Global camera motion causes synchronized offsets of all targets in the frame; tracking query coordinates passed across frames misalign with true targets, breaking matching and causing ID chaos.

Core hypothesis: the global background motion induced by camera rotation has low-dimensional properties and can be uniformly modeled as a global motion bias.

2. Implementation steps + formulas

1. Extract spatial motion priors in heatmap form from the image encoder;
2. Combine tracking queries with motion priors, estimate a unified global motion bias via multi-head attention;
3. Use the motion bias to correct tracking query positional embeddings (coordinates before/after correction respectively);
4. Add a nuclear norm loss constraining the motion bias to remain low-rank, ensuring whole-image motion consistency.

Motion-Informed Denoising Training (MIDT)

Drone frame jitter decomposes into three fundamental transformations—pitch, yaw, and roll. This work abandons random noise and strictly generates denoising queries from ground-truth boxes perturbed according to real aerial motion patterns.

1. Yaw perturbation (horizontal translation):
2. Pitch perturbation (vertical translation): horizontal/vertical offset coefficients; random perturbation ratio.
3. Roll perturbation (rotation around frame center): random rotation matrix; corresponding image width/height.

Training mask constraint prevents information leakage

During training, the total query set (denoising queries / detection queries / tracking queries) uses attention masks to block detection and tracking queries from reading ground-truth information carried by denoising queries, preventing training cheating.

Total loss function: three-part joint optimization

The overall loss combines jitter drift loss, motion regularization loss, and denoising reconstruction loss:

1. Jitter-aware drift loss: up-weights samples with large inter-frame displacement, raising training weight for jitter scenarios;
2. Motion low-rank regularization loss;
3. Denoising reconstruction L2 loss: constrains perturbed boxes to recover ground truth;
4. Final training objective.

Real-machine deployment: perception-planning-flight-control closed-loop pipeline

1. Perception layer: JitTrack visual tracking + MID360 LiDAR point cloud clustering; visual detection boxes matched with radar clusters, Kalman filter outputs stable 3D target coordinates; hardware runs on Jetson Orin NX 16GB onboard computer with real-time computation.
2. Planning layer: Fast-Planner trajectory generator combines target motion and environmental constraints to output collision-free flight trajectories.
3. Localization & flight control layer: FAST-LIO provides drone ego-odometry; trajectory commands sent to PX4 flight controller for fully automatic target following.

Comprehensive Experimental Validation

Basic experimental setup

• Datasets: two industry-standard aerial MOT datasets—VisDrone2019-MOT, UAVDT
• Metrics: MOTA, MOTP, IDF1, ID Switch (lower is better)
• Training config: ResNet50 backbone, Transformer encoder/decoder 6 layers 8 heads each; 4× RTX4090 GPUs, 100 epochs, Adam optimizer with initial learning rate

Horizontal PK with mainstream SOTA algorithms

1. VisDrone2019-MOT: JitTrack achieves 38.4% MOTA, 53.1% IDF1—up 15.6% MOTA and 11.7% IDF1 over the original Transformer baseline MOTR; leading all end-to-end Query-based trackers by a wide margin with sharply reduced ID switches.
2. UAVDT: achieves 46.7% MOTA, 68.1% IDF1, comprehensively surpassing native Transformer tracking solutions like TrackFormer and MOTR, and matching traditional detection+association algorithms.

Highlight: IDF1 improves most significantly—proving the algorithm retains target identity far better under violent frame jitter.

Module ablation study (VisDrone)

Baseline: original MOTR (no optimization, IDF1=41.4%, MOTA=22.8%, ID switches=959)

1. MIDT motion denoising only: IDF1=45.7%, MOTA=28.4% — conclusion: noise matching real drone motion effectively improves jitter robustness
2. MIDT + SQR semantic refinement: IDF1=48.3%, MOTA=36.0% — conclusion: SQR solves missed detection caused by historical queries suppressing new targets
3. MIDT + SQR + MAR (full JitTrack): IDF1=53.1%, MOTA=38.4%, ID switches down to 907 — conclusion: MAR corrects query positions, fundamentally reducing jitter-induced ID switches; all three modules are indispensable.

Denoising strategy comparison: random noise vs. drone motion noise

• Pure baseline: MOTA=22.8%, target recall 28.5%
• Baseline + traditional random Gaussian denoising (DTS): MOTA=25.2%, recall 36.0%
• Baseline + MIDT 3-axis motion perturbation: MOTA=28.4%, recall 41.4%

The comparison shows: perturbation methods simulating real aerial jitter improve target recall more effectively than purely random noise.

Real flight testing

Scenario: drone continuously locks target ID0, with repeated pedestrian occlusion and extreme drone turns creating high-intensity frame jitter.

1. Baseline original algorithm: all 10 repeated tests failed—frequent ID switches, target loss, 0% task success rate;
2. Full JitTrack solution: 9 of 10 experiments completed autonomous follow-flight; target ID0 maintained zero ID switches throughout successful tests.

Direct proof: JitTrack can be deployed directly in onboard real-time systems, perfectly adapting to real aerial complex scenarios like occlusion and aggressive maneuvers.

Summary and Future Outlook

Core conclusions

1. Precisely targets the frame jitter pain point of agile UAV high-speed maneuvers, proposing JitTrack—a motion-aware Query-based tracking framework that needs no complex external camera motion compensation module, adapting to global frame offsets at the query representation level;
2. SQR semantic refinement, MAR motion correction, MIDT motion denoising—three modules work synergistically, significantly improving ID retention on two public aerial benchmarks and sharply suppressing jitter-induced identity switches;
3. Opens the complete “vision-LiDAR fusion-trajectory planning-PX4 flight control” onboard closed-loop pipeline, validated for real-machine deployment—enabling long-duration autonomous continuous target follow-shooting.

Future work directions

1. Develop lightweight motion-aware tracking versions for small, compute-constrained micro drones;
2. Extend tracking optimization to multi-drone cooperative aerial shooting and complex multi-target interaction scenarios.

If you have any questions about this topic, feel free to contact us at [email protected]

Have questions about this article? Feel free to contact us at [email protected] — we’re happy to help!

Frequently Asked Questions

Q1: What is JitTrack and what problem does it solve?
JitTrack is a motion-aware Transformer multi-object tracking framework from Peking University for agile UAVs. It solves the violent viewpoint jitter problem—where camera shake causes large frame-to-frame target offsets, ID switches, and track loss—by natively adapting tracking queries to drone 3-axis camera motion without needing an external camera motion compensation module.

Q2: What are the three core modules of JitTrack?
SQR (Semantic Query Refinement) balances detection and tracking query weights to reduce missed detection of new targets; MAR (Motion-Aware Query Rectification) models global camera motion to correct misaligned tracking queries; MIDT (Motion-Informed Denoising Training) simulates real pitch/yaw/roll perturbations for training to strengthen jitter robustness.

Q3: How well does JitTrack perform on public benchmarks?
On VisDrone2019-MOT it achieves 38.4% MOTA and 53.1% IDF1 (up 15.6%/11.7% over the MOTR baseline); on UAVDT it achieves 46.7% MOTA and 68.1% IDF1, surpassing TrackFormer and MOTR. In real flight tests, JitTrack completed 9 of 10 autonomous follow-flight trials with zero ID switches, versus 0% success for the baseline.

Q4: Does JitTrack require special hardware?
It runs in real time on a Jetson Orin NX 16GB onboard computer. The system fuses RGB vision with MID360 LiDAR point cloud clustering, uses Fast-Planner for trajectory generation, FAST-LIO for ego-odometry, and PX4 flight control—a complete perception-planning-control closed loop.

Q5: Why is IDF1 especially important for drone tracking?
IDF1 measures identity consistency over time. In jittery drone footage, maintaining a target’s identity across violent camera motion is the hardest challenge—ID switches mean the tracker “forgets” who it’s following. JitTrack’s largest gains are in IDF1, proving it keeps targets locked under exactly the conditions that break conventional trackers.