Key Takeaways
- KCF (Kernelized Correlation Filter, 2014) is the algorithm that made correlation-filter tracking genuinely real-time: it trains on circulant-shifted copies of one target patch and uses a kernel trick, hitting 200–300 FPS on a plain CPU.
- The speed does not come from the classifier. It comes from the FFT — spatial convolution becomes pointwise multiplication in the frequency domain, cutting the work by orders of magnitude.
- KCF’s fatal weakness is a fixed box. It cannot adapt to scale, so it drifts as soon as the target moves toward or away from the camera.
- DSST (also 2014) decouples the two problems: a 2D filter finds the center, a separate 1D filter scores 33 scaled patches to pick the size. Paired with KCF it runs at 50–100 FPS.
- Neither handles occlusion, aspect-ratio change or fast motion. That is why deep trackers took over on accuracy, while KCF and DSST remain the default wherever compute is scarce.
In one line: KCF and DSST are the two milestones of the correlation-filter school of classical object tracking. Both are extremely lightweight, extremely fast, and run in real time on CPU alone — which is why they are still the mainstream choice for embedded, industrial and compute-constrained vision systems. KCF solves how to locate the target quickly and accurately; DSST solves what to do when the target changes size. In practice they are used together. Aomway breaks down how each one works and where each one fails.
What Correlation-Filter Tracking Actually Does
Before KCF, you need the underlying paradigm: discriminative correlation filtering. Think of it as a filter tuned specifically to one target, operating in three steps.
- Training. Given an initial target box, learn a filter template such that sweeping the template over the target region produces a sharp response peak, while sweeping it over background produces as low a response as possible. At heart this is a binary classification problem: target versus background.
- Detection. On the next frame, slide the filter across every position and compute the response. The coordinates of the response peak are the target’s new position.
- Why it is fast. Sliding-window correlation is convolution. By the properties of the Fourier transform, convolution in the spatial domain equals pointwise multiplication in the frequency domain, so moving the computation into the frequency domain with an FFT reduces the workload by several orders of magnitude. That — not the classifier — is the real reason correlation filters reach hundreds of FPS.
Early correlation filters had two obvious pain points: far too few training samples (only the initial box counted as a positive), and a linear classifier too weak to fit real target appearance. Accuracy was poor. KCF fixed both problems at once.
KCF: Kernelized Correlation Filter
Published in 2014, KCF is the watershed paper of the correlation-filter field. It was the first to bring circulant samples and the kernel trick into the correlation-filter framework, pushing accuracy and speed up at the same time.
Circulant samples: one image, hundreds of training patches
The traditional way to improve a classifier is to crop many positive and negative samples, which is computationally expensive. KCF’s trick is to apply cyclic shifts to the initial target patch — translating it up, down, left and right, with the parts that shift out wrapping around to the other side — generating hundreds of virtual positive and negative samples automatically. Small shifts produce positives close to the target; large shifts produce negatives close to background.
More importantly, samples generated this way form a circulant matrix in a mathematical sense, and a circulant matrix can be diagonalized directly by FFT. You never actually generate hundreds of images, yet you get the training benefit of a huge sample set at almost no extra cost.
Kernel trick: from linear to non-linear classification
Early correlation filters were all linear classifiers, able to learn only simple edge and grayscale features, and unable to separate a target from a complex background. KCF introduces a Gaussian kernel that maps low-dimensional features into a high-dimensional space, letting the classifier fit far more complex target appearance. Because the kernel matrix inherits the circulant structure, its computation is still FFT-accelerated — speed is not sacrificed.
HOG features instead of raw grayscale
The original KCF uses HOG (histogram of oriented gradients) features in place of grayscale pixels. HOG is more robust to illumination change and slight deformation, describes edges and contours better, and further stabilizes tracking.
What KCF is good at
- Raw speed. 200–300 FPS on CPU alone — top tier among all tracking algorithms.
- Lightweight, no dependencies. Compact code, no deep learning framework, implementable in bare C++.
- Stable on rigid targets. For rigid objects under steady illumination without severe occlusion, accuracy far exceeds contemporaneous mean-shift and Kalman-filter approaches.
Where KCF breaks
- Fixed scale. The filter size never changes, so the tracking box cannot zoom with the target. When the target grows or shrinks, the box drifts and the track dies. This is KCF’s biggest shortcoming.
- No occlusion handling. When the target is occluded, the model keeps updating on background as if it were target, and eventually loses the track entirely.
- Fast motion. If the target moves faster than the filter’s search range, it cannot be captured.
DSST: Discriminative Scale Space Tracking
DSST was also published in 2014. It is not a complete new tracker but a solution aimed squarely at scale adaptation in correlation filtering, and it was the first to propose decoupling translation estimation from scale estimation. It is normally paired with KCF: KCF handles position, DSST handles size.
Two filters, two jobs
The design logic is intuitive: position and size are independent problems, and solving them separately works better and runs faster than solving them jointly.
- Translation filter. Predicts the target’s center coordinates. Identical in logic to KCF — a fixed-size 2D filter, correlate, find the response peak.
- Scale filter. Predicts the scaling ratio, and is a one-dimensional correlation filter whose only job is to decide whether the target got bigger or smaller.
How the scale estimate is computed
- Around the current target center, crop 33 image patches at different sizes on a proportional scaling grid (for example, evenly distributed from 0.5× to 2×).
- Extract HOG features from each patch and build a one-dimensional scale feature sequence.
- Correlate the pre-trained scale filter against that 1D sequence. The scale with the highest response is the target’s true current size.
- The same circulant-sample and frequency-domain acceleration applies, so scale estimation does not slow the pipeline down.
Once the new scale is known, the translation filter’s size is updated to match, and the tracker moves to the next frame.
What DSST is good at
- It fixes the fixed-scale problem. Accuracy improves dramatically when the target moves closer or farther away and its size changes.
- The decoupled design is efficient. Translation and scale are computed separately, which is easy to maintain and easy to optimize in engineering terms.
- Still fast. Tens to roughly a hundred FPS on CPU alone, enough for most real-time scenarios.
Where DSST stops
- Proportional scaling only. It cannot adapt to aspect-ratio change — a pedestrian turning from front view to side view, or a vehicle turning — because the box ratio stays fixed.
- It does not solve the core problems. Occlusion, fast motion and out-of-view re-detection remain, because DSST is fundamentally a supplement to the KCF framework.
- It costs speed. The extra scale computation makes it slower than plain KCF.
Engineering note: DSST is almost never deployed alone. The standard production setup is “KCF translation filter plus DSST scale filter” — the so-called scale-adaptive KCF — which balances accuracy and speed. The later fDSST variant pushed speed further with feature dimensionality reduction.
KCF vs DSST: Side-by-Side
| Dimension | KCF | DSST (used with KCF) |
|---|---|---|
| Core contribution | Introduced circulant samples and the kernel trick; established the high-speed correlation-filter framework | Proposed translation/scale decoupling; added scale adaptation |
| Scale adaptation | None — fixed size | Supports proportional scale change |
| Aspect-ratio adaptation | Fixed ratio | Fixed ratio; width and height cannot be adjusted independently |
| Real-time speed on CPU | 200–300 FPS | 50–100 FPS |
| Typical use case | Close-range tracking of fixed-size targets; simple rigid-body scenes | General tracking where target distance and size change dynamically |
| Role in a system | Base tracking kernel | Functional enhancement module for KCF |
Shared Foundations and Shared Limits
Both belong to the discriminative correlation-filter paradigm, and their core logic is nearly identical.
- Both train the filter by ridge regression, minimizing prediction error.
- Both rely on cyclic shifts to generate dense training samples and work around the small-sample problem.
- Both accelerate inference in the frequency domain via FFT.
- Both update online, fine-tuning the filter frame by frame to follow slow appearance change.
Their shared limits are also what let deep learning tracking become mainstream:
- Hand-crafted features cap accuracy. HOG is far less robust than learned features under complex deformation, abrupt illumination change and similar-background clutter.
- No occlusion awareness. Model updates are indiscriminate, so an occluded target teaches the filter background, causing drift — and once lost, the track cannot recover on its own.
- Single-target only. An initial box must be given manually; there is no detection capability.
- Low fault tolerance. Fast motion, leaving the field of view and heavy occlusion are all poorly handled.
Where These Algorithms Still Win in Production
Deep trackers (the Siam family, Transformer-based trackers) now lead on accuracy across the board, yet KCF and DSST retain irreplaceable engineering value.
- Severe compute constraints. Low-end ARM chips, industrial MCUs, legacy security hardware, GPU-less embedded boards — anywhere a deep model will not run, KCF and DSST are the first choice.
- Detect-and-track pipelines for cost reduction. Pair them with a detector such as YOLO: run detection every few frames and fill the gap with KCF or DSST. This cuts compute cost substantially and raises overall frame rate — the standard approach to multi-object tracking at the edge.
- Simple, stable scenes. Workpiece tracking on an industrial line, fixed-viewpoint small-target surveillance, indoor scenes with steady lighting: KCF and DSST are good enough, with very low maintenance cost.
Summary
KCF is the cornerstone of correlation-filter tracking, using circulant samples and the kernel trick to win on both speed and accuracy. DSST is its most important functional complement, using a decoupled design to close the scale-adaptation gap. Together they form the standard recipe for lightweight classical tracking.
From an engineering standpoint: if compute is tight, the target is mostly rigid and its size barely changes, plain KCF is enough. If the target moves toward and away from the camera, use KCF plus DSST. If the scene is complex, with frequent occlusion and deformation, it is time to move up to a deep tracker.
Have questions about this article? Feel free to contact us at [email protected] — we’re happy to help!
Frequently Asked Questions
Why is KCF so fast on a CPU?
Because the expensive part happens in the frequency domain. Sliding-window correlation is a convolution, and convolution in the spatial domain becomes pointwise multiplication after an FFT. Combined with circulant samples that are diagonalized analytically rather than generated, KCF reaches 200–300 FPS without a GPU.
What problem does the DSST scale filter actually solve?
It removes KCF’s fixed-box limitation. A separate one-dimensional filter scores 33 proportionally scaled patches of the target, and the highest response selects the new size. Translation stays with KCF, so the two problems are estimated independently instead of jointly.
Can KCF handle occlusion?
No. KCF updates its model every frame with no occlusion check, so an occluded target teaches the filter background appearance, which causes drift and permanent track loss. Adding a confidence threshold or a re-detection step is the usual mitigation, but it is a patch, not a fix.
Why combine KCF with a detector like YOLO instead of tracking alone?
Tracking alone has no way to recover after a loss and needs a manual initial box. Running detection every few frames and tracking in between gives you both automatic initialization and cheap interpolation, cutting compute dramatically while keeping frame rate high — the standard edge-deployment pattern.
When should I switch from KCF or DSST to a deep tracker?
Switch when occlusion is frequent, target appearance deforms heavily, aspect ratio changes matter, or the background is cluttered with similar objects. Stay with KCF and DSST when compute is fixed and small, the target is rigid, and lighting is controlled.
Source: original Chinese technical article on KCF and DSST tracking algorithms (WeChat). Original: mp.weixin.qq.com.
About Aomway
Aomway designs and manufactures FPV video transmitters, antennas, cameras and goggle systems for drone pilots worldwide, with more than fifteen years in the UAV industry. Its hardware serves racing, long-range and professional aerial platforms where low-latency video and reliable RF links matter most.

