ToF Camera

Time-of-Flight Depth Camera

Join the Competition Contribute to Benchmark

Standard reconstruction benchmark — forward model perfectly known, no calibration needed. Score = 0.5 × clip((PSNR−15)/30, 0, 1) + 0.5 × SSIM

# Method Score PSNR (dB) SSIM Source
🥇 MPI-Former 0.782 34.0 0.930 ✓ Certified Multi-path interference correction, 2023
🥈 DeepToF 0.742 32.5 0.900 ✓ Certified Marco et al., ECCV 2018
🥉 PnP-ToF 0.617 28.0 0.800 ✓ Certified PnP with depth prior for ToF
4 Phase Unwrap 0.480 24.0 0.660 ✓ Certified Bamji et al., IEEE SSC 2015

Dataset: PWM Benchmark (4 algorithms)

Blind Reconstruction Challenge — forward model has unknown mismatch, must calibrate from data. Score = 0.4 × PSNR_norm + 0.4 × SSIM + 0.2 × (1 − ‖y − Ĥx̂‖/‖y‖)

# Method Overall Score Public
PSNR / SSIM
 Dev
PSNR / SSIM
 Hidden
PSNR / SSIM
 Trust Source
🥇 MPI-Former + gradient 0.677
0.768
31.5 dB / 0.937
0.685
26.89 dB / 0.855
0.578
22.36 dB / 0.705
✓ Certified Multi-path interference correction, 2023
🥈 DeepToF + gradient 0.664
0.749
30.66 dB / 0.926
0.643
24.84 dB / 0.797
0.599
23.17 dB / 0.738
✓ Certified Marco et al., ECCV 2018
🥉 PnP-ToF + gradient 0.614
0.690
26.46 dB / 0.844
0.587
22.71 dB / 0.720
0.564
21.85 dB / 0.684
✓ Certified PnP with depth prior for ToF
4 Phase Unwrap + gradient 0.517
0.595
22.42 dB / 0.708
0.508
19.73 dB / 0.586
0.448
17.92 dB / 0.496
✓ Certified Bamji et al., IEEE SSC 2015

Complete score requires all 3 tiers (Public + Dev + Hidden).

Join the competition →
Scoring: 0.4 × PSNR_norm + 0.4 × SSIM + 0.2 × (1 − ‖y − Ĥx̂‖/‖y‖) PSNR 40% · SSIM 40% · Consistency 20%
Public 5 scenes

Full-access development tier with all data visible.

What you get & how to use

What you get: Measurements (y), ideal forward operator (H), spec ranges, ground truth (x_true), and true mismatch spec.

How to use: Load HDF5 → compare reconstruction vs x_true → check consistency → iterate.

What to submit: Reconstructed signals (x_hat) and corrected spec as HDF5.

Public Leaderboard
# Method Score PSNR SSIM
1 MPI-Former + gradient 0.768 31.5 0.937
2 DeepToF + gradient 0.749 30.66 0.926
3 PnP-ToF + gradient 0.690 26.46 0.844
4 Phase Unwrap + gradient 0.595 22.42 0.708
Spec Ranges (3 parameters)
Parameter Min Max Unit
modulation_freq 19.9 20.2 MHz
multipath -5.0 10.0 %
phase_nonlinearity -2.0 4.0 deg
View Details →
Dev 5 scenes

Blind evaluation tier — no ground truth available.

What you get & how to use

What you get: Measurements (y), ideal forward operator (H), and spec ranges only.

How to use: Apply your pipeline from the Public tier. Use consistency as self-check.

What to submit: Reconstructed signals and corrected spec. Scored server-side.

Dev Leaderboard
# Method Score PSNR SSIM
1 MPI-Former + gradient 0.685 26.89 0.855
2 DeepToF + gradient 0.643 24.84 0.797
3 PnP-ToF + gradient 0.587 22.71 0.72
4 Phase Unwrap + gradient 0.508 19.73 0.586
Spec Ranges (3 parameters)
Parameter Min Max Unit
modulation_freq 19.88 20.18 MHz
multipath -6.0 9.0 %
phase_nonlinearity -2.4 3.6 deg
Submit Reconstruction View Details →
Hidden 5 scenes

Fully blind server-side evaluation — no data download.

What you get & how to use

What you get: No data downloadable. Algorithm runs server-side on hidden measurements.

How to use: Package algorithm as Docker container / Python script. Submit via link.

What to submit: Containerized algorithm accepting y + H, outputting x_hat + corrected spec.

Hidden Leaderboard
# Method Score PSNR SSIM
1 DeepToF + gradient 0.599 23.17 0.738
2 MPI-Former + gradient 0.578 22.36 0.705
3 PnP-ToF + gradient 0.564 21.85 0.684
4 Phase Unwrap + gradient 0.448 17.92 0.496
Spec Ranges (3 parameters)
Parameter Min Max Unit
modulation_freq 19.93 20.23 MHz
multipath -3.5 11.5 %
phase_nonlinearity -1.4 4.6 deg
Submit Algorithm View Details →

Blind Reconstruction Challenge

Challenge

Given measurements with unknown mismatch and spec ranges (not exact params), reconstruct the original signal. A method must be evaluated on all three tiers for a complete score. Scored on a composite metric: 0.4 × PSNR_norm + 0.4 × SSIM + 0.2 × (1 − ‖y − Ĥx̂‖/‖y‖).

Input

Measurements y, ideal forward model H, spec ranges

Output

Reconstructed signal x̂

About the Imaging Modality

ToF cameras measure per-pixel depth by emitting modulated near-infrared light and measuring the phase delay of the reflected signal relative to the emitted signal. In amplitude-modulated continuous-wave (AMCW) ToF, the phase offset phi = 2*pi*f*2d/c encodes the round-trip distance 2d. Multiple modulation frequencies resolve depth ambiguity. Primary degradations include multi-path interference (MPI), motion blur, and systematic errors at depth discontinuities (flying pixels).

Principle

A Time-of-Flight depth camera measures the round-trip time of modulated light (typically near-infrared LEDs at 850 nm) reflected from the scene. The sensor measures the phase shift between emitted and received modulated signals at each pixel, which is proportional to the target distance: d = c·Δφ/(4π·f_mod). Typical modulation frequencies are 20-100 MHz, providing depth ranges of 0.5-10 meters with mm-cm precision.

How to Build the System

Use an integrated ToF camera module (e.g., Microsoft Azure Kinect DK, PMD CamBoard pico, Texas Instruments OPT8241). The module contains the NIR light source, modulation driver, and ToF sensor with per-pixel demodulation circuits. Mount rigidly and calibrate intrinsic parameters (lens distortion, depth offset) and phase-to-depth nonlinearities. For multi-camera setups, synchronize or frequency-multiplex to avoid interference.

Common Reconstruction Algorithms

  • Four-phase demodulation for distance extraction
  • Multi-frequency unwrapping for extended unambiguous range
  • Flying-pixel filtering (mixed pixels at depth discontinuities)
  • Multi-path interference correction
  • Deep-learning depth denoising and completion

Common Mistakes

  • Multi-path interference causing systematic depth errors in concave scenes
  • Flying pixels at depth edges producing incorrect intermediate depth values
  • Phase wrapping ambiguity when objects exceed the unambiguous range
  • Interference from ambient NIR light (sunlight) degrading outdoor performance
  • Systematic depth errors from non-ideal sensor response not calibrated out

How to Avoid Mistakes

  • Use multi-path correction algorithms or multi-frequency modulation
  • Apply flying-pixel detection and removal based on amplitude and neighbor consistency
  • Use dual-frequency operation to extend the unambiguous range
  • Use narrow-band optical filter and higher modulation power for outdoor use
  • Perform per-pixel depth calibration with a known flat reference at multiple distances

Forward-Model Mismatch Cases

  • The widefield fallback produces a 2D intensity image, but ToF cameras measure depth via phase shift of modulated near-infrared light — the distance information (d = c*dphi/(4*pi*f_mod)) is entirely absent from the blurred image
  • ToF measurement involves demodulation of the reflected modulated signal at each pixel, producing amplitude, phase, and confidence maps — the widefield intensity-only blur cannot produce depth or distinguish multi-path interference

How to Correct the Mismatch

  • Use the ToF camera operator that models modulated illumination and per-pixel demodulation: four-phase sampling extracts the phase shift proportional to target distance at each pixel
  • Apply phase-to-depth conversion, multi-path correction, and flying-pixel filtering using the correct modulation frequency, amplitude, and phase measurement model

Experimental Setup — Signal Chain

Experimental setup diagram for Time-of-Flight Depth Camera

Experimental Setup

Instrument: Intel RealSense L515 / Microsoft Azure Kinect DK
Depth Resolution: 640x480
Range M: 0.1-6.0
Frame Rate Fps: 30
Wavelength Nm: 850
Depth Accuracy Mm: 2.0
Modulation: AMCW (amplitude-modulated continuous wave)

Key References

  • Hansard et al., 'Time-of-Flight Cameras: Principles, Methods and Applications', Springer (2013)

Canonical Datasets

  • NYU Depth V2 (Silberman et al.)
  • KITTI depth benchmark (adapted)

Spec DAG — Forward Model Pipeline

P(modulated) → Σ(correlation) → D(g, η₁)

P Modulated Light (modulated)
Σ Correlation Integration (correlation)
D ToF Sensor (g, η₁)

Mismatch Parameters

Symbol Parameter Description Nominal Perturbed
Δf_m modulation_freq Modulation frequency error (MHz) 20 20.1
I_mp multipath Multipath interference intensity (%) 0 5.0
Δφ phase_nonlinearity Phase nonlinearity (deg) 0 2.0

Credits System

40%
Platform Profit Pool
Revenue allocated to benchmark rewards
30%
Winner Share
Top algorithm receives from pool
$100
Min Withdrawal
Minimum payout threshold
Spec Primitives Reference (11 primitives)
P Propagation

Free-space or medium propagation kernel (Fresnel, Rayleigh-Sommerfeld).

M Mask / Modulation

Spatial or spatio-temporal amplitude modulation (coded aperture, SLM pattern).

Π Projection

Geometric projection operator (Radon transform, fan-beam, cone-beam).

F Fourier Sampling

Sampling in the Fourier / k-space domain (MRI, ptychography).

C Convolution

Shift-invariant convolution with a point-spread function (PSF).

Σ Summation / Integration

Summation along a physical dimension (spectral, temporal, angular).

D Detector

Sensor readout with gain g and noise model η (Gaussian, Poisson, mixed).

S Structured Illumination

Patterned illumination (block, Hadamard, random) applied to the scene.

W Wavelength Dispersion

Spectral dispersion element (prism, grating) with shift α and aperture a.

R Rotation / Motion

Sample or gantry rotation (CT, electron tomography).

Λ Wavelength Selection

Spectral filter or monochromator selecting a wavelength band.