Skip to content

Benchmarks & Validation Suite

Welcome to the Zeit Fidelity & Performance Validation Suite. This section provides a comprehensive empirical evaluation of the algorithms implemented in Zeit, assessing both scientific fidelity (how faithfully Zeit reproduces the original tools and published papers it ports) and computational performance (single-core speedup and multi-core CPU scaling via OpenMP).

Executive Summary & Mission

Earth Observation (EO) time-series analysis has historically suffered from fragmented software ecosystems. Canonical algorithms were originally authored across diverse, often proprietary or single-threaded environments:

  • LandTrendr was authored in proprietary IDL (Kennedy et al. 2010).
  • CCDC was authored in MATLAB with compiled Fortran GLMnet (Zhu & Woodcock 2014).
  • BFAST, Phenology, and TWDTW gained widespread adoption through R packages (bfast, phenofit, dtwSat/twdtw).
  • Deep learning models for satellite time-series (TempCNN, LightTAE, U-TAE) were scattered between R sits and standalone PyTorch repositories.
  • Researchers have frequently relied on cloud platforms like Google Earth Engine (GEE), trading off local control, customization, and queue times for scalability.

Zeit was engineered to unify these algorithms into a modern, cloud-native architecture written in high-performance C++ (via Eigen3, SIMD, and OpenMP) and PyTorch (zeit.ai), exposed through Python, Dask, and Xarray.

The Validation Mandate

To verify that Zeit can serve as a drop-in, scientifically reliable replacement for these reference implementations, we subjected Zeit to a rigorous battery of 13 benchmark comparisons (plus one bonus architectural verification). Every algorithm was evaluated against the authentic reference tool across millions of observations, testing:

  1. Numerical and Statistical Fidelity: Do Zeit outputs match the reference tools down to floating-point tolerance, identical breakpoint positions, and matching model categories?
  2. Execution Efficiency: How much faster is Zeit on a single core, how effectively does OpenMP scale across 20 threads, and how does local desktop processing compare against distributed cloud platforms like Google Earth Engine?

Key Performance Indicators

Tested Algorithms
13 + 1
13 canonical tools + 1 bonus segmentation port (U-TAE)
Algorithm Parity
100%
Exact model/vertex/label/codebook match on LandTrendr, CCDC, SNIC, SOM, TWDTW
Max Single-Core Speedup
244×
Phenology vs R phenofit on real 25-yr EVI raster
Vs. Google Earth Engine
20.4×
Compute throughput on full 54.7M px Landsat tile (0 queue delay)

Speed and Fidelity at a Glance

Every port is held to two standards at once: it must give the reference tool's answer, and it must give it faster. The chart lines both up for each algorithm. Hover over (or tab to) a row for the exact numbers; the full matrix follows below.

Single-core speedup over the reference tool, and how closely Zeit reproduces it

Same task, one thread on each side. Log scale: each grid line is ten times the one before. A light extension shows the range over several workloads.

Zeit faster Range across workloads Zeit slower
Identical output Equal within 1e-7 Close: same method, small numerical differences Analog: no reference with the same architecture
Algorithm · reference Fidelity Speedup, one core (log scale) 1× 10× 100× Phenology R phenofit Close 244× LandTrendr IDL original (GDL) Identical 168× CCDC MATLAB original (Octave) Identical 84×–105× TWDTW R twdtw / dtwSat ≤ 1e-7 44× SOM Python minisom Identical 37×–126× Mann-Kendall Python pymannkendall Identical 33×–41× BFAST Monitor R bfast::bfastmonitor Identical 15.8× TempCNN R sits (torch) ≤ 1e-7 11.1× LightTAE R sits (torch) ≤ 1e-7 5.8× BFAST R bfast::bfast Close 4.3× BFAST Lite R bfast::bfastlite Close 2.5× SNIC C reference (snic.c) Identical 1.8×–3.5× U-TAE Official utae-paps Identical 1.08× Siamese CNN Conceptual analog Analog 0.85×

Ten of the fourteen ports return the reference tool's own output, bit for bit or to floating-point rounding, and thirteen are faster on a single core before any parallelism (U-TAE only by 8%). The three "close" ports fit curves or search for breaks with numerical optimisers, where tiny differences in the starting point or in the stopping rule move a date by a few days (Fidelity has the details). U-TAE runs on the same PyTorch backend as its reference, so the two tie; the Siamese detector has no reference with the same architecture.

Comparison Matrix at a Glance

The table below summarizes the scope of our validation suite. Each comparison reflects reproducible benchmarks logged in the Zeit testing repository.

Algorithm Reference Implementation Language / Engine Test Scope Primary Fidelity Metric Single-Core Speedup Status
LandTrendr Kennedy et al. (2010) LandTrendr-2012 Original IDL (via GDL) 330 synthetic series + 54.7M px tile 100% identical vertex years (330/330) 168× compared
CCDC Zhu & Woodcock (2014) GERSL CCDC Original MATLAB (GNU Octave) 200 synthetic + 150 real Landsat px 100% full model match (dates, categories, coeffs) 84×–105× compared
SNIC (Superpixels) Achanta & Süsstrunk (2017) CVPR C reference (snic.c) Reference test fixtures (f32/f64) 100% identical segment labels (bitwise) 1.8×–3.5× compared
SOM Vettigli minisom (online & batch Kohonen SOM) Python (minisom) 1,500 + 20,000 samples, 6 scenarios max weight diff = 0.0 (exact codebook match) 37×–126× tr / 33×–104× pred compared
Phenology (Beck) R phenofit (Zheng et al. 2021) R (nloptr curve fitting) Real EVI raster (638 px × 25 yrs) MAE 3.7d on Start-of-Season (99.1% within 15d) 244× compared
TWDTW Maus et al. (2016) dtwSat / twdtw R (twdtw C core) 45 multi-class temporal series Distances identical to R (1e-12) 43.9× compared
Mann-Kendall pymannkendall (3 variants) Python (pure Python) 270 scenarios (90 series × 3 variants) 100% trend agreement (mean diff p-val = 0.0) 33×–41× compared
BFAST Monitor Verbesselt et al. (2012) bfast R (bfast::bfastmonitor) 40 scenarios (breaks, noise, NaN gaps) 100% break agreement (mag. corr = 1.0) 15.8× compared
TempCNN Pelletier et al. (2019) via R sits R torch (LibTorch backend) 1:1 weight-ported forward passes max abs diff = 5.6e-9 (corr = 1.000000) 11.1× compared
LightTAE (LTAE) Sainte Fare Garnot et al. via R sits R torch (LibTorch backend) 1:1 weight-ported forward passes max abs diff = 8.9e-8 (corr = 1.000000) 5.8× compared
BFAST (Classic) Verbesselt et al. (2010) bfast R (bfast::bfast) 40 scenarios (trend/season breaks) 87.5% break count match (100% pos. match) 4.3× compared
BFAST Lite Jan Hackman et al. bfast R (bfast::bfastlite) 40 scenarios (single/multi breaks) 100% break count match (80% exact index) 2.5× compared
Official U-TAE Garnot & Landrieu (2021) utae-paps Python (official PyTorch repo) Regular and padded sequence paths max abs diff = 0.0 (exact numerical match) 1.08× compared
Siamese CNN Bi-temporal change detection Conceptual analog (R sits DTW) 80 bi-temporal synthetic patches 100% accuracy on respective change tasks 0.85× partial
Foundation ViT ibm-nasa-geospatial/Prithvi-100M HuggingFace transformers HuggingFace Hub remote backbone Wrapper fallback consistent (Conv3d valid) — not_comparable

Status Definitions

  • compared: Exact numerical or statistical benchmark against the official reference tool or canonical implementation.
  • partial: Conceptual analog where no direct equivalent architecture exists in reference libraries (e.g., bi-temporal CNN vs 1D series DTW).
  • not_comparable: Upstream reference package dependency failure preventing direct backbone execution (wrapper fallback path validated).

Hardware & Test Environment

All performance benchmarks were measured on a modern workstation environment:

Benchmark Workstation (Intel Core i5)
  • CPU: Intel Core i5-13600K (14 physical cores: 6 P-cores + 8 E-cores, 20 logical threads)
  • RAM: 64 GB DDR5
  • Operating System: Windows 11 Pro 64-bit
  • R Environment: R 4.4.2 (CRAN `bfast`, `phenofit`, `twdtw`, `torch` 0.17.0, `sits` 1.5.2)
  • Python Environment: Python 3.12.8, PyTorch 2.14.0, Zeit 0.18.0+
  • GDL / Octave: GDL 1.1.2 (native Windows), GNU Octave 11.3 (compiled Fortran GLMnet)

SOM (re-validated in Zeit 0.21.0 after the bit-exact minisom port) was measured on an AMD Ryzen 7 7730U laptop (8 cores / 16 threads, Windows 11, Python 3.10, NumPy 1.26, minisom 2.3.6). Its speedups are ratios measured on that same machine.

Key Scientific & Engineering Insights

1. Parity Demands Meticulous Porting, Not High-Level Mimicry

Empirical validation demonstrates that reproducing published remote sensing algorithms requires rigorous line-by-line fidelity to author reference implementations rather than high-level approximations:

  • LandTrendr: Replicating Kennedy et al. (2010) with 100% vertex year agreement (330/330 series) required reproducing subtle source behaviors: fitting on the despiked series, in-place take_out_weakest2 mutation, whole-ladder Levenberg-Marquardt fallback, flat-line non-significance fallback, and single-precision F-test p-value tie-breaking.
  • CCDC: Matching Zhu & Woodcock (2014) across 350 pixels (599 models, 249 breaks) required a native float32 port of Fortran GLMnet lasso, MATLAB datenum coordinate alignment, and bisquare Tmask outlier detection.
  • SNIC: Achieving bit-for-bit label parity with Achanta & Süsstrunk (2017) required reproducing priority queue heap tie-breaking order, while resolving upstream heap underflow crashes on edge-case inputs.
  • SOM: Reproducing Python minisom codebooks bit-for-bit required matching its RandomState draw sequence, NumPy's pairwise summation inside the distance norm, argmin tie-breaking on square-rooted distances, and sample-order accumulation in the batch update. An earlier Zeit variant that only resembled the batch algorithm (different decay, no learning rate, epoch-based num_iters) agreed with minisom at just ARI = 0.39 and ran 6–10× slower on the same call; the exact port is identical and 37–198× faster.

Across all families, Zeit delivers 100% bitwise and statistical parity with the authoritative reference implementations.

2. High-Performance Local Processing Beats Cloud Queues

The prevailing assumption in remote sensing has been that massive spatial workflows require distributed cloud infrastructure like Google Earth Engine. Our full Landsat tile benchmark demonstrates otherwise:

  • Zeit processing 54,731,482 pixels (41 years) on a single desktop PC completed in 98.44 seconds (556,007 pixels/sec) using OpenMP.
  • Google Earth Engine's distributed cluster executed active server compute on ~38M pixels in 1,396 seconds (27,213 pixels/sec), with an additional 5.08 hours of cloud queue latency.
  • Zeit delivered 20.4× higher compute throughput than GEE's server cluster, with zero queue overhead and zero network egress costs.

LandTrendr on a full Landsat tile: a desktop PC against Google Earth Engine

54.7 million pixels × 41 years on an Intel Core i5-13600K; about 38 million active pixels on Earth Engine's batch workers. Top: throughput while computing. Bottom: the time from submitting the job to having the result.

Zeit, 20 threads Zeit, 1 thread Google Earth Engine Waiting in Earth Engine's queue
Compute throughput, pixels per second 0 200k 400k 600k Google Earth Engine batch workers (EECU) 27k Zeit, 1 thread desktop PC 60k Zeit, 20 threads desktop PC 556k Time to the result, minutes: queue + compute 0 60 120 180 240 300 360 Google Earth Engine batch workers (EECU) queue: 5.1 h 5.47 h Zeit, 1 thread desktop PC 15.1 min Zeit, 20 threads desktop PC 1.6 min

When both sides use every core, the gap stays wide for the statistical tools, whose reference implementations pay for each worker with an interpreter, and closes for the deep-learning models, which run on the same tensor libraries:

Who finishes first with every core

Wall-clock time of Zeit with 20 OpenMP threads and of the reference with its own parallel framework (19 R or Python workers, or intra-op threads for the neural networks). Each row is its own workload; log scale.

Zeit, 20 threads Reference, all workers
Algorithm · workload Wall-clock time, every core (log scale) Zeit finishes 10 ms 100 ms 1 s 10 s 100 s Mann-Kendall 5,000 series 7.3 ms 20.1 s 2,758× sooner BFAST Monitor 20,000 series 14.5 ms 5.0 s 344× sooner Phenology 638 px × 25 years 341 ms 93.5 s 274× sooner TWDTW 3,000 series 13.3 ms 189 ms 14× sooner BFAST 400 series 909 ms 3.3 s 3.7× sooner BFAST Lite 2,000 series 2.0 s 4.6 s 2.3× sooner TempCNN batch of 512 3.9 ms 7.8 ms 2.0× sooner LightTAE batch of 512 14.6 ms 27.0 ms 1.8× sooner U-TAE batch of 32 126 ms 125 ms tied

3. Phenology: Start-of-Season is Robust; Senescence is Ambiguous

Testing Zeit against R phenofit on 25 years of real satellite data (178,710 joined observations across 17 phenometrics) uncovered a vital scientific nuance: - Green-up and Start-of-Season (SOS) metrics agree remarkably well (MAE 3.7 to 5.3 days, with 99.1% of pixels matching within 15 days). - End-of-Season (EOS) and senescence metrics diverge significantly (Dormancy MAE 76.9 days). - This divergence is not an error in either implementation; it reflects fundamental mathematical differences in how asymmetric logistic curves fit noisy, moisture-stressed post-harvest tail observations. Researchers are strongly advised to analyze SOS and EOS metrics separately.

Explore the Deep Dives

To examine the detailed evidence, methodology, and visualizations, navigate to the dedicated benchmark pages:

  • Algorithm Fidelity & Accuracy

    Detailed test batteries, root-cause parity analyses, weight porting verifications, and per-metric accuracy breakdowns for LandTrendr, CCDC, BFAST, Phenology, TWDTW, SOM, and Deep Learning.

    Read Algorithm Fidelity

  • Performance & Parallel Scaling

    Interactive log-scale speedup charts, multi-core CPU scaling tables, the Google Earth Engine Landsat tile benchmark, and absolute execution times.

    Read Performance & Scaling