A series in, a series out, step by step.

01The idea

Janus opens a folder of files as a series, one file a date, the date read from each name. These tools deal with what comes before and after that: files that are not yet one a date, dates that are not evenly spaced, and a series that has to leave Janus for another program.

Build series

File → Build series...: loose files grouped by the date in their names, a date's bands stacked and its tiles mosaicked on one grid; a VRT or a COG per date.

Regularize series

Tools → Data preparation → Regularize series...: one composite per period of a fixed step (8 or 16 days, a month, a year, any number of days), the empty periods filled or left empty; a COG per regular date.

Save series as

File → Save series as...: the active series as a multiband COG, a NetCDF or a Zarr with a time axis, over the whole layer, the visible area or the ROI.

Speckle filter

Tools → Data preparation → Speckle filter...: a radar series smoothed by Quegan's multitemporal filter or by Lee's; a COG per date.

Each runs as a task in Tools → Tasks, writes a folder under %LOCALAPPDATA%\Janus\results with its log and its provenance.json, and opens what it made as a new layer.

02The data

The series of the STAC guide: Planetary Computer's Sentinel-2 L2A over a square of central Rondônia, Brazil, 8.8 km a side, the 107 dates from June 2019 to September 2024 whose scenes had 5% of clouds at most, downloaded as GeoTIFFs (blue, green, red, NIR, the two SWIR and the scene classification, SCL), 877 × 886 cells of 10 m. It opens on the NDVI with the clouds masked by SCL.

Two pins follow the guide, those of the STAC guide: pin 1 (195, 837) in a forest in the south-west, its NDVI 0.80 on average over its 101 clear dates and lowest, 0.25, on the last date, 2024-09-11; pin 2 (179, 761) in a pasture, 0.40 on average and down every dry season, 0.13 at its lowest on the same date.

For section 03, 36 loose files were cut from six dates of that series, as a download would give them (section 03 says how). For section 07, the STAC guide's Sentinel-1 year: 24 dates of 2024 over the same square, VV and VH as linear gamma0, kept as one GeoTIFF a date.

03Build a series from loose files

A Sentinel-2 product comes as one file per band, its 10 m bands apart from its 20 m ones, and an area often needs two tiles. The loose files here: six dates of the 2024 dry season, two tiles (T20LMP, the west 5.2 km of the square, and T20LNP, the east 5.2 km, overlapping by 1.6 km), and three bands named as in an L2A product: red and NIR at 10 m, the scene classification at 20 m:

T20LMP_20240608T142721_B04_10m.tif
T20LMP_20240608T142721_B08_10m.tif
T20LMP_20240608T142721_SCL_20m.tif
T20LNP_20240608T142721_B04_10m.tif
... the same for 2024-06-28, 07-18, 08-02, 08-27 and 09-06

File → Build series..., then Pick a folder... (or Pick files...). Janus reads every file's name and georeference and shows a plan before it writes anything:

Dates

From each name, as when a folder is opened (YYYYMMDD, YYYY-MM-DD, YYYY_MM, a year, MODIS's A2001001...). A file without one is left out.

Bands and tiles

The rest of a name, without its runs of six digits or more (the date, the time 142721), tells files apart. Within a date, files of the same extent are one tile, and that tile's files, in natural order, are the bands; files of another extent are other tiles of the same bands, mosaicked. Files of several bands each keep their bands, their tiles mosaicked.

Band names

What the names do not share, cut at a separator: B04_10m, B08_10m, SCL_20m; then a resolution suffix (_10m, _20m) is dropped when the names stay distinct without it: B04, B08, SCL. Sentinel-2's band names (B04, B08...) get their wavelengths, so the indices find red and NIR. Files of one band a date keep what their names share (lc_2000.tif, lc_2001.tif: lc).

Checks

Every file needs a georeference, an unrotated grid and the first file's CRS (a file in another is left out, with the advice to reproject it first). A date whose tiles do not have the same number of files is left out, and so are dates of another kind than most (another number of bands). What is left out is listed under the dates, with why.

Grid

The union of the files' extents at the finest cell: the 20 m scene classification is read on the 10 m grid.

As chooses what is written in results\series\<name>_<time>:

VRTs

One per date, loose_20240608.vrt..., made with GDAL's own gdalbuildvrt code: a mosaic per band on the grid (-te, -tr), then the bands put together (-separate) with their names. Nothing is copied; the six took 0.1 s.

COGs

The same dates read once into cloud-optimized GeoTIFFs, for a series that no longer depends on the loose files.

04Regularize: one date every 16 days

The 107 dates of the Sentinel-2 series are where the sky was clear: a dozen or more in each dry season, few from October to May. Many tools want a fixed step: BFAST and BEAST read the season from it, the phenology fits a curve per year, a neural network wants the same number of dates everywhere. Tools → Data preparation → Regularize series... cuts the time into periods and gives each one value:

Step

8 days, 16 days, monthly, yearly or any number of days. The periods run as pandas' resample does: every N days from the first date's day, or calendar months or years, the empty ones included. Here 16 days give 121 periods (66 without a date), monthly 64 (27 without a date), yearly 6.

A period gives

Median: each band's median of the period's values. Medoid: the observation nearest the bands' medians, so a composite's bands all come from one date (Zeit's regularize_time_series method). Clearest date: the value of the period's date with the most clear pixels, or of the next clearest where that one has none.

Bands

The quantity shown (a band, an index or an expression) with its quality mask applied, or every band of the series, masked.

Empty periods

None leaves them empty. Linear fills them in time between the values around; Whittaker from a Whittaker curve of smoothing λ through the values (the values themselves are kept, and the ends are not extended). Longest gap: a gap is filled only when the values around it are at most that many days apart; 0 fills any.

Area

The whole layer or the ROI.

Run writes a COG per regular date, s2ro_regular_20190605.tif to s2ro_regular_20240906.tif, in results\regularize, and opens the folder as a series. The whole square, 16 days, the median and Whittaker's λ = 10, took 7.5 s on two processing threads for the 121 dates.

The last period, from September 6, holds two dates, so its median for pin 1 is that of 0.50 and 0.25: 0.38, the forest's lowest value in the regular series. A median per period smooths single dates away; the medoid keeps one real observation per period, with all its bands.

05Save as COG, NetCDF or Zarr

A series in Janus is a folder of files, one a date. File → Save series as... writes the active series as one file with its dates in it, for a program that reads data cubes:

COG

One multiband cloud-optimized GeoTIFF, a band per date with the date in the band's description, so it opens back in Janus as a series. One quantity only.

NetCDF

NetCDF 4 written through GDAL's multidimensional API, following the CF conventions: dimensions time, y, x with their coordinates (lat and lon in degrees for a series in geographic coordinates), time in days since 1970-01-01, the CRS as a grid mapping (here transverse_mercator). Float32 with NaN for no data, in chunks of 1 × 512 × 512, compressed with deflate.

Zarr

A Zarr v2 store with the same axes and chunks, compressed with Blosc LZ4 and byte shuffling, as xarray writes it.

Values

The quantity shown (a band, an index, an expression), its quality mask applied; or, for NetCDF and Zarr, every band, a variable each, named after the bands (Red__B04___665_0_nm).

Area

The whole layer, the visible area or the ROI. To reproject it or clip it by polygons, Tools → Data preparation → Reproject, resample, clip first.

The regular NDVI of section 04 (121 dates of 877 × 886) took 6.4 s as a NetCDF and 1.8 s as a Zarr, on two threads; the ROI of the figure, 350 × 236 cells, every band of the 107 dates, 3.4 s as a NetCDF. What is saved opens back as a layer unless Don't open it when saved is ticked.

06Open it in xarray

The NetCDF and the Zarr are what xarray expects: a data variable over time, y and x, the dates as datetimes. The regular NDVI, opened and read at pin 1:

import xarray as xr

ds = xr.open_dataset("s2ro_ndvi_16d.nc")      # or xr.open_zarr("s2ro_ndvi_16d.zarr")
ndvi = ds["value"]
print(dict(ndvi.sizes), ndvi.attrs["grid_mapping"])
pin1 = ndvi.isel(x=195, y=837)                # pin 1: the forest of the south-west
print(pin1.sel(time=slice("2024-05", "2024-09")).round(2).to_series())
print(pin1.groupby("time.year").min().round(2).to_series())

{'time': 121, 'y': 886, 'x': 877} transverse_mercator
time
2024-05-01    0.88
2024-05-17    0.86
2024-06-02    0.84
2024-06-18    0.82
2024-07-04    0.81
2024-07-20    0.84
2024-08-05    0.70
2024-08-21    0.70
2024-09-06    0.38
Name: value, dtype: float32
year
2019    0.78
2020    0.77
2021    0.76
2022    0.56
2023    0.78
2024    0.38
Name: value, dtype: float32

The variable is value because the regular series' band has no name of its own; the ROI's file of section 05 has a variable per band. open_zarr gives the same array, read lazily by dask in its 1 × 512 × 512 chunks. Read whole, both files hold every value and date of the 121 COGs Janus wrote in section 04: the largest difference is 0, and no pixel has a value in one and not in the other.

07Radar: decibels and speckle

Sentinel-1's backscatter comes as linear powers, 0.01 to 0.3 over land, which are hard to look at: a series of Sentinel-1 RTC from the STAC window opens in decibels, the VVdB index (10 * log10(VV)). Any radar series does with an index of the Display panel (VVdB, VHdB, VHVVdB) or --expr VHdB. The figures here show VH, where pasture and forest differ most.

The values are grainy: each cell sums the echoes of many scatterers that add up or cancel at random. This speckle is averaged away over neighbouring cells, or over dates. Tools → Data preparation → Speckle filter...:

Quegan (multitemporal)

Each date's window mean, times the mean over the dates with a value of each date's value divided by its own window mean (Quegan & Yu 2001). The speckle is averaged through time, so each date keeps its own changes and the cells keep more detail than the window alone would leave. Needs two dates or more.

Lee (spatial)

Each date alone: the window's mean plus the cell's departure from it, weighted by how much more the window varies than speckle of L looks would (Lee 1980). Looks is 4.4 for Sentinel-1's IW GRD scenes; fewer smooths less.

Window

3 × 3 to 15 × 15 cells, cut by the series' edges and counting only the cells with a value.

Values

Every band of every date, on the linear values whatever the layer shows (an average of decibels is not the decibels of the average). The result, a COG per date in results\speckle, opens shown as the series filtered was: in dB.

Quegan's filter, 7 × 7, on the 24 dates of both bands took 2.7 s on two threads.

Over the year, the forest of pin 1 varies by 2.38 dB (the standard deviation of its 24 dates of VH) as read, 1.03 dB after Quegan's filter and 1.16 dB after Lee's; the pasture of pin 2 by 3.32, 1.92 and 2.70 dB. The pasture's own change stays: about −15 dB in the first half of the year, −19 to −20 dB through August and September as the grass dries, back near −13 dB in October. Unfiltered, a single date dipped to −24.1 dB.

08Checked against the originals

Each tool was run against what it stands in for, on the same data, comparing the values and the time (tools/bench_series_io.py, tools/bench_regularize_vs_zeit.py, tools/bench_s1_vs_stackstac.py; the numbers in bench/series-io.json, bench/regularize-vs-zeit.json and bench/s1-vs-stackstac.json). They ran on a 20-thread laptop, Janus on 8 threads; the times of this guide's runs above are on 2.

Build and save, against GDAL and xarray

192 loose files: 24 dates × 2 bands × 4 tiles of 1024 × 1024 cells (Int16). Saving: the 24 dates of one band at 2048 × 2048.

Regularize, against Zeit and sits

An irregular series of 64 dates 16 to 20 days apart, a fifth of every date clouded and four dates clouded whole: one band of 1024 × 1024, or three bands of 512 × 512 for the medoid; to 16 days. Zeit's times read the files but write nothing; Janus's include writing the COGs.

Radar, against stackstac and scipy

The Sentinel-1 year of section 07 (24 dates of 877 × 886). Janus read every date whole as VV in dB in 3.9 s against 5.0 s for stackstac with numpy's 10 * log10 on the same items and grid, every one of the 18.6 million values equal, and equal to rasterio on one scene. The speckle filters, 7 × 7 on both bands, against the same formulas through scipy.ndimage.uniform_filter (numpy on one thread): Quegan's 1.7 s with its COGs written against 2.1 s for the arithmetic alone, 1.3 × 10⁻⁷ apart at most; Lee's 1.7 s against 2.3 s, 2.4 × 10⁻⁷. Over the whole year, VV's spread (its standard deviation over its mean) went from 0.69 to 0.55 with Quegan's filter and 0.49 with Lee's.

09What next

• A regular series for the tools that want one: BFAST, BEAST and the phenology read the season from a fixed step.
• Every date for the others: CCDC, S-CCD and EWMACD suit a series of every date, irregular as it is.
• An index before saving: the raster calculator puts any of 275 indices or an expression on the series; Save series as writes the quantity shown.
• The data: the STAC guide finds and downloads the Sentinel-2 and Sentinel-1 series used here.

10References

• GDAL: gdalbuildvrt, the COG driver and the multidimensional raster API, gdal.org.
• NetCDF Climate and Forecast (CF) Metadata Conventions, cfconventions.org; Zarr storage specification version 2, zarr.dev.
• Hoyer, S., & Hamman, J. (2017). xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software, 5(1), 10.
• Flood, N. (2013). Seasonal composite Landsat TM/ETM+ images using the medoid (a measure of multi-dimensional central tendency). Remote Sensing, 5(12), 6481–6500.
• Eilers, P. H. C. (2003). A perfect smoother. Analytical Chemistry, 75(14), 3631–3636.
• Zeit: change detection and time series tools, github.com/sacridini/zeit-cdts.
• Simoes, R., Camara, G., Queiroz, G., Souza, F., Andrade, P. R., Santos, L., Carvalho, A., & Ferreira, K. (2021). Satellite image time series analysis for big Earth observation data. Remote Sensing, 13(13), 2428.
• Quegan, S., & Yu, J. J. (2001). Filtering of multichannel SAR images. IEEE Transactions on Geoscience and Remote Sensing, 39(11), 2373–2379.
• Lee, J.-S. (1980). Digital image enhancement and noise filtering by use of local statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2(2), 165–168.
• Copernicus Sentinel-2 and Sentinel-1 data (ESA), from Microsoft Planetary Computer (Sentinel-2 L2A; Sentinel-1 RTC processed by Catalyst).