Terraflux Terraflux
Download

Running models

Run, check, test, reproduce, create and convert models from the command line, list the operators and resample a raster.

These are the commands you use every day: run a model, check it first, test it after a change and reproduce a run from its manifest, create a land change model from your maps with new, and a few helpers. The examples use the classic Dinamica EGO example of Sibuyan Island (two land use maps, 1997 and 2011, and thirteen explanatory variables), the outputs shown are real.

terraflux run#

Runs a model and writes its outputs. A model's outputs are its [outputs] (written into the output folder) and any file its nodes write (raster.write, vector.write).

Run a model and write its outputs

Usage: terraflux run [OPTIONS] <MODEL>

Arguments:
  <MODEL>  The model (.toml or .json)

Options:
  -o, --out <OUT>                  The folder of the outputs (default: `output` next to the model)
      --threads <THREADS>          Threads to use (default and most: a quarter of the cores left free; never every core)
      --set <NAME=VALUE>           Replace an input (`name=path` for a file, `name=value` (JSON) for a value) or a node's setting (`node.setting=value`); e.g. `--set seed=7`, `--set patches.isometry=2`
      --scenario <SCENARIO>        Run a scenario of the model (its values, before any --set)
  -q, --quiet                      Print only errors
      --profile                    Print how long every node took and about how much memory what it gave holds, the slowest first
      --tiled                      Run on tiles (maps larger than memory; Dinamica EGO's model): the same maps with any number of threads or machines (the tile size changes the patches a fit measures), statistically equivalent to the exact engine
      --tile <TILE>                The side of the tiles (cells) [default: 1024]
      --radius <RADIUS>            The largest distance of a patch's cell from its seed (cells) [default: 32]
      --disk <DISK>                Keep the steps' layers in this folder instead of memory
      --coordinator <COORDINATOR>  Run the tiles on a coordinator's workers (with --tiled and --disk): the maps and the layers' tiles go to them through the coordinator
      --code <CODE>                The coordinator's pairing code
      --shared-disk                The workers read the model's files and --disk on storage they share with this machine, at the same paths (only sums travel)
  -h, --help                       Print help
> terraflux run model.toml
  simulation: step 13/14 (0.2 s)
  simulation: step 14/14 (0.2 s)
wrote output\maps.tif
wrote output\steps.csv
wrote output\weights.csv
done in 0.2 s

The outputs go to a folder named output next to the model, unless --out names another. Run again, they are overwritten. With them, every run writes manifest.json, the record of the run that terraflux reproduce runs again (it prints wrote output\manifest.json (terraflux reproduce runs it again)).

A model named without its folder

In Terraflux 1.23.0, a model named by its file alone (terraflux run model.toml) stops with cannot make an empty path absolute. Write ./model.toml (or .\model.toml), or the model's folder and name; the next version fixes it.

Changing an input without editing the model#

--set name=value replaces one of the model's [inputs] for this run only, and --set node.setting=value one of a node's settings (container/node.setting for a node inside a loop), as a scenario does. A file input takes a path; a value takes a value written as JSON (a number, true, a list [1, 2], a text in quotes):

terraflux run model.toml --set seed=7 --out seed7          # another seed
terraflux run model.toml --set t0=landuse_2005.tif         # another starting map
terraflux run model.toml --set patches.isometry=2.0        # a node's setting
terraflux run model.toml --set threshold=0.6 --set seed=2  # several at once

This is the simplest way to compare scenarios by hand. For many runs at once, use submit.

Maps larger than memory#

--tiled runs Dinamica EGO's model tile by tile, so a map of billions of cells needs only the memory of a few tiles. With the same tile size and seed, the maps are the same whatever the number of threads or machines; they are statistically equivalent to the exact engine's (the same quantities of change, a similar spatial pattern). The tile size itself changes the maps a little, because the sizes of the patches a fit measures are measured within tiles: keep --tile fixed when you compare runs.

terraflux run model.toml --tiled                            # tiles of 1024 cells, layers in memory
terraflux run model.toml --tiled --tile 2048 --disk layers  # the steps' layers on disk while they are needed
terraflux run model.toml --tiled --disk layers --coordinator http://server:7700   # tiles on many machines

--radius bounds how far a patch grows from its seed (in cells): a patch is grown within one tile and the halo around it. The guide's page on maps larger than memory explains the method.

terraflux check#

Checks a model without running it: the file's format, every operator and setting, every connection (an output must fit the input it feeds), and that every input file exists and can be read.

Check a model and its data without running it

Usage: terraflux check [OPTIONS] <MODEL>

Arguments:
  <MODEL>  

Options:
      --threads <THREADS>  Threads to use (default and most: a quarter of the cores left free; never every core)
  -h, --help               Print help
> terraflux check model.toml
model.toml: no errors

Use it before a long run, after editing a model by hand, and in scripts before a batch.

terraflux test#

Runs models and checks what they give against files of what they should give. A model's tests are [[tests]] in its file, each naming a value (from, a node's output) and the file of what it should be (expected, relative to the model: a raster, a CSV table, a JSON value or a text), within a tolerance (by default 1e-9):

[[tests]]
from = "loss.map"
expected = "expected/loss.tif"

[[tests]]
from = "forest.after"
expected = "expected/forest2011.json"

Test models: run each (a folder: its models) and check what its [[tests]] say against the files of what they should be, and its check.assert nodes; `--update` writes those files from this run instead

Usage: terraflux test [OPTIONS] <MODELS>...

Arguments:
  <MODELS>...  The models, or folders of models

Options:
      --threads <THREADS>  Threads to use (default and most: a quarter of the cores left free; never every core)
      --update             Write the files of what the outputs should be from this run
  -q, --quiet              Print only the failures
      --thaw               Run the frozen nodes too, instead of using their results kept next to the model
  -h, --help               Print help

--update runs the model and writes the expected files from what it gives, once you trust its results; without it, the files are compared. A model passes when it runs to its end (every check.assert that does not only warn passed) and every test holds:

> terraflux test ./model.toml --update
written ./model.toml: 2 test(s), 3 check(s) (0.0 s)
1 model(s) passed

> terraflux test ./model.toml
ok   ./model.toml: 2 test(s), 3 check(s) (0.0 s)
1 model(s) passed

> terraflux test ./model.toml          # after an edit that changed the map
FAIL ./model.toml: loss.map: 243 values differ, the first at 440: 0 instead of 1
Error: 1 of 1 models failed

Give it a folder to test every model in it (its .toml and .json files, not its subfolders'). The command ends with an error when any model fails, so a script can stop on it. A model without [[tests]] is still run and its checks counted. The tutorial Checks, tests and reproducible runs builds a model with its checks and tests step by step.

terraflux reproduce#

Runs again what a run's manifest.json records and tells whether every output is the same, byte for byte. The manifest, written by every terraflux run (and by a whole run of a saved model in the application) beside the outputs, holds the version of Terraflux, the platform, the model file and every input file of [inputs] with their SHA-256 fingerprints, the values of the inputs, the scenario and --set given, every node's setting whose name contains seed, and the SHA-256 of every output of [outputs].

Run again what a run's manifest (manifest.json, written with its outputs) records: the same model, scenario, values and files (told when one changed), and tell whether every output is the same, byte for byte (vectors: the same features)

Usage: terraflux reproduce [OPTIONS] <MANIFEST>

Arguments:
  <MANIFEST>  The manifest.json of the run

Options:
  -o, --out <OUT>          The folder of the outputs (default: `reproduced` next to the manifest)
      --threads <THREADS>  Threads to use (default and most: a quarter of the cores left free; never every core)
  -q, --quiet              Print only what differs
      --thaw               Run the frozen nodes too, instead of using their results kept next to the model
  -h, --help               Print help
> terraflux reproduce output/manifest.json
same    forest2011 (output\reproduced\forest2011.json)
same    loss (output\reproduced\loss.tif)
reproduced: every output the same (0.0 s)

It says first what changed since the run (Terraflux is ... now, ... then, the model ... changed, the input ... changed), runs anyway, and marks each output same or DIFFERS; any difference ends it with an error (1 of 2 outputs differ from the run of the manifest). The outputs go to reproduced, next to the manifest, unless --out names another folder; --quiet prints only what differs. Vectors in GeoPackage are the same when they hold the same features (the file records when it was written). The manifest names the files by their full paths, so it reproduces a run on the computer that made it, or where the study's folders are at the same place; on another computer, use terraflux test, whose paths are relative to the model.

What the next version changes

From the version after 1.23.0, the manifest also holds the files the nodes read (reads, by node.setting) and those they wrote (writes, compared again by terraflux reproduce, but a GeoPackage, which records when it was written), and the frozen results the run used (frozen). Every path is also given from the manifest's folder, so a study's folder moved or copied with its manifest is reproduced where it is now; a file found at neither place is named. terraflux test and terraflux reproduce use the frozen results as terraflux run does, and --thaw runs every node instead.

terraflux new#

Writes a complete land change model from your maps, fitted with the classic methods. The model is written out node by node, each with a note saying why it is there, so you can open it in the application, read it and change any part.

Create a model from a classic land change model

Usage: terraflux new [OPTIONS] <COMMAND>

Commands:
  dinamica  Dinamica EGO's classic model: Markov rates, weights of evidence, Expander and Patcher
  clue      CLUE-S (lulcc's CluesModel): logistic regressions, a linear demand, CLUE allocation
  ordered   The ordered allocation (lulcc's OrderedModel): logistic regressions, a linear demand
  help      Print this message or the help of the given subcommand(s)

Options:
      --threads <THREADS>  Threads to use (default and most: a quarter of the cores left free; never every core)
  -h, --help               Print help

Dinamica EGO's model#

Markov transition rates, weights of evidence of the drivers, and the Expander and Patcher allocation, fitted on two maps of the calibration period:

terraflux new dinamica --t0 landuse.tif:1997 --t1 landuse.tif:2011 \
    --drivers drivers.tif --steps 14 -o model.toml

A map is a file, or a file and a band after a colon: the band's name (landuse.tif:1997) or its 0-based index (landuse.tif:0). --steps is the number of time steps between the two maps (14 years here, one step a year); --simulate how many steps to run (by default the same), from --start (by default the first map).

Dinamica EGO's classic model: Markov rates, weights of evidence, Expander and Patcher

Usage: terraflux new dinamica [OPTIONS] --t0 <T0> --t1 <T1> --drivers <DRIVERS> --steps <STEPS> --output <OUTPUT>

Options:
      --t0 <T0>                      The map at the start of the calibration period: `file.tif` or `file.tif:band` (a name or a 0-based index)
      --threads <THREADS>            Threads to use (default and most: a quarter of the cores left free; never every core)
      --t1 <T1>                      The map at its end
      --drivers <DRIVERS>            The drivers: a raster with one band per variable
      --steps <STEPS>                Time steps in the calibration period
      --simulate <SIMULATE>          Steps to simulate (default: `steps`)
      --start <START>                The map the simulation starts from (default: t0)
      --increment <INCREMENT>        The weights' buffer increment (default: a hundredth of each variable's range)
      --expansion <EXPANSION>        The Expander's share of every transition (default: measured on the observed change)
      --isometry <ISOMETRY>          [default: 1.5]
      --prune-factor <PRUNE_FACTOR>  [default: 10]
      --seed <SEED>                  [default: 1]
  -o, --output <OUTPUT>              The model file to write (.toml or .json)
  -h, --help                         Print help

CLUE-S and the ordered allocation (lulcc)#

Logistic regressions of each class on the drivers, a demand interpolated from the observed maps, and CLUE's iterative allocation, or the ordered allocation:

terraflux new clue --observed observed.tif --times 0,6,14 --drivers drivers.tif \
    --elasticity 0.2,0.2,0.2 -o clue.toml
terraflux new ordered --observed observed.tif --times 0,6,14 --drivers drivers.tif \
    --order 2,1,3 -o ordered.toml

CLUE-S (lulcc's CluesModel): logistic regressions, a linear demand, CLUE allocation

Usage: terraflux new clue [OPTIONS] --observed <OBSERVED> --drivers <DRIVERS> --output <OUTPUT>

Options:
      --observed <OBSERVED>            The observed maps, one band per date (the demand is interpolated from them)
      --threads <THREADS>              Threads to use (default and most: a quarter of the cores left free; never every core)
      --times <TIMES>                  The date of each band of `observed`, e.g. `0,6,14`
      --drivers <DRIVERS>              The drivers: a raster with one band per variable
      --start <START>                  The map the simulation starts from and the regressions are fitted on (default: the first observed band)
      --formulas <FORMULAS>            The drivers of each class's regression as JSON, e.g. `{"1": ["slope"], "2": ["slope", "dist"]}`
      --elasticity <ELASTICITY>        CLUE: the elasticity of each class (one value, or a list in class order), e.g. `0.2,0.2,0.2`
      --order <ORDER>                  Ordered: the order the classes are served in, e.g. `2,1,3`
      --deterministic                  Ordered: take the most suitable cells instead of drawing them
      --settings <SETTINGS>            The allocation's settings as JSON (CLUE: jitter, scale, max_iter, max_diff, mean_diff)
      --allowed <ALLOWED>              The conversion matrix as JSON, in class order, e.g. `[[1,1,0],[0,1,0],[1,1,1]]`
      --neighbourhood <NEIGHBOURHOOD>  Neighbourhood rules as JSON, e.g. `{"rules": {"1": 0.3, "2": 0.2}}`
      --restrictions <RESTRICTIONS>    A map of restricted cells (non-zero keep their class)
      --history <HISTORY>              A map of the steps each cell has had its class
      --seed <SEED>                    [default: 1]
  -o, --output <OUTPUT>                
  -h, --help                           Print help

The settings given as JSON (--formulas, --allowed, --neighbourhood, --settings) are easiest to write inside single quotes on macOS and Linux (--allowed '[[1,1,0],[0,1,0],[1,1,1]]'); in PowerShell, write the inner double quotes twice or keep the JSON in a variable. The tutorials on land change simulation explain every setting.

terraflux methods#

Drafts the methods section of a paper from a model, as File > Methods and citations does in the application: a paragraph naming the methods the model uses with the papers to cite (those of the software Terraflux reproduces, and Terraflux itself), the model step by step, and the references in the APA style. Check the draft and add why each step was chosen.

terraflux methods model.toml                       # in English, on screen
terraflux methods model.toml --language pt         # em português
terraflux methods model.toml -f bibtex -o refs.bib # the references for LaTeX, Zotero, Mendeley
terraflux methods model.toml -f ris -o refs.ris    # for EndNote
-l, --language
en (the default) or pt. In Portuguese the paragraph, the methods' names and the headings are written in Portuguese, and each step is told by its node's note (else its settings): the operators' own descriptions are in English.
-f, --format
text (the default: the paragraph, the steps and the references), bibtex or ris (the references).
-o, --out
A file to write instead of printing.

terraflux plot#

Runs a model and writes one of its Plots (the views kept with the model, see Plots and tables) by its id as a picture of the map, the way Export the view... does in the application: the first map, the maps over it at their opacity, the features with their style and labels, no legend. Each layer is coloured by the palette the Plot names, else the one the Viewer would choose for that output, and the classes in the colours of the model's legend. Hidden layers and differences are left out (told on the screen); a Plot that compares maps gives its map only.

terraflux plot model.toml cover -o cover.png            # a pixel to a cell
terraflux plot model.toml cover -o cover.tif --scale 2  # a GeoTIFF of red, green, blue and alpha, for QGIS
terraflux plot model.toml cover -o cover.png --set seed=7 --scenario dry

Run a model and write one of its Plots (`[[plots]]`, kept with the model) as a picture of its map: the layers at their opacity, the features with their style and labels, as the Viewer draws it (no legend or decorations)

Usage: terraflux plot [OPTIONS] --out <OUT> <MODEL> <ID>

Arguments:
  <MODEL>  
  <ID>     The id of the plot

Options:
  -o, --out <OUT>            The picture: a .png, or a .tif (a GeoTIFF of red, green, blue and alpha on the map's grid, to open in QGIS)
      --threads <THREADS>    Threads to use (default and most: a quarter of the cores left free; never every core)
      --scale <SCALE>        Pixels to a cell of the map: 1 (default), 2 or 4 [default: 1]
      --format <FORMAT>      png or tif (default: by the extension of the file)
      --set <NAME=VALUE>     Replace an input (`name=path` for a file, `name=value` (JSON) for a value) or a node's setting (`node.setting=value`); e.g. `--set seed=7`
      --scenario <SCENARIO>  Run a scenario of the model (its values, before any --set)
  -q, --quiet                Print only errors
  -h, --help                 Print help

--set and --scenario are those of terraflux run.

terraflux convert#

Writes a model in the other format, chosen by the extension: TOML (easy to read and edit) or JSON (easy for programs, and the format landschaft writes from Python).

terraflux convert model.toml model.json
terraflux convert dinamica.json dinamica.toml

terraflux ops#

Lists every operator with what it does; --json gives their full signatures (inputs, outputs, settings, types), the data this website's operator reference is made from.

terraflux ops                      # read on screen
terraflux ops > operators.txt      # into a file
terraflux ops --json > ops.json

terraflux resample#

Puts a raster on the grid of another one (same coordinate reference system): the same cell size, the same cell edges, the same extent. It is what you need before combining maps cell by cell.

Resample a raster to the grid of another (of the same coordinate reference system)

Usage: terraflux resample [OPTIONS] --like <LIKE> --output <OUTPUT> <INPUT>

Arguments:
  <INPUT>  

Options:
      --like <LIKE>        The raster whose grid to take
      --threads <THREADS>  Threads to use (default and most: a quarter of the cores left free; never every core)
  -o, --output <OUTPUT>    
      --method <METHOD>    auto (classes: nearest or mode; continuous: bilinear or average), nearest, mode, bilinear, average [default: auto]
  -h, --help               Print help
terraflux resample drivers.tif --like landuse.tif -o drivers_aligned.tif
terraflux resample slope.tif --like landuse.tif --method average -o slope_100m.tif

With --method auto (the default) Terraflux chooses by the values: classes keep their codes (nearest cell, or the most common one when cells get larger), continuous values are interpolated (bilinear, or averaged when cells get larger). The tutorial One grid for all your maps explains why the choice matters. To also change the coordinate system, use the operator raster.reproject in a model.

terraflux app#

Opens the application, with a model if one is given; the same as double-clicking the program or a model file.

terraflux app
terraflux app model.toml

Suggest a change to this page