TSTK CLI¶

Usage: tstk [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  audit
  bundle
  cp
  downsize
  findpairs   Finds pairs of XXXXXX.{jpg,cr2} or similar with identical...
  gvmosaic
  imgscan
  ingest
  liveingest
  ls
  verify      Verify images from each of EPHEMERAL, ensuring images are in...
  version
  video       Creates a standard timelapse video from timestream

Help is also available for individual commands, e.g. tstk ingest --help

TSTK tools operate on TimeStreams, a format for the storage of time series file data. Typically, timestreams are series of image files (tif, cr2, or jpg typically) from a camera, captures over the life of some field station or lab experiment.

Common Options¶

The following are common options for the various available commands.

--help Show this message and exit
-F, --informat TEXT Input image format (use extension as lower case for raw formats)
-f, --outformat [jpg|png|tif] Output image format
-o, --output TEXT: Output TimeStream
-j, --ncpus INTEGER Number of parallel workers
-b, --bundle [root|year|month|day|hour|none] Level at which to bundle files
--flat: Output all images to a single directory (flat timestream structure)
-x, --rm-script FILE Write a script which deletes input files to FILE
--mv-destination DIR Instead of deleting input files, move files to DIR. (see –rm-script)
-y, --yes Delete/move files without asking

Options to restrict input files to a specific date-time range:

--start-date YYYY-MM-DD
--start-time hhhh
--end-date YYYY-MM-DD
--end-time hhhh

TSTK audit¶

tstk audit scans a timestream, calculating various stats on each image and reporting them as a TSV.

Provides the following:

time
file name and file size
RGB mean values
LAB color space mean values
QR code text (if found)

Usage: tstk audit [OPTIONS] INPUT

Example

tstk audit \
  --informat cr2 \
  --ncpus "${PBS_NCPUS:-1}" \
  --output "/g/data/xe2/datamanagement-workspace/audit/data/picam/GC37_cr2_2020_05_19.tsv \
  "/g/data/xe2/phenomics/camupload/picam/GC37"

bundle¶

Zip images of a type (specifed by --informat) together by time period (specified by --bundle). Valid time periods are year, month, day or hour. Specifying --bundle root zips the entire camera as one file. Specifying --bundle none simply copies files. (? Behaviour TBC. File names standardised?)

Input can also be zipped files (e.g. in resources dir). Specifying --bundle none on zipped input produces unzipped output.

Usage: tstk bundle [OPTIONS] INPUT OUTPUT

Example

tstk bundle \
  --informat cr2 \
  --bundle day \
  /g/data/xe2/camupload/picam/GC37/2020 \
  /g/data/xe2/dir_for_output_zip

cp¶

Copy a subset of files within a range of specified timepoints. Optionally specify an interval to skip files, e.g. only include files taken on the hour. (To do this, specify the interval 60 and provide a start time which is on the hour). Useful for creating a timestream for a specific trial.

Usage: tstk cp [OPTIONS] INPUT OUTPUT

Example

tstk cp \
  --informat cr2 \
  --bundle day \
  --start-date 2020-01-15 \
  --start-time 0500 \
  --end-date 2020-03-09 \
  --end-time 1700 \
  "/g/data/xe2/phenomics/resources/cameras/picam/GC37" \
  "/g/data/xe2/some_dir/TR00678_cr2.zip"

downsize¶

Usage: tstk downsize [OPTIONS] INPUT

Options:

-m, --mode [resize|centrecrop] Either resize whole image to --size, or crop out central --size pixels at original resolution.
-s, --size TEXT: Output size. Use ROWSxCOLS. One of ROWS or COLS can be omitted to keep aspect ratio.

Example

tstk downsize \
  --size 720x \
  --informat cr2 \
  --outformat jpg \
  --ncpus "${PBS_NCPUS:-1}" \
  --output "output_dir/TR00678_GC37L_jpg_720x.zip" \
  "some_input_path/GC37L"

findpairs¶

Finds pairs of XXXXXX.{jpg,cr2} or similar with identical file name and metadata. Use for:

removing extra copies that are lying around
identifying extraneous jpgs for removal (because we have captured a cr2 file and we are not keeping the jpg)

Examples

tstk findpairs --yes

gvmosaic¶

Usage: tstk gvmosaic [OPTIONS] INPUT

Options:

-d, --dims TEXT Dimension of super-image, in units of sub-images, ROWSxCOLS [required]
-O, --order [colsright|colsleft|rowsdown|rowsup] Order in which images are taken (cols or rows, left or right)
--truncate-time TIME Truncate time to TIME
-s, --composite-size TEXT Size of each sub-image in a composite, ROWSxCOLS
-f, --composite-format TEXT File format of composite output images
-o, --composite-output TEXT Output timestream for composite images
-b, --composite-bundling [root|year|month|day|hour|none] Level at which to bundle composite image output
-S, --composite-centrecrop PROPORTION Crop centre of each image. takes centre PROPORTION h x w from each image
--bo, --bundle-output TEXT Output timestream for verbatim import images
--bb, --bundle-level [root|year|month|day|hour|none] Level at which to bundle verbatim images
--ro, --recoded-output TEXT Output timestream for recoded import images
--rf, --recoded-format TEXT File format of images
--rb, --recoded-bundling [root|year|month|day|hour|none] Level at which to bundle recoded images

Examples

TODO

imgscan¶

Usage: tstk imgscan [OPTIONS] [INPUT]...

Options:

-t, --timestreamify-script FILENAME Write script to sort images to FILE.
-d, --timestreamify-destination TEXT -t script moves files to DIR

Examples

TODO

ingest¶

Source files from an input directory (usually camupload) and perform a standard conversion process. Put the resulting (zipped) images in an output dir (usually resouces).

Ingest use cases¶

tstk line for ingest script with variables for file names.

tstk ingest \
  --informat "${FORMAT}" \
  --bundle day \
  --ncpus ${PBS_NCPUS:-1} \
  --downsized-output "${DOWNSIZED_CAM}" \
  --downsized-bundle none \
  --downsized-size 720x \
  --audit-output "${AUDIT_FILE}" \
  --output "${RESOURCES_CAM}" \
  "${CAMUPLOAD_CAM}" \
  > "${LOG_DIR}/${PBS_JOBNAME}_${(date +%Y_%m_%d_%H_%M_%S)}_${PBS_JOBID}.log" 2>&1

verify¶

Verify images from one or more ‘ephemeral’ directories, ensuring images are in dir specified with –resources flag.

Use this after tstk findpairs or tstk ingest, to verify that the files processed and moved are correct, before deleting the original files in the ‘ephemeral’ dir.

tstk verify  \
  --informat cr2 \
  --resource "/g/data/xe2/phenomics/resources/cameras/picam/GC37" \
  --move-dest "/scratch/xe2/phenomics/ingest/trash/picam/GC37" \
  --rm-script "/scratch/xe2/phenomics/REINGEST/rmscripts/"${CAMNAME}" \
  "${REINGEST_CAM}" "${TOSORT_CAM}"

Verify use cases¶

Usage: tstk verify [OPTIONS] [EPHEMERALS]...

Verify images from each of EPHEMERAL, ensuring images are in –resources.

Options:

-r, --resource PATH Archival bundled TimeStream [required]
-p, --pixel-distance FLOAT Fuzzily match images based on distance in pixel units. Formula is abs(X - Y)/maxpixelval/npixel, i.e. 0 for no distance and 1 for all white vs all black.
--distance-file PATH Write log of each ephemeral file’s distance to tsv file
--only-check-exists Only check that a file at the same timepoint exists.

Examples

TODO

version¶

Usage: tstk version

Gets the version number of this build of tstk

Tips & Gotchas¶

Use standardised, lower-case file extensions (cr2, jpg, tif), irrespective of the actual extensions of the source files.
tstk automatically recognises and converts files named with common variants, e.g. *.JPG, *.jpeg, *.tiff