Pipeline Reference

Align time

Audit

exception pyts2.pipeline.audit.ImageMeanColourException

Thrown if image has an unknown pixel matrix shape (i.e. not greyscale or colour)

class pyts2.pipeline.audit.ImageMeanColourStep

Reports key mean color values (intensity, by RGB and LAB if in color) across a timestream image.

class pyts2.pipeline.audit.ScanQRCodesStep

Finds all QR codes in an image and reports them as a list.

class pyts2.pipeline.audit.CalculateEVStep

Reports exposure value for a given image.

Base

exception pyts2.pipeline.base.FatalPipelineError

Thrown for errors stopping the pipeline for all images (compare with AbortPipelineForThisImage).

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception pyts2.pipeline.base.AbortPipelineForThisImage

Thrown for errors stopping the pipeline for the current image being processed only.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class pyts2.pipeline.base.TSPipeline(*args, reporter=None, record_step_times=False, step_id=None, multi_file_pipeline=False)

Core TimeStream pipeline runner.

Parameters

  • *args (PipelineStep) – Pipeline steps to execute

  • reporter (ResultRecorder) – Object to hold reports

  • record_step_times (bool) – whether to record the time take for each step in the file report

  • step_id (str) – (options) when record_step_times is True, what to record the step name as

  • multi_file_pipeline (bool) – Set True to disable per-file reporting and verification, for functions like gvmosaic

add_step(step)

Appends steps to this pipeline.

Parameters

step (PipelineStep) – Step to add.

Returns

self (adding steps can be chained)

Return type

TSPipeline

process_file(file)

Processes all steps in current object. Mirrors PipelineStep, so an entire pipeline can function as a pipeline step.

process(input_stream, ncpus=1, progress=True)

Processes files.

Parameters

  • input_stream (pyts2.timestream.TimeStream) – Timestream of files to process with pipeline.

  • ncpus (int) – Number of threads to use (default 1)

  • progress (bool) – Unimplemented

finish()

Calls finishing method for each PipelineStep and closes out reporting.

class pyts2.pipeline.base.ResultRecorder

Saves pipeline data to a TSV file. Usually used with ResultRecorderStep, where the file path is specified.

class pyts2.pipeline.base.LiveResultRecorder(fileorpath)

Writes results to file in a streaming log as soon as results are sent to it. (compare with ResultRecorder, which stores results and writes them later in a TSV) Unlike ResultRecorder, records file path is determined on init.

Parameters

fileorpath (io.TextIOWrapper or str) – Writable file handle or path to records file

class pyts2.pipeline.base.PipelineStep

A generic base class for pipeline steps.

All pipeline steps should implement a method called process_file that accepts one argument file, and returns either pyts2.timestream.TimestreamFile or a subclass of it.

class pyts2.pipeline.base.ResultRecorderStep(output_file)

Writes out pipeline results at regular intervals.

Parameters

output_file (str) – Path to output TSV file

class pyts2.pipeline.base.TeeStep(other_pipeline, update_report=False)

Execute another step or pipeline with no side effects on each file

Parameters

other_pipeline (pyts2.timestream.TimestreamFile) – Pipeline to process this file, branching off from this point

class pyts2.pipeline.base.WriteFileStep(output)

Write each file to output, without changing the file

Parameters

output (TimeStream) – Writable Timestream instance

class pyts2.pipeline.base.FileStatsStep

Reports file name and size.

class pyts2.pipeline.base.FilterStep(callback, message='Filter excluded image')

Filter out files from the rest of a pipeline based on a callback function.

Parameters

  • callback (function) – Callback function that accepts file as argument and returns true to continue the pipeline and false otherwise.

  • message (str, optional) – Reason for filtering file (added to exception logging)

Raises

AbortPipelineForThisImage – When file is excluded

class pyts2.pipeline.base.ConditionalStep(conditional_callback, step)

Runs a step only if some conditional callback evaluates for an image.

Useful for e.g. conditionally writing an image:

pipeline.add_step(ConditionalStep(lambda im: im.pixels.sum() > 100, WriteFileStep(output)))

will write only images whose sum of all pixel values is greater than 100.

Parameters

  • conditional_callback (function) – Callback function that accepts file as argument and returns true to continue the pipeline and false otherwise.

  • step (PipelineStep) – PipelineStep to run if conditional_callback evaultates to truthy

class pyts2.pipeline.base.ClearFileObjectStep

A helper to remove memory-consuming members of TSFile/TSImage objects

All but required at the end of pipelines when using parallelisation, to prevent e.g. pixel/file content data being pickeled to send back to the coordinating thread, which would significantly impact performance and hog CPU and memory.

class pyts2.pipeline.base.TimeStreamPathRenameStep(name, add_subsecond_field=False, flat_output=False)

A helper step to allow renaming of TimeStreamFiles to a new timestream path filename

Parameters

  • name (str) – name of timetream to use to rename files

  • add_subsecond_field (bool, optional) – Enable for timestreams with sub-second records, using an additional _[00-99] at the end of filenames

  • flat_output (bool, optional) – Store timestream in a flat file structure, instead of Timestream directory structure

Imageio

class pyts2.pipeline.imageio.DecodeImageFileStep(decode_options=None, process_raws=True, raw_use_embedded_jpeg=False)

Pipeline step to decode image pixels from binary file content, optionally converting raw image formats to a standard numpy array.

Parameters

  • decode_options (list, optional) – Options for processing raw images

  • process_raws (bool) – Process raw image

  • raw_use_embedded_jpeg (bool) – Use the jpg embedded in the raw image, if available

class pyts2.pipeline.imageio.EncodeImageFileStep(format='tiff', encode_options=None)

Pipeline step to encode pixels to a file(‘s bytes).

Parameters

  • format (str) – Image type to encode as (tiff/jpg/png)

  • encode_options (list, optional) – Encoding options passed to PIL

class pyts2.pipeline.imageio.TimestreamImage(instant=None, filename=None, fetcher=None, content=None, report=None, pixels=None, exifdata=None)

Image class for all timestreams

Parameters

  • instant (TSInstant object or derived from file path) – Datetime point that this file represents

  • filename (str) – Name of file, can be retrieved from fetcher

  • fetcher (Fetcher, optional) – File fetcher object to retrieve file content from disk or within archive formats

  • content (File content, optional) – File content (usually if creating timestream file in pipelines)

  • report (dict) – Variable to store reports from Timestream pipeline components

  • pixels (np.array or similar, optional) – Pixels as an array format

  • exifdata (None) – Unimplemented

save(outpath)

Writes file to outpath, in whatever format the extension of outpath suggests.

Parameters

outpath (str) – Path of output file

static from_path(path)

Create timestream file from file path.

Parameters

  • path (str) – Path to file

  • instant (time.TSInstant, optional) – Timepoint associated with object. Attempts to use datetime in path if not provided.

Returns

Timestream file

Return type

TimestreamFile

classmethod from_timestreamfile(file, **kwargs)

Convenience creator for TSFile -> TSImage conversion

Resize

pyts2.pipeline.resize.geom2rowcol(geom)

Converts dimension string into integer tuple.

Parameters

geom (str) – Dimensions as “<height>x<width>”

Returns

Tuple of given dimensions

Return type

2-tuple of ints

Raises

ValueError (when invalid string read)

pyts2.pipeline.resize.ceil2(x)

Round up to next even number.

Parameters

x (numeric) – Number to round up

Returns

Rounded number

Return type

int

class pyts2.pipeline.resize.GenericDownsizerStep(rows=None, cols=None, scale=None, geom=None, interpolation='lanczos')

Handles boilerplate of choosing sizes etc. for ResizeImageStep/CropCentreStep.

Requires either:

  • rows and columns, (use None for one dimension to keep aspect ratio)

  • a scale factor, or

  • a dimension string.

Parameters

  • rows (int) – Image height. Has to be an even number.

  • cols (int) – Image width. Has to be an even number.

  • scale (numeric) – Scale image by this factor

  • geom (str) – Dimension string of “<height>x<width>”

class pyts2.pipeline.resize.ResizeImageStep(rows=None, cols=None, scale=None, geom=None, interpolation='lanczos')

Pipeline step which resizes an entire image to rows * cols

Parameters

  • rows (int) – Image height. Has to be an even number.

  • cols (int) – Image width. Has to be an even number.

  • scale (numeric) – Scale image by this factor

  • geom (str) – Dimension string of “<height>x<width>”

class pyts2.pipeline.resize.ResizeImageStepSKI(rows=None, cols=None, scale=None, geom=None, interpolation='lanczos')

Pipeline step which resizes an entire image to rows * cols uses scikit-image only

Parameters

  • rows (int) – Image height. Has to be an even number.

  • cols (int) – Image width. Has to be an even number.

  • scale (numeric) – Scale image by this factor

  • geom (str) – Dimension string of “<height>x<width>”

class pyts2.pipeline.resize.ResizeImageStepPIL(rows=None, cols=None, scale=None, geom=None, interpolation='lanczos')

Pipeline step which resizes an entire image to rows * cols uses Pillow only

Parameters

  • rows (int) – Image height. Has to be an even number.

  • cols (int) – Image width. Has to be an even number.

  • scale (numeric) – Scale image by this factor

  • geom (str) – Dimension string of “<height>x<width>”

class pyts2.pipeline.resize.CropCentreStep(rows=None, cols=None, scale=None, geom=None, interpolation='lanczos')

Pipeline step which resizes an image to rows * cols

Parameters

  • rows (int) – Image height. Has to be an even number.

  • cols (int) – Image width. Has to be an even number.

  • scale (numeric) – Scale image by this factor

  • geom (str) – Dimension string of “<height>x<width>”

Gigavision