Version 1.0.0b4

Version 1.0.0b4

Release date: 2020-07-05. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed

Inference

• Arguments relating to top are renamed to turbulence, and "top" outputs are renamed "turbulence". (#190)
• Change default checkpoint from conditional_mobile-stationary2_effunet6x2-1_lc32_v1.0 to conditional_mobile-stationary2_effunet6x2-1_lc32_v2.0. (#208)
• Status value in EVL outputs extends to final sample (as per specification, not observed EVL files). (#201)
• Rename --nearfield-cutoff argument to --nearfield, add --no-cutoff-at-nearfield argument to control whether the turbulence/bottom line can extend closer to the echosounder that the nearfield line. (#203)
• Improved UI help and verbosity messages. (#187, #188, #203, #204, #207)

Training

• Use 0m as target for surface line for downfacing, not the top of the echogram. (#191)
• Don't include periods where the surface line is below the bottom line in the training loss. (#191)
• Bottom line target during nearfield is now the bottom of the echogram, not 0.5m above the bottom. (#191)
• Normalise training samples separately, based on their own Sv intensity distribution after augmentation. (#192)
• Record echofilter version number in checkpoint file. (#193)
• Change "optimal" depth zoom augmentation, used for validation, to cover a slightly wider depth range past the deepest bottom and shallowest surface line. (#194)
• Don't record fraction of image which is active during training. (#206)

General

• Rename top->turbulence, bot->bottom surf->surface, throughout all code. (#190)
• Convert undefined value -10000.99 to NaN when loading lines from EVL files. (#191)
• Include surface line in transect plots. (#191)
• Move argparser and colour styling into ui subpackage. (#198)
• Move inference command line interface to its own module to increase responsiveness for non-processing actions (--help, --version, --list-checkpoints, --list-colors). (#199)

Fixed

Inference

• Fix depth extent of region boxes. (#186)
• EVL and EVR outputs extend half a timestamp interval so it is clear what is inside their extent. (#200)

Training

• Labels for passive collection times in Minas Passage and Grand Passage datasets are manually set for samples where automatic labeling failed. (#191)
• Interpolate surface depths during passive periods. (#191)
•  Smooth out anomalies in the surface line, and exclude the smoothed version from the training loss. (#191)
•  Use a looser nearfield removal process when removing the nearfield zone from the bottom line targets, so nearfield is removed from all samples where it needs to be. (#191)
• When reshaping samples, don't use higher order interpolation than first for the bottom line with upfacing data, as the boundaries are rectangular (#191)
• The precision criterion's measurement value when there are no predicted positives equals 1 and if there are no true positives and 0 otherwise (previously 0.5 regardless of target). (#195)

Added

Inference

• Add nearfield line to EV file when importing lines, and add --no-nearfield-line argument to disable this. (#203)
• Add arguments to control display of nearfield line, --color-nearfield and --thickness-nearfield. (#203)
• Add -r and -R short-hand arguments for recursive and non-recursive directory search. (#189)
• Add -s short-hand argument for --skip (#189)
• Add two new model checkpoints to list of available checkpoints, conditional_mobile-stationary2_effunet6x2-1_lc32_v1.1 and conditional_mobile-stationary2_effunet6x2-1_lc32_v2.0. (#208)
• Use YAML file to define list of available checkpoints. (#208, #209)
• Default checkpoint is shown with an asterisk in checkpoint list. (#202)

Training

• Add cold/warm restart option, for training a model with initial weights from the output of a previously trained model. (#196)
• Add option to manually specify training and validation partitions. (#205)

Version 1.0.0b3

Release date: 2020-06-25. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed

Inference

• Rename --crop-depth-min argument to --crop-min-depth, and --crop-depth-max argument to --crop-max-depth. (#174)
• Rename --force_unconditioned argument to --force-unconditioned. (#166)
• Default offset of surface line is now 1m. (#168)
• Change default --checkpoint so it is always the same (the conditional model), independent of the --facing argument. (#177)
• Change default --lines-during-passive from "redact" to "predict". (#176)
• Change --sufix-csv behaviour so it should no longer include ".csv" extension, matching how --suffix-file is handled. (#171, #175)
• Change handling of --suffix-var and --sufix-csv to prepend with "-" as a delimiter if none is included in the string, as was already the case for --sufix-file. (#170, #171)
• Include --suffix-var string in region names. (#173)
• Improved UI help and verbosity messages. (#166, #167, #170, #179, #180, #182)
• Increase default verbosity level from 1 to 2. (#179)

Fixed

Inference

• Autocrop with upward facing was running with reflected data as its input, resulting in the data being processed upside down and by the wrong conditional model. (#172).
• Remove duplicate leading byte order mark character from evr file output, which was preventing the file from importing into EchoView. (#178).
• Fix \r\n line endings being mapped to \r\r\n on Windows in evl and evr output files. (#178).
• Show error message when importing the evr file into the ev file fails. (#169).
• Fix duplicated Segments tqdm progress bar. (#180)

Added

Inference

• Add --offset-surface argument, which allows the surface line to be adjusted by a fixed distance. (#168)

Version 1.0.0b2

Release date: 2020-06-18. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed

Inference

• Change default value of --offset to 1m. (#159)
• Use a default --nearfield-cutoff of 1.7m. (#159, #161)
• Show total run time when inference is finished. (#156)
• Only ever report number of skipped regions if there were some which were skipped. (#156)

Fixed

Inference

• When using the "redact" method for --lines-during-passive (the default option), depths were redacted but the timestamps were not, resulting in a temporal offset which accumulated with each passive region. (#155).
• Fix behaviour with --suffix-file, so files are written to the filename with the suffix. (#160).
• Fix type of --offset-top and --offset-bottom arguments from int to float. (#159).
• Documentation for --overwrite-ev-lines argument. (#157).

Added

Inference

• Add ability to specify whether to use recursive search through subdirectory tree, or just files in the specified directory, to both inference.py and ev2csv.py. Add --no-recursive-dir-search argument to enable the non-recursive mode. (#158)
• Add option to cap the top or bottom line (depending on orientation) so it cannot go too close to the echosounder, with --nearfield-cutoff argument. (#159)
• Add option to skip outputting individual evl lines, with --no-top-line, --no-bottom-line, --no-surface-line arguments. (#162).

Version 1.0.0b1

Release date: 2020-06-17. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed

Training

• Built-in line offsets and nearfield line are removed from training targets. (#82).
• Training validation is now against data which is cropped by depth to zoom in on only the "optimal" range of depths (from the shallowest ground truth surface line to the deepest bottom line), using echofilter.data.transforms.OptimalCropDepth. (#83, #109).
• Training augmentation stack. (#79, #83, #106, #124).
• Train using normalisation based on the 10th percentile as the zero point and standard deviation robustly estimated from the interdecile range. (#80).
• Use log-avg-exp for logit_is_passive and logit_is_removed. (#97).
• Exclude data during removed blocks from top and bottom line targets. (#92, #110, #136).
• Seeding of workers and random state during training. (#93, #126).
• Change names of saved checkpoints and log. (#122, #132).
• Save UNet state to checkpoint, not the wrapped model. (#133).
• Change and reduce number of images generated when training. (#95, #98, #99, #101, #108, #112, #114, #127).

Inference

• Change checkpoints available to be used for inference. (#147).
• Change default checkpoint to be dependent on the --facing argument. (#147).
• Default line status of output lines changed from 1 to 3. (#135).
• Default handling of lines during passive data collection changed from implicit "predict" to "redact". (#138).
• By default, output logits are smoothed using a Gaussian with width of 1 pixel (relative to the model's latent output space) before being converted into output probibilities. (#144)
• By default, automatically cropping to zoom in on the depth range of interest if the fraction of the depth which could be removed is at least 35% of the original depth. (#149).
• Change default normalisation behaviour to be based on the current input's distribution of Sv values instead of the statistics used for training. (#80).
• Output surface line as an evl file. (f829cb7)
• Output regions as an evr file. (#141, #142, #143).
• By default, when running on a .ev file, the generated lines and regions are imported into the file. (#152)
• Renamed --csv-suffix argument to --suffix-csv. (#152)
• Improved UI help and verbosity messages. (#81, #129, #137, #145).

General

• Set Sv values outside the range (-1e37, 1e37) to be NaN (previously values lower than -1e6 were set to NaN). (#140).
• Move modules into subpackages. (#104, #130).
• General code tidy up and refactoring. (#85, #88, #89, #94, #96, #146).
• Change code to use the black style. (#86, #87).

Fixed

Training

• Edge-cases when resizing data such as lines crossing; surface lines marked as undefined with value -10000.99. (#90).
• Seeding numpy random state for dataloader workers during training. (#93).
• Resume train schedule when resuming training from existing checkpoint. (#120).
• Setting state for RangerVA when resuming training from existing checkpoint. (#121).
• Running LRFinder after everything else is set up for the model. (#131).

Inference

• Exporting raw data in ev2csv required more EchoView parameters to be disabled, such as the minimum value threshold. (#100).

General

• Fixed behaviour when loading data from CSVs with different number of depth samples and range of depths for different rows in the CSV file. (#102, #103).

Added

Training

• New augmentations: RandomCropDepth, RandomGrid, ElasticGrid, (#83, #105, #124).
• Add outputs and loss terms for auxiliary targets: original top and bottom line, variants of the patches mask. (#91).
• Add option to exclude passive and removed blocks from line targets. (#92).
• Interpolation method option added to Rescale, randomly selected for training. (#79,
• More input scaling options. (#80).
• Add option to specify pooling operation for logit_is_passive and logit_is_removed. (#97).
• Support training on Grand Passage dataset. (#101).
• Support training on multiple datasets. (#111, #113).
• Add stationary2 dataset which contains both MinasPassage and two copies of GrandPassage with different augmentations, and mobile+stationary2 dataset. (#111, #113).
• Add conditional model architecture training wrapper. (#116).
• Add outputs for conditional targets to tensorboard. (#125, #134).
• Add stratified data sampler, which preserves the balance between datasets in each training batch. (#117).
• Training process error catching. (#119).
• Training on multiple GPUs on the same node for a single model. (#123, #133).

Inference

• Add --line-status argument, which controls the status to use in the evl output for the lines. (#135).
• Add multiple methods of how to handle lines during passive data, and argument --lines-during-passive to control which method to use. (#138, #148).
• Add --offset, --offset-top, --offset-bottom arguments, which allows the top and bottom lines to be adjusted by a fixed distance. (#139).
• Write regions to evr file. ([#141](https://github.com/D...

Version 0.1.4

Added

• Add ability to set orientation of echosounder with --facing argument (#77). The orientation is shown to the user if it was automatically detected as upward-facing (#76).

Version 0.1.3

Fixed

• EVL writer needs to output time to nearest 0.1ms. (#72)

Added

• Add --suffix argument to the command line interface of ev2csv. (#71)
• Add --variable-name argument to inference.py (the main command line interface). (#74)

Version 0.1.2

Fixed

• In ev2csv, the files generator needed to be cast as a list to measure the number of files. (#66)
• Echoview is no longer opened during dry-run mode. (#66)
• In parse_files_in_folders (affecting ev2csv), string inputs were not being handled correctly. (#66)
• Relative paths need to be converted to absolute paths before using them in Echoview. (#68, #69)

Added

• Support hiding or minimizing Echoview while the script is running. The default behaviour is now to hide the window if it was created by the script. The same Echoview window is used throughout the the processing. (#67)

Version 0.1.1

Fixed

• Padding in echofilter.modules.pathing.FlexibleConcat2d when only one dim size doesn't match. (#64)

Version 0.1.0

First release of echofilter.