checkpoint file always saved to current working directory

[edg1983]

Hi!

I'm currently running cellbender 0.3.2 using your official docker container (us.gcr.io/broad-dsde-methods/cellbender:0.3.2) and a command like this.

sample=$1

cellbender remove-background \
  --input ${sample}/raw_data/raw_feature_bc_matrix.h5 \
  --output ${sample}/cellbender_results.h5 \
  --checkpoint ${sample}/ckpt.tar.gz \
  --cuda --fpr 0.01 --expected-cells 10000

However, I've noticed that the checkpoint file is always saved in the current directory as ckpt.tar.gz, disregarding the --checkpoint option. Apparently, there is no way to prevent cellbender from creating this file.

If I use the checkpoint option like I did in this example, at the end of the training, I get an error like this.

AssertionError: Checkpoint file /project/immune_variation/sample1/ckpt.tar.gz does not exist

This is caused by the program looking for the checkpoint file in ${sample}/ckpt.tar.gz, while it was actually saved as $PWD/ckpt.tar.gz, thus /project/immune_variation/ckpt.tar.gz.

I think this is also related to #403

Is there a way to set where the checkpoint file will be saved?

The fact that it is always saved in the current working directory creates confusion and may lead to clashes when one wants to analyze multiple samples located in different subfolders.

[edg1983]

Adding to the above issue, I see the same behavior also using version 0.3.0. The checkpoint file is always saved as ckpt.tar.gz in the current working directory.