Changelogs » Caper

Caper

0.4.0

1) Deprecated old flags `--use-docker` and `--use-singularity`.
`--docker` and `--singularity` can work as flags too.
bash
$ caper run atac.wdl --docker   this will use docker image defined in WDL
$ caper run atac.wdl --docker docker_image_location


2) `caper init`
Choose a platform and caper will auto-generate default configuration file (`~/.caper/default.conf`) with only required parameters for a chosen platform.
bash
$ caper init sherlock


3) Others
- A file DB is now compressed.
- Better error checking and troubleshooting.

0.3.20

* Bug fixes
- Deepcopied files still have unlocalized files (JSON/TSV/CSV) again.

0.3.19

* Bug fixes
- Deepcopied files still have unlocalized files (JSON/TSV/CSV).

0.3.18

* Bug fixes
- Localization fails if a JSON, TSV or CSV file is not updated

0.3.17

> **IMPORTANT**: `--deepcopy` has been deprecated. Deepcopy is enabled by default and can be disabled with `--no-deepcopy`. `--use-singularity` and `--use-docker` are deprecated. Instead, `--singularity` and `--docker` can work as flags.

* Deprecated parameters
- `--deepcopy`: Deepcopy is enable by default. `--no-deepcopy` is added to disable it.
- `--use-docker`: `--docker` can work as a flag
- `--use-singularity`: `--singularity` can work as a flag

* New parameters
- `--no-deepcopy`: to disable deepcopy
- `--dry-run`: Caper creates all temporary files but does not take any action.

* New subcommand
- `debug`: an alias of `troubleshoot`, which is too long to type.

* Bug fixes
- Fixed all deepcopy-related bugs

0.3.16

* change of default parameters
- `db-timeout`: 10000 to 30000 (ms)
- `java-heap-run`: 2G to 2G
- `java-heap-server`: 8G to 7G

* added a subcommand `debug` , which is identical to `troubleshoot`.

0.3.15

* bug fix for `db` key error when `--no-file-db` or `-n` is given but `--file-db` or `-d` is defined
* update doc for `sbatch` command
- added `-export=ALL` to `sbatch` command to respect all environment variables (e.g. Conda activated `PATH`)

0.3.14

* added `--db-timeout` to give longer time for Cromwell to connect to DB.
- to prevent `db - Connection is not available` error even when no Caper/Cromwell process is attached to the DB file
- this error can happen when DB file is large
- default: `10000` (ms)

0.3.13

* upgrade google API from `v1` to `v2`: to fix PAPI 10 error
* added `--gcp-zones` to specify GCP zones to provision worker instances
- e.g. `--gcp-zones us-central1-b,us-west1-b`

0.3.12

* Bug fix for v0.3.11 (typo `::` in the code)

0.3.11

**BUGGY. DO NOT USE** USE v0.3.12 INSTEAD

* to support Croo's "soft-linking" on cloud storages
- see [this](https://github.com/ENCODE-DCC/croo/releases/tag/v0.1.6) for details.

0.3.10

* bug fix for `caper list` (non-hashable `list` error)
* update doc for ENCODE workshop 2019 at Seattle
* `caper metadata` prints out Python dict object `{ }` instead of list of dict `[ { } ]` objects when there is only one workflow matched by search condition. Otherwise It keeps printing out lists of objects.

0.3.9

* use input JSON file's basename for string label (`-s` / `--str-label`) if that  is not specified in the command line
* added `-v` / `--version` to show version

0.3.8

* added short parameters `-d` for `--file-db` and `-n` for `--no-file-db`
* fixed bug for `--use-netrc` not working with `submit` command
* added `Minimum required parameters` section to default conf file so that users only need to edit small number of parameters in that section. for TL;DR users.
* `cromwell.jar` will be downloaded on `~/.caper/cromwell_jar/` instead of a directory specified with `--tmp-dir`

0.3.7

* Default `time` runtime attribute for cluster engines (SGE/SLURM/PBS)
- default = 24 hours

0.3.6

* Added parameters
- `--java-heap-server`: Java heap size `java -Xmx` for `caper server` subcommand.
- `--java-heap-run`: Java heap size `java -Xmx` for `caper run` subcommand.

* Bug fixes
- `--docker` not correctly caught when it's already defined in WDL
- added warning when `--docker` is not defined for cloud backends

0.3.5

* Bug fix for `docker`
- Cromwell 42 defaults to use `delegated` flag for docker in mounting volumes but this flag was introduced in docker ce 17.04. Caper will not use this flag for backward compatibility (for docker ce < 17.04).

0.3.4

* final testing on AWS (since isolation issue for aliased tasks is fixed in cromwell-42)
- chip-seq-pipeline worked fine
* documentation for setting up cromwell server on AWS
* fixed bug in `file_exists()` for AWS
- `aws s3 ls` does not show exactly matched files. it also shows partially matched files.

0.3.3

* bug fix for locked copy (deadlock for URL to cloud URIs transfer)
* suppress gsutil/aws warnings (`no URLs matched one of the ...`)

0.3.2

* added `user` to `caper list` format
* users can filter workflows by submission datetime `--hide-result-before` ( or `hide-result-before` in your conf at `~/.caper/default.conf`)
- you can use the same (or shorter in terms of string length) datatime format shown in `caper list`.
- this is just about simple datetime string sorting

For example,

(base) leepc12kadru:/users/leepc12/code/test_wdl$ caper list
id      status  name    str_label       user    submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c    On Hold None    T5      leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b    On Hold None    T4      leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea    On Hold None    T3      leepc12 2019-06-13T11:12:16.349Z
aa08ca19-0ac8-4ec6-995d-53bcfaaa1038    On Hold None    T2      leepc12 2019-06-13T11:12:10.611Z
a6573ee9-5e08-432e-bf7f-5733069b7510    On Hold None    T1      leepc12 2019-06-13T11:11:51.057Z
(base) leepc12kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-13T11:12:16
id      status  name    str_label       user    submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c    On Hold None    T5      leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b    On Hold None    T4      leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea    On Hold None    T3      leepc12 2019-06-13T11:12:16.349Z
(base) leepc12kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-13
id      status  name    str_label       user    submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c    On Hold None    T5      leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b    On Hold None    T4      leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea    On Hold None    T3      leepc12 2019-06-13T11:12:16.349Z
aa08ca19-0ac8-4ec6-995d-53bcfaaa1038    On Hold None    T2      leepc12 2019-06-13T11:12:10.611Z
a6573ee9-5e08-432e-bf7f-5733069b7510    On Hold None    T1      leepc12 2019-06-13T11:11:51.057Z
(base) leepc12kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-14
id      status  name    str_label       user    submission

0.3.1

* Fix for SLURM Stanford Sherlock job failing issue even though Job is actually done successfully. `squeue` doesn't seem to work correctly.
- Job check-alive polling (`squeue -j {job_id}`) somtimes fail when server is busy
- Try `squeue` for every 20 second up to 3 times.
- increase interval for polling 180 sec -> 360 sec

0.3.0

* thread safe inter-storage file transfer
- added a simple spin lock (with `.lock` file) to obtain thread safety for file transfer
- multiple users can share the same `tmp-dir`, `tmp-s3-bucket` and `tmp-gs-bucket`.
- limitation
- possible race condition
- skip copying only if file size of source and target files are same (does not check md5sum)
- source files are not locked to prevent deadlock
- make sure that source files are almost like read-only

0.2.8

* Added `--max-retries` (default:1) to retry failing tasks
* Replaced deprecated `actor factory` for google backend
* Cromwell server submits jobs slowly (1 per 2 second) due to possible socket error for SLURM/other cluster engines.
* Bug fix for `aws` backend (typo in code: out_aws_bucket->out_s3_bucket)
* Upgrade cromwell 40 to 42 (to fix aliased [task isolation issue](https://github.com/broadinstitute/cromwell/issues/4364) on aws)

0.2.7

* Bug fix for singularity in slurm backend (task hanging issue)

0.2.5

* Show more information (even successfully completed tasks if `--show-completed-task` is given) for `caper troubleshoot`
- JOB_ID: PID or JOB_ID (SLURM or SGE) of JOB
- RUN_START: start datetime if task is run
- RUN_END: end datetime if task is run

0.2.4

* added `--use-netrc` to use `~/.netrc` file to handle HTTP username/password more securely

0.2.3.4

* Bug fix for `--file-db` for Caper server when it's defined in cmd line args

0.2.3.3

* Bug fix for Cromwell server IP address. (`None` -> `localhost` by default)
* Update README

0.2.3.2

* Default `singularity_cachedir` at `~/.caper/singularity_cachedir`
* Updated README

0.2.3

* Caper defaults to use a file DB. You can still use MySQL DB.