Parameters
Deprecated parameters:
- `--use-netrc`: Autouri defaults to use `~/.netrc`.
- `--http-user` and `--http-password`: Use `~/.netrc` to access private URLs
Change of parameters:
- `--use-gsutil-over-aws-s3` -> `--use-gsutil-for-s3`: Autouri uses `gsutil` CLI only for direct transfer between S3 and GCS buckets. Otherwise, it always use Python libraries like `google-cloud-storage` and `boto3`.
Added parameters:
- `--debug` and `--verbose`: For better logging.
New features
Localization and preventing repetitive file transfer
- When a new localization module makes a copy of source on destination cache directory, it compares md5 hash of source and destination if a file already exists on destination. All bucket URIs (`s3://`, `gs://`) and most of URLs provide md5 hash information in their headers. If md5 hash of those match, Caper skips unnecessary file transfer. For local paths, Caper calculate md5 hash of them and store md5 hash string in `.md5` file since md5 hash calculation is expensive. This happens only when Caper writes on a local storage (i.e. when localizing files on local cache). `.md5` file is not valid if its modification time (mtime) is older than the file itself.
- If md5sum comparison fails, Caper compares file sizes and mtimes instead. If file sizes match and mtime is newer for destination then Caper skips file transfer.
File locking
- Caper uses a stable file locking tested up to multiple threads (50 for local, 10 for cloud URIs) competing to write on the same file.
Automatic subworkflow zipping
- Fixed bugs in old auto-zipping module.
- Caper can automatically zip subworkflow WDLs imported in the main WDL. It can also be manullay defined by users in command line arguments `--imports`. Caper will skip auto-zipping if `--imports` is defined.
- Enabled for `caper submit` only. i.e. `caper run` does not use any automatic subworkflow zipping since it is assumed that all sub-WDLs are already localized for `caper run`.
Womtool validation
- If `--imports` is defined or there is an auto-zipped subworkflow WDLs, then Caper creates a temporary directory and put the main WDL and unpack the zip file there. And then Caper runs Womtool to validate those WDLs.
- You can still skip Womtool validation with `--ignore-womtool`.