Datalad

Fixes

- Our internal command runner failed to capture output in some cases.
([3656][])
- Workaround in the tests around python in cPython >= 3.7.5 ';' in
the filename confusing mimetypes ([3769][]) ([3770][])

Enhancements and new features

- Prepared for upstream changes in git-annex, including support for
the latest git-annex
- 7.20190912 auto-upgrades v5 repositories to v7. ([3648][]) ([3682][])
- 7.20191009 fixed treatment of (larger/smaller)than in .gitattributes ([3765][])

- The `cfg_text2git` procedure, as well the `--text-no-annex` option
of [create][], now configure .gitattributes so that empty files are
stored in git rather than annex. ([3667][])


0.11.7 (Sep 06, 2019) -- python2-we-still-love-you-but-...

Primarily bugfixes with some optimizations and refactorings.

Fixes

- [addurls][]
- now provides better handling when the URL file isn't in the
expected format. ([3579][])
- always considered a relative file for the URL file argument as
relative to the current working directory, which goes against the
convention used by other commands of taking relative paths as
relative to the dataset argument. ([3582][])

- [run-procedure][]
- hard coded "python" when formatting the command for non-executable
procedures ending with ".py". `sys.executable` is now used.
([3624][])
- failed if arguments needed more complicated quoting than simply
surrounding the value with double quotes. This has been resolved
for systems that support `shlex.quote`, but note that on Windows
values are left unquoted. ([3626][])

- [siblings][] now displays an informative error message if a local
path is given to `--url` but `--name` isn't specified. ([3555][])

- [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't
support all the parameters that Git expects it to. ([3616][])

- Fixed a number of Unicode py2-compatibility issues. ([3597][])

- [download-url][] now will create leading directories of the output path
if they do not exist ([3646][])

Enhancements and new features

- The [annotate-paths][] helper now caches subdatasets it has seen to
avoid unnecessary calls. ([3570][])

- A repeated configuration query has been dropped from the handling of
`--proc-pre` and `--proc-post`. ([3576][])

- Calls to `git annex find` now use `--in=.` instead of the alias
`--in=here` to take advantage of an optimization that git-annex (as
of the current release, 7.20190730) applies only to the
former. ([3574][])

- [addurls][] now suggests close matches when the URL or file format
contains an unknown field. ([3594][])

- Shared logic used in the setup.py files of DataLad and its
extensions has been moved to modules in the _datalad_build_support/
directory. ([3600][])

- Get ready for upcoming git-annex dropping support for direct mode
([3631][])

Primarily bug fixes to achieve more robust performance

Fixes

- Our tests needed various adjustments to keep up with upstream
changes in Travis and Git. ([3479][]) ([3492][]) ([3493][])

- `AnnexRepo.is_special_annex_remote` was too selective in what it
considered to be a special remote. ([3499][])

- We now provide information about unexpected output when git-annex is
called with `--json`. ([3516][])

- Exception logging in the `__del__` method of `GitRepo` and
`AnnexRepo` no longer fails if the names it needs are no longer
bound. ([3527][])

- [addurls][] botched the construction of subdataset paths that were
more than two levels deep and failed to create datasets in a
reliable, breadth-first order. ([3561][])

- Cloning a `type=git` special remote showed a spurious warning about
the remote not being enabled. ([3547][])

Enhancements and new features

- For calls to git and git-annex, we disable automatic garbage
collection due to past issues with GitPython's state becoming stale,
but doing so results in a larger .git/objects/ directory that isn't
cleaned up until garbage collection is triggered outside of DataLad.
Tests with the latest GitPython didn't reveal any state issues, so
we've re-enabled automatic garbage collection. ([3458][])

- [rerun][] learned an `--explicit` flag, which it relays to its calls
to [run][[]]. This makes it possible to call `rerun` in a dirty
working tree ([3498][]).

- The [metadata][] command aborts earlier if a metadata extractor is
unavailable. ([3525][])

Should be faster and less buggy, with a few enhancements.

Fixes

- [create-sibling][] ([3318][])
- Siblings are no longer configured with a post-update hook unless a
web interface is requested with `--ui`.
- `git submodule update --init` is no longer called from the
post-update hook.
- If `--inherit` is given for a dataset without a superdataset, a
warning is now given instead of raising an error.
- The internal command runner failed on Python 2 when its `env`
argument had unicode values. ([3332][])
- The safeguard that prevents creating a dataset in a subdirectory
that already contains tracked files for another repository failed on
Git versions before 2.14. For older Git versions, we now warn the
caller that the safeguard is not active. ([3347][])
- A regression introduced in v0.11.1 prevented [save][] from committing
changes under a subdirectory when the subdirectory was specified as
a path argument. ([3106][])
- A workaround introduced in v0.11.1 made it possible for [save][] to
do a partial commit with an annex file that has gone below the
`annex.largefiles` threshold. The logic of this workaround was
faulty, leading to files being displayed as typechanged in the index
following the commit. ([3365][])
- The resolve_path() helper confused paths that had a semicolon for
SSH RIs. ([3425][])
- The detection of SSH RIs has been improved. ([3425][])

Enhancements and new features

- The internal command runner was too aggressive in its decision to
sleep. ([3322][])
- The "INFO" label in log messages now retains the default text color
for the terminal rather than using white, which only worked well for
terminals with dark backgrounds. ([3334][])
- A short flag `-R` is now available for the `--recursion-limit` flag,
a flag shared by several subcommands. ([3340][])
- The authentication logic for [create-sibling-github][] has been
revamped and now supports 2FA. ([3180][])
- New configuration option `datalad.ui.progressbar` can be used to
configure the default backend for progress reporting ("none", for
example, results in no progress bars being shown). ([3396][])
- A new progress backend, available by setting datalad.ui.progressbar
to "log", replaces progress bars with a log message upon completion
of an action. ([3396][])
- DataLad learned to consult the [NO_COLOR][] environment variable and
the new `datalad.ui.color` configuration option when deciding to
color output. The default value, "auto", retains the current
behavior of coloring output if attached to a TTY ([3407][]).
- [clean][] now removes annex transfer directories, which is useful
for cleaning up failed downloads. ([3374][])
- [clone][] no longer refuses to clone into a local path that looks
like a URL, making its behavior consistent with `git clone`.
([3425][])
- [wtf][]
- Learned to fall back to the `dist` package if `platform.dist`,
which has been removed in the yet-to-be-release Python 3.8, does
not exist. ([3439][])
- Gained a `--section` option for limiting the output to specific
sections and a `--decor` option, which currently knows how to
format the output as GitHub's `<details>` section. ([3440][])

Largely a bug fix release with a few enhancements

Important

- 0.11.x series will be the last one with support for direct mode of [git-annex][]
which is used on crippled (no symlinks and no locking) filesystems.
v7 repositories should be used instead.

Fixes

- Extraction of .gz files is broken without p7zip installed. We now
abort with an informative error in this situation. ([3176][])

- Committing failed in some cases because we didn't ensure that the
path passed to `git read-tree --index-output=...` resided on the
same filesystem as the repository. ([3181][])

- Some pointless warnings during metadata aggregation have been
eliminated. ([3186][])

- With Python 3 the LORIS token authenticator did not properly decode
a response ([3205][]).

- With Python 3 downloaders unnecessarily decoded the response when
getting the status, leading to an encoding error. ([3210][])

- In some cases, our internal command Runner did not adjust the
environment's `PWD` to match the current working directory specified
with the `cwd` parameter. ([3215][])

- The specification of the pyliblzma dependency was broken. ([3220][])

- [search] displayed an uninformative blank log message in some
cases. ([3222][])

- The logic for finding the location of the aggregate metadata DB
anchored the search path incorrectly, leading to a spurious warning.
([3241][])

- Some progress bars were still displayed when stdout and stderr were
not attached to a tty. ([3281][])

- Check for stdin/out/err to not be closed before checking for `.isatty`.
([3268][])

Enhancements and new features

- Creating a new repository now aborts if any of the files in the
directory are tracked by a repository in a parent directory.
([3211][])

- [run] learned to replace the `{tmpdir}` placeholder in commands with
a temporary directory. ([3223][])

- [duecredit][] support has been added for citing DataLad itself as
well as datasets that an analysis uses. ([3184][])

- The `eval_results` interface helper unintentionally modified one of
its arguments. ([3249][])

- A few DataLad constants have been added, changed, or renamed ([3250][]):
- `HANDLE_META_DIR` is now `DATALAD_DOTDIR`. The old name should be
considered deprecated.
- `METADATA_DIR` now refers to `DATALAD_DOTDIR/metadata` rather than
`DATALAD_DOTDIR/meta` (which is still available as
`OLDMETADATA_DIR`).
- The new `DATASET_METADATA_FILE` refers to `METADATA_DIR/dataset.json`.
- The new `DATASET_CONFIG_FILE` refers to `DATALAD_DOTDIR/config`.
- `METADATA_FILENAME` has been renamed to `OLDMETADATA_FILENAME`.

Just a few of important fixes and minor enhancements.

Fixes

- The logic for setting the maximum command line length now works
around Python 3.4 returning an unreasonably high value for
`SC_ARG_MAX` on Debian systems. ([3165][])

- DataLad commands that are conceptually "read-only", such as
`datalad ls -L`, can fail when the caller lacks write permissions
because git-annex tries merging remote git-annex branches to update
information about availability. DataLad now disables
`annex.merge-annex-branches` in some common "read-only" scenarios to
avoid these failures. ([3164][])

Enhancements and new features

- Accessing an "unbound" dataset method now automatically imports the
necessary module rather than requiring an explicit import from the
Python caller. For example, calling `Dataset.add` no longer needs to
be preceded by `from datalad.distribution.add import Add` or an
import of `datalad.api`. ([3156][])

- Configuring the new variable `datalad.ssh.identityfile` instructs
DataLad to pass a value to the `-i` option of `ssh`. ([3149][])
([3168][])

A variety of bugfixes and enhancements

Major refactoring and deprecations

- All extracted metadata is now placed under git-annex by default.
Previously files smaller than 20 kb were stored in git. ([3109][])
- The function `datalad.cmd.get_runner` has been removed. ([3104][])

Fixes

- Improved handling of long commands:
- The code that inspected `SC_ARG_MAX` didn't check that the
reported value was a sensible, positive number. ([3025][])
- More commands that invoke `git` and `git-annex` with file
arguments learned to split up the command calls when it is likely
that the command would fail due to exceeding the maximum supported
length. ([3138][])
- The `setup_yoda_dataset` procedure created a malformed
.gitattributes line. ([3057][])
- [download-url][] unnecessarily tried to infer the dataset when
`--no-save` was given. ([3029][])
- [rerun][] aborted too late and with a confusing message when a ref
specified via `--onto` didn't exist. ([3019][])
- [run][]:
- `run` didn't preserve the current directory prefix ("./") on
inputs and outputs, which is problematic if the caller relies on
this representation when formatting the command. ([3037][])
- Fixed a number of unicode py2-compatibility issues. ([3035][]) ([3046][])
- To proceed with a failed command, the user was confusingly
instructed to use `save` instead of `add` even though `run` uses
`add` underneath. ([3080][])
- Fixed a case where the helper class for checking external modules
incorrectly reported a module as unknown. ([3051][])
- [add-archive-content][] mishandled the archive path when the leading
path contained a symlink. ([3058][])
- Following denied access, the credential code failed to consider a
scenario, leading to a type error rather than an appropriate error
message. ([3091][])
- Some tests failed when executed from a `git worktree` checkout of the
source repository. ([3129][])
- During metadata extraction, batched annex processes weren't properly
terminated, leading to issues on Windows. ([3137][])
- [add][] incorrectly handled an "invalid repository" exception when
trying to add a submodule. ([3141][])
- Pass `GIT_SSH_VARIANT=ssh` to git processes to be able to specify
alternative ports in SSH urls

Enhancements and new features

- [search][] learned to suggest closely matching keys if there are no
hits. ([3089][])
- [create-sibling][]
- gained a `--group` option so that the caller can specify the file
system group for the repository. ([3098][])
- now understands SSH URLs that have a port in them (i.e. the
"ssh://[user]host.xz[:port]/path/to/repo.git/" syntax mentioned
in `man git-fetch`). ([3146][])
- Interface classes can now override the default renderer for
summarizing results. ([3061][])
- [run][]:
- `--input` and `--output` can now be shortened to `-i` and `-o`.
([3066][])
- Placeholders such as "{inputs}" are now expanded in the command
that is shown in the commit message subject. ([3065][])
- `interface.run.run_command` gained an `extra_inputs` argument so
that wrappers like [datalad-container][] can specify additional inputs
that aren't considered when formatting the command string. ([3038][])
- "--" can now be used to separate options for `run` and those for
the command in ambiguous cases. ([3119][])
- The utilities `create_tree` and `ok_file_has_content` now support
".gz" files. ([3049][])
- The Singularity container for 0.11.1 now uses [nd_freeze][] to make
its builds reproducible.
- A [publications][] page has been added to the documentation. ([3099][])
- `GitRepo.set_gitattributes` now accepts a `mode` argument that
controls whether the .gitattributes file is appended to (default) or
overwritten. ([3115][])
- `datalad --help` now avoids using `man` so that the list of
subcommands is shown. ([3124][])