Datalad

Bugfixes

Fixes

- Do not attempt to [update][] a not installed sub-dataset
- In case of too many files to be specified for [get][] or [copy_to][], we
will make multiple invocations of underlying git-annex command to not
overfill command line
- More robust handling of unicode output in terminals which might not support it

Enhancements and new features

- Ship a copy of numpy.testing to facilitate [test][] without requiring numpy
as dependency. Also allow to pass to command which test(s) to run
- In [get][] and [copy_to][] provide actual original requested paths, not the
ones we deduced need to be transferred, solely for knowing the total

A variety of fixes and enhancements

Fixes

- [publish][] would now push merged `git-annex` branch even if no other changes
were done
- [publish][] should be able to publish using relative path within SSH URI
(git hook would use relative paths)
- [publish][] should better tollerate publishing to pure git and `git-annex`
special remotes

Enhancements and new features

- [plugin][] mechanism came to replace [export][]. See [export_tarball][] for the
replacement of [export][]. Now it should be easy to extend datalad's interface
with custom functionality to be invoked along with other commands.
- Minimalistic coloring of the results rendering
- [publish][]/`copy_to` got progress bar report now and support of `--jobs`
- minor fixes and enhancements to crawler (e.g. support of recursive removes)

New features, refactorings, and bug fixes.

Major refactoring and deprecations

- [add-sibling][] has been fully replaced by the [siblings][] command
- [create-sibling][], and [unlock][] have been re-written to support the
same common API as most other commands

Enhancements and new features

- [siblings][] can now be used to query and configure a local repository by
using the sibling name ``here``
- [siblings][] can now query and set annex preferred content configuration. This
includes ``wanted`` (as previously supported in other commands), and now
also ``required``
- New [metadata][] command to interface with datasets/files [meta-data][]
- Documentation for all commands is now built in a uniform fashion
- Significant parts of the documentation of been updated
- Instantiate GitPython's Repo instances lazily

Fixes

- API documentation is now rendered properly as HTML, and is easier to browse by
having more compact pages
- Closed files left open on various occasions (Popen PIPEs, etc)
- Restored basic (consumer mode of operation) compatibility with Windows OS

This release includes a **huge** refactoring to make code base and functionality
more robust and flexible

- outputs from API commands could now be highly customized. See
`--output-format`, `--report-status`, `--report-type`, and `--report-type`
options for [datalad][] command.
- effort was made to refactor code base so that underlying functions behave as
generators where possible
- input paths/arguments analysis was redone for majority of the commands to provide
unified behavior

Major refactoring and deprecations

- `add-sibling` and `rewrite-urls` were refactored in favor of new [siblings][]
command which should be used for siblings manipulations
- 'datalad.api.alwaysrender' config setting/support is removed in favor of new
outputs processing

Fixes

- Do not flush manually git index in pre-commit to avoid "Death by the Lock" issue
- Deployed by [publish][] `post-update` hook script now should be more robust
(tolerate directory names with spaces, etc.)
- A variety of fixes, see
[list of pull requests and issues closed](https://github.com/datalad/datalad/milestone/41?closed=1)
for more information

Enhancements and new features

- new [annotate-paths][] plumbing command to inspect and annotate provided
paths. Use `--modified` to summarize changes between different points in
the history
- new [clone][] plumbing command to provide a subset (install a single dataset
from a URL) functionality of [install][]
- new [diff][] plumbing command
- new [siblings][] command to list or manipulate siblings
- new [subdatasets][] command to list subdatasets and their properties
- [drop][] and [remove][] commands were refactored
- `benchmarks/` collection of [Airspeed velocity](https://github.com/spacetelescope/asv/)
benchmarks initiated. See reports at http://datalad.github.io/datalad/
- crawler would try to download a new url multiple times increasing delay between
attempts. Helps to resolve problems with extended crawls of Amazon S3
- [CRCNS][] crawler pipeline now also fetches and aggregates meta-data for the
datasets from datacite
- overall optimisations to benefit from the aforementioned refactoring and
improve user-experience
- a few stub and not (yet) implemented commands (e.g. `move`) were removed from
the interface
- Web frontend got proper coloring for the breadcrumbs and some additional
caching to speed up interactions. See http://datasets.datalad.org
- Small improvements to the online documentation. See e.g.
[summary of differences between git/git-annex/datalad](http://docs.datalad.org/en/latest/related.html#git-git-annex-datalad)

A bugfix release

Fixes

- [add][] was forcing addition of files to annex regardless of settings
in `.gitattributes`. Now that decision is left to annex by default
- `tools/testing/run_doc_examples` used to run
doc examples as tests, fixed up to provide status per each example
and not fail at once
- `doc/examples`
- [3rdparty_analysis_workflow.sh](http://docs.datalad.org/en/latest/generated/examples/3rdparty_analysis_workflow.html)
was fixed up to reflect changes in the API of 0.5.0.
- progress bars
- should no longer crash **datalad** and report correct sizes and speeds
- should provide progress reports while using Python 3.x

Enhancements and new features

- `doc/examples`
- [nipype_workshop_dataset.sh](http://docs.datalad.org/en/latest/generated/examples/nipype_workshop_dataset.html)
new example to demonstrate how new super- and sub- datasets were established
as a part of our datasets collection

This release includes an avalanche of bug fixes, enhancements, and
additions which at large should stay consistent with previous behavior
but provide better functioning. Lots of code was refactored to provide
more consistent code-base, and some API breakage has happened. Further
work is ongoing to standardize output and results reporting
([1350][])

Most notable changes

- requires [git-annex][] >= 6.20161210 (or better even >= 6.20161210 for
improved functionality)
- commands should now operate on paths specified (if any), without
causing side-effects on other dirty/staged files
- [save][]
- `-a` is deprecated in favor of `-u` or `--all-updates`
so only changes known components get saved, and no new files
automagically added
- `-S` does no longer store the originating dataset in its commit
message
- [add][]
- can specify commit/save message with `-m`
- [add-sibling][] and [create-sibling][]
- now take the name of the sibling (remote) as a `-s` (`--name`)
option, not a positional argument
- `--publish-depends` to setup publishing data and code to multiple
repositories (e.g. github + webserve) should now be functional
see [this comment](https://github.com/datalad/datalad/issues/335#issuecomment-277240733)
- got `--publish-by-default` to specify what refs should be published
by default
- got `--annex-wanted`, `--annex-groupwanted` and `--annex-group`
settings which would be used to instruct annex about preferred
content. [publish][] then will publish data using those settings if
`wanted` is set.
- got `--inherit` option to automagically figure out url/wanted and
other git/annex settings for new remote sub-dataset to be constructed
- [publish][]
- got `--skip-failing` refactored into `--missing` option
which could use new feature of [create-sibling][] `--inherit`

Fixes

- More consistent interaction through ssh - all ssh connections go
through [sshrun][] shim for a "single point of authentication", etc.
- More robust [ls][] operation outside of the datasets
- A number of fixes for direct and v6 mode of annex

Enhancements and new features

- New [drop][] and [remove][] commands
- [clean][]
- got `--what` to specify explicitly what cleaning steps to perform
and now could be invoked with `-r`
- `datalad` and `git-annex-remote*` scripts now do not use setuptools
entry points mechanism and rely on simple import to shorten start up time
- [Dataset][] is also now using [Flyweight pattern][], so the same instance is
reused for the same dataset
- progressbars should not add more empty lines

Internal refactoring

- Majority of the commands now go through `_prep` for arguments validation
and pre-processing to avoid recursive invocations