Zstandard

New Features
------------

* zstd library upgraded from 1.3.6 to 1.3.8.
* ``ZstdCompressor.stream_reader()``, ``ZstdCompressor.stream_writer()``, ``ZstdDecompressor.stream_reader()``, and ``ZstdDecompressor.stream_writer()`` should now completely implement the ``io.RawIOBase`` interface, allowing them to be used as fully-featured streams. One can combine an instance with say an ``io.TextIOWrapper``, ``codecs.StreamReader``, or ``codecs.StreamWriter`` to transparently convert between binary and textual data or apply encodings to data. A stream reader or writer should interoperate with any Python code that accepts a stream-like object. Please note that some implementation details may change slightly in the future in order to ensure better compatibility with the stream interfaces.
* ``ZstdDecompressor.stream_reader()`` can now read inputs that have multiple zstd *frames*. Look for future releases to improve support for reading sources with multiple *frames*.

Backwards Compatibility Notes
-----------------------------

* ``ZstdDecompressor.read()`` now allows reading sizes of ``-1`` or ``0``
and defaults to ``-1``, per the documented behavior of
``io.RawIOBase.read()``. Previously, we required an argument that was
a positive value.
* The ``readline()``, ``readlines()``, ``__iter__``, and ``__next__`` methods
of ``ZstdDecompressionReader()`` now raise ``io.UnsupportedOperation``
instead of ``NotImplementedError``.
* ``ZstdDecompressor.stream_reader()`` now accepts a ``read_across_frames``
argument. The default value will likely be changed in a future release
and consumers are advised to pass the argument to avoid unwanted change
of behavior in the future.
* ``setup.py`` now always disables the CFFI backend if the installed
CFFI package does not meet the minimum version requirements. Before, it was
possible for the CFFI backend to be generated and a run-time error to
occur.
* In the CFFI backend, ``CompressionReader`` and ``DecompressionReader``
were renamed to ``ZstdCompressionReader`` and ``ZstdDecompressionReader``,
respectively so naming is identical to the C extension. This should have
no meaningful end-user impact, as instances aren't meant to be
constructed directly.
* ``ZstdDecompressor.stream_writer()`` now accepts a ``write_return_read``
argument to control whether ``write()`` returns the number of bytes
read from the source / written to the decompressor. It defaults to off,
which preserves the existing behavior of returning the number of bytes
emitted from the decompressor. The default will change in a future release
so behavior aligns with the specified behavior of ``io.RawIOBase``.
* ``ZstdDecompressionWriter.__exit__`` now calls ``self.close()``. This
will result in that stream plus the underlying stream being closed as
well. If this behavior is not desirable, do not use instances as
context managers.
* ``ZstdCompressor.stream_writer()`` now accepts a ``write_return_read``
argument to control whether ``write()`` returns the number of bytes read
from the source / written to the compressor. It defaults to off, which
preserves the existing behavior of returning the number of bytes emitted
from the compressor. The default will change in a future release so
behavior aligns with the specified behavior of ``io.RawIOBase``.
* ``ZstdCompressionWriter.__exit__`` now calls ``self.close()``. This will
result in that stream plus any underlying stream being closed as well. If
this behavior is not desirable, do not use instances as context managers.
* ``ZstdDecompressionWriter`` no longer requires being used as a context
manager (57).
* ``ZstdCompressionWriter`` no longer requires being used as a context
manager (57).
* The ``overlap_size_log`` attribute on ``CompressionParameters`` instances
has been deprecated and will be removed in a future release. The
``overlap_log`` attribute should be used instead.
* The ``overlap_size_log`` argument to ``CompressionParameters`` has been
deprecated and will be removed in a future release. The ``overlap_log``
argument should be used instead.
* The ``ldm_hash_every_log`` attribute on ``CompressionParameters`` instances
has been deprecated and will be removed in a future release. The
``ldm_hash_rate_log`` attribute should be used instead.
* The ``ldm_hash_every_log`` argument to ``CompressionParameters`` has been
deprecated and will be removed in a future release. The ``ldm_hash_rate_log``
argument should be used instead.
* The ``compression_strategy`` argument to ``CompressionParameters`` has been
deprecated and will be removed in a future release. The ``strategy``
argument should be used instead.
* The ``SEARCHLENGTH_MIN`` and ``SEARCHLENGTH_MAX`` constants are deprecated
and will be removed in a future release. Use ``MINMATCH_MIN`` and
``MINMATCH_MAX`` instead.
* The ``zstd_cffi`` module has been renamed to ``zstandard.cffi``. As had
been documented in the ``README`` file since the ``0.9.0`` release, the
module should not be imported directly at its new location. Instead,
``import zstandard`` to cause an appropriate backend module to be loaded
automatically.

Bug Fixes
---------

* CFFI backend could encounter a failure when sending an empty chunk into
``ZstdDecompressionObj.decompress()``. The issue has been fixed.
* CFFI backend could encounter an error when calling
``ZstdDecompressionReader.read()`` if there was data remaining in an
internal buffer. The issue has been fixed. (71)

Changes
-------

* ``ZstDecompressionObj.decompress()`` now properly handles empty inputs in
the CFFI backend.
* ``ZstdCompressionReader`` now implements ``read1()`` and ``readinto1()``.
These are part of the ``io.BufferedIOBase`` interface.
* ``ZstdCompressionReader`` has gained a ``readinto(b)`` method for reading
compressed output into an existing buffer.
* ``ZstdCompressionReader.read()`` now defaults to ``size=-1`` and accepts
read sizes of ``-1`` and ``0``. The new behavior aligns with the documented
behavior of ``io.RawIOBase``.
* ``ZstdCompressionReader`` now implements ``readall()``. Previously, this
method raised ``NotImplementedError``.
* ``ZstdDecompressionReader`` now implements ``read1()`` and ``readinto1()``.
These are part of the ``io.BufferedIOBase`` interface.
* ``ZstdDecompressionReader.read()`` now defaults to ``size=-1`` and accepts
read sizes of ``-1`` and ``0``. The new behavior aligns with the documented
behavior of ``io.RawIOBase``.
* ``ZstdDecompressionReader()`` now implements ``readall()``. Previously, this
method raised ``NotImplementedError``.
* The ``readline()``, ``readlines()``, ``__iter__``, and ``__next__`` methods
of ``ZstdDecompressionReader()`` now raise ``io.UnsupportedOperation``
instead of ``NotImplementedError``. This reflects a decision to never
implement text-based I/O on (de)compressors and keep the low-level API
operating in the binary domain. (13)
* ``README.rst`` now documented how to achieve linewise iteration using
an ``io.TextIOWrapper`` with a ``ZstdDecompressionReader``.
* ``ZstdDecompressionReader`` has gained a ``readinto(b)`` method for
reading decompressed output into an existing buffer. This allows chaining
to an ``io.TextIOWrapper`` on Python 3 without using an ``io.BufferedReader``.
* ``ZstdDecompressor.stream_reader()`` now accepts a ``read_across_frames``
argument to control behavior when the input data has multiple zstd
*frames*. When ``False`` (the default for backwards compatibility), a
``read()`` will stop when the end of a zstd *frame* is encountered. When
``True``, ``read()`` can potentially return data spanning multiple zstd
*frames*. The default will likely be changed to ``True`` in a future
release.
* ``setup.py`` now performs CFFI version sniffing and disables the CFFI
backend if CFFI is too old. Previously, we only used ``install_requires``
to enforce the CFFI version and not all build modes would properly enforce
the minimum CFFI version. (69)
* CFFI's ``ZstdDecompressionReader.read()`` now properly handles data
remaining in any internal buffer. Before, repeated ``read()`` could
result in *random* errors. (71)
* Upgraded various Python packages in CI environment.
* Upgrade to hypothesis 4.5.11.
* In the CFFI backend, ``CompressionReader`` and ``DecompressionReader``
were renamed to ``ZstdCompressionReader`` and ``ZstdDecompressionReader``,
respectively.
* ``ZstdDecompressor.stream_writer()`` now accepts a ``write_return_read``
argument to control whether ``write()`` returns the number of bytes read
from the source. It defaults to ``False`` to preserve backwards
compatibility.
* ``ZstdDecompressor.stream_writer()`` now implements the ``io.RawIOBase``
interface and behaves as a proper stream object.
* ``ZstdCompressor.stream_writer()`` now accepts a ``write_return_read``
argument to control whether ``write()`` returns the number of bytes read
from the source. It defaults to ``False`` to preserve backwards
compatibility.
* ``ZstdCompressionWriter`` now implements the ``io.RawIOBase`` interface and
behaves as a proper stream object. ``close()`` will now close the stream
and the underlying stream (if possible). ``__exit__`` will now call
``close()``. Methods like ``writable()`` and ``fileno()`` are implemented.
* ``ZstdDecompressionWriter`` no longer must be used as a context manager.
* ``ZstdCompressionWriter`` no longer must be used as a context manager.
When not using as a context manager, it is important to call
``flush(FRAME_FRAME)`` or the compression stream won't be properly
terminated and decoders may complain about malformed input.
* ``ZstdCompressionWriter.flush()`` (what is returned from
``ZstdCompressor.stream_writer()``) now accepts an argument controlling the
flush behavior. Its value can be one of the new constants
``FLUSH_BLOCK`` or ``FLUSH_FRAME``.
* ``ZstdDecompressionObj`` instances now have a ``flush([length=None])`` method.
This provides parity with standard library equivalent types. (65)
* ``CompressionParameters`` no longer redundantly store individual compression
parameters on each instance. Instead, compression parameters are stored inside
the underlying ``ZSTD_CCtx_params`` instance. Attributes for obtaining
parameters are now properties rather than instance variables.
* Exposed the ``STRATEGY_BTULTRA2`` constant.
* ``CompressionParameters`` instances now expose an ``overlap_log`` attribute.
This behaves identically to the ``overlap_size_log`` attribute.
* ``CompressionParameters()`` now accepts an ``overlap_log`` argument that
behaves identically to the ``overlap_size_log`` argument. An error will be
raised if both arguments are specified.
* ``CompressionParameters`` instances now expose an ``ldm_hash_rate_log``
attribute. This behaves identically to the ``ldm_hash_every_log`` attribute.
* ``CompressionParameters()`` now accepts a ``ldm_hash_rate_log`` argument that
behaves identically to the ``ldm_hash_every_log`` argument. An error will be
raised if both arguments are specified.
* ``CompressionParameters()`` now accepts a ``strategy`` argument that behaves
identically to the ``compression_strategy`` argument. An error will be raised
if both arguments are specified.
* The ``MINMATCH_MIN`` and ``MINMATCH_MAX`` constants were added. They are
semantically equivalent to the old ``SEARCHLENGTH_MIN`` and
``SEARCHLENGTH_MAX`` constants.
* Bundled zstandard library upgraded from 1.3.7 to 1.3.8.
* ``setup.py`` denotes support for Python 3.7 (Python 3.7 was supported and
tested in the 0.10 release).
* ``zstd_cffi`` module has been renamed to ``zstandard.cffi``.
* ``ZstdCompressor.stream_writer()`` now reuses a buffer in order to avoid
allocating a new buffer for every operation. This should result in faster
performance in cases where ``write()`` or ``flush()`` are being called
frequently. (62)
* Bundled zstandard library upgraded from 1.3.6 to 1.3.7.

Bug Fixes
---------

* ``zstd_cffi.py`` added to ``setup.py`` (60).

Changes
-------

* Change some integer casts to avoid ``ssize_t`` (61).

Backwards Compatibility Notes
-----------------------------

* ``ZstdCompressor.stream_reader().closed`` is now a property instead of a
method (58).
* ``ZstdDecompressor.stream_reader().closed`` is now a property instead of a
method (58).

Changes
-------

* Stop attempting to package Python 3.6 for Miniconda. The latest version of
Miniconda is using Python 3.7. The Python 3.6 Miniconda packages were a lie
since this were built against Python 3.7.
* ``ZstdCompressor.stream_reader()``'s and ``ZstdDecompressor.stream_reader()``'s
``closed`` attribute is now a read-only property instead of a method. This now
properly matches the ``IOBase`` API and allows instances to be used in more
places that accept ``IOBase`` instances.

Backwards Compatibility Notes
-----------------------------

* ``ZstdDecompressor.stream_reader().read()`` now consistently requires an
argument in both the C and CFFI backends. Before, the CFFI implementation
would assume a default value of ``-1``, which was later rejected.
* The ``compress_literals`` argument and attribute has been removed from
``zstd.ZstdCompressionParameters`` because it was removed by the zstd 1.3.5
API.
* ``ZSTD_CCtx_setParametersUsingCCtxParams()`` is no longer called on every
operation performed against ``ZstdCompressor`` instances. The reason for this
change is that the zstd 1.3.5 API no longer allows this without calling
``ZSTD_CCtx_resetParameters()`` first. But if we called
``ZSTD_CCtx_resetParameters()`` on every operation, we'd have to redo
potentially expensive setup when using dictionaries. We now call
``ZSTD_CCtx_reset()`` on every operation and don't attempt to change
compression parameters.
* Objects returned by ``ZstdCompressor.stream_reader()`` no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.
* Objects returned by ``ZstdDecompressor.stream_reader()`` no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.

Bug Fixes
---------

* ``ZstdDecompressor.decompressobj().decompress()`` should now return all data
from internal buffers in more scenarios. Before, it was possible for data to
remain in internal buffers. This data would be emitted on a subsequent call
to ``decompress()``. The overall output stream would still be valid. But if
callers were expecting input data to exactly map to output data (say the
producer had used ``flush(COMPRESSOBJ_FLUSH_BLOCK)`` and was attempting to
map input chunks to output chunks), then the previous behavior would be
wrong. The new behavior is such that output from
``flush(COMPRESSOBJ_FLUSH_BLOCK)`` fed into ``decompressobj().decompress()``
should produce all available compressed input.
* ``ZstdDecompressor.stream_reader().read()`` should no longer segfault after
a previous context manager resulted in error (56).
* ``ZstdCompressor.compressobj().flush(COMPRESSOBJ_FLUSH_BLOCK)`` now returns
all data necessary to flush a block. Before, it was possible for the
``flush()`` to not emit all data necessary to fully represent a block. This
would mean decompressors wouldn't be able to decompress all data that had been
fed into the compressor and ``flush()``ed. (55).

New Features
------------

* New module constants ``BLOCKSIZELOG_MAX``, ``BLOCKSIZE_MAX``,
``TARGETLENGTH_MAX`` that expose constants from libzstd.
* New ``ZstdCompressor.chunker()`` API for manually feeding data into a
compressor and emitting chunks of a fixed size. Like ``compressobj()``, the
API doesn't impose restrictions on the input or output types for the
data streams. Unlike ``compressobj()``, it ensures output chunks are of a
fixed size. This makes this API useful when the compressed output is being
fed into an I/O layer, where uniform write sizes are useful.
* ``ZstdCompressor.stream_reader()`` no longer needs to be used as a context
manager (34).
* ``ZstdDecompressor.stream_reader()`` no longer needs to be used as a context
manager (34).
* Bundled zstandard library upgraded from 1.3.4 to 1.3.6.

Changes
-------

* ``zstandard.__version__`` is now defined (50).
* Upgrade pip, setuptools, wheel, and cibuildwheel packages to latest versions.
* Upgrade various packages used in CI to latest versions. Notably tox (in
order to support Python 3.7).
* Use relative paths in setup.py to appease Python 3.7 (51).
* Added CI for Python 3.7.

This release contains packaging improvements. No changes to the Python library code were made in this release.

Backwards Compatibility Notes
-----------------------------

* CFFI 1.11 or newer is now required (previous requirement was 1.8).
* The primary module is now ``zstandard``. Please change imports of ``zstd``
and ``zstd_cffi`` to ``import zstandard``. See the README for more. Support
for importing the old names will be dropped in the next release.
* ``ZstdCompressor.read_from()`` and ``ZstdDecompressor.read_from()`` have
been renamed to ``read_to_iter()``. ``read_from()`` is aliased to the new
name and will be deleted in a future release.
* Support for Python 2.6 has been removed.
* Support for Python 3.3 has been removed.
* The ``selectivity`` argument to ``train_dictionary()`` has been removed, as
the feature disappeared from zstd 1.3.
* Support for legacy dictionaries has been removed. Cover dictionaries are now
the default. ``train_cover_dictionary()`` has effectively been renamed to
``train_dictionary()``.
* The ``allow_empty`` argument from ``ZstdCompressor.compress()`` has been
deleted and the method now allows empty inputs to be compressed by default.
* ``estimate_compression_context_size()`` has been removed. Use
``CompressionParameters.estimated_compression_context_size()`` instead.
* ``get_compression_parameters()`` has been removed. Use
``CompressionParameters.from_level()`` instead.
* The arguments to ``CompressionParameters.__init__()`` have changed. If you
were using positional arguments before, the positions now map to different
arguments. It is recommended to use keyword arguments to construct
``CompressionParameters`` instances.
* ``TARGETLENGTH_MAX`` constant has been removed (it disappeared from zstandard
1.3.4).
* ``ZstdCompressor.write_to()`` and ``ZstdDecompressor.write_to()`` have been
renamed to ``ZstdCompressor.stream_writer()`` and
``ZstdDecompressor.stream_writer()``, respectively. The old names are still
aliased, but will be removed in the next major release.
* Content sizes are written into frame headers by default
(``ZstdCompressor(write_content_size=True)`` is now the default).
* ``CompressionParameters`` has been renamed to ``ZstdCompressionParameters``
for consistency with other types. The old name is an alias and will be removed
in the next major release.

Bug Fixes
---------

* Fixed memory leak in ``ZstdCompressor.copy_stream()`` (40) (from 0.8.2).
* Fixed memory leak in ``ZstdDecompressor.copy_stream()`` (35) (from 0.8.2).
* Fixed memory leak of ``ZSTD_DDict`` instances in CFFI's ``ZstdDecompressor``.

New Features
------------

* Bundlded zstandard library upgraded from 1.1.3 to 1.3.4. This delivers various
bug fixes and performance improvements. It also gives us access to newer
features.
* Support for negative compression levels.
* Support for *long distance matching* (facilitates compression ratios that approach
LZMA).
* Supporting for reading empty zstandard frames (with an embedded content size
of 0).
* Support for writing and partial support for reading zstandard frames without a
magic header.
* New ``stream_reader()`` API that exposes the ``io.RawIOBase`` interface (allows
you to ``.read()`` from a file-like object).
* Several minor features, bug fixes, and performance enhancements.
* Wheels for Linux and macOS are now provided with releases.

Changes
-------

* Functions accepting bytes data now use the buffer protocol and can accept
more types (like ``memoryview`` and ``bytearray``) (26).
* Add includes so compilation on OS X and BSDs works (20).
* New ``ZstdDecompressor.stream_reader()`` API to obtain a read-only i/o stream
of decompressed data for a source.
* New ``ZstdCompressor.stream_reader()`` API to obtain a read-only i/o stream of
compressed data for a source.
* Renamed ``ZstdDecompressor.read_from()`` to ``ZstdDecompressor.read_to_iter()``.
The old name is still available.
* Renamed ``ZstdCompressor.read_from()`` to ``ZstdCompressor.read_to_iter()``.
``read_from()`` is still available at its old location.
* Introduce the ``zstandard`` module to import and re-export the C or CFFI
*backend* as appropriate. Behavior can be controlled via the
``PYTHON_ZSTANDARD_IMPORT_POLICY`` environment variable. See README for
usage info.
* Vendored version of zstd upgraded to 1.3.4.
* Added module constants ``CONTENTSIZE_UNKNOWN`` and ``CONTENTSIZE_ERROR``.
* Add ``STRATEGY_BTULTRA`` compression strategy constant.
* Switch from deprecated ``ZSTD_getDecompressedSize()`` to
``ZSTD_getFrameContentSize()`` replacement.
* ``ZstdCompressor.compress()`` can now compress empty inputs without requiring
special handling.
* ``ZstdCompressor`` and ``ZstdDecompressor`` now have a ``memory_size()``
method for determining the current memory utilization of the underlying zstd
primitive.
* ``train_dictionary()`` has new arguments and functionality for trying multiple
variations of COVER parameters and selecting the best one.
* Added module constants ``LDM_MINMATCH_MIN``, ``LDM_MINMATCH_MAX``, and
``LDM_BUCKETSIZELOG_MAX``.
* Converted all consumers to the zstandard *new advanced API*, which uses
``ZSTD_compress_generic()``
* ``CompressionParameters.__init__`` now accepts several more arguments,
including support for *long distance matching*.
* ``ZstdCompressionDict.__init__`` now accepts a ``dict_type`` argument that
controls how the dictionary should be interpreted. This can be used to
force the use of *content-only* dictionaries or to require the presence
of the dictionary magic header.
* ``ZstdCompressionDict.precompute_compress()`` can be used to precompute the
compression dictionary so it can efficiently be used with multiple
``ZstdCompressor`` instances.
* Digested dictionaries are now stored in ``ZstdCompressionDict`` instances,
created automatically on first use, and automatically reused by all
``ZstdDecompressor`` instances bound to that dictionary.
* All meaningful functions now accept keyword arguments.
* ``ZstdDecompressor.decompressobj()`` now accepts a ``write_size`` argument
to control how much work to perform on every decompressor invocation.
* ``ZstdCompressor.write_to()`` now exposes a ``tell()``, which exposes the
total number of bytes written so far.
* ``ZstdDecompressor.stream_reader()`` now supports ``seek()`` when moving
forward in the stream.
* Removed ``TARGETLENGTH_MAX`` constant.
* Added ``frame_header_size(data)`` function.
* Added ``frame_content_size(data)`` function.
* Consumers of ``ZSTD_decompress*`` have been switched to the new *advanced
decompression* API.
* ``ZstdCompressor`` and ``ZstdCompressionParams`` can now be constructed with
negative compression levels.
* ``ZstdDecompressor`` now accepts a ``max_window_size`` argument to limit the
amount of memory required for decompression operations.
* ``FORMAT_ZSTD1`` and ``FORMAT_ZSTD1_MAGICLESS`` constants to be used with
the ``format`` compression parameter to control whether the frame magic
header is written.
* ``ZstdDecompressor`` now accepts a ``format`` argument to control the
expected frame format.
* ``ZstdCompressor`` now has a ``frame_progression()`` method to return
information about the current compression operation.
* Error messages in CFFI no longer have ``b''`` literals.
* Compiler warnings and underlying overflow issues on 32-bit platforms have been
fixed.
* Builds in CI now build with compiler warnings as errors. This should hopefully
fix new compiler warnings from being introduced.
* Make ``ZstdCompressor(write_content_size=True)`` and
``CompressionParameters(write_content_size=True)`` the default.
* ``CompressionParameters`` has been renamed to ``ZstdCompressionParameters``.