Changelogs » Blosc

PyUp Safety actively tracks 263,166 Python packages for vulnerabilities and notifies you when to upgrade.



  * Added `<unistd.h>` in vendored zlib 1.2.8 for compatibility with Python 3.8 in recent Mac OSX.  For details, see:


  * More saftey checks have been implemented so that potential flaws
  discovered by new fuzzers in OSS-Fuzzer are fixed now.  Thanks to
  Nathan Moinvaziri (nmoinvaz).
  * BloscLZ updated to 2.3.0. Expect better compression ratios for faster
  codecs.  For details, see our new blog post:
  * Fixed the `_xgetbv()` collision. Thanks to Michał Górny (mgorny).
  * The chunk format has been fully described so that 3rd party software
  may come with a different implementation, but still compatible with
  C-Blosc chunks.


  - The length of automatic blocksizes for fast codecs (lz4, blosclz) has
  been incremented quite a bit (up to 256 KB) for better compression ratios.
  The performance in modern CPUs (with at least 256 KB in L2 cache) should
  be better too (for older CPUs the performance should stay roughly the same).
  - Continuous integration has been migrated to GitHub actions and much
  more scenarios are tested (specially linking with external codecs).
  Also, a new OSS-Fuzz workflow has been added for increased detection
  of possible vulnerabilities.  Thanks to Nathan Moinvaziri.
  - For small buffers that cannot be compressed (typically < 128 bytes),
  `blosc_compress()` returns now a 0 (cannot compress) instead of a negative
  number (internal error).  See
  Thanks to kalvdans for providing the initial patch.
  - blosclz codec updated to 2.1.0.  Expect better compression ratios and
  performance in a wider variety of scenarios.
  - `blosc_decompress_unsafe()`, `blosc_decompress_ctx_unsafe()` and
  `blosc_getitem_unsafe()` have been removed because they are dangerous
  and after latest improvements, they should not be used in production.
  - zstd codec updated to 1.4.5.
  - Conan packaging has been deprecated (from now on, we should try
  to focus on supporting wheels only).


  - Fixed the copy of the leftovers of a chunk when its size is not a
  multiple of the typesize.  Although this is a very unusual situation,
  it can certainly happen (e.g.


  - Fixed the copy of the leftovers of a chunk when its size is not a
  multiple of the typesize.  Although this is a very unusual situation,
  it can certainly happen (e.g.


  - Zstd codec updated to 1.4.4.
  - LZ4 codec updated to 1.9.2.


  - LZ4 codec updated to 1.9.1.
  - Zstd codec updated to 1.4.1.
  - BloscLZ codec updated to 2.0.0.  Although this should be fully backward
  compatible, it contains important changes that affects mainly speed, but
  also compression ratios.  Feedback on how it behaves on your own data is


  - Fix for building for clang with -march=haswell. See PR 262.
  - Fix all the known warnings for GCC/Clang.  Still some work to do for MSVC
  in this front.
  - Due to some problems with several CI systems, the check for library symbols
  are deactivated now by default.  If you want to enforce this check, use:
  `cmake .. -DDEACTIVATE_SYMBOLS_CHECK=ON` to re-activate it.


- Correct the check for the compressed size when the buffer is memcpyed.  This
  was a regression introduced in 1.16.0.  Fixes 261.


- Fixed a regression in 1.16.0 that prevented to compress empty buffers
  (see 260).
  - Zstd updated to 1.3.8 (from 1.3.7).


- Now the functions that execute Blosc decompressions are safe by default
  for untrusted/possibly corrupted inputs.  The additional checks seem to
  not affect performance significantly (see some benchmarks in 258), so
  this is why they are the default now.
  The previous functions (with less safety) checks are still available with a
  '_unsafe' suffix.  The complete list is:
  - blosc_decompress_unsafe()
  - blosc_decompress_ctx_unsafe()
  - blosc_getitem_unsafe()
  Also, a new API function named blosc_cbuffer_validate(), for validating Blosc
  compressed data, has been added.
  For details, see PR 258.  Thanks to Jeremy Maitin-Shepard.
  - Fixed a bug in `blosc_compress()` that could lead to thread deadlock under
  some situations.  See 251.  Thanks to wenjuno for the report and the fix.
  - Fix data race in shuffle.c host_implementation initialization.  Fixes 253.
  Thanks to Jeremy Maitin-Shepard.


This is a maintenance release that adds a workaround for Visual Studio
  2008's lack of a stdint.h file to blosclz.c.
  For more info, please see the release notes in:


In this release, the `blosc_compress()` and `blosc_decompress()` interfaces
  are now fork-safe, preventing child-process deadlocks in fork-based
  multiprocessing applications. These interfaces with BLOSC_NOLOCK were, and
  continue to be, fork-safe. `_ctx` interface context reuse continues to be
  unsafe in the child process post-fork.  Thanks to Alex Ford.
  Also, a few bugs have been fixed, more compatibility with oldish compilers
  and LZ4 and Zstd codecs have been updated to their latest versions.
  For more info, please see the release notes in:


  - Added a new `DEACTIVATE_SSE2` option for cmake that is useful for disabling
  SSE2 when doing cross-compilation (see 236).
  - New check for detecting output buffers smaller than BLOSC_MAX_OVERHEAD.
  Fixes 234.
  - The `complib` and `version` parameters for `blosc_get_complib_info()` can be
  safely set to NULL now.  This allows to call this function even if the user is
  not interested in these parameters (so no need to reserve memory for them).
  Fixes 228.
  - In some situations that a supposedly blosc chunk is passed to
  `blosc_decompress()`, one might end with an `Arithmetic exception`.  This
  is probably due to the chunk not being an actual blosc chunk, and divisions
  by zero might occur.  A protection has been added for this. See 237.


The main change for this release is that the pthreads library is
  not needed on any Windows build anymore.  Thanks to Steven G. Johnson.
  Also, the internal Zstd sources have been updated to 1.3.4; expect
  an important performance boost (can be up to 10%, specially for low
  compression levels).
  For more info, please see the release notes in:


- Reverted the $Configuration var in CMake configuration for Windows so
  as to restore the compatibility with MS VisualStudio compilers.


- Fixed a bug that caused C-Blosc to crash on platforms requiring strict
  alignment (as in some kinds of ARM CPUs).  Fixes 223.  Thanks to Elvis
  Stansvik and Michael Hudson-Doyle for their help.
  - Fixed a piece of code that was not C89 compliant.  C89 compliance is
  needed mainly by MS VS2008 which is still used for creating Python 2
  - Remove the (spurious) $Configuration var in cmake config for Windows.
  Thanks to Francis Brissette for pointing this out.


- New split mode that favors forward compatibility.  That means that,
  from now on, all the buffers created starting with blosc 1.14.0 will
  be forward compatible with any previous versions of the library --at
  least until 1.3.0, when support for multi-codec was introduced.
  So as to select the split mode, a new API function has been introduced:
  Also, the BLOSC_SPLITMODE environment variable is honored when using
  the `blosc_compress()` function.  See
  There is a dedicated blog entry about this at:
  More info in PR 216.
  **Caveat Emptor:** Note that Blosc versions from 1.11.0 to 1.14.0 *might*
  generate buffers that cannot be read with versions < 1.11.0, so if
  forward compatibility is important to you, an upgrade to 1.14.0 is
  - All warnings during cmake build stage are enabled by default now.
  PR 218.  Thanks to kalvdans.
  - Better checks on versions of formats inside Blosc.  PR 219.  Thanks
  to kalvdans.
  - The BLOSC_PRINT_SHUFFLE_ACCEL environment variable is honored now.
  This is useful for determining *at runtime* whether the different SIMD
  capabilities (only for Intel processors) are available to Blosc to get
  better performance during shuffle/bitshuffle operation.  As an example,
  here it is the normal output for the simple.c example:
  $ ./simple
  Blosc version info: ($Date:: 2018-02-15 $)
  Compression: 4000000 -> 41384 (96.7x)
  Decompression succesful!
  Succesful roundtrip!
  and here with the BLOSC_PRINT_SHUFFLE_ACCEL environment variable set:
  Blosc version info: ($Date:: 2018-02-15 $)
  Shuffle CPU Information:
  SSE2 available: True
  SSE3 available: True
  SSSE3 available: True
  SSE4.1 available: True
  SSE4.2 available: True
  AVX2 available: True
  AVX512BW available: False
  XSAVE available: True
  XSAVE enabled: True
  XMM state enabled: True
  YMM state enabled: True
  ZMM state enabled: False
  Compression: 4000000 -> 41384 (96.7x)
  Decompression succesful!
  Succesful roundtrip!
  Blosc only currently leverages the SSE2 and AVX2 instruction sets, but
  it can recognize all of the above.  This is useful mainly for debugging.


New conan binaries publicly accessible in
  Still experimental, but feedback is appreciated.  No changes have been
  made to codebase, so yo don't need to update.
  For more info, please see the release notes in:


Fixed a buffer overrun that happens when compressing small buffers and
  `len(destination_buffer) < len(source_buffer) + BLOSC_MAX_OVERHEAD'.
  Thanks to Ivan Smirnov for reporting this.


Tests should work now when external compressors are located in non-system locations.  Fixes 210.  Thanks to Leif Walsh.


- C-Blosc can be compiled on CentOS 6 now.
  - LZ4 internal codec upgraded to 1.8.1.
  For more info, please see the release notes in:


Fixed a bug uncovered by the python-blosc test suite: when a buffer is
  to be copied, then we should reserve space for the header, not block
  For more info, please see the release notes in:


A serious optimization of memory copy functions (see `blosc/fastcopy.c`) .
  This benefits the speed of all the codecs, but specially the BloscLZ one.
  As a result of the above, the BloscLZ codec received a new adjustment of
  knobs so that you should expect better compression ratios with BloscLZ too.
  Also, the LZ4 codec internal sources have been updated to 1.8.0. And internal
  Zstd sources have been updated to 1.3.3 too.


Backported BloscLZ parameters that were fine-tuned for C-Blosc2.
  You should expect better compression ratios and faster operation,
  specially on modern CPUs.  See:


In order to avoid problems with platforms with incomplete development
  library codecs, the default is now to compile the internal codecs.
  Users can still compile Blosc with external codecs by properly setting
  Also, the internal Zstd codec has been updated to 1.3.0.
  For more info, please see the release notes in:


Enabled use as a CMake subproject, exporting shared & static library targets for super-projects to use. See PRs 178, 179 and 180.  Thanks to Kevin Murray.
  Also, internal codecs have been updated.  LZ4 and LZ4HC codecs to 1.7.5 and Zstd to 1.1.2.
  For more info, please see the release notes in:


  - Internal C-Blosc updated to 1.20.1.  This fixes, and also brings  many new updates in internal codecs, providing interesting bumps in performance in some cases.
  - Due to recent addition of more cores in new CPUs, the number of  internal threads to be used by default has been increased from 4 to 8.
  - Allow zero-copy decompression by allowing bytes-like input.  See PR:  Thanks to Lehman  Garrison.
  - Fix DeprecationWarning due to invalid escape sequence and use array.tobytes for Python 3.9.


  - Disable the attempt to include support for SSE2 and AVX2 on non-Intel
  platforms, allowing the build on such platforms (see 244).  Thanks
  to Lehman Garrison.


  - Dropped support for Python 2.7 and 3.5.
  - Fixed the copy of the leftovers of a chunk when its size is not a
  multiple of the typesize.  Although this is a very unusual situation,
  it can certainly happen (e.g.