Changelogs » Arctic

PyUp Safety actively tracks 232,000 Python packages for vulnerabilities and notifies you when to upgrade.



* Feature: Allow list_library caching to be tunable via a settings collection.
  * Bugfix: 777: Handle empty columns in dataframes


* Bugfix: Pandas convert multiindexes to unicode if flag set.


* Bugfix: 765 Pandas data columns require encoding to convert to unicode.


* Bugfix: Use the default encoding (ascii) instead of bytes for unpickling.


* Bugfix: pypandoc not rendering README correctly for PYPI
  * Bugfix: 744 get_info on an empty dataframe raises an exception
  * Bugfix: 751 Handle unpickling py2 datetime and other object types with latin1
  * Bugfix: 598 Add an option to convert py2 str to py3 str (unicode type) on read.


* Bugfix: 712 Pandas deprecation warning in chunkstore serializer
  * BugFix: 670 Lots of pycodestyle fixes
  * BugFix: 691 Fix arrays_to_mgr for pandas 0.23.4+
  * BugFix: 698 Fix no handler in logging warning
  * BugFix: 705 Use sort_index instead of sortlevel_
  * BugFix: 710: Initialize SEGMENT_SHA in versions for writes and appends
  * Feature: 669 Experimental initial implementation of async arctic
  * Feature: 704 Do not enable sharding by default in BSONStore.
  * Feature: 688 Remove Chunkstore check for issue 422
  * Feature: 717 Add a best effort cache for list_libraries
  * Bugfix: 722 Exception while reading a column from chunkstore with a column missing in the chunk


* Bugfix: 658 Write/append errors for Panel objects from older pandas versions
  * Feature: 653 Add version meta-info in arctic module
  * Feature: 663 Include arctic numerical version in the metadata of the version document
  * Feature: 650 Implemented forward pointers for chunks in VersionStore (modes: enabled/disabled/hybrid)


* Feature: 577 Added implementation for incremental serializer for numpy records
  * Bugfix: 648 Fix issue with Timezone aware Pandas types, which don't contain hasobject attribute


* Bugfix: 645 Fix write errors for Pandas DataFrame that has mixed object/string types in multi-index column


* Bugfix: 157 Assure that serialized dataframes remain value-equivalent (e.g. avoid NaN --> 'nan' in mixed string columns)
  * Bugfix: 608 Ensure Arctic performs well with MongoDB 3.6 (sorting)
  * Bugfix: 629 Column kwarg no longer modified
  * Bugfix: 641 DateRange.intersection open/closed range fix
  * Feature: 493 Can pass kwargs when calling MongoClient, e.g. for ssl
  * Feature: 590 Faster write handler selection for DataFrames with objects
  * Feature: 604 Improved handling handling for pickling serialization decidions


* Docs: VersionStore documentation
  * Bugfix: Issue 612 ThreadPool should be created by process using it
  * Feature: Upsert option on appends in ChunkStore


* Feature: 553 Compatibility with both the new and old LZ4 API
  * Feature: 571 Removed the Cython LZ4 code, use the latest python-lz4
  * Feature: 557 Threadpool based compression. Speed imrpovement and tuning benchmarks.
  * Bugfix: fix tickstore unicode handling, support both unicode and utf-8 arrays
  * Bugfix: 591 Fix tickstore reads not returning index with localized timezone
  * Feature: 595 add host attribute to VersionedItem.
  * Bugfix: 594 Enable sharding on chunkstore


* Bugfix: 579 Fix symbol corruption due to restore_version and append
  * Bugfix: 584 Fix list_versions for a snapshot after deleting symbols in later versions


* Bugfix: 561 Fix PickleStore read corruption after write_metadata


* Bugfix: 168 Do not allow empty string as a column name
  * Bugfix: 483 Remove potential floating point error from datetime_to_ms
  * Bugfix: 271 Log when library doesnt exist on delete
  * Feature: MetaDataStore: added list_symbols with regexp, as_of and metadata fields matching filters
  * Feature: Support for serialization of DataFrames in Pandas 0.23.x


* Bugfix: 534 VersionStore: overwriting a symbol with different dtype (but same data format) does not
  raise exceptions anymore
  * Bugfix: 531 arctic_prune_versions: clean broken snapshot references before pruning
  * Bugfix: develop in a conda environment on Mac
  * Feature: 490 add support to numpy 1.14


* Bugfix: 521 Clang 6.0 compiler support on macOS
  * Feature: 510 VersionStore: support multi column in pandas DataFrames


* Bugfix: 517 VersionStore: append does not duplicate data in certain corner cases
  * Bugfix: 519 VersionStore: list_symbols speed improvement and fix for memory limit exceed


* Feature: 288 Mapping reads and writes over chunks in chunkstore
  * Bugfix: 508 VersionStore: list_symbols and read now always returns latest version
  * Bugfix: 512 Improved performance for list_versions
  * Bugfix: 515 VersionStore: _prune_previous_versions now retries the cleanup operation


* Bugfix: 503 ChunkStore: speedup check for -1 segments
  * Feature: 504 Increasing number of libraries in Arctic to 5000.


* Bugfix: Increase performance of invalid segment check in chunkstore
  * Bugfix: 501 Fix the spurious data integrity exceptions at write path, due to moving chunks form the balancer


* Bugfix: 491 roll back the use of frombuffer to fromstring, fixes the read-only ndarray issue


* Feature: 206 String support for tickstore
  * Bugfix: 486 improve mongo_retry robustness with failures for version store write/append


* Bugfix: 468 Re-adding compatibility for pandas 0.20.x
  * Bugfix: 476 Ensure we re-auth when a new MongoClient is created after fork


* Bugfix:  439 fix cursor timeouts in chunkstore iterator
  * Bugfix:  450 fix error in chunkstore delete when chunk range produces empty df
  * Bugfix:  442 fix incorrect segment values in multi segment chunks in chunkstore
  * Feature: 457 enchances fix for 442 via segment_id_repair tool
  * Bugfix:  385 exceptions during quota statistics no longer kill a write
  * Feature: PR161 TickStore.max_date now returns a datetime in the 'local' timezone
  * Feature: 425 user defined metadata for tickstore
  * Feature: 464 performance improvement by avoiding unnecessary re-authentication
  * Bugfix:  250 Added multiprocessing safety, check for initialized MongoClient after fork.
  * Feature: 465 Added fast operations for write only metadata and restore symbol to a version


* Bugfix:  440 Fix read empty MultiIndex+tz Series


* Perf:    408 Improve memory performance of version store's serializer
  * Bugfix   394 Multi symbol read in chunkstore
  * Bugfix:  407 Fix segment issue on appends in chunkstore
  * Bugfix:  Inconsistent returns on MetadataStore.append
  * Bugfix:  412 pandas deprecation and 289 improve exception report in numpy record serializer
  * Bugfix:  420 chunkstore ignoring open interval date ranges
  * Bugfix:  427 chunkstore metadata not being correctly replaced during symbol overwrite
  * Bugfix:  431 chunkstore iterators do not handle multi segment chunks correctly


* Bugfix:  397 Remove calls to deprecated methods in pymongo
  * Bugfix:  402 Append to empty DF fails in VersionStore


* Feature: 396 now supports as_of argument
  * Bugfix:  397 Pin pymongo==3.4.0


* Feature: 392 MetadataStore
  * Bugfix:  384 sentinels missing time data on chunk start/ends in ChunkStore
  * Bugfix:  382 Remove dependency on cython being pre-installed
  * Bugfix:  343 Renaming libraries/collections within a namespace/database


* BugFix: Rollback 363, as it breaks multi-index dataframe
  * Bugfix:  372 OSX build improvements


* Feature: Re-introduce 363 `concat` flag, essentially undo-ing 1.45
  * BugFix: 377 Fix broken `replace_one` on BSONStore and add `bulk_write`


* Feature: 374 Shard BSONStore on `_id` rather than `symbol`


* BugFix: Rollback 363, which can cause ordering issues on append


* Feature: 364 Expose compressHC from internal arctic LZ4 and remove external LZ4 dependency
  * Feature: 363 Appending older data (compare to what's exist in library) will raise. Use `concat=True` to append only the
  new bits
  * Feature: 371 Expose more functionality in BSONStore


* Bugfix:  350 remove deprecated pandas calls
  * Bugfix:  360 version incorrect in empty append in VersionStore
  * Feature: 365 add generic BSON store


* Bugfix: 346 fixed daterange subsetting error on very large dateframes in version store
  * Bugfix: 351 $size queries can't use indexes, use alternative queries


* Bugfix: 334 Chunk range param with pandas object fails in chunkstore.get_chunk_ranges
  * Bugfix: 339 Depending on lz4<=0.8.2 to fix build errors
  * Bugfix: 342 fixed compilation errors on Mac OSX
  * Bugfix: 344 fixed data corruption problem with concurrent appends


* BugFix: 330 Make Arctic._lock reentrant


* Feature:  329 Add reset() method to Arctic


* Bugfix:  324 Datetime indexes must be sorted in chunkstore
  * Feature: 290 improve performance of tickstore column reads


* Bugfix:  300 to_datetime deprecated in pandas, use to_pydatetime instead
  * Bugfix:  309 formatting change for DateRange __str__
  * Feature: 313 set and read user specified metadata in chunkstore
  * Feature: 319 Audit log support in ChunkStor
  * Bugfix:  216 Tickstore write fails with named index column


* Feature: Default to hashed based sharding
  * Bugfix: retry socket errors during VersionStore snapshot operations


* Bugfix:  296 Cannot compress/decompress empty string


* Feature: 294 Move per-chunk metadata for chunkstore to a separate collection
  * Bugfix:  292 Account for metadata size during size chunking in ChunkStore
  * Feature: 283 Support for all pandas frequency strings in ChunkStore DateChunker
  * Feature: 286 Add has_symbol to ChunkStore and support for partial symbol matching in list_symbols


* Feature:    275 Tuple range object support in DateChunker
  * Bugfix:     273 Duplicate columns breaking serializer
  * Feature:    267 Tickstore.delete returns deleted data
  * Dependency: 266 Remove pytest-dbfixtures in favor of pytest-server-fixtures


* Feature: 260 quota support on Chunkstore
  * Bugfix: 259 prevent write of unnamed columns/indexes
  * Bugfix: 252 pandas 0.19.0 compatibility fixes
  * Bugfix: 249 open ended range reads on data without index fail
  * Bugfix: 262 VersionStore.append must check data is written correctly during repack
  * Bugfix: 263 Quota: Improve the error message when near soft-quota limit
  * Perf:   265 VersionStore.write / append don't aggressively add indexes on each write


* Bugfix: 247 segmentation read fix in chunkstore
  * Feature: 243 add get_library_type method
  * Bugfix: more cython changes to handle LZ4 errors properly
  * Feature: 239 improve chunkstore's get_info method


* Feature: 235 method to return chunk ranges on a symbol in ChunkStore
  * Feature: 234 Iterator access to ChunkStore
  * Bugfix: 236 Cython not handling errors from LZ4 function calls


* Bugfix: 228 Mongo fail-over during append can leave a Version in an inconsistent state
  * Feature: 193 Support for different Chunkers and Serializers by symbol in ChunkStore
  * Feature: 220 Raise exception if older version of arctic attempts to read unsupported pickled data
  * Feature: 219 and 220 Support for pickling large data (>2GB)
  * Feature: 204 Add support for library renaming
  * Feature: 209 Upsert capability in ChunkStore's update method
  * Feature: 207 Support DatetimeIndexes in DateRange chunker
  * Bugfix:  232 Don't raise during VersionStore append(...) if the previous append failed


* Bugfix: 195 Top level tickstore write with list of dicts now works with timezone aware datetimes


* Bugfix: 187 Compatibility with latest version of pytest-dbfixtures
  * Feature: 182 Improve ChunkStore read/write performance
  * Feature: 162 Rename API for ChunkStore
  * Feature: 186 chunk_range on update
  * Bugfix: 189 range delete does not update symbol metadata


* Bugfix: Faster TickStore querying for multiple symbols simultaneously
  * Bugfix: now respects `allow_secondary=True`
  * Bugfix: 147 Add get_info method to ChunkStore
  * Bugfix: Periodically re-cache the library.quota to pick up any changes
  * Bugfix: 166 Add index on SHA for ChunkStore
  * Bugfix: 169 Dtype mismatch in chunkstore updates
  * Feature: 171 allow deleting of values within a date range in ChunkStore
  * Bugfix: 172 Fix date range bug when querying dates in the middle of chunks
  * Bugfix: 176 Fix overwrite failures in Chunkstore
  * Bugfix: 178 - Change how start/end dates are populated in the DB, also fix append so it works as expected.
  * Bugfix: 43 - Remove dependency on hardcoded Linux timezone files


* Bugfix: Ensure that Tickstore.write doesn't allow out of order messages
  * Bugfix: VersionStore.write now allows writing 'None' as a value


* Bugfix: Backwards compatibility reading/writing documents with previous versions of Arctic


* Bugfix: 109 Ensure stable sort during Arctic read
  * Feature: New benchmark suite using ASV
  * Bugfix: 129 Fixed an issue where some chunks could get skipped during a multiple-symbol TickStore read
  * Bugfix: 135 Fix issue with different datatype returned from pymongo in python3
  * Feature: 130 New Chunkstore storage type


* Bugfix: 106 Fix Pandas Panel storage for panels with different dimensions


* Feature: 98 Add initial_image as optional parameter on tickstore write()
  * Bugfix: 100 Write error on end field when writing with pandas dataframes


* Feature: Add python 3.3/3.4 support
  * Bugfix: 95 Fix raising NoDataFoundException across multiple low level libraries


* Bugfix: 81 Fix broken read of multi-index DataFrame written by old version of Arctic
  * Bugfix: 49 Fix strifying tickstore


* Feature: Add timezone suppport to store multi-index dataframes
  * Bugfix:  Fixed broken sdist releases


* Feature: ArticTransaction now supports non-audited 'transactions': `audit=False`
  with ArcticTransaction(Arctic('hostname')['some_library'], 'symbol', audit=False) as at:
  This is useful for batch jobs which read-modify-write and don't want to clash with
  concurrent writers, and which don't require keeping all versions of a symbol.


* Feature: get_info API added to version_store.




* Bugfix: correct version detection for Pandas >= 0.18.
  * Bugfix: retrying connection initialisation in case of an AutoReconnect failure.


* Bugfix: Improve performance of saving multi-index Pandas DataFrames
  by 9x
  * Bugfix: authenticate should propagate non-OperationFailure exceptions
  (e.g. ConnectionFailure) as this might be indicative of socket failures
  * Bugfix: return 'deleted' state in VersionStore.list_versions() so that
  callers can pick up on the head version being the delete-sentinel.


* Bugfix: could do the wrong thing with
  TimeZones (which aren't yet supported for date_range slicing.).


* Bugfix: fix authentication race condition when sharing an Arctic
  instance between multiple threads.


* Bugfix: compatibility with both 3.0 and pre-3.0 MongoDB for
  querying current authentications


* Feature: Add support for reading a subset of a pandas DataFrame
  in by passing in an
  * Bugfix: Reauth against admin if not auth'd against a library a
  specific library's DB.  Sometimes we appear to miss admin DB auths.
  This is to workaround that until we work out what the issue is.


* Feature: Add support for multi-index Bitemporal DataFrame storage.
  This allows persisting data and changes within the DataFrame making it
  easier to see how old data has been revised over time.
  * Bugfix: Ensure we call the error logging hook when exceptions occur


* Always use the primary cluster node for 'has_symbol()', it's safer


* Bugfixes for timezone handling, now ensures use of non-naive datetimes
  * Bugfix for tickstore read missing images


* Improvements to command-line control scripts for users and libraries
  * Bugfix for pickling top-level Arctic object


* Allow snapshotting a range of versions in the VersionStore, and
  snapshot all versions by default.


* Bugfix for backwards-compatible unpickling of bson-encoded data
  * Added switch for enabling parallel lz4 compression


*  Initial public release