Tiledb

Disk Format

Bump to version 19 (.vac vacuum files now use relative filenames). [4024](https://github.com/TileDB-Inc/TileDB/pull/4024)

API changes

C API

* C API dimension handle. [3947](https://github.com/TileDB-Inc/TileDB/pull/3947)
* Query Plan API. [3921](https://github.com/TileDB-Inc/TileDB/pull/3921)
* Error out no-op set_config calls, FragmentInfo. [3885](https://github.com/TileDB-Inc/TileDB/pull/3885)
* Error out no-op set_config calls, Array. [3892](https://github.com/TileDB-Inc/TileDB/pull/3892)
* Error out no-op set_config calls, Group. [3878](https://github.com/TileDB-Inc/TileDB/pull/3878)
* Error out no-op set_config calls, Query. [3884](https://github.com/TileDB-Inc/TileDB/pull/3884)

C++ API

* Add a `Group` constructor that accepts a config in the C++ API. [4010](https://github.com/TileDB-Inc/TileDB/pull/4010)
* Add `get_option` method to the `Filter` class that returns the option value. [4139](https://github.com/TileDB-Inc/TileDB/pull/4139)

New features

* Support in place update of group members. [3928](https://github.com/TileDB-Inc/TileDB/pull/3928)
* Query Condition NOT support. [3844](https://github.com/TileDB-Inc/TileDB/pull/3844)
* Add new Delta Encoder to filters. [3969](https://github.com/TileDB-Inc/TileDB/pull/3969)
* Support new `STSProfileWithWebIdentityCredentialsProvider` S3 credential provider for use of WebIdentity S3 credentials. [4137](https://github.com/TileDB-Inc/TileDB/pull/4137)
* Enable query condition on dimensions for dense arrays. [3887](https://github.com/TileDB-Inc/TileDB/pull/3887)

Configuration changes

* Add `vfs.azure.max_retries`, `vfs.azure.retry_delay_ms` and `vfs.azure.max_retry_delay_ms` options to control the retry policy when connecting to Azure. [4058](https://github.com/TileDB-Inc/TileDB/pull/4058)
* Remove the `vfs.azure.use_https` option; connecting to Azure with HTTP will require specifying the scheme in `vfs.azure.blob_endpoint` instead. [4058](https://github.com/TileDB-Inc/TileDB/pull/4058)
* Removed `sm.mem.reader.sparse_global_order.ratio_query_condition` config option. [3948](https://github.com/TileDB-Inc/TileDB/pull/3948)
* "vfs.s3.no_sign_request" to allow unsigned s3 API calls, useful for anonymous s3 bucket access. [3953](https://github.com/TileDB-Inc/TileDB/pull/3953)

Improvement

* Filter pipeline: make chunk range path default and remove other. [3857](https://github.com/TileDB-Inc/TileDB/pull/3857)
* Prevent combining query conditions with dimension label queries. [3901](https://github.com/TileDB-Inc/TileDB/pull/3901)
* Add attribute data order to attribute capnp. [3915](https://github.com/TileDB-Inc/TileDB/pull/3915)
* Dense reader: adjust memory configuration parameters. [3918](https://github.com/TileDB-Inc/TileDB/pull/3918)
* Move array_open_for_writes from StorageManager to Array. [3898](https://github.com/TileDB-Inc/TileDB/pull/3898)
* Refactor the POSIX VFS and improve error messages. [3849](https://github.com/TileDB-Inc/TileDB/pull/3849)
* Update ArraySchema Cap'n Proto serialization to include dimension labels. [3924](https://github.com/TileDB-Inc/TileDB/pull/3924)
* Refactor readers to use `std::optional<QueryCondition>`. [3907](https://github.com/TileDB-Inc/TileDB/pull/3907)
* Move schema loading functions from StorageManager to ArrayDirectory. [3906](https://github.com/TileDB-Inc/TileDB/pull/3906)
* Migrate BufferList to Handle-based API. [3911](https://github.com/TileDB-Inc/TileDB/pull/3911)
* Avoid storing a separate vector of tiles when loading groups. [3975](https://github.com/TileDB-Inc/TileDB/pull/3975)
* Sparse unordered w/ dups: overflow fix shouldn't include empty tile. [3985](https://github.com/TileDB-Inc/TileDB/pull/3985)
* Sparse unordered w/ dups: smaller units of work. [3948](https://github.com/TileDB-Inc/TileDB/pull/3948)
* Dense reader: read next batch of tiles as other get processed. [3965](https://github.com/TileDB-Inc/TileDB/pull/3965)
* Sparse global order reader: defer tile deletion until end of merge. [4014](https://github.com/TileDB-Inc/TileDB/pull/4014)
* Silence warnings about unqualified calls to std::move on Apple Clang. [4023](https://github.com/TileDB-Inc/TileDB/pull/4023)
* Enable serialization for partial attribute writes. [4021](https://github.com/TileDB-Inc/TileDB/pull/4021)
* Fix vac files to include relative file names. [4024](https://github.com/TileDB-Inc/TileDB/pull/4024)
* Update Subarray Cap'n Proto serialization to include label ranges. [3961](https://github.com/TileDB-Inc/TileDB/pull/3961)
* Fix uses of `noexcept` in the C API code. [4040](https://github.com/TileDB-Inc/TileDB/pull/4040)
* Sparse readers: fixing null count on incomplete queries. [4037](https://github.com/TileDB-Inc/TileDB/pull/4037)
* Sparse readers: adding memory budget class. [4042](https://github.com/TileDB-Inc/TileDB/pull/4042)
* Use the new Azure SDK for C++. [3910](https://github.com/TileDB-Inc/TileDB/pull/3910)
* [superbuild] Set Curl CA path on Darwin. [4059](https://github.com/TileDB-Inc/TileDB/pull/4059)
* Enable support for estimating result size on nullable, remote arrays. [4079](https://github.com/TileDB-Inc/TileDB/pull/4079)
* Fix intermittent failure in Group Metadata, delete. [4097](https://github.com/TileDB-Inc/TileDB/pull/4097)
* Add Subarray::add_range to core stats. [4045](https://github.com/TileDB-Inc/TileDB/pull/4045)
* Query.cc: Fix function ordering. [4129](https://github.com/TileDB-Inc/TileDB/pull/4129)
* Improvements to the Azure VFS. [4058](https://github.com/TileDB-Inc/TileDB/pull/4058)
* Array directory: parallelize large URI vector processing. [4133](https://github.com/TileDB-Inc/TileDB/pull/4133)
* Remove all TileDB artifacts when performing Array and Group deletes. [4081](https://github.com/TileDB-Inc/TileDB/pull/4081)

Defects removed

* Sparse global order reader: tile size error includes hilbert buffer. [3956](https://github.com/TileDB-Inc/TileDB/pull/3956)
* Sparse unordered w/ dups: fix error on double var size overflow. [3963](https://github.com/TileDB-Inc/TileDB/pull/3963)
* Dense reader: fix copies for schema evolution. [3970](https://github.com/TileDB-Inc/TileDB/pull/3970)
* Sparse global order reader: fix read progress update for duplicates. [3937](https://github.com/TileDB-Inc/TileDB/pull/3937)
* Fix check during remote unordered write finalization. [3903](https://github.com/TileDB-Inc/TileDB/pull/3903)
* Avoid narrowing conversion warning by downcasting computed values. [3931](https://github.com/TileDB-Inc/TileDB/pull/3931)
* Fix support for empty strings for Dictionary and RLE encodings. [3938](https://github.com/TileDB-Inc/TileDB/pull/3938)
* Check for failed allocation in `TileBase` constructor and throw if so. [3955](https://github.com/TileDB-Inc/TileDB/pull/3955)
* Fix logger initialization [3962](https://github.com/TileDB-Inc/TileDB/pull/3962)
* Sparse global order reader: deletes and overflow can give wrong results. [3983](https://github.com/TileDB-Inc/TileDB/pull/3983)
* Fix buffer size check in sparse unordered with duplicates reader during partial reads. [4027](https://github.com/TileDB-Inc/TileDB/pull/4027)
* Initialize MemFS in the VFS. [3438](https://github.com/TileDB-Inc/TileDB/pull/3438)
* Fix array/group metadata deletion when they are inserted in the same operation. [4022](https://github.com/TileDB-Inc/TileDB/pull/4022)
* Fix Query Conditions for Boolean Attributes. [4046](https://github.com/TileDB-Inc/TileDB/pull/4046)
* Allow empty AWS credentials for reading public S3 data. [4064](https://github.com/TileDB-Inc/TileDB/pull/4064)
* Serialize coalesce ranges for Subarray. [4043](https://github.com/TileDB-Inc/TileDB/pull/4043)
* Use `std::map::insert_or_assign` to replace values in maps. [4054](https://github.com/TileDB-Inc/TileDB/pull/4054)
* Adds serialization for webp filter options. [4085](https://github.com/TileDB-Inc/TileDB/pull/4085)
* Subarray serialize coalesce ranges as true by default. [4090](https://github.com/TileDB-Inc/TileDB/pull/4090)
* Don't throw an exception from the Group destructor. [4103](https://github.com/TileDB-Inc/TileDB/pull/4103)
* Fix type `and` to be `&&` in experimental group C++ API. [4106](https://github.com/TileDB-Inc/TileDB/pull/4106)
* Fix dense reader error when query conditions reference var sized fields. [4108](https://github.com/TileDB-Inc/TileDB/pull/4108)
* Remove `TILEDB_EXPORT` from private C API functions. [4120](https://github.com/TileDB-Inc/TileDB/pull/4120)
* Filter out empty s3 subfolders [4102](https://github.com/TileDB-Inc/TileDB/pull/4102)
* Fix REST Client buffer management [4135](https://github.com/TileDB-Inc/TileDB/pull/4135)
* Adds checks to ensure subarray is within domain bounds during initialization of DenseReader. [4088](https://github.com/TileDB-Inc/TileDB/pull/4088)
* Fix C++ `Filter::set_option` and `Filter::get_option` methods to accept `tiledb_datatype_t` as a valid option type. [4139](https://github.com/TileDB-Inc/TileDB/pull/4139)

Build system changes

* Add support for building dependencies with vcpkg ([3920](https://github.com/TileDB-Inc/TileDB/pull/3920), [#3986](https://github.com/TileDB-Inc/TileDB/pull/3986), [#3960](https://github.com/TileDB-Inc/TileDB/pull/3960), [#3950](https://github.com/TileDB-Inc/TileDB/pull/3950), [#4015](https://github.com/TileDB-Inc/TileDB/pull/4015), [#4128](https://github.com/TileDB-Inc/TileDB/pull/4128), [#4093](https://github.com/TileDB-Inc/TileDB/pull/4093), [#4055](https://github.com/TileDB-Inc/TileDB/pull/4055))
* Quiet catch2 super build by only printing on failure. [3926](https://github.com/TileDB-Inc/TileDB/pull/3926)
* Add .rc file containing version info to windows build. [3897](https://github.com/TileDB-Inc/TileDB/pull/3897)
* Use sccache in CI. [4016](https://github.com/TileDB-Inc/TileDB/pull/4016)
* Fix building aws-sdk-cpp on Ubuntu 22.04. [4092](https://github.com/TileDB-Inc/TileDB/pull/4092)

Test changes

* Use storage-testbench as the GCS emulator. [4132](https://github.com/TileDB-Inc/TileDB/pull/4132)
* Add unit tests for tiledb/common/platform.h [3936](https://github.com/TileDB-Inc/TileDB/pull/3936)
* Adjust a config buffer parameter for two tests for success in both 32bit and 64bit environments [3954](https://github.com/TileDB-Inc/TileDB/pull/3954)
* Add TEST_CASE blocks that check for possible regression in sparse global order reader [3933](https://github.com/TileDB-Inc/TileDB/pull/3933)
* Use temporary directories from the OS in tests. [3935](https://github.com/TileDB-Inc/TileDB/pull/3935)
* Adds workflow to trigger REST CI. [4123](https://github.com/TileDB-Inc/TileDB/pull/4123)

C++ library changes

* Introduce `synchronized_optional`, a multiprocessing-compatible version of `std::optional` [3457](https://github.com/TileDB-Inc/TileDB/pull/3457)
* Task graph filtering experimental example [3894](https://github.com/TileDB-Inc/TileDB/pull/3894)

Defects removed

* Fix dense reader error when query conditions reference var sized fields. [4108](https://github.com/TileDB-Inc/TileDB/pull/4108)
* Sparse readers: fixing null count on incomplete queries. [4037](https://github.com/TileDB-Inc/TileDB/pull/4037)
* Sparse unordered w/ dups: Fix incomplete queries w/ overlapping ranges. [4027](https://github.com/TileDB-Inc/TileDB/pull/4027)

Improvements

* Enable support for estimating result size on nullable, remote arrays [4079](https://github.com/TileDB-Inc/TileDB/pull/4079)

Defects removed

* Adds serialization for webp filter options [4085](https://github.com/TileDB-Inc/TileDB/pull/4085)
* Serialize coalesce ranges for Subarray [4043](https://github.com/TileDB-Inc/TileDB/pull/4043)

Defects removed

* Allow empty AWS credentials for anonymous access. [4064](https://github.com/TileDB-Inc/TileDB/pull/4064)
* Fix Query Conditions for Boolean Attributes. [4046](https://github.com/TileDB-Inc/TileDB/pull/4046)

Build

* Set Curl CA path on Darwin [4059](https://github.com/TileDB-Inc/TileDB/pull/4059)
* Fix invalid CMake syntax due to empty variable [4026](https://github.com/TileDB-Inc/TileDB/pull/4026)

New features

* Introduce "vfs.s3.no_sign_request" to allow unsigned s3 API calls, useful for anonymous s3 bucket access. [3953](https://github.com/TileDB-Inc/TileDB/pull/3953)

Defects removed
* Fix support for empty strings for Dictionary and RLE encodings [3938](https://github.com/TileDB-Inc/TileDB/pull/3938)
* Sparse global order reader: fix read progress update for duplicates. [3937](https://github.com/TileDB-Inc/TileDB/pull/3937)
* Sparse unordered w/ dups: fix error on double var size overflow. [3963](https://github.com/TileDB-Inc/TileDB/pull/3963)
* Dense reader: fix copies for schema evolution. [3970](https://github.com/TileDB-Inc/TileDB/pull/3970)
* Sparse unordered w/ dups: overflow fix shouldn't include empty tile. [3985](https://github.com/TileDB-Inc/TileDB/pull/3985)

API changes

C++ API

* Add a `Group` constructor that accepts a Config in the C++ API. [4011](https://github.com/TileDB-Inc/TileDB/pull/4011)

Disk Format

* Add FloatScaleFilter to format spec (documentation only; filter released in 2.11.0) [3494](https://github.com/TileDB-Inc/TileDB/pull/3494)

Breaking C API changes

* Remove deprecated `Query` methods. [3841](https://github.com/TileDB-Inc/TileDB/pull/3841)
* Remove deprecated API functions about buffers [3733](https://github.com/TileDB-Inc/TileDB/pull/3733)
* Remove deprecated C API `tiledb_fragment_info_load_with_key` [3740](https://github.com/TileDB-Inc/TileDB/pull/3740)
* Remove extra experimental set buffer methods [3761](https://github.com/TileDB-Inc/TileDB/pull/3761)
* Remove C API functions `tiledb_array_open_at` and variations. [3755](https://github.com/TileDB-Inc/TileDB/pull/3755)
* Remove C API functions `tiledb_array_consolidate_metadata` and `tiledb_array_consolidate_metadata_with_key` [3742](https://github.com/TileDB-Inc/TileDB/pull/3742)
* Remove deprecated C API function `tiledb_coords`. [3743](https://github.com/TileDB-Inc/TileDB/pull/3743)
* Remove bitsort filter (feature-flagged filter). [3852](https://github.com/TileDB-Inc/TileDB/pull/3852)

Breaking behavior

* An error is now raised when setting a subarray on a query that is already initialized. [3668](https://github.com/TileDB-Inc/TileDB/pull/3668)
* An error is now raised when setting a subarray on a write to a sparse array. [3668](https://github.com/TileDB-Inc/TileDB/pull/3668)

New features

* Partial attribute writes into a single fragment. [3714](https://github.com/TileDB-Inc/TileDB/pull/3714)
* Query condition support for TIME types [3784](https://github.com/TileDB-Inc/TileDB/pull/3784)
* Enable dimension labels for the C API (provisional) [3824](https://github.com/TileDB-Inc/TileDB/pull/3824)
* Add dimension label C++ API (provisional) [3839](https://github.com/TileDB-Inc/TileDB/pull/3839)

Improvements

* Add creation of ArrayDirectory on open to stats [3769](https://github.com/TileDB-Inc/TileDB/pull/3769)
* Return underlying errors in Win::remove_dir [3866](https://github.com/TileDB-Inc/TileDB/pull/3866)
* Removes tile-aligned restriction for remote global order writes by caching tile overflow data from submissions. [3762](https://github.com/TileDB-Inc/TileDB/pull/3762)
* Implement S3 buffering support for remote global order writes [3609](https://github.com/TileDB-Inc/TileDB/pull/3609)
* Add support for dimension labels on an encrypted array [3774](https://github.com/TileDB-Inc/TileDB/pull/3774)
* Refactor relevant fragments into a separate class. [3738](https://github.com/TileDB-Inc/TileDB/pull/3738)
* Implement duration instrument for measuring times in stats. [3746](https://github.com/TileDB-Inc/TileDB/pull/3746)
* Move attribute order check to writer base [3748](https://github.com/TileDB-Inc/TileDB/pull/3748)
* Fragment consolidator: use average cell size for buffer allocation. [3756](https://github.com/TileDB-Inc/TileDB/pull/3756)
* Show URL in logger trace before results are returned [3745](https://github.com/TileDB-Inc/TileDB/pull/3745)
* Refactor tiledb_set_subarray call in test/support/src/helpers.cc [3776](https://github.com/TileDB-Inc/TileDB/pull/3776)
* Move array open methods from StorageManager to Array [3790](https://github.com/TileDB-Inc/TileDB/pull/3790)
* Split Tile class into different classes for read and write. [3796](https://github.com/TileDB-Inc/TileDB/pull/3796)
* Update ArrayDirectory to use ContextResources [3800](https://github.com/TileDB-Inc/TileDB/pull/3800)
* Use `EncryptionType::NO_ENCRYPTION` instead of casting the C enum. [3545](https://github.com/TileDB-Inc/TileDB/pull/3545)
* Tile metadata generator: fix buffer overflow on string comparison. [3821](https://github.com/TileDB-Inc/TileDB/pull/3821)
* Ordered writer: process next tile batch while waiting on write. [3797](https://github.com/TileDB-Inc/TileDB/pull/3797)
* Query v3: Reduce array open operations on Cloud [3626](https://github.com/TileDB-Inc/TileDB/pull/3626)
* Read tiles: get rid of extra VFS allocation. [3848](https://github.com/TileDB-Inc/TileDB/pull/3848)
* Clean-up `QueryBuffer` and move it away from statuses. [3840](https://github.com/TileDB-Inc/TileDB/pull/3840)
* Storage manager: remove passthrough read/write functions. [3853](https://github.com/TileDB-Inc/TileDB/pull/3853)
* Fix partial attribute write test. [3862](https://github.com/TileDB-Inc/TileDB/pull/3862)
* Query v2: Add array directory and fragment meta ser/deser behind config flag [3845](https://github.com/TileDB-Inc/TileDB/pull/3845)
* Ordered dimension label reader: handle empty array. [3869](https://github.com/TileDB-Inc/TileDB/pull/3869)
* Fix comments in nullable attributes example. [3870](https://github.com/TileDB-Inc/TileDB/pull/3870)
* RLE and dictionary filter only enabled for UTF8 since format version 17. [3868](https://github.com/TileDB-Inc/TileDB/pull/3868)
* Move more resources into ContextResources [3807](https://github.com/TileDB-Inc/TileDB/pull/3807)
* Better errors for failed dimension label queries. [3872](https://github.com/TileDB-Inc/TileDB/pull/3872)
* Dense reader: process smaller units of work. [3856](https://github.com/TileDB-Inc/TileDB/pull/3856)
* Use different config variable for enabling open v2 and query v3 [3879](https://github.com/TileDB-Inc/TileDB/pull/3879)
* Fragment consolidation: using correct buffer weights. [3877](https://github.com/TileDB-Inc/TileDB/pull/3877)
* Show expected and actual version numbers in the error message [3855](https://github.com/TileDB-Inc/TileDB/pull/3855)
* Fix shallow test-only bug in unit_rtree.cc [3830](https://github.com/TileDB-Inc/TileDB/pull/3830)
* Fix stack buffer overflow bug in unit-curl.cc [3832](https://github.com/TileDB-Inc/TileDB/pull/3832)
* Fix stack use after free bug in unit_thread_pool [3831](https://github.com/TileDB-Inc/TileDB/pull/3831)

Defects removed

* Fail if fragment info objects are accessed before loading them. [3846](https://github.com/TileDB-Inc/TileDB/pull/3846)
* Let ConsolidationPlan::dump() produce valid JSON [3751](https://github.com/TileDB-Inc/TileDB/pull/3751)
* Fix exceptions thrown during array schema validation for WebP filter. [3752](https://github.com/TileDB-Inc/TileDB/pull/3752)
* Modify GlobalStats::reset() to support compensation for registered stats being effectively leaked. [3723](https://github.com/TileDB-Inc/TileDB/pull/3723)
* Avoid pwrite bug on macOS Ventura 13.0 on Apple M1 [3799](https://github.com/TileDB-Inc/TileDB/pull/3799)
* Fix ASAN-detected UAF in sparse global order reader [3822](https://github.com/TileDB-Inc/TileDB/pull/3822)
* Improve WebP validation, fixes SC-24766, SC-24759 [3819](https://github.com/TileDB-Inc/TileDB/pull/3819)
* Deregister a remote array from the Consistency multimap before reopening. [3859](https://github.com/TileDB-Inc/TileDB/pull/3859)
* Fix buffer size error in deserializing query [3851](https://github.com/TileDB-Inc/TileDB/pull/3851)
* Fix segfault in unit_consistency [3880](https://github.com/TileDB-Inc/TileDB/pull/3880)
* Avoid segfault in unit_array [3883](https://github.com/TileDB-Inc/TileDB/pull/3883)
* VFS CAPIHandle class [3523](https://github.com/TileDB-Inc/TileDB/pull/3523)

API changes

C API

* Deprecate tiledb_query_submit_async [3802](https://github.com/TileDB-Inc/TileDB/pull/3802)
* Deprecate `tiledb_fragment_info_get_name`. [3791](https://github.com/TileDB-Inc/TileDB/pull/3791)
* Deprecate `tiledb_query_submit_async`. [3802](https://github.com/TileDB-Inc/TileDB/pull/3802)
* Add `tiledb_array_delete_fragments_list` API [3798](https://github.com/TileDB-Inc/TileDB/pull/3798)
* Add `tiledb_string_handle_t` so that C API functions can output strings with independent lifespans. [3792](https://github.com/TileDB-Inc/TileDB/pull/3792)
* Add `tiledb_dimension_label_handle_t` and `tiledb_dimension_label_t` handles to C-API [3820](https://github.com/TileDB-Inc/TileDB/pull/3820)
* Add `tiledb_subarray_has_label_ranges` and `tiledb_subarray_get_label_name`. [3858](https://github.com/TileDB-Inc/TileDB/pull/3858)
* Add `tiledb_fragment_info_get_fragment_name_v2`. [3842](https://github.com/TileDB-Inc/TileDB/pull/3842)
* Add function to access the dimension label attribute name. [3867](https://github.com/TileDB-Inc/TileDB/pull/3867)
* Change query set buffer methods to set label buffers in experimental builds [3761](https://github.com/TileDB-Inc/TileDB/pull/3761)

C++ API

* Move deprecated constructors into new array_deprecated.h file, manipulating them to use the new TemporalPolicy and EncryptionAlgorithm classes. [3854](https://github.com/TileDB-Inc/TileDB/pull/3854)
* Add function to access the dimension label attribute name. [3867](https://github.com/TileDB-Inc/TileDB/pull/3867)
* Add experimental `set_data_buffer` API that handles dimension labels [3882](https://github.com/TileDB-Inc/TileDB/pull/3882)

Build system changes

* Set ASAN options for TileDB [3843](https://github.com/TileDB-Inc/TileDB/pull/3843)
* Allow building TileDB with GNU GCC on MacOS [3779](https://github.com/TileDB-Inc/TileDB/pull/3779)

Experimental features

* Set of three exploratory but fully consistent schedulers for TileDB task graph library. [3683](https://github.com/TileDB-Inc/TileDB/pull/3683)
* Version 0.1 of specification task graph. [3754](https://github.com/TileDB-Inc/TileDB/pull/3754)

Full Changelog:

* https://github.com/TileDB-Inc/TileDB/compare/2.14.0...2.15.0