Tiledb

Disk Format

* Update on-disk format because of the new available compressor for Dictionary-encoding of strings [3042](https://github.com/TileDB-Inc/TileDB/pull/3042)

New features

* Add virtual filesystem ls_with_sizes function [2971](https://github.com/TileDB-Inc/TileDB/pull/2971)
* Add new CMake build option for `TILEDB_EXPERIMENTAL_FEATURES` to compile time protect experimental features. [2748](https://github.com/TileDB-Inc/TileDB/pull/2748)
* Forwardport Group API [3058](https://github.com/TileDB-Inc/TileDB/pull/3058)
* Support Dictionary-encoding filter for string dimensions and attributes [3077](https://github.com/TileDB-Inc/TileDB/pull/3077)
* Use legacy sparse global order reader for 2.9 [3096](https://github.com/TileDB-Inc/TileDB/pull/3096)
* Add libmagic to build process. [3088](https://github.com/TileDB-Inc/TileDB/pull/3088)
* New file storage APIs (tiledb_filestore_...) [3121](https://github.com/TileDB-Inc/TileDB/pull/3121)

Improvements

* Convert FilterPipeline deserialize function to static factory function [2799](https://github.com/TileDB-Inc/TileDB/pull/2799)
* Convert array metadata deserialize function to factory function [2784](https://github.com/TileDB-Inc/TileDB/pull/2784)
* A new thread pool with modern C++ compatible API and exception-safe behavior. [2944](https://github.com/TileDB-Inc/TileDB/pull/2944)
* Smart pointer conversion: ArraySchema Domain [2948](https://github.com/TileDB-Inc/TileDB/pull/2948)
* Declare all C API functions `noexcept`. Put existing C API functions inside exception safety wrappers to meet the declaration change. [2961](https://github.com/TileDB-Inc/TileDB/pull/2961)
* Add support for compile-time assertion configuration [2962](https://github.com/TileDB-Inc/TileDB/pull/2962)
* Refactored tiledb::sm::serialization::attribute_from_capnp to be C41 compliant [2937](https://github.com/TileDB-Inc/TileDB/pull/2937)
* Smart pointer conversion: ArraySchema Dimension [2926](https://github.com/TileDB-Inc/TileDB/pull/2926)
* Refactored tiledb::sm::serialization::filter_pipeline_from_capnp to be C41 compliant [2943](https://github.com/TileDB-Inc/TileDB/pull/2943)
* Enable sparse global order reader by default. [2997](https://github.com/TileDB-Inc/TileDB/pull/2997)
* Add API for FragmentInfo::get_fragment_name [2977](https://github.com/TileDB-Inc/TileDB/pull/2977)
* add validity file format specification [2998](https://github.com/TileDB-Inc/TileDB/pull/2998)
* Convert Domain class deserialize function to factory function [2800](https://github.com/TileDB-Inc/TileDB/pull/2800)
* Dense reader: fix user buffer offset computation for multi-index queries. [3002](https://github.com/TileDB-Inc/TileDB/pull/3002)
* Sparse readers: using zipped coords buffers for fragment version < 5. [3016](https://github.com/TileDB-Inc/TileDB/pull/3016)
* Extra UTs on string RLEs [3024](https://github.com/TileDB-Inc/TileDB/pull/3024)
* Bump Catch2 version to 2.13.8 [3027](https://github.com/TileDB-Inc/TileDB/pull/3027)
* Split consolidator in multiple classes. [3004](https://github.com/TileDB-Inc/TileDB/pull/3004)
* HTML-render the existing format-spec Markdown docs. [3043](https://github.com/TileDB-Inc/TileDB/pull/3043)
* Add more detailed doc for schema evolution timestamp range functions. [3029](https://github.com/TileDB-Inc/TileDB/pull/3029)
* Run doc-render job on doc-only PRs, and not on non-doc PRs [3045](https://github.com/TileDB-Inc/TileDB/pull/3045)
* Support curl POSTing >2GB data to REST [3048](https://github.com/TileDB-Inc/TileDB/pull/3048)
* Dense reader: do not sort input ranges. [3036](https://github.com/TileDB-Inc/TileDB/pull/3036)
* Support consolidating non-contiguous fragments. [3037](https://github.com/TileDB-Inc/TileDB/pull/3037)
* Introduce dictionary-encoding as an enum option for filters [3042](https://github.com/TileDB-Inc/TileDB/pull/3042)
* Move `Range` to new `tiledb::type` namespace [3059](https://github.com/TileDB-Inc/TileDB/pull/3059)
* Convert tdb shared to shared [2965](https://github.com/TileDB-Inc/TileDB/pull/2965)
* Add `StatusException`, an exception class to be thrown instead of returning `Status` [3050](https://github.com/TileDB-Inc/TileDB/pull/3050)
* Cherry-pick 3061 [3064](https://github.com/TileDB-Inc/TileDB/pull/3064)
* Typo fix in group.cc [3078](https://github.com/TileDB-Inc/TileDB/pull/3078)
* Rename tiledb time.h/math.h to avoid possible conflicts with standard header files. [3087](https://github.com/TileDB-Inc/TileDB/pull/3087)
* Convert ArraySchema's deserialize to a factory function [3012](https://github.com/TileDB-Inc/TileDB/pull/3012)
* varying_size_datum_at: fixing comparison error. [3127](https://github.com/TileDB-Inc/TileDB/pull/3127)
* Global writes: check global order on write continuation. [3109](https://github.com/TileDB-Inc/TileDB/pull/3109)

Bug fixes

* [bug] Fix SC-17415: segfault due to underflow in for loop [3143](https://github.com/TileDB-Inc/TileDB/pull/3143)
* Sparse global order reader: prevent dims from being unfiltered twice. [3150](https://github.com/TileDB-Inc/TileDB/pull/3150)
* compare nullptr, avoid catch2 comparison warning failure [2970](https://github.com/TileDB-Inc/TileDB/pull/2970)
* Check that array is open before getting non_empty_domain [2980](https://github.com/TileDB-Inc/TileDB/pull/2980)
* Fix assertion failure in GCS, debug build [3001](https://github.com/TileDB-Inc/TileDB/pull/3001)
* Fix missing stats on cloud queries. [3009](https://github.com/TileDB-Inc/TileDB/pull/3009)
* Sparse unordered w/ dups reader: coord tiles management fix. [3023](https://github.com/TileDB-Inc/TileDB/pull/3023)
* Incorrect validity result count in REST query [3015](https://github.com/TileDB-Inc/TileDB/pull/3015)
* use different API approach to avoid possible file sharing violation [3056](https://github.com/TileDB-Inc/TileDB/pull/3056)
* avoid some potentially invalid vector references [2932](https://github.com/TileDB-Inc/TileDB/pull/2932)
* Sparse Global Order Reader Fix: Decrement Total Cells [3046](https://github.com/TileDB-Inc/TileDB/pull/3046)

API additions

C++ API
* Add function to check if Config contains a parameter [3082](https://github.com/TileDB-Inc/TileDB/pull/3082)

Bug fixes
* All ranges tile overlap: skip computation for default dimensions. [3080](https://github.com/TileDB-Inc/TileDB/pull/3080)
* Filter pipeline: fixing empty pipeline, multi chunk, refactored queries. [3149](https://github.com/TileDB-Inc/TileDB/pull/3149)
* Unordered writer: fixing segfault for empty writes. [3161](https://github.com/TileDB-Inc/TileDB/pull/3161)

Bug fixes

* Sparse unordered w/ dups reader: fix incomplete reason for cloud reads. [3104](https://github.com/TileDB-Inc/TileDB/pull/3104)

Improvements

* Add golang annotation to capnp spec file [3089](https://github.com/TileDB-Inc/TileDB/pull/3089)
* Update group metadata REST request to standardize cap'n proto class usage [3095](https://github.com/TileDB-Inc/TileDB/pull/3095)

Bug fixes

* Sparse Index Reader Fix: Check For Empty Buffer [3051](https://github.com/TileDB-Inc/TileDB/pull/3051)
* Reset group metadata only based on end timestamp to ensure its always reset to now [3091](https://github.com/TileDB-Inc/TileDB/pull/3091)

Disk Format

* Add Metadata to groups [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)
* Add Group on disk structure for members [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)


New features

* Support gs:// as an alias for gcs:// [2864](https://github.com/TileDB-Inc/TileDB/pull/2864)
* Eliminate LOG_FATAL use from codebase [2897](https://github.com/TileDB-Inc/TileDB/pull/2897)
* Collect missing docs [2922](https://github.com/TileDB-Inc/TileDB/pull/2922)
* Support `tiledb://` objects in the Object API [2954](https://github.com/TileDB-Inc/TileDB/pull/2954)
* RLE compression support for var-length string dimensions [2938](https://github.com/TileDB-Inc/TileDB/pull/2938)
* Add Metadata to groups [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)
* Add robust API to groups for adding and removing members of a group [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)

Improvements

* Support top-level cap'n proto array object [2844](https://github.com/TileDB-Inc/TileDB/pull/2844)
* Nicer error message for tiledb fragment listing [2872](https://github.com/TileDB-Inc/TileDB/pull/2872)
* Removing Buffer from Tile. [2852](https://github.com/TileDB-Inc/TileDB/pull/2852)
* Splitting Writer class into 3 separate classes. [2884](https://github.com/TileDB-Inc/TileDB/pull/2884)
* Adding a compressor algorithm for RLE encoding of strings [2857](https://github.com/TileDB-Inc/TileDB/pull/2857)
* Reader: treating empty string range as expected. [2883](https://github.com/TileDB-Inc/TileDB/pull/2883)
* Add a compression algorithm for dictionary encoding of strings [2880](https://github.com/TileDB-Inc/TileDB/pull/2880)
* Adds an ArrayDirectory class to manage all URIs within the array directory. [2909](https://github.com/TileDB-Inc/TileDB/pull/2909)
* Remove accidental addition of writer.cc. [2917](https://github.com/TileDB-Inc/TileDB/pull/2917)
* Tile metadata generator: code cleanup. [2919](https://github.com/TileDB-Inc/TileDB/pull/2919)
* Listing improvements: new directory structure for array. [2918](https://github.com/TileDB-Inc/TileDB/pull/2918)
* ArraySchema's Attribute smart pointer conversion [2887](https://github.com/TileDB-Inc/TileDB/pull/2887)
* Add option for tile level filtering [2906](https://github.com/TileDB-Inc/TileDB/pull/2906)
* Switch to smart pointers and const references for `ArraySchema`, and avoid fetching the latest array schema twice. [2923](https://github.com/TileDB-Inc/TileDB/pull/2923)
* Move vfs_helpers.cc and helper.cc into separate library with target that can be referenced elsewhere. [2929](https://github.com/TileDB-Inc/TileDB/pull/2929)
* Avoid calling `generate_uri` in `ArraySchema` accessors [2928](https://github.com/TileDB-Inc/TileDB/pull/2928)
* Global order writer: initialize last_var_offsets_. [2930](https://github.com/TileDB-Inc/TileDB/pull/2930)
* Wrap some C API functions with exception handlers. [2650](https://github.com/TileDB-Inc/TileDB/pull/2650)
* Fragment metadata: add min/max/sun/null count. [2934](https://github.com/TileDB-Inc/TileDB/pull/2934)
* Filter pipeline: incorrect stopping point during chunk parallellization. [2955](https://github.com/TileDB-Inc/TileDB/pull/2955)
* Adding support to consolidate ok/wrt files. [2933](https://github.com/TileDB-Inc/TileDB/pull/2933)
* Tile medatada: treating TILEDB_CHAR as TILEDB_STRING_ASCII. [2953](https://github.com/TileDB-Inc/TileDB/pull/2953)
* Fragment metadata: treating TILEDB_CHAR as TILEDB_STRING_ASCII. [2958](https://github.com/TileDB-Inc/TileDB/pull/2958)

* Global order writer: fixing multi writes for var size attributes. [2963](https://github.com/TileDB-Inc/TileDB/pull/2963)
* VFS: adding configuration for vfs.max_batch_size. [2960](https://github.com/TileDB-Inc/TileDB/pull/2960)
* Fixing build errors using MacOSX12.3.sdk. [2981](https://github.com/TileDB-Inc/TileDB/pull/2981)
* Support reading all consolidated fragment metadata files. [2973](https://github.com/TileDB-Inc/TileDB/pull/2973)
* Fixing compute_results_count_sparse_string for multiple range threads. [2983](https://github.com/TileDB-Inc/TileDB/pull/2983)
* Global writes: fixing OOM on write continuation. [2993](https://github.com/TileDB-Inc/TileDB/pull/2993)
* Do not store offsets when RLE is used on string dimensions [2969](https://github.com/TileDB-Inc/TileDB/pull/2969)
* Dynamically infer bytesizes for run length and strings for strings RLE compression [2984](https://github.com/TileDB-Inc/TileDB/pull/2984)
* Add ability to store (optional) name with group member [3068](https://github.com/TileDB-Inc/TileDB/pull/3068)

Bug fixes

* Avoid thread starvation by removing std::future usage in S3 multipart upload [2851](https://github.com/TileDB-Inc/TileDB/pull/2851)
* windows_sanity fix [2870](https://github.com/TileDB-Inc/TileDB/pull/2870)
* Adds missing pthreads link to dynamic memory unit test [2888](https://github.com/TileDB-Inc/TileDB/pull/2888)
* Remove `common.h` from `arrow_io_impl.h` [2915](https://github.com/TileDB-Inc/TileDB/pull/2915)
* Range::set_start and set_end should throw instead of empty returning [2913](https://github.com/TileDB-Inc/TileDB/pull/2913)
* Global writer: fixing write continuation for fixed sized attributes. [3062](https://github.com/TileDB-Inc/TileDB/pull/3062)
* `tiledb_serialize_array_metadata` should load metadata if its not loaded before serializing [3065](https://github.com/TileDB-Inc/TileDB/pull/3065)
* `tiledb_serialize_group_metadata` should load group metadata if its not loaded. [3070](https://github.com/TileDB-Inc/TileDB/pull/3070)

API additions

C API

* Introduce experimental `tiledb_ctx_alloc_with_error` to return error when context alloc fails [2905](https://github.com/TileDB-Inc/TileDB/pull/2905)
* Add `tiledb_group_*` APIs for robust group support [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)

C++ API

* Add missing cstddef include to fix compile w/ GCC 7 [2885](https://github.com/TileDB-Inc/TileDB/pull/2885)
* Add `Group::*` APIs for robust group support [2966](https://github.com/TileDB-Inc/TileDB/pull/2966)

Bug fixes

* Sparse unordered w/ dups reader: fixing memory management for tiles. [2924](https://github.com/TileDB-Inc/TileDB/pull/2924)