Biothings

Highlights:
- New and updated tutorials and documentation
- CURIE id support in the annotation endpoint
- Added new jmespath_exclude_empty option
biothings.web improvements:
- Jmespath now handles list as parent_path field
biothings.hub improvements:
- Added Elasticsearch index selector
biothings.cli improvements:
- Fixed missing data plugin manifest template
documentation improvements:
- Added new tutorial for Biothings CLI
- Added tutorial for creating docker based data plugin
Misc improvements:
- Upgraded Tornado to 6.4.0
- Upgraded elasticsearch-py and elasticsearch-dsl to 8.x

Highlights:
- Implemented the new filter parameter for the query handler (296)
biothings.web improvements:
- Enabled post_filter parameter for POST queries (208)
- Improved jmespath transformation with better support root-level transformation
biothings.hub improvements:
- Allow passing and customizing merge_struct with the new inclusion and exclustion parameters
- Allow customization in Builder's merge order with the new merge_order method
- Fixed the deprecated import of networkx module
- Fixed the security issue when calling check_output https://github.com/biothings/biothings.api/pull/301
- Switched to use orjson in utils.dotfield module https://github.com/biothings/biothings.api/pull/300
- Changed default release date format in DockerContainerDumper

Bugfixes:
- 7e9583ec Fix dict default vals to work with dicts. (292)
- 77c577cc Break Circular Dependency (291)
- 7044abba Set plugin name to the folder name for Advanced plugins. (290)

Bugfixes:
- biothings.cli: fixed "logger is not defined" error when typer is not installed
- biothings.web: fixed an error when build_date is None during the metadata merging
- biothings.web: fixed an edge case when attempting to read ES6 mapping with doc_type

Highlights:
- New CLI feature for developing and testing data plugins https://github.com/newgene/biothings.api/pull/56
- New Docker Dumper based data plugins https://github.com/newgene/biothings.api/pull/83
- Support manifest-only date plugin (with a pre-defined parser) https://github.com/biothings/biothings.api/issues/259
- Support Python 3.11 and dropped Python 3.6 support

biothings.web improvements
- Support Google Analytics 4 https://github.com/newgene/biothings.api/pull/16
- Added a new `jmespath` query parameter for response transformation https://github.com/biothings/biothings.api/commit/31898fac7cd86b5c05520622885a3c0852f2494c
- Added date_format query parameter validation https://github.com/newgene/biothings.api/pull/97
- Added min for numeric query parameter validation (max is already supported) https://github.com/newgene/biothings.api/pull/97
- Fixed tornado.autoreload issue to support task-based reload hook https://github.com/newgene/biothings.api/pull/95
- Prevent biothing_type field in metadata from returning a single-item list https://github.com/biothings/biothings.api/issues/272
- Fixed the response formatter for histogram aggregation results https://github.com/biothings/biothings.api/issues/267
- Fixed an issue when only ES_INDEX is provided, not ES_DOC_TYPE https://github.com/biothings/biothings.api/commit/5c6d6dc6057073fb040dd7ae5b11b1b9967bf4f4
- Retired the use of pyinotify in autoreloader https://github.com/biothings/biothings.api/issues/260
- Logging all hanlders at server start https://github.com/biothings/biothings.api/commit/e9b3f9e6f8761746eb1e6f476dd9500b8bf8fe0e
- Adjusted sentry traces_sample_rate setting for better tracking tracebacks https://github.com/biothings/biothings.api/commit/eec03385e79394d1d99a7ed8537966ce2e995645

biothings.hub improvements
- New CLI feature for developing and testing data plugins https://github.com/newgene/biothings.api/pull/56
- New Docker Dumper based data plugins https://github.com/newgene/biothings.api/pull/83
- Support multiple target Elasticsearch hosts in data releases https://github.com/biothings/biothings.api/issues/251
- Allow to set AUTOHUB_VALIDATOR_CLASS to customize release validation logic https://github.com/newgene/biothings.api/pull/105
- Allow to override AutoHubFeature.validate_release method to add metadata validation https://github.com/biothings/biothings.api/issues/284
- Allow to manually mark a data source dump as success, e.g. manual downloads https://github.com/newgene/biothings.api/pull/101
- Verify existing snapshot config matches SNAPSHOT_CONFIG setting and allow to recreate if needed https://github.com/biothings/biothings.api/issues/275
- Support addtional filters (e.g. status) for the build list https://github.com/newgene/biothings.api/pull/94
- Implemented hub's Auto_Archive feature using AUTO_ARCHIVE_CONFIG https://github.com/biothings/biothings.api/issues/261
- Improved data inspection rules for validating field names and values https://github.com/newgene/biothings.api/pull/87
- Added a new re_index helper function https://github.com/biothings/biothings.api/issues/269
- Added a new helper function to generate local SSL certificate (e.g. to run dev on https) https://github.com/newgene/biothings.api/pull/84
- Allow Uploader.load_data method to be called with either data-folder or data-file path https://github.com/newgene/biothings.api/pull/81
- Support manifest-only date plugin (with a pre-defined parser) https://github.com/biothings/biothings.api/issues/259
- Support biothing_type and display_name in data plugin's manifest file https://github.com/biothings/biothings.api/issues/254
- Fixed cursor timeout issue in utils.mongo's doc_feeder helper function https://github.com/biothings/biothings.api/issues/282
- Fixed hub logging file names and returned sorting order https://github.com/newgene/biothings.api/pull/88
- Fixed CVE-2007-4559 when using tarfile module https://github.com/newgene/biothings.api/pull/74
- Fixed Autobuild is not correctly triggered when last build failed https://github.com/biothings/biothings.api/issues/271
- Replaced yapf with black as code formatter when exporting data plugins https://github.com/biothings/biothings.api/commit/fce0864b70802326d52ccc2dc71fb6fa1e4c1292

Experimental features
- Automatically clean up Elasticsearch snapshots via AUTO_SNAPSHOT_CLEANUP_CONFIG https://github.com/biothings/biothings.api/issues/276
- Allow to delete an existing snapshot https://github.com/newgene/biothings.api/pull/103

Misc improvements
- Added a new biothings.utils.common.DummyConfig class to create a temp config module https://github.com/newgene/biothings.api/pull/79
- Fixed all code-styling issue using fourmat (flake8 + black + isort)
- Print out the current test url and biothings version in BiothingsWebTest https://github.com/biothings/biothings.api/issues/273
- Added API documentation for biothings.hub module https://github.com/biothings/biothings.api/issues/265
- The hub commands documentation is now sorted by command names https://github.com/newgene/biothings.api/pull/76

Hub improvements:
- use pickle protocol 4 as the pickle.dump default
Hub bug fixes:
- Fixed a JSON serialization error during incremental release https://github.com/newgene/biothings.api/pull/65
- Resolved a hub error when installing a full release https://github.com/biothings/biothings.api/issues/257
- Fixed a quick_index error when a data source has multiple uploaders https://github.com/newgene/biothings.api/pull/66