Apache-iotdb

New Features

* [IOTDB-4572] support order by time in align by device
* [IOTDB-4816] Support Show Queries command
* [IOTDB-4817] Support kill query command
* [IOTDB-5077] Support new command formats in SHOW REGIONS
* [IOTDB-5108] Support region migration
* [IOTDB-5202] Support show regions with specific database
* [IOTDB-5282] Add SQL show variables, which can display the current cluster variables
* [IOTDB-5341] Support GROUP BY VARIATION in aggregation query
* [IOTDB-5372] Support data type cast in SELECT INTO
* [IOTDB-5382] Support DIFF as built-in scalar function in IoTDB
* [IOTDB-5393] Show Region creation time when execute show regions
* [IOTDB-5456] Implement COUNT_IF built-in aggregation function
* [IOTDB-5515] Support GROUP BY CONDITION in aggregation query
* [IOTDB-5555] Enable modify dn_rpc_port and dn_rpc_address in datanode-engine.properties
* Support docker deployment
* Rename SeriesSlotId to SeriesSlotNum in show regions

Improvements

* [IOTDB-4497] Improve NodeStatus definition
* [IOTDB-5066] Upgraded the GetSlots SQLs
* [IOTDB-5161] Add output type check for WHERE & HAVING clause and refuse expressions whose return type are not boolean exist in WHERE & HAVING clause
* [IOTDB-5185] Fixed that old snapshot is not deleted in IoTConsensus
* [IOTDB-5287] Added status “Discouraged” to RegionGroup
* [IOTDB-5449] Wait for query resource while the query queue is full instead of throwing exception directly
* change STARTUP_RETRY_INTERVAL_IN_MS from 30s to 3s

Bug Fixes

* [IOTDB-4684] Fix devices with the same name but different alignment properties are compacted into the wrong alignment property
* [IOTDB-5061] Add initialize state check when create snapshot
* [IOTDB-5090] Fix the NPE when executing stop-datanode.sh
* [IOTDB-5112] Fixed IoTConsensus synchronization stuck under low load or during restart
* [IOTDB-5126] Fix that show-regions show IP rather than hostname even though the hostname is used when registering
* [IOTDB-5165] Fix fail to pass compaction validation when resources degrade to FileTimeIndex
* [IOTDB-5170] Fix the problem that datanode is closed when executing stop-confignode.bat on Windows platform
* [IOTDB-5199] Fix NPE in StorageExector inLoading process
* [IOTDB-5216] Fix order by timeseries doesn't take effect in aligned last query
* [IOTDB-5228] NPE if the file does not exist while creating TsFileSequenceReader instance
* [IOTDB-5240] Fix schema template timeseries read/write error after restart
* [IOTDB-5245] Fix data is incomplete in last query
* [IOTDB-5277] Fix DataNode restart failure when SchemaRegion loading snapshot
* [IOTDB-5278] Fix JDBC Driver cannot connect to the dbeaver
* [IOTDB-5285] Fix TimePartition error when restarting with different time partition configuration
* [IOTDB-5269] Fix wrong data of devices ought to be deleted after executing `delete from` sql
* [IOTDB-5324] Fix wal cann't be deleted in destDataNode after region migration when data_replication_factor is 1 in IoTConsensus
* [IOTDB-5389] Cause DataNode startup to fail when wal_mode is disabled in IoTConsensus
* [IOTDB-5414] Fix timeseries with alias deleted success but can still be queried by alias
* [IOTDB-5426] Cannot trigger flush for sequence file when timed flush enabled
* [IOTDB-5441] Fix NPE while fetch schema that is not in template used by related device
* [IOTDB-5469] Fix get schema info failure after creating template with backquote characters successfully
* [IOTDB-5474] Fix count nodes using level bug
* [IOTDB-5480] IoTConsensus sync lag may be negative under single copy
* [IOTDB-5488] Support set system to readonly on local
* [IOTDB-5492] Skip broken tsfile when recovering system
* [IOTDB-5512] Fixed IoTConsensus may repeatedly send some log when restarting
* [IOTDB-5516] Fix creating timeseries failure after dropping all databases
* [IOTDB-5526] Fix deleting timeseries failure when it belongs to a device activating template
* [IOTDB-5548] Fix dropping template failure when using pure number as template name
* [IOTDB-5592] Fix unexpected error while using full path(starting with root) in having/where clause
* [IOTDB-5620] Fix flush stuck when there is a lot of time partitions in each DataRegion
* [IOTDB-5639] Fix file not found exception in cross space compaction selector
* [IOTDB-5657] Fix LIMIT&OFFSET does not take effect in last query
* [IOTDB-5684] Fix log folder generation when using ConfigNode's Simple consensus protocol
* [IOTDB-5685] Fix error msg of failing to create a timeseries on an existing path when ReadOnly state
* [IOTDB-5686] Fix devices with the same name but different alignment properties meets error in inner seq compaction
* [IOTDB-5700] Clean temporary files created by UDF query after it finishes
* Add SHOW_CQ privilege

New Features

* [IOTDB-5102] Support explain the sqls using align by device clause
* [IOTDB-5282] Add SQL show variables, which can display the current cluster variables

Improvements

* [IOTDB-4595] Add real-time monitoring data for tsfile merge
* [IOTDB-5125] Stop the process when port is already in use when launching ConfigNode or DataNode
* [IOTDB-5185] Fixed that old snapshot is not deleted in IoTConsensus
* [IOTDB-5202] Support showing regions regarding database
* [IOTDB-5205] Construct Cluster Node startup protocol, reject start or restart request of ConfigNode or DataNode when using wrong data
* [IOTDB-5208] Fix increase of file handles when TsFileResource is degraded and compacted with fast performer
* [IOTDB-5223] Cause DataNode startup to fail when multiple disks are configured in the data directory and the dataRegion consensus protocol uses Ratis
* [IOTDB-5232] Add Schema Memory Usage metric monitor
* [IOTDB-5249] Add interface to set datatype when users customize the payload formatter of mqtt
* [IOTDB-5258] Add timeout parameter for ExportCSV/ExportTsFile Tool
* [IOTDB-5306] Change the default port of ConfigNode and DataNode
* [IOTDB-5326] Add cluster_name parameter to distinguish different cluster
* [IOTDB-5389] Cause DataNode starup to fail when wal_mode is disabled in IoTConsensus

Bug Fixes

* [IOTDB-4978] Fix possible deadLock while using select into
* [IOTDB-5090] Fix the NPE when executing stop-datanode.sh
* [IOTDB-5116] Fix wrong empty result set in aggregation query while there is no data in queried timeseries
* [IOTDB-5118] Fix that datanode opens too many file handles （TYPE DIR）
* [IOTDB-5174] Fixed restart failure when IoTConsensus hostname containing '-'
* [IOTDB-5126] Fix that show-regions show IP rather than hostname even though the hostname is used when registering
* [IOTDB-5135] Fix the Sync snapshot failure (also blocks ConfigNode snapshot) when sync file is empty
* [IOTDB-5141] Fix fast compaction scheduler bug
* [IOTDB-5144] Fix classCastException in IoTDBDatabaseMetadata of jdbc
* [IOTDB-5158] Fix failure of getting devices from tsfile
* [IOTDB-5170] Fix the problem that datanode is closed when executing stop-confignode.bat on Windows platform
* [IOTDB-5210] Fix concurrent bug caused by init and close of driver
* [IOTDB-5212] Upgrade an illegal CQ verification prompt from v0.14.0 to v1.0
* [IOTDB-5215] Fix the wrong usage of chunkPointNumLowerBound in IoTDBDescriptor
* [IOTDB-5216] Fix order by timeseries doesn't take effect in aligned last query
* [IOTDB-5222] Fix the load of some metric related configuration
* [IOTDB-5231] Fix that Datanode could not start when binding 9091 error
* [IOTDB-5233] Correct the message of unsupported thrift interface of schema template
* [IOTDB-5236] Fix DeleteData Python API cannot work
* [IOTDB-5240] Fix schema template timeseries read/write error after restart
* [IOTDB-5261] Fix the number of clients in client pool of AsyncIoTConsensusServiceClient can not be changed through dn_max_connection_for_internal_service
* [IOTDB-5266] Refine the code of cross selector and fix the bug when using FileTImeIndex
* [IOTDB-5269] Fix wrong data of devices ought to be deleted after executing `delete from` sql
* [IOTDB-5277] Fix DataNode restart failure when SchemaRegion loading snapshot
* [IOTDB-5278] Fix JDBC Driver cannot connect to the dbeaver
* [IOTDB-5285] Fix TimePartition error when restarting with different time partition configuration
* [IOTDB-5286] Fix target files are selected to participate in other compaction task
* [IOTDB-5313] Fix Python client doesn't handle REDIRECTION_RECOMMEND correctly
* [IOTDB-5452] Add leastDataRegionGroupNumSetByUser examination to ensure that leastDataRegionGroupNum will not be changed by system
* [IOTDB-5498] Fix SessionPool OOM problem when the number of devices and sessions is large
* [IOTDB-5501] Fix memory leak in MemoryPool
* [IOTDB-5525] Fix the partition error of device after partition fetcher

New Features

* New architecture that supports standalone and cluster mode with two types of nodes: ConfigNode, DataNode
* Support ConfigNode management: Start/Add, Stop, Remove
* Support DataNode management: Start/Add, Stop, Remove
* Support replication of ConfigNode, Schema and Data
* Support Consensus Protocol: Simple, IoT, Ratis
* Support Cluster management sql: show cluster, show regions
* Support administration in Cluster: User, Permission, Role management
* Authorization when login and executing a command
* Support MPP(massively parallel process) framework in cluster
* Support insertion, deletion and all query types in Cluster
* Support CSV import/export tools
* Support TsFile import/export tools
* Support create/show/count/delete database
* Support show/count devices
* Support create/show/count timeseries
* Support schema template management
* Support flush on local/cluster
* Support clear cache on local/cluster
* Support metric for DataNode and ConfigNode with output to IoTDB, Prometheus, and JMX
* Support DBAPI in python client
* Support RestApi, MQTT for cluster
* Support having clause and between expression in query
* Support order by timeseries in last query
* Support hot configuration of data_dirs
* Support stateful, stateless Trigger
* Support UDF framework
* Support new UDF 'change_points
* Support Continuous Query

The 0.14.0-preview1 version only contains the new cluster version.
New Features

- The new cluster contains two types of nodes: ConfigNode, DataNode
- Support ConfigNode Deploy: Start(Add), Stop
- Support DataNode Deploy: Start(Add), Stop
- Data and Schema replication management
- Support Consensus Protocol: Standalone, MultiLeader, Ratis
- Schema Management: Memory, SchemaFile, RocksDB
- Cluster monitor sql: show cluster, show regions
- Administration: User, Permission, Role management
- Authorization when login and executing a command
- Load Balance for data and schema
- Native API for reading and writing
- Support CSV import/export tools
- Support create/show/count/delete Storage Group
- Support create/show/count timeseries
- Support show/count devices
- Flush is supported in cluster
- Support monitor DataNode and ConfigNode through Prometheus and Grafana
- Support Data insertion and deletion
- Query Types: Raw data Query, Aggregation, Group By Time, Group By Level, Last query
- Support Fill null values in the query result
- Support order by time and limit, offset, slimit, soffset
- Support value filter and expression
- Python API: Add support for DBAPI


Incompatible changes
- Rename the following scripts and configuration files

| Previous | Now |
|------------------------------|--------------------------------|
| conf/iotdb-engine.properties | conf/iotdb-datanode.properties |
| conf/iotdb-env.sh & bat | conf/datanode-env.sh & bat |

- Rename the DataNode configuration parameters

| Previous | Now |
|----------------------------------|---------------------|
| internal_ip | internal_address |
| target_confignode / config_nodes | target_config_nodes |

- Rename the ConfigNode configuration parameters

| Previous | Now |
|----------------------------------|---------------------|
| target_confignode / config_nodes | target_config_nodes |

Acknowledgements:

543202718, 23931017wu, Alima777, Beyyes, Bigreybear, cchen1115, chaow, Chengjianyun, ChengShengHan, ChenSiHou, Choubenson, chuchaofan, ChuShaoHuan, Cigarl, CloudWise-Lukemiao, Cmlmakahts, Cornmonster, Cpaulzy, CRZbulabula, dafei1288, Ericpai, Flashzxi, FrankHWD, greenhandatsjtu, HeimingZ, HTHou, huanhuanerer, huzk8, irvine0109, iskytek, JackieTien97, jamber001, JerryZhangZZY, jiazhiren, jixuan1989, jun0315, Krystal-xiao, leety1228, leonardodalinky, Liuminghui233, LIU-WEI-git, ljn55966005, lyssom, MarosZyk, MinaQin, MrQuanzy, Mychaow, Noorall, OneSizeFitsQuorum, Plutooooooo, Qiaojialin, RYH61, Samperson1997, SilverNarcissus, SpriCoder, SteveYurongSu, swpulk, SzyWilliam, THUMarkLau, tisonkun, trin1t, Tsunghanjacktsai, wallezhang, wangchao316, wanghui42, wayblink, Wei-hao-Li, Xiaoyehanren, Xingtanzjr, Xinzhongtianxia, Xuanronaldo, Yifuzhou, Yschengzi, Yuyuankang, yyt86, ZhanGHanG9991, zjx1230