Dpgen

What's Changed

Breaking change

* breaking: use `shlex.quote` to quote shell commands by njzjz in https://github.com/deepmodeling/dpgen/pull/1396
Single quotes (') can be automatically escaped. One does not need to escape single quotes in command manually anymore. To those who have escaped single quotes in command manually, this is a breaking change.

Enhancement
* exclude docs and tests from sdist by njzjz in https://github.com/deepmodeling/dpgen/pull/1385
* tests: add UT for run_model_devi by njzjz in https://github.com/deepmodeling/dpgen/pull/1386
* throw error if training_iter0_model_path is not found by njzjz in https://github.com/deepmodeling/dpgen/pull/1391
* pwscf: write floaing point number for atomic masses by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1387
* model devi: download LAMMPS log files for pimd by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1406
* only log the current iteration when restarting by njzjz in https://github.com/deepmodeling/dpgen/pull/1420
* support collecting MultiSystems by njzjz in https://github.com/deepmodeling/dpgen/pull/1422
* keep original model_devi.out files for pimd by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1472

Documentation
* docs: add links to external documentation by njzjz in https://github.com/deepmodeling/dpgen/pull/1392
* docs: add the link for how to setup the conda environment by njzjz in https://github.com/deepmodeling/dpgen/pull/1484
* fp: add document and example for Quantum Espresso by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1405
* dpgen simplify: add document for fp style pwscf by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1413

CI/CD
* CI: Setup Dependabot for GitHub Actions by njzjz-bot in https://github.com/deepmodeling/dpgen/pull/1492
* CI: Bump codecov/codecov-action from v3 to v4 by njzjz-bot in https://github.com/deepmodeling/dpgen/pull/1491

Bugfix

* Fix UTs destroyed by pymatgen's update by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1471
* Bugfix for pimd: sorting model_devi files by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1470
* issue1475: a small bug of iterdict (generator/lib/cp2k.py) by yohsama in https://github.com/deepmodeling/dpgen/pull/1476

New Contributors
* yohsama made their first contribution in https://github.com/deepmodeling/dpgen/pull/1476
* njzjz-bot made their first contribution in https://github.com/deepmodeling/dpgen/pull/1492

**Full Changelog**: https://github.com/deepmodeling/dpgen/compare/v0.12.0...v0.12.1

Breaking Changes
* drop Python 3.8 support by njzjz in https://github.com/deepmodeling/dpgen/pull/1264
* breaking: remove default values for training reuse arguments by njzjz in https://github.com/deepmodeling/dpgen/pull/1330

New features
* add dp_train_skip_neighbor_stat argument by njzjz in https://github.com/deepmodeling/dpgen/pull/1179
* support `mixed:N` batch size by njzjz in https://github.com/deepmodeling/dpgen/pull/1183
* support one_h5 for simplify model_devi step by njzjz in https://github.com/deepmodeling/dpgen/pull/1185
* simplify: allow initial models trained by initial data by njzjz in https://github.com/deepmodeling/dpgen/pull/1201
* support automatic training_reuse_old_ratio by njzjz in https://github.com/deepmodeling/dpgen/pull/1209
* split make_fp and share `make_fp_calculation` between run and simplify by njzjz in https://github.com/deepmodeling/dpgen/pull/1235
* add fp_style custom by njzjz in https://github.com/deepmodeling/dpgen/pull/1244
* simplify: support model deviation of energy per atom by njzjz in https://github.com/deepmodeling/dpgen/pull/1312
* simplify: support using true error as error indicator by njzjz in https://github.com/deepmodeling/dpgen/pull/1321
* dprc: add `restart_from_iter` option by njzjz in https://github.com/deepmodeling/dpgen/pull/1322
* support electronic tempeture data used in simplify; refactor post_fp_vasp by njzjz in https://github.com/deepmodeling/dpgen/pull/1329
* support training with -f or -t argument by njzjz in https://github.com/deepmodeling/dpgen/pull/1340
* Add `dpgui` entry point and `dpgen gui` CLI by njzjz in https://github.com/deepmodeling/dpgen/pull/1349
* model_devi: add support for pimd by Yi-FanLi in https://github.com/deepmodeling/dpgen/pull/1366

Enhancement
* refactor(abacus): modify poscar2stru in auto_test.lib.abacus by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1193
* simplify: improve performance of selecting frames by njzjz in https://github.com/deepmodeling/dpgen/pull/1202
* Use get_global_number_of_atoms() instead by njzjz in https://github.com/deepmodeling/dpgen/pull/1256
* load files according to filename suffix by njzjz in https://github.com/deepmodeling/dpgen/pull/1255
* check arguments with 2D lists by njzjz in https://github.com/deepmodeling/dpgen/pull/1258
* throw RuntimeError if no system is found by expand_sys_str by njzjz in https://github.com/deepmodeling/dpgen/pull/1292
* rm old files when performing `dp model-devi` by njzjz in https://github.com/deepmodeling/dpgen/pull/1332
* argcheck: restrict the type of elements in a list by njzjz in https://github.com/deepmodeling/dpgen/pull/1364
* Update {Elastic,run,surf,Vasp,vasp}.py: To be compatible with Pymatgen by ZhouXY-PKU in https://github.com/deepmodeling/dpgen/pull/1302

Bugfixes
* fix(abacus): fix deepks bug in autotest by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1163
* fix(abacus): get ION step by searching "ION=" in relax job by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1169
* fix a bug in `convert_training_data_to_hdf5` by njzjz in https://github.com/deepmodeling/dpgen/pull/1167
* fix batch size for simplify by njzjz in https://github.com/deepmodeling/dpgen/pull/1184
* fix(abacus): Compatibility with kspacing set to 3 values by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1197
* simplify: do not symlink if the file does not exist by njzjz in https://github.com/deepmodeling/dpgen/pull/1191
* Fix the numerical mismatch for 02.fp/task/cp2k_output after update dpdata by robinzyb in https://github.com/deepmodeling/dpgen/pull/1219
* fix(abacus): put pp/orb/dpks files to folder pporb in 02fp. by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1214
* fix bug in make_supercell_abacus by Liu-RX in https://github.com/deepmodeling/dpgen/pull/1212 and https://github.com/deepmodeling/dpgen/pull/1229
* len(all_sys) should > 0 for post_fp_cp2k by robinzyb in https://github.com/deepmodeling/dpgen/pull/1245
* fix a typo in new_to_old_ratio message by njzjz in https://github.com/deepmodeling/dpgen/pull/1270
* fix a bug in the number of frames for auto ratio by njzjz in https://github.com/deepmodeling/dpgen/pull/1276
* Using dict.get methods to get machine.local_root by dingzhaohan in https://github.com/deepmodeling/dpgen/pull/1287
* fix KeyError in init_reaction by njzjz in https://github.com/deepmodeling/dpgen/pull/1352
* fix Python 3.12 compatibility by njzjz in https://github.com/deepmodeling/dpgen/pull/1367
* abacus: fix bug of finding the final relax STRU by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1344

Documentation
* fix the links of DP-GUI by njzjz in https://github.com/deepmodeling/dpgen/pull/1187
* change size and apply dark mode for the logo by njzjz in https://github.com/deepmodeling/dpgen/pull/1190
* extend the doc of cp2k external_input_path in detail by robinzyb in https://github.com/deepmodeling/dpgen/pull/1225
* fix: description in README.md by cherushui in https://github.com/deepmodeling/dpgen/pull/1241
* docs: move `ratio_failed` out of fp variant by njzjz in https://github.com/deepmodeling/dpgen/pull/1243
* docs: fix docs for fp_task_min by njzjz in https://github.com/deepmodeling/dpgen/pull/1257
* update deepmd input to v2; format JSON files by njzjz in https://github.com/deepmodeling/dpgen/pull/1315
* update issue templates by njzjz in https://github.com/deepmodeling/dpgen/pull/1368
* docs: update installation documentation by njzjz in https://github.com/deepmodeling/dpgen/pull/1372

CI/CD
* bump pymatgen to a stable version by njzjz in https://github.com/deepmodeling/dpgen/pull/1186
* bypass the upstream pymatgen error for CI by njzjz in https://github.com/deepmodeling/dpgen/pull/1233
* bump pymatgen to 2023.5.31 by njzjz in https://github.com/deepmodeling/dpgen/pull/1250
* use OIDC to publish packages by njzjz in https://github.com/deepmodeling/dpgen/pull/1298
* test: pin pymatgen-analysis-defects to <2023.08.22 by njzjz in https://github.com/deepmodeling/dpgen/pull/1313
* skip Sphinx 7.2.5 by njzjz in https://github.com/deepmodeling/dpgen/pull/1324

Lint
* apply ruff rules and fix lots of bugs by njzjz in https://github.com/deepmodeling/dpgen/pull/1240
* move to ruff formatter by njzjz in https://github.com/deepmodeling/dpgen/pull/1375

New Contributors
* pre-commit-ci made their first contribution in https://github.com/deepmodeling/dpgen/pull/1171
* ZhouXY-PKU made their first contribution in https://github.com/deepmodeling/dpgen/pull/1302
* Yi-FanLi made their first contribution in https://github.com/deepmodeling/dpgen/pull/1366

**Full Changelog**: https://github.com/deepmodeling/dpgen/compare/v0.11.1...v0.12.0

Breaking changes:
* breaking: remove old dispatcher by njzjz in https://github.com/deepmodeling/dpgen/pull/1097


Enhancements:
* Add parameter 'srtab_file_path' so that one can use DP-ZBL potential successfully by Chengqian-Zhang in https://github.com/deepmodeling/dpgen/pull/918
* Refactor: remove association between ABACUS/STRU and the element number in param.json by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1076
* choose numb_steps/stop_batch automatically by njzjz in https://github.com/deepmodeling/dpgen/pull/1086
* replace distutils.version.LooseVersion by packaging.version.Version by njzjz in https://github.com/deepmodeling/dpgen/pull/1095
* Support forward files for simplify by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/1092
* Replace "python3" in command by sys.executable by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/1105
* add an option to merge data to one H5 file by njzjz in https://github.com/deepmodeling/dpgen/pull/1119
* add and run pre-commit to format codes by njzjz in https://github.com/deepmodeling/dpgen/pull/1124
* refactor(abacus): link the pp/orb files defined in STRU not jdata by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1127
* Raise warnings if there is no system in the path listed in sys_configs by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/1130
* Updated the `revise_lmp_input_dump` function. by panxiang126 in https://github.com/deepmodeling/dpgen/pull/1137
* add isort to sort imports by njzjz in https://github.com/deepmodeling/dpgen/pull/1133
* always pass codecov status by njzjz in https://github.com/deepmodeling/dpgen/pull/1157


Documentation, argument parsing and examples
* add placeholders for variants
by njzjz in https://github.com/deepmodeling/dpgen/pull/1068
* Doc(abacus): modify the description of parameters by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1080
* fix doc: "structures" should be a list by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/1091
* docs: add logo to docs and README by njzjz in https://github.com/deepmodeling/dpgen/pull/1128
* docs: add links to DP-GUI
by njzjz in https://github.com/deepmodeling/dpgen/pull/1131

Bug fixings:

* Fix: remove the reading of `ntype` when treating ABACUS related info by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1075
* fix pass action by njzjz in https://github.com/deepmodeling/dpgen/pull/1082
* fix compatibility with NumPy and pymatgen by njzjz in https://github.com/deepmodeling/dpgen/pull/1085
* Fix(abacus): parse orbital information after parsing of INPUT by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1079
* fix_bug simplify.py by Vibsteamer in https://github.com/deepmodeling/dpgen/pull/1113
* fix_bug simplify.py by Vibsteamer in https://github.com/deepmodeling/dpgen/pull/1114
* fix compatibility with the latest pymatgen by njzjz in https://github.com/deepmodeling/dpgen/pull/1126
* fix restart trajectory for `merge_traj` by wankiwi in https://github.com/deepmodeling/dpgen/pull/1111
* fix a bug in make_abacus_relax by maki49 in https://github.com/deepmodeling/dpgen/pull/1150
* fix(abacus): do not write to deepmd raw/npy, if no abacus data in post_fp_abacus_scf() by pxlxingliang in https://github.com/deepmodeling/dpgen/pull/1153
* bug_fix reproduce.py by Vibsteamer in https://github.com/deepmodeling/dpgen/pull/1156
* fix numpydoc styles by njzjz in https://github.com/deepmodeling/dpgen/pull/1134

New Contributors
* wankiwi made their first contribution in https://github.com/deepmodeling/dpgen/pull/1111

**Full Changelog**: https://github.com/deepmodeling/dpgen/compare/v0.11.0...v0.11.1

Breaking changes:
- bump requires-python to `>=3.8` (1014)

New features:
- gamma line calculation for bcc and fcc systems (801)
- support several CALYPSO run in different pressure in one iteration (814)
- autotest: add the interface of ABACUS in autotest (834)
- support merge_traj for lammps (838 961 )
- support HDF5 files as initial data system (973)
- simplify: support list for pick_data (994)

Enhancements:
- refactor: delete relax_kspacing and md_kspacing in abacus init step (848)
- check run_tasks before submission in run_md_model_devi (854 )
- UT: add UT for create_disturb_abacus_dev (852)
- Automatically set the relative atomic mass according to "type_map" (867 893)
- add tests to check machine files (874)
- UT: update test_disturb_abacus.py (879)
- add an error remind in dpgen_fp when NSW is not set to 1 (903)
- move some imports into runtime functions (894)
- support forward files for init_surf (909)
- fp: check failed tasks (882)
- remove requirements.txt (948)
- migrate from setup.py to pyproject.toml (995)
- _crd2frag: use OB native PBC implementation (1006)
- skip mirror gitee when organization is not deepmodeling (1048)
- add pass step to GitHub Actions (1058)

Documentation, argument parsing and examples
- args: ensembles -> ensemble (805)
- args: `model_devi_v_trust_lo` and `model_devi_v_trust_hi` are optional (804)
- removed `model_devi_e_trust_lo` and `model_devi_e_trust_hi`, which have been removed in 79 (807)
- args: add `shuffle_poscar` to run jdata (806)
- fix readme and qm7.json (817)
- args: trust level for model_devi_job should be optional (818)
- args: model_devi_nopbc should be optional (819)
- args: `fp_aniso_kspacing` and `cvasp` should be optional (820)
- args: add entire training parameters (829)
- add entire arguments of gaussian style (830)
- fix deprecated bare `pair_coeff` (831)
- UT checks examples (821 870 )
- change 'dpgen test' to 'dpgen autotest' (875)
- args: remove duplicated argument (839)
- update abacus input parameters (836)
- update the type of `fp_aniso_kspacing` as `List of float` (843)
- document: add the information of ABACUS in `arginfo.fp_style_variant_type_args` (845)
- add DP-GEN Structure.md to doc (844)
- doc: add FAQs (856)
- remove documentation for old dispatcher (862)
- add dargs to install_requires (863)
- example: add abacus examples (853)
- update run/dp2.x-lammps-cp2k example (864)
- add simplify example of MAPbI3-scan based on lebesgue (840)
- docs: add authors section for credits (869)
- docs: add init surf (890 904)
- docs: ddd a Troubleshooting.md file (888)
- docs: user guide (876)
- docs: add numpydoc; set proper info (895)
- example_dp2.x_lmp_gaussian (887)
- docs: update common-errors.md (897)
- fix typos and optimize autotest part in README (899)
- add DPRc (model_devi_engine `amber` and fp_style `amber/diff`) arguments (911)
- update simplify doucment (912)
- add init-bulk in dpgen documents (907)
- docs: clarify the usage of skip_relax in init_surf (929)
- docs: optimize autotest document based on 891 (923)
- manage TOC of DPGEN's document (932)
- add publication link to docs (939)
- move ch4 folder to examples directory (949)
- fix the description of fp_task (950)
- loose check dpgen run jdata (952)
- optimize document and error report based on FAQs (967)
- remove personal info in dp2.x-lammps-gaussian example (972)
- args: add `init_bulk_jdata_arginfo` and `init_surf_jdata_arginfo` (936)
- strict check simplify jdata (951)
- fix : autotest relaxation TOC (982)
- merge run-process.rst to index.rst (966)
- docs: add links to keys (981)
- docs: init_surf forward_files (1025)
- refactor readme and remove some duplicated contents (946)
- add arguments for lammps revise_template (1015)
- docs: add information about Python requirements (1060)

Bug fixings:
- fix post_fp_vasp and make_fp_configs for simplify (803 )
- fix UT test_make_md (812)
- fix bug of matrix multiply in `create_disturbs_abacus_dev` (815)
- set type_map for MultiSystems in model_devi step (825)
- test: using `assertEqual` to compare two value in test_vasp.py (828)
- autotest: delete the redundant print in `test_vasp.py` (835)
- fix bug of no funding fp_params (842)
- fix a bug in run_abacus_md and delete redundant line about gamma_only. (850)
- fix: incorrect using of `range` in auto_test/lib/abacus.py (884)
- fix: auto_test: equi_stress should have the same unit as stress (910)
- fix: skip cell_type if from_poscar is True in init_bulk (930)
- fix: fix forward_files for merge_traj in run_md_model_devi (947)
- remove `_link_fp_vasp_pp` from `make_fp_gaussian` (971)
- fix incompatibility of autotest module with latest pymatgen>2022.7.19 (963)
- fix: modify the reading of parameter 'fp_dpks_descriptor' in run.py (989)
- CVE-2007-4559 Patch (990)
- fix ImportError (1001)
- python3.10 support: collection.Mapping -> collection.abc.Mapping (1020)
- check data instead of tasks (1031)
- assign type_map for all systems (1033)
- fix codecov (1034)
- ignore dpdata 0.2.11 (1062)

**Full Changelog**: https://github.com/deepmodeling/dpgen/compare/v0.10.6...v0.11.0

New features:
* add DPRc implementation by njzjz in https://github.com/deepmodeling/dpgen/pull/729

Enhancements:
* add link to DP-GUI by njzjz in https://github.com/deepmodeling/dpgen/pull/724
* add many docs and examples by njzjz in https://github.com/deepmodeling/dpgen/pull/736 , https://github.com/deepmodeling/dpgen/pull/758 , https://github.com/deepmodeling/dpgen/pull/751 , https://github.com/deepmodeling/dpgen/pull/759 , https://github.com/deepmodeling/dpgen/pull/762 , https://github.com/deepmodeling/dpgen/pull/747 , https://github.com/deepmodeling/dpgen/pull/788 , https://github.com/deepmodeling/dpgen/pull/787
* refactor simplify by njzjz in https://github.com/deepmodeling/dpgen/pull/730
* supports non-list mdata by njzjz in https://github.com/deepmodeling/dpgen/pull/748
* Create Performing the Run process.md by LiangWenshuo1118 in https://github.com/deepmodeling/dpgen/pull/714
* upgrade all tasks to dpdispatcher by njzjz in https://github.com/deepmodeling/dpgen/pull/749 , https://github.com/deepmodeling/dpgen/pull/755
* update: add a new option of absolute volume in /dpgen/dpgen/auto_test/EOS.py by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/741
* remove `train_param` from examples, which was deprecated in 68 by njzjz in https://github.com/deepmodeling/dpgen/pull/789
* Add netcdf4 dependency to conda recipe by AnguseZhang in https://github.com/deepmodeling/dpgen/pull/792


Bug fixings:
* fix a bug in make_abacus_scf_input by Liu-RX in https://github.com/deepmodeling/dpgen/pull/754
* correct the wrong spelling of 'failure' by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/764
* fix ‘post_fp_cp2k’, add param rfailed by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/765
* Fix simplify by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/767 , https://github.com/deepmodeling/dpgen/pull/768
* fix ZeroDivisionError raised by run.py by HuangJiameng in https://github.com/deepmodeling/dpgen/pull/769
* fix bug for sys_idx dependent trust level and add option for model_devi_job by Cloudac7 in https://github.com/deepmodeling/dpgen/pull/786

New Contributors
* HuangJiameng made their first contribution in https://github.com/deepmodeling/dpgen/pull/741

**Full Changelog**: https://github.com/deepmodeling/dpgen/compare/v0.10.5...v0.10.6

Breaking changes:
- ABACUS versions < 2.2 are not supported anymore.

New features:
- support Calypso structure search in the exploration stage (663)
- add bcc interstitial calculation in autotest (696)
- Add ABACUS/lcao deepks interface to dpgen init and run, upgrade to support ABACUS >= 2.2. (705)

Enhancements:
- change some default parameters in lmp when doing autotest (696)
- batch replace 'except:' with 'except Exception:' (703)
- doc: dpgen run machine parameters (710)
- Delete information printed by `dpgen -h` to enable `pip install dpgen` (718)

Bug fixings:
- Update param_CH4_deepmd-kit-2.0.1.json (694)
- Fix the usage of `pair_coeff` in `auto_test` (696 698)
- Fix QE pwscf AssertionError (685)