Tokenizer

Latest version: v3.4.3

Safety actively analyzes 613750 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 8

3.4.3

* Various minor fixes. Now requires Python 3.8 or later.

**Full Changelog**: https://github.com/mideind/Tokenizer/compare/3.4.2...3.4.3

3.4.2

- Some abbreviations and phrases added
- META_BEGIN token added to help users distinguish between metatokens and regular tokens

3.4.1

* Improved performance on large input chunks

3.4.0

* Improved handling and normalization of punctuation

3.3.3

* Better support for token-level errors

3.3.2

* Internal refactoring
* Fixes in paragraph handling

Page 1 of 8

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.