Zigpy-znp

Bugfixes

- Fix joining error with firmware versions >= 20210707 (https://github.com/zigpy/zigpy-znp/commit/ddf79f67d5318870381a2b342d861576b0c4f3de)
- Increase startup LED response timeout (https://github.com/zigpy/zigpy-znp/commit/c9cb5765b5553b5d6898a7751e49235ac73ae158)

Bugfixes
- Detect the CC2538 and include the Z-Stack build in the device model string (https://github.com/zigpy/zigpy-znp/commit/a5717265991256d6df444344e8ede96bea1e412a, #72).
- Properly handle older Z-Stack `SYS.SetTxPower` responses (82)
- Permit deserialization failures with the undocumented `ZDO.ParentAnnceRsp` command (https://github.com/zigpy/zigpy-znp/commit/aaf0c31fb3a6ded29e3c8e6036478a64d6011c84, https://github.com/home-assistant/core/issues/50005).
- Silence "Unhandled command" warnings for `ZDO.ParentAnnceRsp`, `ZDO.ConcentratorInd`, and `ZDO.MgmtNWKUpdateNotify` (https://github.com/zigpy/zigpy-znp/issues/75).
- Joins through specific routers will not also open the coordinator for joining on Z-Stack `20210708` (properly fixes 53)

New features
- Treat messages from unknown devices as implicit joins. This allows for existing networks to be mapped out without explicitly re-joining most devices (73)
- Normalize command logging for NVRAM reads and writes (https://github.com/zigpy/zigpy-znp/commit/0adeb9b608a812d77f71e5854cce6cefa2e98a41)

Bugfixes
- Command line tools failed to write JSON/binary output to STDOUT and read it from STDIN: 8300994439bb2cd75e3c2ea58aecef65bc1230fd

New Features
- Huge thank you to castorw from the Z2M project for working with me to bring concurrent support for the [Open Coordinator Backup format](https://github.com/zigpy/open-coordinator-backup/) to Z2M. This allows for bidirectional migration between any coordinators supported by zigpy-znp. More documentation is available in [`TOOLS.md`#Backup and restore](https://github.com/zigpy/zigpy-znp/blob/6b8a6d8978fda40c7c5fee320486aee3c4159c48/TOOLS.md#backup-and-restore).
- Multicast and broadcast requests are sent immediately
- Log a warning when using the Z2M default network key.

Changes
- Maximum time to wait for a device announcement after a device join before sending the zigpy `device_joined` signal has been increased from 2 seconds to 5 seconds.
- Write both `PRECFGKEYS_ENABLE` and `EXTENDED_PAN_ID` OSAL NVRAM items to ensure Z2M works with adapters commissioned with zigpy-znp.
- Retry using manual source routing first, before route discovery.
- If an exception is encountered when passing a frame from the `UART` object to `ZNP`, log the entire frame.
- If an unknown frame is passed to `ZNP`, log it but do not throw an exception.
- The `-c` command line flag has been removed from `zigpy_znp.tools.nvram_reset` and is now the default.

Internal Changes
- Add a `force_form` keyword argument to `ControllerApplication.startup`.
- All CC2531-specific types have been removed with the addition of the `CStruct` class, which computes struct padding and alignment during serialization/deserialization. This is autodetected at startup.
- The `ZNP` class can now be used more independently of `ControllerApplication`, especially without a formed network.
- `ControllerApplication._reset` has been moved to `ZNP.reset`.
- Complete network information is extracted from NVRAM at startup and populates the `ZNP` instance's `network_info` attribute.
- New helper methods to read and write NVRAM tables.

Breaking changes
- Serialization and deserialization of complex types is strongly encouraged to be done via `znp.nvram.serialize`/`deserialize` instead of the object's `serialize`/`deserialize` methods, which takes into account structure alignment and padding.
- `item_type` is now a required keyword argument for all NVRAM reads and no trailing data after deserialization is permitted. Old behavior can be preserved by passing `item_type=t.Bytes`.

New Features
- The default LED mode is now `off`.
- Support for `ZDOCmd.Unbind_req` (54).
- Very verbose `TRACE`-level debug logging is now hidden behind the logger object in `zigpy_znp.logger`. To enable it again in Home Assistant, add the following to your logging configuration:
YAML
logger:
logs:
zigpy_znp.logger: debug

- Periodically pings Z-Stack to detect soft lockups and treat this failure as a disconnect. For sticks that reset upon reconnect, like [slaesh's CC2652RB stick](https://slae.sh/projects/cc2652/), this will make recovery automatic.
- Log network settings on startup under the `INFO` log level. For security, the network key will be logged under the `DEBUG` level.
- During request retrying, always attempt source routing, even if a device's `relays` are not present.

Bugfixes
- `permit_ncp` is now a no-op and all calls to `ControllerApplication.permit` will open joins on the coordinator, even when joins are permitted only through another router (https://github.com/zigpy/zigpy-znp/issues/53). A full solution will require changes to Z-Stack.
- Do not attempt manual child aging on older coordinators, as their firmware does not have the required command and MT serialization inconsistencies will cause errors to be logged.
- Prevent the installation of `pyserial_asyncio==0.5` on Windows, which breaks writes (https://github.com/pyserial/pyserial-asyncio/issues/69).

Security Fixes
- CC2531 coordinators running Z-Stack Home 1.2 firmware erroneously permit joins on startup, even after being reset. Joins are now explicitly disabled every time the coordinator is started (https://github.com/home-assistant/core/issues/43245).

New Features
- Handle errors internal to Z-Stack by retrying requests within zigpy-znp. Missing routes are explicitly re-discovered using a few different methods, including manual source routing as a last resort. Thanks zigbee-herdsman!
- Check for and re-discover missing routes before sending ZDO requests.
- Upon receipt of a Trust Center key exchange notification, `handle_join` will be delayed for at most two seconds, until a device sends a ZDO Parent Announce broadcast.

Bugfixes
- NVRAM backup and restore did not read/write tables except for the first and last items. This is now properly handled but the backup format is now backwards-incompatible.
- `ControllerApplication.force_remove` does not throw exceptions (https://github.com/zigpy/zigpy/issues/563).
- Fix communication with devices using nonstandard profiles by always defaulting to endpoint 1 (43)
- Fix `ZDOCmd.Mgmt_Rtg_req` ZDO converter incorrectly sending `ZDO.MgmtLqiReq` instead of `ZDO.MgmtRtgReq`.

Internal Changes
- `NwkNvIds` and `OsalNvIds` were improperly named. `NwkNvIds` are now `OsalNvIds` and `OsalNvIds` are now `ExNvIds`. The new-style and legacy NVRAM operations are now exposed via `ZNP.nvram.read` and `ZNP.nvram.legacy_read`, respectively. The `ZNP.nvram` object transparently handles items that require multiple reads/writes to fully populate and bypasses NVRAM read protection for secure items when used with certain builds of Z-Stack Home 1.2.
- Shield request/response/callback coroutines from cancellation in order to reduce the number of logged `unhandled command` warnings.
- Remove handling of `"auto"` as a device path.