Salt-nornir

Major change in this release - introduction of Pydantic models to validate execution/sate/runner modules arguments to implement fail fast approach and significantly improve overall reliability as all arguments are now validated by Pydantic ensuring semantic and functional validity.

Another change are a switch to Poetry for dependency management and switch to Rocky Linux with Python3.9 as a main development and testing environment.

CHANGES

1. Added pydantic==1.9.0 to base requirements
2. Added these pydantic models to verify execution/state/runner modules function arguments using yangdantic module from nornir-salt:

- model_exec_nr_cli
- model_exec_nr_task
- model_exec_nr_cfg
- model_exec_nr_tping
- model_exec_nr_test
- model_exec_nr_nc
- model_exec_nr_http
- model_exec_nr_do
- model_exec_nr_file
- model_exec_nr_learn
- model_exec_nr_find
- model_exec_nr_diff
- model_exec_nr_nornir_fun
- model_exec_nr_gnmi
- model_state_nr_workflow
- model_runner_nr_inventory
- model_runner_nr_call
- model_runner_nr_event
- model_runner_nr_cfg
- model_nornir_config

3. Proxy Module uses nornir-salt `modelTestsProcessorSuite` Pydantic model to verify tests suite items prior to running them, that way `nr.test` will fail immediately if any of the test items is not compliant
4. `nr.do` changed to omit searching for actions in master config, minion config and grains, it only sources them from pillar or text files from master, this is to speed up `nr.do dir` calls and overall `nr.do` actions execution
5. Added multiprocessing lock to files download, now only one worker at a time can download files from master
6. File downloading from master with `cp.get_url` was updated to use files cached on Minion, instead of downloading files over and over again from master if content is the same.
7. Moved dependency management to Poetry, removed all requirements files replacing them with `pyproject.toml` definition
8. For test environment added `fakenos` container to run scale/performance tests
9. Moved test environment to Rocky Linux container running Python 3.9 - all development will be done using that setup
10. Runner nr.call makes use of _get_hosts_minions function to get a list of hosts and proxy minion to target before running the task, that is to ensure that we only targeting alive proxy minions without unnecessary retry attempts on disconnected.downed minions
11. Test/Development containers environment updated to run on Rocky Linux using Python 3.9

FEATURES

1. Added Pydantic package and Python versions to `nr.nornir version` report
2. Introduced Pydantic models for Salt-Nornir modules functions' input validation, that is to make sure that parameters supplied comply with required format - increase Salt-Nornir proxy robustness/reliability and enforce fail-fast approach
3. Salt-Nornir updated to use Poetry for packaging and dependency management and for virtual env management
4. Added tree output to runner `nr.inventory` to visualize output for minions, devices and machines where they are hosted
5. Added model_nornir_config pydantic model to run configuration parameters verification prior to starting proxy minion

BUGS

1. Fixed issue 9 bug with simultaneous files download from master
2. When doing 'nr.file rm filegrouop_name' was getting tf_index file overwritten probably by various task threads trying to remove files simultaneously, introduced file lock support mechanism passing it onto the 'files' task and 'ToFileProcessor' from proxy minion
3. Fixed nr.cfg runner to honor Fx filters while filtering hosts to configure
4. nr.cfg runner fixed batches calculations to account for cases when using odd numbers of hosts and of batch size e.g. 7 / 3 should produce 3 batches while previously was resulted in 2 batches

CHANGES

1. Cleaned up base installation requirement leaving only nornir, nornir-salt and psutil.
2. Added installation flavors:
- dev - packages for development
- prodmin - minimum set of tested packages for production use to interact with devices over NETCONF, SSH or RESTCONF using Ncclient, Netmiko and requests respectively
- prodmax - set of tested packages to incorporate support of all salt-nornir supported features
3. Nornir Proxy Module render and download function changed, now, instead of throwing error and stopping altogether, new behavior is to continue with task execution but exclude failed hosts, once task completed, failed hosts added back to the results with exception details
4. Previously, when calling `nr.cfg_gen`, `nr.cfg` or `nr.cli` with filename argument pointing to non existing file it returns traceback, now it returns errors on a per-host basis and continue execution for non failed hosts. This is the result of implementing change 3.
5. `nr.workflow` updated to include hard requirement for global filter options to be set if proxy minion `nornir_filter_required` parameter is True, if `nornir_filter_required` is True, and no filters given in workflow global options, it will fail to target any hosts and will exit with error.
6. Updated runner behavior to identify hosts to minions mapping before sending tasks out, that allows to target only minions that actually manage given hosts.
7. Runner `nr.event` replaced `mode` argument with `progress` argument
8. Execution module - added `user` key to job identity dictionary to include it in events emitted by SaltEventProcessor.
9. State module, updated to generate uuid jid for the tasks for jobs tracking, by default state modules does not get jid id assigned, also, extracted user name value from __opts__ to include it in identity dictionary.
10. `nr.event` runner extended to display user name for the progress `bars` and `logs` modes
11. Dependency updated `nornir-scrapli==2022.1.30a1`, previously was `nornir-scrapli==2021.7.30`, reason - `scrapli-netconf==2022.1.30a1` fails to work with `nornir-scrapli==2021.7.30` netconf tasks if doing custom tasks.
12. Changed requirement from `psutil==5.8.*` to `psutil>=5.8.*, <=5.9.*` to allow newer psutil version, tested it with 3002 and 3004 SaltStack versions

BUGS

1. `nr.workflow` state module updated to exit with comment if global filters option provided but matched no hosts, previously it was continuing with execution returning empty report
2. Change 3 fixed `nr.workflow` if first step is to get file from master and it fails for the host, this error was not included in report.
3. Change 3 fixed `nr.workflow` if first step fails and it targeted only one host, empty report produced
4. Nornir Proxy module `list_hosts` function updated to honor `nornir_filter_required` parameter to raise an error if no filters provided but `nornir_filter_required` set to true.
5. Fixing bug 4 fixed issue with nr.workflow when it was continuing on with tasks execution when `nornir_filter_required` was True but no global filters provided in options.
6. Runner nr.event was fixed to better handle events for failed hosts, previously it was failing with key error
7. Runner fixed bug to raise command execution error when no minions matched by tgt and tgt_type
8. Fixed docs spelling using pylint spelling check plugin (pyenchant) and sphinxcontrib-spelling RST plugin
9. When was running tasks with runner other than RetryRunner netmiko_send_command failed with - `TypeError: send_command() got an unexpected keyword argument 'connection_name'`, added check if not using RetryRunner to remove `connection_name` argument from task kwargs.

FEATURES

1. Added support for `nr.workflow` to produce text tables in github, jira or some pretty format out of workflow report summary section using tabulate module.
2. Proxy module `render` function updated to support arguments that refer to dictionary keyed by host name with values being config templates to render.
3. Added new runner function nr.cfg that supports loading devices configuration from directory with files or dictionary, it also supports interactive mode allowing operator to control tasks execution.
4. Updated proxy module and SaltEvetProcessor to send worker_id in events
5. Created runner `nr.make_plugin` function to create boilerplate code for various Nornir Plugins, added Nornir `task` and `test` - TestsProcessor custom function
6. Runner module added `bars` progress mode to `nr.event`, as a result `nr.call` and `nr.cfg` also support `bars` mode. bars mode uses Rich library progress bars.
7. `nr.workflow` added support to specify common step `kwargs` in `options` section, this is to simplify adding common arguments to all steps, like `event_progress`
8. Added new stats - `tasks_completed` and `tasks_failed` counters, as well as `worker_tasks_completed` and `worker_tasks_failed` to `nr.nornir stats` and `nr.nornir workers stats`
functions output respectively.
9. Execution module table `brief` and `terse` format now support headers argument e.g. `table=brief headers="host, result"`
10. Added `nr.nornir connect` function to be able to connect to devices on demand, as well as to retry various connection parameters such as credentials, port number etc. using `reconnect` argument list
11. Execution module tasks added support for RetryRunner task parameters - `run_connect_retry`, `run_task_retry`, `run_creds_retry`, `run_num_workers`, `run_num_connectors` to influence Nornir task run logic
12. Replaced cp.get_file_str with cp.get_url for downloading functions to allow sourcing files from any of supported URL schemes: salt://, http://, https://, ftp://, s3://, swift:// and file:// (local filesystem). Updated proxy module and execution modules to provide that support.

Significant changes in this release to improve task execution throughput and unveil new capabilities to manage network devices inspired by tbotnz's netpalm project workers handling.

BUGS

1. Fixed `nr.nornir` unknown function call - string formatter was broken
2. Fixed nr.test handling of integer patterns converting the to string
3. Fixed bug with 'nr.nornir refresh' function not updating proxy parameters
4. Fixed `proxy_always_alive` is False, connection through the proxy does not terminated completely and becomes unusable after first task
5. Fixed Salt3002 local client destroy method call

FEATURE

1. Added support to execution module for `tf_skip_failed` `ToFileProcessor` argument to not save results for failed tasks
2. Added support for multiple Nornir Worker instances to improve task execution throughput
3. Added proxy parameter `nornir_workers` to control the number of Nornir Worker Instances, default is 3
4. Added support for `worker` execution module argument to delegate task execution to certain Nornir worker
5. Added `nr.nornir workers stats` function to retrieve workers stats
6. Added `dir` function to `nr.nornir` to list supported functions
7. Added new proxy `connections_idle_timeout` to control for how long devices' connections can stay open, by default it never timeouts
9. Added `queues_utils` function to proxy module
10. Added `workers_utils` function to proxy module
11. Added `results_queue_dump` to `nr.nornir` execution module function

CHANGES

1. Due to introduction of multiple Nornir workers had to make these changes:
- Task identity now includes worker number if worker=all
- hcache and dcache handling updated to make sure all worker instances have same data cached
- nr.nornir inventory add/remove/read hosts done for all workers by default now
- nr.nornir connections returns connections data for all workers by default now
- each worker tracks its own job stats, global stats computed using per-worker stats now
2. Updated requirements to latest releases for test environment:
- paramiko==2.9.2 was 2.7.2
- nornir-utils - removed
- nornir-netmiko==0.1.2 was 0.1.1
- nornir==3.2.0 was 3.1.1
- genie==22.1 was 21.10
- pyats==22.1 was 21.10
- pygnmi==0.6.8 was 0.5.3
- ttp==0.8.4 was 0.8.2
- ttp-templates==0.1.3 was 0.1.2
- psutil==5.9.0 was 5.9.0
- pyyaml==6.0 was 5.4.1
3. Execution module `nr.learn` updated to use `tf_skip_failed=True` by default to not save failed tasks

BUGS

1. Fixed issue with _get_or_import_task_fun task checking already imported modules logic, where if task has same name it will not be imported again e.g. nornir_salt.plugins.tasks.netmiko_send_config and nornir_netmiko.tasks.netmiko_send_config - same task function name netmiko_send_config, the fix is to check
if already imported task __module__ attribute equals to the one we importing.
2. Fixed watchdog to call `kill_nornir` function after `shutdown` in case if FD or RAM limit exceeded

FEATURES

1. Added support for `pyats` plugin to `nr.cli`
2. Added support for `pyats` plugin to `nr.cfg`
3. `nr.do dir` now support glob pattern as a second argument to filter action names: `salt nrp1 nr.do dir "*namefilter*"`
4. Added support for `InventoryFun` calls using worker thread functions handling

CHANGES

1. Test/dev environment - added nrp2 proxy to talk with Cisco always on lab devices.
2. Test/dev - changed directories structure to simplify code maintenance
3. Test/dev environment - updated docker-compose to account for new directories structure
4. Removed proxy minion `inventory_data` function
5. Renamed proxy minion hosts_list to list_hosts function and updated it to use `InventoryFun`
6. Updated state nr.workflow to use `nr.inventory hosts` functions

FEATURES

1. Added `hcache` keyword to save task execution results in host's inventory data
2. Added `dcache` keyword to save task execution results in defaults inventory data
3. Added `nr.nornir clear_hcache` call to clear cached task results data from hosts' inventory
4. Added `nr.nornir clear_dcache` call to clear cached task results data from defaults inventory
5. Added hosts list to nornir proxy minion grains
6. Added support for nr.cli plugins to use `_br_` to send enter key to the cli e.g. "reload" "_br_" becomes "reload" "\n", but "\n" cannot be supplied using salt cli
7. `nr.workflow` caches tasks results in `hcache` and `dcache` data by default now, added global state options `hcache` and `dcache` to control that behavior, as a result each and every workflow step has access to previous steps results using Nornir inventory data.
8. Added support for text table output to `nr.workflow` to collect it as part of results detailed report
9. Added `dump` argument to `nr.test` execution module function so that final test results can be saved to file using `DumpResults` nornir-salt function
10. Added support for proxy module `nr_data` function to return values for multiple keys
11. Added `sortby` and `reverse` keywords support for `nr.test` table formatting
12. Added `napalm` plugin support to `nr.cli` execution module function allowing to send commands at once or one by one to devices using NAPALM `cli` method
13. Added `ntfsm` keyword to parse `nr.cli` output using DataProcessor `ntfsm` function that uses TextFSM ntc-templates
14. `nr.nornir version` report now returns version for ttp-template and ntc-templates
15. Added `nr.call` function to Runner Module to be able to all any Execution Module function using `salt-run nr.call xyz FB=*`, this function prints execution progress to terminal screen by default
16. Added `event_progress` argument to execution module to emit events on task execution progress.
17. Added runner `nr.event` task to listen to events emitted by execution module called with `event_progress` argument.
18. Added `run_ttp` `remove_tasks` argument support.
19. Added Proxy minion pillar setting `event_progress_all` to control default behavior for emitting events for all tasks.
20. `nr.test` test suite items now have support for `wait_timeout` and `wait_interval` to wait until test result success evaluates to True

BUGS

1. Fixed salt 3003 support for run_ttp and iplkp arguments - file download was broken for them.
2. Fixed `refresh` and `shutdown` functions to make them actually work by moving them back to worker thread for calling within main process. Previously moved them to calling as
proxy function but that does not work as call happens from within child process that deals with nr object copy and not with original one, which is only accessible from main
process.
3. Fixed state `nr.workflow` to include "non-compatible" steps results in summary report as they were missing before.
4. Fixed support for SaltStack 3004 as they changed api for loader import
5. Fixed nr.test suite argument handling to support suite being a list in addition to be a string or using inline commands
6. On startup nornir-proxy report that watchdog failed to acquire connections lock, this is because watchdog was starting before lock created, fix the order.
7. When `nr.test` was using suite with `task: run_ttp` it was sending `run_ttp` command string to devices, fixed to not use `run_ttp` as a command string.

CHANGES

1. PROXY Context loader now handled using decorator function
2. Improved `nr.nornir refresh` function to sync all modules and refresh pillar, but still, noticed that need to call `nr.nornir refresh` twice to refresh `nr.do` actions
3. Runner module - removed `nr.cli` function in favor of using `nr.call` function
4. Execution module passes `jid` to `execute_job` proxy minion function, this is required by SaltEventProcessor, all execution module tasks now need to extract jid from kwargs.
5. Introduced `identity` dictionary with keys "uuid4", "jid", "function_name" to identify task results in results queue produced by worker thread, this is to make sure if `multiprocessing=False` and threads used instead task results can be uniquely identified, in addition `SaltEventProcessor` uses `identity` dictionary to add jid and function name
data to events.