NUTS defines a desired network state and checks it against a real network using pytest and nornir.

INSRapperswil, updated 🕥 2023-02-08 01:31:58

NetTowel Network Unit Testing System

Introduction

The NetTowel Network Unit Testing System or "nuts" in short is the testing component of the NetTowel Project, which is developed at the Institute of Networked Solutions in Rapperswil, Switzerland. It draws on the concept of unit tests, known from the domain of programming, and applies it to the domain of networking.

One major difference between unit tests in programming and network tests is the definition of what a test actually is. In programming, unit tests normally focus on testing edge cases, since the amount of non-edge cases is not definable. In the network testing domain, tests are less about edge cases, but more about testing existing network states with pre-defined test cases. Such a single test case might be "can host A reach neighbors X, Y, Z?" on many different devices. This is what nuts tries to achieve: Apply test cases based on your pre-defined network topology to your actual network and have the tests confirm the correct state.

Installation Instructions

Using pip

Run pip install nuts

Using poetry

Nuts uses poetry as a dependency manager.

Install poetry.
Clone this repository.
Run $ poetry install

How It Works: Test Bundles and Test Definitions

The project relies on the pytest framework to setup and execute the tests. Nuts itself is written as a custom pytest plugin. In the background, nornir executes specific network tasks for the actual tests.

Nuts treats the test definition and the so-called test bundle as separate entities. The test definition is modeled as a custom pytest.Class, and a predefined set of test definitions can be found in the nuts module base_tests. New test definitions can be added easily by the user of the plugin.

The test bundle is a file that is parsed by pytest. The file provides data on the desired network state and describes which test definitions should be collected and executed by pytest. The structure of the test bundle should enable people without in-depth python knowledge to add new test bundles or update existing ones to reflect changes in the network.

While the readme here is only a short overview, find the documentation of nuts on readthedocs.

Test Bundle Structure

Currently only yaml files are supported as test bundles, but other sources such as other file formats or database entries can be considered in later nuts versions.

Each test bundle contains the following structure: ```yaml

test_module: # optional test_class: label: # optional test_execution: # optional test_data: ... ``test_module: The full path of the python module that contains the test class to be used. This value is optional if the test class is registered inindex.pyof the pytest-nuts plugin. Note that it can be relevant in which directorypytestis started if local test modules are used. Usingtest_modules` allows you to write your own test classes. Note: We currently do not support self-written test modules, since upcoming refactorings might introduce breaking changes.

test_class: The name of the python class which contains the tests that should be executed. Note that currently every test in this class will be executed.

label: Additional identifier that can be used to distinguish between multiple occurrences of the same test class in a test bundle.

test_execution: Data that is exposed as part of the nuts_parameters property. By convention this contains additional information that is passed directly to the nornir task in the background. Therefore the key-value pairs must be consistent with the key-value pairs of the specific nornir task. As an example, the test definition napalm_ping.py calls a nornir task to execute napalm's ping-command. This allows the additional max_drop parameter in test_execution, since it is in turn pre-defined by napalm.

test_data: Data that is used to parametrize the tests in the test class which have the pytest.mark.nuts annotation. It is additionally part of the nuts_parameters property.

Example: CDP Neighbors

Example of a test bundle for TestNetmikoCdpNeighbors which tests that R1 is a CDP Neighbor of both R2 and R3. This example creates three different tests, one for each entry in the test_data list.

```yaml

test_module: nuts.base_tests.netmiko_cdp_neighbors test_class: TestNetmikoCdpNeighbors test_data:
- host: R1 local_port: GigabitEthernet3 destination_host: R2 management_ip: 172.16.12.2 remote_port: GigabitEthernet2
- host: R1 local_port: GigabitEthernet4 destination_host: R3 management_ip: 172.16.13.3 remote_port: GigabitEthernet2
- host: R2 local_port: GigabitEthernet2 destination_host: R1 management_ip: 172.16.12.1 remote_port: GigabitEthernet3 ... ```

How the Test Bundle Is Converted to a Pytest Test

When nuts is executed, pytest converts the test bundles (the yaml files) into tests. During test collection, the custom pytest marker nuts uses the data that has been defined in the test bundle mentioned above. This annotation is a wrapper around the pytest.mark.parametrize annotation and allows the plugin to use the data entries from the test bundle. For each entry in the test_data section of the test bundle, the custom marker generates a single test case. To achieve this, the plugin transforms the entries into n-tuples, since pytest.mark.parametrize expects a list of n-tuples as input.

The custom nuts marker takes two arguments: The first argument of the annotation determines the required fields. For each entry in test_data these fields are extracted and transformed to a tuple considering the correct order. If any of these fields are not present in an entry of test_data, the corresponding test case will be skipped. A second argument determines optional fields that can also be used in a test case as well - non-present values are passed into the function as None.

The following test-run of CDP neighbors for example checks the local port:

python class TestNetmikoCdpNeighbors: @pytest.mark.nuts("remote_host,local_port") def test_local_port(self, single_result, remote_host, local_port): assert single_result.result[remote_host]["local_port"] == local_port

The required fields are host, remote_host and local_port - they must be present in the custom marker, but also be provided as argument to the test method itself.

single_result uses the host field and provides the result that has been processed via the specific context of a test.

Test classes and their context

Each test module implements a context class to provide module-specific functionality to its tests. This context class is a NutsContext or a subclass of it. This guarantees a consistent interface across all tests for test setup and execution. Currently, the predefined test classes use nornir in order to communicate with the network devices. Those test classes derive all from a more specific NornirNutsContext, which provides a nornir instance and nornir-specific helpers. In the example above, it is a class called CdpNeighborsContext that derives from NornirNutsContext.

If you want to learn more how nuts works but do not have a running network in the background, there's a nuts showcase - an offline test class that displays the basic functionality of nuts. See the tutorial for further information.

Develop Your Own Test Classes

Nuts is essentially designed as a pytest-plugin and it is possible to add your own, self-written test classes. A dev documentation on how to write your own test classes is planned for a future release. Still, it is possible to write your own test classes nevertheless, even if we cannot guarantee that upcoming planned refactorings do not introduce breaking changes.

Community-provided test classes

Cisco IOSXE Test Class by briantsaunders

Thanks

Matthias Gabriel, who laid the foundations of nuts.
Florian Bruhin (The Compiler) for invaluable feedback and advice.

Issues

Bump cryptography from 3.4.7 to 39.0.1

opened on 2023-02-08 01:31:58 by dependabot[bot]

Bumps cryptography from 3.4.7 to 39.0.1.

Changelog

Sourced from cryptography's changelog.

39.0.1 - 2023-02-07
* **SECURITY ISSUE** - Fixed a bug where ``Cipher.update_into`` accepted Python
  buffer protocol objects, but allowed immutable buffers. **CVE-2023-23931**
* Updated Windows, macOS, and Linux wheels to be compiled with OpenSSL 3.0.8.
.. _v39-0-0:
39.0.0 - 2023-01-01
BACKWARDS INCOMPATIBLE: Support for OpenSSL 1.1.0 has been removed. Users on older version of OpenSSL will need to upgrade.

BACKWARDS INCOMPATIBLE: Dropped support for LibreSSL < 3.5. The new minimum LibreSSL version is 3.5.0. Going forward our policy is to support versions of LibreSSL that are available in versions of OpenBSD that are still receiving security support.

BACKWARDS INCOMPATIBLE: Removed the encode_point and from_encoded_point methods on :class:~cryptography.hazmat.primitives.asymmetric.ec.EllipticCurvePublicNumbers, which had been deprecated for several years. :meth:~cryptography.hazmat.primitives.asymmetric.ec.EllipticCurvePublicKey.public_bytes and :meth:~cryptography.hazmat.primitives.asymmetric.ec.EllipticCurvePublicKey.from_encoded_point should be used instead.

BACKWARDS INCOMPATIBLE: Support for using MD5 or SHA1 in :class:~cryptography.x509.CertificateBuilder, other X.509 builders, and PKCS7 has been removed.

BACKWARDS INCOMPATIBLE: Dropped support for macOS 10.10 and 10.11, macOS users must upgrade to 10.12 or newer.

ANNOUNCEMENT: The next version of cryptography (40.0) will change the way we link OpenSSL. This will only impact users who build cryptography from source (i.e., not from a wheel), and specify their own version of OpenSSL. For those users, the CFLAGS, LDFLAGS, INCLUDE, LIB, and CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS environment variables will no longer be respected. Instead, users will need to configure their builds as documented here_.

Added support for :ref:disabling the legacy provider in OpenSSL 3.0.x<legacy-provider>.

Added support for disabling RSA key validation checks when loading RSA keys via :func:~cryptography.hazmat.primitives.serialization.load_pem_private_key, :func:~cryptography.hazmat.primitives.serialization.load_der_private_key, and :meth:~cryptography.hazmat.primitives.asymmetric.rsa.RSAPrivateNumbers.private_key. This speeds up key loading but is :term:unsafe if you are loading potentially attacker supplied keys.

Significantly improved performance for :class:~cryptography.hazmat.primitives.ciphers.aead.ChaCha20Poly1305

... (truncated)

Commits

d6951dc changelog + security fix backport (#8231)
138da90 workaround scapy bug in downstream tests (#8218) (#8228)
69527bc bookworm is py311 now (#8200)
111deef backport main branch CI to 39.0.x (#8153)
338a65a 39.0.0 version bump (#7954)
84a3cd7 automatically download and upload circleci wheels (#7949)
525c0b3 Type annotate release.py (#7951)
46d2a94 Use the latest 3.10 release when wheel building (#7953)
f150dc1 fix CI to work with ubuntu 22.04 (#7950)
8867724 fix README for python3 (#7947)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/INSRapperswil/nuts/network/alerts).

Bump certifi from 2020.12.5 to 2022.12.7

opened on 2022-12-08 08:49:28 by dependabot[bot]

Bumps certifi from 2020.12.5 to 2022.12.7.

Commits

9e9e840 2022.12.07
b81bdb2 2022.09.24
939a28f 2022.09.14
aca828a 2022.06.15.2
de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...
b8eb5e9 2022.06.15.1
47fb7ab Fix deprecation warning on Python 3.11 (#199)
b0b48e0 fixes #198 -- update link in license
9d514b4 2022.06.15
4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump lxml from 4.6.5 to 4.9.1

opened on 2022-07-06 21:15:53 by dependabot[bot]

Bumps lxml from 4.6.5 to 4.9.1.

Changelog

Sourced from lxml's changelog.

4.9.1 (2022-07-01)

Bugs fixed

A crash was resolved when using iterwalk() (or canonicalize()) after parsing certain incorrect input. Note that iterwalk() can crash on valid input parsed with the same parser after failing to parse the incorrect input.

4.9.0 (2022-06-01)

Bugs fixed

GH#341: The mixin inheritance order in lxml.html was corrected. Patch by xmo-odoo.

Other changes

Built with Cython 0.29.30 to adapt to changes in Python 3.11 and 3.12.

Wheels include zlib 1.2.12, libxml2 2.9.14 and libxslt 1.1.35 (libxml2 2.9.12+ and libxslt 1.1.34 on Windows).

GH#343: Windows-AArch64 build support in Visual Studio. Patch by Steve Dower.

4.8.0 (2022-02-17)

Features added

GH#337: Path-like objects are now supported throughout the API instead of just strings. Patch by Henning Janssen.

The ElementMaker now supports QName values as tags, which always override the default namespace of the factory.

Bugs fixed

GH#338: In lxml.objectify, the XSI float annotation "nan" and "inf" were spelled in lower case, whereas XML Schema datatypes define them as "NaN" and "INF" respectively.

... (truncated)

Commits

d01872c Prevent parse failure in new test from leaking into later test runs.
d65e632 Prepare release of lxml 4.9.1.
86368e9 Fix a crash when incorrect parser input occurs together with usages of iterwa...
50c2764 Delete unused Travis CI config and reference in docs (GH-345)
8f0bf2d Try to speed up the musllinux AArch64 build by splitting the different CPytho...
b9f7074 Remove debug print from test.
b224e0f Try to install 'xz' in wheel builds, if available, since it's now needed to e...
897ebfa Update macOS deployment target version from 10.14 to 10.15 since 10.14 starts...
853c9e9 Prepare release of 4.9.0.
d3f77e6 Add a test for https://bugs.launchpad.net/lxml/+bug/1965070 leaving out the a...
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump paramiko from 2.7.2 to 2.10.1

opened on 2022-03-29 22:08:59 by dependabot[bot]

Bumps paramiko from 2.7.2 to 2.10.1.

Commits

286bd9f Cut 2.10.1
4c491e2 Fix CVE re: PKey.write_private_key chmod race
aa3cc6f Cut 2.10.0
e50e19f Fix up changelog entry with real links
02ad67e Helps to actually leverage your mocked system calls
29d7bf4 Clearly our agent stuff is not fully tested yet...
5fcb8da OpenSSH docs state %C should also work in IdentityFile and Match exec
1bf3dce Changelog enhancement
f6342fc Prettify, add %C as acceptable controlpath token, mock gethostname
3f3451f Add to changelog
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Make pytest AssertionErrors more understandable

opened on 2022-01-19 07:29:57 by lucmurer

The pytest details delivered when a test fails are rather programmer-specific and need know-how of the internas of pytest. Depending on how the assertion condition is written, the assertion error details printed out by pytest will rather confuse the NUTS user instead of hinting him to the real problem.

Examples:

ARP-Range test: If the ARP table entries count is outside the range, pytest will print <min or max> != <actual> Expected :<min or max> Actual :<actual> This needs some interpretation of the NUTS user to comprehend which part of the test (min or max of the range) really failed.
ARP-Entries test: When using the assert <element> in <array> notation, pytest will print <element> != <array> as reason in the error message. For sure a single element does not equal the whole array, the hint that this results from a contains-check comes some lines further down, where the respective test function is printed. This confused even me a lot when I first saw this. I am not sure whether this is something pytest could improve, anyhow it highlights the point I want to make with this issue. For this specific test I hopefully improved the usefulness of the error message for NUTS users by using assert <array>.count(<element>) >= 1 which will print Expected: 1, Actual: 0 on error. This is probably better understandable from the test context.

Possible Solution:

There is the possibility to change how assertions are rewritten, as described in https://docs.pytest.org/en/6.2.x/assert.html#assertion-introspection-details and the links provided there. This could be a possibility to improve the error message the user gets printed on error.

Publishing Release on GitHub fails

opened on 2021-12-08 09:31:06 by lucmurer

Log for v3.0.2: https://github.com/INSRapperswil/nuts/runs/4455004888?check_suite_focus=true

It seems this is a permissions issue with the GITHUB_TOKEN, for more information on how to set permissions on this see: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token

Releases

Release Notes for Nuts 3.1.0 2022-01-21 09:48:24

The latest 3.1.0 release for Nuts contains some new tests contributed by @marcom4rtinez and @sevigrimm, and fixes some security (dependency) issues.

Add test bundle for ARP table testing (#57)
Add test bundle for VLAN testing (#57, #29)
Add test bundle for Config testing (startup vs. running config) (#57)
Add count tests for CDP/LLDP neighbors (#57)
Add existence test for network instances to network instance test bundle (#57)
Fix some dependency-security issues (#56)
Removed @bytinbit as a maintainer

Further upcoming Changes: * Next big release will probably be NUTS 4.0 (basically do the required refactoring for fixing #19 aka "DINO"), but I have no timeline for this yet. It mostly depends on
the priorities I get from my employer.

Release Notes for Nuts 3.0.2 2021-12-08 09:11:58

The latest 3.0.2 release for Nuts contains mainly fixes to the core and improvements to the documentation.

Add @lucmurer to the project maintainers (#42)
Add some diagrams describing the data flow through NUTS (#12)
Add more self-tests (#15, #16)
Refactor: Move implementation of method nornir_filter() up to NornirNutsContext (#51)
Fix several type annotations (#18)
Fix several dependency issues (#14, #43)

Further upcoming Changes: * Sadly @bytinbit stops being a maintainer of the NUTS project by the end of the year. @lucmurer will take over as maintainer and main developer of NUTS. * @lucmurer and @bytinbit discussed the specifics of release 4.0 (fixing #19 aka "DINO") which will require breaking changes to the architecture of NUTS. Based on this, the issue raised by PR #34 will be much easier to integrate into NUTS, so the merge has been deferred until release 4.0.

Release Notes for Nuts 3.0.1 2021-07-12 07:38:37

The latest 3.0.1 release for nuts revamps its core and self-tests. Changes:

Added short development documentation to help you write your own tests.
Self-tests refactored, added integration tests for test bundles.
Migrated from testdir to pytester in self-tests.
Improved error-handling. Previously, all errors thrown by nuts had been subsumed in a NutsResult, which made debugging difficult. Now errors that are not related to the information gathering process are clearly visible and allow for better error reporting and debugging.
single_result fixture refactored. A new AbstractResultExtractor class now takes care of transforming the gathered, raw data into a structure used by the test classes. The new structure also simplifies using the custom nuts pytest marker.
Improvements in project management.
Set-up release pipeline

Planned for release 3.0.2:

Add more integration tests for "bad" results
Add API documentation

Release Notes for Nuts 3.0.0 2021-05-17 09:04:06

Nuts 3.0.0 is a completely new version of nuts, rewritten from scratch and not compatible with earlier versions of nuts. Nuts 3.0.0. is designed as a pytest-plugin to run tests and uses nornir to interact with a network in the background. It requires python 3.7 and above. In case you wonder where version 2.0.0 went: Nuts 2 was a study project written in spring 2020, which used nornir, but not pytest to conduct tests.

Features of nuts 3.0.0

Write test bundles as yaml files that contain a desired state of the network. Nuts comes with a basic set of test definitions - see the documentation for details. Nuts then checks the desired state against the actual state of the network using pytest.
Enhance nuts: Nuts is designed as pytest plugin. You can write your own, custom test definitions and test bundles to suit your specific needs. Note: It is currently not supported (but nevertheless possible), since future releases might introduce breaking changes.
If you write your own test definitions/test classes and want to share it with the community, feel free to contact the developers and they will happily link to it in the repo readme.

Planned for 3.0.1

Improve error handling - the current implementation has an opaque error handling and is not really informative or helpful.
Refactor self-tests - the current self-tests are hard to read for non-devs and must be improved.

Planned for 4.0.0 (aka the far future)

Nuts is deeply intertwined with pytest to create test classes out of the yaml file. It digs deep into pytest to use its functionalities without really committing to pytest. It smells of future problems with project growth. Version 4 will try to solve that, but this may also introduce breaking changes.

v3.0.0-alpha-1 2021-04-27 06:38:06

NUTS has been completely rewritten and now released in a very early version: Nuts 3 uses pytest and nornir to perform network configuration checks against a real network.

Initial code is by @MatthiasGabriel, @ubaumann and @bytinbit, the project is now maintained by @bytinbit.

Many thanks to @The-Compiler who provided invaluable feedback.

Please note that this repository here is a mirror of another repo.

Nuts v1.2.1 2017-09-21 10:04:52

Fixed

Rename testSchema.yaml to test_schema.py

INS Institute for Network and Security

GitHub Repository

networking automation python pytest network-programming testing nornir hacktoberfest