Assessment/research to evolve e2e tests to support NoviFlow switches as a target platform

[viniarck]

Currently, only OvS is supported as a target when instantiating switches, it's desirable to evolve e2e target NoviFlow switches seamlessly via mininet too, so ideally the underlying grpc NoviFlow API should be used with a new topology. Ultimately, developers who write tests it shouldn't be that different and the API is expected to translate mininet commands/calls to the specific target.

@italovalcy will lead this effort, Italo, feel free to add more information and/or break down in more items as you think it's more convenient.

[italovalcy]

The assessment for evolving the e2e tests to support NoviFlow switches was finished, and as a result, we came up with a Mininet Switch abstraction class that can help us run the end-to-end tests in NoviFlow switch as a target platform. In this first phase, we focused on using the Virtual Noviflow Switches, but the way it was done made it simple to adapt and run on real switches.

The source code for the Noviflow integration is available here: https://github.com/kytos-ng/kytos-end-to-end-tests/tree/feature/noviswitch

The end-to-end testing code also has to be updated to be able to run transparently on Noviflow and OpenVSwitch: https://github.com/kytos-ng/kytos-end-to-end-tests/tree/feature/noviswitch_adapt_tests

After those changes, we can run the end-to-end tests with Noviflow target platform with the following sequence of steps:

1. Create a SSH key for authentication in the server which runs the Virtual Noviflow Switches (this ssh key will be used in the container which runs the pytest for end-to-end-tests -- NoviSwitch Mininet abstraction runs remote commands using Paramiko lib over SSH)
2. Import the SSH key into the container which runs end-to-end tests
3. Install openssh-client and paramiko python lib on the container, which runs end-to-end tests
4. Clone the branch for NoviSwitch and the end-to-end tests update (https://github.com/kytos-ng/kytos-end-to-end-tests/tree/feature/noviswitch_adapt_tests)
5. Setup an environment variable to enable NoviSwitch as the target platform and run the tests

export SWITCH_CLASS=NoviSwitch
./kytos-init.sh

During the tests, we faced many scalability issues. Some of the issues were already addressed by working with the Noviflow Support team (we deployed a new virtual machine and used a more powerful server). Some other issues still need to be investigated. One issue that needs more investigation is the test tests/test_e2e_10_mef_eline.py::TestE2EMefEline::test_080_create_and_remove_ten_circuits_ten_times. This test always fails and even running manually, the Noviflow virtual switch takes up to 30seconds to remove the flows for the created EVCs. I've also tested running with the Ryu controller, and the same behavior was seen. In the physical switches, we don't see such behavior. We will need to engage Noviflow on this problem.

The next steps to deploy such a solution would be:

1. Investigate issues we are having with test_080_create_and_remove_ten_circuits_ten_times . Maybe we can leverage a manual execution of ofctl/barrier (https://www.openvswitch.org/support/dist-docs/ovs-ofctl.8.txt) to make sure all the delete operations were executed
2. Migrate from SSH to gRPC to run the provisioning on the Noviflow switches
3. We have to come up with a plan on how to match the number of virtual switches we have and the requested switches on each topology for the tests (currently, I've added a static mapping on each one of the topologies we can run).

[italovalcy]

I've created Issue #200 to deal with all the details regarding the deployment of this feature into the testing pipeline.