OpenFlow multi-table pipeline processing

[Alopalao]

This discussion is for issue #186: Blueprint for OpenFlow multi-table pipeline processing
First I wanted to confirm some concepts here before beginning implementation.

Issue

Implement Multi-table pipeline processing which requires adjustments on mef_eline, of-lldp, coloring. The concept is installing multiple flows in a switch with different table number which will match with packets. This will distribute the packets to different flows allowing simultaneous processing.
(Image simplified representation, maybe not skipping tables?)

Specification

There are sections on the Openflow specification that describe the openflow pipeline processing on the switches. The most important parts in my opinion are the rules for consistency on the pipeline (section 5.1.1 and forward) and required instruction, next-table-id (section 5.9).

Consistency

There are consistency rules to follow so its ensure the network functionality. Table-miss should be present on every flow table thus inconsistency will likely cause some packets to be dropped.

• What is going to be the matching criteria to forward the packets to the next table?
• Also are we going to allow users to set table number to flows? We will probably need to check consistency to restrict inputs that will not make sense on the installed topology.

Next table

According to the specifications, the table number always goes up.

• Are we allowed to skip a table or more? It looks like a great opportunity to take opportunity of the pipeline. If that is the case what type of rule would require such exclusivity to be processed on its own table?

Thoughts:

• Looks like dl_vlan is not the best match criteria for multi tables since we have priority already.

[viniarck]

What is going to be the matching criteria to forward the packets to the next table?

Each NApp that pushes flows, if it's configured to use a table other than 0, it's expected that this NApp will also install one or more flows in the table zero to forward to whichever table it's using. It's also desirable that the forwarding match from table 0 to its "X" table to be as specific as possible whenever applicable, just so this minimize interference with other NApps that might also be using a different table. In a case where there are multiple flows with different types of matches, like mef_eline, maybe we'll need a more generalized match trying to match all with a low priority (aligned with a priority that this NApps already uses). In your illustrated example, you're highlighting mef_eline using more than two tables. Theoretically, with OpeFlow that should be doable, but, in practice, from what I understand of the use cases that we're trying to cover, for the multi-table feature, from a perspective of a single NApp or feature, it's expected that it'll use only table 0 + another one to simplify.

Ultimately, I'll suggest you to document how each of these NApps will have matches when using another table than 0. To get this started, I'll encourage you to get started with one of the simplest that is of_lldp:

 cookie=0xab00000000000001, duration=259.679s, table=0, n_packets=86, n_bytes=3612, send_flow_rem priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535

If you look on of_lldp matching fields, dl_type is always expected to be 0x88cc, and dl_vlan is 3799 configured by default. So, you could potentially follow the same approach, except the actions would be actions=goto_table:X, where X is the table this NApp is configured to be, and then on table X, it pushes its flow forwarding to the controller.

Ultimately, one of the advantages of also having this multi table feature, is that network engineers are allowed to partition their tables with respective matching (with wildcard/mask or not) and actions capabilities, to achieve a configuration that will allow them to maximize the number of supported flow entries. So, once network engineers know that for instance that of_lldp will always need to math on dl_type and optionally on dl_vlan, they will know upfront that at least dl_type match needs to be supported on table 0 to use this feature, so they have to take this into account when engineering their pipelines.

We're implementing this feature generically, but AmLight is driving it, @italovalcy is very experienced with how pipeline and its partitioning works, he has presented some information about it before, that we could cross reference here to get an idea (please ask him to get this info). Italo will be one of the reviewers of this issue discussion, so ultimately we can double check if the matches/actions being used will end up being also aligned with the expectations/use case.

[viniarck]

Also are we going to allow users to set table number to flows? We will probably need to check consistency to restrict inputs that will not make sense on the installed topology.

I think we should only keep and maintain a single option on settings on which table to use, if it's different than 0, then the NApp can derive it's using multi-table. Which tables are being configured should be responsibilities of network engineers. The configured valued should be a positive int from 0 to OFPTT_MAX in this iteration that we'll ship. We can do the best that we can to keep specific matches but if they end up configuring multiple NApps using the same table, where these NApps also have generic matching flows then it can end up conflicting, however the priority we've been using for flows tend to avoid conflicts in the first place.

• coloring will have specific match
• of_lldp will have specific match
• mef_eline has very different possible set of match flows, I have an impression that it'll need a generic "match any" flow.

On Kytos-ng version 2023.1 we're also going to have a new NApp telemetry_int, that depends on mef_eline and pretty much works hand-in-hand with it. Notice that telemetry_int is already using multi-table flow entries. As you can read more in its blueprint, it's assuming in this version that mef_eline will be on table 0, and it's pushing higher priority flows on table 0, in addition to other flows on table 2. Since this is still being built in this version, we might not going to be able to make it configurable in this version considering our development team development dependencies. But this is something that needs to be aligned, does telemetry_int also ever plan to use any other table than 0 and 2 in the next version 2023.2? Please, align it with @jab1982, just so we keep have this on our radar too.

[viniarck]

According to the specifications, the table number always goes up.

Are we allowed to skip a table or more? It looks like a great opportunity to take opportunity of the pipeline. If that is the case what type of rule would require such exclusivity to be processed on its own table?

Since each NApp will only use table 0 and only another extra table, then we got part of this guaranteed and simplified.

[viniarck]

Also this simplifies any potential margin for inconsistencies that you highlighted in terms of table-miss flow entry

[viniarck]

@Alopalao since sdntrace_cp is also a dependency of mef_eline, it's good to also verify that it'll continue to work, at the moment it's querying for all stored flows, so in a different table it should still continue to work.

[italovalcy]

Hello @viniarck and @Alopalao, what a great discussion here! Please consider my half-bit contribution:

Goal: optimize resources for the switch match-action pipeline by 1) having more specific match fields and exact match for each table (which allows more entries to be allocated to a table on Tofino); 2) splitting the packet processing pipeline into multiple tables and aggregating entries throughout the multiple tables (which makes it possible to have one entry on table 1 that correspond to multiple entries on table 0, saving space).

The ideal solution will require us to define a very complex pipeline to be really optimized. We will not pursue the ideal solution now, that will be delivered as part of the migration to P4 (another ongoing project).

For now, the best way would be to optimize only what is easy to do and create the infrastructure to support multiple tables. From my point of view, optimizing what is easy to do means choosing the Napps that have a very well-defined set of matches and actions, allowing us to create tables with an exact match for each one (see examples later). On the other hand, creating the infrastructure means: creating table-miss entries to ensure the packet forwarding process will work smoothly and the consistency routine will do its job correctly. Other Napps that could benefit from this are BFD, OF-LLDP, Mirror, coloring, etc.

Thus, it looks like what we actually need is:

• Napps must support a configuration option to define which table they will send flow-mods to
• The network operator should define the rules to navigate between tables. This config will define how table miss entires will be installed (if any). Illustrative example: [{"table": 0, "priority": 100, "match": {}, "next_table": 1}, {"table": 1, "priority": 10000, "match": {"dl_vlan": 3379}, "next_table": "4"}, ...]. Maybe we can have another Napp handle this or add this responsibility to flow_manager. This config will be translated to flows to be installed in the switch (with consistency guarantees).
• flow_manager must be aware of multiple tables, and the consistency check should work correctly. Using the config above, we could even check for the existence of the requested tables (fail to reload the config in that case)
• flow_stats must be aware of the multiple tables. We should add the TableStats request (along with FlowStats and PortStats)

Note that, as mentioned by @viniark before, another approach would be decentralizing the configuration of the multi-table pipeline and delegate to each Napp the creation of the flows. My only concern with this option is to make it more complex to manage.

[viniarck]

Since it's desirable for one NApp to potentially be able to use "n" tables with flow misses working smoothly, here's my 2 cents:

• It makes a lot of sense to introduce a new NApp with the following responsibilities: 1) install flow miss entries on each table accordingly to a pipeline that could be set, that way all of the complexity of chaining tables and flow misses are simplified and not fully distributed so boundaries and responsibilities are clear, also network engineers will have the benefit to have everything well defined and clear in a single place when it comes to architecting and setting the pipeline and tables (this assumes that a single pipeline existing at a time is sufficient). 2) also potentially centralize on this NApp to configure each NApp on which tables should be used when pushing flows for a particular functionality 3) be responsible to also publish events to each NApp informing them which of their "match functionalities" when applicable like 'all', 'epl', 'evpl' (this is just an example, we'll need to properly group them and name) which table it should use. This also implies that each NApp who pushes flows should also listen for events and document which sort of functionalities it can parametrize to a particular table (if a NApp is simple enough where it only supports just one table then we could define a default key like 'all' or something similar). All of this must be set as early as possible though, otherwise NApps might still push flows to a wrong table as they're getting loaded.
• I wouldn't recommend to have a NApp to try to proxy requests or events to flow_manager, just so we keep the critical IO hot-path easy optimizable and each NApps being responsible to batch accordingly. We can keep flow_manager as it is without extra responsibilities, which already supports goto_table instruction with consistency check.
• If we go with this approach of having a centralized NApp that will early on also publish the configuration for NApp the order of how NApps are loaded will matter, and we might need to also slightly improve a part here where maybe we should initialize all of the event listeners first just to events can't be lost, and then continue to start each NApp thread as we've been doing

What do you guys think? @Alopalao @italovalcy. With that said, we'll also need to clearly specify how much we'll implement on this release, we could maybe only implement the listeners as if this new NApp will be introduced in the future, with the clear events and listening for it, or maybe we could also introduce a simplified version of it with a basic pipeline, this needs further discussion @italovalcy @Alopalao

[viniarck]

flow_stats (future kytos_stats) with TableStats is being addressed on this blueprint #351 that @gretelliz is authoring. Gretel, you might need to also augment some flow stats related endpoint to also include which table though, and adapt the openapi.yml spec since currently which table each flow belong we haven't included in the structure yet.

[italovalcy]

Here one scenario for example. Considering the following napps and their match fields:

mef_eline: in_port + vlan; in_port
of_lldp: ether_type, vlan, in_port (future)
mirror: (must be flexible, supporting more criteria)
telemetry_int: (flows will be created based on mef_eline)
bfd: vlan (untagged), in_port, ether_type, ip_proto, udp_port
coloring: dl_src

[viniarck]

This example and illustration was very helpful, Italo.

[Alopalao]

Looks like we are inclining for a new NApp few operation. Its responsibilities are related to flow_manager. Also, it is desirable to not send request or events to flow_manager. But when I roughly broke down the steps to achieve these responsibilities, I see that there is a necessity to communicate to flow_manager.

• Ensures the best table for a match criteria. Process:

  1. Listener gets flow information from other NApps.
  2. Gets the flows from the switch (from flow_manager)
  3. Decides the table number on specified rules (e.g. table number always goes up)
  4. Send flow with table to be installed by flow_manager or Send table number result to previous NApp.

• Ensures that each table has a miss/default flow (install flow):

  1. When a flow is received check the miss table, needs to be created or changed or deleted*?.
  2. If so, send two flows to flow_manager to be installed or one removed. One will be the miss/default flow. Maybe send these information to the previous NApp too?

From asked requirements, I see that the best option is to communicate back and forth to the NApp that wants to install certain flow. Overall, this is just speculating from my part. To understand this better, how would the process work for this new Napp in terms of handling the data and communication with flow_manager?
Finally, what will be the name?

[viniarck]

@Alopalao, since this feature is turning out to be larger than expected, it might need a blueprint since we'll need a minimum viable product (scope) and then also have clear scope for how it'll evolve in the future. Initially I had the impression it'd be just an issue discussion since I had the impression that a NApp would only use table 0 + only one extra table which isn't the case.

It's OK to continue discussing here, but we'll need to move this as a blueprint, OK? Thanks.

[Alopalao]

Got it.

[Alopalao]

• What Network Engineers should do when they ever need to change the pipeline?

Could we provide to engineers with a copy of the pipeline to be configurable before being deployed? This would be out of scope and will require API requests. My idea is to have pipelines to spare, stored (after being analyzed by multi_tables) and able to be deployed as a chunk of flows sent to flow_manager. Currently the table for each NApp are established but maybe once they are modifiable.

[viniarck]

Could we provide to engineers with a copy of the pipeline to be configurable before being deployed? This would be out of scope and will require API requests. My idea is to have pipelines to spare, stored (after being analyzed by multi_tables)

I think it's a great idea, we need to confirm with network engineers how they will plan to switch to a new pipeline and confirm this workflow. But, yeah, structing this new NApp in a way where new pipelines can be created/listed/deleted makes a lot of sense. However we still need to differentiate creation from activation. We'll need to clearly define what we'll mean by activation of a pipeline. Currently the easiest way to activate a new pipeline is right after a kytosd restart and NApps getting load and switches completing their handshake. Potentially in the future, we might need to also offer a deactivation controlled mechanism, basically removing flow miss tables that belong to the current pipeline and publish events to NApps to also remove their flows.

[viniarck]

@Alopalao another requirement we need to confirm with @italovalcy is if the same pipeline being architected will be equally used in all switches or not. It seems that will be the case, but if it's not, it'll increase greatly the complexity of managing all of that (and would need another assessment to confirm viability).

[Alopalao]

Does this diagram looks right? For the blueprint, I want to make sure the involvement of multi_tables in the process of creating the table miss flow entries.

EDIT: This diagram was thought under the impression that the table system is different for each switch which out of the scope and, therefore, incorrect.

[viniarck]

@Alopalao that could be an approach, however I wouldn't recommend structuring this way. Essentially this way here is making multi_table as a proxy to flow_manager, which is something that'd recommend to avoid, this make error handling harder and to optimize and batch flow requests will be very difficult.

Please, take a look on one of my previous replies. multi_table or multi_tables since it'll own the pipeline it can push the flow entries directly to flow_manager, and then it can instruct (publish) events to NApps early on (or whenever a new pipeline is activate) telling them which table they should use for a particular feature set (something that I was calling table groups), see if that makes sense, we can have a call if that makes easier to discuss the ideas.

[Alopalao]

The following was settled:

• The table system is global. This means that every switch will have the same pipeline configuration
• The priority of the miss flow entries is higher than zero but should be very low.

With this new information I made another diagram but first few things to clarify:

• multi_table will post miss flow entries for every table in every switch. The event KytosEvent("of_core.handshake.completed") gives enough information to install these flows.
• Since the priority of miss flow entries is not 0 there is a possible scenario that a flow with lower priority is installed by the user. Maybe it can be a future restriction but for now multi_table will try to manage this scenario by listenning to these flows and adjusting the existent miss flow entry by deleting it and adding a new one.

[viniarck]

@Alopalao, overall looks good, nicely done.

Let's start to move it to a blueprint now that most of the requirements have been sorted out.

Parts remaining here is also to define the concept of activating a pipeline (to make it clear how it'll work, and how deletions are expected to happen), and how many will be allowed to be configured. Also did you confirm with Italo that indeed we're dealing with a pipeline that's expected to be used by all switches?

[Alopalao]

Yes, it should be a global pipeline.

[viniarck]

Got it. Cool.

[viniarck]

Closing this discussion. PR landed on #359