[Feature] index stored_flows by cookie

[viniarck]

Fixes #34

Description of the change

Added

• Added lock to avoid race flow mod race conditions in the consistency check

Changed

• stored_flows are now indexed by cookie, issue Adapt the flow_list to store the flow states indexed by cookie instead of OF commands.  #34
• Changed the flow_persistence data structured on storehouse
• Refactored the consistency checks methods accordingly to use cookie indexes

Deprecated

• The prior flow_persistence data structure isn't supported anymore. It's required to delete the kytos.flow.persistence folder, upgrading won't be supported this time.

Benchmark

• I've run some executions to measure how long the consistency check was taking to confirm some speedups that we would expect and stressed tests with 500 flows on dpid 1 using cookies with a relative random distribution, by installing them and monitoring how long it was taking for the consistency checks to run:

• In the execution below using the current master branch, you can see that the consistency check on 00:00:00:00:00:00:00:01 was taking 6+ seconds, while the other switches that had fewer flows < 100ms:

2021-11-05 16:48:05,054 - INFO [kytos.napps.kytos/flow_manager] (Thread-1816) check_consistency on switch 00:00:00:00:00:00:00:02 has started
2021-11-05 16:48:05,084 - INFO [kytos.napps.kytos/flow_manager] (Thread-1816) check_consistency on switch 00:00:00:00:00:00:00:02 is done
2021-11-05 16:48:05,075 - INFO [kytos.napps.kytos/flow_manager] (Thread-1815) check_consistency on switch 00:00:00:00:00:00:00:03 has started
2021-11-05 16:48:05,155 - INFO [kytos.napps.kytos/flow_manager] (Thread-1815) check_consistency on switch 00:00:00:00:00:00:00:03 is done

2021-11-05 16:48:06,154 - INFO [kytos.napps.kytos/flow_manager] (Thread-1833) check_consistency on switch 00:00:00:00:00:00:00:01 has started
2021-11-05 16:48:13,234 - INFO [kytos.napps.kytos/flow_manager] (Thread-1833) check_consistency on switch 00:00:00:00:00:00:00:01 is done

2021-11-05 16:49:04,921 - INFO [kytos.napps.kytos/flow_manager] (Thread-2158) check_consistency on switch 00:00:00:00:00:00:00:02 has started
2021-11-05 16:49:04,959 - INFO [kytos.napps.kytos/flow_manager] (Thread-2158) check_consistency on switch 00:00:00:00:00:00:00:02 is done
2021-11-05 16:49:04,934 - INFO [kytos.napps.kytos/flow_manager] (Thread-2159) check_consistency on switch 00:00:00:00:00:00:00:03 has started
2021-11-05 16:49:04,970 - INFO [kytos.napps.kytos/flow_manager] (Thread-2159) check_consistency on switch 00:00:00:00:00:00:00:03 is done

2021-11-05 16:49:06,360 - INFO [kytos.napps.kytos/flow_manager] (Thread-2179) check_consistency on switch 00:00:00:00:00:00:00:01 has started
2021-11-05 16:49:15,231 - INFO [kytos.napps.kytos/flow_manager] (Thread-2179) check_consistency on switch 00:00:00:00:00:00:00:01 is done

• In the execution below using this current branch of this PR, you can see that the consistency check on 00:00:00:00:00:00:00:01 is always under < 100ms, and now we also have additional locks which trade off correctness for a bit of latency exec, I think in the future we can try do it in parallel once we can leverage a more powerful DB:

2021-11-05 17:07:45,682 - INFO [kytos.napps.kytos/flow_manager] (Thread-1430) check_consistency on switch 00:00:00:00:00:00:00:02 has started
2021-11-05 17:07:45,711 - INFO [kytos.napps.kytos/flow_manager] (Thread-1430) check_consistency on switch 00:00:00:00:00:00:00:02 is done
2021-11-05 17:07:45,779 - INFO [kytos.napps.kytos/flow_manager] (Thread-1431) check_consistency on switch 00:00:00:00:00:00:00:03 has started
2021-11-05 17:07:46,064 - INFO [kytos.napps.kytos/flow_manager] (Thread-1431) check_consistency on switch 00:00:00:00:00:00:00:03 is done
2021-11-05 17:07:46,854 - INFO [kytos.napps.kytos/flow_manager] (Thread-1451) check_consistency on switch 00:00:00:00:00:00:00:01 has started
2021-11-05 17:07:46,890 - INFO [kytos.napps.kytos/flow_manager] (Thread-1451) check_consistency on switch 00:00:00:00:00:00:00:01 is done

2021-11-05 17:08:45,638 - INFO [kytos.napps.kytos/flow_manager] (Thread-1783) check_consistency on switch 00:00:00:00:00:00:00:03 has started
2021-11-05 17:08:45,687 - INFO [kytos.napps.kytos/flow_manager] (Thread-1783) check_consistency on switch 00:00:00:00:00:00:00:03 is done
2021-11-05 17:08:45,753 - INFO [kytos.napps.kytos/flow_manager] (Thread-1784) check_consistency on switch 00:00:00:00:00:00:00:02 has started
2021-11-05 17:08:45,796 - INFO [kytos.napps.kytos/flow_manager] (Thread-1784) check_consistency on switch 00:00:00:00:00:00:00:02 is done
2021-11-05 17:08:46,043 - INFO [kytos.napps.kytos/flow_manager] (Thread-1797) check_consistency on switch 00:00:00:00:00:00:00:01 has started
2021-11-05 17:08:46,088 - INFO [kytos.napps.kytos/flow_manager] (Thread-1797) check_consistency on switch 00:00:00:00:00:00:00:01 is done

Here's the confirmation of the 500+ stored flows:

❯ curl http://127.0.0.1:8181/api/kytos/storehouse/v1/kytos.flow.persistence/7ad9270176924bcb98a1eeb9170890a2 | jq '.flow_persistence["00:00:00:00:00:00:00:01"]' | jq length
502

[italovalcy]

Looks good to me. I just made a few comments to collaborate with the proposed solution.

[italovalcy]

Looks good to me! Thanks @viniarck for this great contribution!

[viniarck]

Thank you both for reviewing. I'll merge this PR shortly.