[Fix] Optimization in the consistency check to skip recent deleted flows

[viniarck]

Fixes #69

Description of the change

This PR introduced an optimization opportunity that @italovalcy has found on this comment, so, now in the same way that consistency tries to skip a pass if a recent flow was added it's doing the same thing for recent deleted flows. I've also cherry picked some commits from PR #45, that I worked sometime ago for archived flows, however, the main difference to avoid putting more pressure on store house is that I only kept the parts we'd need for this feature with storage in memory, so no additional IO/DB pressure is expected. I think it's a win-win situation since now we've got the archived flows partly landed as well, and then when we have a new NoSQL backend it's a matter of writing and reading from there.

Release notes

[2022.1.2] - 2022-02-07

Changed

• Adapted consistency check to skip recent deleted flows
• Extracted is_recent_flow static method
• Changed _del_matched_flows_store to also archive flows
• Changed consistency check to also archive alien flows

Added

• Added archived_flows and its lock to store in memory
• Added _add_Archived_flows method

Local tests

I've also tried to reproduce the issue with the commands and scripts that @italovalcy provided on this comment, in three executions, no alien deleted consistency checks were observed. Also, exploring archived_flows in memory, the 200 flow entries were there, all of them with delete reason:

kytos $> len(controller.napps[('kytos', 'flow_manager')].archived_flows['00:00:00:00:00:00:00:01'])
Out[8]: 200


kytos $> set([v['reason'] for v in controller.napps[('kytos', 'flow_manager')].archived_flows['00:00:00:00:00:00:00:01'].values()])
Out[9]: {'delete'}

[viniarck]

For the record, here's the result of end-to-end tests with this branch:

+ python3 -m pytest tests/
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /builds/amlight/kytos-end-to-end-tester/kytos-end-to-end-tests
plugins: timeout-2.0.2
collected 189 items
tests/test_e2e_01_kytos_startup.py ..                                    [  1%]
tests/test_e2e_05_topology.py .................                          [ 10%]
tests/test_e2e_10_mef_eline.py .........X.......x....x.....              [ 24%]
tests/test_e2e_11_mef_eline.py .....                                     [ 27%]
tests/test_e2e_12_mef_eline.py .....xx.                                  [ 31%]
tests/test_e2e_13_mef_eline.py .....xxxx......xxxx.XXxX.xxxx..x......... [ 53%]
...                                                                      [ 55%]
tests/test_e2e_14_mef_eline.py x                                         [ 55%]
tests/test_e2e_15_maintenance.py ........................                [ 68%]
tests/test_e2e_20_flow_manager.py ................                        [76%]
tests/test_e2e_21_flow_manager.py ..                                      [77%]
tests/test_e2e_22_flow_manager.py ...............                        [85%]
tests/test_e2e_23_flow_manager.py X......................                [ 97%]
tests/test_e2e_30_of_lldp.py ....                                        [100%]
=============================== warnings summary ===============================
------------------------------- start/stop times -------------------------------
==== 165 passed, 19 xfailed, 5 xpassed, 560 warnings in 9798.94s (2:43:18) =====

[italovalcy]

Looks good to me. It is very nice that you were able to quickly design a solution for this optimization. The set of tests presented in the PR description shows the solution is secure and effective. Thanks for fixing this, @viniarck

[viniarck]

Looks good to me. It is very nice that you were able to quickly design a solution for this optimization. The set of tests presented in the PR description shows the solution is secure and effective. Thanks for fixing this, @viniarck

Awesome. Thanks for reviewing, Italo.