Set timer to reduce load #424

Alopalao · 2024-01-29T11:32:38Z

Close #373

Summary

Set a timer to reduce load to database.
The timer will stop first event, while updating last received interface event. When timer ends, the last event will be applied.
Example in used:

2024-01-30 07:20:56,482 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_155) TIMING -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_down            <----- Start time here for 02:3
2024-01-30 07:20:56,487 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_130) RETURN -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,487 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_14) TIMING -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_down             <----- Start time here for 01:4
2024-01-30 07:20:56,489 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_112) RETURN -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,506 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_63) RETURN -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_down
2024-01-30 07:20:56,506 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_39) RETURN -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,508 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_22) RETURN -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_down
2024-01-30 07:20:56,509 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_73) RETURN -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,533 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_6) RETURN -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_down
2024-01-30 07:20:56,533 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_9) RETURN -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_down
2024-01-30 07:20:56,535 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_54) RETURN -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,566 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_54) RETURN -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_up
2024-01-30 07:20:56,583 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_155) PROCESS -> 00:00:00:00:00:00:00:02:3 -> kytos/of_core.switch.interface.link_up             <----- Finnished 02:3
2024-01-30 07:20:56,589 - WARNING [kytos.napps.kytos/mef_eline] (thread_pool_sb_14) PROCESS -> 00:00:00:00:00:00:00:01:4 -> kytos/of_core.switch.interface.link_up              <----- Finnished 01:4

End-To-End Tests

============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.4.0
rootdir: /tests
plugins: timeout-2.1.0, rerunfailures-10.2, anyio-3.6.2
collected 256 items

tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ....................                       [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 24%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 46%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 73%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]

viniarck

It's a good start, but there's a potential concurrency issue. Check out my comment.

Also, don't forget to update the changelog and ship unit tests once the implemenation is working as expected.

main.py

Alopalao · 2024-01-30T14:44:43Z

APM results:
Laps were sent every 1 second.
Previous mef_eline implementation. Peak 144tpm:

Current implementation, TIME_INTERFACE = 1. Peak 56tpm:

Current implementation, TIME_INTERFACE = 10. Peak 24tpm:

viniarck

Nicely done @Alopalao, it's looking great, very neat how it ended up not even needing a deque at this time, I asked trivial implementation detail changes though, check them out. Did you also have a chance to also run a stress link flap test with this last code while monitoring with the APM? (the chart with 1 flap / sec are looking great)

settings.py

main.py

viniarck

@Alopalao, excellent PR. It's great to see simple and elegant solutions like this one here. Consider it approved.

I'm just waiting these things before approving and merging it:

Fix linter errors
Make sure e2e mef eline tests are passing
Update changelog

main.py

tests/unit/test_main.py

viniarck

Leaving it pre-approved. It'll get merged once changelog is updated.

@Alopalao, in the last e2e you posted here there was a RRF on topology, it's probably unrelated, but investigate it and open an issue on e2e repo if needed. Thanks.

Alopalao · 2024-01-31T16:27:14Z

E2E test results. The previous result was from an outdated e2e repo.

============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.4.0
rootdir: /tests
plugins: timeout-2.1.0, rerunfailures-10.2, anyio-3.6.2
collected 256 items

tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ....................                       [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 24%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 46%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 73%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]

viniarck · 2024-01-31T16:29:12Z

E2E test results. The previous result was from an outdated e2e repo.

============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.4.0
rootdir: /tests
plugins: timeout-2.1.0, rerunfailures-10.2, anyio-3.6.2
collected 256 items

tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ....................                       [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 24%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 46%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 73%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]

Cool.

Regarding the changelog though, there's a git conflict if you can solve it.

viniarck

Nicely done. This PR will land by tomorrow.

Alopalao added 2 commits January 29, 2024 06:28

Set timer to reduce load

01b3776

Fixed last event not updating

cae1d2d

Alopalao marked this pull request as ready for review January 29, 2024 17:27

Alopalao requested a review from a team as a code owner January 29, 2024 17:27

viniarck requested changes Jan 29, 2024

View reviewed changes

main.py Outdated Show resolved Hide resolved

main.py Outdated Show resolved Hide resolved

Simplified database load restriction

0fd13cf

viniarck self-requested a review January 30, 2024 14:24

viniarck reviewed Jan 30, 2024

View reviewed changes

settings.py Outdated Show resolved Hide resolved

main.py Outdated Show resolved Hide resolved

main.py Outdated Show resolved Hide resolved

main.py Outdated Show resolved Hide resolved

main.py Show resolved Hide resolved

Added unit test

3665785

viniarck self-requested a review January 30, 2024 17:51

viniarck reviewed Jan 30, 2024

View reviewed changes

main.py Outdated Show resolved Hide resolved

viniarck reviewed Jan 30, 2024

View reviewed changes

tests/unit/test_main.py Show resolved Hide resolved

Tested out of order event

f15ddb7

viniarck self-requested a review January 31, 2024 15:34

viniarck approved these changes Jan 31, 2024

View reviewed changes

Updated changelog

7239ae8

Merge branch 'master' into database_load

9a12b9d

viniarck self-requested a review January 31, 2024 16:31

viniarck approved these changes Jan 31, 2024

View reviewed changes

viniarck merged commit 6305644 into master Feb 1, 2024
2 checks passed

viniarck deleted the database_load branch February 1, 2024 13:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set timer to reduce load #424

Set timer to reduce load #424

Alopalao commented Jan 29, 2024 •

edited

Loading

viniarck left a comment

Alopalao commented Jan 30, 2024 •

edited

Loading

viniarck left a comment •

edited

Loading

viniarck left a comment

viniarck left a comment

Alopalao commented Jan 31, 2024 •

edited

Loading

viniarck commented Jan 31, 2024

viniarck left a comment

Set timer to reduce load #424

Set timer to reduce load #424

Conversation

Alopalao commented Jan 29, 2024 • edited Loading

Summary

End-To-End Tests

viniarck left a comment

Choose a reason for hiding this comment

Alopalao commented Jan 30, 2024 • edited Loading

viniarck left a comment • edited Loading

Choose a reason for hiding this comment

viniarck left a comment

Choose a reason for hiding this comment

viniarck left a comment

Choose a reason for hiding this comment

Alopalao commented Jan 31, 2024 • edited Loading

viniarck commented Jan 31, 2024

viniarck left a comment

Choose a reason for hiding this comment

Alopalao commented Jan 29, 2024 •

edited

Loading

Alopalao commented Jan 30, 2024 •

edited

Loading

viniarck left a comment •

edited

Loading

Alopalao commented Jan 31, 2024 •

edited

Loading