Skip to content

Commit

Permalink
Merge pull request #146 from kytos-ng/chore/remove_active_from_collec…
Browse files Browse the repository at this point in the history
…tions

chore: removed `active` from models and included `scripts/unset_active.py`
  • Loading branch information
viniarck authored Jun 22, 2023
2 parents a40348a + 6fd4200 commit f975191
Show file tree
Hide file tree
Showing 5 changed files with 199 additions and 9 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ Fixed
Changed
=======
- Stopped storing interface and link ``active`` field in the DB
- Removed ``active`` from the application DB models

General Information
===================
- ``@rest`` endpoints are now run by ``starlette/uvicorn`` instead of ``flask/werkzeug``.
- Added ``scripts/unset_active.py`` to ``$unset`` ``active`` from ``links`` and ``switches`` collections that will no longer be in the database. If you are upgrading to ``2023.1`` you should run this script, however, if you don't, it'll still work as intended since the application won't read from or update these ``active`` values

[2022.3.0] - 2022-12-15
***********************
Expand Down
5 changes: 0 additions & 5 deletions db/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ class InterfaceSubDoc(BaseModel):

id: str
enabled: bool
active: bool
mac: str
speed: float
port_number: int
Expand All @@ -48,7 +47,6 @@ class SwitchDoc(DocumentBaseModel):
"""Switch DB Document Model."""

enabled: bool
active: bool
data_path: Optional[str]
hardware: Optional[str]
manufacturer: Optional[str]
Expand All @@ -73,7 +71,6 @@ def projection() -> dict:
"_id": 0,
"id": 1,
"enabled": 1,
"active": 1,
"data_path": 1,
"hardware": 1,
"manufacturer": 1,
Expand Down Expand Up @@ -106,7 +103,6 @@ class LinkDoc(DocumentBaseModel):
"""Link DB Document Model."""

enabled: bool
active: bool
metadata: dict = {}
endpoints: conlist(InterfaceIdSubDoc, min_items=2, max_items=2)

Expand All @@ -117,7 +113,6 @@ def projection() -> dict:
"_id": 0,
"id": 1,
"enabled": 1,
"active": 1,
"metadata": 1,
"endpoint_a": {"$first": "$endpoints"},
"endpoint_b": {"$last": "$endpoints"},
Expand Down
96 changes: 92 additions & 4 deletions scripts/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
## Topology's scripts
## `topology` scripts

This folder contains Topology's related scripts.
This folder contains Topology's related scripts:

### Data migration from `storehouse` to MongoDB
<details><summary><h3>Data migration from <code>storehouse</code> to MongoDB</h3></summary>

[`storehouse_to_mongo.py`](./storehouse_to_mongo.py) is a script to migrate the data entries from certain namespaces from `storehouse` to MongoDB.

#### Pre-requisites

- There's no additional Python libraries dependencies required, other than installing the existing `topology`'s requirements-dev.txt file.
- There's no additional Python libraries dependencies required, other than installing the existing `topology`'s, or if you're running in development locally then installing `requirements/dev.in`
- Make sure you don't have `kytosd` running with otherwise topology will start writing to MongoDB, and the application could overwrite the data you're trying to insert with this script.
- Make sure MongoDB replica set is up and running.
- Export MongoDB related variables that [db/client.py](../db/client.py) uses, make sure the hosts names can be resolved:
Expand Down Expand Up @@ -65,3 +65,91 @@ CMD=insert_links_metadata python3 scripts/storehouse_to_mongo.py
CMD=insert_switches_metadata python3 scripts/storehouse_to_mongo.py
CMD=insert_interfaces_metadata python3 scripts/storehouse_to_mongo.py
```

</details>

<details><summary><h3> <code>$unset active</code> from DB switches and links collections </h3></summary>

[`unset_active.py`](./unset_active.py) is a script to `$unset` `active` and certain metadata from `links` and `switches` collections.


#### Pre-requisites

- There's no additional Python libraries dependencies required, other than installing the existing `topology`'s, or if you're running in development locally then installing `requirements/dev.in`
- Make sure you don't have `kytosd` running with otherwise topology will start writing to MongoDB, and the application could overwrite the data you're trying to insert with this script.
- Make sure MongoDB replica set is up and running.
- Export the following MongnoDB variables accordingly in case your running outside of a container

```
export MONGO_USERNAME=
export MONGO_PASSWORD=
export MONGO_DBNAME=napps
export MONGO_HOST_SEEDS="mongo1:27017,mongo2:27018,mongo3:27099"
```

- The following `CMD` commands are available:

```
aggregate_unset_links
unset_links
aggregate_unset_switches_and_intfs
unset_switches_and_intfs
```

It's recommended that you run the `"aggregated_*"` commands first, just so you can preview the resulting aggregation with similar `$unset` key values. If the results of the aggregation are looking coherent, then you can proceed with the `"unset_*"` commands

#### Examples

- Previewing aggregated changes on `links` collection:

```
❯ CMD=aggregate_unset_links python scripts/unset_active.py
Aggregating links $unset active and metadata[last_status_is_active|last_status_change|notified_up_at]
{'_id': '4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30', 'enabled': True, 'endpoints': [{'id': '00:00:00:00:00:00:00:02:3'}, {'id': '00:00:00:00:00:00:00:03:2'}], 'id': '4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 51, 726000), 'metadata': {}, 'updated_at': datetime.datetime(2023, 6, 20, 20, 13, 53, 696000)}
{'_id': 'c8b55359990f89a5849813dc348d30e9e1f991bad1dcb7f82112bd35429d9b07', 'enabled': True, 'endpoints': [{'id': '00:00:00:00:00:00:00:01:4'}, {'id': '00:00:00:00:00:00:00:03:3'}], 'id': 'c8b55359990f89a5849813dc348d30e9e1f991bad1dcb7f82112bd35429d9b07', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 51, 730000), 'metadata': {}, 'updated_at': datetime.datetime(2023, 6, 20, 20, 13, 59, 707000)}
{'_id': '78282c4d5b579265f04ebadc4405ca1b49628eb1d684bb45e5d0607fa8b713d0', 'enabled': True, 'endpoints': [{'id': '00:00:00:00:00:00:00:01:3'}, {'id': '00:00:00:00:00:00:00:02:2'}], 'id': '78282c4d5b579265f04ebadc4405ca1b49628eb1d684bb45e5d0607fa8b713d0', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 51, 732000), 'metadata': {}, 'updated_at': datetime.datetime(2023, 6, 20, 20, 13, 53, 700000)}
```

- Running `$unset` to update many on `links` collection:

```
❯ CMD=unset_links python scripts/unset_active.py
Trying to $unset links 'active' and metadata[last_status_is_active|last_status_change|notified_up_at]...
Modified 3 links objects
```

- Running `$unset` to update many on `links` collection again, but expecting no changes:

```
❯ CMD=unset_links python scripts/unset_active.py
Trying to $unset links 'active' and metadata[last_status_is_active|last_status_change|notified_up_at]...
Modified 0 links objects
```

- Previewing aggregated changes on `switches` collection:

```
❯ CMD=aggregate_unset_switches_and_intfs python scripts/unset_active.py
Aggregating switches and interfaces $unset active
{'_id': '00:00:00:00:00:00:00:03', 'connection': '127.0.0.1:53680', 'data_path': 's3', 'enabled': True, 'hardware': 'Open vSwitch', 'id': '00:00:00:00:00:00:00:03', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 50, 467000), 'interfaces': [{'id': '00:00:00:00:00:00:00:03:4294967294', 'enabled': True, 'mac': 'c2:9d:dd:f0:f1:4f', 'speed': 0.0, 'port_number': 4294967294, 'name': 's3', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:03', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:03:1', 'enabled': True, 'mac': '6a:79:35:c4:9b:a3', 'speed': 1250000000.0, 'port_number': 1, 'name': 's3-eth1', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:03', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:03:2', 'enabled': True, 'mac': '2a:db:cc:f6:40:a0', 'speed': 1250000000.0, 'port_number': 2, 'name': 's3-eth2', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:03', 'link': '4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:03:3', 'enabled': True, 'mac': '86:62:23:d9:7e:06', 'speed': 1250000000.0, 'port_number': 3, 'name': 's3-eth3', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:03', 'link': 'c8b55359990f89a5849813dc348d30e9e1f991bad1dcb7f82112bd35429d9b07', 'link_side': None, 'metadata': {}, 'updated_at': None}], 'manufacturer': 'Nicira, Inc.', 'metadata': {}, 'ofp_version': '0x04', 'serial': 'None', 'software': '3.1.1', 'updated_at': datetime.datetime(2023, 6, 20, 20, 14, 48, 360000)}
{'_id': '00:00:00:00:00:00:00:02', 'connection': '127.0.0.1:53696', 'data_path': 's2', 'enabled': True, 'hardware': 'Open vSwitch', 'id': '00:00:00:00:00:00:00:02', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 50, 469000), 'interfaces': [{'id': '00:00:00:00:00:00:00:02:4294967294', 'enabled': True, 'mac': '7e:93:b8:64:eb:47', 'speed': 0.0, 'port_number': 4294967294, 'name': 's2', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:02', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:02:1', 'enabled': True, 'mac': '5a:e7:1b:02:f3:c3', 'speed': 1250000000.0, 'port_number': 1, 'name': 's2-eth1', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:02', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:02:2', 'enabled': True, 'mac': '32:75:61:02:93:a7', 'speed': 1250000000.0, 'port_number': 2, 'name': 's2-eth2', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:02', 'link': '78282c4d5b579265f04ebadc4405ca1b49628eb1d684bb45e5d0607fa8b713d0', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:02:3', 'enabled': True, 'mac': 'ea:a0:51:8a:e5:70', 'speed': 1250000000.0, 'port_number': 3, 'name': 's2-eth3', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:02', 'link': '4d42dc0852278accac7d9df15418f6d921db160b13d674029a87cef1b5f67f30', 'link_side': None, 'metadata': {}, 'updated_at': None}], 'manufacturer': 'Nicira, Inc.', 'metadata': {}, 'ofp_version': '0x04', 'serial': 'None', 'software': '3.1.1', 'updated_at': datetime.datetime(2023, 6, 20, 20, 14, 48, 362000)}
{'_id': '00:00:00:00:00:00:00:01', 'connection': '127.0.0.1:53674', 'data_path': 's1', 'enabled': True, 'hardware': 'Open vSwitch', 'id': '00:00:00:00:00:00:00:01', 'inserted_at': datetime.datetime(2023, 6, 20, 19, 54, 50, 551000), 'interfaces': [{'id': '00:00:00:00:00:00:00:01:4294967294', 'enabled': True, 'mac': '42:42:fb:8f:2b:44', 'speed': 0.0, 'port_number': 4294967294, 'name': 's1', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:01', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:01:4', 'enabled': True, 'mac': '72:bb:4f:ad:1f:22', 'speed': 1250000000.0, 'port_number': 4, 'name': 's1-eth4', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:01', 'link': 'c8b55359990f89a5849813dc348d30e9e1f991bad1dcb7f82112bd35429d9b07', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:01:1', 'enabled': True, 'mac': '22:2c:ae:78:ce:7d', 'speed': 1250000000.0, 'port_number': 1, 'name': 's1-eth1', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:01', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:01:2', 'enabled': True, 'mac': '72:5c:af:10:fe:05', 'speed': 1250000000.0, 'port_number': 2, 'name': 's1-eth2', 'nni': False, 'lldp': True, 'switch': '00:00:00:00:00:00:00:01', 'link': '', 'link_side': None, 'metadata': {}, 'updated_at': None}, {'id': '00:00:00:00:00:00:00:01:3', 'enabled': True, 'mac': '8e:d3:93:64:34:be', 'speed': 1250000000.0, 'port_number': 3, 'name': 's1-eth3', 'nni': True, 'lldp': True, 'switch': '00:00:00:00:00:00:00:01', 'link': '78282c4d5b579265f04ebadc4405ca1b49628eb1d684bb45e5d0607fa8b713d0', 'link_side': None, 'metadata': {}, 'updated_at': None}], 'manufacturer': 'Nicira, Inc.', 'metadata': {}, 'ofp_version': '0x04', 'serial': 'None', 'software': '3.1.1', 'updated_at': datetime.datetime(2023, 6, 20, 20, 14, 48, 359000)}
```

- Running `$unset` to update many on `switches` collection:

```
❯ CMD=unset_switches_and_intfs python scripts/unset_active.py
Trying to $unset switches and interfaces 'active'
Modified 3 switches objects
```

- Running `$unset` to update many on `switches` collection again, but expecting no changes:

```
❯ CMD=unset_switches_and_intfs python scripts/unset_active.py
Trying to $unset switches and interfaces 'active'
Modified 0 switches objects
```

</details>
103 changes: 103 additions & 0 deletions scripts/unset_active.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
import os

from kytos.core.db import Mongo


def unset_links(mongo: Mongo) -> None:
"""Unset links"""
print(
"Trying to $unset links 'active' and metadata[last_status_is_active|last_status_change|notified_up_at]..."
)
db = mongo.client[mongo.db_name]
res = db.links.update_many(
{},
{
"$unset": {
"active": 1,
"metadata.last_status_is_active": 1,
"metadata.last_status_change": 1,
"metadata.notified_up_at": 1,
}
},
)
print(f"Modified {res.modified_count} links objects")


def aggregate_unset_links(mongo: Mongo) -> None:
"""Aggregate unset links."""
db = mongo.client[mongo.db_name]
res = db.links.aggregate(
[
{
"$unset": [
"active",
"metadata.last_status_is_active",
"metadata.last_status_change",
"metadata.notified_up_at",
]
}
]
)
print(
"Aggregating links $unset active and metadata[last_status_is_active|last_status_change|notified_up_at]"
)
for doc in res:
print(doc)


def unset_switches_and_intfs(mongo: Mongo) -> None:
"""Unset switches and interfaces"""
print("Trying to $unset switches and interfaces 'active'")
db = mongo.client[mongo.db_name]
res = db.switches.update_many(
{},
{
"$unset": {
"active": 1,
"interfaces.$[].active": 1,
}
},
)
print(f"Modified {res.modified_count} switches objects")


def aggregate_unset_switches_and_intfs(mongo: Mongo) -> None:
"""Aggregate unset switches and interfaces."""
db = mongo.client[mongo.db_name]
res = db.switches.aggregate(
[
{
"$unset": [
"active",
"interfaces.active",
]
}
]
)
print(
"Aggregating switches and interfaces $unset active"
)
for doc in res:
print(doc)


if __name__ == "__main__":
mongo = Mongo()
cmds = {
"aggregate_unset_links": aggregate_unset_links,
"unset_links": unset_links,
"aggregate_unset_switches_and_intfs": aggregate_unset_switches_and_intfs,
"unset_switches_and_intfs": unset_switches_and_intfs,
}
try:
cmd = os.environ["CMD"]
cmds[cmd](mongo)
except KeyError:
print(
f"Please set the 'CMD' env var. \nIt has to be one of these: {list(cmds.keys())}"
)
sys.exit(1)
2 changes: 2 additions & 0 deletions tests/unit/test_topo_controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,8 @@ def test_upsert_switch(self) -> None:
arg1, arg2 = self.topo.db.switches.find_one_and_update.call_args[0]
assert arg1 == {"_id": self.dpid}
for key, value in switch_dict.items():
if key == "active":
continue
assert arg2["$set"][key] == value

def test_upsert_link(self) -> None:
Expand Down

0 comments on commit f975191

Please sign in to comment.