Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: added scripts/drop_compound_index.py #162

Merged
merged 5 commits into from
Aug 22, 2023
Merged

Conversation

viniarck
Copy link
Member

@viniarck viniarck commented Aug 7, 2023

Closes #160

I'll send a backport to 2022.3.2 shortly

Summary

See updated changelog for more information

Local Tests

  • On version 2022.3.2 I ran this script:
❯ CMD=drop_index python3 drop_compound_index.py      
None

If you try to run again, it'll error out as expect since the index switch_1_flow.cookie_1_state_1_inserted_at_1_updated_at_1 doesn't exist anymore:

❯ CMD=drop_index python3 drop_compound_index.py
pymongo error: index not found with name [switch_1_flow.cookie_1_state_1_inserted_at_1_updated_at_1], full error: {'ok': 0.0, 'errmsg': 'index not found with name [switch_1_flow.cookie_1
_state_1_inserted_at_1_updated_at_1]', 'code': 27, 'codeName': 'IndexNotFound', '$clusterTime': {'clusterTime': Timestamp(1691420918, 2), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}, 'operationTime': Timestamp(1691420918, 2)}
  • On master (future 2023.1) I ran this script as well:
❯ CMD=drop_index python3 drop_compound_index.py      
None
rs0 [direct: primary] napps> db.flows.getIndexes()
[
  { v: 2, key: { _id: 1 }, name: '_id_' },
  {
    v: 2,
    key: { flow_id: 1 },
    name: 'flow_id_1',
    background: true,
    unique: true
  },
  {
    v: 2,
    key: { 'flow.cookie': 1 },
    name: 'flow.cookie_1',
    background: true
  },
  {
    v: 2,
    key: { 'flow.priority': -1 },
    name: 'flow.priority_-1',
    background: true
  },
  { v: 2, key: { state: 1 }, name: 'state_1', background: true },
  {
    v: 2,
    key: {
      switch: 1,
      'flow.cookie': 1,
      state: 1,
      'flow.priority': -1,
      inserted_at: 1,
      updated_at: 1
    },
    name: 'switch_1_flow.cookie_1_state_1_flow.priority_-1_inserted_at_1_updated_at_1',
    background: true
  }
]
rs0 [direct: primary] napps> 
rs0 [direct: primary] napps> db.flows.countDocuments()
1009

During these local tests I always had 1000+ flows installed, no other issues were observed including during kytosd restarts

End-to-End Tests

I'll dispatch an execution, but I don't expect surprises.

@viniarck viniarck requested a review from italovalcy August 7, 2023 15:58
@viniarck viniarck requested a review from a team as a code owner August 7, 2023 15:58
("state", pymongo.ASCENDING),
("flow.priority", pymongo.DESCENDING),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be future work to further augment and/or optimize index #161, but for now I decided to only cover the minimum necessary to minimize changes and risk to impact on other queries. I've changed priority to be descending since that's how it's being sorted and queried when finding flows.

@viniarck viniarck merged commit f6e0ea9 into master Aug 22, 2023
@viniarck viniarck deleted the chore/db_index_script branch August 22, 2023 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

potential issue: upgrade from 2022.3.1 to 2022.3.2 didn't load the NApp
1 participant