Nostrdb

[alltheseas]

This is a technical (non user-centered) story/ticket

Overview

Nostrdb should enable performance improvements across the board inluding, but not limited to:
-reduced memory use
-more intelligent querying of relays (do not re-query/re-fetch same events over, and over)

it replaces our in-memory event storage with lmdb-backed flatbuffer events. incoming events get encoded as flatbuffers and stored in lmdb. anytime we need to access a note we have direct memory-mapped data backed with the fastest database in the world

https://github.com/google/flatbuffers/tree/master/swift

https://github.com/apple/swift-lmdb/tree/main

better way to do protobuffers

expected benefits & unlocks

Local DB
Increased memory efficiency

User stories enabled by nostrdb:

1. Multiple profiles (accounts) Multiple Profiles (multiple accounts) #403
2. 👻 mode (view as): "👻 mode" / "View as" toggle in other users profile #1157
3. Gossip model (consume relay hints NIP 19) Consume relay hints NIP-19 #1147
4. Negentropy syncing (?)

dev commentary

initial nostrdb code with swift bindings is in the codebase now if anyone wants to check it out.

it just implements the mapping to the compact representation in memory and some initial broken yet fast json->compact conversion. see NdbTests

I was surprised how far I could go with zero-copy perf in swift. eventually when you need a String you can get one with a copy but you can do lots of stuff without building a String, like iterating through tags to find a specific ref, etc.

but yeah this sets us up for a pretty insane performance boost. the idea is incoming json gets verified and is immediately encoded into a compact representation that can exist in lmdb or in memory. the NdbNote class provides a swift-friendly interface to accessing this data. Think of NdbNote as an eventual NostrEvent replacement.

so incoming json get immediately inserted into lmdb virtual memory. querying will be as fast as a hashtable lookup and we won't need to convert between the database object and the swift representation. this is going to be so cool

oh yeah and re: nostrdb, this should mean that we will get a pretty insane memory usage reduction. we no longer need to store megs of events with tons of strings and arrays on the heap. Tags are now iterators into the compact representation

I think we can do fulltext search in nostrdb as well by mapping words to notes (inverted index)

[alltheseas]

Implemented for profiles

Note nostrdb implementation is pending

[alltheseas]

migrate to local relay #1851

[jb55]

Closing in favor of

• Local Relay Model (nostrdb) #2041