Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate different NoSQL storage backends #25

Open
jsstevenson opened this issue Jan 27, 2023 · 5 comments
Open

Investigate different NoSQL storage backends #25

jsstevenson opened this issue Jan 27, 2023 · 5 comments
Labels
enhancement New feature or request keep alive exempt issue from staleness checks stale

Comments

@jsstevenson
Copy link
Contributor

ClinGen team found that Redis wasn't cost-effective for caching at scale. They moved to RocksDB -- we may want to consider moving our NoSQL support efforts in that direction.

@jsstevenson jsstevenson added the enhancement New feature or request label Jan 27, 2023
@holtgrewe
Copy link

In case it is useful: I'm using the following utils code when bulk importing variants into rocksdb (in rust). In particular when you import all of gnomad, you will see memory usage issues unless you use hierarchical index data structures and bloom filters.

https://github.com/bihealth/rocksdb-utils-lookup

HTH

@jsstevenson
Copy link
Contributor Author

@holtgrewe nice! We expect to focus on relational storage for now, but we do want to reevaluate that at some point (this may be crazy, but we'd like to maintain support for differing backends depending on whether users want to optimize for more complex searches or for pure key-value retrieval)

@theferrit32
Copy link
Contributor

@holtgrewe this is useful, thanks. We have hit issues before with rocksdb memory growth when using the default config and had to make similar config changes.

https://github.com/clingen-data-model/genegraph/blob/e43086b7efb393013759cecb260eb705470191e6/src/genegraph/rocksdb.clj#L21-L28

I haven't tried tweaking the bloom filter settings, but the use case above is not trying to optimize for reads, it's doing a lot of writing too.

Copy link

github-actions bot commented Jan 4, 2024

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Copy link

github-actions bot commented May 1, 2024

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the stale label May 1, 2024
@larrybabb larrybabb added the keep alive exempt issue from staleness checks label May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request keep alive exempt issue from staleness checks stale
Projects
None yet
Development

No branches or pull requests

4 participants