Architecture
+Architecture
SlateDB is a log-structured merge-tree (LSM-tree). If you are unfamiliar with LSM-trees, we recommend reading the following resources:
- RocksDB Overview diff --git a/docs/configuration/index.html b/docs/configuration/index.html index 7635813..ba3d193 100644 --- a/docs/configuration/index.html +++ b/docs/configuration/index.html @@ -4,11 +4,11 @@
Configuration
+Configuration
Database Options
/// Configuration options for the database. These options are set on client startup.
#[derive(Clone)]
pub struct DbOptions {
/// How frequently to flush the write-ahead log to object storage (in
/// milliseconds).
///
/// When setting this configuration, users must consider:
///
/// * **Latency**: The higher the flush interval, the longer it will take for
/// writes to be committed to object storage. Writers blocking on `put` calls
/// will wait longer for the write. Readers reading committed writes will also
/// see data later.
/// * **API cost**: The lower the flush interval, the more frequently PUT calls
/// will be made to object storage. This can increase your object storage costs.
///
/// We recommend setting this value based on your cost and latency tolerance. A
/// 100ms flush interval should result in $130/month in PUT costs on S3 standard.
///
/// Keep in mind that the flush interval does not include the network latency. A
/// 100ms flush interval will result in a 100ms + the time it takes to send the
/// bytes to object storage.
pub flush_ms: usize,
/// How frequently to poll for new manifest files (in milliseconds). Refreshing
/// the manifest file allows writers to detect fencing operations and allows
/// readers to detect newly compacted data.
///
/// **NOTE: SlateDB secondary readers (i.e. non-writer clients) do not currently
/// read from the WAL. Such readers only read from L0+. The manifest poll intervals
/// allows such readers to detect new L0+ files.**
pub manifest_poll_interval: Duration,
/// Write SSTables with a bloom filter if the number of keys in the SSTable
/// is greater than or equal to this value. Reads on small SSTables might be
/// faster without a bloom filter.
pub min_filter_keys: u32,
/// The minimum size a memtable needs to be before it is frozen and flushed to
/// L0 object storage. Writes will still be flushed to the object storage WAL
/// (based on flush_ms) regardless of this value. Memtable sizes are checked
/// every `flush_ms` milliseconds.
///
/// When setting this configuration, users must consider:
///
/// * **Recovery time**: The larger the L0 SSTable size threshold, the less
/// frequently it will be written. As a result, the more recovery data there
/// will be in the WAL if a process restarts.
/// * **Number of L0 SSTs**: The smaller the L0 SSTable size threshold, the more
/// L0 SSTables there will be. L0 SSTables are not range partitioned; each is its
/// own sorted table. As such, reads that don't hit the WAL or memtable will need
/// to scan all L0 SSTables. The more there are, the slower the scan will be.
/// * **Memory usage**: The larger the L0 SSTable size threshold, the larger the
/// unflushed in-memory memtable will grow. This shouldn't be a concern for most
/// workloads, but it's worth considering for workloads with very high L0
/// SSTable sizes.
/// * **API cost**: Smaller L0 SSTable sizes will result in more frequent writes
/// to object storage. This can increase your object storage costs.
/// * **Secondary reader latency**: Secondary (non-writer) clients only see L0+
/// writes; they don't see WAL writes. Thus, the higher the L0 SSTable size, the
/// less frequently they will be written, and the longer it will take for
/// secondary readers to see new data.
///
/// We recommend setting this value to a size that will result in one L0 SSTable
/// per-second. With a default compaction interval of 5 seconds, this will result
/// in 4 or 5 L0 SSTables per compaction. Thus, a writer putting 10MiB/s of data
/// would configure this value to 10 * 1024 * 1024 = 10_485_760 bytes.
pub l0_sst_size_bytes: usize,
/// Configuration options for the compactor.
pub compactor_options: Option<CompactorOptions>,
}
Read Options
diff --git a/docs/introduction/index.html b/docs/introduction/index.html index 63b91a4..b06da67 100644 --- a/docs/introduction/index.html +++ b/docs/introduction/index.html @@ -4,11 +4,11 @@Introduction
+Introduction
SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes all data to object storage.
Vision
Object storage is an amazing technology. It provides highly-durable, highly-scalable, highly-available storage at a great cost. And recent advancements have made it even more attractive:
diff --git a/docs/quickstart/index.html b/docs/quickstart/index.html index 7df2282..a0126bc 100644 --- a/docs/quickstart/index.html +++ b/docs/quickstart/index.html @@ -4,11 +4,11 @@Quickstart
+Quickstart
SlateDB is a Rust library. It doesn't currently ship with any language bindings, so you must use Rust or generate your own bindings.
Installation
SlateDB is not published to crates.io yet. Add the following to your Cargo.toml:
An embedded database built on object storage.
Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage to provide bottomless storage capacity, high durability, and easy replication.
Object Store Durability
SlateDB inherits your object store's durability. Yup, 99.999999999% durability.
Zero-Disk Architecture
SlateDB runs without disks. No more disk failures, no more disk corruption.
Simple Replication
Why write complex replication protocols when you can let your object store handle it?
Tunable Performance
Configure SlateDB to optimize for low latency, low cost, or high durability.
Scalable Readers
Supports a single writer and multiple readers. SlateDB detects and fences zombie writers.
Built in Rust
SlateDB is an embeddable library built in Rust. Use SlateDB with the language of your choice.
Get Started
SlateDB currently only supports the Rust ecosystem, and is not published to crates.io. Add the code snippet to your dependencies and you are off to the races!
An embedded database built on object storage.
Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage to provide bottomless storage capacity, high durability, and easy replication.
Object Store Durability
SlateDB inherits your object store's durability. Yup, 99.999999999% durability.
Zero-Disk Architecture
SlateDB runs without disks. No more disk failures, no more disk corruption.
Simple Replication
Why write complex replication protocols when you can let your object store handle it?
Tunable Performance
Configure SlateDB to optimize for low latency, low cost, or high durability.
Scalable Readers
Supports a single writer and multiple readers. SlateDB detects and fences zombie writers.
Built in Rust
SlateDB is an embeddable library built in Rust. Use SlateDB with the language of your choice.