TidesDB v0.1.0 BETA

[guycipher]

Official v0.1.0 beta release of TidesDB.

Features

• Concurrent multiple threads can read and write to the storage engine. The skiplist uses an RW lock which means multiple readers and one true writer. SSTables are sorted, immutable and can be read concurrently they are protected via page locks. Transactions are also protected via a lock.
• Column Families store data in separate key-value stores.
• Atomic Transactions commit or rollback multiple operations atomically.
• Cursor iterate over key-value pairs forward and backward.
• WAL write-ahead logging for durability. As operations are appended they are also truncated at specific points once persisted to an sstable(s).
• Multithreaded Compaction manual multi-threaded paired and merged compaction of sstables. When run for example 10 sstables compacts into 5 as their paired and merged. Each thread is responsible for one pair - you can set the number of threads to use for compaction.
• Background flush memtable flushes are enqueued and then flushed in the background.
• Chained Bloom Filters reduce disk reads by reading initial pages of sstables to check key existence. Bloomfilters grow with the size of the sstable using chaining and linking.
• Zstandard Compression compression is achieved with Zstandard. SStable entries can be compressed as well as WAL entries.
• TTL time-to-live for key-value pairs.
• Configurable many options are configurable for the engine, and column families.
• Error Handling API functions return an error code and message.
• Easy API simple and easy to use api.

Building

Using cmake to build the shared library.

cmake -S . -B build
cmake --build build
cmake --install build

Include

#include <tidesdb.h>

Usage

Each database method returns a tidesdb_err* which returns an error code and message. If no error, TidesDB returns NULL.

typedef struct
{
    int code;
    char* message;
} tidesdb_err_t;

Opening a database

To open a new database you need to create a configuration and then open the database.

tidesdb_config_t* tdb_config = (malloc(sizeof(tidesdb_config_t)));
if (tdb_config == NULL)
{
    /* handle error */
    return;
}

tdb_config->db_path = "the_dir_you_want_to_store_the_db"; /* tidesdb will create the directory if not exists */
tdb_config->compressed_wal = false; /* whether you want WAL(write ahead log) entries to be compressed */

tidesdb_t tdb = NULL;
tidesdb_err_t* e = tidesdb_open(tdb_config, &tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* close the database */
e = tidesdb_close(tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* free the configuration */
free(tdb_config);

Creating a column family

In order to store data in TidesDB you need a column family.
You pass

• the database
• the name of the column family
• memtable flush threshold in bytes. Example below is 128MB.
• skiplist max level. Example below is 12.
• skiplist probability. Example below is 0.24.
• whether column family data is compressed

/* create a column family */
tidesdb_err_t *e = tidesdb_create_column_family(tdb, "your_column_family", (1024 * 1024) * 128, 12, 0.24f, false);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Dropping a column family

/* drop a column family */
tidesdb_err_t *e = tidesdb_drop_column_family(tdb, "test_cf");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Putting a key-value pair

You pass

• the database
• the column family name
• the key
• the key size
• the value
• the value size
• when the key-value pair should expire. If -1 then it never expires.

/* put a key-value pair */
uint8_t key[] = "key";
uint8_t value[] = "value";

tidesdb_err_t *e = tidesdb_put(tdb, "your_column_family", key, strlen(key), value, strlen(value), -1);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Putting a key-value pair with TTL

/* put a key-value pair with TTL */
uint8_t key[] = "key";
uint8_t value[] = "value";

time_t ttl = time(NULL) + 10; /* 10 seconds */
tidesdb_err_t *e  = tidesdb_put(tdb, "your_column_family", key, strlen(key), value, strlen(value), ttl);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Getting a key-value pair

You pass

• the database
• the column family name
• the key
• the key size
• a pointer to the value
• a pointer to the value size

size_t value_len = 0;
uint8_t* value_out = NULL;
uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_get(tdb, "your_column_family", key, strlen(key), &value_out, &value_len);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Deleting a key-value pair

You pass

• the database
• the column family name
• the key
• the key size

uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_delete(tdb, "your_column_family", key, strlen(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Transactions

You can perform a series of operations atomically. This will block other threads from reading or writing to the database until the transaction is committed or rolled back.

You begin a transaction by calling tidesdb_txn_begin.

You pass

• the transaction
• the column family name

tidesdb_txn_t* transaction;
tidesdb_err_t *e = tidesdb_txn_begin(&transaction, "your_column_family");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Now we can add operations to the transaction.

const uint8_t key[] = "example_key";
const uint8_t value[] = "example_value";
tidesdb_err_t *e = tidesdb_txn_put(transaction, key, sizeof(key), value, sizeof(value), -1); /* you can pass a ttl, similar to put */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* you can add delete operations as well */
e = tidesdb_txn_delete(transaction, key, sizeof(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* now we commit */
e = tidesdb_txn_commit(tdb, transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* before you free, you can rollback */
tidesdb_err_t *e = tidesdb_txn_rollback(tdb, transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* free the transaction */
tidesdb_txn_free(transaction);

Cursors

You can iterate over key-value pairs in a column family.

tidesdb_cursor_t* c;
tidesdb_err_t *e = tidesdb_cursor_init(tdb, "your_column_family", &c);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
    return;
}

key_value_pair_t kv;

/* Iterate forward */
while ((e = tidesdb_cursor_next(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &kv);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use kv.key and kv.value */
    free(kv.key);
    free(kv.value);
}

if (e != NULL && e->code != 1062) /* 1062 means "At end of cursor" */
{
    /* handle error */
    tidesdb_err_free(e);
}

/* Iterate backward */
while ((e = tidesdb_cursor_prev(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &kv);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use kv.key and kv.value */
    free(kv.key);
    free(kv.value);
}

if (e != NULL && e->code != 1085) /* 1085 means "At beginning of cursor" */
{
    /* handle error */
    tidesdb_err_free(e);
}

tidesdb_cursor_free(c);

Compaction

You can manually compact sstables.

tidesdb_err_t *e = tidesdb_compact_sstables(tdb, "your_column_family", 10); /* use 10 threads */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Errors

Error Code	Error Message
1000	Failed to allocate memory for new db
1001	Config is NULL
1002	TidesDB is NULL
1003	DB path is NULL
1004	Failed to create db directory
1005	Failed to unlock flush lock
1006	Failed to join flush thread
1007	Failed to destroy flush lock
1008	Failed to sort sstables
1009	Failed to replay wal
1010	Failed to initialize flush queue
1011	Failed to copy memtable
1012	Failed to get wal checkpoint
1013	Failed to initialize column families lock
1014	Failed to start flush thread
1015	Column family name is NULL
1016	Column family name is too short
1017	Flush threshold is too low
1018	Max level is too low
1019	Probability is too low
1020	Failed to create new column family
1021	Failed to add column family
1022	Failed to lock column families lock
1024	Failed to acquire sstables lock
1025	Failed to unlock sstables
1026	Key is NULL
1027	Value is NULL
1028	Column family not found
1029	Max threads is too low
1030	Failed to acquire sstables lock
1031	Key not found
1032	Failed to create compaction thread
1033	Failed to allocate memory for thread arguments
1034	Failed to deserialize bloom filter
1035	Failed to initialize sstable cursor
1036	Failed to read sstable
1037	Failed to deserialize key value pair
1038	Key value pair is NULL
1040	Failed to signal flush condition
1041	Failed to load column families
1042	Failed to open wal
1043	Failed to destroy flush condition
1044	Failed to destroy column families lock
1045	Failed to allocate memory for queue entry
1046	Failed to initialize flush lock
1047	Failed to initialize flush condition variable
1048	Failed to reallocate memory for column families
1049	Failed to append to wal
1050	Failed to put into memtable
1051	Not enough sstables to compact
1052	Failed to allocate memory for transaction
1053	Failed to lock flush lock
1054	Transaction is NULL
1055	Failed to read bloom filter
1056	Failed to reallocate memory for transaction operations
1057	Failed to allocate memory for cursor
1058	Failed to initialize memtable cursor
1059	Failed to initialize sstable cursor
1060	Failed to get key value pair from cursor
1061	Cursor is NULL
1062	At end of cursor
1063	At start of cursor
1064	Key has a tombstone value. To be deleted on next compaction
1065	Key has expired. To be deleted on next compaction
1066	Key's value cannot be a tombstone
1067	Failed to allocate memory for wal
1068	Failed to lock compaction or flush lock
1069	Failed to allocate memory for value copy
1070	Failed to allocate memory for tombstone
1071	Transaction pointer is NULL
1072	Failed to allocate memory for column family name
1073	Failed to initialize transaction lock
1074	Failed to acquire transaction lock
1075	Failed to allocate memory for operation
1076	Failed to allocate memory for key-value pair
1077	Failed to allocate memory for key
1078	Failed to allocate memory for value
1079	Failed to allocate memory for rollback operations
1080	Failed to allocate memory for rollback column family name
1081	Failed to allocate memory for rollback key-value pair
1082	Failed to allocate memory for rollback key
1083	Failed to acquire memtable lock for commit
1084	Failed to skip initial pages
1085	At beginning of cursor

What's Changed

• updated read me contributions section by @guycipher in updated read me contributions section #9
• pager write and read corrections for page locking by @guycipher in pager write and read corrections for page locking #10
• minor corrections on AVL tree, WAL, and removing flush counter, useless by @guycipher in minor corrections on AVL tree, WAL, and removing flush counter, useless #11
• • correction within LSMT::flushMemtable to set and deset isFlushing by @guycipher in - correction within LSMT::flushMemtable to set and deset isFlushing #12
• PR#5 by @guycipher in PR#5 #13
• PR6 by @guycipher in PR6 #14
• • Remove table iteration standard output writes regarding pager sizes by @guycipher in - Remove table iteration standard output writes regarding pager sizes #16
• On reload of ssttables, sort last modified in descending order and cachedSize incrementation correction for skiplist by @guycipher in On reload of ssttables, sort last modified in descending order and cachedSize incrementation correction for skiplist #19
• • new pager instances shall be using std::ios::binary by @guycipher in - new pager instances shall be using std::ios::binary #20
• Fixes for reopening, recovery, and general clean up by @guycipher in Fixes for reopening, recovery, and general clean up #21
• Equi method optimizations and few comments by @guycipher in Equi method optimizations and few comments #22
• we should skip overflow pages and not break out of iterator because w… by @guycipher in we should skip overflow pages and not break out of iterator because w… #23
• c bindings by @guycipher in c bindings #24
• rough ffi implementations by @guycipher in rough ffi implementations #25
• additions to C bindings for WAL ( Wal_NewWithPath, Wal_Recover, Wal_C… by @guycipher in additions to C bindings for WAL ( Wal_NewWithPath, Wal_Recover, Wal_C… #26
• SkipList atomic operation corrections and retrieval operation concurr… by @guycipher in SkipList atomic operation corrections and retrieval operation concurr… #27
• Updates to datastructures, added more LSMT operations, clean up by @guycipher in Updates to datastructures, added more LSMT operations, clean up #28
• Removing skiplist tests, and some AVL tree enhancements, corrections by @guycipher in Removing skiplist tests, and some AVL tree enhancements, corrections #29
• LSMT constructor corrections, remove unused and wont be used functions and use hpp header extension for library by @guycipher in LSMT constructor corrections, remove unused and wont be used functions and use hpp header extension for library #30
• Compression, logging by @guycipher in Compression, logging #31
• correction on readme regarding compression by @guycipher in correction on readme regarding compression #32
• TTL implementation by @guycipher in TTL implementation  #34
• flushThreadFunc method correction by @guycipher in flushThreadFunc method correction #35
• Added library CMakeLists. Conversion to gtest for testing by @guycipher in Added library CMakeLists. Conversion to gtest for testing #36
• further gtest implementations by @guycipher in further gtest implementations #37
• further gtest implementations by @guycipher in further gtest implementations #38
• LSMT constructor corrections by @guycipher in LSMT constructor corrections #39
• Wal::backgroundThreadFunc and Wal::Recover to use wal compress member by @guycipher in Wal::backgroundThreadFunc and Wal::Recover to use wal compress member #40
• LSMT closure corrections, WAL closure corrections, function placement corrections by @guycipher in LSMT closure corrections, WAL closure corrections, function placement corrections #42
• backgroundThreadFunc corrections by @guycipher in backgroundThreadFunc corrections #43
• bloomfilter implementation by @guycipher in bloomfilter implementation #45
• Rewrite in C by @guycipher in Rewrite in C #59
• Further TidesDB implementation by @guycipher in Further TidesDB implementation #60
• corrections and todo additions by @guycipher in corrections and todo additions #61
• Corrections and test additions by @guycipher in Corrections and test additions #62
• Corrections, further implementation and todo's by @guycipher in Corrections, further implementation and todo's #63
• remove _uchar_arr_to_uint8, _uint8_arr_to_uchar by @guycipher in remove _uchar_arr_to_uint8, _uint8_arr_to_uchar #64
• Corrections, commenting by @guycipher in Corrections, commenting #65
• Delete logic corrections, wal replay corrections, transaction corrections and test additions by @guycipher in Delete logic corrections, wal replay corrections, transaction corrections and test additions #66
• Compaction logic corrections, initial testing by @guycipher in Compaction logic corrections, initial testing #67
• Serialization, compaction corrections and TidesDB tests additions by @guycipher in Serialization, compaction corrections and TidesDB tests additions #68
• Finalize some error messages/codes and correct leaky poitners on free/destroy methods by @guycipher in Finalize some error messages/codes and correct leaky poitners on free/destroy methods #69
• cmake error corrections by @guycipher in cmake error corrections #70
• working on memory handling, also ran formatting by @guycipher in working on memory handling, also ran formatting #71
• Finalizing to initial version, finalizing code style, correcting minor logic where need be. by @guycipher in Finalizing to initial version, finalizing code style, correcting minor logic where need be. #72
• tidesdb_get, test_bloomfilter_add_check corrections, skiplist formatting, and more by @guycipher in tidesdb_get, test_bloomfilter_add_check corrections, skiplist formatting, and more #73
• bloom filter corrections by @guycipher in bloom filter corrections #75
• new id generator internal library. This library is used for sstable … by @guycipher in new id generator internal library. This library is used for sstable … #77
• add artwork by @guycipher in add artwork #78
• Minor corrections and implementations by @guycipher in Minor corrections and implementations #79
• tidesdb_cursor_get implementation by @guycipher in tidesdb_cursor_get implementation #80
• Finalizing errors, tidesdb_cursor_next by @guycipher in Finalizing errors, tidesdb_cursor_next #81
• tidesdb_cursor_prev implementation by @guycipher in tidesdb_cursor_prev implementation #82
• adding to and finalizing tidesdb__tests by @guycipher in adding to and finalizing tidesdb__tests #83
• Corrections tidesdb.c, and tidesdb__tests by @guycipher in Corrections tidesdb.c, and tidesdb__tests #85
• test_cursor corrections by @guycipher in test_cursor corrections #86
• • cursor_get handling of expired or deleted keys by @guycipher in - cursor_get handling of expired or deleted keys #87
• on cursor start at latest sstable when going through sstables on next… by @guycipher in on cursor start at latest sstable when going through sstables on next… #88
• merge, and compaction tests corrections, reformating by @guycipher in merge, and compaction tests corrections, reformating #89
• Sanitizer run corrections and start of windows support by @guycipher in Sanitizer run corrections and start of windows support #90
• Serialization reformating, bloom filter corrections, test corrections, fsanitize passing across all tests but tidesdb__tests by @guycipher in Serialization reformating, bloom filter corrections, test corrections, fsanitize passing across all tests but tidesdb__tests #91
• Furthering tdb to initial version by @guycipher in Furthering tdb to initial version #92
• LeakSanitizer, logic corrections and test corrections by @guycipher in LeakSanitizer, logic corrections and test corrections #93
• More work towards v0.1.0 by @guycipher in More work towards v0.1.0 #94
• Correcting all current tidesdb__tests, commenting across libraries and correction on minor leak on deserialize_bloomfilter method by @guycipher in Correcting all current tidesdb__tests, commenting across libraries and correction on minor leak on deserialize_bloomfilter method #95
• update read me with some usage examples by @guycipher in update read me with some usage examples #96
• read me corrections by @guycipher in read me corrections #97
• Further TidesDB implementation to stable version by @guycipher in Further TidesDB implementation to stable version #99
• Better concurrency handling and read me correction by @guycipher in Better concurrency handling and read me correction #100
• Finalizing error codes, and updated read me by @guycipher in Finalizing error codes, and updated read me #101
• tidesdb cursor tidesdb_cursor_init and tidesdb_cursor_free methods by @guycipher in tidesdb cursor tidesdb_cursor_init and tidesdb_cursor_free methods #102
• tidesdb_cursor_next, tidesdb_cursor_prev rough implementations by @guycipher in tidesdb_cursor_next, tidesdb_cursor_prev rough implementations #103
• Cursor corrections by @guycipher in Cursor corrections #104
• cursor corrections by @guycipher in cursor corrections #105
• cursor finalization's and read me example by @guycipher in cursor finalization's and read me example #106
• cursor test, beta release ready by @guycipher in cursor test, beta release ready #107

New Contributors

• @guycipher made their first contribution in updated read me contributions section #9

Full Changelog: https://github.com/tidesdb/tidesdb/commits/v0.1.0b

---

This discussion was created from the release TidesDB v0.1.0 BETA.