Working towards v0.2.0

[guycipher]

Hello everyone! Alex here. I’d like to let you all know that I’m in the background taking what I’ve learned from the TidesDB beta and finishing up v0.2.0.

TidesDB v0.2.0 is similar to the beta, but I’ve reworked a few things. For example, we no longer use a pager; instead, we use a block manager. The reason being we can append blocks, no overflow logic, you'll see what I mean, way simpler and WAY faster! Additionally, there is no more chaining on bloom filters. We now determine the proper size before flushing or compacting, based on the number of entries in the memtable or sstable.

I’ve also noticed that flushing in the background may not be as efficient as flushing on the main thread for a column family. For instance, once we flush, we want to truncate the WAL. However, having multiple memtables in a queue can be troublesome and more resource-intensive for no reason, so I’ve simplified this process. Now, we can recover a memtable from the WAL for a column family, and once it reaches the set memtable threshold, we flush the memtable to a column family sstable and truncate the WAL. Simple.

Compaction is mostly the same, multi-threaded, and manually activated. There are a few other changes, such as keeping tombstones on disk until a compaction.

I've also just been really taking the time to design the code in a way it's very easily understood. Lots of comments, etc. You'll see I changed some type definitions to use tidesdb_ and _tidesdb prefix, I like this as when you import the library you know what is tied to TidesDB. Anything with an _ before it is internal and methods like tidesdb_get are public.

I’ll write up more details as I go through the beta code and the upcoming v0.2.0 code. I plan to push one large PR in the coming days. Since I work full-time, I only have time to work on TidesDB during my free time, so I hope you all understand. :)

[guycipher]

Great news! I'll probably be releasing this big PR today. Too much to list currently but I rewrote everything from nothing once again the past few weeks going along with the beta I put together. Making sure everything is thoroughly tested, verified, and extensively reviewed. I've removed a lot of complexity once again and cleaned up a lot of code. I'm excited to share with you all :)