K4 comparison

[sxwebdev]

Hello

Is tidesdb a new product that was inspired by k4?
Will k4 continue to be developed?
What are the main advantages of one over the other?

Thanks

[guycipher]

Hi @sxwebdev thank you for the question. TidesDB derived from K4 if you review the code it’s quite similar. The design started from LSMT then K4. My main problem with K4 is that there is GC lag and the C library isn’t as performant as it should be. K4 is an amazing storage but it’s not as finalized as TidesDB at this time. Well, actually it is rather finalized and for what it is it’s very fast. I do plan on doing more reviews and releases of K4 though we will see how the TidesDB Go package goes up against the K4 package! Right now primarily the focus is low level TidesDB. I wouldn’t leave K4 out the picture though :):)

[sxwebdev]

Thank you for your quick response and for your open source contribution. I hope everything works out, I am watching your development with interest!

[guycipher]

Thank you for your quick response and for your open source contribution. I hope everything works out, I am watching your development with interest!

Thank you for the kind words!

[sxwebdev]

Do you think it is possible to use this database to store information from the blockchain? With a simple key - transaction hash.
I made comparisons between rocksdb, lmdb, pebble and k4.

lmdb shows itself to be the fastest for writing, but after 20-30 million data, a strong drop in performance is felt due to the fact that a large amount of RAM is needed. lmdb caches the entire tree for data that is not needed in memory.

rocksdb needs to be prepared, but not everything is so smooth there, it also needs a large amount of RAM.

pebble works very fast, microseconds for insertion without degradation in performance, but the search leaves much to be desired.

k4 instantly eats up all the RAM on the same data and at the same time the insertion is very long.

I have not tried tidesdb yet and have no idea whether it is suitable for tens of billions of data or not. and how it will affect RAM at the same time.

in my case golang stack, nvme disks are used, but the amount of RAM is limited to 16gb. what can you say about this, is it suitable for such a task?

[guycipher]

@sxwebdev I just want to bring up TidesDB is still in beta. TidesDB 1 will be the version I'd recommend for non beta projects. Currently we are stabilizing everything and writing binding packages.

[sxwebdev]

@guycipher looks promising. waiting for release for testing

[guycipher]

@sxwebdev what language do you use normally, GO? I'll make sure to prioritize that binding for ya.

[sxwebdev]

yes, we use golang in the project

[guycipher]

Perfect!! Thank you

[sxwebdev]

I also did a comparative test of writing 50000000 data where the key and value are the corresponding numbers.

pebble did it in 3 seconds.
lmdb did it in 30 seconds.
k4 hung for more than 10 minutes, consumed more than 50 GB of RAM, but never completed the test.

[guycipher]

Interesting stuff. K4 again is not finalized as Pebble or LMDB. It could be but I wanted to do it in C. I wonder what your configurations are its taking so long?

[sxwebdev]

I can come back to the code in 2 days and share it in this thread. But as far as I remember, I took the code from the readme from the repository and used transactions as in the case of lmdb. There are no transactions in pebble, so I used batches there

[guycipher]

Yeah batches will be faster for sure! Same thing with K4 if you put multiple write operations within an atomic transaction. Same thing with TidesDB actually. Right on, well I'll update yeah. Thank you very much !!

[guycipher]

@sxwebdev GO binding initial release is out and TidesDB is at v0.7.0b I'd say if you wanna give that workload a whirl, why not give it a try. Lots of memory optimizations and more have went into the past few releases. Reads as well are now optimized with block indices. I will say though make take more memory but you can choose to disable on build.

[guycipher]

I agree but most wont use transactions and batch chunks. In my honest opinion a better test would be writing sequentially random keys. This is where LSM tree's shine. LMDB in all my benchmarks come last for writes. If you want a PR I can whip one up for you later on today. Even then you can still write transactions and batches but random keys, you'll see a remarkable difference.

[guycipher]

The reason I say this, in reality you wont get writes of key_1-10000000 you'll get completely random stuff right? :)

[guycipher]

This is why LMDB is coming first, do it the above and the LSM Tree's will be on top !! 🗡️

[guycipher]

To add to this TidesDB has TTL implemented in the in-memory data structures. This adds some latency as we check to expire keys on traversals. It's still wicked fast. If we removed the TTL yes it would be number one for writes on that list. Maybe today I will add another data structure that can be used if no TTL is required. I'll write another skip list and hash table with no ttl, this will increase write speed 20-30x. Right now on my system 1 million random keys takes 2 seconds no matter what removing ttl it goes down to 300ms for skip list, the old one I have. Thats what I'm trying to say. It's all about minor optimizations !! Thank you for your work @sxwebdev

[guycipher]

func randomKey(n int) []byte {
	rand.Seed(time.Now().UnixNano())
	b := make([]byte, n)
	for i := range b {
		b[i] = byte(rand.Intn(256))
	}
	return b
}

func BenchmarkPut(b *testing.B) {
	defer os.RemoveAll("testdb")
	db, err := Open("testdb")
	if err != nil {
		b.Fatalf("Failed to open database: %v", err)
	}
	defer db.Close()

	err = db.CreateColumnFamily("test_cf", 1024*1024*128, 0, 0.0, false, int(TDB_NO_COMPRESSION), true, int(TDB_MEMTABLE_HASH_TABLE))
	if err != nil {
		b.Fatalf("Failed to create column family: %v", err)
	}
	defer db.DropColumnFamily("test_cf")

	value := []byte("value")

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		key := randomKey(16)
		err = db.Put("test_cf", key, value, -1)
		if err != nil {
			b.Fatalf("Failed to put key-value pair: %v", err)
		}
	}
}

// goos: linux
// goarch: amd64
// pkg: github.com/tidesdb/tidesdb-go
// cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
// BenchmarkPut
// BenchmarkPut-16    	  112352	     10121 ns/op
// PASS

func BenchmarkGet(b *testing.B) {
	defer os.RemoveAll("testdb")
	db, err := Open("testdb")
	if err != nil {
		b.Fatalf("Failed to open database: %v", err)
	}
	defer db.Close()

	err = db.CreateColumnFamily("test_cf", 1024*1024*128, 0, 0.0, false, int(TDB_NO_COMPRESSION), true, int(TDB_MEMTABLE_HASH_TABLE))
	if err != nil {
		b.Fatalf("Failed to create column family: %v", err)
	}
	defer db.DropColumnFamily("test_cf")

	value := []byte("value")

	keys := make([][]byte, b.N)
	for i := 0; i < b.N; i++ {
		keys[i] = randomKey(16)
		err = db.Put("test_cf", keys[i], value, -1)
		if err != nil {
			b.Fatalf("Failed to put key-value pair: %v", err)
		}
	}

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		_, err = db.Get("test_cf", keys[i])
		if err != nil {
			b.Fatalf("Failed to get value: %v", err)
		}
	}
}

// goos: linux
// goarch: amd64
// pkg: github.com/tidesdb/tidesdb-go
// cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
// BenchmarkGet
// BenchmarkGet-16    	 1890178	       623.9 ns/op
// PASS