Without better (english) docs it will be hard to get adoption, unfortunately. 8x perf gain over rocksdb is... a lot... unless you're poking at particularly bad metrics.
It depends on workloads, in most cases, there is at least 3x perf gain. One of our customer get 20x perf gain(point search).
Another significant improvement is `select count(*) from SomeTable` in MyTopling, I optimized such query by 30x+ faster than MyRocks, 10x+ faster than InnoDB, while the multi-thread scaling is near linear -- better than InnoDB.
If you are really 20x faster, you are very unlikely to sell it as an open source project, since you already give away your competitive advantage. If you are not that fast, and you want to sell it open source, then you need to make a large group of people understand what you are doing which I did not see easily here.
We keep the core component as private repo, which builds the ToplingZipTable files -- the SST with the searchable compression algo for speed. -- Although even without this component, ToplingDB is still outperforms RocksDB.
Would be really interesting to have faster compilation and more simplicity (auto tuning parameters etc.) compared to rocksdb. In my experience rocksdb performance is very good and it is reliable but it is a pain to integrate into the build process and has too many configurations
Sure, this is one of the reason we develop the framework of json/yaml conf and the embedded http server, with this framework, we discover several RocksDB bugs by this framework(we have contributed 100+ PRs and 60+ issues for RocksDB).
With embedded http server, all DB configurations and status are visually displayed, for example: the compaction progress, the SST file list with the compacting files are highlighted -- with different colors for different compaction jobs, different icons for different compaction types(https://github.com/topling/sideplugin-wiki-en/wiki/Compactio...)
Like RocksDB from which this appears to be forked, the primary usage is as a storage engine for other applications/databases. Compared to rocksdb, it seems like ToplingDB has added more facilities to better support distributed use-cases.
Some databases that utilize RocksDB for their storage engine:
https://kvrocks.apache.org/ - Redis/ValKey compatible distributed database with disk persistence via RockDB.
https://github.com/pingcap/tidb - MySQL compatible distributed database. Mentioned elsewhere in this thread.
https://github.com/tikv/tikv - Distributed, transactional, key value store. Originally by the same company as TiDB.
In theory you could use it as an in-process KV store similar to how SQLite provides an in process sql database, but the api is far from ergonomic for that use case.
To the OP who's developing it: I suggest polishing your README. Provide a simple installation tutorial, maybe a trial offering like tidbcloud.com, and comparative benchmark results, since you advertise your performance.
It’s RocksDB but faster because data can be searched while still compressed allowing you to load more records in less cache/ram leading to up to 10x performance of RocksDB. It adds an embedded http control plane as well as supporting other extensions like MyRocks (MySQL) and Todis (redis compatibility).
Or at least thats what I got from it correct me if I am wrong rockeet.
From what I gather it has an embedded http control plane, yaml/json config for plugins, prometheus integration, and distributed compaction workers on separate, potentially serverless, hosts.
Yes, ToplingDB embeds an http server for showing DB info and online conf changes, this http server is very light weight and does not introduce any visible overhead.
-- prometheus is supported by this this http server.
Without better (english) docs it will be hard to get adoption, unfortunately. 8x perf gain over rocksdb is... a lot... unless you're poking at particularly bad metrics.
It depends on workloads, in most cases, there is at least 3x perf gain. One of our customer get 20x perf gain(point search).
Another significant improvement is `select count(*) from SomeTable` in MyTopling, I optimized such query by 30x+ faster than MyRocks, 10x+ faster than InnoDB, while the multi-thread scaling is near linear -- better than InnoDB.
If you are really 20x faster, you are very unlikely to sell it as an open source project, since you already give away your competitive advantage. If you are not that fast, and you want to sell it open source, then you need to make a large group of people understand what you are doing which I did not see easily here.
We keep the core component as private repo, which builds the ToplingZipTable files -- the SST with the searchable compression algo for speed. -- Although even without this component, ToplingDB is still outperforms RocksDB.
Would be really interesting to have faster compilation and more simplicity (auto tuning parameters etc.) compared to rocksdb. In my experience rocksdb performance is very good and it is reliable but it is a pain to integrate into the build process and has too many configurations
Sure, this is one of the reason we develop the framework of json/yaml conf and the embedded http server, with this framework, we discover several RocksDB bugs by this framework(we have contributed 100+ PRs and 60+ issues for RocksDB).
With embedded http server, all DB configurations and status are visually displayed, for example: the compaction progress, the SST file list with the compacting files are highlighted -- with different colors for different compaction jobs, different icons for different compaction types(https://github.com/topling/sideplugin-wiki-en/wiki/Compactio...)
For the laymen folks reading this - what are the ideal use cases for this?
Like RocksDB from which this appears to be forked, the primary usage is as a storage engine for other applications/databases. Compared to rocksdb, it seems like ToplingDB has added more facilities to better support distributed use-cases.
Some databases that utilize RocksDB for their storage engine: https://kvrocks.apache.org/ - Redis/ValKey compatible distributed database with disk persistence via RockDB. https://github.com/pingcap/tidb - MySQL compatible distributed database. Mentioned elsewhere in this thread. https://github.com/tikv/tikv - Distributed, transactional, key value store. Originally by the same company as TiDB.
In theory you could use it as an in-process KV store similar to how SQLite provides an in process sql database, but the api is far from ergonomic for that use case.
Sure, MyTopling(MySQL) is based on ToplingDB and get the perf gains. There are other DBs also based on ToplingDB.
Ah I see! Thanks for explanation :)
Very extensive, great work on TerarkDB and Topling!
Thank you!
What does it have to do with external storage in this context, does it mean S3. Initially I thought it is a db for thumb drives?
A distributed KV-store plus a relational layer makes it a competitor to NewSQL databases like TiDB, which is also based on Facebook's RocksDB.
It doesn't look like it's very actively developed: https://github.com/topling/toplingdb/pulse/monthly
To the OP who's developing it: I suggest polishing your README. Provide a simple installation tutorial, maybe a trial offering like tidbcloud.com, and comparative benchmark results, since you advertise your performance.
It's quite active. They just aren't using GitHub pull requests in their workflow, which is what GitHub Pulse measures. https://github.com/topling/toplingdb/commits/memtable_as_log...
This is failing my "Can I figure out what the hell it is in 60 seconds?" test.
Sometimes that means I'm just not the target market. I do do web dev (among other things) so that doesn't seem to be the case at first glance?
It’s RocksDB but faster because data can be searched while still compressed allowing you to load more records in less cache/ram leading to up to 10x performance of RocksDB. It adds an embedded http control plane as well as supporting other extensions like MyRocks (MySQL) and Todis (redis compatibility).
Or at least thats what I got from it correct me if I am wrong rockeet.
[dead]
I'm confused what makes this cloud native?
From what I gather it has an embedded http control plane, yaml/json config for plugins, prometheus integration, and distributed compaction workers on separate, potentially serverless, hosts.
Yes, you have exactly got the key points.
It has an embedded http server?
Yes, ToplingDB embeds an http server for showing DB info and online conf changes, this http server is very light weight and does not introduce any visible overhead.
-- prometheus is supported by this this http server.