Bolt doesn't support compression unless you implement it yourself (i.e. in your application layer). Bolt also does not support concurrent writers. I don't need MVCC nor transactions, which have overhead. Finally, tree-based systems aren't very good at doing large, sequential reads.
Edit: Again, not exactly sure about the downvoting, so here is a better explanation. If I'm wrong, please let me know!
Let's say you insert many points for a few metrics in time order. Because of the way B-tree page allocation works, you can't guarantee that the points for a metric will be stored in adjacent pages on disk (correct me if I'm wrong). I want the guarantee that if I'm going to read points sequentially off a disk, I will do so without seeking around to different pages. Catena does that, and keeps points very compact.
When I used Bolt with a single key-value pair per point, I saw disk usage that was something like 50 bytes per point (on average). With Catena's file partitions with compression, I see around 3 bytes per point (on average). These points are 8 byte timestamps with 4 byte values. That's a huge benefit.
To be clear, I'm not saying you should have used it, just that some of the functionalities are already there (and I love Bolt so I easily see parallels). It's always interesting to implement stuff from scratch, if only to understand how it works.
Also, I'm only talking about using Bolt as the on-disk partition store, ie the read-only partitions. You'd only write them when flushing the in-mem partitions, so no need for concurrent writers.
> tree-based systems aren't very good at doing large, sequential reads.
It all depends on the implementation you've chosen. B+trees exist exactly for efficient sequential reads, and Bolt uses one. (Take a look at this [0]: 40 ns/op per k/v, and putting the database in memory doesn't even improve it).
Trees also allow you to search for specific timestamps so you can
I like Bolt too. It was my first choice :). Using it for read-only partitions sounds like a great idea. I think the current implementation looks like a tree if you "squint" a bit. There's a hierarchy of metrics and sources, and the arrays of points are like the leaf nodes.
You're very correct about B+trees being efficient with sequential reads. We use MySQL with InnoDB for all of our time series where I work[0]. I started this way just to get a good understanding of what works and what doesn't. I think I'm on track to implementing something like a B+tree by using smaller extents of points.
Bolt author here. I don't think I'd agree that Bolt isn't good for time series. In fact, it has fast appends and fast sequential reads so it works really well for time sorted data. Having a sequential page ordering (like Catena) will have a huge boost when running on spinning disk but it's much less of an issue on SSDs.
All that being said, I do like the approach of Catena. I think it works well for best case scenarios and you can get some large perf gains from tailoring the format to a specific type of data. However, some of the trade offs such as dropping writes after the in-memory window are not going to be acceptable to many people.
As far as disk usage goes, it sounds like you could improve the utilization by increasing Bucket.FillPercent to 1.0. By default Bolt splits pages in half since it assumes random writes. You can do some optimizations if you're writing in order.
Another option that'd be interesting to look at would be doing Catena on top of Bolt. You can write your large, compressed Catena-ized data as a single value in Bolt and it'll be laid out sequentially on disk. I know you mentioned that you don't need transactions but I find them to be really useful once you start distributing data across a cluster and need some guarantees about writes.
Edit: Again, not exactly sure about the downvoting, so here is a better explanation. If I'm wrong, please let me know!
Let's say you insert many points for a few metrics in time order. Because of the way B-tree page allocation works, you can't guarantee that the points for a metric will be stored in adjacent pages on disk (correct me if I'm wrong). I want the guarantee that if I'm going to read points sequentially off a disk, I will do so without seeking around to different pages. Catena does that, and keeps points very compact.
When I used Bolt with a single key-value pair per point, I saw disk usage that was something like 50 bytes per point (on average). With Catena's file partitions with compression, I see around 3 bytes per point (on average). These points are 8 byte timestamps with 4 byte values. That's a huge benefit.