At the end of last month’s update, I said we were working towards alpha 1 of BonsaiDb, and we are very close. With the holidays, however, we’re now anticipating the alpha to be released either the first or second week of January. With the remaining upstream changes being related to cryptography crates, we definitely are wanting to ensure the changes are done correctly rather than quickly.
What is BonsaiDb?
BonsaiDb is a developer-friendly database written in Rust. It offers:
- ACID-compliant transactional document storage
- Schema described using Rust types
- Map/Reduce powered views written as trait implementations
- Supports any serialization format, but has easy accessors for serde-powered types
- Lightweight atomic Key/Value storage
- Pub/Sub
- Ability to use offline a-la SQLite
- Networked server
- Multiuser support with an extensible RBAC permissions system
- Access via QUIC and/or WebSockets
- Extensible wire protocol, enabling a custom API to be added
- … And more
Why pre-alpha for so long?
To me, data is sacred. I couldn’t in good faith publish a database and promote it for usage until I was comfortable using it myself. And until August, my highest priority tended to be missing features instead of working down my list of “remaining questions that might break compatibility”. The freedom of experimenting while knowing it’s OK if you break file formats is important early on.
For the last few months, my focus has been on trying to front-load answering these hard questions before reaching this alpha milestone. I’m hopeful that there will be no data loss due to intentional code changes, and I’m hopeful that the chances of data loss due to a bug are minimal.
Watch me build an app
BonsaiDb was originally introduced at the Rust Gamedev Meetup, and I wanted to build a small “game” to show off BonsaiDb. I use the word game lightly here – it’s really just a few buttons and some text. But, this little demo has a lot going for it:
- Native and WASM game clients
- HTTP-served homepage and leaderboard
- Gameplay powered by a custom api
- Full TLS networking
- Automatic certificate acquisition via LetsEncrypt
- QUIC connections for the native client
- WebSockets for the WASM client
- Automatic deployment via Github Actions
The video is 9 hours long, but has chapter bookmarks to enable jumping between specific topics/tasks:
Or, you can see my presentation at the meetup, where the “game” was demoed:
Highlighted Changes
There have been 122 commits with 7,899 additions and 3,455 deletions across 153 files since the last update, as well as changes in other related repositories. Here’s a quick rundown on some notable changes:
Pot v1.0.0-rc.1
Pot is a serialization format we invented that aims to be more compact than CBOR while still remaining fully self-describing. This release candidate signifies that I’ve reached my comfort level with the format itself.
This release adds a Value type, allowing for serializing and deserializing arbitrary data without corresponding Rust datatypes. Additionally, this release has significantly better test coverage and minor, non-breaking bug fixes relating to number compaction.
Fabruic Updates
Fabruic is our networking implementation for reliable, ordered payload delivery over QUIC. It uses quinn, which released a new update which enabled us to bring Fabruic to rustls 0.20.
Additional Feature Flags
We want BonsaiDb to be fairly lightweight. However, it’s a batteries-included database, and there are a lot of dependencies. We’ve added additional feature flags allowing mutli-user support and at-rest encryption to be disabled.
Tuples as View Keys
BonsaiDb now supports specifying a View’s Key as a tuple containing up to 8 elements that implement the Key trait.
Views in BonsaiDb are just BTrees under the hood. This means that for ranged or ordered queries to work, the keys must be able to be sorted and compared in their binary representation.
Let’s ponder how a (String, u8) key might need to be encoded to be able to be sorted reliably. Let’s ponder the tuples ("a", 255), ("aa", 255), and ("b", 255). Their constituent bytes are:
| String | u8 | |
|---|---|---|
("a", 255) |
0x61 | 0xFF |
("aa", 255) |
0x6161 | 0xFF |
("b", 255) |
0x62 | 0xFF |
If we concatenate the bytes together and sort them, the result would be this:
-
0x6161FF:("aa", 255) -
0x61FF:("a", 255) -
0x62FF:("b", 255)
That’s not right! We need to also encode the length of the String, but now we’ve got a problem: how big should we encode the length? In general, for performance you want to keep your keys as small as possible.
To solve this, I introduced a new crate ordered-varint. This crate allows packing integers in such a way that they remain sortable in their binary representation. BonsaiDb uses this crate to encode length information, when needed, to support composite keys automatically.
New Examples
As part of preparing for a new release, I’ve been working on better documentation. The user guide is still a work in progress, but I’ve spent some time updating it for the release. One new section it contains is a walkthrough of the new CustomApi example.
Another new example shows how to extend the built-in command line interface.
Backup/Restore
I’ve re-introduced the backup/restore functionality in a new way that is able to be extended to support more than just filesystem backups. I have issues tracking S3 support and incremental backup.
Happy Holidays
If you find yourself trying BonsaiDb out and you have any questions, don’t hesitate to open a thread here, an issue on Github, or join us on our Discord. I hope everyone enjoys their holidays!