ActorDB is very cool, but from what I gather it's — at least nominally — designed to work with many small databases (one per "actor") as opposed one large one, and that the constraints it puts on the query model means you have to carefully design your data model to be heavily compartmentalized from the beginning. It also poses some problems for database-wide queries, which have to fan out to every separate actor database. Whereas rqlite acts like one single SQLite instance, and can work with existing schemas.
Please do correct me if I'm wrong. Would ActorDB be happy and perform well with, say, just one actor?
To me, that just sounds like it forces you to do the Right Thing from the beginning. Eventually, you will need to shard your data in some way†.
If you're already locked into a "global queries against tables containing Everyone's Stuff" data model, you end up doing this through Stupid Database Tricks like "see-other" redirection or silent persistent-hash load-balancing (with constant ops-heavy rebalancing as you grow.)
If you think about sharding from the beginning, though, you end up just dividing your data into little naturally-atomic "worlds." Like sharding email [headers, not bodies] by user, or sharding StackOverflow posts by community.
If you don't think you'll ever scale to the point where you'll be forced to make those calls, just pick a different database. (But then, if you don't think you'll ever reach that scale, then literally any database will do.)
---
† Unless you're EVE Online. They could probably write a really good whitepaper about how they've scaled their single-node MSSQL database so far. I presume they're mostly following the same patterns you would when running e.g. Oracle on big iron; but—according to most sources—they're just using some heavily-loaded commodity hardware with no fancy IO offloading et al. No idea how they do it.
Yes, that's what I said: for most businesses (that aren't trying to scale to infinity the way that VC startups talk about), literally any database will do. Even single-node SQLite will do, because there won't be enough contention for its locking to ever matter.
Absolutely. Keep in mind that ActorDB actually requires that you supply the sharding key yourself whenever you do a query. It's not like, say, Elasticsearch, which shards both reads and writes transparently.
Many apps do require large queries that span many "partitions": For example, listing all of a StackOverflow member's contributions across all communities, sorted by time. With ActorDB, you would have to plan for this by denormalizing a bit (biokoda might correct me here): Each community would be an actor with a table of QA items, and each user would also have an actor containing a table listing their own QA items for all communities. Since ActorDB apparently has transactions, you can maintain this duplicate data atomically, though you can't maintain foreign-key constraints across actors.
I work with CouchDB, and it sounds similar: many queries end up being map-reduce functions run across many CouchDB "databases" (partitions) on the same node/cluster.
One of the nice things about this approach, from my CouchDB experience, is that each DB/partition has its own permissions (fully extensible through design-documents shoved into that DB), so instead of needing to carefully write your business layer to ensure that users can only ever query their own data out of a table, you just do all your work for a particular user in a table that only contains stuff that user has a right to access to begin with.
It's a lot like working with S3 buckets, now that I think about it. Buckets containing tables, rather than buckets containing objects.
Please excuse my ignorance - how do I enforce nontrivial system-wide invariants with lots of little databases rather than a single consolidated one?
The invariants I care about are:
(0) Referential integrity. If the table `Foo` has a `FOREIGN KEY (BarID) REFERENCES Bar (BarID)`, then no row in `Foo` must be seen as having a `BarID` whose value can't be found in the table `Bar`.
(1) Linearizability. There must exist a total order on the entire transaction history of a database, such that, starting from the empty database, and executing the transactions nonconcurrently in the given total order, the result is the current state of the database. (NOTE: The transaction history need not be physically stored anywhere. So this invariant can't be “tested” - it has to be proven to hold.)
These guarantees are so basic, so fundamental in my everyday use of RDBMSs, that I need to be convinced that they hold.
I have no experience in developing in ERP systems. I do have experience on being at the end of a very poorly working one.
These systems seem (from my outside view) to have a tendency to become giant monoliths. So when developing you must fight against increasing monolithic complexity. Using something like ActorDB can be somewhat of a beneficial constraint. It forces you to maintain a clean design.
I would force sales, marketing, shipping, product planning (taking from wikipedia here..), to be their own separate actors with their own schemas. Then if possible shard within those types. So if something is customer service, have an actor per customer and have all his data there. If you're developing multiple products, every product has an actor.
> These systems seem (from my outside view) to have a tendency to become giant monoliths. So when developing you must fight against increasing monolithic complexity.
You aren't wrong, that's my experience as well. It's just as annoying for programmers (or, at least, for me) as it is for users. The following question has popped out countless times in my head: “Why do I have to rely on an implicit convention that this application module never touches this database table?” There was never a good answer.
The only reason why I put up with such things is that I have no idea how to prevent more modular designs from turning into a data integrity nightmare. (I'll freely admit my lack of education is to blame here.) For instance, let's say we have three modules: inventory, sales and shipping. Furthermore, let's assume each module is its own actor and uses its own backend database. We must implement the use case “enter a sale in the system”:
(0) The sales module queries the inventory module whether there is enough of a product in stock to satisfy a customer order. The expected sequence of actions is:
(1) The inventory module “locks” the requested quantity/amount of the product [so that it can't be used, say, for another sale], and gives the sales module a “token” that can be used to confirm or cancel the withdrawal.
(2) The sales module queries the shipping module if there are enough available trucks/ships/whatever to ship the product to the customer's location by a given delivery date.
(3) The shipping module “locks” however many trucks/ships/whatever it deems necessary to ship the product, and gives the sales modules a “token” that can be used to confirm or cancel the shipping.
(4) The sales module queries the user for the customer's credit card number and verification code, interfaces with the bank's system, blablabla...
(5) The sales module confirms to the inventory and shipping modules [in this specific order] that the product will be withdrawn and shipped.
Now some exception handling:
(6) If step 3 fails, the sales module cancels the product withdrawal.
(7) If step 4 fails, the sales module cancels the product withdrawal and shipping.
(8) If any system [inventory, sales, shipping] goes down, neither the product nor the trucks/ships/whatever can be kept locked forever. So each lock must have a timeout, and, if it's neither explicitly confirmed nor explicitly cancelled by the sales module, it will be implicitly cancelled by the inventory and/or shipping module when the timeout elapses.
(9) It may happen [unlikely, but not impossible], that the inventory and shipping module's clocks get unsynchronized in such a way that, when the product withdrawal has been been confirmed, the shipping lock has already elapsed. Oh, the nightmare.
Implementing all of this correctly in all cases is actually tricky! And if anything is implemented slightly wrong, the whole system goes kaboom! With a monolithic database, there is no need to “lock” any resources, nor issue “confirmation tokens” - just use the DBMS's built-in transaction system!
> Furthermore, let's assume each module is its own actor and uses its own backend database.
I would have every module an actor type. There can be multiple types each type has its own schema. Within an actor type many actors. An actor for every product for instance. So all X widgets are in one actor.
> With a monolithic database, there is no need to “lock” any resources, nor issue “confirmation tokens” - just use the DBMS's built-in transaction system!
ActorDB has distributed ACID transactions so I would use that. You can create a transaction over multiple actors. The reason I would split it into many actors is that you're always locking small parts of the system for the duration of the transaction not the entire DB.
The ones that I use at home (PostgreSQL) and at work (SQL Server) allow serializable transactions. Of course, in many cases it's overkill, but it's good to know that it's there when needed.
You can actually get very far by not sharding. But it forces you to eventually throw out everything that makes relational databases awesome and turn it into a glorified KV store.
It would perform well with one actor when compared to a single SQLite instance. Not compared to PostgreSQL/MySQL due to the concurrency model being optimized for many concurrent actors, not concurrent access to a single actor. A single actor can still execute thousands of queries per second however.
Compared to a single SQLite instance ActorDB has two major advantages. No write multiplication due to using LMDB and compression which means reads/writes are significantly smaller. SQLite will always write everything twice. First to WAL, then to the SQLite file and it has no compression capabilities. It should completely trounce rqlite in the performance department.
Yes ActorDB takes sqlite3.c, takes out the wal.c code and replaces it with calls to LMDB. There is no redo log, LMDB is designed in such a way that it does not require it. It is actually verified as the safest storage engine design out there.
Please do correct me if I'm wrong. Would ActorDB be happy and perform well with, say, just one actor?