SQLite based job registry

[soxofaan]

Common theme or question recently is how to get a driver setup that does not require a fully functional ElasticSearch based job registry (EJR), which is not trivial.

The only simpler alternative we have is currently an "in memory" registry, based on an (in process) dictionary. This is fine for unit testing and simple one-off cases but might be too simple and limited for slightly more advanced cases, because there is no persistence.

I was thinking if we couldn't make a SQLite based job registry, which would offer a couple of interesting features:

• there sqlite support in the python stdlib, so this would not drag in extra dependencies
• standard SQL interface, which hopefully makes the implementation easy and low on boilerplate
• persistence without dependency on external service
• it both supports purely in-memory databases (e.g. interesting for unit testing) and persistence to disk as a single file (which allows various deployment options: work with local file, work on a docker mount to persist the data outside a container, ...)