12.0.0.md

RxDB 12.0.0

For the last few months, I worked hard on the new RxDB version 12 release. I mostly focused on performance related features and refactored much of the code.

Removed the core plugin

In the past, RxDB exported all bundled plugins when doing import from 'rxdb';. This increased the bundle size, so optionally people could import from 'rxdb/plugins/core'; to create a custom build that only contains the plugin that they really need. But very often this lead to accidental imports of 'rxdb'. For example, when the code editor auto imported methods. So now, the default import from 'rxdb'; only exports RxDB core. Every plugin must be imported afterwards if needed.

Unified the replication primitives and the GraphQL replication plugin

Most of the GraphQL replication code has been replaced by using the replication primitives plugin internally. This means many bugs and undefined behavior that was already fixed in the replication primitives, are now also fixed in the GraphQL replication.

Also, the GraphQL replication now runs push in bulk. This means you either have to update your backend to accept bulk mutations, or set push.batchSize: 1 and transform the array into a single document inside push.queryBuilder().

Added the cleanup plugin

To make replication work, and for other reasons, RxDB has to keep deleted documents in storage. This ensures that when a client is offline, the deletion state is still known and can be replicated with the backend when the client goes online again.

Keeping too many deleted documents in the storage can slow down queries or fill up too much disk space. With the cleanup plugin, RxDB will run cleanup cycles that clean up deleted documents when it can be done safely.

Allow to set a specific index

By default, the query will be sent to RxStorage, where a query planner will determine which one of the available indexes must be used. But the query planner cannot know everything and sometimes will not pick the most optimal index. To improve query performance, you can specify which index must be used, when running the query.

const queryResults = await myCollection
    .find({
      selector: {
        age: {
          $gt: 18
        },
        gender: {
          $eq: 'm'
        }
      },
      /**
       * Because the developer knows that 50% of the documents are 'male',
       * but only 20% are below age 18,
       * it makes sense to enforce using the ['gender', 'age'] index to improve performance.
       * This could not be known by the query planner which might have chosen ['age', 'gender'] instead.
       */
      index: ['gender', 'age']
    }).exec();

Enforce primaryKey in the index

For various performance optimizations, like the EventReduce algorithm, RxDB needs a deterministic sort order for all query results. To ensure a deterministic sorting, RxDB now automatically adds the primary key as last sort attribute to every query, if it is not there already. This ensures that all documents that have the same attributes on all query relevant fields, still can be sorted in a deterministic way, not depending on which was written first to the database.

In the past, this often lead to slow queries, because indexes where not constructed with that in mind. Now RxDB will add the primaryKey to all indexes that do not contain it already. If you have any collection with a custom index set, you need to run a migration when updating to RxDB version 12.0.0 so that RxDB can rebuild the indexes.

Fields that are used in indexes need some meta attributes

When using a schema with indexes, depending on the field type, you must have set some meta attributes like maxLength or minimum. This is required so that RxDB is able to know the maximum string representation length of a field, which is needed to craft custom indexes on several RxStorage implementations.

const schemaWithIndexes = {
  version: 0,
  primaryKey: 'id',
  type: 'object',
  properties: {
      id: {
          type: 'string',
          maxLength: 100 // <- the primary key must set `maxLength`
      },
      firstName: {
          type: 'string',
          maxLength: 100 // <- string-fields that are used as an index, must set `maxLength`.
      },
      active: {
          type: 'boolean'
      },
      balance: {
          type: 'number',

          // number fields that are used in an index, must set `minimum`, `maximum` and `multipleOf`
          minimum: 0,
          maximum: 100000,
          multipleOf: '0.01'
      }
  },
  required: [
      'active' // <- boolean fields that are used in an index, must be required. 
  ],
  indexes: [
    'firstName',
    ['active', 'firstName']
  ]
};

Introduce _meta field

In the past, RxDB used a hacky way to mark documents as being from the remote instance during replication. This is needed to ensure that pulled documents are not sent to the backend again. RxDB crafted a specific revision string and stored the data with that string. This meant that it was not possible to replicate with multiple endpoints at the same time. From now on, all document data is stored with an _meta field that can contain various flags and other values. This makes it easier for plugins to remember stuff that belongs to the document.

In the future, the other meta fields like _rev, _deleted and _attachments will be moved from the root level to the _meta field. This is not done in release 12.0.0 to ensure that there is a migration path.

Removed RxStorage RxKeyObjectInstance

In the past, we stored local documents and internal data in a RxStorageKeyObjectInstance of the RxStorage interface. In PouchDB, this has a slight performance improvement compared to storing that data in 'normal' documents because it does not have to handle the revision tree. But this improved performance is only possible because normal document handling on PouchDB is so slow. For every other RxStorage implementation, it does not really mather if documents are stored in a query-able way or not. Therefore, the whole RxStorageKeyObjectInstance is removed. Instead, RxDB now stores local documents and internal data in normal storage instances. This removes complexity and makes things easier in the future. For example, we could now migrate local documents or query them in plugins.

Refactor plugin hooks

In the past, an RxPlugin could add plugins hooks which where always added as last. This meant that some plugins depended on having the correct order when calling addRxPlugin(). Now each plugin hook can be either defined as before or after to specify at which position of the current hooks the new hook must be added.

Local documents must be activated per RxDatabase/RxCollection

For better performance, the local document plugin does not create a storage for every database or collection that is created. Instead, you have to set localDocuments: true when you want to store local documents in the instance.

// activate local documents on a RxDatabase
const myDatabase = await createRxDatabase({
    name: 'mydatabase',
    storage: getRxStoragePouch('memory'),
    localDocuments: true // <- activate this to store local documents in the database
});

myDatabase.addCollections({
  messages: {
    schema: messageSchema,
    localDocuments: true // <- activate this to store local documents in the collection
  }
});

Added Memory RxStorage

The Memory RxStorage is based on plain in-memory arrays and objects. It can be used in all environments and is made for performance.

RxDB Premium

You can now purchase access to additional RxDB plugins that are part of the RxDB Premium package.

If you have sponsored RxDB in the past (before the April 2022), you can get free lifetime access to RxDB Premium by writing me via Twitter

• RxStorage IndexedDB a really fast RxStorage implementation based on IndexedDB. Made to be used in browsers.
• RxStorage SQLite a really fast RxStorage implementation based on SQLite. Made to be used on Node.js, Electron, React Native, Cordova or Capacitor.
• RxStorage Sharding a wrapper around any other RxStorage that improves performance by applying the sharding technique.
• migrateRxDBV11ToV12 A plugin that migrates data from any RxDB v11 storage to a new RxDB v12 database. Use this when you upgrade from RxDB 11->12 and you have to keep your database state.

Other changes

• The Dexie.js RxStorage is no longer in beta mode.
• Added RxDocument().toMutableJSON()
• Added RxCollection().bulkUpsert()
• Added optional init() function to RxPlugin.
• dev-mode: Add check to ensure all top-level fields in a query are defined in the schema.
• Support for array field based indexes like data.[].subfield was removed, as it anyway never really worked.
• Refactored the usage of RxCollection.storageInstance to ensure all hooks run properly.
• Refactored the encryption plugin so no more plugin specific code is in the RxDB core.
• Removed the encrypted export from the json-import-export plugin. This was barely used and made everything more complex. All exports are now non-encrypted. If you need them encrypted, you can still run by encryption after the export is done.
• RxPlugin hooks now can be defined as running before or after other plugin hooks.
• Attachments are now internally handled as string instead of Blob or Buffer
• Fix (replication primitives) only drop pulled documents when a relevant document was changed locally.
• Fix dexie.js was not able to query over an index when keyCompression: true

Changes to RxStorageInterface:

• RxStorageInstance must have the RxStorage in the storage property.
• The _deleted field is now required for each data interaction with RxStorage.
• Removed RxStorageInstance.getChangedDocuments() and added RxStorageInstance.getChangedDocumentsSince() for better performance.
• Added doesBroadcastChangestream() to RxStorageStatics
• Added withDeleted parameter to RxStorageKeyObjectInstance.findLocalDocumentsById()
• Added internal _meta property to stored document data that contains internal document related data like last-write-time and replication checkpoints.

You can help!

There are many things that can be done by you to improve RxDB:

• Check the BACKLOG for features that would be great to have.
• Check the breaking backlog for breaking changes that must be implemented in the future but where I did not have the time yet.
• Check the TODOs in the code. There are many small improvements that can be done for performance and build size.
• Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
• Improve the documentation. In the last user survey, many users told me that the documentation is not good enough. But I reviewed the docs and could not find clear flaws. The problem is that I am way too deep into RxDB so that I am not able to understand which documentation a newcomer to the project needs. Likely I assume too much knowledge or focus writing about the wrong parts.
• Update the example projects many of them are outdated and need updates.
• Help the next PouchDB release to improve RxDBs performance.