Backward-compatible schema changes are very hard

[sicking]

Use case: Support making backwards-compatible changes to the schema without requiring users to reload existing tabs.

Details: Right now, any time you need to make schema changes you have to bump the version number on the database. Generally speaking this requires existing tabs to be reloaded.

This makes a lot of sense when a schema change makes backwards incompatible changes to the database. I.e. where the code running in existing tabs would not be able to correctly read or write data to the database after the schema change.

However most schema changes are likely backwards compatible. For example they add an object store or an index.

In this case existing code in any existing tabs would be able to keep using the database ignoring the new objectStores/indexes.

It is technically possible to deploy code that uses indexedDB to support forward-compatible schema changes already with the current API.

You could for example use the following code to open the database:

let dbVersion = 1
let db = undefined
let dbPromise = undefined
function openDatabase() {
  if (!dbPromise) {
    dbPromise = new Promise((resolve, reject) => {
      const dbRequest = indexedDB.open("mydb", dbVersion)
      dbRequest.onupgradeneeded = () => { ... }
      dbRequest.onsuccess = () => {
        let db = dbRequest.result
        db.onversionchange = (event) => {
          if (event.newVersion && event.newVersion < 1000) {
            dbVersion = event.newVersion
            db.close()
            dbPromise = undefined
          }
        }
        resolve(db)
      }
    }
  }
  return dbPromise
}

In this code the page assume that any versions below 1000 are versions that have a schema compatible with the code in this tab. So if we get a versionchange event to such a version we remember that version number, close and then reopen the database using this new version.

This way when you deploy new IndexedDB-using code, you can make that code upgrade to version 2 if they know version 2 is backwards compatible. If you need to make a backwards incompatible schema change you can use version 1000 which will cause existing tabs to hold on to their database connections and prevent dataloss or data corruption, but will require those tabs to be closed/reloaded.

However this code has at least two subtle bugs:

• If the tab that tries to update the plugin fails to do the update (due to IO error or other bugs), then the existing tabs will now open the database with the updated version number thus causing them to bump the version number. And they will do so without making the schema changes that should go along with that update.
• If the database in the old tab isn't opened upon page load, then it won't receive the version update notifications and so will just receive an error when trying to open the database with the old version number. And even if you do open the database during page load, you are likely racing with a new tab being opened which uses the new version.

Fixing these problems is doable but quite tricky.

Another way to approach this problem is to use BroadcastChannel to send out messages about when a backwards compatible change to the schema is about to happen. This message can contain information about what the new version number is. But again there seems to be tricky edge cases involved in implementing this approach.

What is even worse is that websites are required to deploy this logic with the initial release of their IndexedDB usage. I.e. you are required to realize ahead of time that you should think about how to support both backwards compatible and backwards incompatible version changes.

The reality is likely that most websites do not implement a solution like this when they first roll out IndexedDB.

It would be great to have some solution in the API specifically for allowing a backward compatible schema change. This could help in several ways:

• Existing connections to the database could remain open and undisrupted (possibly other than that no trasactions could start until the schema change is done)
• Existing tabs which try to open using the "old" schema version could be allowed to succeed
• Users wouldn't have to think about how to handle backward compatible schema changes until they actually make one.

We could even enable websites to treat the first version upgrade as a backward compatible schema change (since any schema is compatible with "database not used at all"). So this could be an opportunity to create schema-changing API which is friendlier than the current API.

If we did this then the version handling in IndexedDB could feel much more smooth. For "backward compatible" changes, including creating the database in the first place, a simpler API could be used.

For backward incompatible changes users would still have to use the current heavy-handed API. But would arguably make more sense since for backward incompatible changes you have to force existing tabs to be reloaded or at least stop using the database.

[inexorabletash]

TPAC 2019 Web Apps Indexed DB triage notes:

Hey Jonas!

Potentially scary/complex change to implementations and adds API confusion, but the use case is clear. FYI we're looking at #288 (comment) which doesn't address the big use case here (avoiding reloading everywhere), but does solve the minor version problem. We think using Web Locks could make the BroadcastChannel-style workaround easier.

Leaving this open to gather more thoughts.

[sicking]

Here's one potential solution:

Add a new "schemachange" transaction type. The scope of the "schemachange" transaction would always be the whole database, which means that two "schemachange" transactions can't run in parallel (this could be somewhat relaxed, but let's start simple).

During a "schemachange" transaction the page would be allowed to add and remove objectStores and indexes just as during a "versionchange" transaction.

When a "schemachange" transaction is committed, it updates the list of objectStores and indexes only of the current IDBDatabase object. Any other open IDBDatabase object will not get an updated list of objectStores or indexes. However any attempt at reading or writing from a objectStore or index that no longer exists, or which doesn't have a "compatible schema" will result in a failed request.

A "compatible schema" for objectStores means has the same value for keyPath and autoIncrement flag. For an index it means that the index's objectStore has a "compatible schema". (We could tighten the requirement for indexes to require that they have the same keyPath, unique flag, and/or multiEntry flag, but I think it's not strictly necessary)

One unfortunate aspect is that the function to create a "schemachange" transaction has to be asynchronous. I.e. it can't synchronously return a IDBTransaction object. This is because it must return an IDBTransaction object with an up-to-date list of objectStores and indexes so that the page can check if the necessary changes have already been done.

If a "schemachange" transaction is aborted, the IDBDatabase object is reverted to contain the objectStores and indexes before the "schemachange" transaction was started (which might be different from the actual set of objectStores and indexes, if the schema was changed using another IDBDabase object).

So the only needed changes in the API would be

• Add "schemachange" to IDBTransactionMode
• Add a new function similar to transaction, but which is async and doesn't take any arguments

This effectively creates a separate method of handling schemas where the webpage takes on the responsibility to handle schema versioning, rather than the current one based on version/onupgradeneeded/onblocked/onversionchange

To make this method safer we could say that you're only allowed to add objectStores and indexes during a "schemachange" transaction. Not remove objectStores or indexes, and not change any existing data. But I think that partially defeats the purpose of adding a "I know what I'm doing" API.

I think it'd be more interesting to add API surface to enable a page that wants to use "schemachange" transactions to avoid having to use version/onupgradeneeded/onblocked/onversionchange when they open the database the first time. Possibly by simply allowing 0 to be passed a version to IDBFactory.open.