The basic problem is something like (assuming that one is in an environment where there are multiple clients using the same record store and that software upgrades are performed in a rolling fashion):

1. A meta-data is created, let's say at version 1.
2. The first version of the code with that meta-data is deployed, and a store is created with the meta-data, and it stores in the store header in the database that it was created with meta-data version 1.
3. The code is updated so that a new meta-data version is used, let's say to version 2.
4. A rolling upgrade is performed, and the first instance that is upgrade upgrades the store to use meta-data version 2.
5. All existing instances continue having only version 1 of the meta-data, so whenever they access the store, they fail with a "stale meta-data exception".

Some proposed solutions are things like:

1. Don't store the meta-data in code but instead in some database somewhere. The FDBMetaDataStore works for that (but so would storing the meta-data in an external system), and indeed, if you store the meta-data there currently, you avoid this problem. However, to avoid reloading the meta-data each time, you want to be able to cache it, but the cache isn't quite invalidated correctly, so you can end up reading the cached stale meta-data each time. This is alluded to in Allow for swapping meta-data to a newer version after calling checkVersion #284.
2. Continue storing the meta-data in code, but be more diligent about when you upgrade. For example, you could imagine having two meta-data objects, an old one and a new one. The first version of the code contains only the old one, but then the second version contains both. It continues using the old one until "something" makes it change (either by, say, deploying a third version of the code that only has the new one or by changing some configuration property or something). At this point, it "knows" to use the new one. And the key here is that you somehow have to communicate to the meta-data provider what version is currently in the database, and if the version matches the newer meta-data, you use that (even if your configuration logic says to prefer the older one).

So, getting a little bit more in the weeds, one problem here is that our RecordMetaDataProvider object only has one method, namely getRecordMetaData(). However, what we kind of want is getRecordMetaData(int metaDataVersion), where the metaDataVersion matches the version we saw in the database (or is, say, -1 if the store is new). Then you could do something like if you stored the meta-data in a database but with a cache, invalidate the cache based on the metaDataVersion provided and refetch, or if you did something like store two meta-data versions in code, use the provided version to know which version to return.

There are a couple of other wrinkles here:

1. We probably want to assume that getting the meta-data may require returning a future, as it may require database reads, so it's probably more like getRecordMetaDataAsync
2. We may also want to include the "user version". This is a version that is included in the store header, but the record layer doesn't do anything with it directly other than let the user provide a "user version checker" hook to read it. In theory, the user can do something interesting with this version, I suppose, like make decisions on what meta-data version to provide. But it's somewhat nebulous. Nevertheless, we may want to add the user version to the list of versions provided, though it's somewhat weird as the current user version checker allows the user to provide a new user version, so the new method may return a more complicated struct with both a new meta-data and a new struct.
3. The FDBRecordStore currently doesn't store the meta-data but instead it stores an instance of the meta-data provider. This means that if the meta-data provider changes what version of the meta-data it vends midway through the life of an FDBRecordStore, the store can change its effective meta-data, possibly without performing appropriate actions like marking new indexes as write-only or updating the store header. We may want to start caching a RecordMetaData object and then create a new method on the store for upgrading it if we want to allow users to update the meta-data on an open store. It's possible that this should be a separate Issue, though.