This document contains a log of changes to the FoundationDB Record Layer. It aims to include mostly user-visible changes or improvements. Within each minor release, larger or more involved changes are highlighted first before detailing the changes that were included in each build or patch version. Users should especially take note of any breaking changes or special upgrade instructions which should always be included as a preface to the minor version as a whole before looking at changes at a version-by-version level.
As the versioning guide details, it cannot always be determined solely by looking at the version numbers whether one Record Layer version contains all changes included in another. In particular, bug fixes and backwards-compatible changes might be back-ported to or introduced as patches against older versions. To track when a patch version has been included in the main release train, some releases will say as a note that they contain all changes from a specific patch.
Support for the Protobuf 2 runtime has been removed as of this version. All artifacts now use Protobuf version 3. Note that the choice of Protobuf runtime version is distinct from the choice of Protobuf message syntax, and that users wishing to retain Protobuf 2 behavior can still achieve the same semantics (including optional field behavior) as long as they specify the syntax on their Protobuf file as proto2. Note that the Maven artifacts using Protobuf version 3 used to be suffixed with -pb3. Existing Protobuf 3 users must remove that suffix from their dependency declarations (e.g., fdb-record-layer-core-pb3 should now be fdb-record-layer-core).
Starting with version 3.4.455.0, the semantics of UnnestedRecordType were changed in response to Issue #2512. It was identified that extraneous synthetic records were being produced when one of the children was empty. This did not match the semantics of FanOut expressions, and so the unnesting calculation was changed. This means that any index on an existing UnnestedRecordType requires rebuilding to clear out any such entries from older indexes.
-
DefaultTextTokenizer(andTextTokenizers that use it) now usesLocale.ROOTfor tokenizing instead of the default locale. This could have an impact if it's being used in an environment with a default locale that is not compatible withLocale.ROOTfor this purpose. For example if the default locale isth, Thai text may be tokenized differently with this version. This also means that if you have TEXT indexes using one of these tokenizers, they will need to be rebuilt with the new code. -
Bug fix DefaultTextTokenizer behaved differently depending on system locale (Issue #2966)
-
Bug fix Sort FieldInfos attributes before serializing, reducing unique FieldInfos (Issue #2968)
-
Feature Support temporary table planning and execution (Issue #2962)
- Bug fix Make test not timeout in Nightly build (Issue #2909)
- Bug fix Fix concurrency issues with
MapPipelinedCursor.close(Issue #548) - Bug fix Fix concurrency issues with
FlatMapPipelinedCursor.close(Issue #2953) - Feature Add more lucene exception handling tests (Issue #2939)
- Feature Methods interacting with delayed futures can now supply their own
ScheduledExecutorService(Issue #2929)
- Bug fix Expand
ForEachquantifier semantics (Issue #2930) - Performance The old planner can now be configured to avoid generating IN-union plans with IN-sources that are not pushed into the underlying scan (Issue #2941)
- Breaking change Planning logic for IN-joins in the old planner means that some plans may change in scenarios where no index is able to be matched to the plan. Only users who have set the maximum number of replans for in-to-join planner configuration parameter to greater than or equal to 0 should be affected (PR #2942)
- Bug fix Missing subspace provider information added to
FDBRecordStorelogs (Issue #2936) - Performance avoid creating useless/redundant intersections when planning data access (Issue #2940)
- Feature Add Lucene lock duration to store timer (Issue #2951)
- Feature indexer: Add information to "build index online" log message (Issue #2946)
- Feature Translate IOException to/from RecordCoreException for Lucene (Issue #2934)
- Feature asyncToSync without exception mapping for Lucene (Issue #2926)
- Feature Allow converting a single/multi target indexing session to mutual one (Issue #2917)
-
For implementors of
IndexMaintainer, there is a new methodclearUniquenessViolationsthat is used to clear any uniqueness violations if the index is not unique. By default this is a no-op, butStandardIndexMaintainerclears the violations it maintains. This can be utilized to change an index from unique to non-unique without having to rebuild, by keeping everything else about the index the same, includinglastModifiedVersionand thesubspaceKey. However, this functionality should only be used if you know the the associatedIndexMaintainerproperly implementsclearUniquenessViolations. -
Bug fix Scope the IndexUniquenessCommitCheck by subspace (Issue #2919)
-
Feature Mark READABLE_UNIQUE_PENDING indexes to READABLE if they are not unique (Issue #1991)
- Bug fix The new
TupleHelpers.prefixLengthOfSizemust catch more errors. (Issue #2921)
- Feature Feature 1 (Issue #2904)
- Bug fix In-union plans should now do a better job of not incorrectly de-duping groups in the presence of repeated fields (Issue #2901)
- Bug fix Infinite recursion in
TreeLike#replaceLeavesMaybe(Issue #2884)
- Bug fix TransformedRecordSerializer fails to compress when serialized record is very very small. (Issue #2896)
- Bug fix DimensionsKeyExpression not handled enough by QueryToKeyMatcher (Issue #2846)
- Feature MultidimensionalIndex update operation is not thread safe (Issue #2849)
- Performance Add timing breakdown to Repartitioned records log (Issue #2886)
- Bug fix Remove uncommitted version mutations during
deleteRecordsWhereto avoid corrupting record stores if there are outstanding record saves whendeleteRecordsWhereis called (Issue #2275) - Bug fix ensure that all compensations get planned with their necessary quantifiers (Issue #2881)
- Performance Lucene partition balancing: reduce the number of retries (Issue #2878)
- Bug fix Expose
Tuple-basedMIN_EVERandMAX_EVERindexes to Cascades (Issue #2874) - Bug fix Log Repartitioned records after writing them (Issue #2867)
- Bug fix Lucene merges: false no merges found (Issue #2864)
- Feature Deprecate special IndexingByRecords functions (Issue #2259)
- Bug fix LazyOpener throws more precise exceptions (Issue #2852)
- Bug fix Fix flaky test - testMutualIndexingWeirdBoundaries (Issue #2854)
- Bug fix Unordered union implementation pre condition is too strict (Issue #2856)
- Bug fix Lucene exception management (Issue #2850)
- Bug fix Fix sporadic timeout in test (Issue #2839)
- Feature IndexOperation for getting Lucene index metadata (Issue #2826)
- Bug fix
Valuesubsumption should default to semantic equality (Issue #2825) - Bug fix
Valuesimplification is not constructing the top-level simplified expression correctly (Issue #2837) - Bug fix Memoization is not behaving correctly in some edge cases (Issue #2836)
- Bug fix Fail fast when Batch GRV rate limit exceeded (Issue #2813)
- Bug fix
ArithmeticValue::semanticEqualsnow includes the operator in its calculation (Issue #2189) - Bug fix subsumedBy is only looking at value type in most cases (Issue #2818)
- Feature Support for basic arithmetic functions in added as default
FunctionKeyExpressions with query support in both the old and Cascades planners (Issue #2663)
- Feature add Support for nested DynamicMessage in ConstantObjectValue (Issue #2806)
- Performance improve performance for index anding in AbstractDataAccessRule (Issue #2804)
- Breaking change New the representation of aggregate plans (Issue #2763)
- Bug fix
TextCollatorRegistryICU.getTextCollatorshould freeze theCollator(Issue #2798) - Performance Covering optimization with invertible index fields (Issue #2801)
- Feature Add more complete direction control to
LogicalSortExpression(Issue #2796)
- Feature Key expression support for reverse ordering and nulls last (Issue #2722)
- Feature OnlineIndexer / OnlineIndexScrubber: move the sync session lease length to Config (Issue #2794)
- Performance The index scrubber now attempts to take a transaction time limit into account when deciding whether to commit early (Issue #2787)
- Breaking change Online indexer and index scrubber configuration has been extracted into its own top-level class (PR #2788)
- Feature Improve modeling of Ordering, OrderingPart (Issue #2127)
- Bug fix Fix out of bound exception with repartitioning (Issue #2784)
- Bug fix Only log repartitioning records when there's repartitioning to do (Issue #2782)
- Bug fix Make exceptions from LuceneIndexMaintainer more verbose (Issue #2768)
- Feature Merge partitions when they become too small (Issue #2739)
- Feature Include cause with AgilityContext already closed exception (Issue #2774)
- Bug fix Fix flaky test - break loop into smaller commits (Issue #2756)
- Bug fix FDBDirectoryLock won't fail if AgilityContext is flushed while closing (Issue #2754)
- Feature Reproduce abandoned FileLock due to conflict issue (Issue #2731)
- Bug fix The
ArithmeticValueandNumericAggregationValueclasses now uses a fixedLocalewithtoUpperCasewhen encapsulating a specified function by name to avoid mismatches when running in the Turkish locale (Issue #2750) - Bug fix Inconsistent nullability status for Array's element type (Issue #2732)
- Feature Make Lucene block cache size configurable (Issue #2749)
- Feature Validate filelock on every transaction to reduce ttl (Issue #2747)
- Feature In-join and in-union plans now support using
ConstantObjectValues as the basis of theirInSource(Issue #2717)
- Bug fix Error messages reported when a store is missing a header now indicate whether the store has record or index data correctly (Issue #2715)
- Bug fix Missing Lucene error wrapper when index is partitioned (Issue #2725)
- Bug fix Fix issue with LuceneLock exception (Issue #2721)
- Feature Add isScannable(), isReadableUniquePending() to RecordStoreState (Issue #2718)
- Bug fix Make Lucene lock error retryable(Issue #2700)
- Feature Test FDBDirectory consistency while cross-transaction merges are ongoing (Issue #2701)
- Feature Online Indexing: improve allowTakeoverContine to distinguish type conversion cases (Issue #2703)
- Bug fix Corrected field to index calculation resolving records from index entries with permuted min/max indexes (Issue #2705)
- Bug fix Lucene directory: treat close as a potential recovery path (Issue #2707)
- Bug fix Lucene: FileLock should handle close as a possible recovery path (Issue #2692)
- Performance Lucene: merges should run in default transaction priority (Issue #2696)
- Breaking change The default collation order is now based on the root locale instead of the system default. Users relying on a specific locale should update any indexes using ta collation function to include the locale in the function arguments before updating (Issue #2678)
- Bug fix LuceneIndexTest.findStartingPartitionTest(false) false to decode tuple (Issue #2684)
- Bug fix Graceful Degradation breaks when index is being built (Issue #2668)
- Feature Add logging information to the runner used when repartitioning (Issue #2685)
- Feature The
TransformedRecordSerializernow validates that compressed data decompresses to the expected size (Issue #2689) - Feature The
TransformedRecordSerializernow includes checksums in compressed data that is validated during decompression (Issue #2689) - Feature Updated the set of known protocol versions for client log event parsing to include FDB 7.3 (Issue #2682)
- Bug fix IndexingMerger: protect form early "open store" exception (Issue #2676)
- Performance LuceneOptimizedPostingsReader: skip checkIntegrity when dis-allowed (Issue #2666)
- Feature Index Scrubber: convert "reset range" log message from info to debug (Issue #2664)
- Feature Record serializers now have a new default method that allows for validating bytes can be deserialized back to their original message (Issue #2680)
- Feature Agility Context: check/commit context before the user operation (Issue #2626)
- Feature AgilityContext: add applyInRecoveryPath test (Issue #2636)
- Feature Select Partition based on query filter (Issue #2616)
- Bug fix Equality index scan can match additional suffix starting with zero byte (Issue #2650)
- Performance Lucene: cleanup logs and lock release path (Issue #2648)
- Bug fix Improve logging for Lucene file locks (Issue #2645)
- Bug fix Review Lucene's use of executors (Issue #2638)
- Bug fix AgilityContext: Prevent operations after failed commit (Issue #2642)
- Bug fix LucenePartitioner.getNextNewerPartitionInfo() sometimes returns invalid (null) result (Issue #2640)
- Bug fix Log total records scanned and time during index build (Issue #2629)
- Bug fix AgilityContext: prevent other threads writes after abort (Issue #2634)
- Bug fix Typobug: LUCENE_FILE_LOCK_TIME_WINDOW_MILLISECONDS has duplicated property key (Issue #2632)
- Bug fix IndexingMerger: assume a write-only context by using a non synchronized runner (Issue #2611)
- Bug fix AgilityContext.accept: release write lock in finally (Issue #2619)
- Bug fix Commit after successful calls to AgilityContext.apply (Issue #2618)
- Bug fix Rename LuceneLogMessageKeys.PARTITION -> INDEX_PARTITION (Issue #2614)
- Bug fix Index Merge: Handle if the store fails to open (Issue #2609)
- Bug fix Record type key comparisons now can be used on synthetic types (Issue #2587)
- Bug fix Scan comparison ranges on the synthetic record type primary key do not break apart list key expression components (Issue #2588)
- Breaking change Union plans on synthetic types can choose different union comparison keys from before so continuations from plans from earlier builds may not be re-used on those queries (Issue #2588)
- Bug fix Lucene merge flush group/partition info AgileContext before executing a merge (Issue #2605)
- Bug fix Lucene merges no longer leave a corrupted index if it fails part way through (Issue #2600)
- Feature Always defer merges during online indexing (Issue #2602)
- Bug fix Calls to AsyncUtil static methods should pass in an executor where possible (Issue #2592)
- Bug fix Lucene file lock may not be cleaned up in a failed merge (Issue #2575)
- Bug fix from is greater than to in a partition info (Issue #2580)
- Bug fix Log proper file count during listAllFiles (Issue #2584)
- Bug fix If an error happens during merge with an agile context, it may leak contexts (Issue #2574)
- Performance Lucene: avoid concurrent merges (Issue #2570)
- Performance Add metric for the size of list of files loaded by loadFileReferenceCacheForMemoization (Issue #2566)
- Feature Lucene: Merge periodically when rebalancing a lot of documents (Issue #2567)
- Performance Allow IN-union plans to be used to plan distinct queries with ordering constraints (Issue #2556)
- Feature IndexDeferredMaintenanceControl: Change the default to defer index operations (Issue #2559)
- Bug fix Revert #2557 to avoid picking union ordering keys with duplicates (Issue #2563)
- Bug fix Use metrics instead of info log when rebalancing lucene partitions (Issue #2509)
- Feature Allow Non-Unique Lucene Partitioning Field Values (Issue #2541)
- Bug fix Avoid protobuf validation errors when expanding nested repeated index expressions (Issue #2552)
- Performance Allow IN-union plans to be used to plan distinct queries with ordering constraints (Issue #2556)
- Feature IndexingMerger: if merge fails, retry with a "skip rebalancing" flag (Issue #2537)
- Bug fix Permuted min and max indexes on repeated fields now index all entries if defined on a
FanOutexpression (Issue #2543) - Performance Implement a proper copy-on-write
replaceforTreeLike(Issue #2500) - Feature API for robust matching of
Values (Issue #2499) - Breaking change Permuted min or max indexes on repeated fields need to be rebuilt to ensure completeness (Issue #2543)
- Bug fix The
PlanOrderingKeycan now handle multiple instances of the same expression appearing within the same key (Issue #2452) - Bug fix The value portion of a
KeyWithValueExpressionno longer contributes to thePlanOrderingKey(Issue #2469) - Bug fix Synthetic indexes built during
checkVersionuse the correct index maintainers. Adopters using synthetic records may need to rebuild or scrub older indexes (Issue #2530) - Feature Support deleteRecordsWhere for some join indexes (Issue #2532)
- Feature Add an option for other non-time size events (Issue #2525)
- Feature FDBDirectoryLockFactory: Implement heartbeat during ensureValid (Issue #2539)
- Feature Harden Lucene continuations with respect to background work (Issue #2523)
- Feature FDBDirectoryLockFactory: better handling of lock's TTL (Issue #2535)
- Feature The planner will now select in-union plans in cases when the index contains additional columns that are not specified in the requested ordering (Issue #2493)
- Bug fix Coercion logic does not handle
Enumtype correctly (Issue #2517)
- Bug fix The index scrubbers now can handle value indexes on synthetic record types (Issue #2513)
- Bug fix OnlineIndexer.IndexingPolicy.toBuilder is missing some elements (Issue #2521)
- Bug fix Indexes on synthetic types are now included in
deleteRecordsWhereoperations (Issue #2502) - Bug fix Use Locale.ROOT when parsing lucene queries (Issue #2503)
- Feature Unnested record types now support range deletion via
deleteRecordsWhere(Issue #2502) - Feature Disabled indexes no longer participate in
deleteRecordsWhereoperations (Issue #2505) - Breaking change Unnested record types no longer produce synthetic records for empty repeated fields (Issue #2512)
- Bug fix Aggregate index match candidate does not name the grouping keys in the underlying select expression (Issue #2506)
- Feature Add map support for Maps in Lucene (Issue #2488)
- Feature Optimize insertion of older Lucene documents to avoid unnecessary partition re-balancing (Issue #2494)
- Feature Use multiple partitions to perform rebalancing & merge all partitions (Issue #2491)
- Bug fix Sync files referencing FieldInfosId (Issue #2455)
- Bug fix Support closing FDBIndexOutput (Issue #2457)
- Bug fix
FDBExceptionsnow correctly wrapsInterruptedExceptionsinstead of throwing anIllegalArgumentException(Issue #2485) - Bug fix Fix topological sort instability (Issue #2476)
- Performance Cascades Pre-order Iterable is inefficient (Issue #2481)
- Feature Allow running Lucene's tests with a random seed & multiple iterations (Issue #2479)
- Feature Support rearranging documents between Lucene partitions (Issue #2465)
- Feature The Cascades planner can now plan against aggregate indexes with repeated grouping columns (Issue #2472)
- Feature Online Indexing: add reverse order option (Issue #2474)
- Bug fix The
PlanOrderingKeynow can tolerate a field appearing in both the equality bound keys and the non-equality bound ones (Issue #2462) - Bug fix Normalization of the plan ordering keys now avoids breaking up columns coming from a
ListKeyExpression(Issue #2463) - Performance Boolean normalization now handles (non-repeated) nested fields allowing for additional query plans to be matched (Issue #2452)
- Feature There is now a proto serialization of the
RecordQueryPlannerConfigurationclass to support planning with the same configuration with multiple JVMs (Issue #2461)
- Feature Support for Cross-Partition Queries (Issue #2432)
- Feature add feature to serialize/deserialize plans (Issue #2443)
- Feature IndexMerge should include key_space_path in logs (Issue #2436)
- Breaking change Support for the Protobuf 2 runtime has been removed from all artifacts (Issue #1637)
The Guava dependency version has been updated to 31.1. Projects may need to check for compatibility with their own versions of the dependency, or they may need to consider using the shaded artifact if the new version is incompatible.
- Bug fix
OfTypeValuedoes not work correctly with array types (Issue #2440)
- Feature Permuted min and max indexes are now available during planning and aggregate function execution (Issue #2418)
- Feature Add implementations of all sub-classes of
BaseIndexFileFormatTestCase(Issue #2421) - Feature Lucene merge with agility context: allow retry with diminished quotas upon failures (Issue #2429)
- Bug fix Fixe Unbalanced log info exception (Issue #2392)
- Bug fix Make stored fields test less flaky (Issue #2401)
- Feature Add lucene partitioning metadata (Issue #2390)
- Feature Lucene: cross transactional merge path - extra cleanup (Issue #2415)
- Feature FDBDirecotry: remove references to caller context to avoid future mis-usage (Issue #2412)
- Feature LuceneScaleTest: Add merge options to test configuration (Issue #2410)
- Bug fix To avoid breaking continuations during a deployment, the improvement to union key matching added in 3.3.348.0 is now guarded by a planner configuration flag (Issue #2408)
- Feature Lucene Property: Add a property to disable agility context (Issue #2403)
- Bug fix Fix Unbalanced log info exception (Issue #2392)
- Performance Support segment merging across transactions & larger segments (Issue #2375)
- Performance The old planner chooses a better plan ordering key for union comparisons to allow ordered union plans in more cases (Issue #2336)
- Bug fix
IndexAggregateFunctionCall.isGroupingPermutableis now too liberal (Issue #2388)
- Bug fix Synthetic record type plans are now only generated if the type has at least one index that requires maintenance (Issue #2377)
- Bug fix Store file content on own file references (Issue #2295)
- Bug fix Store FieldInfo on file reference (Issue #2284)
- Performance Cache parsed FieldInfos in lucene index (Issue #2374)
- Feature Add exception info to the applicable IndexingMerger log messages (Issue #2372)
- Bug fix Fix
Columnhash calculation (Issue #2369)
- Feature indexMerge: perform the merging in multiple transaction (Issue #2343)
- Bug fix Convert FDBDirectory.writeData to a regular func (Issue #2351)
- Bug fix Fix BufferOverflowException triggered by FDBIndexOutput (Issue #2345)
- Feature Introduce the synthetic record type
UnnestedRecordTypeto allow for more expressive indexes on nested messages (Issue #2313)
- Feature IndexAggregateFunctionCall.extractEqualityBoundFields does not recognize one-of components (Issue #2341)
- Bug fix desensitize plan hash computations from output types (Issue #2332)
- Performance clean up computation of plan hashes (Issue #2346)
- Bug fix RecordQueryAbstractDataModificationPlan does not produce stable hash with CoercionTrieNode (Issue #2330)
- Bug fix mutateRecord in RecordQueryAbstractDataModificationPlan fails when underlying metadata ref changes. (Issue #2337)
- Performance secondary node slot index for r-trees (Issue #2215)
- Feature Lucene: add metric to measure Lucene's find merges (Issue #2333)
- Bug fix Set the UPDATE's PipelineSize to 1 (Issue #2324)
- Bug fix Fix hash code instability in
RecordQueryAbstractDataModificationPlan(Issue #2325) - Bug fix Rethrow original IOExceptions exceptions in lucene code (Issue #2293)
- Bug fix Add proper clone to LuceneStoredFieldsReader (Issue #2297)
- Performance Lucene performance: avoid merge calculation before deferring merge [(Issue #2319)](FoundationDB#2319
- Feature LuceneLoggerInfoStream should check if trace is enabled in isEnabled (Issue #2317)
- Feature planner support for multidimensional indexes (Issue #2305)
- Performance Lucene: avoid segments merge at closing (Issue #2279)
- Feature Support Java UDFs in Cascades (Issue #2307)
- Bug fix Throw RecordCoreException when missing FileReference (Issue #2291)
- Bug fix Add cause to LuceneRecordCursor "Failed to get document" exception (Issue #2283)
- Bug fix segregate different types of lucene querying (Issue #2276)
- Bug fix Fix coerceArray to work with list of primitives (Issue #2274)
- Feature Provide support for empty list in InOpValue (Issue #2270)
- Performance Remove unnecessary supplyAsync from FDBDirectory.writeSchema (Issue #2237)
- Performance Indexing throttle: reduce limit faster during repeating failures (Issue #2240)
- Feature multidimensional index maintainer (Issue #2215)
- Bug fix COALESCE does not work with non-scalar types (Issue #2242)
- Bug fix Change log level of "found lucene analyzer" to debug (Issue #2250)
- Feature Re-word misleading error message "unable to build index (Issue #2247)
- Bug fix Resolve Lucene readSchema deadlock (Issue #2234)
- Bug fix Splitting disjunction into union produces duplicates (Issue #2230)
- Performance Lucene: allow skipping checkIntegrity for all PostingsFormat inheritors (Issue #2224)
- Feature OnlineIndexer: IndexingThrottle: re-write limit handling (Issue #2106)
- Bug fix Lucene: allow fieldsProducer close (and commit) only if it was initialized (Issue #2222)
- Performance a wider index should be preferred over a narrower index if choosing an index scan over the wider index avoids a fetch (Issue #2218)
- Feature Stores can now be configured to update their state cacheability during store opening or creation (Issue #2203)
- Feature Add more metrics to Lucene operations (Issue #2210)
- Feature Support Greatest and Least scalar functions (Issue #2206)
- Feature Support coalesce scalar function (Issue #2206)
- Feature Lucene: add context property to skip data integrity check (Issue #2198)
- Feature Add a way to relax stop qord queries (Issue #2196)
- Bug fix Parameterized dual-planner test results do not show test name (Issue #2201)
- Feature Lucene: add context property to skip data integrity check (Issue #2198)
- Bug fix Lifting Java type to Cascades type is inconsistent (Issue #2176)
- Bug fix Fix grouping requested order if it is empty (Issue #2150)
- Performance only break ORs into UNIONs if beneficial for index matching (Issue #2147)
- Bug fix flatmap should honor skips and limits (Issue #2140)
- Feature cascades planner needs to enumerate OR factorizations properly (Issue #2131)
- Bug fix Removed infinite recursion from
LuceneIndexQueryPlan'stoStringimplementation (Issue #2136) - Feature Record versions are now queryable by the Cascades planner, including mathcing on version indexes (Issue #2089)
- Feature Allow placeholders in aggregate index match candidate's underlying
SelectExpression(Issue #2135)
- Bug fix The plan string for
RecordQueryMapPlanis reverted back to its value before 3.3.375.0 (Issue #2132)
- Bug fix Filtered match candidate is incorrectly considered for unfiltered scan (Issue #2118)
- Feature The
PlanStringRepresentationclass separates out plan explain printing fromRecordQueryPlan::toStringincluding adding logic to create length-limited strings (Issue #2112)
- Feature Add throttling tags to context config (Issue #2115)
- Feature Add optional conflict tracking to context config (Issue #2116)
- Feature Refactor KeySpaceTreeResolver out of KeySpaceCountTree (Issue #2121)
- Bug fix FDBRecordStore: keep userVersionChecker during copyFrom store (Issue #2109)
- Feature Use Index.predicate as an index maintenance filter when appropriate (Issue #2069)
- Bug fix Captured constraints of filtered indexes are too restrictive (Issue #2104)
- Bug fix Restore a removed method from
CascadesPlannerand mark it for deprecation (Issue #2102)
- Bug fix A bug in mutual indexing may cause low performance (Issue #2100)
- Feature Literal extraction and plan constraints in Cascades (Issue #2094)
- Bug fix Don't write to disabled Synthetic Indexes (Issue #2099)
- Bug fix Add support for
int32in record metadataValue(Issue #2091)
- Performance Mutual indexing - better handle final iteration conflicts (Issue #2082)
- Feature The
LocatableResolverinterface has been enhanced with read-only and transaction-scoped APIs (Issue #2062) - Feature Indexing stamp operations: return the post change values (Issue #2080)
- Feature Remove old Lucene auto-complete functionality and create a proper composable query clause for auto-complete functionality (Issue #2083)
- Breaking change Remove old Lucene auto-complete functionality and deprecated auto-complete-related indexing options (Issue #2083)
- Bug fix Improve covering field path computation for the case where a normalized field still takes up more than position (Issue #2085)
- Bug fix Fixes correcting issues introduced in #2058 (Issue #2061)
- Feature Feature 1 (Issue #2071)
- Feature improve join enumeration and plan memoization (Issue #2061)
- Bug fix Make testMutualIndexingBlocker less volatile (Issue #2066)
- Feature mutual indexing - add fragment information to throttle logs (Issue #2059)
- Feature Pull up getRecordState to the FDBRecordStoreBase (Issue #2065)
- Performance An unnecessary recount of records on new stores with a record count key has been removed (Issue #2052)
- Feature Supports bitset_contains in Lucene query
- Bug fix The
missingRanges method inRangeSets now adds more precise read conflict ranges to avoid introducing unneccessary conflicts (Issue #2024)
- Bug fix resolves #2035: lucene index auto-complete should adhere to limits as well as should populate primary key info correctly (Issue #2035)
- Feature refactor lucene index auto complete and spell check (Issue #2035)
- Bug fix Cascades planner no longer trips over
VALUEindexes which contain all fields in the primary key (Issue #2015)
- Feature Make Index.options immutable. This replaces the internal data structure of Index.options with an immutable map so that a user cannot call getOptions().put()(Issue #2016)
- Feature Online Indexing: Add transaction time quota and initial scanned records limit (Issue #2013)
- Bug fix OnlineIndexer: make stamp operations public (Issue #2027)
- Feature improve logic to push plans through a fetch (Issue #2020)
- Bug fix The
StoreSubTimerclass now forwards metrics to an underlying timer whenaddis called, which previously could cause metrics reported only on commit to disappear entirely (Issue #2010)
- Bug fix throttleDelayAndMaybeLogProgress causes NullPointerException (Issue #2006)
- Feature Sparse index support in Cascades (Issue #1997)
- Feature Indexing typestamp: add query + flow control (Issue #1980)
- Bug fix The
EMPTY_SCANSmetric no longer double counts empty scans that callhasNextmultiple times (Issue #2002) - Performance Additional instrumentaiton of the
RangeSetis added to account for time spent during index builds (Issue #1995) - Performance Metrics that are dependent on transaction commit will only be recorded after its associated transaction is successfully committed (Issue #1998)
- Performance Non-unique indexes no longer read or clear the uniqueness violation space during maintenance (Issue #1973)
- Performance Adds store timer metrics to the indexer progress metrics message (Issue #1984)
- Performance Instrumentation is added for delays inserted during retries and index builds (Issue #1993)
- Feature planner should only attempt in-to-join if plan uses correlated predicate as a search argument for a scan (Issue #1985)
- Performance Default value of
LOAD_RECORDS_VIA_GETSproperty reverted tofalseto match behavior prior to 3.3.327.0 (Issue #1982)
- Bug fix Cascades Aggregate index plan matching ignores predicates (Issue #1974)
- Performance Add optional mechanism for loading records via single-key gets instead of a scan (Issue #1967)
- Performance A new counter,
EMPTY_SCANS, tracks how many empty range reads are executed (Issue #1970)
- Feature Refactor KeyValueCursor to allow extension (Issue #1957)
- Feature Expose version change info from FDBRecordStore (Issue #1965)
- Feature Mutual concurrent indexing (Issue #1853)
- Feature Indexing: Throw PartlyBuiltException when appropriate (Issue #1961)
- Bug fix fix bug to allow non-covering lucene index scans over synthetic records (Issue #1959)
- Feature Support for IN predicates in Cascades planner (Issue #1955)
- Bug fix Refactor ThenKeyExpression#normalizeKeyForPositions to reduce hotspot detected by removing stream() implementation to reduce new object creation (Issue #1940)
- Performance Unrolled record deletes (introduced in 3.3.309.0 are now on by default (Issue #1942)
- Bug fix Fix byte counting bug for remote fetch (Issue #1934)
- Performance Reduce the number of extraneous clear ranges issued during
checkVersion(Issue #1936)
- Bug fix Indexing: periodic "Built Range" log message is broken (Issue #1898)
- Performance The number of array copies when constructing continuations should be decreased by relying more on
ByteStringinternally (Issue #1923)
- Bug fix Fix covering plan for a Lucene index over a Synthetic Record Type (Issue #1927)
- Bug fix Sorting and comparing nested fields get confused (Issue #1907)
- Feature Non-idempotent target indexes can now be built from an existing index (Issue #1430)
- Feature Support additional streaming modes: LARGE, MEDIUM, SMALL (Issue ##1915)
- Feature Lucene search with highlighting the terms (Issue #1862)
- Bug Pulling up a Value through Ordering is not entirely correct (Issue #1905)
- Feature Support planning aggregate indexes in Cascades. (Issue #1885)
- Bug fix ensure that partial records are created as proper protobuf messages (Issue #1893)
- Bug fix Account for FDBStoreTimer.Counts.BYTES_READ in remote fetch (Issue #1889)
- Bug fix The
RecordQueryPlannernow chooses more efficient indexes for and-queries with one-of-them predicates on repeated nested fields (Issue #1876) - Performance Single record deletes can now be configured to be executed as multiple single-key clears instead of one range clear, which should be an improvement for LSM-based FDB storage engines (Issue #1493)
- Feature Add a method to construct a single record from a raw indexed record (Issue #1865)
- Feature Joined record types now allow some kinds of function key expressions in their join conditions (Issue #1872)
- Feature Allow users to name enum types in
TypeRepository(Issue #1847) - Feature Support planning of (covering) index scans, fetches over synthetic record types (Issue #1886)
- Bug fix
TimeWindowLeaderboardWindowUpdaterebuild=NEVERis not honored for new directory (Issue #1878)
- Performance Looking up logical values from
DirectoryLayerDirectorys no longer needs to create new transactions (Issue #1857)
- Breaking change Enable incremental builds. Upgrade guava dependency to 31.1-jre. (Issue #1868)
This version of the Record Layer allows the FDB API version to be configured through the FDBDatabaseFactory. This means that while this version allows the client to be configured to use 7.1 features, it also supports connecting to 6.3 FDB clusters if the API version is set appropriately. Note that setting the API version does restrict the set of potential FDB server versions that can be connected to, so this configuration change should only be made if the FDB server has already been updated.
New index state "READABLE_UNIQUE_PENDING" - the proper way to roll this feature out is:
- The adopter should upgrade to the new Record Layer version and deploy the version everywhere.
- The format version should be set READABLE_UNIQUE_PENDING_FORMAT_VERSION.
- Only after all the possible clients are upgraded to support the new state, the adopter may set the allowPendingState on the indexing policy of new index builds. An index may be in this new state if it is fully built, the unique flag is set, and duplications were found during online indexing. From the code point of view, it is defined as scannable but not readable.
The FoundationDB Java binding dependency has been updated to 7.1 with this release. This means that clients also need to update their main FDB C client to a 7.1 version. Adopters that still wish to connect to an FDB cluster running a 6.3 or 7.0 server version can do so by packaging additional FDB C clients at the appropriate version(s) using the FDB multi-version client feature.
This release also updates downstream dependency versions. Most notably, the proto3 artifacts now require Protobuf version 3.20.1.
- Bug fix incorrect assertion of PK length fails remote fetch with fallback (Issue #1845)
- Feature Support for remote fetch when common primary key is unavailable. This changes the (experimental) API in a non-backwards-compatible way - the common-primary-key is no longer required and can be calculated internally (Issue #1805)
- Feature Merge online indexer sub modules - "by records" and 'multi target by records" (Issue #1432)
- Feature Introduce JMH framework (Issue #1157)
- Bug fix Lower the severity of error when logging incompatible index for remote fetch (Issue #1828)
- Feature Make
FDBRecordContextremember its config (Issue #1830)
- Feature Use different analyzers for different fields in one Lucene index, and enable exclusion of fields for auto-complete (Issue #1824)
- Feature Cascades planning of logical group by (Issue #1791)
- Bug fix Fix flaky test by allowing larger than expected scans (Issue #1788)
- Performance Reverse directory lookups can now borrow a read version from another transaction to avoid GRV costs (Issue #1714)
- Feature Query returns null for unset non-repeated fields (Issue #1718)
- Feature Emit a different KPI for each reason of index rebuilds (Issue #1798)
- Feature Trigger a log message for every index state change (Issue #1809)
- Feature Support READABLE_UNIQUE_PENDING IndexState (Issue #1611)
- Bug fix Update client event parsing to handle tenant (Issue #1802)
- Performance A new index scan type variant has been added that make better use of existing caches in repeated executions of the same query (Issue #1758)
- Bug fix Add log for fallback triggered (Issue #1782)
- Bug fix Fix fallback for scanIndexRecords (Issue ##1761)
- Bug fix Proximity search does not works well when the term has multi-word synonyms (Issue #1752)
- Feature Index Remote Fetch for index scan methods (Issue #1751)
- Bug fix Lucene auto-complete should apply the limit to the result cursor (Issue #1745)
- Bug fix FDB Java dependency version updated to 7.1.10 to incorporate fixes over older 7.1 versions (Issue #1743)
- Feature Use Lucene query component to pass in the parameter for auto-complete's highlighting option (Issue #1740)
- Feature Online Indexer: allow fallback to by-record if the previous stamp is multi-target (Issue #1736)
- Bug fix Fix double counting of limits in remote fetch (Issue #1730)
- Feature Improve remote fetch fallback to allow fallback after a few records have been returned (Issue #1727)
- Feature API version for tests changed to 7.1 (Issue #1734)
- Bug fix Cascades does not reconstruct a KeyWithValue correctly (Issue #1725)
- Feature Publish test jar as part of the regular distribution (Issue #1703)
- Feature UserVersionChecker should be getting RecordMetaDataProto.DataStoreInfo (Issue #1710)
- Bug fix Failed no-ops no longer log at
ERROR(Issue #1692) - Performance Lucene auto-complete is now handled by running queries on the main index to allow it to avoid needing a separate directory (Issue #1682)
- Feature Cursors now can map updates to their continuations via the
.mapContinuationmethod (Issue #1663)
- Performance Lucene merge based on probability (Issue #1676)
- Bug fix Synchronize TransactionalRunner.close (Issue #1659)
- Feature The FDB API version can now be configured through the
FDBDatabaseFactory(Issue #1639) - Feature Match index hints in a query (Issue #1671)
- Feature Remote Fetch feature using FDB's getMappedRange new API (Issue #1560)
- Breaking change Extenders of StandardIndexMaintainers will inherit the scanRemoteFetch implementation. If this is not suppoted for the subclass, override the method (Issue #1560)
- Bug fix FDBRecordContext::newRunner drops property storage (Issue #1645)
- Breaking change The FDB dependency has been updated to require version 7.1 (Issue #1636)
- Breaking change Additional downstream dependency versions have been upgraded, including Protobuf (Issue #1640)
This version of the Record Layer changes the Java source and target compatibility to version 11. Downstream projects need to be aware that the byte code produced is of class file version 55.0 going forward.
- Feature Create a transactional runner that does not retry (Issue #1615)
- Feature Support auto-complete suggestions for phrases (Issue #1630)
- Feature Have separate analyzer pairs for Lucene full-text search and auto-complete suggestions (Issue #1627)
- Performance An optimized BlendedInfixSuggester for auto-complete suggestion, that does not store term vectors (Issue #1624)
- Performance A file sequence counter used to map file names to an internal ID by the Lucene directory implementation is now cached in memory (Issue #1577)
- Bug fix Scans of empty Lucene index now return an empty cursor instead of throwing and error (Issue #1597)
- Feature The
LuceneIndexMaintainernow supports delete where operations if the filter is on the grouping columns of the index (Issue #1582)
- Bug fix FDBDatabaseRunnerImpl correctly trends towards maxDelayMillis, not 0 (Issue #1565)
- Bug fix Delete records where limits indexes on grouping key expressions to predicates that can be satisfied by only the grouping columns (Issue #1583)
- Bug fix Blocking calls within the Lucene index maintainer implementation now use
asyncToSyncto wrap exceptions, control timeouts, and report metrics (Issue #1571) - Performance The Lucene directory implementation now caches the complete list of files in a directory (Issue #1575)
- Performance Improve performance of chooseK algorithm by using integer counters for state management instead of Iterator heap objects (Issue #1590)
- Bug fix "rebuilding index failed" does not include primary_key (Issue #1572)
- Bug fix Delete records where now handles indexes on key-with-value expressions that split at locations that are in the middle of function key expressions (Issue #1563)
- Feature Lucene plans should support simple comparisons on indexed fields and fielded search (Issue #1238)
- Breaking change Adjust Lucene index to use new scan bounds (Issue #1517)
- Breaking change Lucene index entry grouping key consistency (Issue #1528)
- Breaking change Lucene specific code should be in the lucene module (Issue #1529)
- Feature Type system for Values (Issue #1545
- Feature Index scrubber - return the count of bad entries found (Issue #1547)
- Feature Allow scrubbing of non-"value" indexes (Issue #1551)
- Feature Support custom additional synonyms. This introduces a new SynonymMapRegistry.
New synonym maps should implement
SynonymMapConfig. See exampleEnglishSynonymMap.
- Feature Reconsider IndexMaintainer.scan signature (Issue #1506)
- Feature Expose IndexQueryabilityFilter for Aggregate planning (Issue #1520)
- Breaking change As part of (Issue #1520) implementers
of
FDBRecordStoreBaseneed to implement a new overload ofgetSnapshotRecordCountForRecordTypeandevaluateAggregateFunctionthat takes anIndexQueryabilityFilter. In addition some methods onIndexFunctionHelperandComposedBitmapIndexAggregatenow take anIndexQueryabilityFilter; to preserve backwards compatibility, if all indexes are valid,IndexQueryabilityFilter.TRUEcan be used.
- Feature Support auto complete suggestions for Lucene search (Issue #1504)
- Feature Chooser and Comparator plans (Issue #1471)
- Feature Handle Comparator plan with keys that resolve to repeated fields (Issue #1501)
- Breaking change As part of a refactoring of log keys, the log key
LogMessageKeys.ORIGINAL_DATA_SIZEnow displays asoriginal_data_sizeinstead oforiginal_data-sizeand a typo has been fixed in the name ofLogMessageKeys.DIRECTORY(fromDIRECTOY) (Issue #1500)
- Breaking change Clean up LuceneDocumentFromRecord. Changes definition format for full-text fields; indexes will need to be redefined and rebuilt. (Issue #1235)
- Bug fix Use 2 separate analyzers for indexing time and query time in LuceneIndexMaintainer (Issue #1486)
- Feature Suppor synonym for Lucene indexing (Issue #1488)
- Feature Passing properties configured by adopter into FDBRecordContext (Issue #1478)
- Online Indexer: add time limit to OnlineIndexer.Config (Issue #1459)
- Breaking change Support compression for Lucene data. Backwards compatibility is not supported, so upgrading requires rebuilding Lucene indexes during which search requests cannot be completed. (Issue #1466)
- Feature Make QueryPlanResult and QueryPlanInfo immutable (Issue #1456)
- Breaking change Up java source and target compatibility to 11 (Issue #1454)
- Breaking change Codec with a few optimizations for speeding up compound files sitting on FoundationDB. Backwards compatibility is not supported, so rebuilding Lucene indexes with shutting down search requests handling is needed for transition. (Issue #1457)
This version of the Record Layer removes some legacy elements of the API that were deprecated in previous releases. Most notably, it removes the methods on the RecordCursor interface that were compatible with Java Iterators. That API was deprecated in version 2.6 to make it easier for adopters to reason about continuations in asynchronous code by associating each value returned by the cursor with that value's continuation. Adopters still using the deprecated API can either use the onNext() and getNext() methods on the RecordCursor interface or call asIterator() to get a RecordCursorIterator, which retains compatibility with the Iterator interface.
Another, smaller change that has been made is that by default, new indexes added to existing stores (that cannot be built in-line) are now initialized with a DISABLED IndexState whereas the index used default to a WRITE_ONLY state. This means that any records written to the record store prior to the index being built will not perform any I/O to update the index, which is effectively wasted work. However, all indexes must be put in the WRITE_ONLY state while they are being built in order to ensure that any updates to the index during the build are captured. This is something that the OnlineIndexer should be able to handle automatically for most users, but users of the ERROR_IF_DISABLED_CONTINUE_IF_WRITE_ONLY index state precondition may start seeing additional RecordCoreStorageExceptions with the message "Attempted to build non-write-only index" when attempting to build an index. That IndexStatePrecondition is not reccommended, however, and users should switch over to using a different IndexStatePrecondition (like the default index state precondition, BUILD_IF_DISABLED_CONTINUE_IF_WRITE_ONLY) instead or explicitly set the index state on the index to WRITE_ONLY prior to building the index. Users can also replicate the old behavior by supplying a UserVersionChecker implementation with an appropriate implementation of needRebuildIndex to the FDBRecordStore.Builder.
- Bug fix allow planner rules to be disabled via planner configuration (Issue #1445)
- InstrumentedTransaction: Distinguish reads from range_reads and deletes from range_deletes (Issue #1435)
- Bug fix Uniqueness violations encountered in new indexes built during store opening times now fail the index build (Issue #1371)
- Feature Streaming aggregate operator implementation (Issue #1376)
- Feature Online Indexer: support multi target indexing (Issue #1398)
- Feature Added store timers to track compression efficacy (Issue #1412)
- Feature Publish per-transaction I/O metrics and commit failure metrics (Issue #1402)
- Bug fix Fix potential null recordTypeKey in RecordType.getRecordTypeKeyTuple() (Issue #1395)
- Feature integrate boolean normalization into planning process (Issue #1301)
- Index Scrubber: missing index entry”: Add the (virtual) missing index’s key to the log message (Issue #1392)
- Bug fix Protect against corrupt input in GeophileQueryTest (Issue #1377)
- Bug fix relax conditions enforced in ImplementInUnionRule (Issue #1369) *OnlineIndexScrubberTest.testScrubberLimits - reduce number of records (Issue #1363)
- OnlineIndexScrubber: clear ranges when exhausted (Issue #1367)
- Feature Prototype non-index sorting (Issue #1161)
- OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange timeout failures (Issue #1267)
- Feature Cascades planner can plan IN as a dynamic UNION (Issue #1339)
- Breaking change InUnion uses a different plan hash (Issue #1339)
- Index scrubber: log warnings when repair is not allowed (Issue #1345)
- Use IndexingPolicy instead of indexStatePrecondition (Issue #1336)
- Feature Indexes can now be marked as being replaced by other indexes (Issue #1315)
- Feature PlanOrderingKey for Intersection / (ordered) Union (Issue #1319)
- Feature
AutoContinuingCursorcontinues on retriable exceptions (Issue #1328) - Feature Validate and repair LocatableResolver mappings (Issue #1316)
- Feature Provide a method to clean readable indexes build data (Issue #1334)
- Bug fix Fix a typo in DataAccessRule.comparisonKey() (Issue #1324)
- Feature Added RecordCursor.asList() with continuation support (Issue #1307)
- Bug fix IN that cannot be made into OR with incompatible sort gets stack overflow (Issue #1283)
- Feature IN join with Union instead of concat (Issue #1286)
- Bug fix Index scrubber should consider record filtering (Issue #1313)
- Feature Allow stores to check the total size (in bytes) when determining index states rather than the number of records (Issue #1239)
- Feature Method to convert to ByteString in RecordCursorContinuation (Issue #1299)
- Feature FDB server request tracing is now exposed through one of the context configuration options (Issue #1304)
- Feature Add indexing validation & repair (Issue #1266)
- Bug fix Scanning a record store with split records and a scan limit could sometimes result in errors with the message
attempted to return a result with NoNextReason of SOURCE_EXHAUSTED but a non-end continuation(Issue #1294)
- Feature Multi-valued queryable functions (Issue #1289)
- Bug fix OneOfThemWithComparison.withOtherComparison lost emptyMode (Issue #1284)
- Feature Define query compatibility using store state and pre-bound parameters (Issue #1275)
- Breaking change IndexQueryabilityFilter is not a functional interface anymore (Issue #1275)
- Bug fix The
AutoServiceannotations are no longer retained at in published jars as was the case prior to 3.0.181.0 (Issue #1281)
- Feature Methods to estimate the size of a record store are now exposed (Issue #1229)
- Breaking change The iterator style
RecordCursorAPI is removed (Issue #1136) - Breaking change The
RecordCursor::limitTomethod has been removed in favor ofRecordCursor::limitRowsTo(Issue #1189) - Breaking change The
RecordCursor::orElseandRecordCursor::flatMapPipelinedmethods have been removed in favor of variants that allow for continuations to be correctly handled (Issue #1189) - Breaking change Remove non-controversial deprecated methods (Issue #1191)
- Breaking change Indexes on existing stores will now default to the
DISABLEDIndexStateinstead ofWRITE_ONLYif they cannot be built inline (Issue #1213) - Breaking change Increased the versions of our dependencies, including Protobuf (to 3.15.6) and Guava (to 30.1-jre) (Issue #1193)
- Breaking change Requires a minimum FoundationDB client and server version of 6.3 (Issue #1201)
In this release, the various implementations of the RecordQueryPlan interface have moved to API stability level INTERNAL. This means that individual implementations may change without notice. Clients that are not creating RecordQueryPlan objects directly (but instead using the planner to create plans) should not be affected.
- Bug fix LiteralKeyExpression should ignore Value for planHash and queryHash (Issue #1273)
- Bug fix Uses of FDBDatabaseRunner include logging details for exception logging (Issue #1255)
- Bug fix Passing a
RecordContextto aLocatableResolvercould fail with a non-retriable error when comparing committed and non-committed data in an in-memory cache (Issue #1258)
- Add indexing method to build 'index online log' message (Issue #1242)
- Feature Rebuild index when prev by-index continuation isn't possible (Issue #1232)
- Feature Indexing refactor: change OnlineIndexer.IndexFromIndexPolicy to a general OnlineIndexer.IndexingPolicy (Issue #1223)
- Feature Indexing: Handle the case of requested method mismatches a previous session. Options are: use previous, rebuild, and error (Issue #1198)
- Bug fix allow NULL_UNIQUE as a stand-in for fields that are used to compute covering index optimizations (Issue #1220)
- Feature Indexing: log throttle delay (Issue #1218)
- Bug fix Planner's key from index should include primary keys (Issue #1138)
- Performance RecordQueryPlanner.getKeyForMerge can build redundant merge keys (Issue #1154)
- Feature Need a way to tell whether the result of planning a query is only sorted by the requested keys (Issue #1155)
- Bug fix represent an aggregation function as using a set of aggregation keys (Issue #1175)
- Bug fix
GroupingValidatoronly does one level of nesting (Issue #1172) - Feature Add QueryPlanResult to hold planned query and PlanInfo. (Issue #1176)
- Feature Rank by grouped map-like values (Issue #1183)
- Feature
VersionstampSaveBehavior.IF_PRESENTallows the user to specify that a record should be saved with a version only if one is explicitly provided (Issue #958)
- Bug fix
RecordQueryPlanner.AndWithThenPlanner.planChilddoes not have a case likeplanAndWithNesting(Issue #1140) - Bug fix Covering check when chosen index has duplicate fields (Issue #1139)
- Bug fix
LiteralKeyExpressionfailshasProperInterfaces(Issue #1152) - Bug fix
@DualPlannerTestdoes not distinguish tests (Issue #1150) - Performance represent an aggregation function using a set of aggregation keys rather than a KeyExpression (Issue #1175)
- Bug fix Fix LiteralKeyExpression hash code (Issue #1066)
- Bug fix Stable plan from cost perspective (Issue #1148)
- Feature Allow creating a target index by iterating a source index of the same type (Issue #1078)
- Feature Add persistent continuation and parallel indexing to indexing by index (Issue #1107)
- Feature Add configuration for Cascades planner, protect from queue explosion and infinite loop, use interface instead of RecordQueryPlanner (Issue #1126)
- No changes from version 2.10.155.0
- Feature Certain system keys are exposed through the new
FDBSystemOperationsAPI (Issue #949) - Performance Query plan costing model is adjusted to not prefer full record scans over index scans with the same residual filters (Issue #1130)
- Bug fix Scanned records counter was not cleared properly when clearing the index (Issue #1115)
- Bug fix Fix the instruction document of building Record Layer (Issue #1061)
- Feature Permuted grouped aggregate (Issue #1085)
- Feature integrated index matching and planning in the Cascades planner (Issue #1083)
- Feature Implement QueryHash to identify created queries and match them to generated plans (Issue #1091)
- Breaking change The addition of QueryHashable and the implementation of it by various classes will force existing extenders of some classes to implement the queryHash() method (Issue #1091)
- Feature Count indexes can be cleared when decremented to zero (Issue #737)
- Performance Improvement: OnlineIndexer: Add index name to "build index online" log message (Issue #1081)
- Feature Implement different kinds of planHash for use with plan hash that ignores literals (Issue #1072)
- Breaking change planHash() is now a default method implementation in PlanHashable, delegating to planHash(PlanHashKind). As a result, code will have to be recompiled to use this default (Issue #1072)
- Feature General index matching framework and mechanics needed for the new planner implementation. (Issue #1026)
- Bug fix "make unbuilt index readable" exception blocks opening a record store (Issue #1054)
- Feature Allow degenerate composed bitmaps from single index (Issue #1068)
- Feature Composed bitmaps from map-like fields (Issue #1057)
- Bug fix Record scans with a time limit could result in errors or missing versions (Issue #1046)
- Feature Support checking whether or not an index is being built (Issue #1043)
- Bug fix Need grouping / grouped key canonical form (Issue #1052)
- Performance preloadRecordAsync() caches negative results (Issue #1048)
- Bug fix IndexFunctionHelper.getGroupedKey / getGroupingKey can get array bounds error (Issue #1050)
- Performance Covering filter optimizations and index-applicable filters are now considered as advantageous during planning (Issue #1039)
- Bug fix Sampled key read operation can log a lot (Issue #1020)
- Feature FDB client run-loop profiling can be enabled on the
FDBDatabaseFactory(Issue #1041)
- Bug fix Resolves concurrency issue in executing plans with delayed record fetches in multi-threaded environments (Issue #1024)
- Bug fix Requested full text fields always contain the full text, not just a single token (Issue #1017)
- Performance Covering filter optimizations can now be performed below union, intersection, and distinct operators (Issue #1016)
- Feature Avoid repeating index endpoints conflicts (Issue #1019)
- Feature BITMAP_VALUE index cursors can be bit-wise merged (Issue #1012)
- Breaking change Result type of planCoveringAggregateIndex has changed, requiring a recompile of callers (Issue #1012)
- Bug fix OnlineIndexer does not backoff properly when it retries (Issue #1007)
- Feature Bitmap value indexes (Issue #1010)
- Breaking change Several API changes in
RecordQueryPlanimplementors, which are also nowINTERNALAPIs. (Issue #987)
This version of the Record Layer requires a FoundationDB server version of at least version 6.2. Attempting to connect to older versions may result in the client hanging when attempting to connect to the database.
Additionally, builds for the project now require JDK 11. The project is still targetting JDK 1.8 for both source and binary compatibility, so projects importing the library that have not yet upgraded to the newer JDK should still be able to import the project as before, but developers may need to update their local development environment if they have not already done so.
FDBRecordStore does not have addUniquenessCheck anymore, which is replaced by checkUniqueness in StandardIndexMaintainer now; IndexMaintainer does not have updateUniquenessViolations anymore, which is replaced by addUniquenessViolation and removeUniquenessViolationsAsync in StandardIndexMaintainer now; StandardIndexMaintainer now has updateOneKeyAsync in place of updateOneKey .
OrElseCursor now uses a structured continuation instead of the previous pass-through style. This breaks the continuations of all OrElseCursors from previous builds. Furthermore, the fluent RecordCursor.orElse() method is deprecated in favor of a static method that also takes a continuation.
- Performance Union, intersection, and distinct operations can now be applied to index entries before fetching the primary record (Issue #991)
- Performance Better to scan with OR filter than to Union multiple identical scans with individual filters (Issue #1004)
- Performance Some filters can be performed with covering index scan's partial record (Issue #1000)
- Bug fix Avoid excessively long keys/values logged during assertion checks (Issue #989)
- Feature BooleanNormalizer could eliminate some redundant disjuncts (Issue #993)
- Feature OnlineIndexer limits amount of work by transaction size (Issue #703)
- Performance Fixes #977: RecordMetaData#recordTypesForIndex is N^2 with Universal index probing (Issue #977)
- Feature FDBDatabaseRunner should use FDBRecordContextConfig more (Issue #983)
- Feature Get rid of finalizer (Issue #985)
- Bug fix Event analysis is fooled by incomplete version stamps (Issue #980)
- Performance Fixes #962: Reuse Ciphers vs. creating them new each time... (Issue #962)
- Bug fix False positive of uniqueness violations prevents building indexes (Issue #901)
- Bug fix Incomplete record versions are no longer cached in a way that can lead to cache entries for one record store polluting the values for another (Issue #952)
- Bug fix
OrElseCursorproperly support continuations when using the else branch (Issue #974) - Breaking change Requires a minimum FoundationDB client and server version of 6.2 (Issue #702)
- Breaking change Builds now require JDK 11 (Issue #910)
- Breaking change
FDBRecordStoredoes not haveaddUniquenessCheckanymore, which is replaced bycheckUniquenessinStandardIndexMaintainernow;IndexMaintainerdoes not haveupdateUniquenessViolationsanymore, which is replaced byaddUniquenessViolationandremoveUniquenessViolationsAsyncinStandardIndexMaintainernow;StandardIndexMaintainernow hasupdateOneKeyAsyncin replace ofupdateOneKey(Issue #901) - Breaking change Methods interacting with the local version cache on an
FDBRecordContexthave been removed as they were previously unsafe and should have been internal only (Issue #952) - Breaking change
OrElseCursorhas a new continuation format that is incompatible with the previous continuation format (Issue #974) - Breaking change Guava and protobuf dependency versions have been upgraded to 29.0 and 3.12.2 respectively (Issue #967)
The FDBRecordStore can now use the global meta-data key that was added to FoundationDB in version 6.1 to cache state that otherwise must be read from the database at store initialization time. When opening any record store, the user can supply an instance of the new MetaDataVersionStampStoreStateCache class. (Alternatively, the FDBDatabase can be configured with a MetaDataVersionStampStoreStateCache that will be shared by all record stores opened using that database.) This class uses the value of the cluster's global meta-data key to detect if cache entries are stale, and it will invalidate older entries whenever the value of that key changes.
This cache invalidation strategy is safe as record stores can now be configured to update the global meta-data version key whenever any of the cached information is changed. To do so, the user must configure their record store to have cacheable meta-data by calling setStoreStateCacheability(true). The record store will then update the global meta-data key whenever the cached state changes. If the record store has not been so configured, then the MetaDataVersionStampStoreStateCache will not cache the store initialization information in memory, so opening the store will require re-reading this information from the database each time the store is opened.
If the client is only accessing a small number of record stores on each cluster, then using the MetaDataVersionStampStoreStateCache can speed up record store initialization time as most record stores should now be able to be opened without reading anything from the database. Additionally, enabling this feature can help with users facing scalability problems on large record stores if loading this information at store creation time becomes a performance bottleneck. However, note that if there are many record stores coexisting on the same cluster, enabling this feature on all record stores may cause the meta-data key to change with high frequency, and it can degrage the effectiveness of the cache. Users are therefore encouraged to consider carefully which record stores require this feature and which ones do not before enabling it.
Additionally, each record store now allows users to set custom fields in the store header. For background, every record store includes one extra message stored in a special location with extra information about the record store that the Record Layer uses to control behavior when manipulating that record store. For example, it includes the version of the RecordMetaData that was used the last time the record store was accessed. The header is then read every time a record store is opened, and then it can be used to detect if, for example, the meta-data verion has changed and new index builds need to be scheduled. (There are also ways of caching this information in a way that is transactionally consistent with the Record Store itself to avoid creating a performance bottleneck. See: Issue #537.) The new feature adds a mechanism for the user to set those fields with data that the user may want to update transactionally with the data but that they do not want to pay the cost of performing an additional read against the database with every operation. For example, sample use cases might include:
- Adding a flag to indicate whether a migration of data from an older record type to a newer type has completed, and then using this flag to control whether the old record type or the new record type is queried.
- Information about the history of the data in the record store, such as how many times the data have been copied between different FoundationDB clusters.
- Frequently read data like custom application-speicific options that might differ between different record stores.
Importantly, it is rather expensive to write this data (as all concurrent operations to the record store will fail with a conflict), and as the data in these fields are read with every transaction, it is not advised that the user write to these fields too often or store too much data in them. However, if there is a relatively small amount of data with a very low write-rate, it might be appropriate to use this feature. This feature also requires that the user set the format version for a record store to HEADER_USER_FIELDS_FORMAT_VERSION. The default format version was not updated to that version.
A client can now specify a custom IndexQueryabilityFilter that determines which indexes should be considered by the query planner.
A new format version, CACHEABLE_STATE_FORMAT_VERSION, was introduced in this version of the Record Layer. Users who wish to experience zero downtime when upgrading from earlier versions should initialize all record stores to the previous maximum format version, SAVE_VERSION_WITH_RECORD_FORMAT_VERSION, until all clients have been upgraded to version 2.8. If the update is done without this measure, then older clients running 2.7 or older will not be able to read from any record stores written using version 2.8. The new format version is also required to use the new MetaDataVersionStampStoreStateCache class to cache a record store's initialization state.
This version of the Record Layer requires a FoundationDB server version of at least version 6.1. Attempting to connect to older versions may result in the client hanging when attempting to connect to the database.
Constructors of the RecordQueryUnionPlan and RecordQueryIntersectionPlan have been marked as deprecated in favor of static initializers. This will allow for more flexibility as work on the new planner develops.
The non-static RecordCursor::flatMapPipelined() method has been deprecated because it is easy to mis-use (by mistaken analogy to the mapPipelined() method) and cannot be used with continuations. See Issue #665 for further explanation.
The FDBDatabase::getReadVersion() method has been replaced with the FDBRecordContext::getReadVersionAsync() and FDBRecordContext::getReadVersion() methods. Though not strictly necessary, users should also replace any uses of Transaction::getReadVersion() and Transaction::setReadVersion() (on the Transaction interface provided by the FoundationDB Java bindings) with FDBRecordContext::getReadVersionAsync() and FDBRecordContext::setReadVersion() on any transactions created by the Record Layer. This allows the Record Layer to track the version in Java memory which both can then be used to skip a JNI hop if the read version is needed, but it also allows the Record Layer to more accurately track when read versions are retrieved from the database if the user has enabled read version tracking.
- Performance Reduce lock contention on StoreTimer updates (Issue #953)
- Performance Meta Conflict Range Additions hotspot during large operations (Issue #960)
- Feature Errors from
SplitHelpernow have more helpful log info (Issue #932)
- Bug fix TupleKeyCountTree.addPrefixChild only ever adds one child (Issue #947)
- All changes from version 2.8.118.30
- Feature Support edge ordering in planner graphs (Issue #928)
- Bug fix Latency injection counters published when not in use (Issue #922)
- Feature Add key/value size enforcement assertions to FDBRecordContext (Issue #925)
- Bug fix A race condition was fixed in the
RankIndexMaintainerthat could result in corrupt index data if a record had multiple entries inserted into the index with the same grouping key (Issue #482)
- Bug fix Opening a store with
ERROR_IF_NO_INFO_AND_HAS_RECORDS_OR_INDEXESnow allows opening a record store if the data is in theINDEX_BUILD_SPACE(Issue #941)
- Bug fix
FDBDatabaseRunners will now refetch a fresh read version on retry to avoid hitting the same conflict multiple times (Issue #905)
- Releases for the
fdb-record-layer-spatiallibrary have been re-enabled (Issue #884) - All changes from 2.8.110.29
- Performance The
LocatableResolver::resolvemethods have overloads that allow ancillary transactions started for key space path resolution to avoid starting another read version request (Issue #864) - Performance The metrics for text indexes now contain more accurate numbers for reads and writes (Issue #876)
- Performance Index states are now preloaded in fewer round trips (Issue #881)
- Feature Replace some aggregate metrics with accurate counters (Issue #866)
- Releases for the
fdb-record-layer-spatiallibrary have been temporarily disabled (Issue #884)
- All changes from 2.8.104.28
- All changes from 2.8.104.26
- All changes from version 2.8.104.24
- Feature Enhance KeySpaceCountTree to allow for application-specific resolution (Issue #852)
- Feature Rank index adds option for how to handle ties (Issue #806)
- Feature StoreTimer aggregates can return their componeent events (Issue #857)
- Breaking change StoreTimer aggregates now require getComponentEvents() method (Issue #857)
- Feature The transaction timeout option can now be set on
FDBRecordContexts (Issue #848) - Breaking change Additional methods,
setTransactionTimeoutMillisandgetTransactionTimeoutMillis, were added to theFDBDatabaseRunnerinterface that implementors will need to react to (Issue #848) - Breaking change In resolving Issue #848, the semantics of setting a transaction ID were slightly modified so that an explicit
nullID now also checks the MDC (PR #849)
- Performance When an
INquery is transformed to anOR, other query predicates are now normalized to allow for better plans (Issue #888)
- Feature Expose random hash as an index option (Issue #869)
- Feature Rank index adds option for how to handle ties (Issue #806)
- All changes from versions 2.8.102.23 and 2.8.102.24
- Performance Aggregate metrics added for the number of reads, writes, and deletes (Issue #839)
- Feature New exception type
ScanNonReadableIndexExceptionfor scanning non-readable indexes (Issue #850)
- Feature Add an index queryability function to
RecordQuery(Issue #841) - Feature Expose bytes/records scanned through
ExecuteState(Issue #835) - Breaking change
ByteScanLimiterandRecordScanLimiterare now interfaces. Instances with various concrete behavior are constructed through factory classes. (PR #836)
- Performance If an IN predicate cannot be planned using a nested loop join, we now attempt to plan it as an equivalent OR of equality predicates. (Issue #860)
- Feature The
RecordQueryPlannernow has a dedicated object for specifying configuration options. (Issue #861)
- Performance Aggregate metrics added for the number of reads, writes, and deletes (Issue #839)
- Bug fix NoSuchDirectoryException should include full directory path (Issue #797)
- Bug fix StoreTimer.getDifference() should not return unchanged metrics (Issue #832)
- Performance Allow setting hash function used by
RankedSet(Issue #828)
- Bug fix If the
FunctionKeyExpression.Registryfails to initialize, then the error is now propagated to the user when a function is requested (Issue #826) - Bug fix COMMIT_READ_ONLY not part of COMMITS aggregate (Issue #830)
- Bug fix Remove reverse directory cache scan fallback (Issue #821)
- Feature Fix test to show when function index queries work (Issue #810)
- Feature Support for aggregate performance metrics (Issue #801)
- Feature Support tracking index build progress by
IndexBuildState(Issue #817) - Feature System keyspace client log parsing (Issue #816)
- Bug fix Revert #794 until #489 is worked out. (Fixes potentially index entry lose if two metadata updates happen in different record stores over the same data in the same transaction.) (Issue #808)
- Performance Improved loggings for
OnlineIndexerandFDBDatabaseRunner(Issue #810)
- Feature Emit
TIME_WINDOW_LEADERBOARD_OVERLAPPING_CHANGEDwhenever rebuild needed due to overlapping, not just when rebuilding immediately (Issue #799) - Feature Clear existing index data when
rebuildIndexWithNoRecord(Issue #794)
- Feature Support for custom Executor per FDBRecordContext (Issue #787)
- Feature Better detailed metrics in
RankedSet.add(Issue #792) - Feature Upon a failed record deserialization, the file descriptor is only logged if the logger is at
TRACEinstead ofDEBUG(Issue #789)
- Bug fix OnlineIndexer does not always end the synchronized session after use (Issue #780)
- Bug fix Two inequality predicates on a single field index will now be planned as a single range scan on the index (Issue #765)
- Bug fix Primary keys where one key contained a string or byte array that was a prefix of another could lead to deserialization errors when loading (Issue #782)
- Feature Support forcefully releasing synchronized session locks and stopping ongoing online index builds (Issue #748)
- Bug fix Indexes not marked as write-only before being built online (Issue #773)
- Bug fix
TimeWindowLeaderboardIndexMaintainer.negateScoreForHighScoreFirsthandlesLong.MIN_VALUE(Issue #776) - Breaking change A leaderboard index with high-score-first containing the lowest possible score of
Long.MIN_VALUEwill have indexed that entry as the highest possible score and should be forcibly reindexed. (Issue #776)
- Feature Support building indexes online in a safer and simpler way (Issue #727)
- Feature Read and write conflict ranges can now be added on individual records (Issue #767)
- Breaking change Building indexes with
OnlineIndexerhas now been changed to be safer and simpler. One can setindexStatePreconditiontoERROR_IF_DISABLED_CONTINUE_IF_WRITE_ONLYand setuseSynchronizedSessiontofalseto make the indexer follow the same behavior as before if necessary. (Issue #727)
- Feature SubspaceProviderByKeySpacePath can return the KeySpacePath(Issue #759)
- Breaking bug fix Protobuf serialization of
LiteralKeyExpressionis now correct forLongs andbyte[]s. (Issue #756) - Feature Fetch the config parameters of online indexer in the beginning of each transaction (Issue #731)
- Feature
FDBRecordVersionnow has agetDBVersionmethod for extracting the database commit version (Issue #718)
- Performance The InExtractor was making duplicates that can easily be avoided (Issue #735)
- Bug fix
Field.oneOfThem(OneOfThemEmptyMode mode)for when the repeated field can otherwise distinguish null or null behavior is not desired (Issue #725) - Feature
FDBRecordContextnow exposes theMAX_TR_ID_SIZEconstant containing the maximum admissable length of a transaction ID (Issue #721)
- Bug fix Transaction IDs that would exceed the maximum FDB limit are now truncated or ignored instead of throwing an error (Issue #705)
- Feature Support validating missing index entries in index entry validation (Issue #712)
- Feature The store header now supports allowing the user to set custom fields with application-specific information (Issue #704)
- Performance The
OnlineIndexernow instruments its read versions so that performance issues that may be linked to high read version latencies can now be better diagnosed (Issue #696) - Feature The transaction priority can be set through the
FDBTransactionPriorityenum, and theOnlineIndexerexposes that as an option on index builds (Issue #697) - Breaking change The
FDBDatabase::getReadVersionmethod has been deprecated and replaced withFDBRecordContext::getReadVersionAsync(Issue #698)
- Feature Support synchronized sessions, where only one of them is allowed to work at a time (Issue #637)
- Feature The new
MAX_EVER_VERSIONindex type tracks the maximum ever value for key expressions including a record's version (Issue #670) - Feature A new store existence check level only warns if there is no store info header but the store contains no records (Issue #707)
- Breaking change
FDBDatabaseRunneris changed to an interface, which can be contructed byFDBDatabaseRunnerImpl(Issue #637)
- Feature Add the capability to take a snapshot of a timer and to subtract said snapshot from the timer. (Issue #692)
- Feature Store state information can now be cached using the meta-data key added in FoundationDB 6.1 (Issue #537)
- Feature Trace log format can be set in the
FDBDatabaseFactory(Issue #584) - Breaking change The version of the
commons-lang3dependency has been upgraded to version 3.9 (Issue #606) - Breaking change The stabilities of
FDBDatabaseRunnerconstructors have been downgraded toINTERNAL(Issue #681) - Breaking change The slf4j dependency has been added to
fdb-extensions(Issue #680) - Breaking change The Record Layer now requires a minimum FoundationDB version of 6.1 (Issue #551)
The Guava version has been updated to version 27. Users of the shaded variants of the Record Layer dependencies should not be affected by this change. Users of the unshaded variant using an older version of Guava as part of their own project may find that their own Guava dependency is updated automatically by their dependency manager.
- Bug fix Fields no longer compare their
NullStandInvalue in their equality check (Issue #682)
- Bug fix Fix sorting with null-is-unique indexes (Issue #655)
- Bug fix
NullStandinshould be set inFieldKeyExpression::toProto(Issue #626) - Deprecated The non-static
RecordCursor::flatMapPipelined()method has been deprecated (Issue #665) - All changes from versions 2.7.73.21 and 2.7.73.22
- Bug fix Work around SpotBugs bug (Issue #659)
- Feature Add experimental join index support (Issue #159)
- Bug fix Revert fix for #646 (in 2.7.75.0) until a complete resolution is worked out (PR #654)
- All changes from version 2.7.73.20
- Bug fix Fix key comparison in
checkPossiblyRebuildRecordCounts(Issue #646)
- Bug fix Wrap the
RecordStoreStatein anAtomicReference(Issue #563) - Performance Blocking on a future that should already be complete can now be detected and logged (Issue #641)
- Performance The
performNoOpmethod onFDBDatabaseallows for instrumenting the delay from queuing work onto the network thread (Issue #650)
- Feature Include timeout counters in
StoreTimer::getKeysAndValues(Issue #672) - Feature Add
TRACE-level log details for inspecting theKeyValueUnsplitter(Issue #673)
- Bug fix The
SizeStatisticsCursorContinuationmust not access the continuation of its underlying cursor after it is exhausted (Issue #656)
- Bug fix Revert #627 until a complete version of #647 is worked out. (Issue #646)
- Bug fix Union fields for record types are now assigned based on field number rather than declaration order (Issue #620)
- Feature The
MetaDataEvolutionValidatornow permits record types to be renamed as long as all occurences of the record type in the meta-data definition are also renamed (Issue #508)
- Bug fix
NullStandinshould be set inFieldKeyExpression::toProto(Issue #626) - Feature Added support for named, asynchronous post-commit hooks (Issue #622)
- Bug fix
OnlineIndexer.buildEndpointsshould not decrease the scan limits on conflicts (Issue #511) - Bug fix
KeyValueCursor.Builder.buildnow requiressubspace,contextandscanPropertiesare specifed, throws RecordCoreException if not (Issue #418) - Feature Added a counter counting indexes that need to be rebuilt and changed related logging to
DEBUG(Issue #604)
- Breaking change The deprecated
TimeLimitedCursorandRecordCursor.limitTimeTo()methods have been removed (Issue #582) - Breaking change The Guava dependency has been upgraded to version 27.0.1 (Issue #216)
- Deprecated The
OnlineIndexer::asyncToSyncmethod has been deprecated (Issue #473)
The formerly experimental API for advancing a RecordCursor while ensuring correct continuation handling is now preferred. Specifically, the RecordCursor.onNext() method is now stable. Meanwhile, the AsyncIterator-style methods (such as onHasNext(), hasNext(), and next()) are deprecated. Clients should transition away from these methods before the 2.7 release.
To ease the transition, a new AsyncIterator called RecordCursorIterator provides these method and can built form a RecordCursor using RecordCursor.asIterator().
The IndexEntry class now contains a reference to its source index. This meant breaking the constructor on that class. This class should probably only have been instantiated from within the Record Layer, so its old constructors have been removed, and the new constructors are now marked as INTERNAL.
The API stability annotations have been moved to their own package. This allows for references to the stability levels within Javadoc pages to link to the annotation documentation. Any external users who had also used these annotations will be required to modify their imports to match the new package. Because these annotations have CLASS retention level, this change also requires that any client who depends on the Record Layer update any transitive dependency on fdb-extensions to 2.6 to avoid compile-time warnings about missing annotation classes.
The IndexMaintainer class now has a new public method validateEntries. Subclasses inheriting it should implement the new method. It does not break classes extending StandardIndexMaintainer which has a default implementation that does nothing.
The SubspaceProvider interface is changed to no longer require an FDBRecordContext at construction. Instead, methods that resolve subspaces are directly provided with an FDBRecordContext. This provides implementations with the flexibility to resolve logical subspace representations against different FoundationDB backends.
This version of the Record Layer requires a FoundationDB server version of at least version 6.0. Attempting to connect to older versions may result in the client hanging when attempting to connect to the database.
The getTimeToLoad and getTimeToDeserialize methods on FDBStoredRecord have been removed. These were needed for a short-term experiment but stuck around longer than intended.
Methods for retrieving a record from a record store based on an index entry generally took both an index and an index entry. As index entries now store a reference to their associatedindex, these methods have been deprecated in favor of methods that only take an index entry. The earlier methods may be removed in a future release. The same is true for a constructor on FDBIndexedRecord which no longer needs to take both an index and an index entry.
While not deprecated, the MetaDataCache interface's stability has been lessened from MAINTAINED to EXPERIMENTAL. That interface may undergo further changes as work progresses on Issue #280. Clients are discouraged from using that interface until work evolving that API has progressed.
The static loadRecordStoreStateAsync methods on FDBRecordStore were deprecated in 2.6.61.0 and removed in 2.6.66.0. Users are encouraged to use getRecordStoreState instead.
The RecordStoreState constructors have been deprecated or marked INTERNAL as part of the work adding store state caching (Issue #410). They are likely to be further updated as more meta-data is added to the information stored on indexes within each record store (Issue #506).
- Bug fix Deleting all records no longer resets all index states to
READABLE(Issue #399) - Bug fix Rebuilding all indexes with
FDBRecordStore::rebuildAllIndexesnow updates the local cache of index states (Issue #597) - Feature Include index information and UUID in OnlineIndexer logs (Issue #510)
- Feature Ability to repair records missing the unsplit record suffix (Issue #588)
- Feature Provide a way for
FDBDatabaseto use a non-standard locality API (Issue #504)
- Bug fix OnlineIndexer re-increasing limit log changed to info (Issue #570)
- Bug fix Updating the store header now updates the
last_update_timefield with the current time (Issue #593) - Feature Include index information and UUID in OnlineIndexer logs (Issue #510)
- Bug fix
FilterCursors now schedule all asychronous work using the cursor's executor (Issue #573) - Bug fix
FDBRecordStore.checkRebuildIndexesnow uses the count of all records to determine overall emptiness (Issue #577)
- Bug fix A new store with
metaDataVersionof0would not set that field, confusing a later load (Issue #565) - Breaking change Remove the
loadRecordStoreStateAsyncstatic methods (Issue #559)
- Feature The
FDBRecordStoreStateCacheallows for users to initialize a record store from a cached state. (Issue #410)
- Breaking change The experimental
TimeWindowLeaderboardScoreTrimoperation now takesTupleinstead ofIndexEntry(Issue #554)
- Feature Add
FDBRecordStore.getAllIndexStates(Issue #552)
- Bug fix
preloadRecordAsyncgets NPE if key doesn't exist (Issue #541) - Performance
OnlineIndexerinheritsWeakReadSemantics(Issue #519) - Performance Values are cached by the
AsyncLoadingCacheinstead of futures to avoid futures sharing errors or timeouts (Issue #538) - Performance
ProbableIntersectionCursors andUnorderedUnionCursors now throw an error if they take longer than 15 seconds to find a next state as a temporary debugging measure (Issue #546) - Feature The bytes and record scanned query limits can now be retrieved through the
ExecutePropertiesobject (Issue #544) - Breaking change Deprecate the static
loadRecordStoreStateAsyncmethods fromFDBRecordStore(Issue #534)
- All changes from version 2.5.54.18
- Performance Traced transactions restore MDC context (Issue #529)
- Feature Automatically add a default union to the record meta-data if missing (Issue #204)
- Feature Ability to inject latency into transactions (Issue #521)
- Feature The new
ConcatCursorallows clients to chain the results of two cursors together (Issue #13)
- All changes from version 2.5.54.17
- Bug fix getVersionstamp() future can complete a tiny bit after the commit() future (Issue #476)
- Feature A new cursor type
AutoContinuingCursorwhich can iterate over a cursor across transactions (Issue #397) - Feature A new
AsyncIteratorthat wrapsRecordCursorprovides an easy migration path away from the old methods onRecordCursor(Issue #368) - Feature The
getNextmethod onRecordCursors now provides a blocking version ofonNext(Issue #408) - Breaking change
IndexEntryobjects now contain a reference to their associated index (Issue #403) - Breaking change Remove deprecated
KeySpacePathAPIs (Issue #169) - Breaking change
IndexMaintainers should implementvalidateEntriesto validate orphan index entries (Issue #383) - Breaking change The API stability annotations have been moved into
com.apple.foundationdb.annotation(Issue #406) - Breaking change
SubspaceProviderreceives anFDBRecordContextwhen a subspace is resolved instead of when constructed. (Issue #338) - Breaking change The
MetaDataCacheinterface is nowEXPERIMENTAL(Issue #447) - Breaking change The
AsyncIteratormethods onRecordCursorare now deprecated (Issue #368) - Breaking change The Record Layer now requires a minimum FoundationDB version of 6.0 (Issue #313)
- Breaking change Remove per-record time-to-load (Issue #461)
In order to simplify typed record stores, the FDBRecordStoreBase class was turned into an interface and all I/O was placed in the FDBRecordStore class. This makes the FDBTypedRecordStore serve as only a type-safe wrapper around the FDBRecordStore class. As part of this work, the IndexMaintainer interface and its implementations lost their type parameters and now only interact with Message types, and the OnlineIndexerBase class was removed. Additionally, the FDBEvaluationContext class was removed in favor of using EvaluationContext (without a type parameter) directly. That class also no longer carries around an FDBRecordStore reference, so query plans now require an explicit record store in addition to an evaluation context. Finally, the evaluate family of methods on key expressions no longer take evaluation contexts. Users should switch any uses of the OnlineIndexerBase to a generic OnlineIndexer and will need to update any explicit key expression evluation calls. Users should also switch from calling RecordQueryPlan::execute to calling FDBRecordStore::executeQuery if possible as that second API is more stable (and was not changed as part of the recent work).
The asyncToSync method of the OnlineIndexer has been marked as INTERNAL. Users should transition to using one of the asyncToSync methods defined on either FDBDatabase, FDBRecordContext, or FDBDatabaseRunner. This method may be removed from our public API in a later release (see Issue # 473).
- Bug fix
ProbableIntersectionCursors andUnorderedUnionCursors should no longer get stuck in a loop when one child completes exceptionally and another with a limit (Issue #526)
- Bug fix Added an optional limit on the number of suppressed exceptions (Issue #512)
- Bug fix The preload cache in
FDBRecordStorenow invalidates entries corresponding to updated records. (Issue #494) - Feature Include index subspace key in "Attempted to make unbuilt index readable" message (Issue #470)
- Feature
FDBMetaDataStoreclass now has convenience methods for adding and deprecating record types and fields (Issue #376) - All changes from version 2.5.49.16
- Bug fix KeyValueLogMessage now converts
"to'in values (Issue #472) - Feature Add
RecordMetaDataBuilder.addFormerIndex(Issue #485) - Deprecated The
asyncToSyncmethod of theOnlineIndexerhas been markedINTERNAL(Issue #474)
- Bug fix
OnlineIndexerfails togetPrimaryKeyBoundarieswhen there is no boundary (Issue #460) - Bug fix The
markReadableIfBuiltmethod ofOnlineIndexernow waits for the index to be marked readable before returning (Issue #468)
- Feature Add methods in
OnlineIndexerto support building an index in parallel (Issue #453)
- Feature OnlineIndexer logs number of records scanned (Issue #479)
- Feature OnlineIndexer includes range being built in retry logs (Issue #480)
- Feature Add an
AsyncIteratorCursorbetweenIteratorCursorandKeyValueCursor(Issue #449) - Feature OnlineIndexer can increase limit after successful range builds (Issue #444)
- Bug fix Use ContextRestoringExecutor for FDBDatabaseRunner (Issue #436)
- Feature Add a method to get primary key boundaries in
FDBRecordStore(Issue #439) - All changes from version 2.5.44.14
- All changes from version 2.5.44.15
- Feature
RankedSetand theRANKandTIME_WINDOW_LEADERBOARDindex types that use it now support operations for getting the rank of a score regardless of whether the set contains that value (Issue #425), (Issue #426)
- Bug fix Record type and index subspace keys with identical serializations are now validated for collisions (Issue #394)
- Bug fix Byte scan limit now reliably throws exceptions when
ExecuteProperties.failOnScanLimitReachedistrue(Issue #422) - Feature By default,
FDBMetaDataStores are now initialized with an extension registry with all extensions fromrecord_metadata_options.proto(Issue #352) - Feature
FDBMetaDataStoreclass now has convenience methods foraddIndex,dropIndexandupdateRecords(Issue #281) - Feature Index subspace keys can now be assigned based on a counter (Issue #11)
- Bug fix The
AsyncLoadingCachecould cache exceptional futures in rare scenarios (Issue #395) - Feature A
MetaDataEvolutionValidatornow can be used to ensure that meta-data changes are compatible with the schema evolution guidelines (Issue #85)
- Bug fix The
UnorderedUnionCursorshould now propagate errors from its children instead of sometimes swallowing the exception (Issue #437)
- Feature Optionally log successful progress in OnlineIndexer (Issue #429)
- Feature The
BooleanNormalizernow gives up early if given an expression that is too complex (Issue #356)
- Bug fix ChainedCursor does not obey byte/record/time scan limits (Issue #358)
- Bug fix
FDBRecordStore.IndexUniquenessCheck.checkneeds to be async (Issue #357) - Bug fix The
TimeLimitedCursorcould throw an error when setting its result'sNoNextReasonif the inner cursor was exhausted exactly as the time limit was reached (Issue #380) - Feature Collating indexes (Issue #249)
- Feature Build record meta-data using user's local / evolved meta-data (Issue #3)
- Feature Validation checks now reject a
RecordMetaDataobject that defines no record types (Issue #354) - Feature A new cursor type allows for intersecting cursors with incompatible orderings (Issue #336)
- Feature The text search query API now exposes
containsAllPrefixesandcontainsAnyPrefixpredicates (Issue #343)
- Bug fix Directory layer cache size was accidentally defaulted to zero (Issue #251)
- Performance Remove blocking calls from LocatableResolver constructors (Issue #261)
- Feature Tracing diagnostics to detect blocking calls in an asynchronous context (Issue #262)
- Feature New limit on the number of bytes scanned while executing a cursor (Issue #349)
- All changes from version 2.3.32.13
- Feature A KPI to track occurrences of index entries that do not point to valid records (Issue #310)
- Feature The shaded artifacts now include the
fdb-extensionslibrary and no longer include guava as a transitive dependency (Issue #329)
- Feature The
TextCursorclass now reports metrics on keys read (Issue #242) - Breaking change Typed record stores are now wrappers around generic record stores, and several classes no longer take a type parameter (Issue #165)
- All changes from version 2.4.35.12
The RecordCursor API has been reworked to make using continuations safer and less error prone as well as generally making safer use easier. Previously, a continuation could be retrieved from a RecordCursor by calling the getContinuation method. However, this method was only legal to call at certain times which made it painful and error-prone to use in asynchronous contexts. (Similar problems existed with the getNoNextReason method.) The new API is designed to make correct use more natural. Users are now encouraged to access elements through the new onNext method. This method returns the next element of the cursor if one exists or a reason why the cursor cannot return an element if it cannot. It also will always return a valid continuation that can be used to resume the cursor. As the RecordCursor interface is fairly fundamental to using the Record Layer, the old API has not yet been deprecated.
The GlobalDirectoryResolverFactory class was removed. This is part of a larger effort to migrate away from the ScopedDirectoryLayer class in favor of the ExtendedDirectoryLayer class when resolving DirectoryLayerDirectories as part of a record store's KeySpacePath. Users can still access the global directory layer by calling ExtendedDirectoryLayer.global().
The version of the Protocol Buffers dependency was updated to 2.5.0 and 3.6.1 for artifacts declaring proto2 and proto3 dependencies respectively. While the on-disk format of those versions is backwards-compatible with prior versions, the generated Java files may differ in incompatible ways. Users may therefore need to update their own Protocol Buffer dependencies to the new versions and regenerate any generated Java files created using prior versions.
The KeySpacePath class previously required an FDBRecordContext at object creation time. This was used to then resolve certain elements of the path and replace them with interned integers stored by the directory layer within the FoundationDB cluster. However, this meant that a KeySpacePath could not outlive a single transaction and therefore could not be reused when connecting to the same record store multiple times. To address this, KeySpacePaths should now be constructed without a transaction context, and new methods that do take an FDBRecordContext should be used to resolve a path into a subspace or tuple. The older constructor and methods have been deprecated.
Several constructors of the RecordMetaDataBuilder class have been deprecated. Users should transition to calling RecordMetaData.newBuilder() instead and calling the appropriate setter methods to specify the record descriptor or MetaData proto message. This new pattern allows for additional control as to which transitive dependencies of the record file descriptor are serialized into MetaData proto message as well as allowing for additional optional parameters to be added in the future in a backwards-compatible way.
- Feature A new RecordCursor API makes continuations less error-prone and safe use easier (Issue #109)
- All changes from version 2.4.35.11
- Bug fix The visibility of the
PathValueclass has been increased topublicto match the visibility of theresolveAsyncmethod ofKeySpacePath(Issue #266)
- Bug fix The FlatMapPipelinedCursor could return a null continuation when the inner cursor was done but was not ready immediately (Issue #255)
- Feature The
RecordMetaDataBuildernow performs additional validation by default before returning aRecordMetaData(Issue #201) - Feature Including a repeated field within a meta-data's union descriptor now results in an explicit error (Issue #237)
- All changes from version 2.3.32.10
- Feature The experimental
SizeStatisticsCollectorclass was added for collecting statistics about record stores and indexes (Issue #149) - Feature Key expressions were added for accessing elements of Protocol Buffer map fields (Issue #89)
- Feature
MetaDataproto messages can now specify transitive dependencies and therefore support self-contained meta-data definitions (Issue #114) - Feature Appropriate covering indexes will now be chosen when specified with concatenated fields (Issue #212)
- Feature The
KeySpacePathclass no longer includes a transaction object and old code paths are deprecated (Issue #151) - All changes from versions 2.3.32.8 and 2.3.32.9
- Feature
KeySpacePaths now support listing selected ranges (Issue #199)
- Breaking change Protobuf dependencies updated to versions 2.5.0 and 3.6.1 (PR #251)
- Breaking change Support removed for migrating from the global directory layer (Issue #202)
The behavior of creating a new record store and opening it with the createOrOpen method has been made safer in this version. Previously, a record store that had not been properly initialized may be missing header information describing, for example, what meta-data and format version was used when the store was last opened. This meant that during format version upgrades, a store might be upgraded to a newer format version without going through the proper upgrade procedure. By default, createOrOpen now will throw an error if it finds a record store with data but no header, which is indicative of this kind of error. This should protect users from then writing corrupt data to the store and more easily detect these kinds of inconsistencies.
The OnlineIndexBuilder has been renamed to the OnlineIndexer. This was to accommodate a new builder class for the OnlineIndexer as its constructors had become somewhat cumbersome. Users should be able to change the name of that class where referenced and to react to this change.
As the Index class now tracks the created and last modified version separately, the getVersion method has been deprecated. Users should switch to using getAddedVersion and getLastModifiedVersion as appropriate.
- Bug fix The
UnorderedUnionCursornow continues returning results until all children have hit a limit or are exhausted (Issue #332)
- Bug fix The FlatMapPipelinedCursor could return a null continuation when the inner cursor was done but was not ready immediately (Issue #255)
- Feature Accessing a non-empty record store without its header now throws an error by default (Issue #257)
- Bug fix The
FlatMapPipelinedCursorcould return a null continuation when the outer cursor returned no results (Issue #240) - Bug fix The
IntersectionCursorcould return a null continuation if a child cursor hit an in-band limit at the wrong time (Issue #241)
- All changes from version 2.2.29.7
- Feature Indexes and FormerIndexes now remember additional meta-data about when an index was created or last modified and what its name was (Issue #103)
- Bug fix The
MetaDataValidatornow validates the primary key expression of each record type (Issue #188) - Feature The OnlineIndexer class now utilizes a builder pattern to make setting up index rebuilds more straightforward (Issue #147)
- Feature The
FDBRecordStoreclass now has a method for checking whether with a given primary key exists in the record store (Issue #196) - Feature The FDBReverseDirectoryCache now logs its metrics (Issue #12)
The FDBRecordStore class now has additional methods for saving records that can check pre-conditions about whether a record with the same primary key as the new record does or does not already exist in the record store before saving. For example, the insertRecord method will throw an error if there is already a record present while updateRecord will throw an error if there is not already a record present (or if the type changes). Users are encouraged to switch to these methods to avoid accidentally deleting data if they can reason about whether a record does or does not exist prior to saving it. There are asynchronous variants of all of the new methods, and the existing saveRecord method remains available without any changes.
A new query plan, the RecordQueryUnorderedUnionPlan, allows the planner to produce union plans with “incompatibly ordered” subplans. In particular, the planner was previously only able of producing a union plan if all of its cursors were guaranteed to return records in the same order. For example, if there were one index on some field a of some record type and another on field b of that type, then the query planner would produce a union plan if the query were of the form:
Query.or(Query.field("a").equalsValue(value1), Query.field("b").equalsValue(value2))(In particular, it would plan two index scans over the indexes on a and b and then take their union. Each index scan returned results ordered by the primary key of the records, so the subplans were “compatibly ordered”.) However, it could not handle a query like this with a union:
Query.or(Query.field("a").lessThan(value1), Query.field("b").greaterThan(value2))This would instead result in a plan that scanned the full record store. (In this case, the two candidate index scans would each return records in a different order—one index scan by the value of the field a and the other index scan by the value of the field b.) The new query plan can handle subplans that return results in any order and essentially returns results as soon as any subplan returns a result. The trade-off is that unlike a RecordQueryUnionPlan, the RecordQueryUnorderedUnionPlan does not remove duplicates. This means that: (1) if the query is run with the setRemoveDuplicates flag set to true (as it is by default) then there will almost certainly be a RecordQueryUnorderedPrimaryKeyDistinctPlan as part of the query plan (which is more memory intensive) and (2) duplicates are never removed across continuation boundaries.
Index options have been refactored into their own class (appropriately titled the IndexOptions class). Option names and their semantics were preserved, but users who referenced the options via the constants in the Index class should transition their code to reference them through the IndexOptions class.
- Bug fix The
FlatMapPipelinedCursorcould return a null continuation when the inner cursor completed for an in-band reason even when not exhausted (Issue #222)
- Bug fix Scans by record type key were sometimes preferred by the planner in inappropriate circumstances (Issue #191)
- Feature A new union cursor and plan allows for incompatibly ordered plans to be executed without a full table scan (Issue #148)
- Feature The logging level for text indexing was decreased from INFO to DEBUG (Issue #183)
- Peformance Get range queries for unsplitting records in single-record requests are now done at the
WANT_ALLstreaming mode (Issue #181) - Feature Variants of the saveRecord method were added to the FDBRecordStore class to check whether records exist before saving a new record (Issue #115)
- Feature A streaming mode can now be given to the
ScanPropertiesobject so that users can optimize certain queries (Issue #173) - Feature Additional methods on the RecordMetaData class allow the user to supply strings for record type names (Issue #157)
- Feature Constants defined for index options have been moved from the Index to IndexOptions class (Issue #134)
- Feature The Record Layer can now be built and run using Java 11 (Issue #142)
- Feature A text index can now be configured not to write position lists to save space (Issue #144)
- Feature The number of levels used within a
RankedSet's skip list can now be configured (Issue #95)
A new key expression type on the “record type” of a record allows the user to lay their data out in a way that is more akin to the way tables are laid out in a traditional relational database. In particular, if the user prefixes all primary keys with this expression type, then each record type will live in contiguous segments within the database. This can speed up index build times (as only the range containing the records of the corresponding time needs to be scanned) and improve query performance without secondary indexes. As part of this work, the user can specify a shortened “record type key” as part of the record type definition so that the cost is typically only two bytes per key.
The capability and reliability of text queries on more sophisticated indexes has been greatly improved. In particular, it allows for the user to specify queries on a map field that query on the value of portion of the map (when grouped by key) as well as choosing to produce a covering plan if the text index covers all of the required fields of a given query. Queries can also be more reliably combined with other predicates through an “and” or “or” clause.
- Performance The RankedSet data structure now adds fewer conflict ranges by being more selective about initializing each level (Issue #124)
- Performance The RankedSet data structure now adds fewer conflict ranges when inserting or removing elements from the structure (Issue #122)
- Feature The Record Layer now publishes a second set of artifacts which shade their guava and Protocol Buffer dependencies (Issue #73)
- Feature Queries can now ask whether a nested message field is null (Issue #136)
- Performance “Or” predicates can now be used with text queries without always reverting to a full scan of the record store (Issue #19)
- Bug fix Multi-type text indexes now add type predicates if the query only requires a subset of the types on which the index is defined (Issue #126)
- Bug fix Combining a “not” predicate with a text query would previously throw an error during planning (Issue #127)
- All changes from version 2.1.14.6
- Feature Add a strict compatibility mode for the query-planning
IndexScanPreferenceoption (Issue #119)
- Bug fix Certain predicates could result in an error to be thrown during planning when combined with text index scans (Issue #117)
- Feature Text indexes now support querying the values of maps (Issue #24)
- Bug fix The FDBDatabase class now correctly threads through the datacenter ID if specified (Issue #106)
- Feature The time window leaderboard index now emits additional KPIs (Issue #105)
- Feature Add a strict compatibility mode for the query-planning
IndexScanPreferenceoption (Issue #119)
- Bug fix The planner previously would chose to use a FanOut index even if this could result in missing or duplicated entries in certain circumstances (Issue #81)
- Feature Covering index plans have a more human-readable string representation (PR #77)
- Bug fix A
FanOutindex could have been chosen by the query planner if no other index existed instead of a record scan even though that could have resulted in missing results (Issue #75)
- Feature Text indexes can now be included as part of covering index plans if all required results in the query are covered by the index (Issue #25)
- Feature Records can now be preloaded and cached to acheive better pipelining in otherwise blocking contexts (PR #72)
- Feature A new record type key expression allows for structuring data in a record store more akin to how tables are stored in a traditional relational database (Issue #27)