Skip to content

Latest commit

 

History

History
364 lines (311 loc) · 18.3 KB

CHANGELOG.md

File metadata and controls

364 lines (311 loc) · 18.3 KB
Raw

redb - Changelog

2.3.0 - 2024-12-10

  • Add WriteTransaction::set_two_phase_commit()
  • Add WriteTransaction::set_quick_repair() which enables a faster repair mechanism at the cost of slower transaction commits
  • Durability::Paranoid is now deprecated. Use set_two_phase_commit(true) instead
  • Fix various bugs when repairing the database after an unclean shutdown. These could result in panics, leaked space in the database file, or database corruption

2.2.0 - 2024-10-27

  • Implement TableHandle for ReadOnlyTable
  • Fix bug in write cache, which caused pages to be evicted randomly. Pages are now evicted based on how recently they have been accessed

2.1.4 - 2024-10-10

  • Optimize first() and last() to be almost 2x faster
  • Improve in-memory cache algorithm to resolve edge cases where certain pages could become uncacheable under cache pressure
  • Fix bug in read cache where the read cache could become disabled. This was likely to occur in multithreaded workloads when the read cache was smaller than the database file. This bug lead to 5-10x performance degradations for some workloads

2.1.3 - 2024-09-14

  • Significant performance optimizations to compact()
  • Fix some additional cases where compact() did not fully compact the database
  • Fix a panic that could occur in commit() or abort() after an IO error. StorageError::PreviousIo is now returned
  • Fix a potential panic that could occur when repairing the database after a crash

2.1.2 - 2024-08-25

Major fixes:

  • Fix leak of database space that could occur when calling restore_savepoint()
  • Fix leak of database space when calling delete_multimap_table()
  • Fix database corruption which could occur when restoring a savepoint. This edge case is rare, and could only occur if the database was less than approximately 4TiB when the savepoint was created, and greater than 4TiB when the savepoint was restored
  • Fix edge case where a transient I/O error that occurred during transaction commit, but then did not reoccur when the Database was dropped, could cause database corruption

Important: If your application has called restore_savepoint(), delete_multimap_table(), or you suspect it may have experienced a transient I/O error during transaction commit. It is recommended that you run check_integrity() after upgrading to this version. This will both detect corruption and clean up any leaked space.

Other changes and fixes:

  • Optimize page freeing to reduce the size of the database file
  • Fix several cases where check_integrity() would return Ok(false) instead of Ok(true)
  • Fix some cases where compact() did not fully compact the database
  • Make the metadata overhead returned by WriteTransaction::stats() more accurate
  • Return StorageError::ValueTooLarge when a key-value pair exceeds a total of 3.75GiB. Previously, a panic would occur for key-value pairs that were approximately 4GiB.
  • Downgrade several info! log messages to debug!
  • Improve documentation

2.1.1 - 2024-06-09

  • Fix panic that occurred when calling compact() when a read transaction was in progress
  • Fix ReadTransaction::close() to return Ok when it succeeds
  • Performance optimizations

2.1.0 - 2024-04-20

  • Implement Key and Value for String
  • Allow users to implement ReadableTableMetadata, ReadableTable, and ReadableMultimapTable

2.0.0 - 2024-03-22

Major file format change

2.0.0 uses a new file format that optimizes len() to be constant time. This means that it is not backwards compatible with 1.x. To upgrade, consider using a pattern like that shown in the upgrade_v1_to_v2 test.

Other changes

  • check_integrity() now returns a DatabaseError instead of a StorageError
  • Table metadata methods have moved to a new ReadableTableMetadata trait
  • Rename RedbKey to Key
  • Rename RedbValue to Value
  • Remove lifetimes from read-only tables
  • Remove lifetime from WriteTransaction and ReadTransaction
  • Remove drain() and drain_filter() from Table. Use retain, retain_in, extract_if or extract_from_if instead
  • impl Clone for Range
  • Add support for [T;N] as a Value or Key as appropriate for the type T
  • Add len() and is_empty() to MultimapValue
  • Add retain() and retain_in() to Table
  • Add extract_if() and extract_from_if() to Table
  • Add range() returning a Range with the 'static lifetime to read-only tables
  • Add get() returning a range with the 'static lifetime to read-only tables
  • Add close() method to ReadTransaction

1.5.1 - 2024-03-16

  • Fix check_integrity() so that it returns Ok(true) when no repairs were preformed. Previously, it returned Ok(false)

1.5.0 - 2024-01-15

  • Export TableStats type
  • Export MutInPlaceValue which allows custom types to support insert_reserve()
  • Add untyped table API which allows metadata, such as table stats, to be retrieved for at table without knowing its type at compile time
  • Fix compilation on uncommon platforms (those other than Unix and Windows)

1.4.0 - 2023-11-21

  • Add Builder::set_repair_callback() which can be used to set a callback function that will be invoked if the database needs repair while opening it.
  • Add support for custom storage backends. This is done by implementing the StorageBackend trait and using the Builder::create_with_backend function. This allows the database to be stored in a location other than the filesystem
  • Implement RedbKey and RedbValue for char
  • Implement RedbKey and RedbValue for bool
  • Implement TableHandle for Table
  • Implement MultimapTableHandle for MultimapTable
  • Fix panic that could occur when inserting a large number of fixed width values into a table within a single transaction
  • Fix panic when calling delete_table() on a table that is already open
  • Improve performance for fixed width types
  • Support additional platforms

1.3.0 - 2023-10-22

  • Implement RedbKey for Option<T>
  • Implement RedbValue for Vec<T>
  • Implement Debug for tables
  • Add ReadableTable::first() and last() which retrieve the first and last key-value pairs, respectively`
  • Reduce lock contention for mixed read-write workloads
  • Documentation improvements

1.2.0 - 2023-09-24

  • Add Builder::create_file() which does the same thing as create() but takes a File instead of a path
  • Add stats() to tables which provides informational statistics on the table's storage
  • Fix WriteTransaction::stats() to correctly count the storage used by multi-map tables
  • Fix panics that could occur when operating on savepoints concurrently from multiple threads on the same WriteTransaction
  • Implement Send for WriteTransaction
  • Change MSRV to 1.66
  • Performance optimizations

1.1.0 - 2023-08-20

  • Fix panic when calling compact() on certain databases
  • Fix panic when calling compact() when an ephemeral Savepoint existed
  • Improve performance of compact()
  • Relax lifetime requirements on arguments to insert()

1.0.5 - 2023-07-16

  • Fix a rare panic when recovering a database file after a crash
  • Minor performance improvement to write heavy workloads

1.0.4 - 2023-07-01

  • Fix serious data corruption issue when calling drain() or drain_filter() on a Table that had uncommitted data

1.0.3 - 2023-06-30

  • Fix panic when re-opening databases of certain, small, sizes

1.0.2 - 2023-06-29

  • Fix panic when recovering some databases after a forceful shutdown
  • Fix panic when recovering databases with multimaps that have fixed width values after a forceful shutdown

1.0.1 - 2023-06-26

  • Fix panic that could occur after an IO error when reopening a database
  • Fix panic that could occur after an IO error when opening a table
  • Improve error message when opening a table twice to include a more meaningful line number
  • Performance improvements

1.0.0 - 2023-06-16

Announcement

redb has reached its first stable release! The file format is now gauranteed to be backward compatible, and the API is stable. I've run pretty extensive fuzz testing, but please report any bugs you encounter.

The following features are complete:

  • MVCC with a single WriteTransaction and multiple ReadTransactions
  • Zero-copy reads
  • ACID semantics, including non-durable transactions which only sacrifice Durability
  • Savepoints which allow the state of the database to be captured and restored later

Changes from 0.22.0:

  • Stabilize file format
  • Improve performance of restore_savepoint()

0.22.0 - 2023-06-12

  • Fix panic while repairing a database file after crash
  • Fix rare panic in restore_savepoint()

0.21.0 - 2023-06-09

  • Improve cache heuristic. This asymptotically improves performance on large databases. Benchmarks show 30% to 5x+
  • Fix rare crash that could occur under certain conditions when inserting values > 2GiB
  • Fix crash when growing database beyond 4TiB
  • Fix panic when repairing a database containing a multimap table with fixed width values
  • Performance optimizations
  • File format simplifications

0.20.0 - 2023-05-30

  • Export TransactionError and CommitError. These were unintentionally private
  • Implement std::error::Error for all error enums

0.19.0 - 2023-05-29

  • Remove Clone bound from range argument type on drain() and drain_filter()
  • File format changes to improve future extensibility

0.18.0 - 2023-05-28

  • Improve errors to be more granular. Error has been split into multiple different enums, which can all be implicitly converted back to Error for convenience
  • Rename savepoint() to ephemeral_savepoint()
  • Add support for persistent savepoints. These persist across database restarts and must be explicitly released
  • Optimize restore_savepoint() to be ~30x faster
  • Add experimental support for WASI. This requires nightly
  • Implement RedbKey for ()
  • Fix some rare crash and data corruption bugs

0.17.0 - 2023-05-09

  • Enforce a limit of 3GiB on keys & values
  • Fix database corruption bug that could occur if a Durability::None commit was made, followed by a durable commit and the durable commit crashed or encountered an I/O error during commit()
  • Fix panic when re-openning a database file, when the process that last had it open had crashed
  • Fix several bugs where an I/O error during commit() could cause a panic instead of returning an Err
  • Change length argument to insert_reserve() to u32
  • Change Table::len() to return u64
  • Change width of most fields in DatabaseStats to u64
  • Remove K type parameter from AccessGuardMut
  • Add Database::compact() which compacts the database file
  • Performance optimizations

0.16.0 - 2023-04-28

  • Combine Builder::set_read_cache_size() and Builder::set_write_cache_size() into a single, Builder::set_cache_size() setting
  • Relax lifetime constraints on read methods on tables
  • Optimizations to Savepoint

0.15.0 - 2023-04-09

  • Add Database::check_integrity() to explicitly run repair process (it is still always run if needed on db open)
  • Change list_tables() to return a TableHandle
  • Change delete_table() to take a TableHandle
  • Make insert_reserve() API signature type safe
  • Change all iterators to return Result and propagate I/O errors
  • Replace WriteStrategy with Durability::Paranoid
  • Remove Builder::set_initial_size()
  • Enable db file shrinking on Windows
  • Performance optimizations

0.14.0 - 2023-03-26

  • Remove Builder::create_mmapped() and Builder::open_mmapped(). The mmap backend has been removed because it was infeasible to prove that it was sound. This makes the redb API entirely safe, and the remaining File based backed is within a factor of ~2x on all workloads that I've benchmarked
  • Make Table implement Send. It is now possible to insert into multiple Tables concurrently
  • Expose AccessGuardMut, Drain and DrainFilter in the public API
  • Rename RangeIter to Range
  • RenameMultimapRangeIter to MultimapRange
  • Rename MultimapValueIter to MultimapValue
  • Performance optimizations

0.13.0 - 2023-02-05

  • Fix a major data corruption issue that was introduced in version 0.12.0. It caused databases greater than ~4GB to become irrecoverably corrupted due to an integer overflow in PageNumber::address_range that was introduced by commit b2c44a824d1ba69f526a1a75c56ae8484bae7248
  • Add drain_filter() to Table
  • Make key and value type bounds more clear for tables

0.12.1 - 2023-01-22

  • Fix open() on platforms with OS page size != 4KiB
  • Relax lifetime requirements on argument to range() and drain()

0.12.0 - 2023-01-21

  • Add pop_first() and pop_last() to Table
  • Add drain() to Table
  • Add support for Option<T> as a value type
  • Add support for user defined key and value types. Users must implement RedbKey and/or RedbValue
  • Change get(), insert(), remove()...etc to take arguments of type impl Borrow<SelfType>
  • Return Error::UpgradeRequired when opening a file with an outdated file format
  • Improve support for 32bit platforms
  • Performance optimizations

0.11.0 - 2022-12-26

  • Remove [u8] and str type support. Use &[u8] and &str instead.
  • Change get(), range() and several other methods to return AccessGuard.
  • Rename AccessGuard::to_value() to value()
  • Add a non-mmap based backend which is now the default. This makes Database::create() and Database::open() safe, but has worse performance in some cases. The mmap backend is available via create_mmapped()/open_mmapped(). There is no difference in the file format, so applications can switch from one backend to the other.
  • Better handling of fsync failures

0.10.0 - 2022-11-23

  • Remove maximum database size argument from create(). Databases are now unbounded in size
  • Reduce address space usage on Windows
  • Remove set_dynamic_growth()
  • Add set_initial_size() to Builder
  • Optimize cleanup of deleted pages. This resolves a performance issue where openning a Database or performing a small transaction, could be slow if the last committed transaction deleted a large number of pages
  • Remove set_page_size(). 4kB pages are always used now
  • Add iter() method to Table and MultimapTable
  • Fix various lifetime issues with type that had a lifetime, such as &str and (&[u8], u64)

0.9.0 - 2022-11-05

  • Add support for dynamic file growth on Windows
  • Add support for tuple types as keys and values
  • Remove Builder::set_region_size
  • Save lifetime from Savepoint
  • Fix crash when using create() to open an existing database created with WriteStrategy::TwoPhase
  • Fix rare crash when writing a mix of small and very large values into the same table
  • Performance optimizations

0.8.0 - 2022-10-18

  • Performance improvements for database files that are too large to fit in RAM
  • Fix deadlock in concurrent calls to savepoint() and restore_savepoint()
  • Fix crash if restore_savepoint() failed
  • Move savepoint() and restore_savepoint() methods to WriteTransaction
  • Implement Iterator for the types returned from range() and remove_all()

0.7.0 - 2022-09-25

  • Add support for Windows
  • Add Database::set_write_strategy which allows the WriteStrategy of the database to be changed after creation
  • Make Database::begin_write block, instead of panic'ing, if there is another write already in progress
  • Add Database::savepoint and Database::restore_savepoint which can be used to snapshot and rollback the database
  • Rename DatabaseBuilder to Builder
  • Performance optimizations for large databases

0.6.1 - 2022-09-11

  • Fix crash when Database::open() was called on a database that had been created with WriteStrategy::TwoPhase
  • Change default region size on 32bit platforms to 4GiB

0.6.0 - 2022-09-10

  • Return Err instead of panic'ing when opening a database file with an incompatible file format version
  • Many optimizations to the file format, and progress toward stabilizing it
  • Fix race between read & write transactions, which could cause reads to return corrupted data
  • Better document the different WriteStrategys
  • Fix panic when recovering a database that was uncleanly shutdown, which had been created with WriteStrategy::Checksum (which is the default)
  • Fix panic when using insert_reserve() in certain cases

0.5.0 - 2022-08-06

  • Optimize MultimapTable storage format to use O(k * log(n_k) + v * log(n_v / n_k)) space instead of O(k * log(n_k + n_v) + v * log(n_k + n_v)) space, where k is the size of the stored keys, v is the size of the stored values, n_k is the number of stored keys, n_v is the number of stored values
  • Fix compilation errors for 32bit x86 targets
  • Add support for the unit type, (), as a value
  • Return an error when attempting to open the same database file for writing in multiple locations, concurrently
  • More robust handling of fsync failures
  • Change MultimapTable::range to return an iterator of key-value-collection pairs, instead of key-value pairs
  • Automatically abort WriteTransaction on drop

0.4.0 - 2022-07-26

  • Add single phase with checksum commit strategy. This is now the default and reduces commit latency by ~2x. For more details, see the design doc and blog post. The previous behavior is available via WriteStrategy::Throughput, and can have better performance when writing a large number of bytes per transaction.

0.3.1 - 2022-07-20

  • Fix a bug where re-opening a Table during a WriteTransaction lead to stale results being read

0.3.0 - 2022-07-19

  • Fix a serious data corruption issue that caused many write operations to corrupt the database
  • Make redb region size configurable
  • Implement garbage collection of empty regions
  • Fixes and optimizations to make the file format more efficient

0.2.0 - 2022-06-10

  • Add information log messages which can be enabled with the logging feature
  • Add support for [u8; N] type
  • Optimize storage of fixed width types. The value length is no longer stored, which reduces storage space by ~50% for u64, 2x for u32, and also improves performance.

0.1.2 - 2022-05-08

  • Change insert() to return an Option<V> with the previous value, instead of ()

0.1.1 - 2022-04-24

  • Improved documentation

0.1.0 - 2022-04-23

  • Initial beta release