Releases: facebook/rocksdb
Releases · facebook/rocksdb
RocksDB 6.29.5
6.29.5 (2022-03-29)
Bug Fixes
- Fixed a race condition for
alive_log_files_
in non-two-write-queues mode. The race is between the write_thread_ in WriteToWAL() and another thread executingFindObsoleteFiles()
. The race condition will be caught if__glibcxx_requires_nonempty
is enabled. - Fixed a race condition when mmaping a WritableFile on POSIX.
- Fixed a race condition when 2PC is disabled and WAL tracking in the MANIFEST is enabled. The race condition is between two background flush threads trying to install flush results, causing a WAL deletion not tracked in the MANIFEST. A future DB open may fail.
- Fixed a heap use-after-free race with DropColumnFamily.
- Fixed a bug that
rocksdb.read.block.compaction.micros
cannot track compaction stats (#9722).
RocksDB 7.0.3
7.0.3 (2022-03-25)
Bug Fixes
- Fixed a major performance bug in which Bloom filters generated by pre-7.0 releases are not read by early 7.0.x releases (and vice-versa) due to changes to FilterPolicy::Name() in #9590. This can severely impact read performance and read I/O on upgrade or downgrade with existing DB, but not data correctness.
- Fixed a bug that
Iterator::Refresh()
reads stale keys after DeleteRange() performed.
Public API changes
- Added pure virtual FilterPolicy::CompatibilityName(), which is needed for fixing major performance bug involving FilterPolicy naming in SST metadata without affecting Customizable aspect of FilterPolicy. For source code, this change only affects those with their own custom or wrapper FilterPolicy classes, but does break compiled library binary compatibility in a patch release.
- Since RocksDB 7, RocksJava now requires Java 8 (previously Java 7).
RocksDB 6.29.4
6.29.4 (2022-03-22)
Bug Fixes
- Fixed a bug caused by race among flush, incoming writes and taking snapshots. Queries to snapshots created with these race condition can return incorrect result, e.g. resurfacing deleted data.
- Fixed a bug that DisableManualCompaction may assert when disable an unscheduled manual compaction.
- Fixed a bug that
Iterator::Refresh()
reads stale keys after DeleteRange() performed. - Fixed a race condition when disable and re-enable manual compaction.
- Fix a race condition when cancel manual compaction with
DisableManualCompaction
. Also DB close can cancel the manual compaction thread. - Fixed a data race on
versions_
betweenDBImpl::ResumeImpl()
and threads waiting for recovery to complete (#9496) - Fixed a read-after-free bug in
DB::GetMergeOperands()
. - Fixed NUM_INDEX_AND_FILTER_BLOCKS_READ_PER_LEVEL, NUM_DATA_BLOCKS_READ_PER_LEVEL, and NUM_SST_READ_PER_LEVEL stats to be reported once per MultiGet batch per level.
RocksDB 7.0.2
Rocksdb Change Log
7.0.2 (2022-03-12)
- Fixed a bug that DisableManualCompaction may assert when disable an unscheduled manual compaction.
RocksDB 7.0.1
Rocksdb Change Log
7.0.1 (2022-03-02)
Bug Fixes
- Fix a race condition when cancel manual compaction with
DisableManualCompaction
. Also DB close can cancel the manual compaction thread. - Fixed a data race on
versions_
betweenDBImpl::ResumeImpl()
and threads waiting for recovery to complete (#9496) - Fixed a bug caused by race among flush, incoming writes and taking snapshots. Queries to snapshots created with these race condition can return incorrect result, e.g. resurfacing deleted data.
7.0.0 (2022-02-20)
Bug Fixes
- Fixed a major bug in which batched MultiGet could return old values for keys deleted by DeleteRange when memtable Bloom filter is enabled (memtable_prefix_bloom_size_ratio > 0). (The fix includes a substantial MultiGet performance improvement in the unusual case of both memtable_whole_key_filtering and prefix_extractor.)
- Fixed more cases of EventListener::OnTableFileCreated called with OK status, file_size==0, and no SST file kept. Now the status is Aborted.
- Fixed a read-after-free bug in
DB::GetMergeOperands()
. - Fix a data loss bug for 2PC write-committed transaction caused by concurrent transaction commit and memtable switch (#9571).
- Fixed NUM_INDEX_AND_FILTER_BLOCKS_READ_PER_LEVEL, NUM_DATA_BLOCKS_READ_PER_LEVEL, and NUM_SST_READ_PER_LEVEL stats to be reported once per MultiGet batch per level.
Performance Improvements
- Mitigated the overhead of building the file location hash table used by the online LSM tree consistency checks, which can improve performance for certain workloads (see #9351).
- Switched to using a sorted
std::vector
instead ofstd::map
for storing the metadata objects for blob files, which can improve performance for certain workloads, especially when the number of blob files is high. - DisableManualCompaction() doesn't have to wait scheduled manual compaction to be executed in thread-pool to cancel the job.
Public API changes
- Require C++17 compatible compiler (GCC >= 7, Clang >= 5, Visual Studio >= 2017) for compiling RocksDB and any code using RocksDB headers (previously required C++11). See #9388.
- Require Java 8 for compiling RocksJava (previously Java 7). See #9541
- Removed deprecated automatic finalization of RocksJava RocksObjects, the user must explicitly call
close()
on their RocksJava objects. See #9523. - Added
ReadOptions::rate_limiter_priority
. When set to something other thanEnv::IO_TOTAL
, the internal rate limiter (DBOptions::rate_limiter
) will be charged at the specified priority for file reads associated with the API to which theReadOptions
was provided. - Remove HDFS support from main repo.
- Remove librados support from main repo.
- Remove obsolete backupable_db.h and type alias
BackupableDBOptions
. Use backup_engine.h andBackupEngineOptions
. Similar renamings are in the C and Java APIs. - Removed obsolete utility_db.h and
UtilityDB::OpenTtlDB
. Use db_ttl.h andDBWithTTL::Open
. - Remove deprecated API DB::AddFile from main repo.
- Remove deprecated API ObjectLibrary::Register() and the (now obsolete) Regex public API. Use ObjectLibrary::AddFactory() with PatternEntry instead.
- Remove deprecated option DBOption::table_cache_remove_scan_count_limit.
- Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit.
- Remove deprecated API AdvancedColumnFamilyOptions::hard_rate_limit.
- Remove deprecated API DBOption::base_background_compactions.
- Remove deprecated API DBOptions::purge_redundant_kvs_while_flush.
- Remove deprecated overloads of API DB::CompactRange.
- Remove deprecated option DBOptions::skip_log_error_on_recovery.
- Remove ReadOptions::iter_start_seqnum which has been deprecated.
- Remove DBOptions::preserved_deletes and DB::SetPreserveDeletesSequenceNumber().
- Remove deprecated API AdvancedColumnFamilyOptions::rate_limit_delay_max_milliseconds.
- Removed timestamp from WriteOptions. Accordingly, added to DB APIs Put, Delete, SingleDelete, etc. accepting an additional argument 'timestamp'. Added Put, Delete, SingleDelete, etc to WriteBatch accepting an additional argument 'timestamp'. Removed WriteBatch::AssignTimestamps(vector) API. Renamed WriteBatch::AssignTimestamp() to WriteBatch::UpdateTimestamps() with clarified comments.
- Changed type of cache buffer passed to
Cache::CreateCallback
fromvoid*
toconst void*
. - Significant updates to FilterPolicy-related APIs and configuration:
- Remove public API support for deprecated, inefficient block-based filter (use_block_based_builder=true).
- Old code and configuration strings that would enable it now quietly enable full filters instead, though any built-in FilterPolicy can still read block-based filters. This includes changing the longstanding default behavior of the Java API.
- Remove deprecated FilterPolicy::CreateFilter() and FilterPolicy::KeyMayMatch()
- Remove
rocksdb_filterpolicy_create()
from C API, as the only C API support for custom filter policies is now obsolete. - If temporary memory usage in full filter creation is a problem, consider using partitioned filters, smaller SST files, or setting reserve_table_builder_memory=true.
- Remove support for "filter_policy=experimental_ribbon" configuration
string. Use something like "filter_policy=ribbonfilter:10" instead. - Allow configuration string like "filter_policy=bloomfilter:10" without
bool, to minimize acknowledgement of obsolete block-based filter. - Made FilterPolicy Customizable. Configuration of filter_policy is now accurately saved in OPTIONS file and can be loaded with LoadOptionsFromFile. (Loading an OPTIONS file generated by a previous version only enables reading and using existing filters, not generating new filters. Previously, no filter_policy would be configured from a saved OPTIONS file.)
- Change meaning of nullptr return from GetBuilderWithContext() from "use
block-based filter" to "generate no filter in this case."- Also, when user specifies bits_per_key < 0.5, we now round this down
to "no filter" because we expect a filter with >= 80% FP rate is
unlikely to be worth the CPU cost of accessing it (esp with
cache_index_and_filter_blocks=1 or partition_filters=1). - bits_per_key >= 0.5 and < 1.0 is still rounded up to 1.0 (for 62% FP
rate)
- Also, when user specifies bits_per_key < 0.5, we now round this down
- Remove class definitions for FilterBitsBuilder and FilterBitsReader from
public API, so these can evolve more easily as implementation details.
Custom FilterPolicy can still decide what kind of built-in filter to use
under what conditions. - Also removed deprecated functions
- FilterPolicy::GetFilterBitsBuilder()
- NewExperimentalRibbonFilterPolicy()
- Remove default implementations of
- FilterPolicy::GetBuilderWithContext()
- Remove public API support for deprecated, inefficient block-based filter (use_block_based_builder=true).
- Remove default implementation of Name() from FileSystemWrapper.
- Rename
SizeApproximationOptions.include_memtabtles
toSizeApproximationOptions.include_memtables
. - Remove deprecated option DBOptions::max_mem_compaction_level.
- Return Status::InvalidArgument from ObjectRegistry::NewObject if a factory exists but the object ould not be created (returns NotFound if the factory is missing).
- Remove deprecated overloads of API DB::GetApproximateSizes.
- Remove deprecated option DBOptions::new_table_reader_for_compaction_inputs.
- Add Transaction::SetReadTimestampForValidation() and Transaction::SetCommitTimestamp(). Default impl returns NotSupported().
- Add support for decimal patterns to ObjectLibrary::PatternEntry
- Remove deprecated remote compaction APIs
CompactionService::Start()
andCompactionService::WaitForComplete()
. Please useCompactionService::StartV2()
,CompactionService::WaitForCompleteV2()
instead, which provides the same information plus extra data like priority, db_id, etc. ColumnFamilyOptions::OldDefaults
andDBOptions::OldDefaults
are marked deprecated, as they are no longer maintained.- Add subcompaction callback APIs:
OnSubcompactionBegin()
andOnSubcompactionCompleted()
. - Add file Temperature information to
FileOperationInfo
in event listener API. - Change the type of SizeApproximationFlags from enum to enum class. Also update the signature of DB::GetApproximateSizes API from uint8_t to SizeApproximationFlags.
- Add Temperature hints information from RocksDB in API
NewSequentialFile()
. backup and checkpoint operations need to open the source files withNewSequentialFile()
, which will have the temperature hints. Other operations are not covered.
Behavior Changes
- Disallow the combination of DBOptions.use_direct_io_for_flush_and_compaction == true and DBOptions.writable_file_max_buffer_size == 0. This combination can cause WritableFileWriter::Append() to loop forever, and it does not make much sense in direct IO.
ReadOptions::total_order_seek
no longer affectsDB::Get()
. The original motivation for this interaction has been obsolete since RocksDB has been able to detect whether the current prefix extractor is compatible with that used to generate table files, probably RocksDB 5.14.0.
New Features
- Introduced an option
BlockBasedTableOptions::detect_filter_construct_corruption
for detecting corruption during Bloom Filter (format_version >= 5) and Ribbon Filter construction. - Improved the SstDumpTool to read the comparator from table properties and use it to read the SST File.
- Extended the column family statistics in the info log so the total amount of garbage in the blob files and the blob file space amplification factor are also logged. Also exposed the blob file space amp via the
rocksdb.blob-stats
DB property. - Introduced the API rocksdb_create_dir_if_missing in c.h that calls underlying file system's CreateDirIfMissing API to create the directory.
- Added last level and non-last level read statistics:
LAST_LEVEL_READ_*
,NON_LAST_LEVEL_READ_*
. - Experimental: Add support for new APIs ReadAsync in ...
RocksDB 6.29.3
Rocksdb Change Log
6.29.3 (2022-02-17)
Bug Fixes
- Fix a data loss bug for 2PC write-committed transaction caused by concurrent transaction commit and memtable switch (#9571).
6.29.2 (2022-02-15)
Performance Improvements
- DisableManualCompaction() doesn't have to wait scheduled manual compaction to be executed in thread-pool to cancel the job.
6.29.1 (2022-01-31)
Bug Fixes
- Fixed a major bug in which batched MultiGet could return old values for keys deleted by DeleteRange when memtable Bloom filter is enabled (memtable_prefix_bloom_size_ratio > 0). (The fix includes a substantial MultiGet performance improvement in the unusual case of both memtable_whole_key_filtering and prefix_extractor.)
6.29.0 (2022-01-21)
Note: The next release will be major release 7.0. See #9390 for more info.
Public API change
- Added values to
TraceFilterType
:kTraceFilterIteratorSeek
,kTraceFilterIteratorSeekForPrev
, andkTraceFilterMultiGet
. They can be set inTraceOptions
to filter out the operation types after which they are named. - Added
TraceOptions::preserve_write_order
. When enabled it guarantees write records are traced in the same order they are logged to WAL and applied to the DB. By default it is disabled (false) to match the legacy behavior and prevent regression. - Made the Env class extend the Customizable class. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method.
Options::OldDefaults
is marked deprecated, as it is no longer maintained.- Add ObjectLibrary::AddFactory and ObjectLibrary::PatternEntry classes. This method and associated class are the preferred mechanism for registering factories with the ObjectLibrary going forward. The ObjectLibrary::Register method, which uses regular expressions and may be problematic, is deprecated and will be in a future release.
- Changed
BlockBasedTableOptions::block_size
fromsize_t
touint64_t
. - Added API warning against using
Iterator::Refresh()
together withDB::DeleteRange()
, which are incompatible and have always risked causing the refreshed iterator to return incorrect results.
Behavior Changes
DB::DestroyColumnFamilyHandle()
will return Status::InvalidArgument() if called withDB::DefaultColumnFamily()
.- On 32-bit platforms, mmap reads are no longer quietly disabled, just discouraged.
New Features
- Added
Options::DisableExtraChecks()
that can be used to improve peak write performance by disabling checks that should not be necessary in the absence of software logic errors or CPU+memory hardware errors. (Default options are slowly moving toward some performance overheads for extra correctness checking.)
Performance Improvements
- Improved read performance when a prefix extractor is used (Seek, Get, MultiGet), even compared to version 6.25 baseline (see bug fix below), by optimizing the common case of prefix extractor compatible with table file and unchanging.
Bug Fixes
- Fix a bug that FlushMemTable may return ok even flush not succeed.
- Fixed a bug of Sync() and Fsync() not using
fcntl(F_FULLFSYNC)
on OS X and iOS. - Fixed a significant performance regression in version 6.26 when a prefix extractor is used on the read path (Seek, Get, MultiGet). (Excessive time was spent in SliceTransform::AsString().)
New Features
- Added RocksJava support for MacOS universal binary (ARM+x86)
RocksDB 6.28.2
6.28.2 (2022-01-31)
Bug Fixes
- Fixed a major bug in which batched MultiGet could return old values for keys deleted by DeleteRange when memtable Bloom filter is enabled (memtable_prefix_bloom_size_ratio > 0). (The fix includes a substantial MultiGet performance improvement in the unusual case of both memtable_whole_key_filtering and prefix_extractor.)
6.28.1 (2022-01-10)
Bug Fixes
- Fixed compilation errors on newer compiler, e.g. clang-12
6.28.0 (2021-12-17)
New Features
- Introduced 'CommitWithTimestamp' as a new tag. Currently, there is no API for user to trigger a write with this tag to the WAL. This is part of the efforts to support write-commited transactions with user-defined timestamps.
Bug Fixes
- Fixed a bug in rocksdb automatic implicit prefetching which got broken because of new feature adaptive_readahead and internal prefetching got disabled when iterator moves from one file to next.
- Fixed a bug in TableOptions.prepopulate_block_cache which causes segmentation fault when used with TableOptions.partition_filters = true and TableOptions.cache_index_and_filter_blocks = true.
- Fixed a bug affecting custom memtable factories which are not registered with the
ObjectRegistry
. The bug could result in failure to save the OPTIONS file. - Fixed a bug causing two duplicate entries to be appended to a file opened in non-direct mode and tracked by
FaultInjectionTestFS
. - Fixed a bug in TableOptions.prepopulate_block_cache to support block-based filters also.
- Block cache keys no longer use
FSRandomAccessFile::GetUniqueId()
(previously used when available), so a filesystem recycling unique ids can no longer lead to incorrect result or crash (#7405). For files generated by RocksDB >= 6.24, the cache keys are stable across DB::Open and DB directory move / copy / import / export / migration, etc. Although collisions are still theoretically possible, they are (a) impossible in many common cases, (b) not dependent on environmental factors, and (c) much less likely than a CPU miscalculation while executing RocksDB.
Behavior Changes
- MemTableList::TrimHistory now use allocated bytes when max_write_buffer_size_to_maintain > 0(default in TrasactionDB, introduced in PR#5022) Fix #8371.
Public API change
- Extend WriteBatch::AssignTimestamp and AssignTimestamps API so that both functions can accept an optional
checker
argument that performs additional checking on timestamp sizes. - Introduce a new EventListener callback that will be called upon the end of automatic error recovery.
Performance Improvements
- Replaced map property
TableProperties::properties_offsets
with uint64_t propertyexternal_sst_file_global_seqno_offset
to save table properties's memory. - Block cache accesses are faster by RocksDB using cache keys of fixed size (16 bytes).
Java API Changes
- Removed Java API
TableProperties.getPropertiesOffsets()
as it exposed internal details to external users.
RocksDB 6.27.3
6.27.3 (2021-12-10)
Bug Fixes
- Fixed a bug in TableOptions.prepopulate_block_cache which causes segmentation fault when used with TableOptions.partition_filters = true and TableOptions.cache_index_and_filter_blocks = true.
- Fixed a bug affecting custom memtable factories which are not registered with the
ObjectRegistry
. The bug could result in failure to save the OPTIONS file.
6.27.2 (2021-12-01)
Bug Fixes
- Fixed a bug in rocksdb automatic implicit prefetching which got broken because of new feature adaptive_readahead and internal prefetching got disabled when iterator moves from one file to next.
6.27.1 (2021-11-29)
Bug Fixes
- Fixed a bug that could, with WAL enabled, cause backups, checkpoints, and
GetSortedWalFiles()
to fail randomly with an error likeIO error: 001234.log: No such file or directory
6.27.0 (2021-11-19)
New Features
- Added new ChecksumType kXXH3 which is faster than kCRC32c on almost all x86_64 hardware.
- Added a new online consistency check for BlobDB which validates that the number/total size of garbage blobs does not exceed the number/total size of all blobs in any given blob file.
- Provided support for tracking per-sst user-defined timestamp information in MANIFEST.
- Added new option "adaptive_readahead" in ReadOptions. For iterators, RocksDB does auto-readahead on noticing sequential reads and by enabling this option, readahead_size of current file (if reads are sequential) will be carried forward to next file instead of starting from the scratch at each level (except L0 level files). If reads are not sequential it will fall back to 8KB. This option is applicable only for RocksDB internal prefetch buffer and isn't supported with underlying file system prefetching.
- Added the read count and read bytes related stats to Statistics for tiered storage hot, warm, and cold file reads.
- Added an option to dynamically charge an updating estimated memory usage of block-based table building to block cache if block cache available. It currently only includes charging memory usage of constructing (new) Bloom Filter and Ribbon Filter to block cache. To enable this feature, set
BlockBasedTableOptions::reserve_table_builder_memory = true
. - Add a new API OnIOError in listener.h that notifies listeners when an IO error occurs during FileSystem operation along with filename, status etc.
- Added compaction readahead support for blob files to the integrated BlobDB implementation, which can improve compaction performance when the database resides on higher-latency storage like HDDs or remote filesystems. Readahead can be configured using the column family option
blob_compaction_readahead_size
.
Bug Fixes
- Prevent a
CompactRange()
withCompactRangeOptions::change_level == true
from possibly causing corruption to the LSM state (overlapping files within a level) when run in parallel with another manual compaction. Note that settingforce_consistency_checks == true
(the default) would cause the DB to enter read-only mode in this scenario and returnStatus::Corruption
, rather than committing any corruption. - Fixed a bug in CompactionIterator when write-prepared transaction is used. A released earliest write conflict snapshot may cause assertion failure in dbg mode and unexpected key in opt mode.
- Fix ticker WRITE_WITH_WAL("rocksdb.write.wal"), this bug is caused by a bad extra
RecordTick(stats_, WRITE_WITH_WAL)
(at 2 place), this fix remove the extraRecordTick
s and fix the corresponding test case. - EventListener::OnTableFileCreated was previously called with OK status and file_size==0 in cases of no SST file contents written (because there was no content to add) and the empty file deleted before calling the listener. Now the status is Aborted.
- Fixed a bug in CompactionIterator when write-preared transaction is used. Releasing earliest_snapshot during compaction may cause a SingleDelete to be output after a PUT of the same user key whose seq has been zeroed.
- Added input sanitization on negative bytes passed into
GenericRateLimiter::Request
. - Fixed an assertion failure in CompactionIterator when write-prepared transaction is used. We prove that certain operations can lead to a Delete being followed by a SingleDelete (same user key). We can drop the SingleDelete.
- Fixed a bug of timestamp-based GC which can cause all versions of a key under full_history_ts_low to be dropped. This bug will be triggered when some of the ikeys' timestamps are lower than full_history_ts_low, while others are newer.
- In some cases outside of the DB read and compaction paths, SST block checksums are now checked where they were not before.
- Explicitly check for and disallow the
BlockBasedTableOptions
if insertion into one of {block_cache
,block_cache_compressed
,persistent_cache
} can show up in another of these. (RocksDB expects to be able to use the same key for different physical data among tiers.) - Users who configured a dedicated thread pool for bottommost compactions by explicitly adding threads to the
Env::Priority::BOTTOM
pool will no longer see RocksDB schedule automatic compactions exceeding the DB's compaction concurrency limit. For details on per-DB compaction concurrency limit, see API docs ofmax_background_compactions
andmax_background_jobs
. - Fixed a bug of background flush thread picking more memtables to flush and prematurely advancing column family's log_number.
- Fixed an assertion failure in ManifestTailer.
Behavior Changes
NUM_FILES_IN_SINGLE_COMPACTION
was only counting the first input level files, now it's including all input files.TransactionUtil::CheckKeyForConflicts
can also perform conflict-checking based on user-defined timestamps in addition to sequence numbers.- Removed
GenericRateLimiter
's minimum refill bytes per period previously enforced.
Public API change
- When options.ttl is used with leveled compaction with compactinon priority kMinOverlappingRatio, files exceeding half of TTL value will be prioritized more, so that by the time TTL is reached, fewer extra compactions will be scheduled to clear them up. At the same time, when compacting files with data older than half of TTL, output files may be cut off based on those files' boundaries, in order for the early TTL compaction to work properly.
- Made FileSystem extend the Customizable class and added a CreateFromString method. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method.
- Clarified in API comments that RocksDB is not exception safe for callbacks and custom extensions. An exception propagating into RocksDB can lead to undefined behavior, including data loss, unreported corruption, deadlocks, and more.
- Marked
WriteBufferManager
asfinal
because it is not intended for extension. - Removed unimportant implementation details from table_properties.h
- Add API
FSDirectory::FsyncWithDirOptions()
, which provides extra information like directory fsync reason inDirFsyncOptions
. File system like btrfs is using that to skip directory fsync for creating a new file, or when renaming a file, fsync the target file instead of the directory, which improves theDB::Open()
speed by ~20%. DB::Open()
is not going be blocked by obsolete file purge ifDBOptions::avoid_unnecessary_blocking_io
is set to true.- In builds where glibc provides
gettid()
, info log ("LOG" file) lines now print a system-wide thread ID fromgettid()
instead of the process-localpthread_self()
. For all users, the thread ID format is changed from hexadecimal to decimal integer. - In builds where glibc provides
pthread_setname_np()
, the background thread names no longer contain an ID suffix. For example, "rocksdb:bottom7" (and all other threads in theEnv::Priority::BOTTOM
pool) are now named "rocksdb:bottom". Previously large thread pools could breach the name size limit (e.g., naming "rocksdb:bottom10" would fail). - Deprecating
ReadOptions::iter_start_seqnum
andDBOptions::preserve_deletes
, please try using user defined timestamp feature instead. The options will be removed in a future release, currently it logs a warning message when using.
Performance Improvements
- Released some memory related to filter construction earlier in
BlockBasedTableBuilder
forFullFilter
andPartitionedFilter
case (#9070)
RocksDB 6.26.1
6.26.1 (2021-11-18)
Bug Fixes
- Fix builds for some platforms.
6.26.0 (2021-10-20)
Bug Fixes
- Fixes a bug in directed IO mode when calling MultiGet() for blobs in the same blob file. The bug is caused by not sorting the blob read requests by file offsets.
- Fix the incorrect disabling of SST rate limited deletion when the WAL and DB are in different directories. Only WAL rate limited deletion should be disabled if its in a different directory.
- Fix
DisableManualCompaction()
to cancel compactions even when they are waiting on automatic compactions to drain due toCompactRangeOptions::exclusive_manual_compactions == true
. - Fix contract of
Env::ReopenWritableFile()
andFileSystem::ReopenWritableFile()
to specify any existing file must not be deleted or truncated. - Fixed bug in calls to
IngestExternalFiles()
with files for multiple column families. The bug could have introduced a delay in ingested file keys becoming visible afterIngestExternalFiles()
returned. Furthermore, mutations to ingested file keys while they were invisible could have been dropped (not necessarily immediately). - Fixed a possible race condition impacting users of
WriteBufferManager
who constructed it withallow_stall == true
. The race condition led to undefined behavior (in our experience, typically a process crash). - Fixed a bug where stalled writes would remain stalled forever after the user calls
WriteBufferManager::SetBufferSize()
withnew_size == 0
to dynamically disable memory limiting. - Make
DB::close()
thread-safe. - Fix a bug in atomic flush where one bg flush thread will wait forever for a preceding bg flush thread to commit its result to MANIFEST but encounters an error which is mapped to a soft error (DB not stopped).
New Features
- Print information about blob files when using "ldb list_live_files_metadata"
- Provided support for SingleDelete with user defined timestamp.
- Experimental new function DB::GetLiveFilesStorageInfo offers essentially a unified version of other functions like GetLiveFiles, GetLiveFilesChecksumInfo, and GetSortedWalFiles. Checkpoints and backups could show small behavioral changes and/or improved performance as they now use this new API.
- Add remote compaction read/write bytes statistics:
REMOTE_COMPACT_READ_BYTES
,REMOTE_COMPACT_WRITE_BYTES
. - Introduce an experimental feature to dump out the blocks from block cache and insert them to the secondary cache to reduce the cache warmup time (e.g., used while migrating DB instance). More information are in
class CacheDumper
andCacheDumpedLoader
atrocksdb/utilities/cache_dump_load.h
Note that, this feature is subject to the potential change in the future, it is still experimental. - Introduced a new BlobDB configuration option
blob_garbage_collection_force_threshold
, which can be used to trigger compactions targeting the SST files which reference the oldest blob files when the ratio of garbage in those blob files meets or exceeds the specified threshold. This can reduce space amplification with skewed workloads where the affected SST files might not otherwise get picked up for compaction. - Added EXPERIMENTAL support for table file (SST) unique identifiers that are stable and universally unique, available with new function
GetUniqueIdFromTableProperties
. Only SST files from RocksDB >= 6.24 support unique IDs. - Added
GetMapProperty()
support for "rocksdb.dbstats" (DB::Properties::kDBStats
). As a map property, it includes DB-level internal stats accumulated over the DB's lifetime, such as user write related stats and uptime.
Public API change
- Made SystemClock extend the Customizable class and added a CreateFromString method. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method.
- Made SliceTransform extend the Customizable class and added a CreateFromString method. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method. The Capped and Prefixed transform classes return a short name (no length); use GetId for the fully qualified name.
- Made FileChecksumGenFactory, SstPartitionerFactory, TablePropertiesCollectorFactory, and WalFilter extend the Customizable class and added a CreateFromString method.
- Some fields of SstFileMetaData are deprecated for compatibility with new base class FileStorageInfo.
- Add
file_temperature
toIngestExternalFileArg
such that when ingesting SST files, we are able to indicate the temperature of the this batch of files. - If
DB::Close()
failed with a non aborted status, callingDB::Close()
again will return the original status instead of Status::OK. - Add CacheTier to advanced_options.h to describe the cache tier we used. Add a
lowest_used_cache_tier
option toDBOptions
(immutable) and pass it to BlockBasedTableReader. By default it isCacheTier::kNonVolatileBlockTier
, which means, we always use both block cache (kVolatileTier) and secondary cache (kNonVolatileBlockTier). By set it toCacheTier::kVolatileTier
, the DB will not use the secondary cache. - Even when options.max_compaction_bytes is hit, compaction output files are only cut when it aligns with grandparent files' boundaries. options.max_compaction_bytes could be slightly violated with the change, but the violation is no more than one target SST file size, which is usually much smaller.
Performance Improvements
Java API Changes
- Add Java API bindings for new integrated BlobDB options
keyMayExist()
supports ByteBuffer.- Fix multiget throwing Null Pointer Exception for num of keys > 70k (#8039).
RocksDB 6.26.0
6.26.0 (2021-10-20)
Bug Fixes
- Fixes a bug in directed IO mode when calling MultiGet() for blobs in the same blob file. The bug is caused by not sorting the blob read requests by file offsets.
- Fix the incorrect disabling of SST rate limited deletion when the WAL and DB are in different directories. Only WAL rate limited deletion should be disabled if its in a different directory.
- Fix
DisableManualCompaction()
to cancel compactions even when they are waiting on automatic compactions to drain due toCompactRangeOptions::exclusive_manual_compactions == true
. - Fix contract of
Env::ReopenWritableFile()
andFileSystem::ReopenWritableFile()
to specify any existing file must not be deleted or truncated. - Fixed bug in calls to
IngestExternalFiles()
with files for multiple column families. The bug could have introduced a delay in ingested file keys becoming visible afterIngestExternalFiles()
returned. Furthermore, mutations to ingested file keys while they were invisible could have been dropped (not necessarily immediately). - Fixed a possible race condition impacting users of
WriteBufferManager
who constructed it withallow_stall == true
. The race condition led to undefined behavior (in our experience, typically a process crash). - Fixed a bug where stalled writes would remain stalled forever after the user calls
WriteBufferManager::SetBufferSize()
withnew_size == 0
to dynamically disable memory limiting. - Make
DB::close()
thread-safe. - Fix a bug in atomic flush where one bg flush thread will wait forever for a preceding bg flush thread to commit its result to MANIFEST but encounters an error which is mapped to a soft error (DB not stopped).
New Features
- Print information about blob files when using "ldb list_live_files_metadata"
- Provided support for SingleDelete with user defined timestamp.
- Experimental new function DB::GetLiveFilesStorageInfo offers essentially a unified version of other functions like GetLiveFiles, GetLiveFilesChecksumInfo, and GetSortedWalFiles. Checkpoints and backups could show small behavioral changes and/or improved performance as they now use this new API.
- Add remote compaction read/write bytes statistics:
REMOTE_COMPACT_READ_BYTES
,REMOTE_COMPACT_WRITE_BYTES
. - Introduce an experimental feature to dump out the blocks from block cache and insert them to the secondary cache to reduce the cache warmup time (e.g., used while migrating DB instance). More information are in
class CacheDumper
andCacheDumpedLoader
atrocksdb/utilities/cache_dump_load.h
Note that, this feature is subject to the potential change in the future, it is still experimental. - Introduced a new BlobDB configuration option
blob_garbage_collection_force_threshold
, which can be used to trigger compactions targeting the SST files which reference the oldest blob files when the ratio of garbage in those blob files meets or exceeds the specified threshold. This can reduce space amplification with skewed workloads where the affected SST files might not otherwise get picked up for compaction. - Added EXPERIMENTAL support for table file (SST) unique identifiers that are stable and universally unique, available with new function
GetUniqueIdFromTableProperties
. Only SST files from RocksDB >= 6.24 support unique IDs. - Added
GetMapProperty()
support for "rocksdb.dbstats" (DB::Properties::kDBStats
). As a map property, it includes DB-level internal stats accumulated over the DB's lifetime, such as user write related stats and uptime.
Public API change
- Made SystemClock extend the Customizable class and added a CreateFromString method. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method.
- Made SliceTransform extend the Customizable class and added a CreateFromString method. Implementations need to be registered with the ObjectRegistry and to implement a Name() method in order to be created via this method. The Capped and Prefixed transform classes return a short name (no length); use GetId for the fully qualified name.
- Made FileChecksumGenFactory, SstPartitionerFactory, TablePropertiesCollectorFactory, and WalFilter extend the Customizable class and added a CreateFromString method.
- Some fields of SstFileMetaData are deprecated for compatibility with new base class FileStorageInfo.
- Add
file_temperature
toIngestExternalFileArg
such that when ingesting SST files, we are able to indicate the temperature of the this batch of files. - If
DB::Close()
failed with a non aborted status, callingDB::Close()
again will return the original status instead of Status::OK. - Add CacheTier to advanced_options.h to describe the cache tier we used. Add a
lowest_used_cache_tier
option toDBOptions
(immutable) and pass it to BlockBasedTableReader. By default it isCacheTier::kNonVolatileBlockTier
, which means, we always use both block cache (kVolatileTier) and secondary cache (kNonVolatileBlockTier). By set it toCacheTier::kVolatileTier
, the DB will not use the secondary cache. - Even when options.max_compaction_bytes is hit, compaction output files are only cut when it aligns with grandparent files' boundaries. options.max_compaction_bytes could be slightly violated with the change, but the violation is no more than one target SST file size, which is usually much smaller.
Performance Improvements
Java API Changes
- Add Java API bindings for new integrated BlobDB options
keyMayExist()
supports ByteBuffer.- Fix multiget throwing Null Pointer Exception for num of keys > 70k (#8039).