-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restore improvements #3986
base: master
Are you sure you want to change the base?
Restore improvements #3986
Commits on Sep 2, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 948adde - Browse repository at this point
Copy the full SHA 948addeView commit details -
refactor(restore): wrap smaller stages into separate functions
This decreases main function complexity. Noticed by: cognitive complexity 66 of func `(*tablesWorker).restore` is high (> 50) (gocognit).
Configuration menu - View commit details
-
Copy full SHA for d99b4b5 - Browse repository at this point
Copy the full SHA d99b4b5View commit details -
feat(restore_test): validate that compaction and tombstone_gc are dis…
…abled and enabled As a preparation for restoring data, SM should disable tombstone_gc and compaction. They should be re-enabled after the restore finishes.
Configuration menu - View commit details
-
Copy full SHA for 5a3c05a - Browse repository at this point
Copy the full SHA 5a3c05aView commit details -
feat(restore): new indexing and batching approach
The idea is to first index all files to be restored, so that we can create better batches. Restore workload is aggregated first by table, then by remote sstable dir. From batching we expect: - batch contains X*shard_cnt sstables - batch contains similarly sized sstables - batch is created from any manifest/table - no waiting for manifest/table restore to finish - workload across different nodes is evenly distributed - --batch-size is ignored, batches are aiming to have size equal to 5% of expected node workload New flag --table-parallel - it allows for running multiple download and l&s jobs from the same node (no documentation as it might not be exposed later on).
Configuration menu - View commit details
-
Copy full SHA for cf29ed4 - Browse repository at this point
Copy the full SHA cf29ed4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8cb67ff - Browse repository at this point
Copy the full SHA 8cb67ffView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3dfdae6 - Browse repository at this point
Copy the full SHA 3dfdae6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 84f0790 - Browse repository at this point
Copy the full SHA 84f0790View commit details -
Configuration menu - View commit details
-
Copy full SHA for 258310a - Browse repository at this point
Copy the full SHA 258310aView commit details -
Configuration menu - View commit details
-
Copy full SHA for b47e813 - Browse repository at this point
Copy the full SHA b47e813View commit details -
Configuration menu - View commit details
-
Copy full SHA for 96820e5 - Browse repository at this point
Copy the full SHA 96820e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d498511 - Browse repository at this point
Copy the full SHA d498511View commit details -
feat(restoretest): enable --unpin-agent-cpu, --stream-to-all-replicas…
… and --table-parallel flags
Configuration menu - View commit details
-
Copy full SHA for 7fcc9be - Browse repository at this point
Copy the full SHA 7fcc9beView commit details -
Configuration menu - View commit details
-
Copy full SHA for 48154eb - Browse repository at this point
Copy the full SHA 48154ebView commit details -
Configuration menu - View commit details
-
Copy full SHA for 97425bd - Browse repository at this point
Copy the full SHA 97425bdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 98da79f - Browse repository at this point
Copy the full SHA 98da79fView commit details