Skip to content

Commit

Permalink
fix(restore): handle download of fully versioned batch
Browse files Browse the repository at this point in the history
Recent commits changed versioned batch download so that
if any sstable component is versioned, then all sstable components
are downloaded as versioned files.
It was done in that way to allow easier versioned progress calculation
(we don't store per file size, only the whole sstable size).

This brought to light a bug (that existed before, but was more difficult to hit),
in which restoring batch failed when the whole batch was versioned,
as calling RcloneSyncCopyPaths on empty paths parameter resulted in broken download.

We could just skip the RcloneSyncCopyPaths call when the whole batch is versioned,
but this would leave us without the agentJobID which is a part of sort key
in RestoreRunProgress. Without it, we could potentially overwrite one
restore run progress with another - if both of them happened on the same
RemoteSSTableDir, by the same Host, and were fully versioned.
It would also introduce a different path for restoring regular batch and fully versioned batch,
which is not desirable.

That's why I decided to modify rclone server to allow empty path parameter,
so that it still generates agentJobID, but it doesn't do anything except for that.
  • Loading branch information
Michal-Leszczynski committed Sep 26, 2024
1 parent a7705d2 commit bc04376
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 1 deletion.
4 changes: 3 additions & 1 deletion pkg/rclone/rcserver/rc.go
Original file line number Diff line number Diff line change
Expand Up @@ -554,7 +554,9 @@ func rcCopyPaths() func(ctx context.Context, in rc.Params) (rc.Params, error) {
if err != nil {
return nil, err
}

if len(paths) == 0 {
return nil, nil
}
return nil, sync.CopyPaths(ctx, dstFs, dstRemote, srcFs, srcRemote, paths, false)
}
}
Expand Down
3 changes: 3 additions & 0 deletions pkg/scyllaclient/client_rclone.go
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,9 @@ func (c *Client) RcloneCopyPaths(ctx context.Context, host, dstRemoteDir, srcRem
if err != nil {
return 0, err
}
if paths == nil {
paths = make([]string, 0)
}

p := operations.SyncCopyPathsParams{
Context: forceHost(ctx, host),
Expand Down

0 comments on commit bc04376

Please sign in to comment.