PSA: borg does not support hardlink-copied repos

[ThomasWaldmann]

Thought I could be clever and save some disk space when testing borg 1.2.

TL;DR: DON'T USE HARDLINKED REPO COPIES!

That's roughly what I did:

• took a borg 1.1 repo, let's call it repo11
• made a hardlink copy of it (cp -al repo11 repo12, IIRC)
• made a copy of repo12/config, edited the repo id inside to avoid clientside cache confusion and moved it back over repo12/config (making a copy first avoids to edit BOTH repos' configs at the same time as they are also hardlinked)
• of course i am aware of the crypto issues with repo copies, but that's not a big issue here, backup server is trusted. also, did only intend to run both repos in parallel temporarily.
• ran borg1.2 compact --cleanup-commits repo12 (among other borg1.2 commands, like check, list, etc.)

This unintentionally modified my borg11 repo, because of how delete_segment works: it calls truncate_and_unlink, which first modifies the file in place, truncating it to 0 bytes and then unlinks it (deletes it).

If an inode is referenced by 2 filenames (2 hardlinks), that procedure will give:

• a 0 bytes file in borg11 repo
• a deleted file in borg12 repo

In this case, the damage is relatively limited, the 17b files were crap anyway.

But delete_segment is also called for real, with-content segment files when the compaction runs. And that will cause real damage on the other (hardlinked) repo.

repository.py has this, explaining why truncation is used:

File system interaction
-----------------------

LoggedIO generally tries to rely on common behaviours across transactional file systems.

Segments that are deleted are truncated first, which avoids problems if the FS needs to
allocate space to delete the dirent of the segment. This mostly affects CoW file systems,
traditional journaling file systems have a fairly good grip on this problem.

Note that deletion, i.e. unlink(2), is atomic on every file system that uses inode reference
counts, which includes pretty much all of them. To remove a dirent the inodes refcount has
to be decreased, but you can't decrease the refcount before removing the dirent nor can you
decrease the refcount after removing the dirent. File systems solve this with a lock,
and by ensuring it all stays within the same FS transaction.

Truncation is generally not atomic in itself, and combining truncate(2) and unlink(2) is of
course never guaranteed to be atomic. Truncation in a classic extent-based FS is done in
roughly two phases, first the extents are removed then the inode is updated. (In practice
this is of course way more complex).

LoggedIO gracefully handles truncate/unlink splits as long as the truncate resulted in
a zero length file. Zero length segments are considered to not exist, while LoggedIO.cleanup()
will still get rid of them.

At least the corruption in the borg11 repo was rather noticable: borg basically stopped working, just throwing exceptions and hanging.

So guess I'll use the borg12 repo exclusively from now on. :-)

[ThomasWaldmann]

@enkore ^^^

[enkore]

Yeah, this is rather ugly. I suppose one could skip the truncate if st_nlink>1 to avoid clobbering hardlinks, or fail in the first place when seeing st_nlink>1 in the repo (although I don't think we modify segments in-place after-the-fact in any other branch, do we?). Another option could be to only truncate if unlinking fails with ENOSPC and st_nlink=1, failing otherwise. That of course adds code to a failure-only path so not exactly an amazing solution, either.

[ThomasWaldmann]

I don't think we elsewhere modify segments in-place, that's why I had the idea with the hardlinks-copy.

Truncating only on ENOSPC and st_nlink == 1 sounds good.

[MXLBEATS1]

File system interaction

LoggedIO generally tries to rely on common behaviours across transactional file systems.

Segments that are deleted are truncated first, which avoids problems if the FS needs to
allocate space to delete the dirent of the segment. This mostly affects CoW file systems,
traditional journaling file systems have a fairly good grip on this problem.

Note that deletion, i.e. unlink(2), is atomic on every file system that uses inode reference
counts, which includes pretty much all of them. To remove a dirent the inodes refcount has
to be decreased, but you can't decrease the refcount before removing the dirent nor can you
decrease the refcount after removing the dirent. File systems solve this with a lock,
and by ensuring it all stays within the same FS transaction.

Truncation is generally not atomic in itself, and combining truncate(2) and unlink(2) is of
course never guaranteed to be atomic. Truncation in a classic extent-based FS is done in
roughly two phases, first the extents are removed then the inode is updated. (In practice
this is of course way more complex).

LoggedIO gracefully handles truncate/unlink splits as long as the truncate resulted in
a zero length file. Zero length segments are considered to not exist, while LoggedIO.cleanup()
will still get rid of them.``