Chunker permutation

[ncleaton]

Make the buzhash chunker more resistant to fingerprinting by introducing a permutation of the buzhash table, as discussed at #3687

[codecov-io]

Codecov Report

Merging #5070 (6647331) into master (d2673c0) will decrease coverage by 0.00%.
The diff coverage is 92.00%.

@@            Coverage Diff             @@
##           master    #5070      +/-   ##
==========================================
- Coverage   83.15%   83.15%   -0.01%     
==========================================
  Files          38       38              
  Lines       10058    10078      +20     
  Branches     1668     1672       +4     
==========================================
+ Hits         8364     8380      +16     
- Misses       1202     1204       +2     
- Partials      492      494       +2     

Impacted Files	Coverage Δ
src/borg/crypto/key.py	86.71% <90.47%> (+0.11%)	⬆️
src/borg/archive.py	81.44% <100.00%> (-0.14%)	⬇️
src/borg/selftest.py	73.80% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2118f1...6647331. Read the comment docs.

[ThomasWaldmann]

It would be useful, if someone else would also review this.

[textshell]

On first look this seems to silently diverge on the chunking when this key is used in older borg versions? That sounds problematic.

[ThomasWaldmann]

found one cosmetic thing while doing another review.

[ThomasWaldmann]

could be also just break.

[ThomasWaldmann]

@textshell looks to me like:

• if repo was initialized with old borg (e.g. 1.1.14), the PR code will use the null permutation and behave same, compute same chunks, no dedup issue, even if borg is switched between old and new versions.

• if repo was initialized with new PR code, the key material will be longer (132 bytes instead of 100 bytes), the new code would compute different chunks than the old code. So there would be a dedup issue, but we could just document that in changelog.

I have still to check how the old code deals with a key that is longer than expected. I guess it would just silently ignore that.

Do you think it is problematic if we just document "don't use borg 1.1.x on repos created with borg 1.2"?

[ThomasWaldmann]

We could also set item.Key.version to 2 (was: 1), then borg 1.1.x code will reject working with a key made by borg 1.2.

borg 1.2 would check the version and accept version 1 (null permutation, compatibility mode) or version 2 (random permutation).

[ThomasWaldmann]

rebased on current master.