Compression not compressing files

[Provissy]

System information

Type	Version/Name
Distribution Name	Ubuntu Focal Fossa (development branch)
Distribution Version	20.04
Linux Kernel	5.4.0-21-generic
Architecture	x86_64
ZFS Version	0.8.3-1ubuntu8
SPL Version	0.8.3-1ubuntu8

Describe the problem you're observing

Compression seems not working.

I used a single HDD with ZFS for testing purpose and plan to deploy after the raid-z reflux codes are done. After months of use, I found that the compression seems not engaging.

$ zfs list
NAME            USED  AVAIL     REFER  MOUNTPOINT
MY10TB_ZPOOL   8.16T   661G     8.16T  /media/ubuntu/MY10TB_ZPOOL

$ zfs get compressratio
NAME           PROPERTY       VALUE  SOURCE
MY10TB_ZPOOL   compressratio  1.01x  -

Since I am not 100% sure, I designed a small test to compare with btrfs.

Describe how to reproduce the problem

A single MHT file will be used for demonstration, it contains texts and images, the file size is 12GB. The first portion(about 3 GB) is plain UTF-8 texts so the file is supposed to be recognised as "compressible".

First, I cut two partitions on a single HDD for testing.

$parted -l

Number  Start   End     Size    File system  Name      Flags
 1      101GB   150GB   49.0GB  btrfs      
 4      151GB   200GB   49.0GB  zfs       

Then, create filesystems with default options, after that

For ZFS, set

zfs set compression=on testZcomp

$ zfs get compression

NAME       PROPERTY     VALUE     SOURCE
testZcomp  compression  on        local

For BTRFS, mount with
mount -t btrfs -o compress=zstd /dev/sdc1 testBcomp/

$ mount -t btrfs
/dev/sdc1 on /testBcomp type btrfs (rw,relatime,compress=zstd:3,space_cache,subvolid=5,subvol=/)

Finally, copy the test file into each filesystem with

$ rsync -r -v --info=progress2 bigsite.mht /testZcomp/

sending incremental file list
bigsite.mht
 12,845,046,339 100%  125.92MB/s    0:01:37 (xfr#1, to-chk=0/1)

and

$ rsync -r -v --info=progress2 bigsite.mht /testBcomp/

sending incremental file list
bigsite.mht
 12,845,046,339 100%  139.18MB/s    0:01:28 (xfr#1, to-chk=0/1)

Results

$df
testZcomp               46217344   12533248      33684096  28% /testZcomp
/dev/sdc1               47851520    9737884      37641940  21% /testBcomp

$ df -h
testZcomp            45G   12G   33G  28% /testZcomp
/dev/sdc1            46G  9.3G   36G  21% /testBcomp

$ du /testZcomp/
12533045        /testZcomp/

$ du --apparent-size /testZcomp/
12543991        /testZcomp/

$ du /testBcomp/
12544008        /testBcomp/

$ compsize /testBcomp/
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       77%      9.2G          11G          11G
zstd        77%      9.2G          11G          11G

$ compsize --bytes /testBcomp/
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       77%     9897226240   12845047808  12845047808
zstd        77%     9897226240   12845047808  12845047808

Conclusion

As shown above, I assume the file is not being compressed at all. This problem might already exist before I upgraded to Ubuntu Focal Beta, if it is not reproducible, please tell me what more information I should summit.

Finally I want to thank you developers for creating this amazing file system, I suffered a severe data lost when using BTRFS due to bugs(even with Kernel 5.3!). ZFS is just the best FS. I also hope ZSTD could be soon implemented!

Include any warning/errors/backtraces from the system logs

/proc/spl/kstat/zfs# cat dbgmsg | grep testZ
1586848005   mmp.c:248:mmp_thread_start(): MMP thread started pool 'testZcomp' gethrtime 69397796080600
1586848006   spa_history.c:305:spa_history_log_sync(): command: zpool create testZcomp /dev/sdc4
1586848060   spa_history.c:309:spa_history_log_sync(): txg 18 set testZcomp (id 54) compression=1
1586848065   spa_history.c:305:spa_history_log_sync(): command: zfs set compression=on testZcomp
1586848724   metaslab.c:2736:metaslab_condense(): condensing: txg 302, msp[1] ffff8cbaebc75800, vdev id 0, spa testZcomp, smp size 16432, segments 41, forcing condense=FALSE
1586848724   metaslab.c:2736:metaslab_condense(): condensing: txg 302, msp[0] ffff8cbaebc73800, vdev id 0, spa testZcomp, smp size 16432, segments 41, forcing condense=FALSE
1586848726   metaslab.c:2736:metaslab_condense(): condensing: txg 308, msp[3] ffff8cbaebc70000, vdev id 0, spa testZcomp, smp size 16440, segments 11, forcing condense=FALSE
1586848735   metaslab.c:2736:metaslab_condense(): condensing: txg 336, msp[5] ffff8cbaebc75000, vdev id 0, spa testZcomp, smp size 16432, segments 22, forcing condense=FALSE
1586848735   metaslab.c:2736:metaslab_condense(): condensing: txg 337, msp[7] ffff8cbaebc72800, vdev id 0, spa testZcomp, smp size 16424, segments 15, forcing condense=FALSE
1586848735   metaslab.c:2736:metaslab_condense(): condensing: txg 337, msp[6] ffff8cbaebc73000, vdev id 0, spa testZcomp, smp size 16424, segments 15, forcing condense=FALSE
1586848769   metaslab.c:2736:metaslab_condense(): condensing: txg 440, msp[31] ffff8cbaf5c02000, vdev id 0, spa testZcomp, smp size 16408, segments 21, forcing condense=FALSE
1586848780   metaslab.c:2736:metaslab_condense(): condensing: txg 476, msp[1] ffff8cbaebc75800, vdev id 0, spa testZcomp, smp size 16392, segments 103, forcing condense=FALSE
1586848780   metaslab.c:2736:metaslab_condense(): condensing: txg 476, msp[0] ffff8cbaebc73800, vdev id 0, spa testZcomp, smp size 16392, segments 103, forcing condense=FALSE
1586848789   metaslab.c:2736:metaslab_condense(): condensing: txg 506, msp[3] ffff8cbaebc70000, vdev id 0, spa testZcomp, smp size 16416, segments 29, forcing condense=FALSE

[gmelikov]

There are several factors here:

• you compare lz4 in ZFS with zstd in BTRFS
• zstd has better compress ratio
• ZFS 0.8.3 won't write compressed data if it's more than 87.5% of original size (there is a WIP PR on this case zio_compress: introduce max_size threshold #9416 )
• raidz may have a write amplification on small blocks https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz

So if zstd can give you 77% data compression, there is a chance that your data won't be compressed by lz4 better than 87.5%.

Please check that your data compresses better than ~80% (per-block compression in ZFS will be slightly worse).

[Provissy]

Thanks for your reply.

I switched to gzip and tested again, the file is now being properly compressed. Ratio is 1.30x

I never noticed there is an underlying 87.5% "criteria" for compression... It is not documented in Oracle's admin guide. I finally found it in zfs command's man page.

Please close this issue if you think problem is solved, or keep it to track the PR status.

[gmelikov]

Glad to hear that!
I'll close this issue in favor of #9416, because it works as intended now.