[Rootless] Permission denied: unknown on image pull via nerdctl

[inklesspen1rus]

Issue

Tried to pull ubuntu:20.04 via nerdctl using nydus-snapshotter, but got permission denied: unknown:

$ nerdctl --snapshotter nydus image pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 8.2 s                                                                    total:  26.2 M (3.2 MiB/s)                                       
FATA[0008] failed to commit snapshot extract-160833661-2ZUy sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/3/fs/var/cache/apt/archives/partial: permission denied: unknown 

Expected result

alpine:3 pulls fine:

$ nerdctl --snapshotter nydus image pull alpine:3
docker.io/library/alpine:3:                                                       resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 4.9 s                                                                    total:  3.3 Mi (680.0 KiB/s)

Environment

containerd in rootless via user systemd
config.tar.gz from $HOME/.config

$ inxi
CPU: 6-core AMD Ryzen 5 5625U with Radeon Graphics (-MT MCP-)
speed/min/max: 1091/400/4388 MHz Kernel: 6.7.12-1-MANJARO x86_64 Up: 4d 6h 13m
Mem: 6.64/15.01 GiB (44.2%) Storage: 476.94 GiB (66.7% used) Procs: 422
Shell: Zsh inxi: 3.3.33
$ containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8.m
$ nerdctl --version
nerdctl version 1.7.2
$ nydusd --version

Version:        v2.3.0-alpha.1
Git Commit:     93ef71db793ae36b12b0e9e6e08d1b4e9566b498
Build Time:     2023-12-06T01:10:03.515180463Z
Profile:        release
Rustc:          rustc 1.68.2 (9eb3afe9e 2023-03-27)
$ containerd-nydus-grpc --version
Version:     v0.13.11
Revision:    7835988d383d591d4f4b1e0e3a1f0c71f6ac8a77
Go version:  go1.19.6
Build time:  2024-03-22T11:10:30

[imeoer]

Any error logs are output from nydus-snapshotter for the ubuntu:20.04 image? The problem doesn't seem to be nydus related (ubuntu:20.04 is not a nydus image), have you tried removing --snapshotter nydus ?

[inklesspen1rus]

Thank you for reply!

Yes, it works fine with --snapshotter overlayfs (Currently I have default snapshotter - stargz):

$ nerdctl --snapshotter=overlayfs pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 180.0s                                                                   total:  26.2 M (149.3 KiB/s)

With --snapshotter=nydus

$ nerdctl --snapshotter=nydus pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 31.9s                                                                    total:  26.2 M (842.3 KiB/s)                                     
FATA[0032] failed to commit snapshot extract-111208692-sqvL sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/1/fs/var/cache/apt/archives/partial: permission denied: unknown

Here's nydus logs:
nydus-snapshotter.log

[inklesspen1rus]

Is it would be simpler if I share qemu virtual machine image with that issue?

So you won't worry about reproducing bug

[imeoer]

There are no exceptions in the nydus snapshotter logs, please check if it is related to the access permissions of the directory where /home/inklesspen/.local/share/containerd-nydus/snapshots/1/fs/var/cache/apt/archives/partial is located, e.g., the access perm of the directory /home/inklesspen/.local/share/containerd-nydus are not configured correctly.

[inklesspen1rus]

Chmodded 777, still doesn't work

$ cd /home/inklesspen/.local/share/containerd-nydus
$ ls -lah
total 80K
drwx------ 1 inklesspen inklesspen   74 апр 24 16:08 .
drwxr-xr-x 1 inklesspen inklesspen 1,4K апр 25 02:04 ..
drwxr-xr-x 1 inklesspen inklesspen    0 апр 21 16:00 cache
drwxr-xr-x 1 inklesspen inklesspen   42 апр 21 16:00 logs
-rw------- 1 inklesspen inklesspen  64K апр 24 16:08 metadata.db
-rw------- 1 inklesspen inklesspen  64K апр 24 16:01 nydus.db
drwx------ 1 inklesspen inklesspen    0 апр 24 16:08 snapshots
$ ls -lah snapshots
total 0
drwx------ 1 inklesspen inklesspen  0 апр 24 16:08 .
drwx------ 1 inklesspen inklesspen 74 апр 24 16:08 ..
$ chmod -R 777 .
$ ls -lah
total 88K
drwxrwxrwx 1 inklesspen inklesspen   74 апр 24 16:08 .
drwxr-xr-x 1 inklesspen inklesspen 1,4K апр 25 10:59 ..
drwxrwxrwx 1 inklesspen inklesspen    0 апр 21 16:00 cache
drwxrwxrwx 1 inklesspen inklesspen   42 апр 21 16:00 logs
-rwxrwxrwx 1 inklesspen inklesspen  64K апр 25 11:00 metadata.db
-rwxrwxrwx 1 inklesspen inklesspen  64K апр 24 16:01 nydus.db
drwxrwxrwx 1 inklesspen inklesspen    2 апр 25 11:00 snapshots
$ nerdctl --snapshotter=nydus pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    exists         |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 14.1s                                                                    total:  26.2 M (1.9 MiB/s)                                       
FATA[0014] failed to commit snapshot extract-313934468-QzOP sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/4/fs/var/cache/apt/archives/partial: permission denied: unknown 

[changweige]

@inklesspen1rus Looks like you are running rootless container. Nydus-snapshotter is not expected to run in such environment yet. For rootless container, nydus-snapshotter has to help containerd do to UIDMAP mount, however, it is not implemented yet.