Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scripts now run on the host by default #1729

Closed
DaanDeMeyer opened this issue Aug 4, 2023 · 5 comments
Closed

Scripts now run on the host by default #1729

DaanDeMeyer opened this issue Aug 4, 2023 · 5 comments
Labels
announcement

Comments

@DaanDeMeyer
Copy link
Contributor

DaanDeMeyer commented Aug 4, 2023 •
edited
Loading

Build scripts are now executed on the host. See the SCRIPTS section
in the manual for more information. Existing build scripts will need
to be updated to make sure they keep working. Specifically, most paths
in scripts will need to be prefixed with $BUILDROOT to have them
operate on the image instead of on the host system. To ensure the host
system cannot be modified when running a script, most host directories
are mounted read-only when running a script to ensure a script cannot
modify the host in any way. Alternatively to making the script run on
the host, the script can also still be executed in the image itself by
putting the following snippet at the top of the script:

if [ "$container" != "mkosi" ]; then
    exec mkosi-chroot "$SCRIPT" "$@"
fi
@DaanDeMeyer DaanDeMeyer pinned this issue Aug 4, 2023
@iam-TJ
Copy link

iam-TJ commented Aug 5, 2023

"putting the following snippet at the top of the script:"

Please, don't leave us hanging... is this a null snippet or is it really missing?

@iam-TJ
Copy link

iam-TJ commented Aug 5, 2023 •
edited
Loading

(Updated)
And I think I already hit this whilst building latest systemd! Without the latest mkosi commits specifically 9b626c647037bc8a (I was inadvertently on a topic branch that was a few commits behind):

chroot: cannot change root directory to '/work/build-script': No such file or directory

Is this going to break builds of projects like systemd for those with a distro-installed version of mkosi that isn't very recent?

Build log:

‣   Copying cached trees
‣ + stat --file-system --format %T /srv/NAS/Sunny/SourceCode/systemd/systemd/mkosi.cache/debian~bookworm/base.cache
‣ + cp --recursive --preserve=mode,timestamps,links,xattr,ownership --no-target-directory --reflink=auto /srv/NAS/Sunny/SourceCode/systemd/systemd/mkosi.cache/debian~bookw
orm/base.cache /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root
‣ + mount --no-mtab overlay /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root --types overlay --options lowerdir=/srv/NAS/Sunny/SourceCode/systemd/systemd/
.mkosi.tmpql8j70uv/root,upperdir=/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/build-overlay,workdir=/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8
j70uv/build-overlay-workdirsdaw6i4n,userxattr
‣ + umount --no-mtab --lazy /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root
‣   Cleaning up overlayfs
‣    Removing overlay whiteout files…
‣   Running build script…
‣ + mount --no-mtab overlay /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root --types overlay --options ro,lowerdir=/srv/NAS/Sunny/SourceCode/systemd/syste
md/.mkosi.tmpql8j70uv/root,upperdir=/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/build-overlay,workdir=/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmp
ql8j70uv/build-overlay-workdirlffvkung,userxattr
‣ + bwrap --dev-bind / / --chdir /srv/NAS/Sunny/SourceCode/systemd/systemd --die-with-parent --tmpfs /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/run 
--tmpfs /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/tmp --proc /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/proc --dev /srv/NAS/
Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/dev --ro-bind /sys /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/sys --bind /srv/NAS/Sunny/Sou
rceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/passwd /etc/passwd --bind /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/group /etc/group --bind
 /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/shadow /etc/shadow --bind /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/gsha
dow /etc/gshadow --setenv PATH /tmp/mkosi-scriptslyqkji_7:/tmp/mkosi.pathx6vx9ic8:/srv/NAS/Sunny/SourceCode/systemd/systemd/build:/usr/my/sbin:/usr/my/bin:/usr/local/sbin:
/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --bind /tmp/mkosi-var-tmpv7jetwnx /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/var/tmp --ro-bind /srv/NAS
/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/machine-id /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/machine-id sh -c chmod 1777 
/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/tmp /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/var/tmp /srv/NAS/Sunny/SourceCode/s
ystemd/systemd/.mkosi.tmpql8j70uv/root/dev/shm && exec $0 "$@" || exit $? chroot /work/build-script
chroot: cannot change root directory to '/work/build-script': No such file or directory
‣ "chroot /work/build-script" returned non-zero exit code 125.
‣ + umount --no-mtab --lazy /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root
‣    (Cleaning up overlayfs)
‣     (Removing overlay whiteout files…)
Traceback (most recent call last):
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/__main__.py", line 19, in propagate_failed_return
    yield
  File "/usr/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/__main__.py", line 48, in main
    run_verb(args, presets)
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/__init__.py", line 2223, in run_verb
    build_image(args, config)
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/__init__.py", line 1767, in build_image
    run_build_script(state)
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/__init__.py", line 1868, in run_build_script
    bwrap(
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/run.py", line 336, in bwrap
    raise e
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/run.py", line 330, in bwrap
    result = run([*cmdline, *cmd], env=env, log=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/run.py", line 212, in run
    raise e
  File "/srv/NAS/Sunny/SourceCode/mkosi/mkosi/run.py", line 193, in run
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bwrap', '--dev-bind', '/', '/', '--chdir', '/srv/NAS/Sunny/SourceCode/systemd/systemd', '--die-with-parent', '--tmpfs', '/srv/NAS
/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/run', '--tmpfs', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/tmp', '--proc', '/srv/NAS/Sun
ny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/proc', '--dev', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/dev', '--ro-bind', '/sys', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/sys', '--bind', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/passwd', '/etc/passwd', 
'--bind', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/group', '/etc/group', '--bind', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j7
0uv/root/etc/shadow', '/etc/shadow', '--bind', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/gshadow', '/etc/gshadow', '--setenv', 'PATH', '/tmp/m
kosi-scriptslyqkji_7:/tmp/mkosi.pathx6vx9ic8:/srv/NAS/Sunny/SourceCode/systemd/systemd/build:/usr/my/sbin:/usr/my/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sb
in:/bin', '--bind', '/tmp/mkosi-var-tmpv7jetwnx', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/var/tmp', '--ro-bind', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/machine-id', '/srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/etc/machine-id', 'sh', '-c', 'chmod 1777 /srv/NAS/S
unny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/tmp /srv/NAS/Sunny/SourceCode/systemd/systemd/.mkosi.tmpql8j70uv/root/var/tmp /srv/NAS/Sunny/SourceCode/systemd/sys
temd/.mkosi.tmpql8j70uv/root/dev/shm && exec $0 "$@" || exit $?', 'chroot', '/work/build-script']' returned non-zero exit status 125.

@DaanDeMeyer
Copy link
Contributor Author

Yes, check the HACKING guide in systemd, it explicitly documents that systemd depends on the latest mkosi from git.

nosada added a commit to nosada/mkosi-files that referenced this issue Aug 26, 2023
@nosada
Support mkosi >= 15
8fa4d8f 
- Rename mkosi.default to mkosi.conf
- Add mkosi.repart to define root partition
- Add gzip to make sure generating locales
- Add snippet to mkosi.build / mkosi.postinst
  - See: systemd/mkosi#1729
@shuLhan
Copy link

shuLhan commented Oct 10, 2023

Hi @DaanDeMeyer ,

Sorry for my ignorance, I am new to mkosi. Why not make special scripts name, like "mkosi.prepare.chroot", "mkosi.build.chroot", and so on, that if its exist it will run inside chroot?

Adding (if [ "$container" != "mkosi" ]; then) seems like a hack to me. It make sense if this repository is a wrapper of mkosi, but we are in mkosi repository.

What is possible values for "$container" beside "mkosi" that make it need its only IF-condition? Just curious.

@DaanDeMeyer
Copy link
Contributor Author

@shuLhan See https://systemd.io/CONTAINER_INTERFACE/ which recommends container managers to set $container.

I also opened #1969 to implement your suggestion of running scripts with the ".chroot" extension in the image automatically.

@DaanDeMeyer DaanDeMeyer mentioned this issue Oct 11, 2023
Closed
@DaanDeMeyer DaanDeMeyer mentioned this issue Oct 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
announcement
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shuLhan @iam-TJ @DaanDeMeyer