07-05
Pinned
Activity
x-yuri push x-yuri/rfcount
commit sha: b00902bc7ddd42664c38f39d8c735d2a1e9d9212
push time in 1 week agox-yuri push x-yuri/rfcount
commit sha: 297d61280fe41d8a995ec236d3a070865bad9ab4
push time in 1 week agox-yuri push x-yuri/rfcount
commit sha: c435598c0b3ed65c86deb9580e768e335d1abdf8
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: bbe8174e910d2674d9889a4c40a9b0a822c6288a
push time in 2 weeks agox-yuri issue comment moby/moby
Unable to run systemd in docker with ro /sys/fs/cgroup after systemd 248 host upgrade
BUG REPORT INFORMATION
I used to run docker containers with systemd as CMD without having to expose /sys/fs/cgroup as rw; this worked until systemd 248 on the host. Now it fails with
Failed to create /init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
I opened a related issue on the systemd github repo: https://github.com/systemd/systemd/issues/19245
Workarounds
- boot host with systemd.unified_cgroup_hierarchy=0
- remove ro flag from docker run arg -v /sys/fs/cgroup:/sys/fs/cgroup:ro but this contaminates the host cgroup, causing e.g. docker top to get confused:
docker top debian-systemd
Error response from daemon: runc did not terminate successfully: container_linux.go:186: getting all container pids from cgroups caused: lstat /sys/fs/cgroup/system.slice/docker-817dfec3facbeb10c64d7b0fae478804b1177ae949e695e111b7c693569dd21a.scope: no such file or directory
: unknown
Steps to reproduce the issue:
Dockerfile:
FROM debian:buster-slim
ENV container docker
ENV LC_ALL C
ENV DEBIAN_FRONTEND noninteractive
USER root
WORKDIR /root
RUN set -x
RUN apt-get update -y \
&& apt-get install --no-install-recommends -y systemd \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
&& rm -f /var/run/nologin
RUN rm -f /lib/systemd/system/multi-user.target.wants/* \
/etc/systemd/system/*.wants/* \
/lib/systemd/system/local-fs.target.wants/* \
/lib/systemd/system/sockets.target.wants/*udev* \
/lib/systemd/system/sockets.target.wants/*initctl* \
/lib/systemd/system/sysinit.target.wants/systemd-tmpfiles-setup* \
/lib/systemd/system/systemd-update-utmp*
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/lib/systemd/systemd"]
Expected behaviour
systemd 247 (247.4-2-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
$ docker build -t debian-systemd .
$ docker run -t --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro debian-systemd
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 10 (buster)!
Set hostname to <bf431002c7c1>.
Couldn't move remaining userspace processes, ignoring: Input/output error
File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ OK ] Listening on Journal Socket.
...
[ OK ] Reached target Graphical Interface.
Actual behaviour
Since systemd v248
$ /lib/systemd/systemd --version
systemd 248 (248-3-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
$ docker build -t debian-systemd .
$ docker run -t --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro debian-systemd
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 10 (buster)!
Set hostname to <fbb4fc19cb95>.
Failed to create /init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
Output of docker version
:
$ docker version
Client:
Version: 20.10.5
API version: 1.41
Go version: go1.16
Git commit: 55c4c88966
Built: Wed Mar 3 16:51:54 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.16
Git commit: 363e9a88a1
Built: Wed Mar 3 16:51:28 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info
:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)
Server:
Containers: 10
Running: 1
Paused: 0
Stopped: 9
Images: 61
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.11.11-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.712GiB
Name: homepc
ID: 67YO:62DZ:3NIF:TZT3:HTXP:BU6I:YBR3:XETA:7YCB:YGNN:MV6Q:QYN4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
x86_64 Intel hw, Arch Linux 5.11.11-arch1-1
Actually for now I'm planning to employ the hybrid/legacy systemd mode (cgroup v1), which seems tolerable in my case. But podman
sounds like an interesting option (haven't tried it).
x-yuri push x-yuri/rfcount
commit sha: c94b11fba150385cc7763cec4040de1c4154a153
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: a07c38878642b7d81ee32c70a30a21bd92b6ebbe
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: 905667085da1d0f268222fa63917c6a0ecae975d
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: d68fc65b8491684f89cb44fe6ac18695e09d79b8
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: 591c5bb7fcc46aefaa9d8a981edced3255e54e21
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: 6f22173b5505fda6218bda3f2aa0597a6a54c9a2
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: c1f8a723f857a1d027dceb7b24a527e64551e0f3
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: 2dd610ac04920b3a78dc26ebb734779f4e16e029
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: e4f4406df4c1b0314579bafc546bf4c92db5b686
push time in 2 weeks agox-yuri push x-yuri/rfcount
commit sha: b081925643e346c44fe0612b2dd0de0a56b41ed0
push time in 2 weeks agox-yuri issue comment moby/moby
Unable to run systemd in docker with ro /sys/fs/cgroup after systemd 248 host upgrade
BUG REPORT INFORMATION
I used to run docker containers with systemd as CMD without having to expose /sys/fs/cgroup as rw; this worked until systemd 248 on the host. Now it fails with
Failed to create /init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
I opened a related issue on the systemd github repo: https://github.com/systemd/systemd/issues/19245
Workarounds
- boot host with systemd.unified_cgroup_hierarchy=0
- remove ro flag from docker run arg -v /sys/fs/cgroup:/sys/fs/cgroup:ro but this contaminates the host cgroup, causing e.g. docker top to get confused:
docker top debian-systemd
Error response from daemon: runc did not terminate successfully: container_linux.go:186: getting all container pids from cgroups caused: lstat /sys/fs/cgroup/system.slice/docker-817dfec3facbeb10c64d7b0fae478804b1177ae949e695e111b7c693569dd21a.scope: no such file or directory
: unknown
Steps to reproduce the issue:
Dockerfile:
FROM debian:buster-slim
ENV container docker
ENV LC_ALL C
ENV DEBIAN_FRONTEND noninteractive
USER root
WORKDIR /root
RUN set -x
RUN apt-get update -y \
&& apt-get install --no-install-recommends -y systemd \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
&& rm -f /var/run/nologin
RUN rm -f /lib/systemd/system/multi-user.target.wants/* \
/etc/systemd/system/*.wants/* \
/lib/systemd/system/local-fs.target.wants/* \
/lib/systemd/system/sockets.target.wants/*udev* \
/lib/systemd/system/sockets.target.wants/*initctl* \
/lib/systemd/system/sysinit.target.wants/systemd-tmpfiles-setup* \
/lib/systemd/system/systemd-update-utmp*
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/lib/systemd/systemd"]
Expected behaviour
systemd 247 (247.4-2-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
$ docker build -t debian-systemd .
$ docker run -t --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro debian-systemd
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 10 (buster)!
Set hostname to <bf431002c7c1>.
Couldn't move remaining userspace processes, ignoring: Input/output error
File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ OK ] Listening on Journal Socket.
...
[ OK ] Reached target Graphical Interface.
Actual behaviour
Since systemd v248
$ /lib/systemd/systemd --version
systemd 248 (248-3-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
$ docker build -t debian-systemd .
$ docker run -t --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro debian-systemd
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 10 (buster)!
Set hostname to <fbb4fc19cb95>.
Failed to create /init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
Output of docker version
:
$ docker version
Client:
Version: 20.10.5
API version: 1.41
Go version: go1.16
Git commit: 55c4c88966
Built: Wed Mar 3 16:51:54 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.16
Git commit: 363e9a88a1
Built: Wed Mar 3 16:51:28 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info
:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)
Server:
Containers: 10
Running: 1
Paused: 0
Stopped: 9
Images: 61
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.11.11-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.712GiB
Name: homepc
ID: 67YO:62DZ:3NIF:TZT3:HTXP:BU6I:YBR3:XETA:7YCB:YGNN:MV6Q:QYN4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
x86_64 Intel hw, Arch Linux 5.11.11-arch1-1
remove ro flag from docker run arg -v /sys/fs/cgroup:/sys/fs/cgroup:ro
It didn't help. I'm running Ubuntu 21.10 (Impish Indri).
For reference, it is possible with namespace isolation.
@skast96, it didn't help either. I edited /etc/docker/daemon.json
:
{"userns-remap": "default"}
Restarted docker
. The dockremap
user was created, as were the entries in /etc/sub{uid,gid}
. The /var/lib/docker/100000.100000
dir was created. docker image ls
produced no output. Then:
$ docker run -it --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup jrei/systemd-ubuntu
systemd 245.4-4ubuntu3.16 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Ubuntu 20.04.4 LTS!
Set hostname to <1bdd4443336d>.
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
So the only workaround is supposedly to switch to the cgroup v1 mode (systemd.unified_cgroup_hierarchy=0
):
-
/etc/default/grub
:
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0"
-
update-grub
- reboot
x-yuri push x-yuri/rfcount
commit sha: 1f406501b86e7af7a3144dd9bc4d3ed81a044e39
push time in 3 weeks agox-yuri push x-yuri/rfcount
commit sha: 30b704dbf5d5a86f0e6267f49ccd9638b85ae36d
push time in 3 weeks agox-yuri push x-yuri/rfcount
commit sha: ccbce3dd29f8b6b73cc8f72180e20fe795cb4e35
push time in 3 weeks agox-yuri push x-yuri/rfcount
commit sha: ca66c90e7a12d5efd382d927f0e50cf1b68f8865
push time in 3 weeks agox-yuri push x-yuri/rfcount
commit sha: 0f472085ef8afe933d31a9395fda02c2e96fe66b
push time in 3 weeks agox-yuri push x-yuri/rfcount
commit sha: 8c0e75d194d17ca6778548989887e16bac5320f4
push time in 4 weeks agox-yuri push x-yuri/rfcount
commit sha: cc192724a31ba4cc6a71599e07094af3580f7c60
push time in 1 month agox-yuri push x-yuri/rfcount
commit sha: cd004c34b15ee54c986bc03800073fe88da6d169
push time in 1 month agox-yuri push x-yuri/rfcount
commit sha: 4bf4c4c97adfd19aa78c951725430d3b5c79284d
push time in 1 month agox-yuri push x-yuri/rfcount
commit sha: b59e6a0ba23f1f00d088e17f34adc546e2a13e96
push time in 1 month agox-yuri push x-yuri/rfcount
commit sha: 32a4673d0a2eb410a7654f08eea57edec940cb22
push time in 1 month agox-yuri push x-yuri/rfcount
commit sha: dcb7a6dd2d6d2b4c4fe381e9df4b89d96baa96a6
push time in 1 month ago
10-05