Moby: No space left on machine (overlay)
I have docker running on CentOS Linux release 7.0.1406 (Core). Also I'm using overlay instead of devicemapper loopback and I have a few containers running on machine including docker registry container. After few days I had no space left on machine, therefore I do following:
[root@m13 lib]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 59G 53G 3.1G 95% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 153M 1.8G 8% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
tmpfs 395M 0 395M 0% /run/user/0
overlay 59G 53G 3.1G 95% /var/lib/docker/overlay/83b3f48705cf143225818f2cb8ef6a5e5a634a0915a719ee1c5cf8ed42212ff3/merged
overlay 59G 53G 3.1G 95% /var/lib/docker/overlay/2499db1fbf395c19d7de98a417ead5c9002d65ff58f6de14df5bc21506e68129/merged
shm 64M 0 64M 0% /var/lib/docker/containers/40c09a988385341d893fc19a02c49959023a992e5463d12a4b2c2c692b0b080a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/78ec95fecfb21b415fa46161a4ddd9465f4309d906789714e9095e11ac2d6445/shm
overlay 59G 53G 3.1G 95% /var/lib/docker/overlay/d48c155765212df5b4215c555b09e3e2c1a32869b0c628b4b5bdf829d34022e0/merged
shm 64M 0 64M 0% /var/lib/docker/containers/d00ffe0dbcebaf176d60000c7ed91135ace7026b49c2903abf7f7f5b6317e720/shm
overlay 59G 53G 3.1G 95% /var/lib/docker/overlay/54ff581382a6c6c58c7ffe40b9351aaaadbbd5b31acc9487446b3ab5c8bd6fbd/merged
shm 64M 0 64M 0% /var/lib/docker/containers/abc3ff5e5d8f3e60346f5187c464d3148cefce3604fe0716ca27b8d4cd03cb5f/shm
overlay 59G 53G 3.1G 95% /var/lib/docker/overlay/a66aa00a04f0d22a4f63c8bb1f168cf0e0c772a7ac5adeee4b46f4dbc52265b1/merged
shm 64M 0 64M 0% /var/lib/docker/containers/d524c1a653b9e2681738ca1cf5312b58ec31ca0ac2f6c9600081214f2e916e6b/shm
Here you can see that 53G is used by overlay. So I do following:
[root@m13 lib]# du -sh /var/lib/docker
33G /var/lib/docker
Where is additional 20G of space? Or maybe I missed something?
[root@m13 docker]# docker version
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
[root@m13 docker]# docker info
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.5
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.14.1.el7.centos.plus.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.856 GiB
Name: m13
ID: SMZF:64TM:BGWW:NU6U:M6TK:43F3:DHWW:RTFU:ZWIL:WDVV:MBRO:QAQQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
euqen
·
13 Jan 2017
All comments
It's easy for a registry to become so big. Just a question. Are you sharing a volume or mounting a directory into the registry that it's not into the Docker root dir? Where is your registry storing images?
Thanks
gianarb
·
13 Jan 2017
@gianarb I'm not sure, but I'm not sharing volume for registry. Here is what I did:
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/auth:/auth \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-e REGISTRY_STORAGE_DELETE_ENABLED=true \
registry
I'm not sure that it's registry's fault, I think it's can be something else ..
Also I forgot to say, that two containers are running in previleged mode, just to allow run docker into these containers. And yes, I checked these containers: there are no unused images or containers inside.
euqen
·
13 Jan 2017
overlay is reporting the space used on /
Notice how each overlay entry is reporting the same usage, which is also the same usage as /.
Presumably if du /var/lib/docker reports 33G, the rest is being used elsewhere.
Also note that the overlay graphdriver uses an excessive amount of inodes, which would also cause the system report "no space left on device" when inodes are exhausted.
cpuguy83
·
13 Jan 2017
Guys, this issue was resolved, the reason was that containers which are running in privileged mode has some unused volumes. So the following command helped me:
docker volume ls -qf dangling=true | xargs -r docker volume rm inside of each privileged container.
euqen
·
16 Jan 2017
is this the same as docker volume prune ?
I have the same issue but when I removed dangling volumes, it only freed up few mega bytes. What would be the issue here?
When I do df -h I get a big list of 49GB overlay volumes
ahamad37
·
16 Aug 2018
Most helpful comment
is this the same as
docker volume prune?I have the same issue but when I removed dangling volumes, it only freed up few mega bytes. What would be the issue here?
When I do
df -hI get a big list of 49GB overlay volumes