When Images Fill Up Your Home Directory

In one line

Images of rootless podman pile up in ~/.local/share/containers/storage. If the home partition is small, moving the graph root becomes the first operations task.

Why this was needed

Server partition designs like this are common.

/          50G
/home      20G      <- rootless podman 의 이미지가 여기 쌓인다
/data      2T       <- 정작 큰 디스크는 여기

A single CUDA image is 8GB. Pulling just a couple fills the home partition. And when home fills up, not only podman but everything that user does stops.

How it works

What to move

Item	Default path (rootless)	Should it move?
`graphroot`	`~/.local/share/containers/storage`	Yes. Image and container layers
`runroot`	`/run/user/<uid>/containers`	No. tmpfs is correct
Volumes	`<graphroot>/volumes`	They follow when you move graphroot
Configuration	`~/.config/containers`	No. It is small

Procedure

# 1. 현재 상태 확인
podman info --format '{{.Store.GraphRoot}}'
du -sh ~/.local/share/containers/storage

# 2. 새 경로 준비 (소유자가 그 사용자여야 한다)
sudo mkdir -p /srv/podman/podster
sudo chown podster:podster /srv/podman/podster
sudo chmod 700 /srv/podman/podster

# 3. 설정 변경
#    ~/.config/containers/storage.conf
#    [storage]
#    graphroot = "/srv/podman/podster"

# 4. 검증
podman info --format '{{.Store.GraphRoot}}'
podman images

The important point is that after step 3 the new store is empty. The existing images stay on the old path, and podman no longer looks at them. You have two choices.

Pull again — the simplest option if you have internet access.
Move them — create a tar with podman save and run podman load in the new store. Or copy the whole directory (cp -a or rsync -aHAX). A whole-directory copy must preserve SELinux labels and hard links, so you must give the options exactly.

In an air-gapped network, only the latter is possible. And before you delete the old path, make sure podman images on the new path shows the list you expect.

Setting it system-wide

To apply the same policy to several users, write it in /etc/containers/storage.conf. However, a rootless user's personal configuration overrides it, so if standardization is the goal, distribute the personal configuration file or put it in the home skeleton (/etc/skel).

Commands to check

podman info --format '{{.Store.GraphRoot}}'
podman info --format '{{.Store.RunRoot}}'
podman info --format '{{.Store.GraphDriverName}}'
podman system df

podman system df shows how much the images, containers, and volumes each use. It is the first command when choosing what to clean up.

Why storage causes trouble in rootless mode

Most storage problems unique to rootless Pods come from the user namespace and file ownership. Once you know the principle, the symptoms read immediately.

A UID inside the container is a different number on the host. It is shifted by the range written in /etc/subuid. Root (0) inside the container is my UID on the host, and 1000 inside the container is subuid 시작 + 999 on the host (the subuid start value plus 999). That is why the owner of that file looks like a very large number when you view it from the host.

grep "^$USER:" /etc/subuid /etc/subgid
podman unshare cat /proc/self/uid_map

If you mount a host directory, the permissions do not match. A file I created is owned by me on the host, but it appears as nobody inside the container. To change ownership, you must run chown inside that namespace.

podman unshare chown -R 1000:1000 ./data

If you run sudo chown without podman unshare, it is correct only from the host's point of view and still wrong in the container.

What catches you when moving the graph root. If the new location is on a different filesystem, first check whether it supports overlay. overlay does not work on NFS, and then it falls back to vfs, so disk usage grows several times over and it gets slow.

podman info --format '{{.Store.GraphRoot}} {{.Store.GraphDriverName}}'

On distributions with SELinux, you must attach labels. If you do not add :Z to a volume mount, the container cannot read it. :z is a shared label and :Z is a private label.

Clean up what is left regularly. Rootless piles up under the user's home, so the home partition fills easily. Look at temporary images, stopped containers, and unused volumes together.

podman system df
podman system prune --volumes --filter until=168h

What it looks like in the field

You run into this especially often on GPU nodes. CUDA base images are several GB, so pulling just a few fills the home partition. That is why the standard configuration of a GPU server often includes an entry that sends graphroot to a data disk.

Never put graphroot on an NFS home. Overlay-family drivers do not work properly, and file locking problems cause strange failures. It must be a local disk.

Using double the disk because the old path was not deleted. Once you have moved and verified, you must clean up the old path. Note that podman system reset deletes the currently configured store, so you cannot use it to clean up the old path.

What you will do in the next lab

You move podster's graph root to a new path, confirm that the new store is empty, fill it by loading an image archive, and do the final verification with podman info.