Podman

2026-05-06T21:01:33Z

Kris: Add Podman page!

Podman is available on Sapelo2 for researchers who need to run software distributed as [https://opencontainers.org/ OCI container images]. It can be useful when a project already provides a standard container image or when a workflow expects familiar Docker-style commands. [https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics Podman is provided in a rootless configuration] and behaves differently from a typical workstation or server installation. In particular, image storage and some runtime features are constrained by the cluster's file system and stateless node environment. This guide explains the main differences, why they exist, and the practical limitations users should expect when running Podman on compute nodes.

== Podman in Slurm jobs ==
Podman depends on <code>XDG_RUNTIME_DIR</code> (typically <code>/run/user/$UID</code>) for rootless operation. On Sapelo2, Slurm jobs are not associated with a user systemd session, so this directory is '''not created automatically'''.

As a result, Podman will fail in Slurm jobs unless a user session is created. Here is an example error from an interactive job:<syntaxhighlight lang="shell-session">
Failed to obtain podman configuration: lstat /run/user/1337: no such file or directory
</syntaxhighlight>

=== Workaround ===
'''''Important:''' Podman will not work in Slurm jobs unless this step is performed.''

Before running any Podman commands inside a Slurm job, start a background SSH session to localhost<syntaxhighlight lang="shell-session">
ssh -N localhost >/dev/null 2>&1 &
</syntaxhighlight>This creates a user session in the background and initializes <code>/run/user/$UID</code>
This workaround '''requires passwordless SSH access''' to your own account. You must have

* An SSH keypair in <code>~/.ssh/</code>

* Your public key added to <code>~/.ssh/authorized_keys</code>

Test with:<syntaxhighlight lang="shell-session">
ssh localhost
</syntaxhighlight>'''This should not prompt for a password.'''

This works because SSH creates a PAM/systemd user session, which initializes <code>/run/user/$UID</code>. The session remains active as long as the SSH process is running.

Because the SSH process runs in the background of the job, ''it is automatically cleaned up when the job exits''.

==Filesystem limitations==
On Sapelo2, Podman images are effectively '''node-local''', not cluster-global like <code>/apps</code>. Rootless Podman normally expects local filesystem behavior for its image storage, but NFS-backed <code>/home</code> and Lustre-backed <code>/scratch</code> do not provide that behavior. As a result, Podman is configured to use local scratch storage on the compute node where the image is pulled or loaded.

In practice, this means an image you pull on one compute node will not automatically be available on another. Researchers should expect to pull or load images again when moving between nodes.

===Transporting images with <code>podman save</code> and <code>podman load</code>===
A practical workaround is to save an image as a tar archive on shared storage, then load it into Podman on the node where you want to run it. While this does not make images globally shared, it does make them quicker to move between nodes without re-downloading from a registry on each node.<syntaxhighlight lang="shell-session">
podman pull ubuntu
podman save -o ubuntu.tar ubuntu
podman load -i ubuntu.tar
</syntaxhighlight>

==NVIDIA Container Toolkit==
The NVIDIA Container Toolkit is the layer that enables Podman to expose NVIDIA GPUs inside containers. A major feature of the toolkit is CDI, the Container Device Interface. CDI lets containers request GPUs using stable, predictable names such as <code>nvidia.com/gpu=0</code>, <code>nvidia.com/gpu=1</code>, or even <code>nvidia.com/gpu=all</code>. On a heterogeneous cluster, where GPU types and counts may differ across nodes, this is much simpler than manually listing <code>/dev/nvidia*</code> devices and host library mounts for each container.

A typical CDI-based Podman command looks like this:<syntaxhighlight lang="shell-session">
podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi
</syntaxhighlight>

===CUDA validation example===
A practical way to check that CUDA is working inside a container is to run a small PyTorch test rather than relying only on GPU enumeration. In this example, PyTorch reports that CUDA is available, allocates a tensor, and runs a simple arithmetic operation.

'''''Note:''' Due to a known issue with the NVIDIA CDI <code>update-ldcache</code> hook on Sapelo2 (see below), this example includes a manual bind mount of <code>libcuda.so.1</code>, which would not normally be required.''<syntaxhighlight lang="shell-session">
$ podman run --rm -it \
--device nvidia.com/gpu=all \
-v /usr/lib64/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro \
docker.io/pytorch/pytorch:2.9.0-cuda12.8-cudnn9-runtime \
python3
Trying to pull docker.io/pytorch/pytorch:2.9.0-cuda12.8-cudnn9-runtime...
[output truncated]
Writing manifest to image destination
Python 3.11.14 | packaged by conda-forge | (main, Oct 13 2025, 14:09:32) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> (torch.tensor([1.0], device="cuda") * 2).item()
2.0
>>> torch.cuda.get_device_name(0)
'NVIDIA L4'
>>> quit()
$

</syntaxhighlight>

=== Known Issues ===

==== Broken <code>update-ldcache</code> OCI hook ====
The specific hook causing trouble is the NVIDIA CDI <code>update-ldcache</code> hook. In the generated CDI configuration, that hook appears as a <code>createContainer</code> hook that invokes <code>/usr/bin/nvidia-cdi-hook</code> with the <code>update-ldcache</code> argument. When Podman tries to start a container with CDI enabled, this hook can fail preventing container startup. The <code>update-ldcache</code> hook is meant to help make host NVIDIA libraries visible inside the container by updating the dynamic linker cache path used by the containerized environment. In a more typical setup, this helps ensure that GPU-related executables inside the container can locate the right shared libraries without extra manual bind mounts. When the <code>update-ldcache</code> hook fails, CDI may still identify the GPUs correctly, but the automatic library setup path is incomplete. In practice, that means users may need to manually bind-mount some host libraries or utilities into the container instead of relying on the hook to make everything available automatically.

Due to the issues, the <code>update-ldcache</code> OCI hook has been disabled in CDI. While this avoids container startup failures, it also bypasses the mechanism that updates the container’s dynamic linker cache with host NVIDIA library paths. Consequently, GPU-related libraries may not be discovered automatically inside the container, requiring some manual bind mounts or environment configuration to ensure proper runtime behavior.

This effect is clearly seen in the CUDA validation example above, where <code>libcuda.so.1</code> is explicitly bind-mounted (<code>-v /usr/lib64/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro</code>) despite CDI being enabled via <code>--device nvidia.com/gpu=all</code>

Research Computing Center Wiki - User contributions [en]

Podman