Dear maintainer,

autopkgtest-build-podman(1) describes creating a container image
suitable for testing uploads targeting Debian unstable, from scratch,
like this:

    mmdebstrap --variant=minbase sid - \
      | autopkgtest-build-podman --release=sid --tarball=-

This is sufficient for 'autopkgtest-virt-podman sid' to allow you to
speak the autopkgtest protocol with a fresh environment, which is great.
(The context, as you have perhaps guessed, is tag2upload, where our
source package building tool uses the autopkgtest protocol to work in a
clean environment.)

However, there are no instructions about maintaining the container image
and the image store.  In particular:

1. How does one rebuild the image, say in a weekly cronjob?  Do you just
   run the same command again, and the new image will take precedence?

2. How do you prune ~/.local/share/containers/ to avoid it just building
   up forever?
   I found both 'podman image prune' and 'podman container prune'.
   It would be great if a recommendation could be documented.
   We can resort to just 'rm -rf ~/.local/share/containers' but that has
   the disadvantage of service downtime while the new image is built.

Thanks.

Sean writes:

To be completely clear, we would want a way to do this that will work
correctly even if there are concurrent uses of the (old) image.  Those
concurrent uses should not be disturbed.

Likewise, the cleanup operation ought not to disturb in-progress uses,
but that'll be easier since hopefully there's an operation that will
only want to delete "old and unused" things.

Thanks,
Ian.

I'm not sure that autopkgtest-{build,virt}-podman(1) is really the ideal
place for a podman tutorial, any more than -{build,virt}-schroot is the
place for a schroot tutorial: nothing here is specific to autopkgtest.

If autopkgtest is the only use of podman by a particular uid, then
everything it works with is essentially a disposable cache anyway,
so it's OK to clean up anything that is not in active use
(`podman container prune`, `podman image prune`). However, if the same
user is using podman for other things, they might have usage patterns
where that cleanup would be undesired (like if they are keeping a stateful
"pet" container around for interactive use with podman-toolbox,
in which case `podman container prune` would result in losing that state),
so it's difficult to give a one-size-fits-all recommendation.

Yes, that should work. The tags created by a-b-podman will be changed
to point to the newer image, and the old image will hang around in the
content-addressed storage with no name (addressable only by its hash)
until it is pruned.

If a podman image is like a git commit, then you can think of podman
tags as being like git refs (they can be used like either git tags or
git branches), and I think what you are looking for in your next question
is the equivalent of `git gc`?

`podman image prune` prunes unused images, which I believe means images
that have no tags pointing to them and also no container instances
running with them as the base (but I could be wrong about this, I am
not an expert on podman). If a container instance is like a schroot
session, then an image is like a stored schroot tarball: you can't enter
it directly, but you can create new container instances that have it as
their starting point. In my git-commit analogy, this is like `git gc`.

List images with `podman image list`. The ones that are listed with
repository and tag "<none>" are the ones that `podman image prune` will
try to delete.

`podman container prune` prunes container instances that exist but are
no longer running. A container instance is like a schroot session: it
has some on-disk resources, and maybe some running processes. List them
with `podman container list -a`. To continue my git commit analogy above,
you could think of a container instance as being like a git worktree with
a detached HEAD (including the checked-out source code, any uncommitted
edits, and any temporary build artifacts that might be in the tree), and
`podman container prune` is like removing all the worktrees that you're
no longer using.

When using autopkgtest-virt-podman, this command should rarely
(never?)  do anything in practice, because a-v-podman runs
`podman stop $container_id` and `podman rm -f $container_id`
during cleanup. `podman stop` kills the container's processes, and
`podman rm -f` cleans up its on-disk resources, similar to `schroot
--end-session`. However, there could be stale container instances if
a-v-podman was terminated uncleanly (a crash or an unexpected system
reboot or similar).

When using podman interactively, by default the container instance will
continue to hang around after you have exited from the interactive shell
or similar, until you explicitly remove it with `podman rm`. This can be
done automatically on container exit by creating the container instance
with `podman run --rm`. I'm actually not sure why a-v-podman doesn't
use `--rm` to make its cleanup more automatic.

A typical interactive podman invocation might be something like this:

    podman run --rm -it -v $HOME/tmp/exchange:/tmp/exchange:rw \
        localhost/autopkgtest/amd64/debian:sid

in order to avoid the container instance being kept indefinitely.
(This also demonstrates how to use -v/--volume to have a way to copy
artifacts in and out of the container.)

I believe `podman image prune` will avoid deleting images that are
currently in use as the base for a container, even if there is no longer
any tag pointing to the image. Certainly that seems to be true from a
brief experiment. Continuing the git commit analogy, this is like the way
`git gc` won't garbage-collect a commit that is checked out as a detached
HEAD in a worktree, even if it's no longer reachable from a branch or tag.

The tags that you use on the a-v-podman command-line are only aliases
(like a git branch or tag), and the "real" content of the image is in
content-addressed storage (like a git commit). I don't think the podman
CLI will let you delete it while it's still in use (certainly not without
applying some --force).

I believe the operation you want is exactly `podman image prune`, perhaps
combined with `podman container prune` if there might have been container
instances that were shut down uncleanly and are no longer wanted.

    smcv

Simon McVittie writes ("Re: Bug#1098352: autopkgtest-build-podman(1): Documentation on rebuilding images & pruning image store"):

Thanks, what an excellent reply.  On a quick reading, I think it
answers all our questions.

I'm sure others would find much of this information (including the
analogues, and the explanation of the model) very helpful.

I'd like to encourage you to find *somewhere* to put it that's not
just this bug report.  Maybe a file in /usr/share/doc, as part of the
podman package?

To be clear, I'm not asking you to do the work of *editing* it to be
more general.  It is very useful even as it is.  I'm sure it can be
improved later.  I'm encouraging you to take just the first step,
which is to put it somewhere where it can be referenced, and further
improved/refined/adapted including by others.

If you think that's a good idea, then we should reassign this bug to
src:podman, to represent the desire to make that change there.

Thanks,
Ian.

Hello,

I would propose adding the following text to the manpage:

    You can simply run this command again to replace the image with one
    that's freshly built.  Then use podman-image-prune(1) to clean up
    old images.  See podman-container-prune(1) if you need to clean up
    containers that didn't shut down properly.

That's a tiny addition and it would have helped this Podman beginner a
great deal.  I can prepare a formal patch if you agree, Simon.

I wrote this with a specific audience in mind, namely you and Sean
(git experts with a lot of schroot experience), and I don't think it's
general-purpose documentation. To a reader who does not have your git
expertise, the analogy to git would likely be more confusing than helpful,
and to a reader outside the Debian bubble, analogies to schroot are also
not going to be helpful.

Sorry, but I don't feel that what I wrote is in a useful form to be in
the podman package, and I would not feel comfortable asking the podman
maintainers to include it: adding some text describing one specific
use-case to the podman package is helpful right now for anyone who
has that exact use-case, but it is unlikely to be particularly
discoverable, will not be kept updated over time if things change, and
doesn't help podman users whose use-case is different.

A general-purpose tutorial on what is meant by images, container
instances, repositories and tags almost certainly already exists in the
podman documentation, and there are probably literally hundreds on the
internet (the concepts and terminology are the same as in the widely-used
Docker suite, and I'm only preferring Podman for autopkgtest purposes
because Docker requires a daemon and normally requires making its user
be root-equivalent).

Some sort of documentation about routine maintenance for a Docker/Podman
host system is probably also something that already exists, but I don't
know where specifically to look for it; and if it doesn't already exist
in the podman package, perhaps it should (ideally in the upstream podman
project rather than being Debian-specific). Sorry, I'm not willing to
volunteer to convert my stream-of-consciousness into formal documentation
that meets the podman maintainers' standards: the list of things in
Debian and open source that I'm required to feel guilty for not having
done is already considerably longer than my ability to clear it.

If you feel that I am wrong about this, anyone else is welcome to submit
a suitably edited version of that text to the podman maintainers, but
only if they make themselves and not me responsible for its content
and correctness. (Please consider what I wrote in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1098352#15 to be
licensed under CC0 or FSFAP, your choice.)

    smcv

My concern about this is the last sentence, and whether it's sufficiently
clear to a podman beginner that for some non-autopkgtest use-cases,
this would be data loss. Because I know you're a schroot expert, I
can use an analogy: this is a bit like calling `schroot --end-session`
on every open session that has not been resumed since the last boot.

If your use-case is that container instances (~= sessions) are essentially
disposable anyway (as in autopkgtest and many schroot use-cases) then
that's fine, but if your use-case is that some container instances carry
important state (as they might when using toolbox(1) in particular),
then pruning stopped instances destroys that state. podman cannot tell
which way you plan to use it.

    smcv

Hello,

Very fair.  How about adding that text without the final sentence?

Hello,

What you say is all reasonable, though, it was much better than a mere
stream-of-consciousness :)