parted 3.x removes most of the ped_file_system_* entry points that
required extensive copies of file system code in libparted, with the
exception of some limited support for HFS+ and FAT resizing (see its
NEWS file for more details; facilities such as file system probing are
still available).  d-i still uses these for important functions.  In
most cases we fall back to external utilities, but if there are any
remaining places where we don't then we'll need to fix those.

However, simply falling back to external utilities is not a
functionally-equivalent option right now, because in general we have no
support for handling progress information from those utilities.  Most of
them have some kind of progress interface, usually requiring some
special option and designed for human consumption rather than machine
parsing.

Rather than the Sisyphean task of giving all the appropriate utilities
machine-parseable progress interface, I instead suggest that we study
the current progress interfaces and try to identify some common factors
that can be parsed heuristically (e.g. solid bar of characters;
"\-/|"-style spinner; percentage output), and write a wrapper utility
that can translate some of the common ones into debconf progress output.
I think this ought to be a single program in partman-base, since there
are only so many ways people have invented for console progress
interfaces and we don't want to keep reinventing the wheel.  We can then
use that to wrap various mkfs, resize, etc. utilities, and in the
process fix some ancient bugs about lack of progress feedback in various
operations since some cases haven't actually used libparted's file
system support for a while.

The relevant parted_server commands to grep for in various partman-*
modules for ideas on the kinds of utilities that need to be wrapped are:

  CREATE_FILE_SYSTEM
  CHECK_FILE_SYSTEM
  RESIZE_PARTITION

(Theoretically also COPY_PARTITION but we have no fallback for that; we
may have to either desupport this or reimplement it by hand somehow.)

I looked at doing this once but unfortunately I seem to have lost the
source code for my attempt; I'd only really made a start at it anyway.
I'm filing this bug as a memory aid, and because people keep asking me
what's happening with packaging of parted 3.x and I want to have
something reasonably coherent to point them to.

I'm confused.  My understanding is that partman was already using
external utilities for everything besides fat, hfs, and ext2 resize,
and that there were already bug reports covering the lack of progress
indication with those.  Parted3 appears to have moved the resize
functions for fat and hfs to a separate library, so partman should be
able to just link to the new library and be no worse off than before,
with the exception of ext2, and well, who really cares about that?

What you describe here seems to be targeted at fixing the lack of
progress indication already covered by other bugs.  While providing
progress indication is nice, I'm not sure why it is still blocking the
migration to parted3.

On Mon, 2012-12-17 at 00:11 +0000, Colin Watson wrote:
[...]
[...]

I have trouble believing that this is going to be easier than patching
the most common utilities to add an option for a machine-readable
progress indicator.

Ben.

Not too long ago I was looking at patching gparted to support this and
while it did not seem difficult to process the output of the e2fsprogs
utilities for the progress information, I was wondering why the
various fs utilities don't already have a flag to generate machine
readable output... and I think some of the utilities supported this,
and some did not, even within e2fsprogs.

IIRC, I found that at least some of them already have a flag to write
output to a specified auxiliary fd to indicate progress, it just
seemed that a standard needed decided on and implemented in all of them.

I have no such trouble; we're going to need a wrapper to translate into
the debconf protocol *anyway*, and having it understand a few different
syntaxes is not that troublesome.  IME getting the file descriptor
plumbing right on these things is usually the most fiddly part. :-)

Quite.  Of course we should use it where it exists.
them all independently anyway, so invoking them in different ways and
even handling different styles of output is not a problem.  Trying to
invent and enforce a standard just takes you down an unnecessary rathole
of months spent trying to convince maintainers to change their
machine-parseable output, any of whom might point out with good reason
that it's machine-parseable and therefore is not supposed to change
frivolously.

Say Colin, it has been over a year so I thought I would ping on this.
 Is this really the only issue that is still preventing the parted3
transition?  If so I think it may be time go go ahead and make the
move rather than continue to block on this, which really is a very
minor issue given that ext4 only takes seconds to format these days
thanks to lazy_itable_init.

Phillip, you've asked me about this a number of times.  I think the work
involved is fairly clear, although I don't seem to have managed to make
time for it personally.  Perhaps, rather than continuing to ask me about
this, you could have a go at the necessary d-i integration?  I don't
think it should be that hard really.

Could you point me in the direction of how to test with my locally
built parted3 udebs?  I tried using debian-installer and putting it in
the local directory but it is ignored I guess because it doesn't go in
the initrd, but is loaded later from the pool.  In the case of the
netinst image, it just downloads the official 2.3 udeb from the mirror
since there is no pool, and when I tried make build_cdrom_isolinux, it
just prepared the kernel and initrd in a directory rather than making
an iso.

You're nearly there; in addition to having it in build/localudebs/, you
also need to add it to some appropriate file in build/pkg-lists/.  For
example build/pkg-lists/local would do.

Or you could just build a monolithic image, which is an image type
intended for this kind of testing.

cdrom is mostly not a very convenient image type to use, since it needs
to be paired with debian-cd or similar to construct the actual CD.  Try
"make rebuild_netboot", or as I say "make rebuild_monolithic" to build
all plausible udebs into the image.