#1123959 manpages: Please consider shipping uncompressed man pages

Package:
manpages
Source:
manpages
Submitter:
Helge Kreutzmann
Date:
2025-12-25 14:35:03 UTC
Severity:
normal
#1123959#5
Date:
2025-12-25 06:07:57 UTC
From:
To:
Hello Tobias,
if you look at mansect(1), the example given does not work in Debian.
I reported this upstream and got the following reply:

Could you consider this?

Less preferred would be a Debian patch to the man pages, stating what
a Debian user would have to do to achieve the purpose of mansect(1).
#1123959#10
Date:
2025-12-25 11:20:25 UTC
From:
To:
Hello Helge, Tobias,

Thanks!

Indeed, compressed manual pages are a pain to work with.  You can't use
regular Unix tools to work with them.  With uncompressed manual pages,
You can go to /usr/share/man, and run a pipe of programs to do a complex
search.  With tools like zgrep(1) and zcat(1), you can do some stuff,
but not everything.

A couple of years ago, I did some measurements, which should be relevant
now:
<https://lore.kernel.org/linux-man/c8cf5be0-04e7-f0a1-179f-eada6182c33e@gmail.com/T/#m272e6ee8939d0836999dd8bb28f2e0e94f48dfc7>

I'll paste some of it:

	$ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-1  | wc -l
	2571
	$ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-9  | wc -l
	2571
	$ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z=                   | wc -l
	2571

	$ du -sh /opt/local/man/*
	5.7M	/opt/local/man/gz__1
	5.5M	/opt/local/man/gz__9
	5.5M	/opt/local/man/gz___
	9.4M	/opt/local/man/man__

	$ export MANPATH=/opt/local/man/gz__1/share/man
	$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
	17; 0.21
	$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
	17; 1.16

	$ export MANPATH=/opt/local/man/gz__9/share/man
	$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
	17; 0.20
	$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
	17; 1.17

	$ export MANPATH=/opt/local/man/man__/share/man
	$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
	17; 0.55
	$ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
	17; 0.01

As you can see, running a pipeline directly in $MANPATH is orders of
magnitude faster with uncompressed pages.  And doing global apropos is
still in the same order of magnitude, even if a bit slower (I suspect it
is actually running zgrep(1) on uncompressed pages, so it probably could
be faster).

Also, if man-db's man(1) could eventually remove the optimizations done
for gzip, we'd get a much simpler (safer) librarian program suite.

Indeed.


Have a lovely day!
Alex
#1123959#15
Date:
2025-12-25 13:47:33 UTC
From:
To:
Am 25.12.25 um 12:20 schrieb Alejandro Colomar:

Hi Helge and Alex,

thanks for your bug report and the provided statistics. I haven't
thought about this up until now, because it violates Debian Policy.
Quoting from Section 12.1
(https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):

"Manual pages should be installed compressed using gzip -9."

And regarding the terminology using the word "should", this is defined
in section 1.1
(https://www.debian.org/doc/debian-policy/ch-scope.html#scope):

"The terms should and should not, and the adjective recommended, denote
best practices. Non-conformance with these guidelines will generally be
considered a bug, but will not necessarily render a package unsuitable
for distribution. These statements correspond to bug severities of
important, normal, and minor. They are collectively called Policy
recommendations."

So by not compressing the man pages, the Debian package would introduce
a bug. Moreover, I'd have to explicitely opt out of automatic
compression in the build stage of the package.

All of this is doable, of course. But I'm a bit hesitant with just
making the switch, given that the manpages package is certainly the
package with the most man pages in the Debian ecosystem -- by a large
margin.

So it might be better to discuss the pros and cons in a broader
audience, trying to understand why the compression has been chosen
initially. Maybe only due to disk space limitations back then, but maybe
there are other reasons as well -- which might still be valid today.

Regards,
Tobias
#1123959#20
Date:
2025-12-25 14:15:03 UTC
From:
To:
Hi,

Yup, I'd like that policy to change.  I've added debian-policy@ to this
mail (and also linux-man@).

For those reading only since this email, please have a look at
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1123959>, which
itself also references an discussion in the Linux man-pages project from
a couple of years ago:
<https://lore.kernel.org/linux-man/c8cf5be0-04e7-f0a1-179f-eada6182c33e@gmail.com/T/#m272e6ee8939d0836999dd8bb28f2e0e94f48dfc7>.

I'll paste again the numbers:

        $ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-1  | wc -l
        2571
        $ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-9  | wc -l
        2571
        $ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z=                   | wc -l
        2571

        $ du -sh /opt/local/man/*
        5.7M    /opt/local/man/gz__1
        5.5M    /opt/local/man/gz__9
        5.5M    /opt/local/man/gz___
        9.4M    /opt/local/man/man__

        $ export MANPATH=/opt/local/man/gz__1/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.21
        $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
        17; 1.16

        $ export MANPATH=/opt/local/man/gz__9/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.20
        $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
        17; 1.17

        $ export MANPATH=/opt/local/man/man__/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.55
        $ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.01

Using uncompressed manual pages is both faster and simpler, by orders of
magnitude, when doing complex searches with pipelines.  In the simple
cases where man(1) is enough, the speed is in the same order of
magnitude.

Also, the compression only cuts storage by half, so not even an order of
magnitude.  In this age, where storage is relatively cheap, systems that
have manual pages installed most likely have room enough for the
uncompressed pages.

Please change Debian policy 12.1 ("Manual pages") to recommend
uncompressed pages.


Have a lovely day!
Alex