#551638 dselect: track seen packages in a new frontend-unified state file

Package:
dselect
Source:
dpkg
Description:
Debian package management front-end
Submitter:
Petr Vandrovec
Date:
2010-05-06 13:33:09 UTC
Severity:
important
#551638#5
Date:
2009-09-06 17:35:07 UTC
From:
To:
Since today's update run dselect is completely confused and does not
remember which packages are new and which are old - it lists all
available and not installed packages as new, again and again.  And not
only that, it also marks some of them for installation although I do not
want them (and some are not installable at all - for some reason it
wants to install libsasl2 which is not installable together with
libsasl2-2).  I see that it updates available & status, they seem to
contain valid data, but on load it somehow ignores them.

Besides that it creates some strange /var/lib/dpkg/updates/tmp.i which
is full of lines saying '##padding', but other than that I do not see
what can be wrong, and unfortunately diff between 1.15.3.1 and 1.15.4 is
quite huge.

Thanks, Petr

#551638#10
Date:
2009-09-07 05:57:40 UTC
From:
To:
Same problem here.
I had to use aptitude to see new packages.

Regards,
Romain

#551638#15
Date:
2009-09-07 07:11:45 UTC
From:
To:
You can use "git bisect" to isolate the problem more precisely. My bet
would be on the code that auto-cleans up the status database, maybe it
applies by error on the available file as well and thus it believes
that all packages are new everytime?

Cheers,

#551638#20
Date:
2009-09-07 07:43:37 UTC
From:
To:
Certainly.  FWIW, the problem occurs also if only dpkg is upgraded to
1.15.4 and dselect stays at 1.15.3.1.

Given that the available package is as big as ever and dselect shows
every package as new even after you run "sync-available" (from the
dctrl-tools package), this does not seem to be the case.

Sven

#551638#25
Date:
2009-09-07 16:13:01 UTC
From:
To:
Raphael Hertzog napsal(a):
commit 224f0285...
Author: Guillem Jover <guillem@debian.org>
Date:   Sun Jul 12 20:11:53 2009 +0200

    Obsolete --forget-old-unavail

    On parse mark not-installed leftover packages for automatic removal from
    the database on next dump ...

    Closes: #333394, #429262


And while we are on it, checkin 7fa96f35... (disable default automake
paths) causes FTBFS on my box.
							Petr

#551638#30
Date:
2009-09-08 07:28:06 UTC
From:
To:
Hi!

Hrmmmf, ok missed that one when doing the automatic forget change on
dpkg...

The installability problems I assume is due to some transition going
on in unstable right now.

The “tmp.i” file is normal, yes.

The problem is that dselect used the status file to track not seen
and seen not-installed packages as either want_unknown or want_purge
want states. And was setting all want_unknown packages to want_purge
on normal exit (not using X keyibindig).

This increases the status file, the parsing and processing time when
doing dependency resolutions and package iterations for everyone, and
just to be able to show new packages on dselect. The correct solution
here is not to revert the change, but to store the seen/not-seen
information in another place, in a similar way as how apt/aptitude do
it. And ideally in a unified place which all front-ends can share, so
we avoid duplication.

I'll put it on the pending stuff to discuss with the front-end
developers.

regards,
guillem

#551638#35
Date:
2009-09-08 15:33:59 UTC
From:
To:
Guillem Jover napsal(a):

IMHO change should be reverted until old functionality can be achieved
through some other way - unless you can achieve such functionality today.
						Petr

#551638#40
Date:
2009-09-09 23:08:48 UTC
From:
To:
Yes please, revert that commit. dselect is currently unusable.

regards,
Domenico
-----[ Domenico Andreoli, aka cavok --[ http://www.dandreoli.com/gpgkey.asc ---[ 3A0F 2F80 F79C 678A 8936 4FEE 0677 9033 A20E BC50
#551638#45
Date:
2009-09-10 07:03:07 UTC
From:
To:
severity 545366 serious
thanks

I'd like to remind everybody that we're looking for someone to step up and
maintain dselect:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=282283

Here's your chance instead of requesting an instant fix from people who
would rather not have to maintain dselect as part of dpkg.

I'll mark this bug as RC so that it doesn't get into testing, you're free
to downgrade dpkg in the mean time.

Cheers,

#551638#52
Date:
2009-09-28 05:46:48 UTC
From:
To:
there is no  /var/cache/apt/available

partial output of dselect update ...
Replacing available packages info, using /var/cache/apt/available.

man dselect
dselect --admindir <directory>
Changes  the  directory  where  the  dpkg `status',`available' and
similar files are located. This defaults to /var/lib/dpkg and normally
there shouldn't be any  need  to change it.

man dpkg
dpkg --update-avail, --merge-avail Packages-file
Update  dpkg's  and  dselect's  idea  of  which  packages are available.
With action --merge-avail, old information is combined with information
from Packages-file.  With action  --update-avail, old information is
replaced with the information in the Packages-file. The Packages-file
distributed with Debian is simply named Packages. dpkg keeps its record
of available packages in /var/lib/dpkg/available.

hth
really
ctl
aka jetscreamer

#551638#57
Date:
2009-10-01 14:39:24 UTC
From:
To:
even downgrading to the version of dselect in lenny does not fix this
behavior. therefore (obviously?) this is not a problem in dselect but
something else? did somebody change dpkg's defaults? beyond me. :(

hope dselect gets fixed soon though. i really don't like aptitude.
nor anything else...

tia
ctl

#551638#62
Date:
2009-10-09 05:16:34 UTC
From:
To:
don't know if this helps, but when i tried cp'ing available to
/var/cache/apt, when i do dselect update, /var/cache/apt/available gets
deleted.

also, cp'ing var/lib/dpkg to /var/cache/apt then setting admindir in the
command line didn't work, nor setting it in /etc/dpkg/*.conf. still deleted.

setting admindir manually in console and also in the conffiles to the
default location didn't work either. :(


anyway
ctl

#551638#77
Date:
2009-10-20 23:46:15 UTC
From:
To:
Today I installed dpkg, dpkg-dev and dselect 1.15.4.1.
I hoped that the error is fixed, but he is still there.
I use _only_ dselect, because I love it!
Can you please fix it?

Sandro

#551638#82
Date:
2009-10-21 06:11:54 UTC
From:
To:
What is precisely the error that you're seeing?

Cheers,

#551638#87
Date:
2009-10-21 11:35:50 UTC
From:
To:
Raphael Hertzog schrieb:
The "original" error:
Since today's update run dselect is completely confused and does not
remember which packages are new and which are old - it lists all
available and not installed packages as new, again and again.  And not
only that, it also marks some of them for installation although I do not
want them (and some are not installable at all - for some reason it
wants to install libsasl2 which is not installable together with
libsasl2-2).  I see that it updates available & status, they seem to
contain valid data, but on load it somehow ignores them.

Sandro

#551638#92
Date:
2009-10-25 10:40:41 UTC
From:
To:
Version: 1.15.4.1

That's what we wanted. Closing this bug again. The fact that it doesn't
keep track of new packages is tracked in #551638 and is not release
critical for us.

Please use the other bug for further discussion.

Cheers,

#551638#97
Date:
2009-10-25 14:45:16 UTC
From:
To:
Raphael Hertzog <hertzog@debian.org>:

While the change was, indeed, intended, it doesn't change the fact
that it does nothing to fix the problem.

#551638#102
Date:
2009-12-08 18:21:03 UTC
From:
To:
your fix changed nothing, everything is still marked new. you were wrong.
downgrading to the lenny version changed nothing, it doesn't work
either. once again, you are wrong.

you broke it yet refuse to fix it. this seems to be becoming typical.
i suggest you take dselect out if you continue in this vein.
oh... it's tied too closely to dpkg you say. we can't fix it.
so you will continue to release buggy non-functional stable releases of
this distro.

re Disk devices may change on reboot, as seen on
http://www.debian.org/releases/stable/debian-installer/...
i found this bug back when the sid kernel was 2.6.13 or so, yet it's
still not fixed. afaict, it won't be. yet this is truly a show stopper.
i have no idea what the bug number is, nor do i care anymore.

i find it TRULY a shame that i've wasted so many years on debian now.
all it is, basically, is ubuntu. if i wanted to run ubuntu, i would.

thank you for your support

#551638#109
Date:
2010-03-21 00:43:42 UTC
From:
To:
I meant bug 551638 instead of 559519.

Regards,
robert

#551638#114
Date:
2010-03-27 09:58:28 UTC
From:
To:
Hi,

I discovered that the previous version of my patch re-added the "purge
ok not-installed" packages into status file. Fortunately the issue
turned out to be quite simple to fix. I'm attaching a new patch split
into two files.

The first one, 1_dselect_559519+556889.patch.gz, causes dselect to treat
all unknown packages all as already seen. This fixes #556889, since
dselect never automatically selects already seen packages for installation.
The change is especially visible at the packages' selection screen: the
non-patched version shows status `n_' (meaning `new purge') for most
packages, for example:
  n_ Xtr x11      choosewm     <none>      0.1.6-1     fake ...
  n_ Xtr x11      compiz-fusio <none>      0.8.4-1     Compiz Fusion ...
  n_ Xtr x11      compiz-fusio <none>      0.8.4-2     Compiz Fusion ...
My version displays the status as '__' (`purge purge').
I hope the patch would be quite safe for applying in dselect.


With the second patch, 2_dselect_551638.patch.gz, dselect stores already
seen packages in packages-seen file - one package name in each line. It
would be nice if you could apply this patch too, even though you might
dislike it e.g. because db is certainly not `fronted united', coding
introduces usage of STL maps, strings and iostreams and doesn't look
like the awful `C with classes' used elsewhere in dselect codes.

Regards,
robert

#551638#119
Date:
2010-03-29 08:51:47 UTC
From:
To:
How can the two patches work together? I mean either the package has been
already seen or it's not.

A new package by definition has no entry in the status file so it's
unknown (and thus seen) according to the first patch. What good does the
second patch do then?

Otherwise I discussed your patch with Guillem quickly the other day and he
would like to store the seen/not-seen information in a common place (i.e.
shared by all frontends). I'm not sure there's enough time to implement
this in the squeeze timeframe. Guillem mentionnend reverting part of the
problematic changes if he can't manage this in time, I'm not sure I like
this, I would rather commit something based on your patch.

In that case, if we store the data in a frontend-specific manner, the file
shall be stored in /var/lib/dselect and not /var/lib/dpkg. Our goal is to
prepare a split so new files should go there IMO.

In the mean time, I think committing your first patch would alleviate the
most annoying side-effects of the current situation. I would be in favor
of that, Guillem what do you think?

Cheers,

#551638#124
Date:
2010-03-30 12:09:17 UTC
From:
To:
Raphael Hertzog writes:

The second patch complements to the first one, i.e. the latter depends
on and extends the former one.

Currently dselect treats all unknown packages as unseen, which is
clearly visible by users and causes the behaviour reported in #556889.

In short the first patch makes dselect to treat all unknown packages as
already seen, whereas the second one adds to it ability to distinguish
between seen and unseen packages (so with the second patch unknown
packages are no longer all seen by default).

As you can see, the second patch contains amongst others the following
change:
-    state->direct= state->original= (pkg->want == pkginfo::want_unknown
 ? pkginfo::want_purge : pkg->want);
+    state->direct= state->original= (pkg->want == pkginfo::want_unknown
 && seen ? pkginfo::want_purge : pkg->want);
 ^^^^^^^


I was aware of Guillem's intentions, after all he put the magic
`frontend-unified state file' phrase in the bug's #551638 description :)
I think it's a nice idea, but probably not for squeeze. Even if you
manage to add such a file into dpkg on time for squeeze, most probably
aptitude, synaptic and any other frontend (besides dselect) won't
switch to using it so fast. So how about implementing temporary
dselect-only solution for now and implementing the proper solution for
squeeze + 1?

That's OK. I've put it in $admindir for simplicity and not to care about
locking the file as the any reads/writes are done where dpkg's status
database in locked. If you put the file somewhere else probably:
- either you would need to add some locking
- or the file location would need to depend on $admindir (like
$admindir/../dselect/, but it's quite strange)
- eventually you would need to drop the --admindir option from dselect
(honestly I've never used it).

I second this.

Regards,
robert

#551638#129
Date:
2010-04-14 19:35:06 UTC
From:
To:
I'd like to vote for getting back the old functionality, where one could
easily see which new software was available in the archives.
If that means Robert's two patches, with a temporary solution and other
frontends following suite after squeeze, by all means make it so.

This is dselect functionality from back when apt did not exist and there
were no other package management frontends in Debian.
Don't leave it broken in a stable release.


KR, Filip