Dear Maintainer,

today I upgraded our famliy+friends email server system from bullseye to
bookworm including the cyrus-imapd upgrade to v3.6.1-4.

Unfortunately, all mails of all users disappeared and new inboxes were
autocreated when users logged in:
-----------------------------

The emails were still in the non-uuid directory of the cyrus spool dir. I did
not figure out how this happened, but I solved it by

1. extracting a list of all sub-mailboxes of all users from the filesystem
structure,
2. re-creating the same sub-mailboxes with the cm command of cyradm,
3. and hard-linking all directory content to the new uuid dirs (with the help
of mbpath)

The issue is now solved for me, but I can provide additional system
information and logs if needed.

(same report goes to https://github.com/cyrusimap/cyrus-imapd/issues/4035)

Dear Kai,
Thank you very much for this report and a possible solution. Exactly the
same happened to me on upgrade to bookworm. Can you please elaborate how
you did this in detail? I really would be in trouble if I lose the emails.
I know this is a big ask but I would really appreciate your help.
Best regards
Michael

PS. I may have a delay answering, I am down under.

Dear Maintainer,

a new upstream bug was filed on github https://github.com/cyrusimap/cyrus-imapd/issues/4532.

Best
-Kai

Dear Michael,
(see below), I added the comments for this post. The script was created
iteratively while figuring out how the recovering might work. I worked on the
live system and relied on my backup. I did not spent any time to make the
script fail-safe or even readable, sorry. (Be sure to backup the spool
directory......)

===============================

#!/bin/bash
# the cyrus spool dir
SPOOLDIR=/var/spool/cyrus/mail/
# first argument is relative spool dir of user e.g.: $ scriptname k/user/kai
# extract user and reformat for cm command of cyadmin
# k/user/kai -> user.kai
USER=$(echo $1|cut -d/ -f 2,3|tr "/" ".")
# find all mailboxes of user and reformat for cm
MBXLIST=$(find $SPOOLDIR$1 -type d|cut -d/ -f 1-8 --complement|tr "/ " "._")
# generate a script for cyradm to create new mailboxes
for MBX in $MBXLIST; do
echo cm $USER.$MBX
done > creatembx.cyradm
echo starting shell to examine the situation
echo creatembx.cyradm created to feed cyradm \(please review\)
echo continue with exit
# start a shell to check cyradmin script and general situation
# you need to feed the cyradmin script to cyadmin with
# $ cat creatembx.cyradm | cyradm --user cyrus localhost
bash
# generate a script to hard-link to the new location
for MBX in $MBXLIST; do
# get path of created mailbox
NEWPATH=$(/usr/lib/cyrus/bin/mbpath $USER.$MBX)
# get original path of mailbox
OLDPATH=$SPOOLDIR$1/${MBX//\./\/}
# link it
echo ln -f ${OLDPATH//_/\\ }/\* $NEWPATH
done | tee linkmbx.bash
echo linkmbx.bash created, please review before executing
echo manual work:
echo 1. might be too many argument, review output
echo 2. main inbox not linked, create inbox_recovered

======================================

(be sure you understand each step of the script and be sure that it fits to
your configuration.)

You need to apply this script for all users on your systems (~20 users on my
system). Then you should see all sub-mailboxes filled with mails.

I treated the INBOX differently because the server was already receiving new
mail and out them into the INBOX. To avoid any interferences I created a
mailbox inbox_recovered for each user with cyradm:

$ cyradm --user cyrus localhost
xxx.xxx> cm user.kai.inbox_recovered
ctrl-d

and hard-linked the mails with

$ cd `mbpath user.kai.inbox_recovered`
$ ln -f /var/spool/cyrus/mail/k/user/kai/* .

(these are the commands from by bash history)

some final thoughts:
- I did all the operations as the user cyrus
- I used hard-linking to avoid copying 10s of Gigs of mails....
- it fails with unusual characters because of not escaping them
- new mailbox use underscore instead of blank (did not want to escape too)

Total time for my system: 6 hours with analysis, learnings, and recovering

Feel free to ask, if anything was too unclear, of if you want to know why I
did sth this way (sometime there might be a reason, sometimes I did not know
better)

Good luck
Kai

Kai,

Thanks so much for your work here, I've been panicking about my email.

This script is a HUGE help. I noticed one thing though, in the linking,
it doesn't really handle nested folders, so I get a ton of:

ln: /var/spool/cyrus/mail/p/user/phil/lists/linuxdriver: hard link not
allowed for directory

And one small thing, it might be useful for others if the 'echo's before
the first bash gave the command to run to feed the file to cyradm, so
you don't have to go back to the script to find it.

Thanks!

I've fixed up the script a bit and thrown it into a public repo, so
others can easily access it and hack on it as well:

https://github.com/jaymzh/recover-cyrus/blob/main/recover_cyrus_user.sh

Credit given at the top of the script.

Kai,

I noticed that no emails in nested folders were recovered. Is that expected?

Aha, I figured out the issue.

Your script does:

   ln -f ${OLDPATH//_/\\ }/\* $NEWPATH

But the substitution there is incorrect - the folders with `_` in them,
actually have that in their file path.

OK I've updated my copy of the script to properly handle both
underscores and spaces:

https://github.com/jaymzh/recover-cyrus/blob/main/recover_cyrus_user.sh

Hope that helps others!

Dear Maintainers

The documentation

https://www.cyrusimap.org/3.6/imap/download/upgrade.html#versions-to-upgrade-from

tells me that we should have upgraded from Cyrus 3.2.6 to 3.2.10 before going to
3.6.x. Bullseye stable didn't take us that way, and perhaps neither did Bullseye-backports.

Perhaps a check should be made, and stop Cyrus 3.2 from being upgraded?

Best wishes,

Hi,

the patch needed to update to 3.6 is included in version
3.2.6-2+deb11u2. So the doc should be updated to explain that "we should
have upgraded from Cyrus 3.2.6-2 to 3.2.6-2+deb11u2 before going to 3.6.x"

Regards,
Yadd

Hello Yadd,

Thanks for all your work on Cyrus-imapd in Debian!

Ellie writes: "From the amount of failures reported by people who have
let apt upgrade Cyrus for them, I guess there's a step missing in
whatever apt does". Is there something missing so far you know?
https://cyrus.topicbox.com/groups/info/Ta01e6935b46972c7-Mbb5e079f8d91464da502f947/upgrade-cyrus-on-debian-11-to-12

Is it a good idea to upgrade first to the backported version in Debian
11, and then later to Debian 12?  (In this way, I can concentrate on
Cyrus first.)

I have many mailboxes on my server, some of them are over 20 years old,
and some of them contain much data.

With regards,
Paul van der Vlis

Unfortunately my #calendars and #adresses mailboxes have also been hit.
They now have Inboxes and all entries are gone. Any experience on how
to recover these?

Thanks,
jOACHIM

The recovery script worked quite nicely. Thanks, I could not have done
it without. However my addressbook is not working yet. It has been
recovered, the links have been created:

mail:~# /usr/lib/cyrus/bin/mbpath 'user.jzobel.#addressbooks.Default'
/var/spool/cyrus/mail/uuid/n/c/ncp127mdldzs1jn8augq24jo
mail:~# ls -lrt /var/spool/cyrus/mail/uuid/n/c/ncp127mdldzs1jn8augq24jo
| head
total 1220
-rw------- 2 cyrus mail    518 Aug  9  2015 2.
-rw------- 2 cyrus mail    536 Aug 16  2015 293.
-rw------- 2 cyrus mail    609 Aug 16  2015 295.
...

But nothing is delivered to my client (evolution).

Sincerely,
Joachim

The recovery of the DAV mailboxes needs an additional call to
/usr/lib/cyrus/bin/dav_reconstruct. After that, my addressbooks and
calendars started working again. Client caching may however be a
problem, I used cadaver for testing.

Sincerely,
Joachim

Hi,

we had the same problem. After upgrade was created only mailboxes, which
had uniqueid in database. We can check it in dump:
/usr/lib/cyrus/bin/cvt_cyrusdb /var/lib/cyrus/mailboxes.db twoskip
/tmp/aaa.txt flat;
folder without uniqueid doesn't have "I" in dump /tmp/aaa.txt.

It can be repaired _before_ upgrade with:
/usr/lib/cyrus/bin/reconstruct -I
...
pokus.cz!test: update uniqueid from header (null) => 7f54216e52e02e7c
...

I hope, that this repaired mailboxes.db can be upgraded without problem
and will check it in few days.

I copy mailboxes.db without "reconstruct -I" to test bookworm system and
run "/usr/lib/cyrus/bin/ctl_cyrusdb -r". After that:
/usr/lib/cyrus/bin/ctl_mboxlist -d | wc -l
458
I run  "reconstruct -I" on test source system and copy to test bookworm
system, and run "/usr/lib/cyrus/bin/ctl_cyrusdb -r". After that:
/usr/lib/cyrus/bin/ctl_mboxlist -d | wc -l
2280



And there is typo in postinst in cyrus-common package (you must test
zero size and compare version in variable $2, not $1):
===
case "$1" in
     configure)
         # Refuse to update if previous version is lower than
3.2.6-2+deb11u2~
         if [ -z "$1" ] || $(dpkg --compare-versions $1 lt
'3.2.6-2+deb11u2~'); then
             echo "You must update cyrus-imapd to at least version
3.2.6-2+deb11u2~" >&2
             echo "before updating it to version 3.6.x and run it, else
your mailboxes" >&2
             echo "may be corrupted" >&2
             exit 1
         fi
===

Regards,
Petr Jurasek


On Sat, 5 Aug 2023 21:55:30 +0200 Paul van der Vlis <paul@vandervlis.nl>
wrote:
 > Hello Yadd,
 >
 > Thanks for all your work on Cyrus-imapd in Debian!
 >
 > On Wed, 19 Jul 2023 06:38:48 +0400 Yadd <yadd@debian.org> wrote:
 > > the patch needed to update to 3.6 is included in version
 > > 3.2.6-2+deb11u2. So the doc should be updated to explain that "we
should
 > > have upgraded from Cyrus 3.2.6-2 to 3.2.6-2+deb11u2 before going to
3.6.x"
 >
 > I have heard about problems, even when upgrading from 3.2.6-2+deb11u2.
 >
 > Ellie writes: "From the amount of failures reported by people who have
 > let apt upgrade Cyrus for them, I guess there's a step missing in
 > whatever apt does". Is there something missing so far you know?
 >
https://cyrus.topicbox.com/groups/info/Ta01e6935b46972c7-Mbb5e079f8d91464da502f947/upgrade-cyrus-on-debian-11-to-12
 >
 > Is it a good idea to upgrade first to the backported version in Debian
 > 11, and then later to Debian 12? (In this way, I can concentrate on
 > Cyrus first.)
 >
 > I have many mailboxes on my server, some of them are over 20 years old,
 > and some of them contain much data.
 >
 > With regards,
 > Paul van der Vlis
 >
 > --
 > Paul van der Vlis Linux systeembeheer Groningen
 > https://vandervlis.nl/
 >
 >

Hi,

thanks for the fix. Did you get this issue when upgrading from
3.2.6-2+deb11u2 or upgrading from 3.2.6-2 ?

Hi,

Dne 26. 08. 23 v 5:13 Yadd napsal(a):

it was upgrade from buster, 3.0.8-6+deb10u6. Now I know, that it's not
unsupported, but with this typo it was possible. But I tested this
upgrade on few instances without problem. This was first situation and I
think, that this mailbox database inconsistency was the reason.

Regards,
Petr Jurasek

Hi,

Confirming this issue also arises when coming from 3.2.6-2+deb11u2.
After conducting an update from a fully updated Bullseye to Bookworm, no
more mails were in any user's account, and new mailboxes were created
for users when logging in.

All accounts are many years old. As I had awareness of potential upgrade
issues, I even tried to run reconstruct -G -V max before conducting the
upgrade, which though did not help to prevent the desaster.

Suggestion:
Bullseye (oldstable) should be updated to a Cyrus version 3.2.10+, as
this is the only confirmed way as per Cyrus' upgrade documentation to
avoid these issues when moving to v3.6.1 as contained in Bookworm.
https://www.cyrusimap.org/3.6/imap/download/upgrade.html#versions-to-upgrade-from

I currently have no awareness of what else an admin could do to prevent
this from happening when updating directly from 3.2.6-2+deb11u2
(Standard Bullseye). The most promising hints I've read in other threads
is to run the reconstruct with additional parameters (reconstruct -f -G
-I -V max), whereas I would suggest anyone not to test this without a 
very solid backup.

For others for whom it's too late and who are here for solutions, here
is what I did to recover after the desaster:

1) su'ed to user cyrus
2) Downloaded rescue script by @larsimmisch from here and saved to local
folder:
https://gist.github.com/larsimmisch/f0c02977ffc9a9c72781716eaa3d7334
3) In the same folder:
git clone https://github.com/larsimmisch/python_cyrus python_cyrus
4) python3 ./cyrreconstruct.py -a <cyrus admin user> <mailbox username>

Step 4 will only work if the mailbox has already been created again
(what happens automatically - as an empty one - when a user logs in). If
the mailbox isn't present yet, use cyradm and
cm user.<username>
to create it before running the cyrreconstruct.py script.

Special thanks to @larsimmisch for saving my life, and to the package
maintainers for acting swiftly for preventing others from this nightmare.

Best,
Bastian

Hi,

after a few tests I can reproduce error on test environment.

mailboxes.db are broken only if I make connections to cyrus during
upgrade. I think, that some postinst restarts cyrus without all needed
packages upgraded or some other coincidence.

But repair is simple - copy mailboxes.db from backup and restart cyrus
:-). This will run "ctl_cyrusdb -r" and repair mailboxes.db.

Folders, that was badly created during upgrade, will be still empty and
you must run recover on them. All mails will be flagged as unseen. If
you use "delete_mode: delayed", users will have "delayed deleted" mails
in folders.

If you can repair mailboxes.db on different server (without physical
mailboxes in /var/lib/cyrus),  cyrus will include only folders, that
have uid included in mailboxes.db. Folders without uid you can see:
/usr/lib/cyrus/bin/cvt_cyrusdb /var/lib/cyrus/mailboxes.db twoskip
/tmp/aaa.txt flat; cat /tmp/aaa.txt | egrep -a -v '( I |\%\(I)'
But you can add uid before upgrade with:
/usr/lib/cyrus/bin/reconstruct -I
and this mailboxes.db will include all folders if you run it on test
machine without physical mailboxes in /var/lib/cyrus.

Prevention is block imap/pop3 in firewall during upgrade.

Regards,
Petr Jurasek

Dne 26. 08. 23 v 5:13 Yadd napsal(a):

Some additional information:
 - You can roll-back to package 3.2 after the upgrade to 3.6, the
   mailboxes will re-appear again
 - You might encounter some junk mailboxes from the failed conversion
   attempt, the remedy is to convert mailboxes.db to text, remove the
   offending lines, and convert it back to its original format.
 - The proper may to migrate to 3.6 without hassle is:
   - Stop cyrus-imapd service
   - Download and compile 3.4, run ./imap/ctl_cyrusdb -r once (make a
     copy of /var/lib/cyrus/mailboxes.db beforehand, of course)
 - Then proceed with the update

It seems that the debian package, despite claiming that it has the
necessary patches to make the upgrade possible, doesn't.

Op 06-02-2024 om 11:34 schreef Jens Georg:
I still have to do the upgrade on a few machines..

Did you do this on Debian 11 or Debian 12?

What did you download exactly?  Maybe this?
https://github.com/cyrusimap/cyrus-imapd/tree/cyrus-imapd-3.4.6

After downloading and go to the right directory something like:
systemctl stop cyrus-imap
./configure
make
sudo make install

Did you use any Debian patches?

sudo cp -a /var/lib/cyrus/mailboxes.db /path/to/backup/

./imap/ctl_cyrusdb -r

How did you remove this version?

Then install the new Debian packages?

Bye,
Paul

Hi,

I had the same problem (mails were not displayed in the mail client any
more after upgrading from Bullseye to Bookworm).

What worked in the end for me was this:

- restored /var/lib/cyrus/ and /var/spool/cyrus/ from backup, to the old
state from before the Debian upgrade.
- built Cyrus 3.4.6, and installed and started it temporarily. The mails
were now visible in the mail client again.
- upgraded to the normal Cyrus version from Bookworm. The mails were
still visible in the mail client.

And since I had already kept the server running for some hours with the
broken mail archive, I then also restored the getmail6 status files
(/var/lib/getmail) to the state from before the Debian upgrade. This
caused getmail to re-fetch the mails that were missing.

I don't know how to detect whether the mail archive was really migrated
successfully; but since Cyrus 3.6.1 is now running and Thunderbird can
connect and sees all mails, I suppose the migration was successful?


Regarding the build of Cyrus 3.4.6, in the end I built it using the
existing Debian packaging data, and using the "debocker" tool to build
inside a clean Docker container.

Unfortunately debocker will always run the "lintian" tool to check the
package, which failed for me with error "E: cyrus-common:
depends-on-obsolete-package Depends: lsb-base". I didn't know how to fix
this error or how to correctly disable the lintian step; so I edited the
debocker files to disable this check. Very ugly, but it worked.

So from my notes, these must have been the steps that I did for building
the Cyrus 3.4.6 Debian package:
- installed "debocker" and "devscripts" packages: `sudo apt install
debocker devscripts`
- downloaded the Debian cyrus-imapd packaging info: `debcheckout
cyrus-imapd`
- downloaded the original source package: `wget
https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.4.6/cyrus-imapd-3.4.6.tar.gz`
- renamed the source package so it would be found in the next steps: `mv
cyrus-imapd-3.4.6.tar.gz cyrus-imapd_3.4.6.orig.tar.gz`
- `cd cyrus-imapd/`
- started new Git branch at the state of Cyrus 3.4.3-4: `git checkout -b
cyrus-3.4.6 debian/3.4.3-4`
- added changelog entry: `dch -v '3.4.6-1.1' "use new upstream version
3.4.6"`
	- the warning about missing DEBEMAIL can be ignored
- committed change: `git commit debian/changelog -m 'update changelog'`

- modified the "debocker" template files to disable the lintian step:
`sudo nano /usr/share/debocker/bundle-files/steps/05-build` and then
commented out the line for "lintian --pedantic --display-info *.changes"

- created build bundle: `debocker bundle --image debian:bookworm -f "-b
-us -uc"`
- did the actual build: `sudo debocker build-bundle
../cyrus-imapd_3.4.6-1.1_bundle.tar`
	- whether the "sudo" is necessary depends on your local Docker setup
	- the build took a while (half an hour or more?); and then there was a
message like "LOG Build successful", and there were lots of Cyrus
packages in the current directory.

I then copied cyrus-clients_3.4.6-1.1_amd64.deb,
cyrus-common_3.4.6-1.1_amd64.deb and cyrus-imapd_3.4.6-1.1_amd64.deb to
the server and installed them.

Remember to undo the change to the debocker file
(/usr/share/debocker/bundle-files/steps/05-build) after the build.

Kind regards,
Oliver

Hello,

I also got hit by this bug - old mailboxes were still in the place, but
new empty mailboxes started to appear in uuid folders. And mailboxes.db
got corrupted/erased.

The easy way is to add to /etc/imapd.conf

mailbox_legacy_dirs: yes

BEFORE starting the upgrade. See:
https://www.cyrusimap.org/3.6/imap/download/upgrade.html#storage-changes

So the easy way would be to add this information to NEWS.Debian.gz

More complicated (but more secure) way, would be to add this option
automatically if upgrading from version < 3.6.0