Hi,

Sorry to bother you and for my poor English. I hope you understand.

I work with Debian 12 and the following products:

  *   patroni 4.0.7
  *   etcd Version: 3.4.23
  *   pgBackRest 2.56.0
  *   postgresql 17

In the latest DEBIAN update, the products have been updated to:

  *   patroni 4.1.0
  *   etcd Version: 3.4.23
  *   pgBackRest 2.57.0
  *   postgresql 17

Since then, PATRONI has stopped working. The LEADER switches between servers.

Before migration :
+ Cluster: PG1 (7537608679052463361) ----+-----------+----+-----------+
| Member     | Host            | Role    | State     | TL | Lag in MB |
+------------+-----------------+---------+-----------+----+-----------+
| vdn-pg-t11 | 148.196.186.106 | Leader  | running   | 47 |           |
| vdn-pg-t12 | 148.196.186.107 | Replica | streaming | 47 |         0 |
+------------+-----------------+---------+-----------+----+-----------+

Start UPDATE :
root@vdn-pg-t11:~# apt update -y
Hit:1 http://ftp.ch.debian.org/debian bookworm InRelease
Get:2 http://security.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [107 kB]
Get:4 http://ftp.ch.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:5 http://security.debian.org/debian-security bookworm-security/main amd64 Packages [284 kB]
Get:6 http://security.debian.org/debian-security bookworm-security/main Translation-en [172 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [403 kB]
Fetched 1,069 kB in 1s (1,656 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: http://apt.postgresql.org/pub/repos/apt/dists/bookworm-pgdg/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

root@vdn-pg-t11:~# apt upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  bind9-dnsutils bind9-host bind9-libs intel-microcode patroni pgbackrest
6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.7 MB of archives.
After this operation, 51.2 kB of additional disk space will be used.
Get:1 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 patroni all 4.1.0-1.pgdg12+1 [287 kB]
Get:2 http://security.debian.org/debian-security bookworm-security/main amd64 bind9-host amd64 1:9.18.41-1~deb12u1 [54.7 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 pgbackrest amd64 2.57.0-1.pgdg12+1 [476 kB]
Get:4 http://security.debian.org/debian-security bookworm-security/main amd64 bind9-dnsutils amd64 1:9.18.41-1~deb12u1 [155 kB]
Get:5 http://security.debian.org/debian-security bookworm-security/main amd64 bind9-libs amd64 1:9.18.41-1~deb12u1 [1,181 kB]
Get:6 http://security.debian.org/debian-security bookworm-security/non-free-firmware amd64 intel-microcode amd64 3.20250812.1~deb12u1 [11.6 MB]
Fetched 13.7 MB in 1s (18.8 MB/s)
(Reading database ... 45586 files and directories currently installed.)
Preparing to unpack .../0-bind9-host_1%3a9.18.41-1~deb12u1_amd64.deb ...
Unpacking bind9-host (1:9.18.41-1~deb12u1) over (1:9.18.33-1~deb12u2) ...
Preparing to unpack .../1-bind9-dnsutils_1%3a9.18.41-1~deb12u1_amd64.deb ...
Unpacking bind9-dnsutils (1:9.18.41-1~deb12u1) over (1:9.18.33-1~deb12u2) ...
Preparing to unpack .../2-bind9-libs_1%3a9.18.41-1~deb12u1_amd64.deb ...
Unpacking bind9-libs:amd64 (1:9.18.41-1~deb12u1) over (1:9.18.33-1~deb12u2) ...
Preparing to unpack .../3-intel-microcode_3.20250812.1~deb12u1_amd64.deb ...
Unpacking intel-microcode (3.20250812.1~deb12u1) over (3.20250512.1~deb12u1) ...
Preparing to unpack .../4-patroni_4.1.0-1.pgdg12+1_all.deb ...
Unpacking patroni (4.1.0-1.pgdg12+1) over (4.0.7-3.pgdg12+1) ...
Preparing to unpack .../5-pgbackrest_2.57.0-1.pgdg12+1_amd64.deb ...
Unpacking pgbackrest (2.57.0-1.pgdg12+1) over (2.56.0-2.pgdg12+1) ...
Setting up bind9-libs:amd64 (1:9.18.41-1~deb12u1) ...
Setting up intel-microcode (3.20250812.1~deb12u1) ...
intel-microcode: microcode will be updated at next boot
Setting up pgbackrest (2.57.0-1.pgdg12+1) ...
Setting up patroni (4.1.0-1.pgdg12+1) ...
Installing new version of config file /etc/patroni/config.yml.in ...
Job for patroni.service failed because a timeout was exceeded.
See "systemctl status patroni.service" and "journalctl -xeu patroni.service" for details.
invoke-rc.d: initscript patroni, action "restart" failed.
● patroni.service - Runners to orchestrate a high-availability PostgreSQL
     Loaded: loaded (/lib/systemd/system/patroni.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: timeout) since Thu 2025-10-30 09:21:19 CET; 6ms ago
    Process: 8164 ExecStart=/usr/bin/patroni /etc/patroni/config.yml (code=exited, status=0/SUCCESS)
   Main PID: 8164 (code=exited, status=0/SUCCESS)
        CPU: 1.478s
dpkg: error processing package patroni (--configure):
installed patroni package post-installation script subprocess returned error exit status 1
Setting up bind9-host (1:9.18.41-1~deb12u1) ...
Setting up bind9-dnsutils (1:9.18.41-1~deb12u1) ...
Processing triggers for libc-bin (2.36-9+deb12u13) ...
Processing triggers for initramfs-tools (0.142+deb12u3) ...
update-initramfs: Generating /boot/initrd.img-6.1.0-40-amd64
W: initramfs-tools configuration sets RESUME=/dev/mapper/vg--root-lg--swap
W: but no matching swap device is available.
Errors were encountered while processing:
patroni
E: Sub-process /usr/bin/dpkg returned an error code (1)

After migration :

root@vdn-pg-t11:~# patronictl list
+ Cluster: PG1 (7537608679052463361) ----+-----------+----+-------------+-----+-------------+-----+
| Member     | Host            | Role    | State     | TL | Receive LSN | Lag |  Replay LSN | Lag |
+------------+-----------------+---------+-----------+----+-------------+-----+-------------+-----+
| vdn-pg-t11 | 148.196.186.106 | Replica | streaming | 48 | 6C/B60001E0 |   0 | 6C/B60001E0 |   0 |
| vdn-pg-t12 | 148.196.186.107 | Leader  | running   | 48 |             |     |             |     |
+------------+-----------------+---------+-----------+----+-------------+-----+-------------+-----+
…
root@vdn-pg-t11:~# patronictl list
+ Cluster: PG1 (7537608679052463361) ----+-----------+----+-------------+-----+-------------+-----+
| Member     | Host            | Role    | State     | TL | Receive LSN | Lag |  Replay LSN | Lag |
+------------+-----------------+---------+-----------+----+-------------+-----+-------------+-----+
| vdn-pg-t11 | 148.196.186.106 | Replica | streaming | 69 | 6C/CA0001E0 |   0 | 6C/CA0001E0 |   0 |
| vdn-pg-t12 | 148.196.186.107 | Leader  | running   | 69 |             |     |             |     |
+------------+-----------------+---------+-----------+----+-------------+-----+-------------+-----+
…
root@vdn-pg-t11:~# patronictl list
+ Cluster: PG1 (7537608679052463361) ----+-----------+-----+-------------+-----+-------------+-----+
| Member     | Host            | Role    | State     |  TL | Receive LSN | Lag |  Replay LSN | Lag |
+------------+-----------------+---------+-----------+-----+-------------+-----+-------------+-----+
| vdn-pg-t11 | 148.196.186.106 | Replica | streaming | 277 | 6D/980001A8 |   0 | 6D/980001A8 |   0 |
| vdn-pg-t12 | 148.196.186.107 | Leader  | running   | 277 |             |     |             |     |
+------------+-----------------+---------+-----------+-----+-------------+-----+-------------+-----+

The various logs are attached.

Thank you very much and best regards


Paolo Daniele
Ingénieur système

Hi,

Well, it was an update of the PostgreSQL community packages from
apt.postgresql.org (Debian 12 has patroni 3.0.2-1), but opening a bug
here is fine.

You run etcd on the local nodes apparantly, right?
[...]
[...]

You only attached logs for one of the machines (vdn-pg-t11), right? Can
you also attach the ones for the other?

It is hard to figure out what went wrong from them, but the beginning of
the patroni logs has this:

|2025-10-30 09:27:20,725 ERROR: Request to server http://148.196.186.106:2379 failed:ReadTimeoutError("HTTPConnectionPool(host='148.196.186.106', port=2379): Read timed out. (read timeout=3.3328031433333343)")

Which sounds like an etcd problem. Did you try to install etcd-client
and run the following:

etcdctl member list
etcdctl endpoint status
etcdctl endpoint health

It also looks like pgbackrest is very unhappy:

|2025-10-30 10:51:59.912 P00  ERROR: [045]: WAL file '000000C40000006D00000048' already exists in the repo1 archive with a different checksum
|2025-10-30 10:52:00.013 CET - 3 - 28698 -  - @ - 0LOG:  archive command failed with exit code 45
|2025-10-30 10:52:00.013 CET - 4 - 28698 -  - @ - 0DETAIL:  The failed archive command was: pgbackrest --stanza=stanza_PG archive-push /db/PG1/pgdata/pg_wal/000000C40000006D00000048

In general, I am not (yet) convinced this is a problem with Patroni 4.1.0,
there's not enough logs to say something for now.


Cheers,

Michael

Hi Daniele,

please keep the bug report CC´d, somebody else might be able to help you
better than me.

Ok.

Thanks. However, logs_vdn-pg-t12/logs/after_update/postgresql-Mon.log
just seems to be the beginning of
logs_vdn-pg-t11/logs/after_update/postgresql-Mon.log, so there are still
no logs from logs_vdn-pg-t12 it seems.

[...]

That all looks fine.

Looking at the logs, the upgrade was at or shortly before 2025-10-30
09:27:20,725 (first log line in
logs_vdn-pg-t11/logs/after_update/patroni.log) , is that right?

logs_vdn-pg-t11/logs/after_update/postgresql-Mon.log already shows
unhealthy replication a week ago:

|2025-10-27 08:51:06.332 CET - 1 - 1844 - 148.196.186.107 - replicator@[unknown] - 0ERROR:  requested starting point 6C/99000000 is ahead of the WAL flush position of this server 6C/98000218
|2025-10-27 08:51:06.332 CET - 2 - 1844 - 148.196.186.107 - replicator@[unknown] - 0STATEMENT:  START_REPLICATION SLOT "vdn_pg_t12" 6C/99000000 TIMELINE 40

As the logs are still incomplete (there are also no patroni logs from
vdn-pg-t12), I can't say what the problem is, but it might be that the
replication (and also archiving) was already somewhat bad before the
upgrade.


Michael

Hello Michael,

The Patroni log file was empty yesterday. I've attached today's log.

I reproduced the update problem in a test, but this occurred in our production environment. And these environments had been running without any issues for months.

Regards,

Paolo

[E-MAIL EXTERNE]

Hi Daniele,

please keep the bug report CC´d, somebody else might be able to help you better than me.

Ok.

Thanks. However, logs_vdn-pg-t12/logs/after_update/postgresql-Mon.log
just seems to be the beginning of
logs_vdn-pg-t11/logs/after_update/postgresql-Mon.log, so there are still no logs from logs_vdn-pg-t12 it seems.

[...]

That all looks fine.

Looking at the logs, the upgrade was at or shortly before 2025-10-30
09:27:20,725 (first log line in
logs_vdn-pg-t11/logs/after_update/patroni.log) , is that right?

logs_vdn-pg-t11/logs/after_update/postgresql-Mon.log already shows unhealthy replication a week ago:

|2025-10-27 08:51:06.332 CET - 1 - 1844 - 148.196.186.107 -
|replicator@[unknown] - 0ERROR:  requested starting point 6C/99000000 is
|ahead of the WAL flush position of this server 6C/98000218
|2025-10-27 08:51:06.332 CET - 2 - 1844 - 148.196.186.107 -
|replicator@[unknown] - 0STATEMENT:  START_REPLICATION SLOT "vdn_pg_t12"
|6C/99000000 TIMELINE 40

As the logs are still incomplete (there are also no patroni logs from vdn-pg-t12), I can't say what the problem is, but it might be that the replication (and also archiving) was already somewhat bad before the upgrade.


Michael

Hi,

Can you describe a full reproducible example again of this if you
managed at least once? I could not reproduce this so far.


Michael