Steps to reproduce:

$ echo '<A/>' | xmlstarlet fo --omit-decl; echo $?

Expected result:

<A/>
0

Actual result:

<A/>
5

What's really weird is that the returned error code is always
identical to the number of bytes written to the output.

This bug seems to only occur if --omit-decl is used.

Control: tags -1 upstream fixed-upstream
That's very observant of you; I've investigated this and found that XMLStarlet's new maintainership has fixed this issue accidentally. Here's where it comes from.

In xml_format.c as shipped by Debian, there's a branch inside the foProcess() function that is buggy:

This 'ret' variable becomes the exit status of the process, and depending on how big it is, it may seem strange when its least-significant eight bits are used as the exit status in a shell.
The reason why is that the foMain() function effectively contains

so the return value from xmlOutputBufferClose() propogates to foProcess(), which propogates to foMain(), and ultimately this becomes the exit status of the process, because xml.c defines the actual main() function which has something like
https://gnome.pages.gitlab.gnome.org/libxml2/html/xmlIO_8h.html#a36fe9cce2bb1dff29f251a993ccb745c

The new code at https://github.com/xmlstarlet/xmlstarlet/blob/master/src/xml_format.c has been largely tweaked and doesn't have this problem anymore, even if this was fixed unintentionally.