On s390x (big endian), Buffer.prototype.toString('utf8') silently decodes a
code point as U+0000 when all of the following hold:
- the code point's low byte is zero (U+0100, U+2000, U+3000, U+4E00,
U+AC00, U+FB00, ... 230 code points in the BMP);
- it is at the very start of the buffer;
- it is followed only by NUL bytes;
- the buffer is at least 32 bytes long.
No exception is thrown, no U+FFFD is produced, and the decoded string has the
correct length -- only the code unit is wrong. Code points whose low byte is
non-zero are unaffected, which is why this went unnoticed.
$ node -e 'const b=Buffer.alloc(32); b.write("\u3000",0,"utf8");
console.log(b.toString("utf8").charCodeAt(0).toString(16))'
s390x: 0
amd64: 3000
Full matrix on s390x (amd64 is correct everywhere):
U+3000 + NUL*13 len=16 -> U+3000 ok
U+3000 + NUL*28 len=31 -> U+3000 ok
U+3000 + NUL*29 len=32 -> U+0000 BROKEN
U+3000 + NUL*64 len=67 -> U+0000 BROKEN
U+3000 + 'a'*29 len=32 -> U+3000 ok
'a'*29 + U+3000 + NUL*29 len=61 -> U+3000 ok
'a'*29 + U+3000, no NUL len=32 -> U+3000 ok
TextDecoder.decode() and Buffer.prototype.utf8Slice() are affected in the same
way, so the bug is in the shared UTF-8 decoding path.
simdutf has been ruled out: a native reproducer calling
simdutf::convert_utf8_to_utf16{,le,be} directly on the same input returns the
correct code unit for every length tested (active implementation "fallback",
libsimdutf 8.2.0-1). Sources and outputs of all reproducers attached.
The failure signature -- the low byte survives, the high byte is lost, and only
for code points whose low byte is zero -- suggests a byte-order confusion in a
"does this string fit in one byte (Latin-1)" fast path.
Impact: breaks node-tar test on s390x
https://ci.debian.net/packages/n/node-tar/testing/s390x/73799909/
Found after many tries with the help of Claude-Code