#992666 /usr/bin/cut: -n ignored => mangles output

Package:
coreutils
Source:
coreutils
Description:
GNU core utilities
Submitter:
наб
Date:
2021-08-28 18:30:02 UTC
Severity:
normal
#992666#5
Date:
2021-08-22 00:16:28 UTC
From:
To:
Dear Maintainer,

POSIX.1-2008 says:
-- >8 --
-n
    Do not split characters. When specified with the -b option,
	each element in list of the form low-high
	(<hyphen-minus>-separated numbers) shall be modified as follows:

	*	If the byte selected by low is not the first byte
		of a character, low shall be decremented to select
		the first byte of the character originally selected by low.
		If the byte selected by high is not the last byte of a character,
		high shall be decremented to select the last byte of the character
		prior to the character originally selected by high,
		or zero if there is no prior character.
		If the resulting range element has high equal to zero
		or low greater than high, the list element shall be
		dropped from list for that input line without causing an error.

	Each element in list of the form low- shall be treated as above
	with high set to the number of bytes in the current line,
	not including the terminating <newline>.
	Each element in list of the form -high shall be treated as above
	with low set to 1.
	Each element in list of the form num (a single number)
	shall be treated as above with low set to num and high set to num.
-- >8 --


With a more succinct exemplary text driving the point home:
-- >8 --
Earlier versions of the cut utility worked in an environment
where bytes and characters were considered equivalent
(modulo <backspace> and <tab> processing in some implementations).
In the extended world of multi-byte characters,
the new -b option has been added.
The -n option (used with -b) allows it to be used to act on bytes
rounded to character boundaries.
The algorithm specified for -n guarantees that:
	cut -b 1-500 -n file > file1
	cut -b 501- -n file > file2
ends up with all the characters in file appearing exactly once
in file1 or file2. (There is, however, a <newline> in both
file1 and file2 for each <newline> in file.)
-- >8 --


So, compare a conforming implementation:
-- >8 --
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 1-5
яй
ЯЙ
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 6-
цо
ЦО
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 1-4
яй
ЯЙ
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 5-
цо
ЦО
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 1-3
я
Я
$ printf 'яйцо\nЯЙЦО' | ./out/cmd/cut -nb 4-
йцо
ЙЦО
-- >8 --

With the garbage that GNU cut spews:
-- >8 --
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-5
яй�
ЯЙ�
$ printf 'яйцо\nЯЙЦО' | cut -nb 6-
�о
�О
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-4
яй
ЯЙ
$ printf 'яйцо\nЯЙЦО' | cut -nb 5-
цо
ЦО
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-3
я�
Я�
$ printf 'яйцо\nЯЙЦО' | cut -nb 4-
�цо
�ЦО
-- >8 --

Or, without the luxury of REPLACEMENT CHARACTER:
-- >8 --
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-5 | hexdump -C
00000000  d1 8f d0 b9 d1 0a d0 af  d0 99 d0 0a              |............|
0000000c
$ printf 'яйцо\nЯЙЦО' | cut -nb 6-  | hexdump -C
00000000  86 d0 be 0a a6 d0 9e 0a                           |........|
00000008
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-4 | hexdump -C
00000000  d1 8f d0 b9 0a d0 af d0  99 0a                    |..........|
0000000a
$ printf 'яйцо\nЯЙЦО' | cut -nb 5-  | hexdump -C
00000000  d1 86 d0 be 0a d0 a6 d0  9e 0a                    |..........|
0000000a
$ printf 'яйцо\nЯЙЦО' | cut -nb 1-3 | hexdump -C
00000000  d1 8f d0 0a d0 af d0 0a                           |........|
00000008
$ printf 'яйцо\nЯЙЦО' | cut -nb 4-  | hexdump -C
00000000  b9 d1 86 d0 be 0a 99 d0  a6 d0 9e 0a              |............|
0000000c
-- >8 --


If we consult the manual, we can see:
-- >8 --
$ man cut | grep -C3 -- -n
              select  only  these fields;  also print any line that contains no delimiter character, unless the -s op‐
              tion is specified

       -n     (ignored)

#992666#10
Date:
2021-08-28 18:28:07 UTC
From:
To:
Hi!

A better and more exhaustive test dataset can be as simple as:
-- >8 --
for i in $(seq 10); do
  echo "-$i;$((i+1))-" | paste - \
    <(printf 'яйцо\nЯЙЦО' | cut -nb -$i) \
    <(printf 'яйцо\nЯЙЦО' | cut -nb $((i+1))-)
done
-- >8 --

With a correct implementation, this yields:
-- >8 --
-1;2-		яйцо
		ЯЙЦО
-2;3-	я	йцо
	Я	ЙЦО
-3;4-	я	йцо
	Я	ЙЦО
-4;5-	яй	цо
	ЯЙ	ЦО
-5;6-	яй	цо
	ЯЙ	ЦО
-6;7-	яйц	о
	ЯЙЦ	О
-7;8-	яйц	о
	ЯЙЦ	О
-8;9-	яйцо	
	ЯЙЦО
-9;10-	яйцо	
	ЯЙЦО
-10;11-	яйцо	
	ЯЙЦО
-- >8 --

With the current coreutils implementation, this yields:
-- >8 --
-1;2-	�	�йцо
	�	�ЙЦО
-2;3-	я	йцо
	Я	ЙЦО
-3;4-	я�	�цо
	Я�	�ЦО
-4;5-	яй	цо
	ЯЙ	ЦО
-5;6-	яй�	�о
	ЯЙ�	�О
-6;7-	яйц	о
	ЯЙЦ	О
-7;8-	яйц�	�
	ЯЙЦ�	�
-8;9-	яйцо	
	ЯЙЦО
-9;10-	яйцо	
	ЯЙЦО
-10;11-	яйцо	
	ЯЙЦО
-- >8 --

Or, without replacement characters:
-- >8 --
00000000  2d 31 3b 32 2d 09 d1 09  8f d0 b9 d1 86 d0 be 0a  |-1;2-...........|
00000010  09 d0 09 af d0 99 d0 a6  d0 9e 0a 2d 32 3b 33 2d  |...........-2;3-|
00000020  09 d1 8f 09 d0 b9 d1 86  d0 be 0a 09 d0 af 09 d0  |................|
00000030  99 d0 a6 d0 9e 0a 2d 33  3b 34 2d 09 d1 8f d0 09  |......-3;4-.....|
00000040  b9 d1 86 d0 be 0a 09 d0  af d0 09 99 d0 a6 d0 9e  |................|
00000050  0a 2d 34 3b 35 2d 09 d1  8f d0 b9 09 d1 86 d0 be  |.-4;5-..........|
00000060  0a 09 d0 af d0 99 09 d0  a6 d0 9e 0a 2d 35 3b 36  |............-5;6|
00000070  2d 09 d1 8f d0 b9 d1 09  86 d0 be 0a 09 d0 af d0  |-...............|
00000080  99 d0 09 a6 d0 9e 0a 2d  36 3b 37 2d 09 d1 8f d0  |.......-6;7-....|
00000090  b9 d1 86 09 d0 be 0a 09  d0 af d0 99 d0 a6 09 d0  |................|
000000a0  9e 0a 2d 37 3b 38 2d 09  d1 8f d0 b9 d1 86 d0 09  |..-7;8-.........|
000000b0  be 0a 09 d0 af d0 99 d0  a6 d0 09 9e 0a 2d 38 3b  |.............-8;|
000000c0  39 2d 09 d1 8f d0 b9 d1  86 d0 be 09 0a 09 d0 af  |9-..............|
000000d0  d0 99 d0 a6 d0 9e 09 0a  2d 39 3b 31 30 2d 09 d1  |........-9;10-..|
000000e0  8f d0 b9 d1 86 d0 be 09  0a 09 d0 af d0 99 d0 a6  |................|
000000f0  d0 9e 09 0a 2d 31 30 3b  31 31 2d 09 d1 8f d0 b9  |....-10;11-.....|
00000100  d1 86 d0 be 09 0a 09 d0  af d0 99 d0 a6 d0 9e 09  |................|
00000110  0a                                                |.|
00000111
-- >8 --

Best,
наб