#1082617 amanda-common: extremely inefficient output reading during restore/verify

Package:
amanda-common
Source:
amanda-common
Description:
Advanced Maryland Automatic Network Disk Archiver (Libs)
Submitter:
Damyan Ivanov
Date:
2024-09-23 12:00:02 UTC
Severity:
normal
Tags:
#1082617#5
Date:
2024-09-23 11:57:03 UTC
From:
To:
Hi,

Currently, amrestore reads the "tar" program output and writes it to a debug
file. The problem is that that output is read one byte at a time, which is very
very inefficient and takes unreasonable amount of time for big dumps.

I run verification of the completed dumps and it is much slower than the dump
itself - days vs. hours. The system load is heavily CPU-bound on the restore
process.

The patch below reads the output in larger chunks and uses the Perl's
instruments for splitting it into lines which are then forwarded to the debug
logs as usual.

This makes the dump verification complete in a reasonable time and during that
the system load is as expected - I/O and decompression.


Thanks for considering,
    Damyan
------------------8<----------8<-------------------
--- /usr/lib/x86_64-linux-gnu/amanda/perl/Amanda/Restore.pm-ORIG	2024-09-16 11:00:39.045205405 +0300
+++ /usr/lib/x86_64-linux-gnu/amanda/perl/Amanda/Restore.pm	2024-09-23 14:41:25.973046593 +0300
@@ -551,7 +551,7 @@
     $self->{'all_filter'}->{$src} = 1;
     $src->set_callback( sub {
 	my $b;
-	my $n_read = POSIX::read($fd, $b, 1);
+	my $n_read = POSIX::read($fd, $b, 4096*16);
 	if (!defined $n_read) {
 	   return;
 	} elsif ($n_read == 0) {
@@ -563,9 +563,8 @@
 	   }
 	} else {
 	   $buffer .= $b;
-	   if ($b eq "\n") {
-		my $line = $buffer;
-		chomp $line;
+	   while ($buffer =~ s/^([^\n]*)\n//) {
+		my $line = $1;
 #		if (length($line) > 1) {
 		   if ($line !~ /trailing garbage ignored/) {
 			my $msg_severity;
@@ -586,7 +585,6 @@
 				line		=> $line));
 		   }
 #		}
-		$buffer = "";
 	   }
 	}
     });
------------------8<----------8<-------------------