#803371 coreutils: cp degrades system performance by wasting cache

Package:
coreutils
Source:
coreutils
Description:
GNU core utilities
Submitter:
Russell Coker
Date:
2015-10-29 11:12:13 UTC
Severity:
normal
#803371#5
Date:
2015-10-29 10:46:54 UTC
From:
To:
If you copy many large files to a slow device the amount of memory used for
cache grows needlessly and impacts overall system performance.  As we already
have the --reflink option I think that a precedent has been set for adding cp
options to improve system performance.

Firstly I think there should be an option to call fdatasync() after completely
writing data for a file.  This would reduce the amount of RAM used for write
back caching to a maximum of the size of one file, in the case where you copy
many MP4 files to an SD card that would reduce the amount of write back cache
to something like 300M instead of something large.  dd has a fdatasync option
to do this.

Next I think there should be an option to avoid caching file data which would
be particularly useful when it is known that the size of the files is larger
than the system RAM (IE caching really can't do any good even if you are
accessing the files twice).  The O_DIRECT option seems good for this and it's
an option to dd so it seems that it would work well for cp.

#803371#10
Date:
2015-10-29 10:59:31 UTC
From:
To:
Not exactly. That option changes the end state of the copy.

While I agree this is an issue, I'm not sure every too should
get control knobs like that. There are system wide settings
to control the amount of write back cache.  I'm not saying
cp definitely should not get such options, though it warrants
discussion, and that discussion should happen at coreutils@gnu.org.

thanks,
Pádraig

#803371#15
Date:
2015-10-29 10:59:31 UTC
From:
To:
Not exactly. That option changes the end state of the copy.

While I agree this is an issue, I'm not sure every too should
get control knobs like that. There are system wide settings
to control the amount of write back cache.  I'm not saying
cp definitely should not get such options, though it warrants
discussion, and that discussion should happen at coreutils@gnu.org.

thanks,
Pádraig