I have 48 MB, after booting about 36 MB are still free. $ dd if=/dev/zero of=/tmp/image bs=1024k count=50 (default pager): dropping data request because of previous paging errors (default pager): dropping data request because of previous paging errors (default pager): dropping data request because of previous paging errors (default pager): dropping data request because of previous paging errors ... scrolls heavily, dies, hangs. Then I reboot, and look at /tmp/image. It's about 34MB big (so about the size of memory I had free). Then I try again, but this time I am more clever. I activate swap. I have a 128 MB swap partition. $ swapon /dev/hd2s1 $ dd if=/dev/zero of=/tmp/image bs=1024k count=50 $ This command takes a long time, btw, partly because it starts to use the swap after 36 MB :) Writing to a file does take as much memory as you write. This shouldn't be the case, when no memory is available, it should start to flush the disk cache or so. I have not tried to write to /dev/null (this is left as an exercise to the reader). Thanks, Marcus
I have reproduced this. (It does not happen with /dev/null, that works fine.) It appears to be some sort of leak in ext2fs (or libdiskfs, I haven't been able to test ufs). The memory-related info from ps is bogus (at least it is for me), but vmstat clearly shows swap being eaten up as the dd runs, and it doesn't come back when it's finished. Running vminfo on the extfs process shows a lot more pages allocated after than before. (If you dd an amount about your amount of physical RAM, you notice some obvious thrashing, too.) Note that with a usage pattern like this, there is a huge explosion of threads in the filesystem (mine has 914). That is a known issue and not actually a leak. But there is also a leak here. Repeating the large dd, my ext2fs process never gets more threads, but it does end up with some more pages allocated in its address space (I think they are anonymous pages).
Date: Tue, 18 May 1999 17:39:22 -0400 From: Roland McGrath <roland@gnu.org> I have reproduced this. (It does not happen with /dev/null, that works fine.) It appears to be some sort of leak in ext2fs (or libdiskfs, I haven't been able to test ufs). The memory-related info from ps is bogus (at least it is for me), but vmstat clearly shows swap being eaten up as the dd runs, and it doesn't come back when it's finished. Running vminfo on the extfs process shows a lot more pages allocated after than before. (If you dd an amount about your amount of physical RAM, you notice some obvious thrashing, too.) I think the fact that the swap doesn't come back when the dd is finished is related to the fact that we use memory objects that may be cached. This means that the kernel keeps the memory object around for a while, even if it is no longer mapped. You can see that the swap comes back if you remove the file created with dd or if you terminate the filesystem server. The thrashing that accours of you dd an amount about the amount of physical RAM indicates that there is something wrong with the way paging is done. Either a kernel bug or a bug in the default pager. Note that with a usage pattern like this, there is a huge explosion of threads in the filesystem (mine has 914). That is a known issue and not actually a leak. But there is also a leak here. Repeating the large dd, my ext2fs process never gets more threads, but it does end up with some more pages allocated in its address space (I think they are anonymous pages). I repeatedly create a 10 MB file using $ dd if=/dev/zero of=file bs=1024k count=10 The first time the number of threads increases a lot, to about 60, and the amount of free swap decreases by approximately 10 MB. The second time it increases to about 80. Then it increases in small steps to about 95. For a while after the dd command finishes the amount of virtual memory used is indeed higher than before, but after a while it stabilizes. The difference, compared to the amount of virtual memory used before the dd command, seems to be very close to 16 * increase of number of threads (in pages). No surprise since the default stack size of a thread is 16 pages. By the way. Do you realize that the explosion of threads in the filesystem is very bad. The stack space alone used by your 914 threads is 16 * 4 * 914 = 60 MB. Are there any thoughts on fixing this explosion of threads? Mark
That is only virtual space, and swap is lazily allocated. I suspect that in reality most or all threads in the filesystem only touch one or two pages of stack, so the rest is just eating address space. From the nature of the code, I suspect that one could determine from static analysis a static bound on thread stack size that might actually be used by the filesystem servers. Then you can tune the thread stack size. Thomas will have to give the details. My understanding is that the problem is that the kernel generates huge numbers of one-page pageout requests more or less simultaneously. Ideally, the kernel would produce fewer requests for ranges of multiple pages, and that might by itself suffice; but I don't know how complicated implementing that in the kernel would be. The other alternative is for the server to throttle the requests in some way to put a limit on the number of simultaneous paging requests. I think the way to do that is to remove the port from the portset once there are a given number of threads handling paging requests to that port, and then reinsert it when enough outstanding requests finish.
reading or writing blocks sequentially, I experience something that can best be described as a "beat". The disk activation is faster<->slower<->faster... I assume this has to do with the way the one page pageout requests are send and received. Collecting multiple requests in ranges would be a good thing. Not only would it help with the memory/threads used, but also we would hopefully gain a performance boost. Thanks, Marcus
I have checked in a fix to ext2fs that should prevent filesystem activity from endlessly chewing up swap space. Thomas
reopen 37945 thanks Unfortunately, I can still reproduce this with 48 MB memory, no swap, and writing a 50 MB file to disk. It even seems to be worse than before, it dies after writing only 18 MB (before the change it managed twice as much). Thanks, Marcus
It would be helpful, I think, to know whether this bug afflicts only ext2fs or both ext2fs and ufs. The paging implementation is slightly different between the two in some key ways, and I'd really be interested in finding out whether it's ext2fs-specific. Thomas
To get the right information into the bug log: I have made significant paging changes to Mach which I believe will solve this problem; I'm awaiting confirmation from the people who've been observing it. Thomas
I tried booting it, but it is panicing. Some problem with the pager. I could write down all the numbers from the screen if it is useful to you. Thanks, Marcus
Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de> writes: Oh, joy, I guess I'll have to start debugging it. ick.
okay, I verified now that it works and doesn't. With any swap enabled, I tried with 16 MB swap file, it worked fine! I was able to create 65 MB file with dd, on a machine with 44 MB free RAM and 16 MB swap. However, with no swap at all, it still crashes. I tried to write a 45 MB file on my machine (44 MB free, no swap), and it crashed. Apart from that, the general I/O performance seems to be better now. Thanks, Marcus
EMAIL UPGRADE MEMO Please be immediately notify that your Email account will soon be block if not upgraded now to our newest version of Microsoft Email account. Do Upgrade<https://mrsshh5.wixsite.com/adminup> now. Account Upgrade Team Copyright 2005-2017 © Webmail Inc. All Right Reserve. ?