Most likely you're already aware that the memory footprint of fsck.fat is quite
large. I'm using it on an embedded system with 16MB of free memory and it
triggers an OOM when checking 8GB partitions.
fsck.fat does three big memory allocations:
4*#clusters bytes to read the first fat (assuming FAT32)
4*#clusters bytes to read the second fat (if it exists)
sizeof(DOS_FILE*)*#clusters bytes to cluster_owner mapping to files
Between the second and the third one, there is actually a free(), so the
maximum real size is 8*#clusters for a 32-bit system (12*#clusters for a 64-bit
system).
The maximum #clusters on a FAT32 filesystem is 255M. So that leads to a maximum
memory allocation of 3GB (which will be reached for an 8TB partition).
Although these maxima are a bit extreme, I already run into trouble with 8GB
partitions with a cluster size of 4K (standard used by Windows if the partition
size is just under 8GB). 2M clusters will occupy 16M of memory and this invokes
the OOM killer.
I have the following ideas for fixing this issue.
For the first two allocations (the FAT tables), there is a relatively easy
workaround: instead of malloc()ing it, it can be mmap()ed. This will allow the
memory to be freed again by the OS, because the OS can just read it from disk
again if it is accessed later on. Clearly, this is only possible with the -w and
-y options, because there is no way to make fixes without writing to disk.
The third allocation (the cluster-to-file table) is a lot more difficult to
solve. This table is used all the time while checking the filesystem, and it is
also updated during the filesystem check. To avoid allocating that large table,
we'd need to keep track of which clusters are in use by some file (in a bitmap,
which would be 32 times smaller, so maximum 32MB and realistically 2MB), and
then only when a double-use cluster is detected iterate over all files again to
find which one was the first one.
I personally don't really feel comfortable with this cluster_owner change. But
I would be able to implement the mmap() solution.
So, what do you think about this? Would you be willing to accept patches for
the mmap() conversion? Any hints about how to approach it? Can I assume that
mmap() is always available?