#502065 Disproportionate use of RAM on big GPX files

Package:
gpscorrelate
Source:
gpscorrelate
Description:
correlates digital photos with GPS data filling EXIF fields (command line)
Submitter:
Enrico Zini
Date:
2019-10-22 11:57:07 UTC
Severity:
normal
#502065#5
Date:
2008-10-13 09:41:49 UTC
From:
To:
Hello,

I am back from a 2 weeks vacation and I'm trying to gpscorrelate the
photos.  I have a 77MB uncompressed GPX file, and even when trying to
correlate a few photos, gpscorrelate eats all the available ram on my
laptop.

I was assuming that the task of gpscorrelate would be simple enough not
to require inordinate resources even for such a big GPX file, but it
turned out that I was wrong.  Now I'll try to split the GPX data into
smaller chunks and kludge things into place one way or another.

Ciao,

Enrico

#502065#10
Date:
2019-10-22 11:43:28 UTC
From:
To:
I looked into the RAM usage with large .gpx files and the problem is libxml2.
A test file with 840K points (115.0 MiB file size) used 848.6 MiB of heap in
libxml2 and only 45 MiB of heap for everything else (on i686; all figures
exclude allocation overhead). gpscorrelate's internal requirements are 48 bytes
per point, which is 38.5 MiB . Even if gpscorrelate reduced its use in half, it
would result in only a 2% reduction in peak usage. I'm not sure there's
anything that can be done about that beyond switching to a different XML
parser.

You can also use the new feature to load multiple gpx files. By splitting
the XML file into multiple pieces memory usage is reduced because the libxml2
memory structures are freed once each file has loaded.