There seems to be a rather important memory leak in e-d-s.  After
habing it running for a few days, here's what top(1) reports:

28781 roland 20 0 670m 235m 4288 S 0.0 23.3 0:39.73 evolution-data-server

  I wouldn't normally use the "important" severity for a memory leak,
but that one is so large that even my swap partitions get saturated
after a few days, which means I can't suspend to disk because there's
no room left.  Of course, I could kill e-d-s before suspend, but then
the Gnome panel crashes...

Roland.

Roland Mas, 2008-01-26 18:11:47 +0100 :

I did some more measuring with ps.  Both the VSZ and the RSS columns
grow by about 17 MB per hour, even when I don't touch the calendar
applet or run Evolution.  That's fairly large.

  On the other hand, it seems the leak is located in the code that
handles CalDAV.  If I disable my CalDAV calendars (I have two) and
restart e-d-s, then e-d-s seems to no longer grow.

  There's a bug in the upstream Bugzilla:
http://bugzilla.gnome.org/show_bug.cgi?id=510949 is about a memory
leak in e-d-s too.  I'm not sure it's the same leak, but it contains a
patch.

Roland.

notforwarded 462664
title 462664 evolution-data-server: multiple massive memory leaks in the caldav backend
thanks

[ Hi Roland, how're ya doing? ]

justification of the "notforwarded": upstream bug report is not related to caldav
(although it is, indeed, related to other memory leaks within E-D-S).


here follows my original bug report which by chance failed to be delivered to submit@bts
before I had a chance to notice Roland's earlier report on the same topic.
------------------
Package: evolution-data-server
Version: 2.22.3-1.1
Severity: important
Justification: try running your system with an E-D-S gobbling up 5G of memory ;)


After a day of use of an Evolution with a single CalDAV calendar, the memory
consumption of E-D-S is very high. Running E-D-S through valgrind yields
a few interesting tracks:

==10847== 505,964 (22,392 direct, 483,572 indirect) bytes in 311 blocks are
definitely lost in loss record 68 of 113
==10847==    at 0x4C20FEB: malloc (vg_replace_malloc.c:207)
==10847==    by 0x7B4092E: xmlXPathWrapNodeSet (in
/usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B4ED93: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B4CEF0: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B4D6AE: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B4FC74: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B529BB: xmlXPathEvalExpression (in
/usr/lib/libxml2.so.2.6.32)
==10847==    by 0xEEFFB8E: xpath_eval (e-cal-backend-caldav.c:497)
==10847==    by 0xEEFFF83: synch_slave_loop (e-cal-backend-caldav.c:700)
==10847==    by 0x900A4D3: (within /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0x9AB8FC6: start_thread (in /lib/libpthread-2.7.so)
==10847==    by 0x9D9D7CC: clone (in /lib/libc-2.7.so)

it seems that in function static gboolean
parse_report_response (SoupMessage *soup_message, CalDAVObject **objs, int
*len) from e-cal-backend-caldav.c, no-one attempts to free the
xmlXPathObjects that are returned by function xpath_eval() (the method to
free is xmlXPathFreeObject())

It does not appear that parse_report_response() frees memory in SVN
evolution either.


==10847== 251,005 (22,080 direct, 228,925 indirect) bytes in 184 blocks are
definitely lost in loss record 69 of 113
==10847==    at 0x4C20FEB: malloc (vg_replace_malloc.c:207)
==10847==    by 0x7B1222D: xmlNewNode (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B165B7: xmlNewDocNode (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B166A0: xmlNewDocRawNode (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B16744: xmlNewTextChild (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0xEEFFD95: synch_slave_loop (e-cal-backend-caldav.c:944)
==10847==    by 0x900A4D3: (within /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0x9AB8FC6: start_thread (in /lib/libpthread-2.7.so)
==10847==    by 0x9D9D7CC: clone (in /lib/libc-2.7.so)
==10847==

==10847== 49,102 (3,456 direct, 45,646 indirect) bytes in 36 blocks are
definitely lost in loss record 75 of 113
==10847==    at 0x4C20FEB: malloc (vg_replace_malloc.c:207)
==10847==    by 0x7B11DBA: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B14D56: xmlSetProp (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0xEEFFDC2: synch_slave_loop (e-cal-backend-caldav.c:946)
==10847==    by 0x900A4D3: (within /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0x9AB8FC6: start_thread (in /lib/libpthread-2.7.so)
==10847==    by 0x9D9D7CC: clone (in /lib/libc-2.7.so)
==10847==

==10847== 38,102 (3,360 direct, 34,742 indirect) bytes in 28 blocks are
definitely lost in loss record 84 of 113
==10847==    at 0x4C20FEB: malloc (vg_replace_malloc.c:207)
==10847==    by 0x7B11CC6: xmlNewText (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B11D6B: xmlNewDocText (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B11E23: (within /usr/lib/libxml2.so.2.6.32)
==10847==    by 0x7B14D56: xmlSetProp (in /usr/lib/libxml2.so.2.6.32)
==10847==    by 0xEEFFDC2: synch_slave_loop (e-cal-backend-caldav.c:946)
==10847==    by 0x900A4D3: (within /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0x9AB8FC6: start_thread (in /lib/libpthread-2.7.so)
==10847==    by 0x9D9D7CC: clone (in /lib/libc-2.7.so)

These three stacks all involve the function caldav_server_list_objects()
(apparently untouched in SVN). It seems to me that nothing ever ties the
root xmlNodePtr (from which all child nodes and attributes are created) to
the xmlDocPtr "doc"; hence when the document is freed using xmlFreeDoc(doc),
the root node (and its children) are lost.
line 934 might read instead
	root = xmlNewDocNode (doc, NULL, (xmlChar *)"calendar-query", (xmlChar*)NULL);


==10847== 3,455 (56 direct, 3,399 indirect) bytes in 1 blocks are definitely lost in loss record 85 of 113
==10847==    at 0x4C20FEB: malloc (vg_replace_malloc.c:207)
==10847==    by 0x6C73079: icalproperty_new_impl (in /usr/lib/libecal-1.2.so.7.2.0)
==10847==    by 0x6C5F28A: icalproperty_new_version (in //usr/lib/libecal-1.2.so.7.2.0)
==10847==    by 0x6C59727: e_cal_util_new_top_level (in /usr/lib/libecal-1.2.so.7.2.0)
==10847==    by 0xEEFE789: pack_cobj (e-cal-backend-caldav.c:1645)
==10847==    by 0xEEFF82C: caldav_create_object (e-cal-backend-caldav.c:1702)
==10847==    by 0x69EBD12: e_cal_backend_sync_create_object (in /usr/lib/libedata-cal-1.2.so.6.0.2)
==10847==    by 0x69EBDDB: (within /usr/lib/libedata-cal-1.2.so.6.0.2)
==10847==    by 0x82D9FA5: ORBit_small_invoke_adaptor (in /usr/lib/libORBit-2.so.0.1.0)
==10847==    by 0x82E927F: (within /usr/lib/libORBit-2.so.0.1.0)
==10847==    by 0x82E9839: (within /usr/lib/libORBit-2.so.0.1.0)
==10847==    by 0x82D3374: giop_thread_queue_process (in /usr/lib/libORBit-2.so.0.1.0)

the routine static char *
pack_cobj (ECalBackendCalDAV *cbdav, ECalComponent *ecomp)
carefully constructs an icalcomponent* (named "calcomp") and asks it to
serialize itself into a string ("objstr"). The string is returned and the
calcomp is dropped as if there was a JVM to pick it up.

This seems untouched in SVN.

pack_cobj(...) should delete the icalcomponent (calling
icalcomponent_free(); perhaps first calling g_strdup() on the objstr,
depending on the ownership rules of icalcomponent_as_ical_string().



==10847== 1,282,280 (1,228,896 direct, 53,384 indirect) bytes in 489 blocks
are definitely lost in loss record 100 of 113
==10847==    at 0x4C200FC: calloc (vg_replace_malloc.c:397)
==10847==    by 0x8FE9BE9: g_malloc0 (in /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0xEEFFFF6: synch_slave_loop (e-cal-backend-caldav.c:711)
==10847==    by 0x900A4D3: (within /usr/lib/libglib-2.0.so.0.1600.5)
==10847==    by 0x9AB8FC6: start_thread (in /lib/libpthread-2.7.so)
==10847==    by 0x9D9D7CC: clone (in /lib/libc-2.7.so)
==10847==
==10847==

in function synchronize_cache(), the result of calling
caldav_server_list_objects (the array of CalDAVObject named "sobjs") is
never freed. It ought to be freed at the end of the for loop "see if we have
to upate or add some objects", line 1305.
It is possible that any server objects found with object->status != 200 are
dropped without destruction.

This seems untouched in SVN.


********************************************
********************************************

furthermore, this only addresses the 4-odd megabytes that were definitely
lost in that session. Valgrind, ran through
  valgrind --leak-check=full \
		/usr/lib/evolution/evolution-data-server-2.22 \
		--oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_InterfaceCheck \
		--oaf-for-fd=25
also reports 280MB of "still reachable" memory (which is still very probably
a case of 'hot leak', a.k.a failure to unreference obsolete things).
I will re-run valgrind with --leak-check=full --show-reachable=yes to
further investigate.

after running valgrind again with --leak-check=full and
--show-reachable=yes, it appears that a great deal of the
formally still reachable, but probably quite lost memory originates from
e_cal_backend_cache_get_components() (e-cal-backend-cache.c line 473
specifically, which calls icalparser_parse_string())
It seems this function is identical in SVN

==1641== 5,393,785 bytes in 181,741 blocks are still reachable in loss
record 102 of 107
==1641==    at 0x4C1FFAB: malloc (vg_replace_malloc.c:207)
==1641==    by 0x9D16DE1: strdup (strdup.c:43)
==1641==    by 0x6C61604: icalvalue_set_text
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C6167D: icalvalue_new_text
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C769E2: icalvalue_new_from_string_with_error
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C688BA: icalparser_add_line
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68CB8: icalparser_parse
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68F20: icalparser_parse_string
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x69DE5D6: e_cal_backend_cache_get_components
(e-cal-backend-cache.c:473)
==1641==    by 0xF4D20EA: synch_slave_loop (e-cal-backend-caldav.c:1259)
==1641==    by 0x8FDDCA3: g_thread_create_proxy (gthread.c:635)
==1641==    by 0x9A8C016: start_thread (pthread_create.c:297)
==1641==
==1641==
==1641== 7,568,054 bytes in 39,280 blocks are still reachable in loss
record 103 of 107
==1641==    at 0x4C1F0BC: calloc (vg_replace_malloc.c:397)
==1641==    by 0x8FBFA52: g_malloc0 (gmem.c:151)
==1641==    by 0x8FABCC8: g_hash_table_new_full (ghash.c:358)
==1641==    by 0x8FCE136: g_scanner_new (gscanner.c:228)
==1641==    by 0x6ED64CA: e_sexp_new
(in /usr/lib/libedataserver-1.2.so.9.1.0)
==1641==    by 0x69DF470: e_cal_backend_sexp_new
(e-cal-backend-sexp.c:1362)
==1641==    by 0x69E61E7: impl_Cal_getQuery (e-data-cal.c:415)
==1641==    by 0x82B3FA5: ORBit_small_invoke_adaptor
(in /usr/lib/libORBit-2.so.0.1.0)
==1641==    by 0x82C327F: (within /usr/lib/libORBit-2.so.0.1.0)
==1641==    by 0x82C3839: (within /usr/lib/libORBit-2.so.0.1.0)
==1641==    by 0x82AD374: giop_thread_queue_process
(in /usr/lib/libORBit-2.so.0.1.0)
==1641==    by 0x82ADB4E: (within /usr/lib/libORBit-2.so.0.1.0)
==1641==
==1641==
==1641== 10,416,032 bytes in 325,501 blocks are still reachable in loss
record 104 of 107
==1641==    at 0x4C1FFAB: malloc (vg_replace_malloc.c:207)
==1641==    by 0x6C7763A: pvl_new_element
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C776A1: pvl_push (in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C6864C: icalparser_add_line
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68CB8: icalparser_parse
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68F20: icalparser_parse_string
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x69DE5D6: e_cal_backend_cache_get_components
(e-cal-backend-cache.c:473)
==1641==    by 0xF4D20EA: synch_slave_loop (e-cal-backend-caldav.c:1259)
==1641==    by 0x8FDDCA3: g_thread_create_proxy (gthread.c:635)
==1641==    by 0x9A8C016: start_thread (pthread_create.c:297)
==1641==    by 0x9D665BC: clone (in /usr/lib/debug/libc-2.7.so)
==1641==
==1641==
==1641== 11,273,560 bytes in 281,839 blocks are still reachable in loss
record 105 of 107
==1641==    at 0x4C1FFAB: malloc (vg_replace_malloc.c:207)
==1641==    by 0x6C778DA: pvl_newlist (in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C6A093: icalproperty_new_impl
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68617: icalparser_add_line
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68CB8: icalparser_parse
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68F20: icalparser_parse_string
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x69DE5D6: e_cal_backend_cache_get_components
(e-cal-backend-cache.c:473)
==1641==    by 0xF4D20EA: synch_slave_loop (e-cal-backend-caldav.c:1259)
==1641==    by 0x8FDDCA3: g_thread_create_proxy (gthread.c:635)
==1641==    by 0x9A8C016: start_thread (pthread_create.c:297)
==1641==    by 0x9D665BC: clone (in /usr/lib/debug/libc-2.7.so)
==1641==
==1641==
==1641== 13,541,304 bytes in 241,809 blocks are still reachable in loss
record 106 of 107
==1641==    at 0x4C1FFAB: malloc (vg_replace_malloc.c:207)
==1641==    by 0x6C6A079: icalproperty_new_impl
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68617: icalparser_add_line
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68CB8: icalparser_parse
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68F20: icalparser_parse_string
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x69DE5D6: e_cal_backend_cache_get_components
(e-cal-backend-cache.c:473)
==1641==    by 0xF4D20EA: synch_slave_loop (e-cal-backend-caldav.c:1259)
==1641==    by 0x8FDDCA3: g_thread_create_proxy (gthread.c:635)
==1641==    by 0x9A8C016: start_thread (pthread_create.c:297)
==1641==    by 0x9D665BC: clone (in /usr/lib/debug/libc-2.7.so)
==1641==
==1641==
==1641== 36,754,968 bytes in 241,809 blocks are still reachable in loss
record 107 of 107
==1641==    at 0x4C1FFAB: malloc (vg_replace_malloc.c:207)
==1641==    by 0x6C764A9: icalvalue_new_impl
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C6166A: icalvalue_new_text
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C769E2: icalvalue_new_from_string_with_error
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C688BA: icalparser_add_line
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68CB8: icalparser_parse
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x6C68F20: icalparser_parse_string
(in /usr/lib/libecal-1.2.so.7.2.0)
==1641==    by 0x69DE5D6: e_cal_backend_cache_get_components
(e-cal-backend-cache.c:473)
==1641==    by 0xF4D20EA: synch_slave_loop (e-cal-backend-caldav.c:1259)
==1641==    by 0x8FDDCA3: g_thread_create_proxy (gthread.c:635)
==1641==    by 0x9A8C016: start_thread (pthread_create.c:297)
==1641==    by 0x9D665BC: clone (in /usr/lib/debug/libc-2.7.so)

line #473 is calling icalparser_parse_string(), which parses a string
and builds a complex icalcomponent* structure, which is then to be
wrapped into an ECalComponent object and stashed into a list. The
ECalComponent's destructor is responsible for deallocating the
icalcomponent* structure.

Line 1259 of e-cal-backend-caldav.c falls (again) within static void
synchronize_cache().
synchronize_cache() attempts to match the GList of ECalComponent
obtained from the ecal backend cache with items retrieved from the
server (using caldav_server_list_objects).

Initially, a hash table of all cached ECalComponent entries is built
(L1262-L1274), without incrementing the ECalComponent refcounts.
For each (server side) CalDAVObject, a match is attempted with the hash
table.
    Each hit causes synchronize_object() to be called, in order to
replicate the remote state into the local ECalComponent. The
ECalComponent is also removed from the GList (and no refcount activity
happens) [**]
    Each miss causes a leak of the CalDAVObject (image of the remote),
as previously noted. The ECalComponent remains in both the GList and the
GHashTable.
At this point, the GList still contains all the ECalComponent for which
either no match from server side was found, or for which matches failed.
Each of these ECalComponent instances is removed from the backend cache,
and if a notification needs to be raised, it is raised. Finally, the
ECalComponent is unref'd.

In the end, both the GList and the GHash are destroyed. In my copy of
libglib2.0-0, neither is unreffing the contained values, since the hash
table is built with a NULL last parameter.

Sketch of a solution:
    * each addition to the hash table should g_object_ref(comp)
    * the hash table should be built with g_object_unref as its last
parameter
    * caldav_object_free(object, FALSE) should also be called in the
"object->status != 200" case.
    * the sobjs lists should be g_free()'d.

On jeu., 2008-11-13 at 21:41 +0100, Cyrille Chépélov wrote:

Hey,

do you still experience such huge memleaks in eds/caldav? If so, could
you open a new bug upstream and provide valgrind logs, using 2.30?

Cheers,