Hello, we are using Debian 7.1 on amd64. We installed a multimaster replication setup. Now if we modifies some attributes and groups-memberships and the memory use of the slapd on the 'master' increase extremely (>10G) until out of memory. This occurs only if the two servers in sync. If we disable the connection (i.e. iptables) between the servers, the memory usages isn't growing. We use the online config. In attachment is the corresponding slapd.conf. best regards Thomas Sesselmann
--On Tuesday, October 01, 2013 1:10 PM +0200 Thomas Sesselmann <thomas.sesselmann@uni-jena.de> wrote: Distribution packages are not meant to be used for production services. There is even an FAQ about this fact on written by one of the previous Debian LDAP packagers on the OpenLDAP website: <http://www.openldap.org/faq/data/cache/1456.html> I would strongly advise you to build your own package of OpenLDAP for production use that live in their own location (/usr/local, /opt, etc). I suggest OpenLDAP 2.4.36 linked to OpenSSL for security reasons. In addition, you may wish to read the OpenLDAP changelog while your packages are building: <http://www.openldap.org/software/release/changes.html> Regards, Quanah
If you don't have any useful responses to this bug (for example, linking to an ITS where this particular issue has been fixed or discussed), or want to help fixing or maintaining the openldap packages in Debian, please refrain from responding. The maintainers of distribution packages in distributions like Debian do intend for them to be used in production use, and openldap is no exception. Otherwise, we wouldn't bother making the packages in the first place.
--On Tuesday, October 01, 2013 12:12 PM -0700 Don Armstrong <don@debian.org> wrote: I guess our definitions of "useful" differ. I'm offering advice that will allow the end user to have a working server. That, to me, is useful. Funny. I suggest you read the FAQ I linked to. It was written for a reason *by* one of the Debian maintainers of the OpenLDAP package. And I also linked to the changelog, which lists all the variety of fixes to OpenLDAP since 2.4.31 was released 1.5 years ago. If Debian could keep a current build available to its users, then maybe I wouldn't have to constantly advise people not to use the Debian package. But as it stands, what Debian provides is not usable for a production service, and it should be avoided at all cost.
Ten years of experience with this package shows me that there is no reason to expect the new versions upstream recommends to be any less buggy than the old ones you constantly slag Debian in our own BTS for shipping. Maybe if there was some evidence of OpenLDAP getting less buggy over time, there would be reason to agree with this assessment. As it is, all the package's history shows is that there is no shortage of critical bugs in any version of the software. It is the height of absurdity to suggest that every other historical version of slapd has had serious bugs, but *this* version, which is brand new and has had no burn-in in which to even discover bugs, is bug-free. That bugs being reported in the Debian version of the package are known bugs is no evidence at all that the current upstream version is better suited to production use. As for that FAQ, Russ is entitled to his opinion about the best way to deploy an OpenLDAP server, as are you. But Russ is no longer a comaintainer of this package in Debian, and it is patently *false* to say that the distribution packages are not *meant* to be used for production services. That upstream has yet to deliver a reasonably bug-free version of the software that users can rely on for any extended period of time is no reflection on the intent of the Debian packagers. Your persistent badmouthing of Debian, its package maintainers, and its processes in our own bug tracker is absolutely uncalled for. If you aren't actually interested in helping Debian improve its packages, then just go away.
--On Tuesday, October 01, 2013 2:33 PM -0700 Steve Langasek <vorlon@debian.org> wrote: Yes, shockingly, software evolves over time. And depending on the feature, yes, some things have had issues needing to be resolved more than others. Has back-bdb/hdb been stable for a long time? Yes. I've back-bdb since 2.2, and back-hdb since 2.3 on. Has MMR been stable? Not particularly. Delta-syncrepl MMR (Introduced in 2.4.27) has been quite stable, however. Essentially if Debian even had 2.4.33 rather than 2.4.31 available, then I doubt you'd see much if any traffic on bugs, as long as the end user used delta-syncrepl MMR if they were doing multi-master. If this is false, I've yet to see any evidence of Debian being capable of producing a package suitable for running a production service. As I said before, if Debian can do that, then I'll stop telling people to stop using it. This is no different than what I tell people running RHEL, SLES, etc. I'm really not aware of *any* distribution that can competently provide an OpenLDAP package to its community. RHEL is many ways is *much* worse than Debian, not only because of the age of their product, but because they also link to the god-awful MozNSS libraries. GnuTLS is at least a step up from that. I'm trying to provide worthwhile advice to someone experiencing problems directly related to using the Debian package. As long as Debian only has 2.4.31 available to its users, then the *only* reasonable advise is to not use that package. Period. If you are blind to that *fact* I cannot help that. If you want to do something about it, since you *are* one of the packagers, then backport a newer version. Either way, you're picking a fight where there isn't one, and you have the ability to resolve the issue for all your users.
Hi Thomas, I think you might be experiencing ITS#7292. What do you think? If you bring up a new server with the same configuration and an empty database and allow it to start replicating, like in the ITS, do you observe the same memory usage? http://www.openldap.org/its/index.cgi/Software%20Bugs?id=7292 It was fixed upstream in 2.4.32 by a series of commits: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=shortlog;h=dc912776;hp=65c0cd19 If it's easy for you to rebuild a package and deploy it in a testing environment, you might try doing so with those changes integrated. I tried to reproduce your report but wasn't successful yet. Is it possible for you to reduce the testcase any further? For example, can you still trigger it if you omit some of the overlays?
Hi Ryan,
On 02.10.2013 00:52, wrote Ryan Tandy:
I don't know, on initial update/sync there are no problems.
The problem exists only if the two master servers are in sync
with the persist connection (syncrepl type=refreshAndPersist).
We now build an independent test-environment. So we can install there
test packages ...
Rebuild a source packages with some diffs is not impossible, but will
spend some time, because we have to learn it first ;)
The modules are loaded but not used in the configuration,
if this should change something?
We modifies the member of some groups (delete/add). There are much users
in the groups (>20000), perhaps the result is so worse.
Here are the Skript we used:
-------modify.pl ----------------
#!/usr/bin/perl
use strict;
use warnings;
use Net::LDAPS;
use Net::LDAP::Util qw(ldap_error_text);
my $hostname = ['ldaptest2.rz.uni-jena.de'];
my $binddn = 'ou=local,dc=uni-jena, dc=de';
my $auth = 'uid=xxxy';
my $passwd = 'xxxx';
my $basedn = "ou=users,dc=uni-jena,dc=de";
my $ldap = Net::LDAP->new($hostname) or die "Unable to connect to LDAP server $hostname: $@\n";
my $mesg = $ldap-> start_tls() or die "Unable to StartTLS: $@\n";
$ldap->bind( "$auth,$binddn", password => $passwd, version => 3 ) or die "Binding to the LDAP-Server";
##searching on LDAP
my $searchresult = $ldap->search(base =>$basedn, filter => "(&(uid=*)(mail=*)(|(mailHost=mail1.rz.uni-jena.de)))", attrs => ['*', 'memberOf']) or die "Searching the
LDAP-Server";
print STDOUT "(".$searchresult->count.")\n";
sub in_array {
my ($arr,$search_for) = @_;
return grep {$search_for eq $_} @$arr;
}
### For all LDAP-USERS ###
foreach my $entry ($searchresult->entries) {
my @AddArray = [];
my @ReplaceArray = [];
my @DeleteArray = [];
print STDOUT $entry->dn, "\n";
my $memberOfs = $entry->get_value('memberOf', asref => 1);
### DELETE IN GROUP SMTP ###
if(in_array($memberOfs,'cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de')){
$ldap->modify('cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de', delete => {member => [$entry->dn]}) or die "Delete group SMTP";
}
### DELETE IN GROUP SMTP-AUTH ###
if(in_array($memberOfs,'cn=smtp-auth,ou=local,ou=groups,dc=uni-jena,dc=de')){
$ldap->modify('cn=smtp-auth,ou=local,ou=groups,dc=uni-jena,dc=de', delete => {member => [$entry->dn]}) or die "Delete group SMTP-AUTH";
}
### DELETE IN GROUP IMAP ###
if(in_array($memberOfs,'cn=imap,ou=local,ou=groups,dc=uni-jena,dc=de')){
$ldap->modify('cn=imap,ou=local,ou=groups,dc=uni-jena,dc=de', delete => {member => [$entry->dn]}) or die "Delete group IMAP";
}
### WRITE USER MODIFICATIONS TO LDAP ###
push @{$ReplaceArray[0]}, mailhost => "mail1-deleted";
$ldap->modify( $entry->dn, changes => [add => @AddArray, replace => @ReplaceArray, delete => @DeleteArray]) or die "Modify User on LDAP";
}
$ldap->unbind();
---------------------------------------
best regards
Thomas
Hi Thomas, Sorry it took me so long to get back to you. I think the problem is that your slapd.conf uses LDAP Sync replication and not delta-syncrepl. I missed that at first because you have an accesslog database configured, so I assumed you were using delta-syncrepl, but your syncrepl consumers are actually not configured for it. I created a basic provider-consumer setup (configuration files attached) and the memory usage was low and stable. When I changed from delta-syncrepl back to LDAP Sync, the memory usage climbed quickly. I reproduced the same behaviour with recent (2.4.36) code from upstream, so it's not a Debian-specific problem. As per slapd.conf(5), the change to switch from delta-syncrepl back to LDAP Sync is to remove the logbase, logfilter, and syncdata attributes from the syncrepl configuration. Reversing that change, I think it might help if you added those three attributes to your syncrepl configurations. See the OpenLDAP admin guide and slapd.conf(5) for details. Your syncprov overlay also doesn't have a syncprov-checkpoint directive, which is optional but useful in case of a crash or unclean shutdown. The reason LDAP Sync is not efficient for your use case is that it transfers entire entries and not just changed attributes. So, if you add or remove a member from a group, the entire list of members for that group has to be transferred, which is of course pretty bad when your groups have a lot of members. Delta-syncrepl is better because it only transfers changed attributes. Does this help?
Hi Ryan,
Am 11.10.2013 00:44, schrieb Ryan Tandy:> Hi Thomas,
we try to configure Delta-syncrepl and run in next issue :(
The slapd on the slaves crashes immediately after modifying a group
on memberof overlay. I can try to start in debug mode an the slave
crashes after the next entry:
ldap-slave# slapd -u openldap -g openldap -F /etc/ldap/slapd.d -d -1
....
525d3d02 => access_allowed: search access to "cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de" "objectClass" requested
525d3d02 <= root access granted
525d3d02 => access_allowed: search access granted by manage(=mwrscxd)
525d3d02 <= test_filter 6
525d3d02 send_ldap_result: conn=-1 op=0 p=3
525d3d02 send_ldap_result: err=0 matched="" text=""
525d3d02 ==> unique_modify <cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de>
525d3d02 unique_modify: administrative bypass, skipping
525d3d02 => bdb_entry_get: ndn: "cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de"
525d3d02 => bdb_entry_get: oc: "(null)", at: "(null)"
525d3d02 bdb_dn2entry("cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de")
525d3d02 => bdb_entry_get: found entry: "cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de"
525d3d02 bdb_entry_get: rc=0
525d3d02 hdb_modify: cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de
525d3d02 bdb_dn2entry("cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de")
525d3d02 bdb_modify_internal: 0x00000056: cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de
525d3d02 <= acl_access_allowed: granted to database root
525d3d02 bdb_modify_internal: softdel member
525d3d02 dnMatch 1
"cn=ldapadmin,ou=local,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 1536
"uid=ku49qeq,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 1536
"uid=ku49haf,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 1536
"uid=ku49lan,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 1536
"uid=ku49ded,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 1536
"uid=ku49bel,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 dnMatch 0
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
"uid=ko35rot,ou=users,dc=uni-jena,dc=de"
525d3d02 bdb_modify_internal: replace entryCSN
525d3d02 bdb_modify_internal: replace modifiersName
525d3d02 bdb_modify_internal: replace modifyTimestamp
525d3d02 => key_change(DELETE,56)
525d3d02 bdb_idl_delete_key: 56
525d3d02 <= key_change 0
525d3d02 => key_change(ADD,56)
525d3d02 bdb_idl_insert_key: 56
525d3d02 <= key_change 0
525d3d02 => entry_encode(0x00000056):
525d3d02 <= entry_encode(0x00000056):
525d3d02 hdb_modify: updated id=00000056 dn="cn=smtp,ou=local,ou=groups,dc=uni-jena,dc=de"
525d3d02 send_ldap_result: conn=-1 op=0 p=3
525d3d02 send_ldap_result: err=0 matched="" text=""
slapd: ../../../../../servers/slapd/overlays/memberof.c:1465: memberof_res_modify: Assertion `0' failed.
Abgebrochen
Sounds like ITS#7487 (http://www.openldap.org/its/index.cgi/Software%20Bugs?id=7487;page=6)
best regards
Thomas
--On Tuesday, October 15, 2013 4:30 PM +0200 Thomas Sesselmann
<thomas.sesselmann@uni-jena.de> wrote:
Hi Thomas,
I'm going to re-iterate again that you will need to upgrade to a current
release if you want to do multi-master replication.
I would also note that you'll need to get the recent fixes to
slapo-memberof around replication that are going into OpenLDAP 2.4.37:
Fixed slapo-memberof to not replicate internal ops (ITS#7710)
Hi Thomas, I agree, it does sound like the same bug. Quanah is correct that several fixes are needed to 2.4.31 in order to have working MMR (and probably several more depending on the overlays you use), and I agree that the fastest way to a working server is probably to build your own from recent upstream code. That said, if you prefer to try to achieve a package based on 2.4.31 with patches applied to satisfy your requirements, I'm happy to provide any help I can. It would probably involve a few iterations of patching and testing (and might be better discussed away from this particular bug since we already diverged from the original subject: memory leak with LDAP Sync). To build a package with some local changes applied, the basic workflow looks like: apt-get build-dep openldap && apt-get source openldap && cd openldap-2.4.31 && <make your changes> && dch -l jena && dpkg-buildpackage -us -uc See http://debian-handbook.info/browse/stable/debian-packaging.html for more guidance. Your other option would be to wait until a newer upstream version is introduced into unstable and then install that, possibly rebuilding it for wheezy if the library dependencies changed.
Hi Ryan, Am 18.10.2013 07:54, schrieb Ryan Tandy: We prefer to use packages for our servers. Also we see the need of a newer upstream release version. thanks for this help This would be the best option for us. Did you know when a new upstream version in unstable or experimental would be released? Until this we have to try to build our own package at 2.4.36 (the first trial is failed). best regards Thomas
--On Friday, October 18, 2013 1:39 PM +0200 Thomas Sesselmann <thomas.sesselmann@uni-jena.de> wrote: 2.4.37 was released on Sunday. I would note that other people have had success building on ubuntu12 with: 18:05] <paco11> i use checkinstall > ./configure ..... ; make depend ; make ; sudo checkinstall -D --showinstall --pkgname=openldap --maintainer= --pkgversion=2.4.37 --pkgrelease=1 --pkglicense=GPL --pkggroup=checkinstall --requires=make,automake,gcc,libtool,libperl-dev,libdb5.1-dev,libssl-dev,libsasl2-dev [18:06] <paco11> it's easy to use [18:09] <paco11> and i modified 2 files from slapd debian package: /etc/default/slapd & /etc/init.d/slapd to have "/usr/local". and then " update-rc.d slapd defaults" [18:09] <paco11> and nothing else You would of course need to use the configure options most relevant to you.
FYI; 2.4.39 was uploaded to unstable a few days ago.
Cheers,
Moritz
tags 725091 + confirmed retitle 725091 slapd: memory leak with non-delta syncrepl thanks Hi Thomas, The good news: the crash involving delta-syncrepl, mirrormode, and memberof (ITS #7487) is definitely fixed. I've reproduced with 2.4.31 and verified with 2.4.39. The bad news: the memory leak, using exactly the same configuration but with delta-syncrepl disabled, does seem to still exist. It's odd, because valgrind doesn't report any leaks, but the memory usage does increase continuously as long as I perform modifications. I'm going to keep looking at this and try to gather enough information for an ITS. In case you'd like to run 2.4.39 on wheezy, let me know if you require any assistance with backporting and building the package. Happy to help. thanks, Ryan
Lieber Benutzer Dein Passwort wird endlich abgelaufen. Klicken Sie hier<https://mail-unterstuetzung1280067543.jimdo.com/>, um Ihre E-Mail zu bestätigen. Vielen Dank Systemadministrator