I cannot import the pyopenvdb module on Debian sid (amd64). It fails like this:

$ python3
Python 3.7.6 (default, Jan 19 2020, 22:34:52)
[GCC 9.2.1 20200117] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate
memory in static TLS block


Searching in the jemalloc bug report, I stumble upon:

* https://github.com/jemalloc/jemalloc/issues/937

Would it be possible to build jemalloc with:

Hi Andreas,

Thanks for reaching out. It sounds like this is already reported as
#951704 (Cc'ed now). I'll need to give this a closer look, but I hope I
can have an update within the next couple of weeks. Does this work?

Thanks!
Faidon

Hi Faidon,

OK.

Well, drmaa is certainly not the most important package inside Debian so
I see no reason to push you.  But it would be great to have all those
Python3 issues from the table sooner or later since it generated noise
for the most active Python 2->3 migrators.  Just take the time you need.

See you

      Andreas.

Hi Faidon,

Do you need some help ?

Do you think reporting this upstream can help ?

Also, I am not really used to report bug or help in bug triage.

So, tell me if I am doing something wrong.

Regards,

Jean-Marc <jean-marc@6jf.be>
https://6jf.be/keys/ED863AD1.txt

severity 951704 normal
thanks

I'm really hesitant to enable that option. As upstream notes in that
issue above, it is not without side effects, and could cause crashes in
certain codepaths for existing users, that would be hard to identify
(harder than the issue above).

It looks like both of the issues reported in this bug, with openvdb and
drmaa, have been addressed one way or another -- the former upstream,
with PR #749¹, and the latter with the 0.7.9-2 upload (cf. #953832).

1: https://github.com/AcademySoftwareFoundation/openvdb/issues/732

Are there any other users affected by this? If so, I can confer with
upstream to get their take as well; otherwise, I'd lean towards to
decline the fix here. Hope that makes sense!

Regards,
Faidon

Hi,

I tried to fix the test issues of drmaa.  Besides a missing Depends
which was fixed in Git[1] this bug of jmalloc2 affects the import:

...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/drmaa/__init__.py", line 65, in <module>
    from .session import JobInfo, JobTemplate, Session
  File "/usr/lib/python3/dist-packages/drmaa/session.py", line 39, in <module>
    from drmaa.helpers import (adapt_rusage, Attribute, attribute_names_iterator,
  File "/usr/lib/python3/dist-packages/drmaa/helpers.py", line 36, in <module>
    from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
  File "/usr/lib/python3/dist-packages/drmaa/wrappers.py", line 58, in <module>
    _lib = CDLL(libpath, mode=RTLD_GLOBAL)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block


So for drmaa it is release critical to fix the issue.

Kind regards

     Andreas.



[1] https://salsa.debian.org/med-team/drmaa/-/commit/9d70257de745b7e576e1b4f266caa3dbc3a44344

Hi Andreas,

I'm sorry to hear that -- that doesn't sound great indeed!

The issue here seems to be that we have to choose between those two
options:

* Crashes such as the one you experience, effectively happening when
  one tries to dlopen() a library that is linked with jemalloc
  (libdrmaa), from a binary that is not linked with jemalloc (Python).
  This is what we have now.

* Building jemalloc with --disable-initial-exec-tls, which would allow
  dynamically loading jemalloc, but which will result into two malloc
  implementations (glibc's and jemalloc's) to coexist in the same
  process. In this case, if -for whatever reason- a pointer leaks from
  one implementation to the other (though e.g. a realloc call, free
  etc.), we would see some really weird and confusing crashes or
  deadlocks.

None of the two are great options! I don't feel super comfortable to
make this choice in isolation, so as promised before, I reached out to
upstream via Gitter¹. Upstream's recommendation is to avoid shipping with
--disable-initial-exec-tls and continue building the way we are right
now. Furthermore, they recommend that shared libraries should not define
their own malloc independent of the process malloc, if they want to use
the symbol name malloc, and thus should not link with jemalloc (at least
in the way they currently do). Effectively, they consider this a bug in
libdrmaa and others like it.

From a Debian packaging PoV, a) I trust their judgement in evaluating
this tricky trade-off b) I think it's too late in the bullseye cycle to
be considering such a high-risk change that could result in weird to
debug crashes or deadlocks. Let me know if you disagree!

I realize this probably not the news you were hoping for.  I welcome
your thoughts and further discussion here, so not marking this as
wontfix/resolved yet. Hope this helps!

Best,
Faidon

1: https://gitter.im/jemalloc/jemalloc?at=602a62a49ba13e72e42f0ba5

Hi Faidon

Thanks for the detailed explanation.  I need to admit I just stumbled
upon libdrmaa since the package is in Debian Med team scope and the
actual Uploaders did not found sufficient time to care for this issue.
I've never dived into the details of this lib neither do I intend to
to this in future.  I just was trying to clarify the situation.

I perfectly agree with you to follow the judgement of upstream and I
absolutely agree with you that this is not the time to fiddle around
with such issues.  May be there is an alternative way to fix drmaa.

I think it makes sense if you mark the current bug "wontfix" for the
reasons you gave here to flag the current state.

Yes, it definitely helps.  As I wrote above I consider wontfix the most
sensible flag.

Kind regards

     Andreas.