Hi,

During a rebuild of all packages in bullseye, your package failed to build
on amd64.

Relevant part (hopefully):
http://qa-logs.debian.net/2021/04/06/sagemath_9.2-2_testing.log

A list of current common problems and possible solutions is available at
http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!

If you reassign this bug to another package, please marking it as 'affects'-ing
this package. See https://www.debian.org/Bugs/server-control#affects

If you fail to reproduce this, please provide a build log and diff it with mine
so that we can identify if something relevant changed in the meantime.

About the archive rebuild: The rebuild was done on EC2 VM instances from
Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
failed build was retried once to eliminate random failures.

Has anyone been able to reproduce this? Attempting to build Sage in a
fresh unstable environment succeeds for me; perhaps the build failure
was spurious.

Hi Scott,

* John Scott <jscott@posteo.net> [2021-05-03 00:35]:

I triggered reproducible builds yesterday:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/sagemath.html

The problem Lucas found is:

* Lucas Nussbaum <lucas@debian.org> [2021-04-07 08:51]:

Success: 41 tests failed, up to 200 failures are tolerated
Success: 5 tests failed, up to 200 failures are tolerated

The 200 is set in debian/rules:

https://salsa.debian.org/science-team/sagemath/-/commit/6088e9f2abc7ae99a8d07760ceee6fb6aac7bb54

and sounds a little arbitrary. Sadly the state upstream seems not to be
much better:

https://github.com/sagemath/sage/tree/9.2

13 failing, 17 cancelled, and 70 successful checks

(I did not look into them.)

So I think the question is rather if the test suite gives an
appropriate view on the quality of the software. If it does, I assume
it is not appropriate for a Debian stable release. Contrary if we assume
the test suite being broken, we could disable it completely rather then
producing random FTBFS.

Cheers Jochen

Hi,

I had a look:
- sagemath build-depends on cython3
- /usr/bin/cython is in cython
- /usr/bin/cython3 is in cython3

so I see two ways out :

(1) change the b-dep to cython (from cython3) [and cython-dbg from
cython3-dbg, I guess] ;

(2) make so cython is called as cython3.

I can't give it a try before at least a week, so if someone wants to
dive in...

JP

Hi,

I tried to create a testing sbuild and compile sagemath 9.2-2 with it,
and it worked so unless I made a mistake in my setup, something made
that bug go away...

Can someone independently give it a try?

JP

Hi Julien,

* Julien Puydt <julien.puydt@gmail.com> [2021-05-17 09:01]:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/sagemath.html

Success: 40 tests failed, up to 200 failures are tolerated
Success: 5 tests failed, up to 200 failures are tolerated

so not much changed comparing to two weeks ago and my conclusion still
holds:

* Jochen Sprickerhof <jspricke@debian.org> [2021-05-04 13:22]:


Cheers Jochen

Hi,

Le lundi 17 mai 2021 à 15:20 +0200, Jochen Sprickerhof a écrit :

Well :

1) Upstream itself uses the testsuite in the sense of "shouldn't have
too many failing tests", and it still allows to detect when a build is
utterly broken, so we shouldn't disable it.

2) It's not an FTBFS, since the sources actually build to a set of
packages.

This points to lowering this bug severity and let sagemath go in
stable. Or at least not preventing it just for this reason.

JP

Hi Julien,

* Julien Puydt <julien.puydt@gmail.com> [2021-05-18 07:56]:

Debian doesn't necessarily need to do the same as upstream here.
Killing test src/sage/misc/cython.py
(14 in total)

That's this line:

https://sources.debian.org/src/sagemath/9.2-2/sage/src/sage/doctest/forker.py/#L1999

So if we add a

except: before the finally: we get:

EXCEPTION: [Errno 24] Too many open files

so running the build with a higher file limit:

ulimit -n 10240

will produce:

Error: 1940 tests failed, up to 200 failures are tolerated

Cheers Jochen

Hi,

Le mardi 18 mai 2021 à 15:31 +0200, Jochen Sprickerhof a écrit :

Upstream manages to ship version with no error because they ship
hundreds of deps to an exact version for which they fitted the
testsuite to pass. We ship those deps as separate packages, because
they're actually not sagemath-specific [look at the list, it's pretty
telling : https://people.debian.org/~thansen/debian-sage-status.html ],
so we might not have the exact same version, and that's enough to
trigger artificial failures.

I don't think we'll ever hit zero. At least not while their testsuite
framework is so... so like it is.

If we keep it with an allowed error rate, we detect the package is
*really* broken before shipping ; if we don't, we detect it is broken
after shipping, because it hurts the users and they complain.

Sagemath is definitely not bug-free, the Debian package for it isn't
bug-free either, but it is working and useful to users, and this
(bringing useful software to users) is what Debian is about.

If I look at the title of this bug, I think "Oh, just patch
src/bin/sage so it calls cython3 and not cython" (see my message #20),
but if you look at the exchange, then it's not clear at all what the
problem actually is.

The buildd logs (
https://buildd.debian.org/status/package.php?p=sagemath&suite=bullseye
), John Cross (message #10), myself (message #25) and the triggered RB
rebuild (message #30) all conclude the package doesn't FTBFS.

I would like to fix the problem, but at that point, I have no clue what
it really is about...

JP

* Julien Puydt <julien.puydt@gmail.com> [2021-05-18 17:47]:

I totally understand your point but please keep in mind that we need to
be able to rebuild packages in Debian as well. If we rebuild a package
and it FTBFS, we can't publish security updates, for example.
Also if the tests depend on so many external dependencies, changing one
of them could render sagemath FTBFS due to hitting the test limit. Would
you then simply bump the limit?

Totally agreed here, I would like to see sagemath in stable as well.
- Tests not being executed due to the open file limit ("Killing test" in
   the log). If you want to use the tests as an indicator if the build
   works, you should make sure the all tests are executed, otherwise 200
   tolerated failures is arbitrary.
- I found a number of segfaults in the tests, like:
   sage -t --long --random-seed=0 src/sage/interfaces/singular.py  # Killed due to segmentation fault
- Looking at the amd64 log of the buildd:
   Error: 202 tests failed, up to 200 failures are tolerated
   Yes: 202 tests failed, up to 400 failures are tolerated for rerun
   Success: 192 tests failed, up to 200 failures are tolerated
   does that mean we ran the test twice and it passed the second time
   cause there where 10 failures less? Can we be sure that this always
   succeeds? 192 is really close to 200.
- I still see cython: not found in the logs, so there are definitely
   tests broken due to that. Maybe it makes sense to define tests which
   are allowed to break and other which should succeed?

I am honestly not sure what the best way forward would be but I think
just downgrading the bug priority is not helping. On the other hand
given the current freeze policy and what will be the policy in stable,
what would be your actions if sagemath FTBFS there? Would you say at
some point it is unfit for a stable release or would you retry the build
and ignore the errors? Maybe it would make sense to ignore the tests
results for the version in stable?

Cheers Jochen

retitle #986527 Testsuite is too fragile - FTBFS randomly
thanks

Now I get the point and I have to agree ; let me retitle this report so
it states the actual problem.

Cheers,

JP

Hi,


I have been struggling with this for quite a while and I don't know how to fix it. It comes and goes and does not seem to affect vanilla upstream builds. This has impacted progress on the package since one cannot properly work on it when the test suite crashes. I tried asking upstream for help here and provided more details:

https://groups.google.com/g/sage-packaging/c/1G_3JiIcbvY


I agree, segfaults and the cython issue should be fixed. The number of failed tests always grows with time when dependencies change and sagemath is not adapted accordingly.

Best,

Tobias

Hi,

friendly ping on the sagemath issue as it will be dropped from testing
on the 18., otherwise.
As far as I read there are a number of proposals (downgrade, ignore test
results..) besides fixing the tests. I'm happy to help with this but
leave it to the maintainers how to approach this.

Cheers Jochen

* Tobias Hansen <thansen@debian.org> [2021-05-19 15:01]:

Hi,

Le mardi 08 juin 2021 à 14:58 +0200, Jochen Sprickerhof a écrit :

I've been convinced that getting a fragile sagemath in next stable
wouldn't be a good thing.

Cheers,

JP

You've put much more effort than I have into maintaining scientific
software in Debian, so I respect your opinion, but is it really
accurate to say that SageMath is fragile as a whole?

With respect to this particular issue, I'd like to share my perspective
wrangling with a package that poses a similar dilemma: GCC (I'm working
on packaging gcc-sh-elf). Like the status quo with SageMath in Debian,
GCC has a test suite where failures are normal, and in general it takes
an individual to watch out for what number of failures counts as "too
many." Rather than hardcode an arbitrary threshold for what number of
failing tests is acceptable, it seems that it's much better, and in the
interest of Debian ports and alternative build environments, to just
let the tests run for informative purposes.

This, I believe, is what the GCC team actually does; the test results
get sent to the team mailing list IIRC. Perhaps we should take a
similar philosophy towards the tests.

At least with GCC and DejaGnu, the test results get written out to a
file, so before a new upload, say, one can do a diff on the old and new
test results and see if any new regressions were introduced. In this
same respect, SageMath test results may be best consulted before new
uploads by hand.

I believe it's in the best interest of Debian users that this bug be
downgraded for Bullseye so Sage can be used in the mostly-wholesome
shape it's in, but since I lack expertise in maintaining it I too will
leave this to someone else.

If you're looking for someone that can be committed to fixing issues in
SageMath for the duration of the stable release cycle, I think I can
step up to the plate, with the caveat that I will have to learn as I go
since I don't yet know Python, but would like to learn and could seize
this opportunity. (C is my forte, so perhaps I can help with some
dependencies.)

I've sent a merge request to add a superficial autopkgtest checking
that 'sage -v' works, as it has broken in the past, and would
appreciate if it could be merged (I believe this is appropriate during
the freeze):
https://salsa.debian.org/science-team/sagemath/-/merge_requests/14

Please let me know if there is any other low-hanging fruit I could work
on as a newcomer to get acquainted with the package.

https://buildd.debian.org/status/package.php?p=sagemath

On armhf, mips64el and ppc64el the build fails reliably with
  Error: critical test failures (e.g. timeout, segfault, etc.)
This seems to be a reasonable build-time smoke test.

The only special case is s390x:
  Checking number of failed tests to determine whether to rerun tests in series...
  No: 504 tests failed, up to 400 failures are tolerated for rerun
It is not a surprise that more tests a failing on big endian,
but there are actually no "critical test failures".

IMHO the best action would be an upload with the following changes:
- your superficial autopkgtest
- raising the limit from 200 to something that makes builds non-flaky
- optionally force build failure on s390x

I could sponsor or NMU such an upload.

cu
Adrian

Hi,

the main problem making the sagemath testsuite flaky is that it randomly aborts due to 'Too many open files'.
Thus only a small part of the test suite gets actually run, when the build is heavily parallelized.
This can be seen by reporting not only the number of failed, but also that of run tests, which shows significant fluctuations.

The problem occurs, because every finished, but not yet logged worker, holds an open fd (a pipe used to read the output of the child actually doing the tests).
Thus when following a long running worker, i.e. logging its messages, while it is still running, so many finished tests can accumulate, that the open files limit (ulimit -n) is reached.

However, there should be no open pipe per finished worker, as the test suite calls 'os.close(self.rmessages)' before waiting for logging the messages.
So this seems to be caused by something that python does behind the scenes.
Removing the single line 'finished.append(w)' in src/sage/doctest/forker.py prevents the open fd increase, though at the cost of hardly logging any test output.

This problem can be avoided by simply logging every finished test, but no running one.

With only the 0001-Report-the-number-of-total-tests-run.patch, the result is something like:
Success: 5 of 71435 tests failed, up to 200 failures are tolerated

Adding the dt-Do-not-follow-a-running-worker.patch, the result becomes:
Success: 194 of 361139 tests failed, up to 200 failures are tolerated

These 194 failures are pretty close to the threshold of 200, so it is not particularly surprising, that this can fail in some environments.
Slightly passing this threshold triggered the build failure in this bug and also the one in bug #983931.

Increasing the threshold to 300 should make that rather unlikely, though.
And considering that there are more than 360 thousand tests, less then 300 failures means more than 99.9 % of the tests succeeded.

The "cython: not found" issue is trivial to fix and important, because otherwise 'sage --cython' does not work and there is no '--cython3' option (unlike e.g. the '--ipython3' option).

After adding the 0002-Tolerate-up-to-300-failing-tests.patch and the u2-Adapt-to-python2-removal.patch the test result is:
Success: 189 of 361139 tests failed, up to 300 failures are tolerated

It would also be a good idea to include a backport of commit 5cf493ca51 ("Avoid libgmp's new lazy allocation") in the next sagemath upload, as that fixes a severe memory leak (see bug #964848).

As to the crashes, I can't reproduce any crash when testing interfaces/singular.py:
sage -t --long --random-seed=0 src/sage/interfaces/singular.py
    [404 tests, 3.87 s]

This crash also does not always happen for the reproducible builds either, e.g. the following log shows it first crashing and then passing this test:
https://tests.reproducible-builds.org/debian/rbuild/bullseye/amd64/sagemath_9.2-2.rbuild.log.gz
[...]
sage -t --long --random-seed=0 src/sage/interfaces/singular.py
    Killed due to segmentation fault
[...]
sage -t --long --random-seed=0 src/sage/interfaces/singular.py
    [404 tests, 21.06 s]
[...]

However, a number of other crashes happen during every test run, but only one of them causes a test failure:
sage -t --long --random-seed=0 src/sage/interfaces/tests.py
**********************************************************************
File "src/sage/interfaces/tests.py", line 34, in sage.interfaces.tests
Failed example:
    subprocess.call("echo syntax error | ecl", **kwds) in (0, 255)
Expected:
    True
Got:
    False
**********************************************************************

Similar crashes sometimes also occur when testing interfaces/lisp.py, but without causing the test to fail.
This is a problem in ecl, which crashes when both stdout and stderr are full, see bug #710953.

Then there is a crash in nauty-gentourng triggered by src/sage/graphs/digraph_generators.py.
For details see bug #991750.

There are also two SIGABRT crashes in mwrank triggered by src/sage/interfaces/mwrank.py.
These seem to be intentional due to invalid input.

Finally, there are some python crashes (5 SIGQUIT, 1 SIGABRT, 1 SIGSEGV) that are all caused intentionally by the test suite.

So none of these crashes are problems in sagemath itself.

Regards,
Ahzo

Hi

Agreed, this really should be fixed, but if anybody is still considering
an upload today, please only make it raise the limit. It's too late for
more invasive changes.

Paul

Thanks a lot for the patches Ahzo. Especially fixing the file handle leak should help a lot.

I guess it's too late for bullseye now, but I can at least upload a fixed package to experimental. I'll also try to fix many of the failing tests by including sage's (large) patch to support pari 2.13 which was finished in June [1]. I have to see if I can backport that to sage 9.2 or if I update to sage 9.4 right away.

Best,
Tobias

[1] https://trac.sagemath.org/ticket/30801

Aug 3, 2021, 11:12 by thansen@debian.org:

Actually that patch was rather a workaround than a fix.

The real cause of the "Too many open files" issue is a behavior change of the multiprocessing.Process class in python3.
It now opens a pipe internally, which it did not do in python2.

The solution is to call the close() method of multiprocessing.Process to close the internal pipe.
Attached u2-close-the-internal-pipe-of-multiprocessing.Process.patch does just that.
However, this method was only added in python 3.7, so attempting to use it fails in earlier versions of python3 (and also in python2).

Regards,
Ahzo

Hi,

Just for the record, I did a rebuild of the package for two transitions
that are ongoing, and it failed on all architectures.

Paul

I started updating sagemath to version 9.4, which is the first version that supports pari 2.13 (which is already in Debian and causes a large part of the failing doctests). I got stuck for now because we need an update of singular (see #992003).

Best,
Tobias

We believe that the bug you reported is fixed in the latest version of
sagemath, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 986527@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Tobias Hansen <thansen@debian.org> (supplier of updated sagemath package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Sat, 18 Dec 2021 00:15:17 +0000
Source: sagemath
Binary: python3-sage python3-sage-dbgsym sagemath sagemath-doc sagemath-jupyter
Architecture: source amd64 all
Version: 9.4-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Math Team <team+math@tracker.debian.org>
Changed-By: Tobias Hansen <thansen@debian.org>
Description:
 python3-sage - Open Source Mathematical Software - Python 3 library
 sagemath   - Open Source Mathematical Software
 sagemath-doc - Open Source Mathematical Software - documentation
 sagemath-jupyter - Open Source Mathematical Software - Jupyter kernel
Closes: 931223 951592 960925 964848 972346 984326 986527 988681 993149 998190
Changes:
 sagemath (9.4-1) unstable; urgency=medium
 .
   * New upstream release. (Closes: #951592, #960925, #964848,
     #984326, #986527, #993149, #998190)
   * Doctests work out of the box now. (Closes: #931223)
   * Build standard python package python3-sage.
   * Install scripts to /usr/bin (Closes: #972346)
   * Activate autopkgtest.
   * Move package to Debian Math Team.
   * New (Build-)Depends:
     - Use dependencies as provided by upstream (drop many).
     - libec-dev (>= 20210503)
     - python3-memory-allocator
     - python3-sphinx (>= 4.3.1-2) (includes fix to dh_sphinxdoc)
     - singular (>= 1:4.2.1-p2)
   * Add dependency on libjs-mathjax to sagemath-doc.
     (Closes: #988681)
   * Run dh_sphinxdoc again.
   * New patches:
     - u0-trac32799-set-MPMATH_SAGE.patch         #32799
     - u0-version-arb-2.21.patch                  #32567
     - u0-version-matplotlib-3.5.patch            #32909
     - u0-version-singular-4.2.1.patch            #32001
     - u0-version-singular-4.2.1p2.patch          #32907
     - u0-version-lcalc-2.0.patch                 #32037
     - u0-version-linbox-1.7.patch                #32959
     - u0-version-gsl-2.7.patch                   #32607
     - u0-version-sphinx-4.3.patch                #32968
     - u1-close-the-internal-pipe-of-multiprocessing.Process.patch
     - dt-version-ipywidgets-6-revert-31517.patch
   * Remove patches (applied upstream):
     - u0-version-pari-2.13-spkg-configure.patch  #30906
     - u0-version-pari-2.13.patch                 #30801
     - u0-version-gap-4.11.patch                  #29314
     - u0-version-flint-2.6.3.patch               #29719
     - d1-no-spkg-builds.patch
     - d1-system-python-packages.patch
     - dt-avoid-pari-timeout.patch
   * Remove patches (no longer required):
     - u1-scripts-dir.patch                       #22731
     - d0-gsl-cblas.patch
     - d0-libgap-path.patch
     - d0-singular.patch
     - d0-docbuild-main.patch
     - d1-sage-env.patch
     - d1-sage.patch
     - dt-avoid-giac-segfault.patch
     - dt-avoid-ecl-timeout.patch
   * Use dephelper-compat level 13.
Checksums-Sha1:
 f091a86dface6bb2d58ca939c70475359d52c486 4792 sagemath_9.4-1.dsc
 471b8f853866658e5ed6ae77d4eb84f879d3a0a8 20664944 sagemath_9.4.orig.tar.xz
 5262e95053841fa07c6b789c511ed3eaa88cbe9a 67188 sagemath_9.4-1.debian.tar.xz
 b098c97fd872d52563fd52bf0702696e48fe2689 152159360 python3-sage-dbgsym_9.4-1_amd64.deb
 4bf26b88f1221c3dc956938c78ef0deb79b806f3 41603232 python3-sage_9.4-1_amd64.deb
 da4d43a087b250a9c7f871c2547900d458d6d207 73434784 sagemath-doc_9.4-1_all.deb
 a18da6e70c7c79c0fe648e01dd529f660bbf6dc3 29344 sagemath-jupyter_9.4-1_all.deb
 694228d138c3443cbedcd5a5b511b03557ed0856 65336 sagemath_9.4-1_all.deb
 7315f2ae24533b121029b4344b3c4528704a85aa 25265 sagemath_9.4-1_amd64.buildinfo
Checksums-Sha256:
 5099cb4d121a6229c634deffb5be163b66cf4e3319fe618d3f06261133dd5bcf 4792 sagemath_9.4-1.dsc
 12d88ecd34957a6ffd8ac1c7bbd28f0612d433b3debf0f249608d2fde1fe730c 20664944 sagemath_9.4.orig.tar.xz
 5a09240310dd61253c9ce9a24b079ba697570e85c978abe03f503b44f9838198 67188 sagemath_9.4-1.debian.tar.xz
 b3f656d57e83336031f08820e0d93c68156f075d3f8579d9424f8515032fcd65 152159360 python3-sage-dbgsym_9.4-1_amd64.deb
 aaca9c324418a7e071a50a552721410756cb8fe8b7ba0eb74dc03b87a5af5cc4 41603232 python3-sage_9.4-1_amd64.deb
 3d6fc35673d547b8e827f6109a3ed58fcebdbc622cd9319f4cefb7065dd2298f 73434784 sagemath-doc_9.4-1_all.deb
 8877bda8ab52936c3d87188349474ec1fdcc8b49acb0d10951ff932b5158350a 29344 sagemath-jupyter_9.4-1_all.deb
 63e01dfa2396510e27b3cb749b379d850adb097bb0dc5d8956dc59d60cef3de1 65336 sagemath_9.4-1_all.deb
 2d128736ca187d55aac47a3c1ac2113c87b1d77c8c72079b4f198d9c11f1147f 25265 sagemath_9.4-1_amd64.buildinfo
Files:
 53d158b827a77019ea01f1afda4455b6 4792 math optional sagemath_9.4-1.dsc
 0c40f1391a7d26ec8703e559e8932fa2 20664944 math optional sagemath_9.4.orig.tar.xz
 343d36d2a5dd9ed44236fb5234e2bfbe 67188 math optional sagemath_9.4-1.debian.tar.xz
 95e7888101fbcd383442de729ea851c7 152159360 debug optional python3-sage-dbgsym_9.4-1_amd64.deb
 197aebcf2ab0267b00da2bb2af5e140d 41603232 python optional python3-sage_9.4-1_amd64.deb
 30dfabe1781449af26ac7f93582da599 73434784 doc optional sagemath-doc_9.4-1_all.deb
 6c8ae73c9a54bb18b489f4f01dd7a320 29344 math optional sagemath-jupyter_9.4-1_all.deb
 ab9603fa1982f1c67321581dcf6c821c 65336 math optional sagemath_9.4-1_all.deb
 71d1ade0238393f9b8e010957eb0861f 25265 math optional sagemath_9.4-1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEoH46ol3M2u2mYo0kjIIWnY7OzSoFAmG9ltEACgkQjIIWnY7O
zSrazA//dZtzBI1YIsXYgMMOQKvEPEDODKskQcidVDd20z4IEkSYhCSel/kC/8QG
u86IXrhvRTg0WphczM9ceohV8K3xEsjRLjwKiOnFh9kPg0ERkugGA5nDTl5SrqHM
gnzK1ftMZumdmv0gfFOcxfo5WwcLpPs53AIHt90Md7Bv/DKijx6BjAIDm9KXO1KX
wV2GnLsI0zThQu69D9IB1KC5UpJTWIU0xPerDlM3SV++X+jPdtqCIznpfvF6yfAz
DxouaFgNaNLwHM/IQBR2F0DRyCeJ76SKW5AuESXRVDxjju/2T3Y8HiWW1NU7Jq88
/Ky5BjDQTBCK59a98VMt+xWxNTdrmA5EBbe6sBc0ZIPenVeSWtxPAJFCbL4UI+i1
zk+1eWOtk5v3qQ97T6BX8PYUve8n4U8mc/MD25HNeYkMfwGGTgzLbONkTTAzDeJ9
0auxTY8xbo55ssU6mTpzDGW7qqiakDGGHS7S+ugOYWJ7hJta2l4wcFSZXbX0jTzj
S8m+EWd2zrfmqUA3/UtCG6qDaV7jyecKO/e3WHyahjz9273l7GVX9Wvs2w63cdAI
VWCX91E2p+Uk4lKfnXTvjAF5cs8NemKI3gkhYbwNojKYAnb7cptYcmfr23G4pqEj
jQhOSJEe6X5fZhcXhcEhsQ5Hp5yh6OZ0scpXdHCzxxIltJBFJP0=
=cpWu
-----END PGP SIGNATURE-----