gitlab 15.10.x needs ruby-google-protobuf 3.22.2 and it'd be great if we
can make a new upload to experimental. If you need help with the update,
I can help.

thanks
Praveen

On Fri, 21 Apr 2023 14:16:18 +0530 Praveen Arimbrathodiyil  <praveen@onenetbeyond.org> wrote:
 > gitlab 15.10.x needs ruby-google-protobuf 3.22.2 and it'd be great
if we
 > can make a new upload to experimental. If you need help with the
update,
 > I can help.

I tried building the updated version, dsc here
https://people.debian.org/~praveen/new/pool/main/p/protobuf/

but seems there is some changes in build system (they are using cmake
now)


CMake Warning at cmake/abseil-cpp.cmake:28 (message):
  protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong
Call Stack (most recent call first):
  CMakeLists.txt:350 (include)

I could skip building googletest easily, but this is not very straight
forward. Any ideas?

On Mon, 24 Apr 2023 12:03:35 +0530 Pirate Praveen  <praveen@onenetbeyond.org> wrote:
 > CMake Warning at cmake/abseil-cpp.cmake:28 (message):
 >   protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong
 > Call Stack (most recent call first):
 >   CMakeLists.txt:350 (include)
 >
 > I could skip building googletest easily, but this is not very
straight
 > forward. Any ideas?

I tried setting -Dprotobuf_ABSL_PROVIDER=package similar to grpc, but
the rules file seems too complex, so I will leave it to the maintainer
for a proper update. In the meanwhile, gitlab can use the version from
rubygems.org (being in contrib).

On Wed, 26 Apr 2023 22:51:08 +0530 Pirate Praveen
<praveen@onenetbeyond.org> wrote:
 > On Mon, 24 Apr 2023 12:03:35 +0530 Pirate Praveen
 > <praveen@onenetbeyond.org> wrote:
 >  > CMake Warning at cmake/abseil-cpp.cmake:28 (message):
 >  >   protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong
 >  > Call Stack (most recent call first):
 >  >   CMakeLists.txt:350 (include)
 >  >
 >  > I could skip building googletest easily, but this is not very
 > straight
 >  > forward. Any ideas?
 >
 > I tried setting -Dprotobuf_ABSL_PROVIDER=package similar to grpc, but
 > the rules file seems too complex, so I will leave it to the
maintainer
 > for a proper update. In the meanwhile, gitlab can use the version
from
 > rubygems.org (being in contrib).

With help from sur5r on irc,

diff -Nur protobuf-3.22.3/debian/control
protobuf-3.22.3-sur5r/debian/control
--- protobuf-3.22.3/debian/control 2023-04-21 12:00:57.000000000 +0200
+++ protobuf-3.22.3-sur5r/debian/control 2023-04-26 19:23:36.007605725
+0200
@@ -10,6 +10,7 @@
  , cmake
 # C/C++
  , zlib1g-dev
+ , libabsl-dev
  , libgmock-dev <!nocheck>
  , libgtest-dev <!nocheck>
 # Python
diff -Nur protobuf-3.22.3/debian/files
protobuf-3.22.3-sur5r/debian/files
--- protobuf-3.22.3/debian/files 1970-01-01 01:00:00.000000000 +0100
+++ protobuf-3.22.3-sur5r/debian/files 2023-04-26 19:23:41.469612410
+0200
@@ -0,0 +1 @@
+protobuf_3.22.3-1_source.buildinfo devel optional
diff -Nur protobuf-3.22.3/debian/rules
protobuf-3.22.3-sur5r/debian/rules
--- protobuf-3.22.3/debian/rules 2023-04-09 07:50:55.000000000 +0200
+++ protobuf-3.22.3-sur5r/debian/rules 2023-04-26 19:20:30.327913541
+0200
@@ -38,12 +38,15 @@
 PYTHON_CROSS_VARS += CXX=$(CXX)
 endif

+override_dh_auto_configure:
+ dh_auto_configure -- -Dprotobuf_ABSL_PROVIDER=package
+
 override_dh_auto_build-arch:
 ## Chicken<->Egg problem: protobuf requires self-generated .pb.go
files to
 ## be built. First we build it normally; then
"generate_descriptor_proto.sh"
 ## generates .pb.go files and replaces 'em if they are different as
well as
 ## invokes "make protoc" which re-builds "src/protoc" if required.
- dh_auto_build --arch
+ dh_auto_build --arch -- -Dprotobuf_ABSL_PROVIDER=package
  bash -x ./generate_descriptor_proto.sh

 ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))

The error changed to, which he suggested needs a newer absl

With a newer libabsl-dev from
https://people.debian.org/~praveen/new/pool/main/a/abseil/, the error
changed to

CMake Error at cmake/tests.cmake:97 (target_link_libraries):
 Target "tests" links to:

   absl::scoped_mock_log

 but the target was not found. Possible reasons include:

   * There is a typo in the target name.
   * A find_package call is missing for an IMPORTED target.
   * An ALIAS target is missing.

Call Stack (most recent call first):
 CMakeLists.txt:401 (include)

Hi,
building taking[0] changes as reference. Patch attached.

This seems to be an issue with abseil itself which is currently
unfixed[1]. The workaround suggested is to add in
"ABSL_BUILD_TESTING=ON" and "BUILD_TESTING=ON" options to cmake flags.

Now, the release of abseil in experimental seems to be doing that, but
only for the static library and not the shared library, because it needs
googltest for the same, and googletest does not vendor any shared
library because the SO lib is not maintained properly upstream[2] and
hence trying to enable the above options for abseil for shared object
compilation chokes with a dh_shlibdeps error shouting that it also
needs a so lib for gtest. The maintainer of abseil has implemented the
same for shared lib in this commit[3] and has explained with a solid reasoning, and you
were involved (as a reporter) in a bug too, regarding the same.

The reason why you get the above issue (as far as i understand) is
because protobuf tries to link with abseil in a shared lib form, and
I found it non-trivial to change that to fit all usecases.
Since this was a test-only thing, the best way for now is to disable
the tests for now until this is fixed properly in abseil.

This was straightforward was to add "-Dprotobuf_BUILD_TESTS=OFF" to
configure options.

Now, I had to do multiple changes to get it working. I'm mentioning them
incase that helps you/someone else who looks at it:

* Refreshed the existing patches
* Added in -Dprotobuf_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=ON to get
  shared lib building as well.
* There's no need of any `-D` in front of dh_auto_build since this
  translates into a `make` command (which fails with -D), removed that.
* Saw some bazel stuff failing, had to add a B-D on bazel-bootstrap and
  also set HOME to /tmp for the descriptior_proto bash script so it
  builds fine in sbuild.
* The build proceeded forward but choked at python build. Turns out due
  to buildsystem change, binaries are no longer available at src/ but in
  obj-* dir. Added a patch to setup.py to link against that dir.
* Now the build moved but choked at ruby's build. Turns out it isn't
  sensible to add entire ruby/ext/google/protobuf_c to d/clean. Adding 4 files
  is enough.
* Now the java part of the build failed. Here, d/p/no_javax.patch has to
  be updated to remove more instances of javax and @Nullable annotation.
* In addition to this change, the java build xml expected protoc binary
  to be in src/ but it changed to obj-$(DEB_HOST_MULTIARCH) so had to
  copy it to src/ before build and removed it after build was done (sort
  of a hack).
* Now the python testsuite choked, has to change LD_LIBRARY_PATH to new
  location. Also needed a B-D on numpy. Had to exclude a generator_test
  here which looked syntactically incorrect to me.
* After this, the install step failed, because no static libs are being
  built. I've removed them from install files for now, because fixing
  them would mean a lot of work in either d/rules or patching cmake.
* Installed some .cmake files for libprotobuf. There is also a compiled
  lib for third_party/utf8_range but I suppose we don't need it. So
  overrided dh_missing to ignore that.

After doing all of this, protobuf finally built. All in all, I had to
look at multiple languages and buildsystems to get this working.

**NB1:** The static libs are not present in the protobuf lib at the moment
because this changed after the build system changed, and needs some work to get both shared
and static libs enabled and it's 5 in the morning and
I'm too tired and sleepy to work on it. For gitlab's usecase, static libs are *probably*
not needed.

**NB2:** bazel buildsystem probably downloads some stuff off the
internet for this package (I did not check in depth) so the debian way
of doing it would be to either package the stuff it needs or download
copies beforehand and use that in build.

PS: Sponsor me a party/dinner when?

[0]: https://people.debian.org/~praveen/new/pool/main/p/protobuf/
[1]: https://github.com/abseil/abseil-cpp/issues/1407
[2]: https://github.com/google/googletest/issues/2184
[3]: https://salsa.debian.org/debian/abseil/-/commit/136c2572f4d6e6ab8ae02f74d691634874458184
[4]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034908#48

Best,
Nilesh

This was actually easy to implement, attached modified patch to get the
static libraries building as well.

The third party utf8's static libs are not installed still
because I don't really know if they are needed.
But if they are, it should be as simple as adding it to an install
file with relevant path, much like how protobuf's static libs are
installed.

That said, my work for the gitlab issue ends here 😃

This still seems to be an issue, though. During build I could see it
fetch some tarballs from github.


Best,
Nilesh

The last comment that I probbaly should add is that you also need to
check if there's api changes between 3.21 and 3.22/3.23 (there probably
are). In such a case you need to bump SONAME in d/rules and also change
soname in binary packages from 32 to 33/34. This should not be
complicated, just need to `mv` some files, that's all (and change soname
in rules/cmakelists).

Best,
Nilesh

Hi Nilesh, Benjamin,
 The reason is that it and protobuf 3.24.2 need abseil 20230125.0 or
newer. While it is available in experimental, it doesn't export
scoped_mock_log targets for cmake. Hence it is not found, I don't know
if newer versions like 20230802.0 fix this issue or not. Can you
comment on this Benjamin and if you are going to package it soon
and/or accept someone as a co-maintainer for abseil? It seems abseil
20230802.0 became available and should be packaged.
[...]
 Building without tests are always a bad idea.
 If you are at DebConf, we may talk and eat together.

Regards,
Laszlo/GCS

I will do an upload of 20230802.0 to experimental today, but I don’t
think it fixes this issue. scoped_mock_log depends on symbols in
GoogleMock, but there’s no good way to express those dependencies in a
hypothetical libabsl_scoped_mock_log.so since libgmock.so doesn’t exist.

The way upstream (Google) intends for all this to work is for protobuf
to link its Abseil and GoogleMock dependencies statically. Is that
possible? If not, I have some alternative ideas (like building a
libabsl_scoped_mock_log.so without a DT_NEEDED entry for GoogleMock),
but they all seem like hacks. I’m also open to other ideas if anybody
has them.

 Thanks for the upload and the information. I'm going to check it soon.
 It is possible, but not recommended. If abseil gets (security) fixes,
protobuf will not get it like in the case of loading fresh shared
libraries.
Also it seems protobuf configure process looks for shared version of
abseil. Need to double check.
 I'm not sure how to handle this properly for everyone. I think I will
solve this on the protobuf side. Use static linking with abseil and
record with the build process which version was used. The hard way
would be to bundle an abseil source tarball with protobuf and use its
shared version of libraries from a private location only for protobuf.
The first should get precedence for which abseil should export static
linking possibilities of scoped_mock_log in its cmake files.

Regards,
Laszlo/GCS