* Package name    : python-mapillary-tools
  Version         : 0.4.0
  Upstream Author : support@mapillary.com
* URL             : https://github.com/mapillary/mapillary_tools
* License         : BSD-2
  Programming Lang: Python
  Description     : Useful tools and scripts related to Mapillary

Mapillary Tools is a library for processing and uploading images to Mapillary.

I had a need for this so had a go at packagaing it.

Turns out the the dependency pymp4 is not in debian, so I packaged that too.


However the tests for pymp4 are giving an error "NameError: name
'String' is not defined ", and if you ignore that and build
mapillary-tools anyway you get test errors there, and if you press on
and then try to use it, you hit the same issue:
  File "/usr/lib/python3/dist-packages/pymp4/parser.py", line 86, in <module>
    "major_brand" / String(4),
NameError: name 'String' is not defined

So something is wrong that needs investigation.

Build log:
Traceback (most recent call last):
  File "/home/wookey/packages/mapillary/pymp4/python-pymp4-1.2.0/setup.py", line 29, in <module>
    setup(name="pymp4",
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 223, in run
    self.run_tests()
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 226, in run_tests
    test = unittest.main(
  File "/usr/lib/python3.9/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.9/unittest/main.py", line 147, in parseArgs
    self.createTests()
  File "/usr/lib/python3.9/unittest/main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "/usr/lib/python3.9/unittest/loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.9/unittest/loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.9/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python3/dist-packages/setuptools/command/test.py", line 56, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/wookey/packages/mapillary/pymp4/python-pymp4-1.2.0/tests/test_box.py", line 21, in <module>
    from pymp4.parser import Box
  File "/home/wookey/packages/mapillary/pymp4/python-pymp4-1.2.0/src/pymp4/parser.py", line 86, in <module>
    "major_brand" / String(4),
NameError: name 'String' is not defined
E: pybuild pybuild:355: test: plugin distutils failed with: exit code=1: python3.9 setup.py test
dh_auto_test: error: pybuild --test -i python{version} -p "3.10 3.9" returned exit code 13
make: *** [debian/rules:10: build] Error 25



Running the tool:
$ mapillary_tools
Traceback (most recent call last):
  File "/usr/bin/mapillary_tools", line 33, in <module>
    sys.exit(load_entry_point('mapillary-tools==0.8.0', 'console_scripts', 'mapillary_tools')())
  File "/usr/bin/mapillary_tools", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/__main__.py", line 6, in <module>
    from .commands import authenticate
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/commands/__init__.py", line 2, in <module>
    from . import process
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/commands/process.py", line 4, in <module>
    from ..insert_MAPJson import insert_MAPJson
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/insert_MAPJson.py", line 9, in <module>
    from . import image_log, types, processing, error
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/processing.py", line 16, in <module>
    from .gpx_from_blackvue import gpx_from_blackvue
  File "/usr/lib/python3.9/dist-packages/mapillary_tools/gpx_from_blackvue.py", line 8, in <module>
    from pymp4.parser import Box
  File "/usr/lib/python3/dist-packages/pymp4/parser.py", line 86, in <module>
    "major_brand" / String(4),
NameError: name 'String' is not defined

This is the offending code:
FileTypeBox = Struct(
    "type" / Const(b"ftyp"),
    "major_brand" / String(4),
    "minor_version" / Int32ub,
    "compatible_brands" / GreedyRange(String(4)),
)

My python-foo is very weak so I have no idea what's up here.

I also had some trouble with a .eggs directory being created but not
cleaned up and pybuild seeming to prevent the normal overriding of
dh_foo targets in the rules file.

I also find python packaging very confusing with disttools and
setuptools and pybuild and eggs and setup.py and various other python
build tools/mechanisms that have come and gone and some that get
layered. I'm never sure a) how things are supposed to work and b) what
tools one is currently supposed to be using.

I guess I should mail the python team for advice on these things.

The main point is that a reasonable basic packaging job has been done
so no need to repeat it. With a bit of help it should be good for
uploading.

I decided that mapillary-tools was a better name for the package (than
python-mapillary-tools) as it's a user-facing tool and this matches
the upstream name people would be looking for.

Wookey

[cc:ing debian-python in case anyone happens to know enough about python3-contruct to provide some clues]

OK, so it turns out that there are problems packaging pymp4.

It depends on construct, a (nice) library for parsing binary
formats. However said library seems to have little interest in
maintaining any sort of stable API so there have been significant
changes between 2.8, 2.9 and 2.10 (and in fact pymp4 needs 2.8.8 quite
specifically, and doesn't even work with 2.8.16).

2.8.8 is from October 2016 and Debian now has 2.10.x in stable, testing and unstable.

There is a bug in construct 2.8.8 which means that pymp4 fails 5 out of 30-odd tests even with that.
A python class moved, so that is trivially fixed with:

--- construct-2.8.8.orig/construct/core.py
+++ construct-2.8.8/construct/core.py
@@ -997,7 +997,7 @@ class Range(Subconstruct):
	 max = self.max(context) if callable(self.max) else self.max
	 if not 0 <= min <= max <= sys.maxsize:
	     raise RangeError("unsane min %s and max %s" % (min, max))
-        if not isinstance(obj, collections.Sequence):
+        if not isinstance(obj, collections.abc.Sequence):
	     raise RangeError("expected sequence type, found %s" % type(obj))
	 if not min <= len(obj) <= max:
	     raise RangeError("expected from %d to %d elements, found %d" % (min, max, len(obj)))


But as no-one cares about 2.8.x anyway this fix doesn't help much.

What's really needed is updating pymp4 to use construct 2.10 (or
switch to a more stable library if such a thing exists ('Kaitai
Struct' was mentioned)).

There is an upstream issue for this:
https://github.com/beardypig/pymp4/issues/3

Which I've just added some info to.

2.9 changes the way Strings work: an encoding is always required, and
explicit flavours of padding (left/right, specifiable padding char)
have been removed. pymp4 uses these padding options (specifying spaces
and right-padding), but may still work fine with the remaining default
behaviour of 'PaddedString' (nulls and rightpading). I don't know
enough about either the MP4 format or the codebase to be sure. I did
know up a patch to update to the new string API.

2.9 also loses Embedded bitwise structs. And 2.10 loses 'Embedded' completely.

I have not really managed to work out exactly what 'Embedded' does. I
can't really work out what the difference between putting a bitwise
struct in the middle of a struct and putting an Embedded bitwise
struct in the middle of a struct is. Mostly this is because the online
docs only cover 2.10, not 2.8 so I don't know what the old definition
was. I spent a couple of hours trying to work it out. It's made more
complicated by the fact that construct also switched from 'bits by
default' to 'bytes by default' for efficiency reasons.

I've put a half-arsed patch in the github issue which is probably OK
for the strings part and almost certainly wrong for the Embedded
part. So example I have no idea how to deal with this which selects
one struct depending on a string:
Box = PrefixedIncludingSize(Int32ub, Struct(
    "offset" / TellMinusSizeOf(Int32ub),
    "type" / Peek(String(4, padchar=b" ", paddir="right")),
    Embedded(Switch(this.type, {
        b"ftyp": FileTypeBox,
        b"styp": SegmentTypeBox,
        b"mvhd": MovieHeaderBox,
        b"moov": ContainerBoxLazy,
	...
	b'afrt': HDSFragmentRunBox
    }, default=RawBox)),
    "end" / Tell
))

changing
-    "type" / Peek(String(4, padchar=b" ", paddir="right")),
to
+    "type" / Peek(PaddedString(4,"ascii")),
is probably equivalent, but what is the equivalent syntax for choosing
the right struct for the 'type' field according to the 1st 4 bytes of
it, without using 'Embedded'?  This was where I decided it was bedtime
and admitted defeat for the time being.

If someone familiar with construct 2.8 to 2.10 upgrades wanted to take
a look at this that would be very helpful. For some things we might
need to know something about the mp4 format too. I'm not sure to what
degree we need to understand the format, or if we can more or less
mechanically update the syntax.

So, for now there is no mapillary-tools in Debian, and without a
response from upstream or some help I'm stuck.

Wookey
--
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/