It appears that the deb822.Deb822.iter_paragraph method gets confused
if there are bogus entries (like a single line) in the file. Below is
a test that shows the behavior. Depending on the policy the excpeted
value is either 2 or 3 (2 if we want to discard invalid stanzas).

It appears that the problem is "while len(x) != 0" in deb822.py, that
will make the parser stop on the first bogus line. Attached is a
possible patch for this that makes the EOF handling explicit.

Thanks,
 Michael
--- a/lib/debian/deb822.py	2010-03-16 02:06:28.000000000 +0100
+++ b/lib/debian/deb822.py	2010-06-10 10:51:30.000000000 +0200
@@ -241,7 +241,7 @@
             (All values are given back as unicode objects, so an encoding is
             necessary in order to properly interpret the strings.)
         """
-
+        self.eof = False
         if hasattr(sequence, 'items'):
             _dict = sequence
             sequence = None
@@ -249,12 +249,11 @@
             _dict = None
         Deb822Dict.__init__(self, _dict=_dict, _parsed=_parsed, _fields=fields,
                             encoding=encoding)
-
         if sequence is not None:
             try:
                 self._internal_parser(sequence, fields)
             except EOFError:
-                pass
+                self.eof = True

         self.gpg_info = None

@@ -285,7 +284,7 @@
         else:
             iterable = iter(sequence)
             x = cls(iterable, fields, encoding=encoding)
-            while len(x) != 0:
+            while not x.eof:
                 yield x
                 x = cls(iterable, fields, encoding=encoding)

--- a/tests/test_deb822.py	2010-03-16 02:06:28.000000000 +0100
+++ b/tests/test_deb822.py	2010-06-10 09:59:59.000000000 +0200
@@ -235,6 +235,12 @@
 -----END PGP SIGNATURE-----
 '''

+BOGUS_DATA = '''
+
+xxx-bogus-entry
+
+'''
+
 class TestDeb822Dict(unittest.TestCase):
     def make_dict(self):
         d = deb822.Deb822Dict()
@@ -705,6 +711,15 @@
         self.assertEqual(utf8_contents, latin1_to_utf8.getvalue())
         self.assertEqual(latin1_contents, utf8_to_latin1.getvalue())

+    def test_bogus(self):
+        text = (UNPARSED_PACKAGE + '\n\n\n' +
+                BOGUS_DATA + '\n\n\n' +
+                UNPARSED_PACKAGE).splitlines()
+        l = []
+        for d in deb822.Deb822.iter_paragraphs(text, use_apt_pkg=True):
+            l.append(d)
+        self.assertEqual(len(l), 3)
+
 class TestPkgRelations(unittest.TestCase):

     def test_packages(self):

Hi Michael,

What is your use case for this?  I'm having a hard time seeing a good
way to handle bogus data consistently.  What should the parser yield
when it encounters a bogus stanza?

It should be noted that this behavior is specific to the native parser
(which is used when you specify use_apt_pkg=False or you don't have
python-apt installed).  When iter_paragraphs uses apt_pkg, it returns a
bogus Deb822 object for the bogus line.  Because of apt_pkg's TagParser
implementation, it may appear to have a key corresponding to the bogus
line, but actually trying to get the value for that key will raise
KeyError.  This is not good behavior - it breaks the map interface - but
unless we check for validity of the data (which would defeat the purpose
of using apt_pkg), I don't know how do make it better.

Interestingly, with your patch, the native parser returns an empty
Deb822 object (essentially {}).  This probably is the best behavior we
can ask for - although I think it should at least raise a warning, and
the behavior should be documented.  And...it would be really nice if we
could make the apt_pkg one do the same thing.

Any ideas?

Hi,

the DDTP gatherer for UDD[1] produced an error in a Danish translation
which contains in line 3048:

  Description-da: : Islamic hijri date and prayer time utilities

I fixed this in the DDTP webform and thus the problem might vanish
sooner or later but I wonder if this is another case of not robust
parsing.  I'm not totally sure about RFC 822 and so I don't know what
the correct behaviour in cases like this should be but the double ':'
obviosely confuses the parser and leads to a KeyError "Key
Description-da not found".  I just catched this exception in the
ddtp_gatherer code but I wonder how to solve this cleanly.

The problem is in

http://ddtp.debian.net/Translation_udd/dists/squeeze/main/i18n/Translation-da.gz
  (as well as in
http://ddtp.debian.net/Translation_udd/dists/sid/main/i18n/Translation-da.gz )

and I think it should properly parse the Description-da key or
at least issue a warning about bogus data.

Kind regards

         Andreas.

PS: If you regard this problem as to different from the previousely
    reported one I can open a new bug report.

[1]svn://svn.debian.org/svn/collab-qa/udd/udd/ddtp_gatherer.py

Hi Andreas,

Thanks for the report!

It should definitely have parsed it as

   {'Description-da': ': Islamic hijri date and prayer time utilities'}

instead of

   {'Description-da:': 'Islamic hijri date and prayer time utilities'}

Both the apt_pkg implementation and the standard message library parse
it as the first dict above, so I regard this as a bug in the native
deb822 parser.  Would you mind opening a new bug to track this?  (I
don't mind opening it if you don't get to it, but I probably won't get
to it for a couple of days.)

In the meantime, if you have python-apt installed, and you're using
deb822.Deb822.iter_paragraphs (without a use_apt_pkg=False argument) to
iterate through all of the paragraphs in the Translation file, you
shouldn't hit this.

If a debian/control file containing a comment line followed by a blank line is processed by Sources.iter_paragraphs with use_apt_pkg=False, it silently fails to iterate stanzas. Processing the same file with Packages.iter_paragraphs succeeds. Stripping either the comment or the blank line succeeds.

Failing example:

```
# A comment

Source: abcd
Section: unknown
Priority: extra
Build-Depends: xyzzy
```

Successful example:
```
# A comment
Source: abcd
Section: unknown
Priority: extra
Build-Depends: xyzzy
```

We are using version 1.40-3