Owner: Jan Gru <j4n6ru@gmail.com> Severity: wishlist * Package name : bulk-extractor Version : 1.6.0 Upstream Author : Simson L. Garfinkel <slgarfin@nps.edu> * URL : https://github.com/simsong/bulk_extractor * License : MIT and CC0 Programming Lang: C++, Python (and Java for the BEViewier, probably not packaged) Description : A stream-based forensics tool for triage and cross-evidence analysis, which scans the media and extracts recognizable content bulk_extractor is a program for bulk data extraction and analysis, it carves for relevant features such as email addresses, credit card numbers, URLs, and other types of information from digital evidence files in a stream-based manner by parallelized processing blocks to omit disk seeking. ** Why is this package relevant? It is a useful tool for forensic investigations, because it is way more than just another file carver. The program provides several unusual capabilities including: - It finds email addresses, URLs and credit card numbers that other tools miss because it can process compressed data (like ZIP, PDF and GZIP files) and incomplete or partially corrupted data. - It can carve JPEGs, office documents and other kinds of files out of fragments of compressed data. It will detect and carve encrypted RAR files. - It builds word lists based on all of the words found within the data, even those in compressed files that are in unallocated space. Those word lists can be useful for password cracking. - It is multi-threaded; running bulk_extractor on a computer with twice the number of cores typically makes it complete a run in half the time. - It creates histograms showing the most common email addresses, URLs, domains, search terms and other kinds of information on the drive. The program is authored by the renowned forensics researcher Simson L. Garfinkel, who is probably most recognized for his work on DFXML at the Naval Postgraduate School (NPS) and the National Institute of Standards and Technology (NIST). It provides rich documentation -- for the end-users as well as for potential contributors [0]. To sum it up, bulk_extractor has great potential for improving triage and automatation workflows within digital forensics and should be therefore included in Debian's package sources. ** Resolved issues bulk_extractor is already packaged in Kali [1], but had licensing issues until recently. To be more precise, it linked code with OpenSSL while not explicitly permitting it and used a the modified MIT-license from the JSON-project, which is considered non-free and not DFSG-compliant. To overcome this issues I resolved this issues in cooperation with upstream by sending two recent patches [2], which were already accepted. ** Maintanance plan I plan to maintain it within the pkg-security-team's repository on salsa, where a lot of forensics packages live [3]. I am looking for a sponsor of this package, who would be ideally a member of the a/m team. Best regards Jan [0] See http://digitalcorpora.org/downloads/bulk_extractor/BEUsersManual.pdf, https://digitalcorpora.s3.amazonaws.com/downloads/bulk_extractor/BEProgrammersManual.pdf and https://digitalcorpora.s3.amazonaws.com/downloads/bulk_extractor/BEWorkedExamplesStandalone.pdf [1] See https://tools.kali.org/forensics/bulk-extractor [2] See https://github.com/simsong/bulk_extractor/issues/168, https://github.com/simsong/bulk_extractor/pull/169 and https://github.com/simsong/bulk_extractor/pull/170 [3] See https://salsa.debian.org/pkg-security-team/
Hello Jan, This would be a great package to have it on Debian, I usually do a quick review to see if I spot any noticeable issues before I do a deep dive on it (which I would to during this weekend), and I notice an issue on d/rules, there are some commands doing: "test -d foo || git clone bar" This is an issue because it goes against our policy of not using network during the build process[0], you can read a recent discussion about it on LWN as well[1]: "For packages in the main archive, no required targets may attempt network access, except, via the loopback interface, to services on the build host that have been started by the build." In order to fix this issue you have two options: 1) Package those projects separately and add them to B-D. 2) Repack the upstream tarball and vendor/bundle them in. You would usually prefer option 1 when the libraries could be reused by other packages and option 2 when they are likely to only be used by your package (usually means the same upstream). But sometimes, even if the library could be used by another package in the future (but it's not currently), you can go with option 1 if it makes more sense. Beware that there is not a clear consensus on this matter so some arguing might be needed (even though we have examples of packages vendoring libraries which are already available in a standalone manner on main). Looking at the three libraries we are talking about: simsong/be13_api simsong/dfxml (watchout cause it looks like this one has just been moved to a different repo) nbeebe/sceadan It looks like it's totally fine to vendor be13_api and dfxml, it seems like sceadan is generic enough to be used by other projects but I didn't do a proper check. I suggest you consider the options here and let us know what you think it's best. Oh, and since you are in contact with upstream, this sort of issue is sometimes solved by upstream providing a release tarball that includes the submodules. The issue is that as far as I know Github does not provide this feature, so they have to use a script to generate the tarball and attach it to the release. This makes the tarball easier to be worked on/packaged by other distros as well[2], but it's also easy for us to workaround so this is a tradeoff between bothering upstream vs repacking on our side. Considering upstream is focused on a rewrite of bulk_extractor, it might be a good idea to repack it ourselves, I just wanted to let you know so you're aware of the ideal fix for this if it happens again in the future. Thanks for your work! [0] https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules [1] https://lwn.net/Articles/700465/ [2] And I guess it's also easier for users who wants to build it themselves, as plain git clone will not checkout the submodules.
Hello Jan, This would be a great package to have it on Debian, I usually do a quick review to see if I spot any noticeable issues before I do a deep dive on it (which I would to during this weekend), and I notice an issue on d/rules, there are some commands doing: "test -d foo || git clone bar" This is an issue because it goes against our policy of not using network during the build process[0], you can read a recent discussion about it on LWN as well[1]: "For packages in the main archive, no required targets may attempt network access, except, via the loopback interface, to services on the build host that have been started by the build." In order to fix this issue you have two options: 1) Package those projects separately and add them to B-D. 2) Repack the upstream tarball and vendor/bundle them in. You would usually prefer option 1 when the libraries could be reused by other packages and option 2 when they are likely to only be used by your package (usually means the same upstream). But sometimes, even if the library could be used by another package in the future (but it's not currently), you can go with option 1 if it makes more sense. Beware that there is not a clear consensus on this matter so some arguing might be needed (even though we have examples of packages vendoring libraries which are already available in a standalone manner on main). Looking at the three libraries we are talking about: simsong/be13_api simsong/dfxml (watchout cause it looks like this one has just been moved to a different repo) nbeebe/sceadan It looks like it's totally fine to vendor be13_api and dfxml, it seems like sceadan is generic enough to be used by other projects but I didn't do a proper check. I suggest you consider the options here and let us know what you think it's best. Oh, and since you are in contact with upstream, this sort of issue is sometimes solved by upstream providing a release tarball that includes the submodules. The issue is that as far as I know Github does not provide this feature, so they have to use a script to generate the tarball and attach it to the release. This makes the tarball easier to be worked on/packaged by other distros as well[2], but it's also easy for us to workaround so this is a tradeoff between bothering upstream vs repacking on our side. Considering upstream is focused on a rewrite of bulk_extractor, it might be a good idea to repack it ourselves, I just wanted to let you know so you're aware of the ideal fix for this if it happens again in the future. Thanks for your work! [0] https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules [1] https://lwn.net/Articles/700465/ [2] And I guess it's also easier for users who wants to build it themselves, as plain git clone will not checkout the submodules.