#915372 RFP: grab-site -- The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Package:
wnpp
Source:
wnpp
Submitter:
"emma.wts"
Date:
2025-11-29 16:42:18 UTC
Severity:
wishlist
#915372#5
Date:
2018-12-03 09:43:10 UTC
From:
To:
Upstream-Dev: https://github.com/ludios/grab-site/

Programming Language: Python

License: MIT license

Description: grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses a fork of wpull for crawling.