#787990 game-data-packager: implement mirror checking scheme

#787990#5
Date:
2015-06-07 14:18:55 UTC
From:
To:
Nothing last forever on the internet. (GeoCities, GoogleCode...)

The URL in .yaml files needs to be actively/passively monitored.

*) active monitoring: this would need a custom spider

*) passive monitoring: set-up a custom web server as a last-resort
   that is appended to the list of servers found in the .yaml file

   This server doesn't even need to store all the files;
   the missing one would be listed as 404 in access.log.

   At this point, the user would have already faced on or
   several other broken links, so an extra one isn't that ugly.

   ~~~

   Then the access.log on the server can be monitored with a
   cron job and a list of broken link regularly computed
   and emailed to gdp@packages.d.o
   (todo: check threshold, stuff it in a DB...).

   I really don't care about gathering IP addresses of random
   g-d-p users; this can be hosted anywhere.

(see attached flowchart)

Alexandre