It would be better to use exclude_urls: /cgi-bin/ .cgi ? .pl as default. because i had a php3 script (memory game) which was looped forever by htdig filling up my discspace... :-( Adding a single question mark to the exclude_urls parameter resolved this.