[PLUG] new wget question (robots.txt)

Brent Rieck bsr at spek.org
Wed Dec 29 04:28:56 UTC 2004


john morgali wrote:
> I have looked and dont seem to be seeing if there is a
> way to make wget ignore the robots.txt file and grab a
> file or directory from a site.  I was just thinking of
> this because of the previous thread on wget.  Anyway,
> let me know and thanks in advance. 

I think you can add a "-erobots=off" to your command line to ignore 
robots.txt.  Be nice about it.

--Brent



More information about the PLUG mailing list