[PLUG] new wget question (robots.txt)
Brent Rieck
bsr at spek.org
Wed Dec 29 04:28:56 UTC 2004
john morgali wrote:
> I have looked and dont seem to be seeing if there is a
> way to make wget ignore the robots.txt file and grab a
> file or directory from a site. I was just thinking of
> this because of the previous thread on wget. Anyway,
> let me know and thanks in advance.
I think you can add a "-erobots=off" to your command line to ignore
robots.txt. Be nice about it.
--Brent
More information about the PLUG
mailing list