[PLUG] wget and politeness
Brent Rieck
bsr at spek.org
Mon Dec 27 16:08:14 UTC 2004
Keith Lofstrom wrote:
> When I manually web to the site, I get:
>
>>You are black listed at the TWiki web site due to excessive access or
>>suspicious activities. Please contact site administrator
>>peter.thoeny at attglobalSTOPSPAM.net if you got on the list by mistake.
>>Black listed IP addresses will be submitted to major blacklist databases.
It's a not uncommon topic for the newsgroups at my web host, many people
find that their website is getting hammered by people with wget (or a
windows equivalent) causing bandwidth overages. And since most of these
websites are just labors of love they'll block wget and blacklist ip
addresses so their users don't cost them more money, the theory being
that humans don't actually look at most of the website they scrape down.
I'd guess that you won't look at most of what's on twiki.org while
offline but that it's impossible to predict what you will look at, so
you want to download it all. twiki.org could be in this bandwidth
situation.
Try emailing Peter to explain what you were trying to do, he might have
a solution for your offline needs.
--Brent
More information about the PLUG
mailing list