[PLUG] wget and politeness

Brent Rieck bsr at spek.org
Mon Dec 27 16:08:14 UTC 2004


Keith Lofstrom wrote:
> When I manually web to the site, I get:
> 
>>You are black listed at the TWiki web site due to excessive access or
>>suspicious activities. Please contact site administrator
>>peter.thoeny at attglobalSTOPSPAM.net if you got on the list by mistake.
>>Black listed IP addresses will be submitted to major blacklist databases.

It's a not uncommon topic for the newsgroups at my web host, many people 
  find that their website is getting hammered by people with wget (or a 
windows equivalent) causing bandwidth overages.  And since most of these 
websites are just labors of love they'll block wget and blacklist ip 
addresses so their users don't cost them more money, the theory being 
that humans don't actually look at most of the website they scrape down. 
  I'd guess that you won't look at most of what's on twiki.org while 
offline but that it's impossible to predict what you will look at, so 
you want to download it all.  twiki.org could be in this bandwidth 
situation.

Try emailing Peter to explain what you were trying to do, he might have 
a solution for your offline needs.

--Brent



More information about the PLUG mailing list