[PLUG] Wget and Cookies
Jeff Schwaber
freyley at gmx.net
Sun Aug 10 23:02:01 UTC 2003
Hi,
As I am currently poor, and lazy, I wish to purchase things which are on
sale, but do not particularly want to go through each grocery store's
circular myself. I'd much rather get an itemized list in my email and be
able to run filters on it and stuff. Of course, if they offered such a
service, it'd be html emails with huge images of what they're selling,
but even so, I'd be willing to parse that. I haven't been able to find
that. So I figured I'd just write a combo bash and python script to wget
their website, and then parse it for their offers.
This should be easily possible. Looking in galeon at their website, all
the information I want is available in some scripts they write to create
mouseovers.
But when I use wget to get the page, I get, instead of the page that I
want, a page called needsCookies.jsp, which tells me that their page
requires cookies.
I checked wget's man page, and it confirmed what I believed (wget
defaults to accepting cookies), but I manually turned them on anyway,
and eventually added my galeon cookies file as wget's loaded cookies.
Still no page.
I'm baffled. Wget should be sending the cookies galeon has collected
which are perfectly valid, and the jsp should be seeing that wget is
accepting cookies, but it's not.
And the bizarre thing is, when I set galeon to warn about cookies, and
then reject them all, I can get through to the page.
The supermarkets I'm looking at are Safeway and Fred Meyer, which put
their circulars on adinsertsonline.com and inserts2online.com
respectively.
Any thoughts, please. =)
Jeff
More information about the PLUG
mailing list