[PLUG-TALK] "Driving" a javascript page

Brent Rieck bsr at spek.org
Sat Mar 25 09:39:59 PST 2006

Mike Witt wrote:
> Is there any way to programatically query a javascript page?
> In other words, if I'm presented with a form created in javascript,
> is there a way to write a program that can fill out that form
> and submit it. If so, I'd be grateful for any pointers to how
> this might be done.

I've had to do this on many occasions, usually for vendors that say they
have an "API", where by "API" they actually mean a web interface that
must be driven by a human using IE6.0 on Windows.

Here's basically what I've done; there may be a better way, probably
some 0.4GB eclipse plugin to automate this:

1) Manually figure out what form fields need to be submitted, this may
involve using the "view rendered source" extension on firefox, usually
the "get form info" feature of the web developer extension is sufficient.
2) Figure out if they need a cookie or not, save it if they are.
3) Write a little glue to generate the name=value pairs for the form
values from my application.
4) Submit data and cookie (if needed) with Curl, save results.

If I'm lucky I'm done, but sometimes these "API"s require you to bounce
through 1, 2, or 3 more URLs to actually complete the transaction, each
bounce is accomplished with some horrific javascript, sometimes setting
cookies along the way.  If that's the case, use the saved result of step
4 to write a regular expression to grab the redirection url, look to see
if they're setting a cookie; use curl to go to next url, sending cookie
along if needed, repeat until you reach the final destination.  Wrap
that all up in a script and you're done!

It's kind of a huge pain in the ass and the solution is usually somewhat
fragile.. ymmv.


More information about the PLUG-talk mailing list