Mechanize – Useful Tips

Mechanize is a handy, high level, programmatic web browser type deal for Python. Use it for all of your screen scraping needs. Here are a few tips I had to figure out on my own:

How to set the user agent:

from mechanize import Browser
browser = Browser()
browser.addheaders = [("User-agent", "Mozilla/5.0 (compatible;
MyProgram/0.1)")]
browser.open('http://www.python.org')

How to see the data you’ve fetched:

from mechanize import Browser
browser = Browser()
browser.open('http://www.python.org')
print browser.response().read()

2 Responses to “Mechanize – Useful Tips”

  1. Titus Brown says:

    Hi, Greg,

    you can do this with twill quite easily; it’s a wrapper around mechanize.

    http://ivory.idyll.org/blog/feb-07/simple-with-twill.html

    cheers,
    –titus

  2. Hi Titus. That’s a nice article. I actually considered using Twill but I just took a glance at it, and it looked like I’d have to learn a new language. But from your article I must have been mistaken, I could have done it in Python too.

    I’ll try out Twill for my next screen scraping project.