Skip to main content.
February 11th, 2007

Mechanize - Useful Tips

Mechanize is a handy, high level, programmatic web browser type deal for Python. Use it for all of your screen scraping needs. Here are a few tips I had to figure out on my own:

How to set the user agent:

from mechanize import Browser
browser = Browser()
browser.addheaders = [("User-agent", "Mozilla/5.0 (compatible;
MyProgram/0.1)")]
browser.open('http://www.python.org')

How to see the data you’ve fetched:

from mechanize import Browser
browser = Browser()
browser.open('http://www.python.org')
print browser.response().read()

Posted by Greg Pinero (Primary Searcher) in Python

This entry was posted on Sunday, February 11th, 2007 at 1:56 am and is filed under Python. You can follow any responses to this entry through the comments RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Mechanize - Useful Tips”

  1. Titus Brown says:

    Hi, Greg,

    you can do this with twill quite easily; it’s a wrapper around mechanize.

    http://ivory.idyll.org/blog/feb-07/simple-with-twill.html

    cheers,
    –titus

    February 12th, 2007 at 4:00 am
  2. Greg Pinero (Primary Searcher) says:

    Hi Titus. That’s a nice article. I actually considered using Twill but I just took a glance at it, and it looked like I’d have to learn a new language. But from your article I must have been mistaken, I could have done it in Python too.

    I’ll try out Twill for my next screen scraping project.

    February 12th, 2007 at 11:40 am