HTTP error 403 in Python 3 Web Scraping

I was trying to scrap a website for practice, but I kept on getting the HTTP Error 403 (does it think I'm a bot)?

Here is my code:

    #import requests
    import urllib.request
    from bs4 import BeautifulSoup
    #from urllib import urlopen
    import re

    webpage = urllib.request.urlopen('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1').read
    findrows = re.compile('<tr class="- banding(?:On|Off)>(.*?)</tr>')
    findlink = re.compile('<a href =">(.*)</a>')

    row_array = re.findall(findrows, webpage)
    links = re.finall(findlink, webpate)

    print(len(row_array))

    iterator = []

The error I get is:

     File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
        return opener.open(url, data, timeout)
      File "C:\Python33\lib\urllib\request.py", line 479, in open
        response = meth(req, response)
      File "C:\Python33\lib\urllib\request.py", line 591, in http_response
        'http', request, response, code, msg, hdrs)
      File "C:\Python33\lib\urllib\request.py", line 517, in error
        return self._call_chain(*args)
      File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
        result = func(*args)
      File "C:\Python33\lib\urllib\request.py", line 599, in http_error_default
        raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 403: Forbidden

This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it's easily detected). Try setting a known browser user agent with:

    from urllib.request import Request, urlopen

    req = Request('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1', headers={'User-Agent': 'Mozilla/5.0'})
    webpage = urlopen(req).read()

This works for me.

By the way, in your code you are missing the () after .read in the urlopen line, but I think that it's a typo.

TIP: since this is exercise, choose a different, non restrictive site. Maybe they are blocking urllib for some reason...

From: stackoverflow.com/q/16627227

Back to homepage or read more recommendations: