1

I have BurpSuite configured as a proxy. I have generated and installed a certificate for Burp in my Mac's keychain. On a Kali box I have also configured Iceweasel to proxy through my Burp Proxy running on my Mac. I can can browse from Kali to both http and https sites and burp intercepts the requests. I forward them through and confirm the security exceptions and the browser eventually displays web pages.

From the Kali command line I exported the following two variables 

export http_proxy=http://172.xx.yy.z:8081
export https_proxy=http://172.xx.yy.z:8081

I wrote one python script:

(env) ojblass@kali:~/effective_python$ cat ojblass_urllib_http.py
import urllib
url = urllib.urlopen("http://www.hotmail.com")
data = url.read()
print data

This correctly hits the BurpSuite proxy; however, when I attempt to run https traffic through the request is not intercepted:

(env) ojblass@kali:~/effective_python$ cat ojblass_urllib_https.py
import urllib
url = urllib.urlopen("https://mail.live.com/default.aspx")
data = url.read()
print data

Instead the following error is printed:

<html><head><title>Burp Suite Professional</title>
<style type="text/css">
body { background: #dedede; font-family: Arial, sans-serif; color: #404042; -webkit-font-smoothing: antialiased; }
#container { padding: 0 15px; margin: 10px auto; background-color: #ffffff; }
a { word-wrap: break-word; }
a:link, a:visited { color: #e06228; text-decoration: none; }
a:hover, a:active { color: #404042; text-decoration: underline; }
h1 { font-size: 1.6em; line-height: 1.2em; font-weight: normal; color: #404042; }
h2 { font-size: 1.3em; line-height: 1.2em; padding: 0; margin: 0.8em 0 0.3em 0; font-weight: normal; color: #404042;}
.title, .navbar { color: #ffffff; background: #e06228; padding: 10px 15px; margin: 0 -15px 10px -15px; overflow: hidden; }
.title h1 { color: #ffffff; padding: 0; margin: 0; font-size: 1.8em; }
div.navbar {position: absolute; top: 18px; right: 25px;}div.navbar ul {list-style-type: none; margin: 0; padding: 0;}
div.navbar li {display: inline; margi-left: 20px;}
div.navbar a {color: white; padding: 10px}
div.navbar a:hover, div.navbar a:active {text-decoration: none; background: #404042;}
</style>
</head>
<body>
<div id="container">
<div class="title"><h1>Burp Suite Professional</h1></div>
<h1>Error</h1><p>Invalid&#32;client&#32;request&#32;received&#58;&#32;First&#32;line&#32;of&#32;request&#32;did&#32;not&#32;contain&#32;an&#32;absolute&#32;URL&#32;&#45;&#32;try&#32;enabling&#32;invisible&#32;proxy&#32;support&#46;</p>
<div class="request">GET&nbsp;https://mail.live.com/default.aspx&nbsp;HTTP/1.0<br>
User-Agent:&nbsp;Python-urllib/1.17<br>
Accept:&nbsp;*/*<br>
<br>
</div><p>&nbsp;</p>
</div>
</body>
</html>

The main content of the error states 'Invalid client request received. First line of request did not contain an absolute URL. I have googled this error but everything I seem to come does not appear relevant. I am new to this and would appreciate any insight you might have.

ojblass
  • 216
  • 2
  • 9

1 Answers1

1

I found the answer. Urllib does not support https proxy. I found this site which has a recipe for connecting properly.

Here is the code in case the link goes dead:

# urllib2 opener to connection through a proxy using the CONNECT method, (useful for SSL)
# tested with python 2.4

import urllib2
import urllib
import httplib
import socket


class ProxyHTTPConnection(httplib.HTTPConnection):

    _ports = {'http' : 80, 'https' : 443}


    def request(self, method, url, body=None, headers={}):
        #request is called before connect, so can interpret url and get
        #real host/port to be used to make CONNECT request to proxy
        proto, rest = urllib.splittype(url)
        if proto is None:
            raise ValueError, "unknown URL type: %s" % url
        #get host
        host, rest = urllib.splithost(rest)
        #try to get port
        host, port = urllib.splitport(host)
        #if port is not defined try to get from proto
        if port is None:
            try:
                port = self._ports[proto]
            except KeyError:
                raise ValueError, "unknown protocol for: %s" % url
        self._real_host = host
        self._real_port = port
        httplib.HTTPConnection.request(self, method, url, body, headers)


    def connect(self):
        httplib.HTTPConnection.connect(self)
        #send proxy CONNECT request
        self.send("CONNECT %s:%d HTTP/1.0\r\n\r\n" % (self._real_host, self._real_port))
        #expect a HTTP/1.0 200 Connection established
        response = self.response_class(self.sock, strict=self.strict, method=self._method)
        (version, code, message) = response._read_status()
        #probably here we can handle auth requests...
        if code != 200:
            #proxy returned and error, abort connection, and raise exception
            self.close()
            raise socket.error, "Proxy connection failed: %d %s" % (code, message.strip())
        #eat up header block from proxy....
        while True:
            #should not use directly fp probablu
            line = response.fp.readline()
            if line == '\r\n': break


class ProxyHTTPSConnection(ProxyHTTPConnection):

    default_port = 443

    def __init__(self, host, port = None, key_file = None, cert_file = None, strict = None):
        ProxyHTTPConnection.__init__(self, host, port)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        ProxyHTTPConnection.connect(self)
        #make the sock ssl-aware
        ssl = socket.ssl(self.sock, self.key_file, self.cert_file)
        self.sock = httplib.FakeSocket(self.sock, ssl)


class ConnectHTTPHandler(urllib2.HTTPHandler):

    def do_open(self, http_class, req):
        return urllib2.HTTPHandler.do_open(self, ProxyHTTPConnection, req)


class ConnectHTTPSHandler(urllib2.HTTPSHandler):

    def do_open(self, http_class, req):
        return urllib2.HTTPSHandler.do_open(self, ProxyHTTPSConnection, req)


if __name__ == '__main__':

    import sys

    opener = urllib2.build_opener(ConnectHTTPHandler, ConnectHTTPSHandler)
    urllib2.install_opener(opener)
    req = urllib2.Request(url='https://192.168.1.1')
    req.set_proxy('192.168.1.254:3128', 'https')
    f = urllib2.urlopen(req)
    print f.read()
ojblass
  • 216
  • 2
  • 9