Troubleshooting HTTPS connections with mitmproxy
Programming Snapshot – mitmproxy
Finding the data zipping back and forth between the browser and server is not only interesting for snooping spies, but also for debugging developers. Mike Schilli gets you started with mitmproxy and shows how to customize it using Python scripts.
If things just don't work when you are developing a web application, the question immediately arises as to what data the browser and web server are actually exchanging. Tools for snooping on the network such as Wireshark (as well as proxies that sit between the client and the server) leave both requests and responses untouched, while logging them for inspection.
mitmproxy
, which stands for man in the middle (MITM) proxy, is the king of the hill in this category; it makes the impossible possible by logging encrypted HTTPS requests. But first, let's look at the simplest, unencrypted case, for which my aging website perlmeister.com that still uses plain old unencrypted HTTP is a great choice.
Figure 1 shows how the browser retrieves the requested page's HTML text, along with some images and JavaScript snippets from the server. The mitmproxy
tool, which is available for download as binary from [1], sports a terminal user interface (UI), which displays a double arrow to the left of the current request, called Flow
in mitmproxy
parlance. When you press the Enter key, the detailed request data come up, as shown in Figure 2.
Using the cursor keys (or, in typical Vi-style: h, j, k, l), you can move to the left/down/up/right and jump between Request
, Response
, and Details
on the request detail page. You can always go back to the next level up with Q. On the main screen listing the flows, use the cursor keys (or J for down and K for up) to select a specific one. Press D to clear the current flow (marked >>
), and hammer D repeatedly to clear the whole screen.
Command-Line Option
There are several options to place mitmproxy
between the client and the server in order to start snooping on the data exchanged. The simplest approach is configuring it on the client side, where applications typically provide options to configure HTTP or SOCKS proxies. To start the proxy, call mitmproxy
from the command line, which will render the currently running terminal black, and log intercepted data, while waiting for user commands from the keyboard.
If you try to persuade Chromium on Ubuntu to use a proxy, officially located under Settings | Advanced/Security | Open Proxy Configuration, you are in for a surprise, because the desktop application won't allow it and outputs an error message (Figure 3). Fortunately, the program accepts the command-line option
$ chromium-browser --proxy-server="localhost:8080"
telling it to use a proxy server on the host (localhost
) and port (8080), which is where mitmproxy
listens by default. The important thing is to run Chromium as a single instance on the Ubuntu desktop. If you already have another instance running, you might find yourself tearing your hair out in frustration because no requests show up in the logs. If it's configured correctly, however, when you access URLs in the browser, the data will appear in the proxy UI for further analysis.
Breaking Encryption
So far, so good. But what happens if the browser sends an HTTPS request? By design, the protocol will prevent sniffing attacks from men in the middle. And, lo and behold, a Chromium browser configured to use mitmproxy
reports that an attack is in progress (Figure 4).
But if you then go to Advanced and click on the Proceed (unsafe) link, Chromium lifts the barrier and lets you continue at your own risk, while mitmproxy
in turn will eagerly log the connection data. This may seem like a silly question, but how does it do that if the client and server are using an encrypted connection?
By using a revolutionary trick, mitmproxy
identifies itself to the client as the server, and to the server as the client (Figure 5). Instead of sending back the actual server response, it sends the client a fake certificate. This leads to the client – assuming it doesn't check or validate the certificate – opening an encrypted connection to the proxy instead of the server, if the user continues despite the warning.
By being in possession of the encryption keys, the proxy can now easily decrypt and log the data going over the wire (Figure 6) that it then encrypts again for forwarding to the server, which thinks it is talking to the client. In other words, a classic MITM attack.
Hard Cases
If the client does not allow communication over obviously hacked channels even if the user tries to force it, the sys admin can get in on the data heist and install a spoofed Certificate Authority (CA) certificate on the client, which mitmproxy
conveniently provides. However, many clients (e.g., mobile phones) do not allow proxy configurations at all or simply ignore them. For these cases, mitmproxy
offers the option of a transparent proxy mode, in which the data are routed to the proxy instead of to the server using lower-level network tools such as iptables. The documentation on [2] illustrates solutions for these trickier cases.
Home-grown programs written in Go rely on the system's CA store for encrypted communications. On Ubuntu, it comes with the ca-certificates package. Out of the box, it doesn't contain the counterfeit mitmproxy
CA certificate, of course, but the sys admin can add it. As an example, Listing 1 [3] submits an HTTPS request to get the TLS-protected website of a well-known online giant. If mitmproxy
is running, the program aborts the request with the following error message if the client is rerouted via the proxy server by setting the HTTP_PROXY
environment variable:
Listing 1
client.go
01 package main 02 03 import ( 04 "fmt" 05 "io/ioutil" 06 "net/http" 07 ) 08 09 func main() { 10 resp, err := http.Get("https://amazon.com") 11 if err != nil { 12 panic(err) 13 } 14 defer resp.Body.Close() 15 body, err := ioutil.ReadAll(resp.Body) 16 fmt.Println(string(body)) 17 }
$ go build client.go $ HTTP_PROXY=localhost:8080 ./client panic: Get https://usarundbrief.com: x509: certificate signed by unknown authority
What happened? Instead of Amazon's certificate, the client received a self-signed certificate, injected by mitmproxy
, and the client rightfully complains about it. Now, to convince this pesky client to accept the certificate anyway (for testing purposes only, of course), the sys admin copies it into the Ubuntu system collection using the command sequence in Figure 7. And, sure enough, the following call to
$ HTTP_PROXY=localhost:8080 ./client
displays the TLS-protected Amazon website without any complaints, while mitmproxy
logs the transaction as an MITM.
By the way, Chromium on Ubuntu does not use the system store for trusted CAs; instead, it comes with its own collection. In order to add the counterfeit certificate to this collection, a few tricks are required, which mitmproxy
explains when the user accesses the magic mitm.it
domain in the browser with the proxy configured, after clicking the button with the desired operating system (Figure 8). It works similarly on Firefox, which offers a configuration menu for adding a new certificate below Options | Privacy & Security | View Certificates | Authorities | Import.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
KaOS Linux 2024.05 Adds Bcachfs Support and More
With updates all around, KaOS Linux now includes support for the bcachefs file system.
-
TUXEDO Computers Unveils New Iteration of the Stellaris Laptop Line
The Stellaris Slim 15 is the 6th generation and includes either an AMD or Intel CPU
-
KDE Releases Plasma 6.0.5
The latest release of the Plasma desktop has arrived with several improvements and the usual bug fixes.
-
Gnome OS Adopting systemd-sysupdate
Gnome OS is about to undergo a major under-the-hood change that promises enhanced security.
-
Endless OS 6 Now Available
After more than a year since the last update, the latest release of Endless OS is now available for general usage.
-
Fedora Asahi 40 Remix Available for Macs with Apple Silicon
If you've been anticipating KDE's Plasma 6 for your Apple Silicon-powered Mac, then you're in luck.
-
Red Hat Adds New Deployment Option for Enterprise Linux Platforms
Red Hat has re-imagined enterprise Linux for an AI future with Image Mode.
-
OSJH and LPI Release 2024 Open Source Pros Job Survey Results
See what open source professionals look for in a new role.
-
Proton 9.0-1 Released to Improve Gaming with Steam
The latest release of Proton 9 adds several improvements and fixes an issue that has been problematic for Linux users.
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.