Part Two
Last updated
Last updated
The chart shows how quickly the internet is becoming more encrypted, based on Firefox page loads. The data comes from Firefox Telemetry, and the chart is hosted by Let's Encrypt.
The chart highlights the need to consider how outbound HTTPS encryption affects security monitoring. Also, HTTPS is faster than HTTP, and encryption continues to grow.
The key is to think about the security risks and decide what works best for your organization. Also, your approach might change over time as things are quickly evolving. Some organizations now accept more monitoring as a necessary part of security, even if they didn't before.
Since HTTP uses untrusted networks, securing its transmission is important. Encryption helps keep the communication private since HTTP is cleartext.
TLS 1.0 is built on SSL 3.0, the last version made by Netscape. The name changed, but TLS is just a new label for SSL since IETF took over. TLS 1.2 is the most used version now, despite being over ten years old and having some security issues. TLS 1.3 is a big update, but it will take time for servers, clients, and apps to support it.
Here are more complete details of the TLS handshake process:
The client starts the handshake by sending a "hello" message to the server. This message lists the TLS version and cipher suites it supports, along with random bytes called "client random."
The "server hello" message is the server's response to the client hello. It includes the server's SSL certificate, the chosen cipher suite, and a random string of bytes created by the server.
Authentication: The client checks the server's SSL certificate with the certificate authority to ensure the server is legitimate and that the client is communicating with the real domain owner.
The client sends a random byte string called the "premaster secret." This secret is encrypted with the server's public key, and only the server's private key can decrypt it. The client gets the public key from the server's SSL certificate.
Private key used: The server decrypts the premaster secret.
Session keys created: Both the client and server use the client random, server random, and premaster secret to generate session keys that match.
Client is ready: The client sends a "finished" message that is encrypted with a session key
Server ready: The server sends a "finished" message encrypted with a session key. The handshake is done, and communication continues with the session keys.
HTTP/2, defined in RFC 7540, aims to improve the performance of HTTP, which is widely used. Enhancing HTTP's speed could greatly affect its usage. HTTP/1.1 was created in 1999, before the rise of major video streaming and social media platforms like Facebook, founded in 2004.
HTTP/2 improves performance by changing how HTTP works. This mainly affects browsers, web servers, and the infrastructure in between, not web applications or their development. There are major changes with HTTP/2.
Even though a browser can send many HTTP requests, it must receive and process responses one at a time. You may have noticed this when a webpage takes a long time to load; sometimes one big image loads first, then all the other images suddenly appear because that large file was slowing things down. Over the years, various tricks have been used to fix this, but HTTP/2 simplifies things. It lets all HTTP traffic go through one TCP connection, allowing requests and responses to be sent and received at the same time.
HTTP headers will be compressed using HPACK to improve performance by reducing the size of repeated headers sent in plain text. HPACK was created for HTTP/2, but not all HTTP/2 data is encrypted by default. Many people wrongly think HTTP/2 automatically means HTTPS everywhere. Although there were talks about requiring encryption for HTTP/2, it’s not a part of the standard.
QUIC is essentially HTTP/2 over UDP using TLS 1.3:
It is a new transport that lowers latency compared to TCP. It works like TCP, TLS, and HTTP/2 but uses UDP instead. Changing TCP is hard because it’s part of the operating system and network devices, but QUIC, built on UDP, doesn't have those limits.
QUIC is often overlooked by many organizations and penetration testers. Here’s a Wireshark screenshot showing the author connecting to images.google.com through Chrome using QUIC.
QUIC is a protocol that replaces TCP with a faster and more efficient UDP method while keeping TCP's reliability. It's always encrypted and uses TLS 1.3 for security. QUIC is already used in Windows 10 with the Edge browser and other apps. With SMB over QUIC, it can replace TCP/IP, providing secure SMB communication, even if SMB encryption isn't turned on, and it uses port 443. Admins can enable this feature by setting up a Windows Server on the network edge, installing a trusted certificate, and turning on the QUIC option, or by using it with Azure Files.
We have two design imperatives for SMB over QUIC:
Secure: Stop hackers from spying or pretending to be someone else and protect sensitive file data and user credentials. All SMB communication, including capabilities, authentication, and messages, happens within the QUIC layer, similar to IPSEC or VPN tunnels, covering NTLM challenges too.
Simple: Users won't have to change their experience with SMB over QUIC from what they're used to in the office, as retraining is costly. Therefore, we won't add new UI options or command-line arguments. Their Windows 10 machines will still try TCP and RDMA first as usual, then briefly switch to QUIC. This way, they'll use faster local connections like RDMA or TCP when possible, and switch to QUIC when traveling or if required by an admin, all without the user noticing any difference.
Microsoft has a good explanation of the TLS handshake and the pre-master secret:
The client sends a "Hello" message to the server with a random value and a list of supported encryption methods.
The server replies with a "Server hello" message and its random value.
The server sends its certificate to the client to verify its identity and might ask for the client's certificate too. Then, the server sends a "Server hello done" message.
If the server asks for a certificate, the client sends it.
The client makes a random Pre-Master Secret, encrypts it with the server's public key, and sends it to the server.
The server gets the Pre-Master Secret. Both the server and client create the Master Secret and session keys from it.
The client tells the server it's ready to use new keys for encryption by sending a "Change cipher spec" message and a "Client finished" message.
The server gets a "Change cipher spec" message and switches to symmetric encryption with the session keys. Then, it sends a "Server finished" message to the client.
The client and server can now share data securely. All messages sent between them are encrypted with a session key.
Web application penetration testers are very familiar with TLS interception/proxying. That's how interception proxies such as ZAP and Burp Suite are able to inspect (and manipulate) HTTPS traffic.
Here is an example of ZAP's (https://www.zaproxy.org/) Root CA certificate:
Wireshark can decrypt TLS traffic if you give it the right secrets. There are two ways to do this:
Key log file using per-session secrets.
Decryption using an RSA private key.
A key log file allows decryption even when Diffie-Hellman is used, unlike the RSA private key, which only works in some cases. This file is created by apps like Firefox, Chrome, and curl when the SSLKEYLOGFILE environment variable is set. Their libraries (NSS, OpenSSL, or BoringSSL) save session secrets to the file, which can be used in Wireshark for decryption.
The IDS uses the web server's private key to decode traffic, but this only works for servers the organization controls, not third-party sites like banks. Wireshark can also decode TLS traffic with the server's private key.
This won't work if the web server uses Perfect Forward Secrecy (PFS), which is required by TLS 1.3. TLS 1.3 can't be inspected passively.
Wireshark's site has new steps for setting up TLS decryption with the RSA private key.
In Wireshark 3.0 and later, go to Edit -> Preferences -> RSA Keys to add a key file or token. You may need to enter a password or configure a provider DLL/.so file for HSM keys.
The RSA key file can be either a PEM text file or a PKCS#12 binary file (.pfx or .p12).
The deprecated RSA keys list might be removed. Use the RSA keys dialog to configure keys. To change the protocol for decrypted data, right-click a TLS packet and use "Decode As." The IP address and Port fields are not used.
The IDS gets a copy of each client’s pre-master secret. Firefox and Chrome can export this secret, but Internet Explorer and Edge do not.
Wireshark's instructions for decrypting TLS traffic via the client pre-master secret:
Close the browser completely (check your task manager just to be sure).
Set environment variable SSLKEYLOGFILE to the absolute path of a writable file.
Start the browser.
Verify that the location from step 2 is created.
In Wireshark, go to Preferences -> Protocols -> TLS, and change the (Pre)-Master-Secret log filename preference to the path from step 2.
Start the Wireshark capture.
Open a website
Check that the decrypted data is visible. For example, using the tls and (http or http2) filter.
If you create a symmetric session key with asymmetric encryption and the private key gets hacked, are old symmetric keys safe? If other session keys are also hacked, do the others stay safe?
Perfect Forward Secrecy ensures that past symmetric session keys stay secure, even if the private key or other session keys are compromised. Some think the term "perfect" is too strong, so "forward secrecy" is also used.
Google implemented forward secrecy in their HTTPS implementation for Gmail:
Forward secrecy means that private keys for a connection aren't stored long-term. If an attacker gets one key, they can't decrypt past connections, and even the server can't unlock old HTTPS sessions.
Network diagnostics and troubleshooting will be greatly affected. To solve complex issues, TLS decryption of network packet traces is necessary to track transactions and find problems. The visibility from out-of-band TLS decryption can't be matched by MITM setups or endpoint diagnostics. Losing this visibility will lead to long outages as support teams have to guess when facing tough problems.
Endpoint Monitoring: This method can't replace the broad network visibility that companies will lose without RSA key exchange. Keeping a monitoring agent on every endpoint is much harder than just having a network traffic inspection tool. For monitoring employee communications, moving to endpoint monitoring creates new security issues, as the main threat comes from the person using the endpoint.
Sticking with TLS 1.2 might work for now, but it could be hard to maintain in the future. As network interception of encrypted traffic becomes tougher, more people will likely use host-based solutions. Since clients can access unencrypted data, applications on the client can also see that data.
The downside of the pre-master secret:
Firefox and Chrome are the only major graphical browsers that support this functionality. Curl also supports this functionality
The decrypting device must have the key log from every supported browser
Just because you can do something doesn’t mean you should or that it's legal. Privacy laws and rules differ widely around the world. Make sure to get approval for any plans to decrypt user traffic from the right people in your organization before moving forward.
Organizations are increasingly decrypting their users' outgoing traffic. This is partly because more tools for decryption are now available. Additionally, as more internet traffic becomes encrypted, organizations need to maintain visibility, which has led to greater adoption of decryption practices.
Many are eager to decrypt everything to find threats, but take a moment to consider. There may be legal rules preventing your organization from decrypting certain traffic. Even without legal issues, your organization might have policies against decrypting specific data. Generally, traffic with employee health or financial information is treated as sensitive and is not decrypted. While organizations try to avoid decrypting this type of traffic, it can depend on the tools used. Often, site categorization rules help decide what can be decrypted, but these rules aren't foolproof.
Some traffic can't be decrypted for technical reasons, even outside of legal or policy issues. Ideally, your decryption tool should have a list of apps or sites that don't work well with decryption. If it doesn't, be ready for that problem. One reason for this issue is certificate pinning, which prevents man-in-the-middle (MITM) attacks but can also cause apps to fail during TLS decryption, which acts like an authorized MITM.
Goal: decrypt TLS 1.2 traffic sent to the Sec511 Attack Map at https://www.sec511.net:8443 (running locally at 127.0.0.1:8443), using the web server's RSA private key.
In Wireshark: let's double-click on the "Loopback: lo" adapter to begin packet capture.
Wireshark will start capturing packets.
In Firefox: let's click the bookmark shortcut for SEC511 Attack Map.
Let's click the Lock icon to view the TLS connection details, then click the ">" next to "Connection secure."
Then click "More information".
Look at the "Technical Details".
This web server uses "TLS_RSA_WITH_AES_128_CBC_SHA" encryption with TLS 1.2. We can decrypt this using Wireshark if we have the server's Private RSA key. Since this is a local web server, we can easily access that key.
The server's private key is found at /etc/ssl/private/sec511_net.key. It can only be read by the root user for security reasons, so Wireshark cannot import it directly. Let's check its permissions:
Let's copy the key to the /labs/tls directory:
Next, we'll change the permissions so Wireshark can read it:
Then set the file's group to 'wireshark' and check the updated permissions.
We changed the file permissions from only being readable by the root user to being readable by the root user and users in the wireshark group. This helps keep the key safe from other accounts. While this isn't crucial for this lab, it shows a better way to handle sensitive RSA private keys.
The student account is part of the Wireshark group, so we can now see the key without using sudo.
Now let's configure Wireshark to use that RSA key. In Wireshark: let's go to Edit -> Preferences
Then let's select "RSA Keys" -> "Add New keyfile…"
Next, navigate to /labs/tls/sec511_net.key, click "Open" and "OK".
Now, let's close Firefox completely and restart it to make sure it starts a new TLS session instead of continuing an old one. Then stop the capture in Wireshark by pressing the red stop square.
To start capturing again, we'll click the green shark fin icon and select "Continue without Saving."
Now, let's repoen Firefox and Inspect the decrypted TLS 1.2 traffic.
Let's enter the following display filter:
Wireshark is now able to decrypt TLS 1.2. Check any "GET" request where the destination port is 8443, and look at the "Transport Layer Security" part in the packet details section.
Let's click on any GET, and then click "Decrypted TLS".
The plain text request is shown.
Let's see the content of country_centroids_primary.csv, we'll do this by finding the GET request in the decrypted HTTP stream. Right-click on it and select Follow -> HTTP Stream. Make sure to pick "HTTP Stream."
Let's scroll down to find the CSV download.
The screenshot above shows decrypted traffic to using the browser's pre-master secret. PayPal uses HTTP/2, which looks different from HTTP/1.1. This demonstrates how the pre-master secret can decrypt TLS traffic, even for sites using TLS 1.3.
