Exposing and Circumventing China's Censorship of ESNI

[net4people]

Exposing and Circumventing China's Censorship of ESNI

[gfw-report]

Starting from Thursday, August 13, 2020 6:32 AM UTC, we could not trigger ESNI blocking from the outside of China to the inside of China anymore from different vantage points. The last observed ESNI blocking triggered from outside-in was Thursday, August 13, 2020 6:27 AM UTC.

We confirm the ESNI blocking can still be triggered inside-out as of Thursday, August 13, 2020 7:50 AM UTC.

Could anyone corroborate our observation?

[wkrp]

EDIT: Triggering an ESNI block from the outside no longer works, since 2020-08-13 06:32. See below.

Because the GFW's ESNI detection is bidirectional, you can easily experiment with the blocking yourself, even if you are located outside of China.

Here is a short payload that triggers blocking. (ffce is the ESNI extension that the GFW is looking for.)

160303003b0100003703035b72616e646f6d72616e646f6d72616e646f6d7261
6e646f6d72616e646f6d5d0000000100000effce000a53754772000000000000

1. Choose any responsive TCP host in China. It doesn't have to be port 443. For example, www.tsinghua.edu.cn:80.
2. Start TCP-pinging the port using, for example, hping or Nping. You will see responses to your pings.

$ sudo hping3 -S www.tsinghua.edu.cn -p 80
HPING www.tsinghua.edu.cn (eth0 166.111.4.100): S set, 40 headers + 0 data bytes
len=44 ip=166.111.4.100 ttl=102 id=56332 sport=80 flags=SA seq=0 win=2105 rtt=279.8 ms
len=44 ip=166.111.4.100 ttl=102 id=63224 sport=80 flags=SA seq=1 win=2105 rtt=283.7 ms
len=44 ip=166.111.4.100 ttl=102 id=19078 sport=80 flags=SA seq=2 win=2105 rtt=287.5 ms
len=44 ip=166.111.4.100 ttl=102 id=1977 sport=80 flags=SA seq=3 win=2105 rtt=279.3 ms
len=44 ip=166.111.4.100 ttl=102 id=41003 sport=80 flags=SA seq=4 win=2105 rtt=283.1 ms

$ nping -4 -c 0 --tcp-connect www.tsinghua.edu.cn -p 80

Starting Nping 0.7.70 ( https://nmap.org/nping )
SENT (0.0752s) Starting TCP Handshake > www.tsinghua.edu.cn:80 (166.111.4.100:80)
RCVD (0.3682s) Handshake with www.tsinghua.edu.cn:80 (166.111.4.100:80) completed
SENT (1.0778s) Starting TCP Handshake > www.tsinghua.edu.cn:80 (166.111.4.100:80)
RCVD (1.3598s) Handshake with www.tsinghua.edu.cn:80 (166.111.4.100:80) completed
SENT (2.0798s) Starting TCP Handshake > www.tsinghua.edu.cn:80 (166.111.4.100:80)
RCVD (2.3584s) Handshake with www.tsinghua.edu.cn:80 (166.111.4.100:80) completed
SENT (3.0818s) Starting TCP Handshake > www.tsinghua.edu.cn:80 (166.111.4.100:80)
RCVD (3.3580s) Handshake with www.tsinghua.edu.cn:80 (166.111.4.100:80) completed
SENT (4.0840s) Starting TCP Handshake > www.tsinghua.edu.cn:80 (166.111.4.100:80)
RCVD (4.3603s) Handshake with www.tsinghua.edu.cn:80 (166.111.4.100:80) completed

3. Send the trigger payload.

printf '\x16\x03\x03\x00\x3b\x01\x00\x00\x37\x03\x03[randomrandomrandomrandomrandom]\x00\x00\x00\x01\x00\x00\x0e\xff\xce\x00\nSuGr\x00\x00\x00\x00\x00\x00' | nc -4 -v www.tsinghua.edu.cn 80

4. Now you will stop receiving replies to your pings for 120 or 180 seconds.

Python 3 program to generate trigger payload

#!/usr/bin/env python3

# Generates a small TLS ClientHello that trigger's the GFW's ESNI detector.
# Writes output to the file minimal.bin.
#
# You can send the ClientHello with, for example,
#     nc -v www.tsinghua.edu.cn 443 < minimal.bin

import struct

from scapy.all import *
load_layer("tls")
from scapy.layers.tls.all import *

# https://tools.ietf.org/html/rfc8446#section-3.4
def var(ceiling, data):
    if ceiling < 256:
        fmt = ">B"
    elif ceiling < 65536:
        fmt = ">H"
    else:
        raise ValueError(ceiling)
    return struct.pack(fmt, len(data)) + data

# https://datatracker.ietf.org/doc/html/draft-ietf-tls-esni-01#section-5
def encrypted_server_name(suite, group, key_exchange, record_digest, encrypted_sni):
    return struct.pack(">HH", suite, group) \
        + var(65535, key_exchange) \
        + var(65535, record_digest) \
        + var(65535, encrypted_sni)

clienthello = TLS(
    msg = TLSClientHello(
        gmt_unix_time = 0x5b72616e, # "[ran"
        random_bytes = b"domrandomrandomrandomrandom]",
        ciphers = [],
        ext = [
            # The GFW detector requires a syntactically valid
            # server_name_extension, but the actual values it contains may be
            # nonsense. Here we use a CipherSuite of 0x5375 ("Su"), a NamedGroup
            # of 0x4772 ("Gr"), and zero-length key_exchange, record_digest, and
            # encrypted_sni.
            TLS_Ext_Unknown(type=0xffce, val=encrypted_server_name(0x5375, 0x4772, b"", b"", b"")),
        ],
    )
)

TLS(bytes(clienthello)).show()
print(bytes(clienthello))

FILENAME = "minimal.bin"
open(FILENAME, "wb").write(bytes(clienthello))
print("output written to {}".format(FILENAME))

[wkrp]

Specifically, we replace the 0xffce in a triggering ClientHello with the values of 0xff02, 0xff03, and 0xff04 respectively. And no blocking is observed after sending such modified ClientHellos.

The GFW requires the 0xffce extension to be syntactically correct. Did you also try to produce syntactically correct 0xff02, 0xff03, and 0xff04 extensions, or did you simply replace 0xffce with each of those values? If it was simple replacement, then it leaves open the possibility that the GFW would block the newer ECH extensions if they were syntactically valid.

To show you what I mean, the syntax of the encrypted_server_name extension is:

 2 bytes  suite
 2 bytes  group
 2 bytes  key_exchange length
variable  key_exchange
 2 bytes  record_digest length
variable  record_digest
 2 bytes  encrypted_sni length
variable  encrypted_sni

Typical observed values when using Firefox and a Cloudflare TLS server are suite=0x1301 (TLS_AES_128_GCM_SHA256), group=0x001d (X25519), key_exchange length=32, record_digest length=32, and encrypted_sni length=292 (260 padded bytes plus 32 bytes of AEAD tag). The short trigger payload has suite=0x5375 ("Su"), group=0x4772 ("Gr"), and key_exchange length, record_digest length, and encrypted_sni length all zero, so it's syntactically correct although meaningless. But in my testing, if you increase encrypted_sni length, for example, without also appending the same amount of data to the extension, it will not trigger blocking.

[gfw-report]

Thank you @wkrp for sharing this step by step tutorial on reproducing the ESNI blocking.

Did you also try to produce syntactically correct 0xff02, 0xff03, and 0xff04 extensions, or did you simply replace 0xffce with each of those values?

We simply replaced the 0xffce with 0xff02, 0xff03 and 0xff04.

If it was simple replacement, then it leaves open the possibility that the GFW would block the newer ECH extensions if they were syntactically valid.

We agreed that this was a very good point and was also a valid concern. We will test the GFW against valid ECHs and get back to this thread.

But in my testing, if you increase encrypted_sni length without also appending the same amount of data to the extension, it will not trigger blocking.

This is a very interesting finding. It reminds us a similar finding we had when trying to determine GFW's minimal triggering condition of SNI-based censorship.

It is a bit off-topic but let us share our findings along with the (dirty) code here:

import struct

"""Extension - Server Name 

The client has provided the name of the server it is contacting, also
known as SNI (Server Name Indication).

Without this extension a HTTPS server would not be able to provide
service for multiple hostnames on a single IP address (virtual hosts)
because it couldn't know which hostname's certificate to send until
after the TLS session was negotiated and the HTTP request was made.

    00 00 - assigned value for extension "server name"
    00 18 - 0x18 (24) bytes of "server name" extension data follows
    00 16 - 0x16 (22) bytes of first (and only) list entry follows
    00 - list entry is type 0x00 "DNS hostname"
    00 13 - 0x13 (19) bytes of hostname follows
    65 78 61 ... 6e 65 74 - "example.ulfheim.net"

"""
def construct_sni(server_name):
    extension_value = bytearray.fromhex("0000")
    bytes_followed_data = struct.pack('>H', len(server_name) + 5)
    bytes_followed_entry = struct.pack('>H', len(server_name) + 3)
    list_entry_type = bytearray.fromhex("00")
    bytes_followed_hostname = struct.pack('>H', len(server_name))
    server_name_hex = bytearray(server_name, 'utf-8')
    return extension_value + bytes_followed_data + bytes_followed_entry + list_entry_type + bytes_followed_hostname + server_name_hex

# source: https://tls.ulfheim.net/
# Minimal ClientHello to trigger the payload.
def contruct_clienthello(server_name):
    record_header = ""
    handshake_header = ""

    # The GFW is not checking if the extension length value is correct or not.
    # But they must have the length.
    CLIENT_VERSION = bytearray.fromhex("0303")
    CLIENT_RANDOM = bytearray.fromhex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f")
    SESSION_ID = bytearray.fromhex("00")
    CIPHER_SUITE = bytearray.fromhex("0020cca8cca9c02fc030c02bc02cc013c009c014c00a009c009d002f0035c012000a")
    COMPRESSION_MEHTODS = bytearray.fromhex("0100")
    extensions_length = struct.pack('>H', len(construct_sni(server_name))) 

    # The GFW checks the Handshake Header (second section): num of bytes of ClientHello follows. 
    # If correct_length - 3 <= x <= correct_length, trigger RST.
    handshake_header = bytearray.fromhex("0100") + struct.pack('>H', 80 + len(server_name))

    # The GFW checks the Recrod Header (first section): total length. 
    # If >= correct_length, trigger RST,
    # < correct_length does not trigger RST. Why?
    record_header = bytearray.fromhex("160301") + struct.pack('>H', 4 + 80 + len(server_name))

    return record_header + handshake_header \
        + CLIENT_VERSION + CLIENT_RANDOM \
        + SESSION_ID \
        + CIPHER_SUITE \
        + COMPRESSION_MEHTODS \
        + extensions_length \
        + construct_sni(server_name)

legit_payload_hex = contruct_clienthello("example.youtube.com")

[klzgrad]

I'd like to start some speculations:

• Why block ESNI? This seems similar to the way it used to block HTTPS yet allow unencrypted HTTP traffic, i.e. a stopgap measure before better traffic inspection system is developed. But various findings above also suggest that a correct TLS 1.3 parser is in production in GFW. So the GFW is probably somewhere between being comfortable with SNI and experimenting with ESNI traffic.
• Why now? According to Wikipedia ESNI is only in production in Firefox since March 2020 and it is not supported in Chrome. Lack of Chrome support indicates poor utility in deploying this policy. But ESNI has been indeed been some time in the making. My bet would put this more on the experimental side.
• Why dropping traffic vs sending TCP resets? I was not aware of this new development, but it makes sense. Sending TCP resets as the way of blocking traffic was shown to be unreliable from time to time as it generates quite some more traffic which can be again lost, for which the resets are often sent in duplets or triplets. Dropping packets would be indeed more reliable. I knew from related Chinese literature that there are route blackholing infrastructures (gateway routers capable of maintaining and updating millions of blackhole rules in real time) linked with traffic detection frontends to handle this type of dynamic blocking, but taking only 1 second to enact such a rule is quite a result. I would suggest trying millisecond-level probes to test the latency from detection events to blocking events, which could be useful in mapping its internal topology. Because it takes some time for a rule to propagate from detection frontend to blocking backend, and longer if the propagation crosses geographical regions. If there are some rare cases that a (source IP, destination IP, and destination port) tuple is routed via both Beijing and Guangzhou, I imagine it would take longer for the Guangzhou data center to enact a rule detected in Beijing.
• Why 120 seconds vs 180 seconds? This could be related to its geographical features. Different data centers have different parameters and unifying it is more trouble than useful.
• Why additional detection event does not renew blocking timers and why it does in Iran? Renewing the timers would require update operations from detection frontends to the blackholing routers. These updates cost traffic, CPUs, resources. Not doing it would be a form of optimization that is more needed for handling China's national traffic than Iran's.

[gfw-report]

Thank you @klzgrad for your informative and inspiring comments.

I would suggest trying millisecond-level probes to test the latency from detection events to blocking events, which could be useful in mapping its internal topology.

Why 120 seconds vs 180 seconds? This could be related to its geographical features.

We will get back to this thread with experiment results.

[wkrp]

• Why now? According to Wikipedia ESNI is only in production in Firefox since March 2020 and it is not supported in Chrome. Lack of Chrome support indicates poor utility in deploying this policy. But ESNI has been indeed been some time in the making.

That's a good question. My take is that the GFW can afford to block ESNI now only because it is not yet widely used. If they waited until ESNI/ECH were essential to a large fraction of connections, then it would be more expensive. This is like a game where the first to move has an advantage.

RFC 8744, "Issues and Requirements for Server Name Identification (SNI) Encryption in TLS," has a requirement "Do Not Stick Out". There are two ways to meet this requirement. One way is to make connections whose SNI is encrypted indistinguishable from connections whose SNI is unencrypted. The other way is to do a sudden, massive deployment, so that even if encrypted-SNI connections are tagged and easily distinguishable, those tags become a feature of normal TLS traffic. If you want to blend in with a crowd, you can change yourself to match the surroundings; or you can change the surroundings to match yourself. I think the IETF was banking on the latter strategy being more likely of success, and I don't necessarily disagree. Deployment of encrypted SNI was always precarious, with a risk of failure.

The good news is that there will be a second chance with ECH (Encrypted Client Hello), which is the name for the latest revision of what was called ESNI. ECH uses different extension values which are not blocked yet, as far as we know.

My bet would put this more on the experimental side.

You may be right about that. We should not assume the ESNI block is permanent. The GFW sometimes institutes a new rule and later walks back. I am thinking of this case in 2016 when the GFW blocked an Azure CDN edge server for about four days, but did not re-block it when the server changed its IP address.

[gfw-report]

I knew from related Chinese literature that there are route blackholing infrastructures (gateway routers capable of maintaining and updating millions of blackhole rules in real time) linked with traffic detection frontends to handle this type of dynamic blocking, but taking only 1 second to enact such a rule is quite a result. I would suggest trying millisecond-level probes to test the latency from detection events to blocking events, which could be useful in mapping its internal topology.

During the time when ESNI blocking could still be triggered from the outside-in direction, we did the following experiment to test the latency from detection events to blocking events. This was a snapshot (one-time measurement) study, and was only tested from one point to another.

In specific, we used a script as follows:

#!/bin/bash

sudo -v

IP="REDUCTED"
PORT="80"

sudo tcpdump host "$IP" and port "$PORT" -Uw "delay.pcap" &

sleep 2

# wait 100000 micro seconds between each SYN
sudo hping3 -S "$IP" -p "$PORT" -i u100000 | tee output_hping.txt &

# send ESNI handshake
sudo python3 esni.py "$IP" "$PORT" | tee esni_output.txt &

sleep 10

sudo pkill python3
sudo pkill hping3

sleep 10

sudo pkill tcpdump

In each experiment, we changed the sending rate of the SYN ping. There is trade-off between the SYN ping rate: faster SYN ping rate allows us to have more precision in the blocking delay; but it may also introduce congestion and/or overwhelm the server with SYN flood.

We analyzed the pcap files captured. In specific, we tried to find the timestamps of the following three events: the time when ESNI sent from the client; the last SYN that got the SYN/ACK; and the first SYN that did not get the SYN/ACK. Blocking should happen between the last SYN that got the SYN/ACK and the first SYN that did not get the SYN/ACK. And the delay of blocking should be at most as long as the interval between ESNI sent from client and the first SYN that did not get the SYN/ACK.

The results, shown in relative timestamp, are as follows:

SYN ping rate: u100000 (wait 100000 micro seconds between each SYN):
1.243874s ESNI sent from client
1.305223s the last SYN that got the SYN/ACK
1.405656s the first SYN that did not get the SYN/ACK

u10000:
0.259244s ESNI sent from client
0.824456s the last SYN that got the SYN/ACK
0.834909s the first SYN that did not get the SYN/ACK

u1000:
0.251383s ESNI sent from client
2.068717s the last SYN that got the SYN/ACK
2.069755s the first SYN that did not get the SYN/ACK

u100 first experiment:
0.283411s ESNI sent from client
2.088801s the last SYN that got the SYN/ACK
2.088914s the first SYN that did not get the SYN/ACK
u100 second experiment:
0.290077s ESNI sent from client
2.114338s the last SYN that got the SYN/ACK
2.114449s the first SYN that did not get the SYN/ACK

In summary, the shortest observed delay of blocking between the time when GFW sees a ClientHello with ESNI and the time when all packets are dropped happened in the experiment when the sending rate was u100000. And the shortest delay was at most 1.405656s - 1.243874s = 0.161782 second.

[wkrp]

Starting from Thursday, August 13, 2020 6:32 AM UTC, we could not trigger ESNI blocking from the outside of China to the inside of China anymore from different vantage points. The last observed ESNI blocking triggered from outside-in was Thursday, August 13, 2020 6:27 AM UTC.

At about 2020-08-13 14:30 UTC, I see the same. I put the short trigger payload in a file minimal.bin, then ran the following commands. The TCP ping kept receiving responses even after sending the trigger payload.

nping -c 0 --tcp-connect www.tsinghua.edu.cn -p 80
ncat -v www.tsinghua.edu.cn 80 < minimal.bin

I had the same results with various destinations: www.tsinghua.edu.cn:80 (166.111.4.100:80), www.tsinghua.edu.cn:443 (166.111.4.100:443), www.china-railway.com.cn:80 (183.131.168.120:80), www.12306.cn:80 (61.147.210.242:80), www.miit.gov.cn:80 (202.106.121.6:80). I am not able to test from inside.

SNI-based TCP RST injection can still be triggered from outside. I tried the following commands. The first command returns a certificate and a working connection. The second command causes three immediate RSTs.

openssl s_client -connect www.tsinghua.edu.cn:443 -servername www.example.com
openssl s_client -connect www.tsinghua.edu.cn:443 -servername www.facebook.com

As a side note, today I am able to visit China-based web sites in Tor Browser (https://www.tsinghua.edu.cn/, http://www.china-railway.com.cn/, https://www.12306.cn/, http://www.miit.gov.cn/). I do not know if it is related. The GFW has, of course, blocked connections to Tor relays for a long time, but for the past few years it has also blocked connections from Tor relays, including exits. I cannot remember exactly when the outside-in blocking of Tor began, but I think it was in 2016 or 2017.