My operation is as follows: let the mobile phone and the computer be in the same LAN (for example, connect to the same wifi), and then set up a proxy on the mobile phone's wifi. The computer uses Charles as the agent, and IP is the local area network IP of the computer. In my environment, the mobile phone IP is172.17.32.117, and the computer IP is. Then set the proxy port to 8888. After setting up the proxy, the next mobile phone request will be sent through the network card proxy request of the computer.

Actually, there is no need to go around like this. The reason why there is an extra agent is because the wifi hotspot mobile phone created by the computer can't be received. This is to make the mobile phone package sniff through the computer network.

The most convenient way is to put a wifi hotspot on the computer to connect to the mobile phone.

After creating the proxy connection, use Wireshark to sniff the network card. For example, I use etho0 network card to access the network here. At this time, when you play with your mobile phone and open several requests, a large number of captured data packets will appear on Wireshark. There are various protocols, including ARP search for you (looking for the physical address corresponding to IP), TCP connection package and HTTP request package.

Here, I set filtering rules for Netease to filter out one request after another, as shown below:

The whole process of a complete HTTPS request is as follows:

Next, the mobile phone is called A (172.17.32.211), and the computer is called B (172.17.32.19).

As the first TCP packet in the whole process, this paper analyzes it in detail to understand the format and content of TCP message. TCP is a transport layer protocol, which is responsible for reliable data communication. Its position in the whole architecture is as follows:

As a transport layer protocol, it mainly provides three functions for the upper layer protocol:

TCP protocol provides basic communication functions for HTTP and SSL protocols. So SSL protocol is based on TCP.

Three-way handshake's content is:

Analyze each package in detail:

A sends a packet with SYN synchronization bit to inform the server to establish a connection.

The results of the first handshake, the data of TCP packets sent out and Wireshark parsing are as follows:

The gray part is the data content of TCP message, and the first two bytes 0x8c85 = 35973 represent the source port.

The format of TCP message is as follows, corresponding to the gray part in the above picture. The non-gray parts are IP header and data frame header respectively.

Referring to Xie Xiren's book "Computer Network" and comparing with the full message format table, this paper explains the binary information and related meanings of the whole TCP message:

At this point, the fixed part of the TCP datagram header ends, and the fixed part I * * * has 20 bytes. In other words, the TCP header must have at least 20 bytes.

After the header is fixed, it is a variable-length option:

The size of the entire TCP packet can be expressed as follows:

A long string of messages after Wireshark points out some main information of TCP messages:

As can be seen from the above analysis, this SYN packet does not carry data, but according to the protocol, a serial number is consumed here.

After sending the SYN packet, terminal A enters the SYN-SENT state.

B receives SYN packet and sends SYN+ACK acknowledgement packet.

This packet is not only a packet to confirm the receipt of the first handshake, but also a synchronization packet sent by terminal B, indicating that it is ready to start sending data.

Compared with the first handshake packet, TCP packets can see some changes:

As you can see, the response timestamp of this packet is the transmission timestamp of the first handshake. It can also be understood that this packet is the one that responds to the first handshake.

Therefore, the receiver A can use this value to calculate the RTT this time. After receiving the packet of the second handshake, calculating the current timestamp minus the response timestamp of the packet is RTT delay.

Although this is an ACK packet, it is also a SYN packet, so it also consumes a sequence number.

After sending this packet, terminal B enters the SYN-REVD state.

Upon receipt, A sends an ACK acknowledgement packet.

The released software packages are as follows:

Here, we have a question Here, the sender A sends connection request information, the receiver B sends confirmation information, and the serial numbers are synchronized with each other. Is it possible to transmit data? But in fact, A must send another ACK acknowledgement message. As shown in the figure, confirm that you have received the data packet sent by B's second handshake. At this point, A and B will formally enter the setup state after this ACK packet. This is the first time in three-way handshake.

Why is this?

Suppose we used two handshakes, and then in the first handshake, A sent the handshake packet for the first time, and then there was a scene: it had been retransmitted once with the passage of time, and no response was received, and then the packet was sent again, and then we called the last packet invalid.

Then we can see:

Therefore, only after the receiver B sends out the handshake packet for the third time from the sender A will it think that the connection has been established and start waiting for the data sent by the sender A, so that the receiver will not be abnormal due to the invalid connection request message.

The timing diagram of TCP three-way handshake is as follows:

Three-way handshake, there are several important tasks. One is the synchronization serial number. Both the receiver and the sender send synchronous packets and inform each other of the initial sequence number, so that the packets received later can be reliably transmitted according to the sequence number. The other is to prepare both the sender and the receiver. Then start transmitting data.

The whole process takes place before the HTTP message is sent. HTTP protocol relies on TCP protocol to manage transmission. TCP can be regarded as its housekeeper, managing the large and small transactions of transmission, such as whether to ensure the order of packets is consistent? When will the contract be awarded? Do you want to collect your bags? TCP is very strict.

At the Java API level, three-way handshake corresponds to the creation of Socket connection (the last call is the native layer of socket creation):

ConnectTimeout here corresponds to the total duration of three-way handshake. If the time limit is exceeded, it will be considered as a connection failure.

For example, in a scenario, after the client sends a SYN message, it does not receive SYN+ACK from the server. At this point, the client triggers the retransmission mechanism, and the retransmission interval is doubled, and no data packet is received. Then, if this time exceeds the connection timeout setting, a connection timeout occurs.

Therefore, if the time spent by three-way handshake is always greater than the connectTimeout time here, this socket cannot establish a connection.

The three-way handshake time we requested this time is around 180ms.

For example, in OkHttp, if the connection in three-way handshake phase times out, there will be a retry mechanism. That is, re-establish the connection and resend the SYN message to initiate the TCP connection. When reconnecting, the route of the connection will change. If there is no alternative route, then it really fails.

Under the default configuration of OkHttp 3.9.0, the connection timeout is 10000ms = 10s. At OkHttpClient. constructor

In practical application, it should be adjusted according to business scenarios.

This requirement, in order to let Wireshark catch the bag of the mobile phone, I use the computer as an agent.

In fact, client A establishes a connection with proxy server B using HTTP protocol. Just like an ordinary HTTP request, it needs to carry IP+ port number, and if there is authentication, it will also carry authorization information, and proxy server B will use the authorization information for authentication. Then the proxy server will connect to the remote host and return to 200 after the connection is successful.

Wireshark captured a packet with two pieces of information, namely, creating a proxy:

Request message:

Response message:

HTTP CONNECT is a new command in HTTP 1. 1 that supports https encryption.

I took this step because I used an agent to grab the bag. If you capture the HTTPS package sent by the browser directly on the PC, there will be no such process.

Then let's consider why the proxy server needs this information, and the host name and port number to connect to.

This is because SSL encryption HTTP protocol was carried out later. Because the proxy server can't get the encryption key, it can't get the HTTP header, so it can't know which host the request is sent to. Therefore, use the CONNECT method to inform the proxy server of the host name and the corresponding port number.

This is also called HTTPS SSL tunneling protocol. After establishing this SSL tunnel, the agent will blindly forward the data.

The whole SSL protocol is actually divided into two layers, SSL recording protocol and other sub-protocols (SSL handshake protocol, SSL password change protocol, SSL warning protocol):

The relationship between these two protocols is actually the relationship of data encapsulation, and SSL handshake encapsulation protocol encapsulates other upper-layer protocols.

Encapsulated handshake protocol:

Encapsulate application data protocols, such as HTTP:

Encapsulated exchange cipher protocol:

Encapsulate alarm protocol:

Therefore, SSL recording protocol is actually the carrier of other protocols and only provides encapsulation function. Its format is:

MAC is a message verification code, which is used to verify the integrity of data and ensure that there is no tampering in the middle. The message verification code is weaker than the digital signature, and the digest is encrypted with a symmetric key. Digital signature uses asymmetric key encryption to distinguish between public key and private key.

The main purpose of the recording protocol is these, and it provides the following services for other SSL sub-protocols:

After the TCP three-way handshake ends and the proxy server connects successfully, the connection is established successfully, and the client A starts to initiate SSL connection, which will first enter the SSL handshake stage.

The main purposes of SSL handshake phase are as follows:

The process of SSL handshake is not static, according to the actual application scenario. There are three main types:

The complete interaction process of SSL handshake is as follows. Here, the server and the client are authenticated:

Our request only verifies the server, so 7, 8 and 9 do not exist.

Now analyze the content of each stage in detail.

Hello, customer

As the first handshake package of SSL handshake, we analyzed and understood the contents of the package in detail.

The following is the packet of this SSL protocol that Wireshark analyzed:

How to interpret this package, according to the analysis of SSL protocol, is actually divided into two parts:

Because it is a handshake process, there is no negotiation key, and it is still transmitted in clear text here. The data carrier of the recording protocol is plaintext SSL handshake protocol.

The format of SSL handshake protocol is:

We can get this information from the packet of handshake protocol:

Cryptographic suite is developed with the development of cryptography, and according to practical application, some passwords may be cracked, which may lead to security problems, so the latest and safest cryptographic suite is generally used.

In the Android system, under normal circumstances, when using SSLSocket to connect, it will bring its own password suite supported by the system by default. However, this one has a disadvantage, such as the encryption algorithm of some password suites is cracked or there are security loopholes, and the response is slow with the system upgrade. OkHttp will use the ConnectionSpec class to provide a series of the latest password suites when shaking hands with SSL. As can be seen from the comments, these cipher suites are fully supported above Chrome 5 1 and Android 7.0.

Then, these cipher suites intersect with the cipher suites supported by Android system and submit them to the server. In this way, in case there is a problem with any password suite, OkHttp official will drop the support. With the iteration of the version, the online library OkHttp library will continue to provide updated password suites and abandon those unsafe password suites. Access the application to update OkHttp immediately, so you don't have to wait for a slow system update.

If all the provided encryption suite servers don't support it, OkHttp has a fallback mechanism, so the next best thing is to choose an older suite.

Hello, server

The server receives the Hello from the client and gives the final configuration information through the configuration information of the client and the server itself.

The parsed Wireshark content is as follows:

Details are as follows:

certificate

The above server Hello developed the following asymmetric encryption algorithm.

The server issues a certificate, and the client verifies the identity of the server and takes out the public key carried by the certificate, which is the public key of the exchange encryption algorithm. That is, the ECDHE (EC Diffie-Hellman) algorithm specified in the Hello stage of the server, which is also commonly known as DH encryption.

This certificate message sends a certificate chain from digital certificate and CA certificate with its own public key. In the Certificate field:

CA is an important part of PKI system and is called certification body.

Then what is a CA certificate? Used to authenticate the validity of the certificate issued by CA center, which can ensure the reliability of the certificate and will not be replaced by intermediaries. But the CA certificate may also be intercepted and forged by the middleman? Then use another certificate to authenticate it. It seems endless. In fact, there is also a root CA certificate, which is stored in the browser or operating system and directly trusted by the system.

The server certificate requires a CA certificate for authentication. Digital signature is still used, which extracts a piece of information from the data and encrypts it with CA certificate. Then, during verification, it is decrypted with the public key of CA certificate, and the data part is summarized with the same summarization algorithm, and compared with the decrypted information.

The client has a process to verify the validity of the server certificate. First of all, you will find the certificate of this certificate, that is, the intermediate CA certificate. Then find the authentication certificate of the intermediate CA certificate, which can be another intermediate CA certificate or the root CA certificate. This goes up to the root CA certificate.

Then start with the root CA certificate and verify the digital signature step by step. For example, there is such a certificate chain: root CA certificate->; Intermediate CA certificate->; Server certificate. Verify the digital signature of the intermediate certificate with the public key of the CA certificate, and then verify the digital signature of the server certificate with the public key of the intermediate certificate. If any link fails to be verified, the certificate can be considered illegal.

This is the authentication process of the whole certificate chain:

Looking at the data of the caught bag, I found that there are only two certificates. It is a server certificate and an intermediate CA certificate. What about the root CA certificate? Follow the clue to find it.

First look at the server certificate. Its contents are as follows:

We can get a glimpse of this information from this certificate:

The first is the contents of the signedCertificate field, that is, the data of the digital certificate:

Then there is the signature information of the certification authority:

From the issuer above, we can know that the CA certificate of the authentication server certificate is GeoTrust SSL CA-G3, and we find the corresponding intermediate certificate from the Certificates as follows (the intermediate certificate can have several levels, but we only have one level here):

The intermediate certificate that can be obtained is called GeoTrust SSL CA-G3, and the certificate authority is GeoTrust Inc

What about the certificate that certifies the CA certificate? Still look at the distribution field, the certification certificate is called GeoTrust Global CA, and the institution is also GeoTrust Inc

Actually, this is the root CA certificate. Not found in this request, but can be found in the browser or operating system. Common browsers and systems will have this CA certificate built in. Therefore, the root certificate is trusted by the browser or the operating system, and no other certificate is needed as a guarantee.

If you want your system to trust the root CA certificate of a non-generic organization, then install it.

For example, my Windows system has GeoTrust Global CA certificate installed:

Just like we usually use Charles to catch HTTPS, this is the principle. Charles' CA certificate is installed in the mobile phone and becomes a trusted root CA certificate.

The basic principle is that Charles agent, as an SSL tunnel, has no transparent transmission, but acts as a middleman to intercept SSL handshake information and modify the CA certificate inside. The fake mobile phone establishes a connection with the real server, obtains the master key, then establishes an SSL connection between the fake server and the mobile phone client, modifies the CA and digital signature of the server certificate, and lets Charles analyze the encrypted HTTP content.

The modified server certificate is as follows. You can see that the publisher was replaced by Charles' certificate.