ProgrammerSought

Do you need to use an email service every day?

Email (email) is a long-established and commonly used form of messaging on the Internet. For most office workers, the first thing to do every day at work is to check for new emails. Although instant messaging tools are rapidly occupying the communication market, email still occupies a mainstream position in business or academic circles.

When you clicked "Send", what did your mailbox do? Today's experiment takes you to see for yourself.

Introduction: Internet Protocol and SMTP Protocol

In a previous article (click here to view), the author discussed several layers of the Internet protocol and built experiments to explore how HTTP servers for serving web pages work. If you also like to explore the principle, and have not done this experiment, then it is strongly recommended that you open the connection and follow the experiment designed in the article to explore the HTTP protocol. Here, we will do a similar experiment to spy on the SMTP protocol used for sending and receiving emails.

A simple summary of the principle: Our Internet is divided into four layers, and the normal work of each layer is based on the foundation of the layer below. Working at the top "application layer", there are HTTP protocols that provide web services, SMTP protocols that provide email sending and receiving, FTP protocols that provide file transfer, and so on. If these protocols want to work properly, they must be based on the support of the "transport layer" below. The most commonly used transport layer is the TCP protocol. In today's experiment, we will observe the SMTP protocol at the SMTP layer and the TCP layer, and construct a simple "SMTP server" on the TCP layer that requires manual control.

This diagram illustrates an SMTP connection. When we say SMTP communication, it is actually a virtual, abstract statement. The real connection is made on the layer below

Experiment 0 (preparatory work): View the email server address

Before actually starting the experiment, let's take a look at how to query the corresponding SMTP server domain name from an email address. For example, [email protected], we know its email domain name is 126.com. But we need to know what is the SMTP server address behind 126.com. In order to facilitate the distinction, 126.com is usually called the  email domain name  , and the SMTP server address behind it is called the  mx domain name  . (mx is short for mail exchange.)

Here we use the tool nslookup to query the mx domain name. Whether you're using Windows, Mac OS, or Linux, nslookup already exists on your computer. The steps to use it are as follows:

1. Open a command line. Windows system: Open the "Start" menu, enter "cmd", and search for the "Command Prompt" tool. After opening, the interface is as follows.

command prompt line in windows

Mac OS find "Terminal" in the application. Linux I won't say where the command line is.

1. Enter nslookup on the command line and press Enter to enter the nslookup tool. Enter set q=mx to specify the query mx domain name. Enter 126.com, press enter, and you will get the query results.

nslookup query the mx domain name of 126.com

As you can see in the picture, there are 4 mx servers behind the 126.com email domain name. In the following discussion, any mx server (such as "126mx01.mxmail.netease.com") will work. Friendly reminder: When copying the mx domain name, be careful not to copy the last period.

Lab 1 (Application Layer): Sending an Email Using Python

Of course, many times such emails will be rejected by the other party's SMTP server. Even if it is received, it may also be placed in spam due to unknown sources. Therefore, readers are not advised to use this to send and receive daily emails. But it's worth doing this once or twice in order to understand the SMTP protocol.

Without further ado, let's take a screenshot of a complete email sent. Note that as the sender, I did not log in to any of my own mailboxes. Also, note the format of the variable s_body. Most mail servers take this format seriously. Emails that do not conform to this format are often rejected. Finally, note that after the server.connect line is run, the back hand speed must be fast. If the next line is executed after a while, the other server is usually disconnected.

Use smtplib to send emails in Python. The recipient address, sender address and email content must be edited and stored in variables. Avoid server disconnection due to timeout not responding after connecting to server

Lab 2 (Transport Layer): Pretending to be an SMTP server using a TCP socket

How can you figure out what server.sendmail did with the server behind your back? Well, another question: Suppose you suspect that your partner will hook up with a beautiful girl/handsome guy on the Internet, how can you catch him/her? One way is to register an online account by yourself, disguise yourself as a beautiful girl/handsome boy, and chat with him/her.

We know that, like requests, if smtplib wants to carry out SMTP communication, it must use the TCP protocol of the transport layer below to establish a TCP connection with the other party's SMTP server. Therefore, like the previous article, we prepare a TCP connection and display the data sent by the other party on the screen. The specific messages and formats are clear at a glance.

Different from http request, server.sendmail is not a  single  request/response, but requires both parties to use the format specified in the protocol to  repeatedly ask and answer several times  to complete the mail sending. Therefore, we also need to carefully set the content of the response in our server to ensure that what is returned conforms to the format, so that the conversation can continue. (That is to say, it is a little harder to pretend to be an SMTP server than to pretend to be an HTTP server, and it is more likely to pass through.) In order to enhance the experience, let us fill in the information manually every time we receive it. Return content.

Let's look at the code first.

"""
This is a virtual server. When any program is introduced to it, it first sends a welcome message WELCOME_MSG,
Then wait for the other party to send the message. Each time the other party sends a message, it will display the message on the screen, and then prompt
We enter the response content. Immediately afterwards, it will add a newline character to the content of our input
, send it back.
"""
SERVER_IP = "localhost"
SERVER_PORT = 25 #One of the default SMTP
MAX_LENGTH = 1023 #Specify the upper limit of the length of each message.
WELCOME_MSG = "220 Virtual Server At Your Service!
" #welcome message
socket_list = []
import socket
def close_sockets(): #Close all ports when the program exits abnormally to avoid port occupation
 for sock in socket_list:
 sock.close()
def main():
 
 sock_listen = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
 socket_list.append(sock_listen)
 sock_listen.bind((SERVER_IP, SERVER_PORT))
 sock_listen.listen(1)
 conn, addr = sock_listen.accept()
 socket_list.append(conn)
 # Once connection is built: send welcome message and print connection
 print("Connection established. From: " + str(addr))
 if WELCOME_MSG is not None and WELCOME_MSG != '': # If WELCOME_MSG is '' or None,
 # No welcome message will be sent, after the link is established
 # Go directly to the state of receiving information
 conn.send(WELCOME_MSG.encode())
 print("Welcome message sent!")
 
 while True:
 print("Waiting for the answer from the other party... Maximum message length: " + str(MAX_LENGTH))
 data = conn.recv(MAX_LENGTH)
 print("Received message:
", data)
 if len(data) == 0:
 break
 reply_msg = input("Your reply: ")
 reply_msg += '
'
 conn.send(reply_msg.encode())
 print("Reply message sent!")
if __name__ == "__main__":
 try:
 main()
 except Exception as err:
 print(str(err))
 close_sockets()
 exit()

This code is relatively straightforward. Code comments or print prompts describe the function of each part of the code.

With this artificial server, we can use it to receive requests from smtplib. The previous demos used Windows and Mac OS computers. In order not to be biased, I will use a Linux computer as a demonstration here. (I won't tell you, it's because my wife is using the Windows computer and the Mac computer is in the office T_T)

1. Server is on. Note that since port 25 (one of the default ports of the SMTP protocol) is used, a port is reserved for the system, so the program requires administrator privileges. In the first picture, the first attempt was denied because it was not sudo with administrator privileges. After adding sudo, the program can be started, and the port is opened, waiting for a connection.

Start the server. The first time was denied for not using admin rights. The second time it is successfully opened, it enters the state of waiting for connection

1. Use Python3's smtplib to connect to the SMTP server. This step is the same as the experiment above. Note that server.connect connects to 'localhost'. At this time, the server in the figure on the right also shows that it has received the connection and sent a welcome message '220 Virtual Server At Your Service!' At this time, from the information received by smtplib, it recognizes this  status code <space> reply message  format, returns a two-element array.

Join the group: 548377875 to get dozens of PDFs and a lot of learning materials!

Python's smtp connection, the server sends a welcome message

Next, we repeat the practice in the above experiment, define s_from, s_to and s_msg, and then send it to the server.sendmail function to send it as an email. See below.

Send mail using smtplib's sendmail

1. The next diagram  is important! It shows the chain of messages  received on the server after the sendmail function is executed   .

A series of SMTP messages sent by the sendmail function to the server. Start with `helo`

Note that in order to display all newline characters, the Bytes type string is not decoded into a normal Python3 string (that is, the decode method is not called). So each piece of information in the picture has a b in front of it.

The conversation goes like this:

sendmail: helo [own address]

Me: 250 Nice to meet you # Note the format is "status code <space> response information".

sendmail: mail FROM:<sender address>

Me: 250 Ok

sendmail: rcpt TO: <address of recipient>

Me: 250 Ok

sendmail: data # This data word is to tell the other party, pay attention, I will start sending the text later!

Me: 354 Go ahead # The status code here is no longer 250, but 354, which means "I'm waiting for you to send a message"

sendmail: <mail body> # Note: The . at the end of this text is the data terminator defined by the SMTP protocol.

Me: 250 Received

At this point, the sendmail function has completed its task and sent an email. In fact, sendmail works normally and analyzes the status codes (250, 354) sent by each requesting party. For example, when sendmail sends data characters, if you still reply 250 instead of 354, sendmail will think that there is a problem with your server and ignore you. In contrast, the specific content of the following response information is not specifically required by the SMTP protocol. So there will be a variety of responses. For example, the content of gmail's server response to helo is "at your service", and what I wrote here is "Nice to meet you".

After everything is sent, sendmail returns the familiar {}, which is an empty dictionary, indicating that the message was sent successfully. Later I call server.quit() to end the conversation. As you can see from the figure below, this function sends a quit message to the server before disconnecting. It only closed the TCP connection after I responded with 221 bye.

After sendmail returns `{}`, the quit function is called to send the "end communication" message

Summarize

This experiment is a bit long. In the preparation stage (lab 0), we learned how to use nslookup to query the MX server address corresponding to an email domain name. In experiment 1, we used Python's smtplib package to send emails, observed the situation on the application layer (SMTP layer), and mastered the use of smtplib. In experiment 2, we dived to the TCP layer and started a simple artificial SMTP server to receive email sending requests from smtplib. I have seen the standard SMTP request messages such as helo , mail FROM , rcpt TO , data , quit , and also learned about the "status code <space> response message" format of the server response information. By the way, in fact, smtplib also provides server.helo , server.mail , server.rcpt , server.data these functions. Interested readers can try it out for themselves.

By doing this experiment by hand, I believe that readers will have a more intuitive understanding of the SMTP protocol. When sending emails in the future, will this series of requests sent by your email management program behind the scenes automatically appear in your mind?

Thank you for your support. If you have any questions, suggestions, or simple experiments to explore computers, please leave me a message!