Foundations of Python Network Programming (eBook)

Brandon Rhodes is a consulting programmer who also teaches the Python language professionally for organizations that are adding the language to their tool set. He has spoken at PyOhio; at PyGotham; at national PyCon conferences in Canada, Ireland, and Poland; and at Django conferences in Portland, Wales, and Warsaw, where he was delighted at the creativity of the organizers, who rented a circus tent for the occasion. He will chair the flagship PyCon North America conference in Portland in 2016–2017. Brandon is interested in how ideas like the Clean Architecture can help programmers organize code more effectively and in what we can learn from writers in other fields about offering kind and actionable critiques of each other’s work. He currently lives in tiny Bluffton, Ohio, with his wife Jackie and their two cats.

Contents at a Glance 3
Contents 358
About the Author 367
About the Technical Reviewers 368
Acknowledgments 369
Introduction 5
Chapter 1: Introduction to Client-Server Networking 9
The Building Blocks: Stacks and Libraries 9
Application Layers 12
Speaking a Protocol 13
A Raw Network Conversation 14
Turtles All the Way Down 17
Encoding and Decoding 18
The Internet Protocol 19
IP Addresses 20
Routing 21
Packet Fragmentation 22
Learning More About IP 23
Summary 24
Chapter 2: UDP 25
Port Numbers 26
Sockets 27
Promiscuous Clients and Unwelcome Replies 31
Unreliability, Backoff, Blocking, and Timeouts 32
Connecting UDP Sockets 36
Request IDs: A Good Idea 37
Binding to Interfaces 38
UDP Fragmentation 40
Socket Options 41
Broadcast 42
When to Use UDP 44
Summary 44
Chapter 3: TCP 46
How TCP Works 46
When to Use TCP 47
What TCP Sockets Mean 48
A Simple TCP Client and Server 49
One Socket per Conversation 52
Address Already in Use 53
Binding to Interfaces 54
Deadlock 55
Closed Connections, Half-Open Connections 60
Using TCP Streams Like Files 61
Summary 61
Chapter 4: Socket Names and DNS 63
Hostnames and Sockets 63
Five Socket Coordinates 64
IPv6 65
Modern Address Resolution 66
Using getaddrinfo() to Bind Your Server to a Port 67
Using getaddrinfo() to Connect to a Service 68
Asking getaddrinfo() for a Canonical Hostname 69
Other getaddrinfo() Flags 70
Primitive Name Service Routines 71
Using getsockaddr() in Your Own Code 71
The DNS Protocol 73
Why Not to Use Raw DNS 75
Making a DNS Query from Python 75
Resolving Mail Domains 77
Summary 79
Chapter 5: Network Data and Network Errors 80
Bytes and Strings 80
Character Strings 81
Binary Numbers and Network Byte Order 84
Framing and Quoting 86
Pickles and Self-delimiting Formats 91
XML and JSON 91
Compression 92
Network Exceptions 93
Raising More Specific Exceptions 95
Catching and Reporting Network Exceptions 96
Summary 97
Chapter 6: TLS/SSL 98
What TLS Fails to Protect 98
What Could Possibly Go Wrong? 99
Generating Certificates 100
Offloading TLS 103
Python 3.4 Default Contexts 104
Variations on Socket Wrapping 108
Hand-Picked Ciphers and Perfect Forward Security 109
Protocol Support for TLS 111
Learning Details 112
Summary 118
Chapter 7: Server Architecture 119
A Few Words About Deployment 119
A Simple Protocol 121
A Single-Threaded Server 124
Threaded and Multiprocess Servers 127
The Legacy SocketServer Framework 128
Async Servers 129
Callback-Style asyncio 132
Coroutine-Style asyncio 134
The Legacy Module asyncore 135
The Best of Both Worlds 137
Running Under inetd 137
Summary 139
Chapter 8: Caches and Message Queues 140
Using Memcached 140
Hashing and Sharding 143
Message Queues 145
Using Message Queues from Python 147
Summary 151
Chapter 9: HTTP Clients 153
Python Client Libraries 153
Ports, Encryption, and Framing 155
Methods 156
Paths and Hosts 157
Status Codes 158
Caching and Validation 160
Content Encoding 163
Content Negotiation 163
Content Type 165
HTTP Authentication 165
Cookies 167
Connections, Keep-Alive, and httplib 168
Summary 169
Chapter 10: HTTP Servers 170
WSGI 170
Asynchronous Server- Frameworks 172
Forward and Reverse Proxies 172
Four Architectures 173
Running Python Under Apache 175
The Rise of Pure-Python HTTP Servers 175
The Benefits of Reverse Proxies 175
Platforms as a Service 176
GET and POST Patterns and the Question of REST 177
WSGI Without a Framework 179
Summary 183
Chapter 11: The World Wide Web 184
Hypermedia and URLs 184
Parsing and Building URLs 185
Relative URLs 187
The Hypertext Markup Language 189
Reading and Writing to a Database 192
A Terrible Web Application (in Flask) 193
The Dance of Forms and HTTP Methods 198
When Forms Use Wrong Methods 200
Safe and Unsafe Cookies 200
Nonpersistent Cross-Site Scripting 202
Persistent Cross-Site Scripting 204
Cross-Site Request Forgery 204
The Improved Application 205
The Payments Application in Django 207
Choosing a Web Framework 211
WebSockets 212
Web Scraping 213
Fetching Pages 214
Scraping Pages 217
Recursive Scraping 219
Summary 223
Chapter 12: Building and Parsing E-Mail 224
E-Mail Message Format 224
Building an E-Mail Message 226
Adding HTML and Multimedia 228
Adding Content 233
Parsing E-Mail Messages 234
Walking MIME Parts 236
Header Encodings 238
Parsing Dates 240
Summary 240
Chapter 13: SMTP 241
E-mail Clients vs. Webmail Services 241
In the Beginning Was the Command Line 241
The Rise of Clients 242
The Move to Webmail 243
How SMTP Is Used 244
Sending E-Mail 246
Headers and the Envelope Recipient 246
Multiple Hops 247
Introducing the SMTP Library 248
Error Handling and Conversation Debugging 249
Getting Information from EHLO 252
Using Secure Sockets Layer and Transport Layer Security 254
Authenticated SMTP 256
SMTP Tips 258
Summary 258
Chapter 14: POP 259
POP Server Compatibility 259
Connecting and Authenticating 260
Obtaining Mailbox Information 263
Downloading and Deleting Messages 264
Summary 266
Chapter 15: IMAP 267
Understanding IMAP in Python 268
IMAPClient 270
Examining Folders 272
Message Numbers vs. UIDs 273
Message Ranges 273
Summary Information 274
Downloading an Entire Mailbox 275
Downloading Messages Individually 277
Flagging and Deleting Messages 283
Deleting Messages 283
Searching 284
Manipulating Folders and Messages 285
Asynchrony 286
Summary 287
Chapter 16: Telnet and SSH 288
Command-Line Automation 288
Command-Line Expansion and Quoting 289
Unix Command Arguments Can Include (Almost) Any Character 290
Quoting Characters for Protection 293
The Terrible Windows Command Line 294
Things Are Different in a Terminal 294
Terminals Do Buffering 297
Telnet 298
SSH: The Secure Shell 302
An Overview of SSH 303
SSH Host Keys 303
SSH Authentication 305
Shell Sessions and Individual Commands 306
SFTP: File Transfer Over SSH 311
Other Features 313
Summary 314
Chapter 17: FTP 315
What to Use Instead of FTP 315
Communication Channels 316
Using FTP in Python 317
ASCII and Binary Files 318
Advanced Binary Downloading 319
Uploading Data 321
Advanced Binary Uploading 322
Handling Errors 323
Scanning Directories 324
Detecting Directories and Recursive Download 326
Creating Directories, Deleting Things 327
Doing FTP Securely 328
Summary 328
Chapter 18: RPC 329
Features of RPC 330
XML-RPC 331
JSON-RPC 337
Self-Documenting Data 340
Talking About Objects: Pyro and RPyC 341
An RPyC Example 342
RPC, Web Frameworks, and Message Queues 344
Recovering from Network Errors 344
Summary 345
Index 346