Pro Python System Administration (eBook)

Rytis Sileika has over ten years experience in system administration and system design. Previously, he worked as a system integration engineer and platform designer for enterprises, designing and building telecom platforms. Currently, Rytis administers systems in one of the largest online betting exchanges.

Title Page 1
Copyright Page 2
Contents at a Glance 4
Table of Contents 5
About the Author 14
About the Technical Reviewer 15
Acknowledgments 16
Introduction 17
Prerequisites for This Book 17
Structure of This Book 18
The Example Source Code 20
CHAPTER 1 Reading and Collecting Performance Data Using SNMP 21
Application Requirements and Design 21
Specifying the Requirements 21
High-Level Design Specification 22
Introduction to SNMP 22
The System SNMP Variables Node 24
The Interfaces SNMP Variables Node 25
Authentication in SNMP 27
Querying SNMP from the Command Line 27
Querying SNMP Devices from Python 31
Configuring the Application 31
Using the PySNMP Library 33
The SNMP GET Command 34
The SNMP SET Command 35
The SNMP GETNEXT Command 36
Implementing the SNMP Read Functionality 38
Storing Data with RRDTool 38
Introduction to RRDTool 38
Using RRDTool from a Python Program 40
Creating a RoundRobin Database 40
Writing and Reading Data from the RoundRobin Database 42
Plotting Graphs with RRDTool 45
Integrating RRDTool with the Monitoring Solution 48
Creating Web Pages with the Jinja2 Templating System 51
Loading Template Files with Jinja2 52
The Jinja2 Template Language 53
Accessing Variables 53
Flow Control Statements 54
Generating Web Site Pages 55
Summary 59
CHAPTER 2 Managing Devices Using the SOAP API 60
What Is the SOAP API? 60
The Structure of a SOAP Message 61
Requesting Services with SOAP 61
Finding Information about Available Services with WSDL 63
SOAP Support in Python 64
Converting WSDL Schema to Python Helper Module 65
Defining Requirements for Our Load Balancer Tool 66
Basic Requirements 67
Code Structure 67
Configuration 68
Accessing Citrix Netscaler Load Balancer with the SOAP API 69
Fixing Issues with Citrix Netscaler WSDL 69
Creating a Connection Object 70
Logging In: Our First SOAP Call 73
Finding What Is Being Returned in the Response from a Web Service 75
How Is the Session Maintained After We Have Logged In? 76
Gathering Performance Statistics Data 78
SOAP Methods for Reading Statistical Data and Their Return Values 78
Reading System Health Data 79
Reading Service Status Data 81
Automating Administration Tasks 84
Device Configuration SOAP Methods 85
Setting a Service State 85
A Word About Logging and Error Handling 87
Using the Python logging Module 87
Logging Levels and Scope 87
Configuring and Using the Logger 89
Handling Exceptions 91
Summary 92
CHAPTER 3 Creating a Web Application for IP Address Accountancy 93
Designing the Application 93
Setting Out the Requirements 93
Making Design Decisions 94
Defining the Database Schema 95
Creating the Application Workflow 97
The Search and Display Functions 97
The Add Function 98
The Delete Function 98
The Modify Function 98
The System Health Check Function 98
The Name Resolution Function 98
The Basic Concepts of the Django Framework 98
What Is Django? 99
The Object-to-Relation Database Mapper 99
The Administration Interface 99
A Flexible Template System 99
Open Source Community Support 99
The Model/View/Controller Pattern 99
The Data Model Component 100
The View Component 100
The Controller Component 100
Installing the Django Framework 100
The Structure of a Django Application 101
Using Django with Apache Web Server 103
Implementing Basic Functionality 106
Defining the Database Model 106
URL Configuration 109
Using the Management Interface 111
Enabling the Management Interface 111
Allowing the Administration Plug-in to Manage New Models 113
Viewing Records 114
Using Templates 117
Deleting Records 120
Adding New Records 121
Modifying Existing Records 123
Summary 124
CHAPTER 4 Integrating the IP Address Application with DHCP 125
Extending the Design and Requirements 125
Extending the Database Schema 127
Additions to the Workflow 128
Adding DHCP Network Data 128
Defining Data Models 128
Additional Workflows 129
The Add Function 129
The Modify Function 131
The Delete Function 131
Extending DHCP Configuration with Address Pools 131
The Address Pool Data Model 132
Displaying DHCP Network Details 132
The Add and Delete Functions 134
Reworking the URL Structure 134
Generating URLs in the Model Class 135
Reverse Resolution of URLs 135
Assigning Names to URL Patterns 136
Using URL References in the Templates 136
Adding Client Classification 138
Additions to the Data Model 138
Using Template Inheritance 139
Class Rules Management 140
Generic Views 140
Displaying a List of Objects 140
A Detailed View of the Object 141
Adding and Modifying New Objects 143
Deleting Objects 144
Generating the DHCP Configuration File 145
Other Modifications 149
Resolving IPs to Hostnames 149
Checking Whether the Address Is In Use 150
Summary 153
CHAPTER 5 Maintaining a List of Virtual Hosts in an Apache Configuration File 154
Specifying the Design and Requirements for the Application 154
Functional Requirements 155
High-Level Design 155
Setting Up the Environment 155
Apache Configuration 156
Creating a Django Project and Application 156
Configuring the Application 157
Defining the URL Structure 158
The Data Model 159
The Basic Model Structure 160
Modifying the Administration Interface 164
Improving the Class and Object Lists 164
Customizing the Class Names 164
Adding New Fields to the Object List 165
Reorganizing the Form Fields 167
Adding Custom Object Actions 169
Generating the Configuration File 171
Summary 174
CHAPTER 6 Gathering and Presenting Statistical Data from Apache Log Files 175
Application Structure and Functionality 175
Application Requirements 176
Application Design 176
Plug-in Framework Implementation in Python 176
The Mechanics of a Plug-in Framework 177
Interface Model 177
Plug-in Registration and Discovery 178
Creating the Plug-in Framework 179
Discovery and Registration 179
Defining the Plug-in Modules 182
Log-Parsing Application 183
Format of Apache Log Files 183
Log File Reader 185
Calling the Plug-in Methods 189
Tagging the Plug-in Classes 190
Plug-in Methods and the Call Mechanism 190
Plug-in Modules 193
Installing the Required Libraries 193
Using the GeoIP Python Bindings 194
Writing the Plug-in Code 195
Summary 196
CHAPTER 7 Performing Complex Searches and Reporting on Application Log Files 197
Defining the Problem 197
Why We Use Exceptions 200
Are Exceptions Always a Bad Sign? 200
Why We Should Analyze Exceptions 200
Parsing Complex Log Files 202
What Can We Find in a Typical Log File? 202
The Structure of an Exception Stack Trace Log 203
Handling Multiple Files 205
Handling Multiple Files 205
Using the Built In Bzip2 Library 209
Traversing Through Large Data Files 209
What Are Generators, And How Do We Use Them? 209
Detecting Exceptions 211
Detecting Potential Candidates 211
Filtering Legitimate Exception Traces 212
Storing Data in Data Structures 213
The Structure of Exception Stack Trace Data 214
Generating an Exception Fingerprint for Unknown Exceptions 214
Detecting Known Exceptions 217
The Configuration File 218
Parsing XML Files with Python 219
Storing and Applying Filters 220
The Benefits of a Precompiled Search Over a Plain-Text Search 222
Producing Reports 222
Grouping Exceptions 223
Producing Differently Formatted Output for the Same Dataset 224
Calculating Group Statistics 224
Summary 226
CHAPTER 8 A Web Site Availability Check Script for Nagios 227
Requirements for the Check System 227
The Nagios Monitoring System 228
Nagios Plug-In Architecture 228
The Site Navigation Check 229
Installing the Beautiful Soup HTML Parsing Library 230
Retrieving a Web Page 230
Parsing the HTML Pages with Beautiful Soup 232
Adding the New Check to the Nagios System 238
Emulating the User Login Process 240
Summary 245
CHAPTER 9 Management and Monitoring Subsystem 246
Design 246
The Components 246
The Monitoring Server 247
The Monitoring Agent 248
The Sensors 248
The Data Objects 248
Configuration 249
Performance Readings 249
Site Configuration 249
Scheduling 249
The Data Structures 249
Introduction to Data Normalization 250
Configuration Data 252
Performance Data 255
Scheduling 256
Site Configuration 257
Representing the Information in an ER Diagram 257
Communication Flows 258
XML-RPC for Information Exchange 259
Structure 259
Python Support 260
CherryPy 261
The Server Process 262
Storing Data in a SQLite3 Database 262
Initializing the Database File 262
Actions 267
Accepting Sensor Readings 267
Supplying a New Configuration 268
Providing New Sensor Code 269
The Server Health Check 270
The Scheduler 270
Actions 270
Running Multiple Processes 270
Multithreading, Multiprocessing, and GIL 271
Basic Usage Patterns and Examples 272
Running Methods at Equal Intervals 275
A Simple Clock Implementation 275
A Cron-Like Scheduler 278
Ticket Dispatcher 279
Summary 282
CHAPTER 10 Remote Monitoring Agents 283
Design 283
The Passive Component 283
Architecture 283
Actions 283
Accepting a New Configuration 284
Upgrading the Sensors 284
Submitting Sensor Readings 284
The Security Model 285
Configuration 285
The ConfigParser Library 285
The File Format 286
Using the ConfigParser Class Methods 286
The Configuration Class Wrapper 291
The Configuration Class Wrapper 291
The Sensor Design 295
Running External Processes 296
Using the subprocess Library 296
Controlling the Running Processes 300
Communicating with External Processes 303
Using File Descriptors 303
Using File Objects 303
Using the Pipe Objects 304
Redirecting Standard Error 305
Automatically Updating Sensor Code 306
Sending and Receiving Binary Data with XML-RPC 306
Working with Files and Archives (TAR and BZip2) 307
Summary 310
CHAPTER 11 Statistics Gathering and Reporting 311
Application Requirements and Design 311
Using the NumPy Library 311
Installing NumPy 312
NumPy Examples 312
Working with Arrays 312
Basic Mathematical and Statistical Operations 316
Calculating the Mean and Standard Deviation 317
Finding the Trend Line of a Dataset 321
Reading and Writing Data to Files 323
Representing Data with matplotlib 323
Installing matplotlib 323
Understanding the Library Structure 324
Plotting Graphs 325
Changing the Appearance of the Plot Primitives 326
Drawing Bars and Using Multiple Axes 327
Working with Text Strings 329
Saving Plots to a File 330
Graphing Statistical Data 331
Collating Data from the Database 331
Displaying Available Hosts 331
Drawing Timescale Graphs 332
The Index Page 333
Host Details Page 334
Graph Collection Pages 337
Plotting Performance Graphs 339
Summary 342
CHAPTER 12 Automatic MySQL Database Performance Tuning 343
Requirements Specification and Design 343
Basic Application Requirements 344
System Design 344
Modifying the Plug-in Framework 346
Changes to the Host Application 346
Modifying the Plug-in Manager 347
Writing the Producer Plug-ins 348
Accessing the MySQL Database from Python Applications 348
Querying the Configuration Variables 353
Querying the Server Status Variables 355
Collecting the Host Configuration Data 356
Writing the Consumer Plug-ins 357
Checking the MySQL Version 358
Checking the Key Buffer Size Setting 360
Checking the Slow Queries Counter 361
Summary 362
CHAPTER 13 Using Amazon EC2/S3 as a Data Warehouse Solution 363
Specifying the Problem and the Solution 363
The Problem 363
Our Solution 364
Design Specifications 364
The Amazon EC2 and S3 Crash Course 365
Authentication and Security 365
Account Identifier 365
Access Credentials 366
X.509 Certificates 366
EC2 Key Pair 366
The Simple Storage System Concepts 366
The Elastic Computing Cloud Concepts 368
Amazon Machine Images and Instances 368
Elastic Block Store 371
Security Groups 372
Elastic IPs and Load Balancers 372
User Interfaces 372
Creating a Custom EC2 Image 373
Reusing Existing Images 373
Making Modifications 374
Install the Additional Packages 374
Create and Setup an Elastic Block Store Volume 374
Configure the MySQL Instance 375
Bundling the New AMI 376
Controlling the EC2 Using the Boto Python Module 378
Setting Up the Configuration Variables 378
Initializing the EC2 Instance Programmatically 379
Launching the EC2 Instance 380
Attaching the EBS Volume 381
Mounting the EBS Device 381
Starting the MySQL Instance 382
Transferring the Data 383
Destroying the EC2 Instance Programmatically 383
Shutting Down the MySQL Instance 383
Unmounting the File System 383
Detaching the EBS Volume 384
Taking a Snapshot of the Volume 384
Shutting Down the Instance 384
The Control Sequence 385
Summary 386
Index 387