The Practice of System and Network Administration

Thomas A. Limoncelli is Director of Operations at Lumeta Corporation, a venture startup that focuses on intranet security. His more than 11 years of experience includes time at Drew University, Mentor Graphics, and seven years supporting the researchers and scientists at Bell Labs. Christine Hogan is an independent consultant. Her 11-plus years of experience have been gained at a variety of different companies including Trinity College in Dublin, Synopsys, and Global Networking and Computing (GNAC). Currently, she is taking time out to earn a Ph.D. at Imperial College in London. 0201702711AB07052001

Preface.


Acknowledgments.


About the Authors.


Introduction.


Do These Now!


Use a Trouble-Ticket System.



Manage Quick Requests Right.



Start Every New Host in a Known State.



Conclusion.
I. THE PRINCIPLES.

1. Desktops.


The Basics.



Loading the System Software and Applications Initially.



Updating the System Software and Applications.



Network Configuration.



Dynamic DNS with DHCP.



The Icing.



High Confidence in Completion.



Involve Customers in the Standardization Process.



A Variety of Standard Configurations.



Conclusion.

2. Servers.


The Basics.



Buy Server Hardware for Servers.



Vendors Known for Reliable Products.



Does Server Hardware Really Cost More?



Maintenance Contracts and Spare Parts.



Data Backups.



Servers Live in the Data Center.



Same, Different, or a Stripped-Down OS on Clients.



Remote Administration Access.



Mirrored Root Disks.



The Icing.



Server Appliances.



Redundant Power Supplies.



Full and n + 1 Redundancy.



Hot-swap Components.



Separate Networks for Administrative Functions.



Opposing View: Many Inexpensive Workstations.



Conclusion.

3. Services.


The Basics.



Customer Requirements.



Operational Requirements.



Open Architecture.



Simplicity.



Vendor Relations.



Machine Independence.



Environment.



Restricted Access.



Reliability.



Single or Multiple Servers.



Centralization and Standards.



Performance.



Monitoring.



Service Rollout.



The Icing.



Dedicated Machines.



Full Redundancy.



Conclusion.

4. Debugging.


The Basics.



Learn the Customer's Problem.



Find the Problem's Cause and Fix It.



Have the Right Tools.



The Icing.



Better Tools.



Formal Training on the Tools.



End-to-End Understanding of the System.



Conclusion.

5. Fixing Things Once.


The Basics.



Fix Things Once, Rather Than Over and Over.



Avoid the Temporary Fix Trap.



Learning from Carpenters.



The Icing.



Conclusion.

6. Namespaces.


The Basics.



Namespaces Need Policies.



Namespaces Need Change Procedures.



Namespace Management Should Be Centralized.



The Icing.



One Huge Database That Drives Everything.



Further Automation.



Customers Do Many of the Updates.



Next-Level Namespace Ubiquity.



Conclusion.

7. Security Policy.


The Basics.



Build Security Using a Solid Infrastructure.



Ask the Right Questions.



Document the Company's Security Policies.



Basics for the Technical Staff.



Management and Organizational Issues.



The Icing.



Make Security Pervasive.



Stay Up-to-Date: Contacts and Technologies.



Produce Metrics.



Organization Profiles.



Small Company.



Medium-Size Company.



Large Company.



E-commerce Site.



University.



Conclusion.

8. Disaster Recovery and Data Integrity.


The Basics.



What Is a Disaster?



Risk Analysis.



Legal Obligations.



Damage Limitation.



Preparation.



Data Integrity.



The Icing.



Redundant Site.



Security Disasters.



Media Relations.



Conclusion.

9. Ethics.


The Basics.



Informed Consent.



Professional Code of Conduct.



Network/Computer User Code of Conduct.



Privileged Access Code of Conduct.



Copyright Adherence.



Working with Law Enforcement.



The Icing.



Setting Expectations on Privacy and Monitoring.



Being Told to Do Something Illegal/Unethical.



Conclusion.

II. THE PROCESSES.

10. Change Management and Revision Control.


The Basics.



Technical Issues.



Communications Structure.



Scheduling.



Process and Documentation.



Quiet Times.



The Icing.



Automated Front-Ends.



Change Management Meetings.



Streamline the Process.



Conclusion.

11. Server Upgrades.


The Basics.



The Steps in Detail.



The Icing.



Add and Remove Services at the Same Time.



Fresh Installs.



Reusing the Tests.



System Changelog.



A Dress Rehearsal.



Install Old and New Versions on the Same Machine.



Minimal Changes From the Base.



Conclusion.

12. Maintenance Windows.


The Basics.



Scheduling.



Planning.



Flight Director.



Change Proposals.



The Master Plan.



Disabling Access.



Mechanics and Coordination.



Deadlines for Change Completion.



Comprehensive System Testing.



Postmaintenance Communication.



Re-enable Remote Access.



Visible Presence the Next Morning.



Postmortem.



The Icing.



Mentoring a New Flight Director.



Trending of Historical Data.



Providing Limited Availability.



High-Availability Sites.



The Similarities.



The Differences.



Conclusion.

13. Service Conversions.


The Basics.



Small Groups First, Then Expand Communication.



Minimize Intrusiveness.



Layers Versus Pillars.



Avoid Flash-Cuts.



Successful Flash-Cuts.



Back-Out Plan.



The Icing.



Instant Roll-Back.



Avoid Explicit Conversions.



Vendor Support.



Conclusion.

14. Centralization and Decentralization.


The Basics.



Guiding Principles.



Candidates for Centralization.



Candidates for Decentralization.



The Icing.



Consolidate Purchasing.



Outsourcing.



Conclusion.

III. THE PRACTICES.

15. Helpdesks.


The Basics.



Have a Helpdesk.



A Friendly Face.



Staff Sizing.



Defined Scope of Coverage.



Defined Processes for Sta.



An Escalation Process.



Helpdesk Software.



The Icing.



Statistical Improvements.



Out of Hours and 24 x 7 Coverage.



Better Advertising for the Helpdesk.



Different "Desks" for Service Provision Versus Problem Resolution.



Conclusion.

16. Customer Care.


The Basics.



Ticket Tracking Software.



Phase A: The Greeting.



Phase B: Problem Identification ("What's Wrong?").



Phase C: Planning and Execution ("Fix It").



Phase D: Verification ("Verify It").



Perils of Skipping a Step.



Team of One.



The Icing.



Training Based on the Model.



The Single Point of Contact.



Increasing Customer Familiarity.



Special Announcements for Major Outages.



Trend Analysis.



Customers That Know the Process.



Architectural Decisions That Match the Process.



Conclusion.

17. Data Centers.


The Basics.



Picking a Location.



Access.



Security.



Power and Air.



Fire Suppression.



Racks.



Wiring.



Labeling.



Communication.



Console Servers.



Workbench.



Tools and Supplies.



Parking Spaces.



The Icing.



Greater Redundancy.



More Space.



Ideal Data Centers.



Tom's Dream Data Center.



Christine's Dream Data Center.



Conclusion.

18. Networks.


The Basics.



The OSI Model.



Clean Architecture.



Network Topologies.



Intermediate Distribution Frame.



Main Distribution Frame.



Demarcation Points.



Documentation.



Simple Host Routing.



Use Network Devices.



Overlay Networks.



Number of Vendors.



Standards-Based Protocols.



Monitoring.



Single Administrative Domain.



The Icing.



Leading-Edge Versus Reliability.



Multiple Administrative Domains.



Conclusion.

19. Email Service.


The Basics.



Privacy Policy.



Namespaces.



Reliability.



Simplicity.



Generality.



Automation.



Basic Monitoring.



Redundancy.



Scaling.



Security Issues.



Communication.



The Icing.



Encryption.



Backup Policy.



Advanced Monitoring.



High-Volume List Processing.



Conclusion.

20. Print Service.


The Basics.



Select the Level of Centralization.



Print Architecture Policy.



Designing the System.



Documentation.



Monitoring.



Environmental Issues.



The Icing.



Automatic Fail-Over and Load Balancing.



Dedicated Clerical Support.



Shredding.



Dealing with Printer Abuse.



Conclusion.

21. Backup and Restore.


The Basics.



Three Reasons for Restores.



The Backup Schedule.



Time and Capacity Planning.



Consumables Planning.



The Restore Process.



Backup Automation.



Centralization.



Tape Inventory.



The Icing.



Firedrills.



Backup Media and Off-Site Storage.



High DB Availability.



Technology Changes.



Conclusion.

22. Remote Access Service.


The Basics.



Remote Access Requirements.



Define a Remote Access Policy.



Define Service Levels.



Centralization.



Outsourcing.



Authentication.



Perimeter Security.



The Icing.



Home Office.



Cost Analysis and Reduction.



New Technologies.



Conclusion.

23. Software Depot Service.


The Basics.



Understand the Justification.



Understand the Technical Expectations.



Set the Policy.



Selecting Depot Software.



Create the Process Manual.



A Unix Example.



A Windows Example.



The Icing.



Different Configurations for Different Hosts.



Local Replication.



Including Commercial Software in the Depot.



Handling Second-Class Citizens.



Conclusion.

24. Service Monitoring.


The Basics.



Historical Data.



Real-Time Monitoring.



The Icing.



Accessibility.



Pervasive Monitoring.



Device Discovery.



End-to-End Tests.



Application Response Time Monitoring.



Scaling.



Conclusion.

IV. MANAGEMENT.

25. Organizational Structures.


The Basics.



Sizing.



Cost Centers.



Management Chain.



Appropriate Skills.



Infrastructure Teams.



Customer Support.



Helpdesk.



Outsourcing.



The Icing.



Consultants and Contractors.



Sample Organizational Structures.



Small Company.



Medium Company.



Large Company.



E-commerce Site.



Universities and Non-Profit Organizations.



Conclusion.

26. Perception and Visibility.


The Basics.



A Good First Impression.



Attitude, Perception, and Customers.



Align Your Priorities with Customer Expectations.



Be the System Advocate.



The Icing.



The System Status Web Page.



Management Meetings.



Be Visible.



Town Meetings.



Newsletters.



Mail to All Customers.



Lunch.



Conclusion.

27. Being Happy.


The Basics.



Organizing for Excellent Follow-Through.



Time Management.



Communication Skills.



Constant Professional Development.



Staying Technical.



The Icing.



Learn to Negotiate.



Loving Your Job.



Managing Your Manager.



Further Reading.



Conclusion.

28. A Guide for Technical Managers.


The Basics.



Responsibilities.



Working with Nontechnical Managers.



Working with Your Employees.



Decisions.



The Icing.



Make Your Team Even Stronger.



Sell Your Department to Senior Management.



Work on Your Own Career Growth.



Do Something You Enjoy.



Conclusion.

29. A Guide for Nontechnical Managers.


The Basics.



Morale.



Communication.



Sta Meetings.



Look for One-Year Plans.



Technical Staff and the Budget Process.



Professional Development.



The Icing.



Have a Five-Year Vision.



Meetings with Single Point of Contact.



Understand the Technical Staff's Work.



Conclusion.

30. Hiring System Administrators.


The Basics.



Job Description.



Skill Level.



Recruiting.



TimingIs Everything.



Team Considerations.



Select the Interview Team.



Interview Process.



Technical Interviewing.



Nontechnical Interviewing.



Sell the Position.



Employee Retention.



The Icing.



Get Noticed.



Conclusion.

31. Firing System Administrators.


The Basics.



Follow Your Corporate HR Policy.



Remove Physical Access.



Remove Remote Access.



Remove Service Access.



Fewer Access Databases.



The Icing.



A Single Authentication Database.



Monitoring System File Changes.



Conclusion.

Epilogue.
Appendix A. The Many Roles of a System Administrator.
Appendix B. What to Do When . . .
Appendix C. Acronyms.
Bibliography.
Index. 0201702711T08072001