Shell Scripting

Steve Parker is an IT consultant specializing in Solaris and GNU/Linux. He has been providing consultancy services for more than a decade. He is the author of the popular Bourne Shell Programming/Scripting Tutorial (http://steve-parker.org/sh/sh.shtml), which sees more than one million visitors a year.

Introduction xxix

Part I: About the Ingredients

Chapter 1: The History of Unix, Gnu, and Linux 3

Unix 3

“Everything is a File” and Pipelines 5

BSD 6

GNU 7

Linux 11

Summary 12

Chapter 2: Getting Started 15

Choosing an OS 15

GNU/Linux 15

The BSDs 17

Proprietary Unix 17

Microsoft Windows 17

Choosing an Editor 18

Graphical Text Editors 18

Terminal Emulation 21

Nongraphical Text Editors 22

Setting Up the Environment 24

The Shell Profile 24

Aliases 26

vim Settings 30

Summary 31

Chapter 3: Variables 33

Using Variables 33

Typing 34

Assigning Values to Variables 35

Positional Parameters 39

Return Codes 42

Unsetting Variables 45

Preset and Standard Variables 47

BASH_ENV 47

BASHOPTS 47

SHELLOPTS 48

BASH_COMMAND 50

BASH_SOURCE, FUNCNAME, LINENO, and BASH_LINENO 51

SHELL 55

HOSTNAME and HOSTTYPE 55

Working Directory 55

PIPESTATUS 55

TIMEFORMAT 56

PPID 57

RANDOM 58

REPLY 58

SECONDS 58

BASH_XTRACEFD 59

GLOBIGNORE 60

HOME 62

IFS 62

PATH 63

TMOUT 64

TMPDIR 65

User Identification Variables 65

Summary 66

Chapter 4: Wildcard Expansion 67

Filename Expansion (Globbing) 67

Bash Globbing Features 70

Shell Options 71

Regular Expressions and Quoting 75

Overview of Regular Expressions 76

Quoting 77

Summary 81

Chapter 5: Conditional Execution 83

If/Then 83

Else 85

elif 85

Test ([) 87

Flags for Test 88

File Comparison Tests 95

String Comparison Tests 96

Regular Expression Tests 98

Numerical Tests 101

Combining Tests 103

Case 105

Summary 109

Chapter 6: Flow Control Using Loops 111

For Loops 111

When to Use for Loops 112

Imaginative Ways of Feeding “for” with Data 112

C-Style for Loops 118

while Loops 119

When to Use while Loops 119

Ways to Use while Loops 119

Nested Loops 125

Breaking and Continuing Loop Execution 126

while with Case 130

until Loops 131

select Loops 133

Summary 137

Chapter 7: Variables Continued 139

Using Variables 139

Variable Types 141

Length of Variables 142

Special String Operators 144

Stripping Variable Strings by Length 144

Stripping from the End of the String 146

Stripping Strings with Patterns 147

Searching Strings 151

Using Search and Replace 151

Replacing Patterns 153

Deleting Patterns 153

Changing Case 153

Providing Default Values 153

Indirection 157

Sourcing Variables 158

Summary 159

Chapter 8: Functions and Libraries 161

Functions 161

Defining Functions 162

Function Output 162

Writing to a File 164

Redirecting the Output of an Entire Function 167

Functions with Trap 171

Recursive Functions 173

Variable Scope 177

Libraries 181

Creating and Accessing Libraries 183

Library Structures 183

Network Configuration Library 187

Use of Libraries 191

getopts 191

Handling Errors 194

getopts within Functions 195

Summary 197

Chapter 9: Arrays 199

Assigning Arrays 199

One at a Time 200

All at Once 200

By Index 201

All at Once from a Source 201

Read from Input 203

Accessing Arrays 205

Accessing by Index 205

Length of Arrays 206

Accessing by Variable Index 206

Selecting Items from an Array 209

Displaying the Entire Array 209

Associative Arrays 210

Manipulating Arrays 211

Copying an Array 211

Appending to an Array 213

Deleting from an Array 214

Advanced Techniques 216

Summary 217

Chapter 10: Processes 219

The ps Command 219

ps Line Length 220

Parsing the Process Table Accurately 220

killall 223

The /proc pseudo-filesystem 225

prtstat 226

I/O Redirection 227

Appending Output to an Existing File 229

Permissions on Redirections 229

exec 229

Using exec to Replace the Existing Program 230

Using exec to Change Redirection 231

Pipelines 237

Background Processing 237

wait 238

Catching Hangups with nohup 239

Other Features of /proc and /sys 242

Version 242

SysRq 242

/proc/meminfo 245

/proc/cpuinfo 245

/sys 246

/sys/devices/system/node 251

sysctl 253

Summary 254

Chapter 11: Choosing and Using Shells 255

The Bourne Shell 256

The KornShell 256

The C Shell 256

The Tenex C Shell 257

The Z Shell 257

The Bourne Again Shell 257

The Debian Almquist Shell 258

Dotfiles 258

Interactive Login Shells 259

Interactive Non-Login Shells 260

Non-Interactive Shells 261

Logout Scripts 262

Command Prompts 262

The PS1 Prompt 262

The PS2, PS3, and PS4 Prompts 264

Aliases 265

Timesavers 265

Modifying Behaviors 265

History 266

Recalling Commands 267

Searching History 267

Timestamps 268

Tab Completion 269

ksh 269

tcsh 270

zsh 270

bash 271

Foreground, Background, and Job Control 272

Backgrounding Processes 272

Job Control 273

nohup and disown 275

Summary 276

Part II: Recipes For Using and Extending System Tools

Chapter 12: File Manipulation 279

stat 279

cat 281

Numbering Lines 282

Dealing with Blank Lines 282

Non-Printing Characters 283

cat Backwards is tac 284

Redirection 285

Redirecting Output: The Single Greater-Than Arrow (>) 285

Appending: The Double Greater-Than Arrow (>>) 286

Input Redirection: The Single Less-Than Arrow (<) 288

Here Documents: The Double Less-Than Arrow (<< EOF) 290

dd 292

df 294

mktemp 295

join 297

install 298

grep 300

grep Flags 300

grep Regular Expressions 301

split 303

tee 304

touch 306

find 307

find-exec 310

Summary 313

Chapter 13: Text Manipulation 315

cut 315

echo 316

dial1 316

dial2 319

fmt 320

head and tail 323

Prizes 323

World Cup 324

od 328

paste 331

pr 334

printf 335

shuf 337

Dice Thrower 337

Card Dealer 338

Travel Planner 340

sort 341

Sorting on Keys 342

Sorting Log Files by Date and Time 344

Sorting Human-Readable Numbers 345

tr 346

uniq 350

wc 351

Summary 352

Chapter 14: Tools For Systems Administration 353

basename 353

date 355

Typical Uses of date 355

More Interesting Uses of date 359

dirname 360

factor 362

identity, groups, and getent 364

logger 367

md5sum 368

mkfi fo 370

Master and Minions 371

Reversing the Order 373

Networking 375

telnet 376

netcat 376

ping 378

Scripting ssh and scp 381

OpenSSL 383

nohup 390

seq 391

Integer Sequences 391

Floating Point Sequences 393

sleep 394

timeout 394

Shutdown Script 396

Network Timeout 399

uname 400

uuencode 401

xargs 402

yes 405

Summary 406

Part III: Recipes For Systems Administration

Chapter 15: Shell Features 409

Recipe 15-1: Installing Init Scripts 409

Technologies Used 410

Concepts 410

Potential Pitfalls 410

Structure 410

Recipe 412

Invocation 414

Summary 414

Recipe 15-2: RPM Report 414

Technologies Used 415

Concepts 415

Potential Pitfalls 415

Structure 415

Recipe 417

Invocation 419

Summary 420

Recipe 15-3: Postinstall Scripts 421

Technologies Used 421

Concepts 421

Potential Pitfalls 422

Structure 422

Recipe 423

Invocation 425

Summary 426

Chapter 16: Systems Administration 427

Recipe 16-1: init Scripts 427

Technologies Used 428

Concepts 428

Potential Pitfalls 429

Structure 430

Recipe 431

Invocation 432

Summary 433

Recipe 16-2: CGI Scripts 433

Technologies Used 433

Concepts 434

Potential Pitfalls 434

Structure 435

Recipe 438

Invocation 441

Summary 445

Recipe 16-3: Configuration Files 445

Technologies Used 445

Concepts 445

Potential Pitfalls 446

Structure 446

Recipe 446

Invocation 447

Summary 448

Recipe 16-4: Locks 448

Technologies Used 448

Concepts 448

Potential Pitfalls 449

Structure 450

Recipe 453

Invocation 455

Summary 458

Chapter 17: Presentation 459

Recipe 17-1: Space Game 459

Technologies Used 459

Concepts 460

Potential Pitfalls 462

Structure 462

Recipe 464

Invocation 469

Summary 470

Chapter 18: Data Storage and Retrieval 471

Recipe 18-1: Parsing HTML 471

Technologies Used 471

Concepts 472

Potential Pitfalls 472

Structure 472

Recipe 473

Invocation 474

Summary 476

Recipe 18-2: CSV Formatting 476

Technologies Used 476

Concepts 476

Potential Pitfalls 477

Structure 477

Recipe 478

Invocation 480

Summary 481

Chapter 19: Numbers 483

Recipe 19-1: The Fibonacci Sequence 483

Technologies Used 483

Concepts 484

Potential Pitfalls 484

Structure for Method 1 485

Recipe for Method 1 486

Invocation of Method 1 486

Structure for Method 2 487

Recipes for Method 2 488

Invocations of Method 2 489

Structure for Method 3 490

Recipe for Method 3 490

Invocation of Method 3 491

Summary 492

Recipe 19-2: PXE Booting 492

Technologies Used 492

Concepts 493

Potential Pitfalls 493

Structure 493

Recipe 494

Invocation 497

Summary 499

Chapter 20: Processes 501

Recipe 20-1: Process Control 501

Technologies Used 501

Concepts 502

Potential Pitfalls 503

Structure 503

Recipe 506

Invocation 511

Summary 516

Chapter 21: Internationalization 517

Recipe 21-1: Internationalization 517

Technologies Used 518

Concepts 518

Potential Pitfalls 519

Structure 520

Recipe 521

Invocation 525

Summary 526

Part IV: Reference

Appendix: Further Reading 529

Shell Tutorials and Documentation 529

Arrays 530

Tools 530

Unix Flavors 531

Shell Services 531

Glossary 533

Index 539