Pro Oracle Database 11g RAC on Linux (eBook)

Julian Dyke is an independent consultant specializing in Oracle Database technology. He has over 20 years of database experience, including more than 15 years as an Oracle database administrator, developer, and consultant. He is chair of the U.K. Oracle User Group Real Application Clusters Special Interest Group (UKOUG RAC SIG) and a member of the OakTable Network. He regularly presents at conferences, seminars, and usergroup meetings in the U.K., Europe, and the U.S. He also maintains www.juliandyke.com, which specializes in Oracle diagnostics, optimization, and internals. He is an Oracle Certified Professional and holds a bachelor of science degree in computation from the University of Manchester Institute of Science and Technology (UMIST), U.K.

Title page 1
Copyright page 2
Contents at a Glance 3
Table of contents 4
About the Author 21
About the Technical Reviewer 22
Acknowledgments 23
CHAPTER 1 Introduction 24
Introducing Oracle Real Application Clusters 24
Examining the RAC Architecture 26
Deploying RAC 27
Maintaining High Availability 28
Providing Fault Tolerance by Redundancy 28
Making Failover Seamless 28
Putting the Technology Stack in Perspective 29
Defining Scalability 29
Scoping Various Levels of Scalability 29
Scaling Vertically vs. Horizontally 30
Increasing Manageability 31
Achieving Manageability Through Consolidation 31
Enabling Database Consolidation 32
Consolidating Servers 33
Assessing the Cost of Ownership 33
Choosing RAC vs. SMP 33
Evaluating Service-Level Agreements 34
Improving Database Management 34
Factoring in the Additional Hardware Cost 34
Assessing the Staff and Training Cost 36
Clustering with Oracle on Linux 36
Running Linux on Oracle 39
Understanding the Role of Unix 39
Liberating Software 40
Developing Linux 41
Expanding the Concept of Free with Open Source 42
Combining Oracle, Open Source, and Linux 43
Drilling Down on Unbreakable Linux 44
Creating and Growing Red Hat Enterprise Linux 45
Extending Red Hat with Oracle Enterprise Linux 46
Drilling Down on SuSE Linux Enterprise Server 47
Taking Linux to Asia 48
Summary 48
CHAPTER 2 RAC Concepts 49
Clustering Concepts 49
Configuring Active/active Clusters 49
Implementing Active/passive Clusters 50
Configuring a Shared-All Architecture 50
Configuring a Shared-Nothing Architecture 51
Exploring the Main RAC Concepts 51
Working with Cluster Nodes 51
Leveraging the Interconnect 52
Using an Ethernet-based Interconnect 52
Implementing an Infiniband-based Interconnect 53
Clusterware/Grid Infrastructure 53
Planning a RAC Installation 55
Choosing a Process Structure 56
Configuring Network Components 59
Setting up Shared Grid Infrastructure Components 60
Implementing the Oracle Cluster Registry and Oracle Local Registry 60
Configuring Voting Disks 61
Leveraging Automatic Storage Management 61
Working with ASM Disks 63
Exploiting ASM Disk Groups 65
Configuring Failure Groups 65
Weighing Your ASM Installation and Administration Options 65
Installing Real Application Clusters 66
Sorting Through Your Installation Options 66
Choosing Between a Single Instance and a RAC Database 67
Working with Database Files 67
Leveraging Online Redo Logs and Archiving 68
Managing Undo Tablespaces 69
Weighing Storage Options for RAC Databases 69
Drilling Down on a RAC Database Instance 70
Storing Information in Control Files 67
Using the Global Resource Directory (GRD) 71
Maintaining Cache Coherence with Global Cache Services (GCS) 72
Managing Global Enqueues with Global Enqueue Services (GES) 72
Transferring Data Between Instances with Cache Fusion 73
Achieving Read Consistency 74
Synchronizing System Change Numbers 74
Exploring the New Features of 11g Release 2 74
Leveraging Grid Plug and Play 75
Modeling Resources with Server Pools 77
Ensuring POSIX Compliance with ACFS 78
Using Oracle Restart Instead of RAC 79
Simplifying Clusterd Database Access with SCAN Listener 81
Summary 82
CHAPTER 3 RAC Architecture 84
Availability Considerations 84
Deciding the Number of Nodes 86
Two-Node RAC Clusters 87
Oracle RAC One Node 87
Multi-Node RAC Systems 88
Online Maintenance and Patching 88
Patching Real Application Clusters Using opatch 90
Opatch Operation Modes 92
All Node Patch Mode 92
Rolling Patch Mode 92
Minimum Downtime Patch Mode 93
Local Patch Mode 93
Instance Recovery in RAC 93
Enqueue Remastering 94
Database Recovery 94
Failover Considerations 95
Transparent Application Failover 96
Failover Modes 96
Failover Methods 97
Fast Connection Failover and Fast Application Notification 97
Scalability Considerations 98
Scalability Enhancers 99
Scalability Inhibitors 100
Standby Databases 102
Introduction to Oracle Standby Databases 103
Types of Standby Database 104
Physical Standby Database 105
Snapshot Standby Database 105
Logical Standby Database 105
Transient Logical Standby Database 106
Active Data Guard 106
Role Transitions 107
Data Protection Modes 109
The Data Guard Broker 109
Extended Distance Clusters 111
Oracle Streams 112
Streams Processing 113
Oracle Streams Prerequisites 114
Cluster Topologies 115
Summary 116
CHAPTER 4 Hardware 117
Oracle Availability 118
Server Processor Architecture 119
x86 Processor Fundamentals 119
x86-64 122
Multicore Processors and Hyper-Threading 123
CPU Cache 126
CPU Power Management 128
Virtualization 130
Memory 131
Virtual Memory 131
Physical Memory 132
NUMA 135
Memory Reliability 144
Additional Platform Features 144
Onboard RAID Storage 145
Machine Check Architectures 145
Remote Server Management and IPMI 146
Network Interconnect Technologies 146
Server I/O 147
PCI 147
PCI-X 148
PCI-Express 148
Private Interconnect 150
Standard Ethernet Interconnects 150
Fully Redundant Ethernet Interconnects 151
Infiniband 154
Private Interconnect Selection Summary 155
Storage Technologies 155
RAC I/O Characteristics 156
Read Activity 156
Write Activity 157
Asynchronous I/O and Direct I/O 160
Hard Disk and Solid State Disk Drive Performance 162
RAID 166
RAID 0 Striping 167
RAID 1 Mirroring 168
RAID 10 Striped Mirrors 168
RAID 0+1 Mirrored Stripes 169
RAID 5 170
Storage Cache 171
RAID Summary 172
Storage Protocols for Linux 172
SCSI 173
Fibre Channel and FCoE 176
iSCSI 177
SATA 178
Using Block-Based Storage 179
Linux I/O Scheduling 179
NFS and NAS 181
Evaluating Storage Performance 181
Summary 183
CHAPTER 5 Virtualization 185
Virtualization Definition and Benefits 185
Oracle VM 188
Oracle VM Server Architecture 188
Paravirtualization 189
Full Virtualization 192
Oracle VM Design 194
Oracle VM Server Installation 198
Oracle VM Manager Installation 203
Oracle VM CLI Installation and Configuration 206
Configuring Oracle VM 207
Network Configuration 207
Server Pool Configuration 212
Enabling High Availability 215
Configuring Shared Storage 215
Cluster Configuration 217
Installing and Configuring Guests 228
Importing a Template 229
Creating a Guest from a Template 230
Accessing a Guest 232
Configuring a Guest for RAC 234
Managing Domains 236
Oracle VM Agent 236
Oracle VM Manager 238
Oracle VM Manager CLI 240
The xm Command-Line Interface 242
Displaying Information 242
Managing Domains 245
Managing Resources 247
Summary 250
CHAPTER 6 Linux Installationand Configuration 251
Selecting the Right Linux Software 251
Reviewing the Hardware Requirements 252
Drilling Down on Networking Requirements 253
Configuring a GNS or a Manual IP 253
Configuring DNS and DHCP 256
Downloading the Linux Software 263
Preparing for a Network Install 263
Installing Oracle Enterprise Linux 5 267
Starting the Installation 267
Installation Media Check 267
Anaconda Installation 267
Install or Upgrade 268
Disk Partitioning 268
Creating a Default Partitioning Scheme 268
Creating a Partition Manually 271
Creating an MBR Partition 271
Creating an EFI Partition 272
Customized Partitioning 272
Configuring Swap Space 273
Configuring RAID 274
Configuring the Boot Loader and Network 279
Selecting a Time Zone 280
Configuring the Root Password 281
Reviewing the Package Installation Defaults 281
Selecting a Package Group 281
Installing Packages 283
Setting the Final Configuration 283
Accepting the License Agreement 283
Configuring the Firewall 283
Configuring SELinux 283
Enabling kdump 284
Setting the Date and Time 284
Creating a User 285
Installing Additional CDs 285
Configuring Oracle Enterprise Linux 5 285
Configuring a Server with the Oracle Validated RPM 286
Configuring YUM 286
Running the Oracle Validated RPM 287
Using the up2date Command 289
Verifying the Oracle Validated RPM Actions 290
Creating the Oracle User and Groups 290
Configuring Kernel Parameters 292
Working with Shared Memory 293
Using Semaphores 295
Setting Network Parameters 296
Message Queues 297
Setting the Number of Open Files 297
Configuring Asynchronous I/O 299
Using Magic SysRq Keys 299
Setting PAM Limits 300
Setting Kernel Boot Parameters 301
Setting Kernel Module Parameters 302
Post Oracle Validated RPM Configuration 302
Setting the Huge Pages Kernel Parameter 302
I/O Fencing and the Hangcheck-Timer Kernel Module 304
Configuring the oracle user 305
Creating the Oracle Software Directories 305
Setting Environment Variables 306
Putting Environment Variables to Work 311
Completing the Linux Configuration for RAC 312
Configuring Hostnames and Name Resolution 312
Using NTP 313
Configuring Secure Shell 315
Configuring Shared Storage 318
Discovering and Configuring SAN Disk 319
Partitioning Disks 320
I/O Multipathing with Device-Mapper 324
Preparing the Partitions for ASM with ASMLIB 328
Preparing the Partitions for ASM with udev 329
Network Channel Bonding 333
I/O Fencing with IPMI 337
Summary 342
CHAPTER 7 Grid Infrastructure Installation 343
Getting Ready for Installation 343
Obtain Software Distribution 343
Configure X Environment 344
Checking Prerequisites 344
Starting an X environment in a console window 344
Starting an X environment using VNC 345
Determining Configuration Type 347
Advanced Installation - Manual Configuration 347
Network Configuration 348
DNS Configuration 349
Choosing an Installation Option 350
Selecting an Advanced or Typical Installation Type 352
Selecting a Language 353
Configuring the Grid Plug and Play 354
Configuring the Cluster Node Information Page 356
Configuring the Network Interface Usage Page 357
Configuring the Storage Option Information Page 358
Creating an ASM Disk Group 360
Specifying an ASM Password 361
Specifying a Username and Password for IPMI 362
Configuring Privileged Operating System Groups 362
Setting the Installation Location 364
Specify the Central Inventory’s Location 365
Performing Prerequisite Checks 365
Identifying Typical Errors 366
Getting More Detail 367
Fixup Scripts 368
Anatomy of Fixup Scripts 369
Addressing Failed Checks 370
Reviewing the Summary Page 371
Setup Page 372
Reviewing Execute Configuration Scripts 372
Execution Order 373
Running the orainstRoot.sh Script 374
Executing the root.sh script 374
Monitoring Configuration Assistants 379
Implementing an Advanced Installation for Automatic Configuration 380
Configuring a Network Configuration 380
Configuring DNS 382
Configuring DHCP 383
Setting up the Grid Plug and Play Information Page 384
Configuring the Cluster Node Information Page 385
The Summary Page 386
Typical Installation 387
Choosing the Installation Type 387
Specifying the Cluster Configuration Page 388
Install Locations Page 389
Reviewing the Summary Page for a Typical Installation 390
Installing a Standalone Server 391
Selecting an Installation Option 392
Creating an ASM Disk Group Page 393
Reviewing the Summary Page for a Standalone Installation 393
Configuring the Execute Configuration Scripts 395
Deinstalling the Grid Infrastructure Software 396
Summary 397
CHAPTER 8 Clusterware 398
Introducing Clusterware 398
Examining the Hardware and Software Requirements 399
Using Shared Storage with Oracle Clusterware 400
Storing Cluster Information with the Oracle Cluster Registry 400
Storing Information in the Oracle Local Registry 401
Fencing with the Voting Disk 401
Recording Information with the Grid Plug and Play Profile 402
Using Background Processes 403
Grid Infrastructure Software Stacks 403
Drilling Down on the High Availability Stack 404
Drilling Down on the Cluster Ready Services Stack 405
Using Grid Infrastructure Agents 407
Initiating the Startup Sequence 408
Managing Oracle Clusterware 410
Using the Enterprise Manager 411
Using the Clusterware Control Utility 411
Managing Resources with srvctl 414
Verifying the Cluster with the CVU 415
Configuring Network Interfaces with oifcfg 419
Administering the OCR and OLR with ocrconfig 419
Checking the State of the OCR and its Mirrors with ocrcheck 419
Dumping Contents of the OCR with ocrdump 419
Defining Server-Side Callouts 420
Protecting Applications with Clusterware 422
Managing Resource Profiles 422
Configuring Active/Passive Clustering for Oracle Database 423
Configuring Active/Passive Clustering for Apache Tomcat 428
Using Oracle Restart 432
Troubleshooting 434
Resolving Startup Issues 434
Failing to Start OHAS 435
Failing to Start Agents Created by OHAS 437
Failing to Start the Cluster Synchronization Services Daemon 438
Failing to Start the Cluster Ready Services Daemon 439
Failing to Start the GPnP Daemon 440
Agents spawned by CRSD 440
Resolving Problems with Java Utilities 441
Patching Grid Infrastructure 441
Adding and Deleting Nodes 445
Adding Nodes 445
Checking the Prerequisites 446
Executing the addNode.sh Script 446
Finishing the Node Addition 448
Adding the RDBMS Software 450
Deleting Nodes 451
Removing the Clustered RDBMS Home 452
Removing the Node from the Cluster 454
Finishing the Node Removal 456
Exploring More Advanced Topics 456
Selecting non-Default Listener Ports 457
Selecting a non-Default SCAN Listener Endpoint 460
Changing the SCAN After Installation 461
Maintaining Voting Disks 462
Restoring Voting Disks 463
Moving Voting Disks into ASM 464
Maintaining Local and Cluster Registry 466
Dealing With a Corrupt or Inaccessible OLR 466
Dealing with a Corrupt or Inaccessible OCR 468
Moving the OCR into ASM 470
Summary 471
CHAPTER 9 Automatic Storage Management 473
Introducing ASM 473
ASM Terminology 474
Supported File Types 475
ASM Management 476
ASM and RDBMS Support 476
ASM Installation 477
ASM Components and Concepts 477
ASM Instances 477
Instance Names 478
SGA 478
Background Processes 479
Parameters 481
Failure Groups 482
ASM Files 483
Naming Files 483
Aliasing Files 484
Managing Control Files 484
Understanding Extents, Allocation Units, Types, and Templates 484
Redundancy 486
Striping 486
Mirroring 487
Intelligent Data Placement 487
Access Control 488
Understanding Access Privileges 488
Preparing the ASM Disk Group 489
Changing File Ownership and Permissions 491
Maintaining ASM 493
Creating an ASM Disk Group 493
Extending an ASM Disk Group 496
Dropping Disks from an ASM Disk Group 497
Enabling Disk Discovery 498
Understanding the ASM Header 498
Installing the Grid Infrastructure 499
Re-creating the ASM Disks 500
ASM Cluster File System 500
Creating and Mounting an ACFS Using ASMCA 502
Creating and Mounting an ACFS Using the Command Line 509
Maintaining the ACFS 512
Using ACFS with Oracle Restart 514
Administering ASM 514
Using SQL*Plus to Administer ASM 515
Setting the Environment 515
Determining the User and Privilege 515
Knowing Your Options 515
Dealing with Connected Clients 516
Checking for Corruption 517
Manually Mounting a Disk Group 517
Dropping a Disk Group 517
Creating ADVM Volumes 517
ASM Administration Using SRVCTL 517
Accessing Files in ASM 518
Using Files Instead of Devices 519
Virtualization and Shared Disks 520
Summary 521
CHAPTER 10 RDBMS Installationand Configuration 522
Installing the RAC Software 522
Start the Installer 522
Configuring the Security Updates Page 523
Configuring the Installation Options Page 523
Configuring the Node Selection Page 524
Configuring the Product Language Selection Page 525
Configuring the Database Editions Page 526
Configuring the Installation Locations Page 527
Configuring the Privileged Operating Systems Group Page 528
Configuring the Prerequisites Check Page 529
Reviewing the Summary Page 529
Executing Configuration Scripts 530
Using the Database Configuration Assistant (DBCA) 531
Starting the DBCA and Choosing an Operation 531
Configuring the Welcome Page 531
Configuring the Operations Page 532
Creating a Database 533
Configuring the Database Templates Page 534
Configuring the Database Identification Page (Admin-Managed) 535
Configuring the Database Identification Page (Policy-Managed) 536
Management Options Page 538
Database Credentials Page 539
Database File Locations Page 540
Setting up the Recovery Configuration Page 541
Configuring the Database Content Page 542
Configuring the Standard Database Components Page 543
Configuring the Initializations Parameters Page 545
Configuring the Database Storage Page 549
Configuring the Creation Options Page 551
Reviewing the Summary Page 552
Configuring the Database Options 553
Deleting a Database 555
Managing Templates 556
Building Database Creation Scripts 556
Setting up Admin-Managed Database Scripts 557
Executing the ADMIN1.sh Script 558
Creating the ADMIN1.sql Script 559
Generating the init.ora File 561
Executing the CreateDB.sql Script 562
Executing the CreateDBFiles.sql Script 563
Executing the CreateDBCatalog.sql Script 564
Executing the JServer.sql Script 565
Executing the xdb_protocol.sql Script 565
Executing the ordinst.sql Script 565
Executing the intermedia.sql Script 566
Executing the emRepository.sql Script 566
Executing the apex.sql Script 566
Executing the CreateClustDBViews.sql Script 567
Executing the lockAccount.sql Script 567
Executing the postDBCreation.sql Script 567
Building Policy-Managed Database Scripts 569
Executing the POLICY_1.sh Script 570
Executing the POLICY_1.sql Script 570
Executing the init.ora File for a Policy-Managed Database 571
Deinstalling the RDBMS Software 572
Summary 574
CHAPTER 11 Workload Management 575
Introducing Services 575
Creating an Administrator Managed Database vs.Policy-Managed Database 576
Managing Services with the Database Scheduler 577
Using Services with Shared Server 579
Managing Services 580
Managing Services with SRVCTL 580
Managing Services with Enterprise Manager 585
Managing Services with DBMS_SERVICE 588
Balancing the Workload 588
Configuring Client-Side Load Balancing 589
Configuring Server-Side Load Balancing 590
Exploring the Load Advisory Framework 592
Using Transparent Application Failover 593
Configuring Transparent Application Failover 594
Using TAF with JDBC Applications 595
Implementing Fast Connection Failover 600
Implementing Fast Application Notification 600
Implementing a Fast Connection Failover Example 605
Establishing Connection Affinity 611
Reading FAN events using the FAN API 613
Using the Resource Manager 613
Caging an Instance 616
Database Resident Connection Pool 617
Summary 620
CHAPTER 12 Oracle Performance Monitoring 622
Enterprise Manager Database Control 623
The Cluster Tab 624
The Database Tab 626
The Performance Tab 626
AWR Reports 628
Interpreting the RAC Statistics of an AWR Report 632
Top 5 Timed Foreground Events 633
Global Cache Load Profile 634
Global Cache Efficiency Percentages 634
Global Cache and Enqueue Services - Workload Characteristics 634
Global Cache and Enqueue Services - Messaging Statistics 635
Cluster Interconnect 635
Foreground Wait Class 636
Wait Event Histogram 636
“SQL Statement” Sections 637
RAC-Related Segment Statistics 637
Dictionary Cache Stats (RAC) 638
Library Cache Activity (RAC) 638
Global Messaging Statistics 639
Global CR Served Statistics 639
Global Current Served Statistics 639
Global Cache Transfer Statistics 640
Interconnect Statistics 641
Dynamic Remastering Statistics 641
Active Session History 642
Automatic Database Diagnostic Monitor 644
Executing an ADDM Report 644
Controlling ADDM 644
The Report Format 646
AWR SQL Report 646
Performance Monitoring Using SQL*Plus 647
GV$ Views 648
System Statistics 648
Segment Statistics 648
Global Caches Services: Consistent and Current Reads 650
Global Cache Services: Current Block Activity 652
Global Enqueue Service 655
Library Cache 656
Dictionary Cache 657
Lock Conversions 657
Automatic Diagnostic Repository 659
Summary 667
CHAPTER 13 Linux Performance Monitoring 668
The uptime and last Commands 668
The ps Command 669
free, ipcs, pmap, and lsof 670
The free Command 670
The /proc File System 671
The /sys/devices/system/node File System 672
The ipcs Command 673
The pmap Command 673
The lsof Command 675
top 675
vmstat 677
strace 678
netstat, ss, and tcpdump 679
Looking at Interface Statistics 679
Summary Statistics 680
Listening Socket Statistics 680
Looking up Well-Known Ports 681
Reporting on Socket Statistics Using ss 681
Capturing and Displaying Network Packets 682
iostat 683
mpstat 684
sar and kSar 685
Configuring sar 685
Invoking sar Directly 686
Graphing the Results 687
Oracle Cluster Health Monitor 689
Installing the Oracle Cluster Health Monitor 689
Starting and Stopping the Oracle Cluster Health Monitor 692
Understanding the Architecture 693
Installing the Client-Side GUI 693
Viewing Current and Captured Activity 694
OSWatcher 695
Installing OSWatcher 695
Starting OSWatcher 696
Stopping OSWatcher 696
Viewing Results Graphically 697
nmon 698
Summary 700
CHAPTER 14 Parallel Execution 701
Parallel Execution Concepts 702
Serial Execution 702
Parallel Execution 703
Producers and Consumers 705
Bloom Filters 710
Partitioning 712
Parallel Execution Configuration 714
cluster_interconnects 714
db_block_size, db_cache_size, and db_file_multiblock_read_count 714
instance_groups and parallel_instance_group 715
large_pool_size, parallel_execution_message_size, andshared_pool_size 716
parallel_adaptive_multi_user 716
parallel_automatic_tuning 717
parallel_degree_limit 717
parallel_degree_policy, parallel_min_time_threshold, andparallel_servers_target 718
Automatic Degree of Parallelism 718
Statement Queuing 719
In-memory Parallel Execution 720
parallel_force_local 721
parallel_io_cap_enabled 721
parallel_max_servers, parallel_min_servers,parallel_threads_per_cpu, and processes 722
parallel_min_percent 722
pga_aggregate_target 723
Parallel Execution Performance 723
AWR Reports 723
SQL*Plus 727
Trace Files 728
Summary 729
CHAPTER 15 Upgrading to Oracle 11g Release 2 730
Upgrading Grid Infrastructure Components 730
Installing the Prerequisites 731
Running the Installer 732
Specifying Options 733
Running the Upgrade 738
Upgrading RAC Software 742
Running the Installer 743
Running Configuration Scripts 745
Preparing for the Database Upgrade 747
Identifying the Upgrade Path 747
Determine Upgrade Method 748
Testing the Upgrade Process 748
Running the pre-Upgrade Tool 749
Performing Other Checks 754
Saving Current Database Parameters 754
Backing up the Database 755
Configuring the Listener Process 756
Upgrading Automatically with DBUA 756
Upgrading a Database Manually 765
Preparing the Parameter Files 767
Preparing Password Files 768
Modifying the Initialization Parameters 768
Restarting the Database in UPGRADE Mode 768
Running the Catalog Upgrade Script 768
Configuring SPFILE 769
Running the post-Upgrade Status Tool 770
Running post-Upgrade Scripts 771
Recompiling Invalid Packages 773
Updating /etc/oratab 775
Updating Environment Variables 775
Updating the Oracle Cluster Registry 775
Setting the Initialization Parameters for the New Release 776
Performing the Necessary post-Upgrade Steps 777
Completing Mandatory post-Upgrade Tasks 777
Performing the Recommended Tasks 778
Resolving Problems in Mixed-Database Environments 780
Using a Swing Kit 781
Summary 782
Index 783