Oracle Database Problem Solving and Troubleshooting Handbook

Tariq Farooq is an Oracle technologist, architect, and problem-solver and has been working with various Oracle Technologies for more than twenty-four years in very complex environments at some of the world’s largest organizations. Having presented at almost every major Oracle conference/event all over the world, Tariq is an award-winning speaker, community leader/organizer, author, forum contributor, and tech blogger. He is the founding president of the IOUG Virtualization & Cloud Computing Special Interest Group and the BrainSurface social network for the various Oracle communities. Tariq founded, organized, and chaired various Oracle conferences, including, among others, the OTN Middle East and North Africa (MENA) Tour, VirtaThon—the largest online-only conference for the various Oracle domains, the CloudaThon & RACaThon series of conferences, and the first-ever Oracle-centric conference at the Massachusetts Institute of Technology (MIT) in 2011. He was the founder and anchor/show host of the VirtaThon Internet Radio series program. Tariq is an Oracle RAC Certified Expert and holds a total of 14 professional Oracle certifications. Having authored more than one-hundred articles, whitepapers, and other publications, Tariq is the coauthor of the Expert Oracle RAC 12c (Apress, 2013), Oracle Exadata Expert’s Handbook (Addison-Wesley, 2015), and Building Database Clouds in Oracle 12c (Addison-Wesley, 2016) Oracle books. Tariq has been awarded the Oracle ACE and ACE Director awards. Mike Ault began working with computers in 1980—following a six-year Navy enlistment in the Nuclear Navy riding submarines—programming in Basic and Fortran IV on the PDP-11 architecture in the nuclear industry. During Mike’s nuclear years he worked with PDP, IBM-PC, Osborne, and later VAX-VMS and HP architectures, as well as with the Informix and Ingres databases. Following the downturn in the nuclear industry, Mike began working with Oracle as the only DBA at the Luka, Mississippi–based Advanced Solid Rocket Motor (ASRM) project for NASA in 1990. Since 1990, Mike has worked with a variety of industries using Oracle both in-house and as a consulting talent. Mike got extensive Flash experience as the Oracle Guru for Texas Memory Systems. Mike transitioned to IBM as the Oracle Guru for the STG Flash group when IBM purchased TMS in 2012. Mike has published over two dozen Oracle-related books, including the 7.0, 8.0, 8i and 9i versions of his Oracle Administration and Management with Wiley, the “Oracle8 Black Book” and “Oracle DBA OCP Exam Cram” series (for versions 8 and 8i) with Coriolis, and multiple titles including Oracle9i RAC and Oracle10g Grid & Real ApplicationClusters with Rampant Technical Press. Mike has written articles for Oracle, Select, DBMS, Oracle Internals, and several other database-related magazines. Mike is also a highly sought-after keynote speaker and expert instructor for local, regional, and international Oracle conferences, such as GOUSERS, SEOUG, RMOUG, NYOUG, NCOUG, IOUG, OOW, ODTUG, UKOUG, and EOUG. Paulo Portugal has more than fifteen years of IT experience as an Oracle DBA. He is an Oracle Certified Master 11g; an Oracle Certified Professional (9i, 10g, 11g, and 12c); an Oracle RAC 10g and 11g Certified Specialist; an Oracle DBA 10g Certified Linux Administrator; Oracle Exadata Implementation Certified; IBM DB2 Certified (8 and 9 “Viper”); an Oracle GoldenGate 10 Certified Implementation Specialist; an Oracle Enterprise Manager Certified Implementation Specialist; and an Oracle 11i Applications Database Administrator Certified Professional. Paulo is the author of the Rampant “Advanced DBMS Packages” and many articles in blogs and some magazines and websites. Paulo has maintained a regular presence on the Oracle conference and speaking circuit: Oracle Open World—San Francisco (2005, 2006, 2011, and 2013); IBM Information on Demand—Los Angeles (2006); Burleson Oracle RAC Cruise (2009); and Oracle Training in Reading—United Kingdom (2011). Currently, Paulo works as an Oracle sales consultant for Oracle Brazil. Paulo has participated in the Oracle Beta Test 11i project using Data Guard and is a specialist in high Availability tools such as Oracle Data Guard, Oracle Streams, Oracle GoldenGate, and Oracle RAC. Mohamed Houri has a Ph.D. in fluid mechanics (scientific computing) from the University of Aix–Marseille II, preceded by an engineer diploma in aeronautics. He has been working around the Oracle database for more than fourteen years for different European customers as an independent Oracle consultant specializing in tuning and troubleshooting Oracle performance problems. Mohamed has also worked with the Naval Architect Society of Japan on the analysis of tsunamis and breaking waves using a powerful signal analysis called Wavelet Transform. He maintains an Oracle blog and is active in the Oracle Worldwide forum and in the French equivalent. He tweets about Oracle topics at @MohamedHouri. Syed Jaffar Hussain is an Oracle Database expert with more than twenty years of IT experience. For more than a year, he has been involved with several local and large-scale international banks where he implements and manages highly complex cluster and Exadata environments with hundreds of business-critical databases. Oracle awarded him the prestigious Best DBA of the Year and Oracle ACE Director status in 2011. He also acquired industry-best Oracle credentials, Oracle Certified Master (OCM), Oracle RAC Expert, OCP DBA 8i, 9i, 10g, and 11g, in addition to ITIL expertise. Syed is an active Oracle speaker who regularly presents technical sessions and webinars at many Oracle events. You can visit his technical blog at http://jaffardba.blogspot.com. In addition to being part of the core technical review committee for Oracle technology–oriented books, he coauthored Oracle 11g R1/R2 Real Application Clusters Essentials (Packt Publishing, 2011), Expert Oracle RAC 12c (Apress, 2013), and Oracle Exadata Expert’s Handbook (Addison-Wesley, 2015). Jim Czuprynski is an Oracle ACE Director with more than thirty-five years of experience in information technology, serving diverse roles at several Fortune 1000 companies in those three-plus decades—mainframe programmer, applications developer, business analyst, and project manager—before becoming an Oracle DBA in 2001.      In his current role as a strategic solutions consultant for OnX Enterprise Solutions, he focuses on providing his expertise to help customers understand how to best leverage Oracle technology to solve their most difficult IT challenges. As a senior Oracle University instructor, Jim has taught Oracle core technologies, Exadata, and GoldenGate to more than two-thousand Oracle DBAs since 2005. He was selected as Oracle Education Partner Instructor of the Year in 2009.      Jim’s most recent book, Oracle Database Upgrade, Migration & Transformation Tips & Techniques (McGraw-Hill Education, 2015), takes a nitty-gritty approach to tackling the best ways to migrate, transform, and upgrade Oracle databases to 12c, Exadata, and beyond. Jim continues to write a steady stream of articles that focus on the myriad facets of Oracle database administration, with more than one-hundred articles to his credit since 2003 at databasejournal.com and ioug.org. Jim’s blog, Generally . . . It Depends, contains his regular observations on all things Oracle. Jim is also a sought-after public speaker on Oracle Database technology features. Since 2008, he has presented topics at Oracle OpenWorld, IOUG’s COLLABORATE, Hotsos Symposium, Oracle Technology Network ACE Tours, and Oracle User Group conferences around the world. Guy Harrison is an executive director of research and development at Dell Software, where he oversees the development of database tools such as Toad and Shareplex. Guy is the author of six books on database technology, including Next Generation Databases (Apress, 2015), O racle Performance Survival Guide (Prentice Hall, 2010), and MySQL Stored Procedure Programming (O’Reilly, 2006). He also writes the “Big Data notes” column for Database Trends and Applications (dbta.com). Guy can be found on the Internet at www.guyharrison.net, and at @guyharrison on Twitter.

Preface xix

Acknowledgments xxi

About the Authors xxv

About the Technical Reviewers and Contributors xxix

 

Chapter 1: Troubleshooting and Tuning LOB Segment Performance 1

Introduction to the LOB Datatype 1

BASICFILE versus SECUREFILE LOBs 8

The Impact of PCTFREE on LOBs 14

Overcoming Poor INSERT Performance 17

Summary 17

 

Chapter 2:  Overcoming Undo Tablespace Corruption 19

Overview of Undo Management 19

DTP, XA, and Rollback Segments 22

Recovering from Undo Tablespace Corruption 24

Summary 32

 

Chapter 3: Handling GC Buffer Busy Wait Events 35

Overview of Buffer Busy Wait Events 35

Leveraging the ORAchk Utility 36

Isolating GC Buffer Busy Waits 40

Isolating GC Buffer Busy Wait Event Issues 45

Fixes for GC Buffer Busy Waits 49

Summary 50

 

Chapter 4: Adaptive Cursor Sharing 51

ACS Working Algorithm 52

ACS in Action 58

ACS Bind-Awareness Monitoring 61

The Bind-Aware Cursor 73

A Practical Case 76

Summary 81

 

Chapter 5: Stabilizing Query Response Time Using SQL Plan Management 83

Getting Started 83

Creating a SQL Plan Baseline 87

Faking Baselines 92

Oracle Optimizer and SPM Interaction 96

SQL Plan Baseline Reproducibility 108

NLS_SORT and SQL Plan Baseline Reproducibility 114

ALL_ROWS versus FIRST_ROWS 117

Adaptive Cursor Sharing and SPM 122

Summary 131

 

Chapter 6: DDL Optimization Tips, Techniques, and Tricks 133

DLL Optimization Concept 133

The DDL Optimization Mechanism 136

DDL Optimization for NULL Columns 147

Summary 152

 

Chapter 7: Managing, Optimizing, and Tuning VLDBs 153

Overview of Very Large Databases 153

Optimal Basic Confi guration 154

VLDB Performance Tuning Principles 162

Gathering Optimizer Statistics 166

Backup and Recovery Best Practices 170

Summary 172

 

Chapter 8: Best Practices for Backup and Recovery with Recovery Manager 173

A Perfect Backup and Recovery Plan 173

An Overview of RMAN 174

Tips for Database Backup Strategies 175

Validating RMAN Backups 186

Backup Optimization and Tuning 187

Using RMAN for RAC Databases 189

Retaining Data in a Recovery Catalog 191

Having a Robust Recovery Strategy 192

Leveraging the Data Recovery Advisor 193

Summary 194

 

Chapter 9: Database Forensics and Tuning Using AWR Analysis: Part I 197

What Is AWR? 197

Knowing What to Look For 199

Header Section 199

RAC-Specific Pages 208

Time Model Statistics 211

Operating System Statistics 212

The SQL Sections 217

Instance Activity Statistics 221

Summary 233

 

Chapter 10: Database Forensics and Tuning Using AWR Analysis: Part II 235

Tablespace I/O Statistics 235

Buffer Pool Statistics 237

PGA Statistics 240

Shared Pool Statistics 244

Other Advisories 245

Buffer Waits Statistics 247

Enqueue Statistics 248

Undo Segment Statistics 250

Latch Statistics 251

Segment Access Areas 255

Library Cache Activity Sections 257

Dynamic Memory Components Sections 260

Process Memory Sections 262

Streams Component Sections 264

Resource Limits Statistics 266

Initialization Parameter Changes 267

Global Enqueue and Other RAC Sections 268

Summary 273

 

Chapter 11: Troubleshooting Problematic Scenarios in RAC 275

Troubleshooting and Tuning RAC 276

A Well-Oiled RAC Ecosystem 279

Troubleshooting RAC with OEM 12c 282

Utilities and Commands for Troubleshooting 283

Summary 288

 

Chapter 12: Leveraging SQL Advisors to Analyze and Fix SQL Problems 289

OEM 12c—SQL Advisors Home 290

SQL Tuning Advisor 290

SQL Access Advisor 295

SQL Repair Advisor 300

SQL Performance Analyzer 301

Summary 302

 

Chapter 13: Extending Data Pump for Data and Object Migration 303

Using Data Pump 303

Working with Private and Public Objects 306

Finding Valid INCLUDE and EXCLUDE Values 309

Exporting Subsets of Data 310

Changing Object Properties 313

Using PL/SQL API with Data Pump 317

Monitoring and Altering Resources 319

Improving Performance 320

Upgrading Databases 321

Summary 322

 

Chapter 14: Strategies for Migrating Data Quickly between Databases 323

Why Bother Migrating? 324

Determining the Best Strategy 324

Considering What Data to Migrate 326

Data Migration Methods 327

Summary 352

 

Chapter 15: Diagnosing and Recovering from TEMPFILE I/O Issues 353

Overview of Temporary Tablespaces 353

Correcting TEMPFILE I/O Waits 359

Summary 365

 

Chapter 16: Dealing with Latch and Mutex Contention 367

Overview of Latch and Mutex Architecture 367

Measuring Latch and Mutex Contention 371

Latch and Mutex Scenarios 375

Intractable Latch Contention 383

Summary 385

 

Chapter 17: Using SSDs to Solve I/O Bottlenecks 387

Disk Technologies: SSD versus HDD 388

The Oracle Database Flash Cache 395

Comparing SSD Options 402

Storage Tiering 410

Flash and Exadata 414

Summary 418

 

Chapter 18: Designing and Monitoring Indexes for Optimal Performance 421

Types of Indexes 421

Multiple Indexes on Identical Columns 431

Index Performance Issues 432

Summary 442

 

Chapter 19: Using SQLT to Boost Query Performance 445

Installing SQLT 446

Using the XTRACT Method 447

Using the XECUTE Method 448

Leveraging Other SQLT Methods 451

A Real-World Example 452

Summary 453

 

Chapter 20: Dealing with XA Distributed Transaction Issues 455

Repairing Common Distributed Transaction Issues 456

Repairing Ghost Distributed Transactions 457

Monitoring Distributed Transactions 462

Summary 464

 

Index 465