Service Availability – Principles and Practice Cloth

Maria Toeroe, Ericsson, Canada Maria Toeroe has been in the telecommunications industry for over 10 years conducting research and standardisation in the area of dependable software, service availability and fault tolerance utilizing the expertise she gained in her prior work in the academia related to formal description techniques and their use in protocol engineering. Having represented Ericsson in the Service Availability Forum since 2005, Maria has led several standardization efforts in different chair positions of the SA Forum Technical Work Group.? Within Ericsson she is also responsible for university collaborations related to her fields. She has been an Adjunct Associate Professor with the Department of Electrical and Computer Engineering of Concordia University since 2007. Maria holds a Diploma degree from the Institute of Engineering Economics Kharkov, a University Doctorate and a PhD from the Faculty of Electrical Engineering and Informatics of the Budapest University of Technology and Economics. Francis Tam, Nokia, Finland Francis Tam has conducted research, development and standardisation in the dependable software area of service availability and fault tolerance in the telecommunications industry for more than 16 years. He has served as industrial adviser to three European Commission funded projects on dependable computer systems and frequently gives lectures on highly available systems to the Master of Engineering students at INSA Lyon in France. Francis holds a BSc in Computer Science from Queen Mary College (University of London), an MSc in Computing Science from North Staffordshire Polytechnic and a Doctor of Science (Technology) in Software Systems from Tampere University of Technology in Finland.

List of Contributors xiii Foreword xv Preface xix Acknowledgments xxv List of Abbreviations xxvii Part I INTRODUCTION TO SERVICE AVAILABILITY 1 Definitions, Concepts, and Principles 3 Francis Tam 1.1 Introduction 3 1.2 Why Service Availability? 4 1.2.1 Dossier on Unavailability of Service 4 1.2.2 Issues and Challenges 5 1.3 Service Availability Fundamentals 6 1.3.1 System, Behavior, and Service 6 1.3.2 Dependable Computing Concepts 8 1.3.3 The Meaning of Availability 10 1.4 Achieving Service Availability 13 1.4.1 Following the Framework of Fault Tolerance 13 1.4.2 Redundancy is a Requisite 14 1.4.3 Dealing with Failures 16 1.4.4 Upgrade Matters 19 1.5 Conclusion 20 2 The Birth of the Service Availability Forum 23 Francis Tam 2.1 Introduction 23 2.2 Technology Environment 23 2.3 Business Environment 24 2.3.1 Ecosystem 25 2.3.2 COTS and Open Systems 26 2.4 The Service Availability Forum Era 27 2.5 Concluding Remarks 28 Part II THE SA FORUM SYSTEM: SERVICES AND FRAMEWORKS 3 Overview of the Service Availability Architecture 33 Dave Penkler 3.1 Introduction 33 3.1.1 Background and Business Context 33 3.1.2 Goals and Requirements 34 3.1.3 Service Availability Architecture Scope and Presentation 36 3.2 HA Concepts Applied 39 3.2.1 To Be or Not to Be High Availability Aware 39 3.2.2 HA Aware Application Perspective 42 3.3 Architecture 43 3.3.1 Basic Architectural Model 43 3.3.2 The AIS Services and Frameworks Architecture 47 3.3.3 Service Dependencies 58 3.4 Open Issues 59 3.4.1 The Optional Features Issue 60 3.4.2 Integrated AIS Service API 60 3.4.3 Common Low Level Communication Facility Interface 60 3.4.4 Common Distributed Process Management Interface 61 3.4.5 System Trace Service 61 3.4.6 Diagnostics Framework 61 3.4.7 Overload Control Framework 61 3.5 Conclusion 62 4 The SA Forum Information Model: The Heart of Control and Monitoring 63 Maria Toeroe 4.1 Introduction 63 4.2 Background 64 4.2.1 Management Models Out There 64 4.2.2 The SA Forum Needs 65 4.3 The SA Forum Information Model 67 4.3.1 Overview of the SA Forum Solution 67 4.3.2 Administrative and Management Aspects 80 4.3.3 Application Information Models 81 4.3.4 Open Issues and Recommendations 81 4.4 Conclusion 83 5 Consistent and High Level Platform View 85 Maria Toeroe 5.1 Introduction 85 5.2 Hardware Platform Interface 86 5.2.1 Background 86 5.2.2 Overview of the Hardware Platform Interface 87 5.2.3 The HPI Model 88 5.2.4 HPI Capability Discovery 93 5.2.5 Error Handling and Administrative Operations 94 5.2.6 Open Issues and Conclusions 95 5.3 Platform Management Service 96 5.3.1 The Conception of PLM 96 5.3.2 Overview of the SA Forum Platform Management 97 5.3.3 The PLM Information Model 98 5.3.4 Tracking of PLM Entities 107 5.3.5 Administrative and Management Aspects 110 5.3.6 Service Interaction 118 5.3.7 Open Issues and Conclusions 120 5.4 Cluster Membership Service 121 5.4.1 Background 121 5.4.2 Overview of the Cluster Membership Service 122 5.4.3 CLM Configuration: The Bootstrap Trap 125 5.4.4 Are You a Member? 126 5.4.5 Administrative and Management Aspects 127 5.4.6 Service Interaction 129 5.4.7 Open Issues 130 5.4.8 Recommendation 131 5.5 Conclusion 131 6 Model Based Availability Management: The Availability Management Framework 133 Maria Toeroe 6.1 Introduction 133 6.2 Background 134 6.2.1 Error Detection and Repair 134 6.2.2 Fault Zones and Error Escalation 135 6.2.3 Separation of Services from Serving Entities 136 6.2.4 Service Provisioning Roles 136 6.2.5 Delicacies of Service State Replication 137 6.3 The Availability Management Framework 138 6.3.1 Overview of the SA Forum Solution 138 6.3.2 Components and Component Service Instances 139 6.3.3 The AMF Information Model 148 6.3.4 Redundancy Models 167 6.3.5 The AMF Administrative Interface 176 6.3.6 Interactions Between AMF and Other AIS Services 187 6.3.7 Open Issues 190 6.3.8 Recommendation 191 6.4 Conclusion 191 7 Communication and Synchronization Utilities 193 Maria Toeroe and Sayandeb Saha 7.1 Introduction 193 7.2 Event Service 194 7.2.1 Background: Event Service Issues, Controversies, and Problems 194 7.2.2 Overview of the SA Forum Event Service 195 7.2.3 Event Service Architecture and Model 196 7.2.4 User Perspective 200 7.2.5 Administrative and Management Aspects 201 7.2.6 Service Interactions 201 7.2.7 Open Issues and Recommendations 202 7.3 Message Service 202 7.3.1 Need for Reliability and Load Distribution 202 7.3.2 Overview of the SA Forum Message Service 203 7.3.3 Message Service Architecture and Model 205 7.3.4 User Perspective 207 7.3.5 Administrative and Management Aspects 210 7.3.6 Service Interaction 210 7.3.7 Open Issues and Recommendations 211 7.4 Checkpoint Service 212 7.4.1 Background: Why Checkpoints 212 7.4.2 Overview of the SA Forum Checkpoint Service 213 7.4.3 Checkpoint Service Model 215 7.4.4 User Perspective 217 7.4.5 Administrative and Management Aspects 220 7.4.6 Service Interaction 221 7.4.7 Open Issues 222 7.4.8 Recommendation 222 7.5 Conclusion 223 7.5.1 Common Issue: Entity Names 223 7.5.2 Conclusion 223 8 Services Needed for System Management 227 Maria Toeroe 8.1 Introduction 227 8.2 Log Service 228 8.2.1 Background: Data, Data, and More Data 228 8.2.2 Overview of the SA Forum Solution 229 8.2.3 The LOG Information Model 231 8.2.4 User Perspective 232 8.2.5 Administrative and Management Aspects 233 8.2.6 Service Interaction 233 8.2.7 Open Issues and Recommendations 235 8.3 Notification Service 236 8.3.1 Background: Issues, Controversies, and Problems 236 8.3.2 Overview of the SA Forum Notification Service 237 8.3.3 User Perspective 239 8.3.4 Correlation of Notifications 241 8.3.5 Administrative and Management Aspects 243 8.3.6 Service Interaction 244 8.3.7 Open Issues and Recommendation 246 8.4 Information Model Management Service 247 8.4.1 Background: Issues, Controversies, and Problems 247 8.4.2 Overview of the SA Forum IMM Solution 249 8.4.3 The Object Manager API 251 8.4.4 The Object Implementer API 255 8.4.5 IMM XML File 258 8.4.6 Administrative and Management Aspects 258 8.4.7 Service Interaction 258 8.4.8 Open Issues 260 8.4.9 Recommendation 261 8.5 Conclusion 262 9 Model-Based Software Management: The Software Management Framework 265 Maria Toeroe 9.1 Introduction 265 9.2 Background 266 9.3 Software Management a la Carte 268 9.3.1 Overview of the SA Forum Solution 268 9.3.2 Entity Types File: Is It Eaten or Drunk by SMF? 271 9.3.3 The Upgrade Campaign and Its Specification 273 9.3.4 Upgrade Campaign Execution Status and Failure Handling 279 9.3.5 Administrative and Management Aspects 285 9.3.6 User Perspective 288 9.3.7 Service Interaction 289 9.3.8 Open Issues 291 9.3.9 Recommendation 292 9.4 Conclusion 294 10 Combining the Services 297 Maria Toeroe 10.1 Introduction 297 10.2 Application Design and Development 297 10.3 Application Platform Design 299 10.4 Operation and Maintenance 301 Part III SA FORUM MIDDLEWARE IN ACTION 11 SA Forum Programming Model and API Conventions 305 Francis Tam 11.1 Introduction 305 11.2 Programming Model 306 11.2.1 AIS Area Service Interfaces 306 11.2.2 Real-Time Support 306 11.2.3 Naming Conventions and Type Definitions 308 11.2.4 Usage Model and Library Life Cycle 309 11.2.5 Tracking 311 11.3 Making Sense of the API Specifications 312 11.3.1 Structure of Service API Specification 314 11.3.2 Administration API 315 11.4 Practical Topics 316 11.4.1 Interacting with POSIX 316 11.4.2 Allocating and Freeing Memory 319 11.4.3 Handling Pointers 319 11.4.4 Finding Out Implementation Limits 320 11.4.5 When an Area Service is Unavailable 321 11.4.6 Backward Compatibility 322 11.5 Concluding Remarks 322 12 SA Forum Java Mappings: Specifications, Usage, and Experience 325 Robert Hyerle and Jens Jensen 12.1 Introduction 325 12.2 Background 325 12.2.1 Early Exploration of Java Mappings in Hewlett Packard 325 12.2.2 Java in Ericsson 326 12.2.3 The SA Forum Java Mapping Initiative 327 12.3 Understanding the Java Mappings 328 12.3.1 Java Application Integration Architecture 328 12.3.2 Naming 329 12.3.3 Package Structure 330 12.3.4 The Underlying Objects 330 12.3.5 Types 331 12.3.6 Parameters, Exceptions, and Method Signatures 332 12.3.7 Factories, Callbacks, and Life-cycles 333 12.3.8 Callbacks and the Selection Object in Java 334 12.4 Using the Java Mappings 335 12.4.1 Integrating AIS Services with Java Applications 335 12.4.2 Integrating AIS Services with Containerized Java Applications 342 12.4.3 AIS Services in Mixed Language and Mixed Implementation Environments 343 12.5 Going Further 343 12.5.1 The Java Mapping Roadmap 343 12.5.2 Related Java Standards and Other References 344 13 SA Forum Middleware Implementations 347 Mario Angelic and Ulrich Kleber 13.1 Introduction 347 13.1.1 OpenHPI 347 13.1.2 OpenSAF 348 13.2 The OpenHPI Project 348 13.2.1 Overview of the OpenHPI Solution 348 13.2.2 User Perspective 351 13.2.3 OpenHPI Tools 353 13.2.4 Open Issues and Recommendations 354 13.3 The OpenSAF Project 355 13.3.1 Background 355 13.3.2 OpenSAF Architecture 356 13.3.3 SA Forum Compliant Services 360 13.3.4 OpenSAF Infrastructure Services 364 13.3.5 Managing OpenSAF 365 13.3.6 Deploying OpenSAF 367 13.4 Conclusion 368 14 Integration of the VideoLAN Client with OpenSAF: An Example 371 Anik Mishra and Ali Kanso 14.1 Introduction 371 14.2 Going Under the Hood: The VLC Workflow 372 14.3 Integrating VLC with OpenSAF 373 14.3.1 Nonproxied-Non-SA-Aware Integration 374 14.3.2 SA-Aware VLC Integration 379 14.3.3 SA-Aware VLC with Service Continuity 384 14.4 Summary and Conclusion 387 15 Migration Paths for Legacy Applications 391 Mario Angelic 15.1 Introduction 391 15.2 Reasons for Migration 392 15.2.1 Benefits for System Owners 392 15.2.2 Benefits for ISVs 392 15.3 Integration Criteria 393 15.3.1 Main Factors 393 15.3.2 Easy Management 394 15.3.3 Streamlined Architecture 396 15.3.4 Code Quality 397 15.3.5 Integration Levels 397 15.4 How to Migrate 399 15.4.1 Availability Integration 399 15.4.2 Manageability Integration 409 15.5 Open Issues 413 15.6 Conclusion 413 16 Overcoming Complexity: Formal Modeling Techniques at the Rescue 415 Maria Toeroe and Ferhat Khendek 16.1 Introduction 415 16.2 Background 416 16.2.1 The Model-Based Approach 416 16.2.2 Starting Points in the Specifications 417 16.3 Model-Based Software Management 419 16.3.1 Configuration Model 419 16.3.2 Configuration Generation 420 16.3.3 Upgrade Campaign Generation 424 16.3.4 Analytical Models and How They Can Help 427 16.4 Conclusion 428 17 Conclusion 431 17.1 Summary 431 17.2 The Future 433 References 435 Index 443