Storage Design and Implementation in vSphere 6

Mostafa Khalil is a senior staff engineer at VMware. He is a senior member of the VMware Engineering team and has worked for VMware for more than 18 years. Prior to joining VMware, he worked at Lotus/IBM. A native of Egypt, Mostafa graduated from the Al-Azhar University’s School of Medicine and practiced medicine in Cairo. He became intrigued by the mini computer system used in his medical practice and began to educate himself about computing and networking technologies. After moving to the United States, Mostafa continued to focus on computing and acquired several professional certifications, including VCDX (3, 4, 5, and 6), VCAP (4, 5, and 6)-DCD, VCAP4-DCA, VCP (2, 3, 4, 5, and 6), MCSE, Master CNE, HP ASE, IBM CSE, and Lotus CLP. As storage became a central element in the virtualization environment, Mostafa became an expert in this field. He has delivered several seminars and troubleshooting workshops at various VMware public events in the United States and elsewhere around the world.

Introduction xxxiii
Part I Storage Protocols and Block Devices
1 Storage Types 1
    History of Storage 1
        Birth of the Hard Disks 3
        Along Comes SCSI 4
        PATA and SATA–SCSI’s Distant Cousins? 4
        Measuring Storage Capacity 7
        Permanent Storage Media Relevant to vSphere 6 8
    Summary 9
2 Fibre Channel Storage Connectivity 11
    SCSI Standards and Protocols 11
        SCSI-2 and SCSI-3 Standards 11
        Fibre Channel Protocol 12
        Decoding an EMC Symmetrix WWPN 23
        Locating a Target’s WWNN and WWPN Seen by ESXi 6 Hosts 24
        SAN Topology 27
        Fabric Switches 31
        FC Zoning 33
        Designing Storage with No Single Points of Failure 37
    Summary 43
3 FCoE Storage Connectivity 45
    FCoE (Fibre Channel over Ethernet) 45
    FCoE Initialization Protocol 47
    FCoE Initiators 49
        Hardware FCoE Adapters 49
        Software FCoE Adapters 50
    Overcoming Ethernet Limitations 51
        Flow Control in FCoE 51
    Protocols Required for FCoE 52
        Priority-Based Flow Control 52
        Enhanced Transmission Selection 52
        Data Center Bridging Exchange 53
        10GigE–A Large Pipeline 53
    802.1p Tag 54
    Hardware FCoE Adapters 55
    Confi guring HW FCoE Adapters 56
    Implementing SW FCoE in ESXi 6 61
    Confi guring SW FCoE Network Connections 63
    Enabling a Software FCoE Adapter 67
    Removing or Disabling a Software FCoE Adapter 71
        Using the UI to Remove a Software FCoE Adapter 71
        Using the CLI to Remove a Software FCoE Adapter 72
    Troubleshooting FCoE 72
        ESXCLI 72
        FCoE-Related Logs 75
        How to Verify SW FCoE Connectivity 90
        Proc Nodes Can Be Useful, Too 91
    Parting Tips 93
    Summary 94
4 iSCSI Storage Connectivity 95
    iSCSI Protocol 95
        Customizing Storage Network Confi guration 151
        Troubleshooting iSCSI Connectivity 175
    Summary 177
5 vSphere Pluggable Storage Architecture 179
    Native Multipathing 179
    Storage Array Type Plug-in (SATP) 180
        Listing SATPs on an ESXi 6 Host 182
    Path Selection Plug-in (PSP) 182
        Listing PSPs on an ESXi 6 Host 184
    Third-Party Plug-ins 184
    Multipathing Plug-ins (MPPs) 185
    Anatomy of PSA Components 186
    I/O Flow Through PSA and NMP 187
        Classifi cation of Arrays Based on How They Handle I/O 187
        Paths and Path States 189
        Preferred Path Setting 189
        Flow of I/O Through NMP 191
    Listing Multipath Details 191
        Listing Paths to a LUN by Using the UI 192
        Listing Paths to a LUN by Using the CLI 194
        Identifying Path States and on Which Path the I/O Is Sent–FC 198
        Example of Listing Paths to an iSCSI-Attached Device 198
        Identifying Path States and on Which Path the I/O Is Sent–iSCSI 200
        Example of Listing Paths to an FCoE-Attached Device 200
        Identifying Path States and on Which Path the I/O Is Sent–FC 202
    Claim Rules 203
    MP Claim Rules 204
    Plug-in Registration 207
    SATP Claim Rules 208
    Modifying PSA Plug-in Confi gurations Using the UI 213
        Which PSA Confi gurations Can Be Modifi ed Using the UI? 213
    Modifying PSA Plug-ins Using the CLI 215
        Available CLI Tools and Their Options 215
        Adding a PSA Claim Rule 217
        Deleting a Claim Rule 231
        Masking Paths to a Certain LUN 233
        Unmasking a LUN 236
        Changing PSP Assignment via the CLI 238
    Summary 242
6 ALUA 243
    ALUA Defi nition 244
        ALUA Target Port Groups 244
        Asymmetric Access State 245
        ALUA Management Modes 247
        ALUA Followover 248
        Identifying Device ALUA Confi guration 252
        Troubleshooting ALUA 259
    Summary 263
7 Multipathing and Failover 265
    What Is a Path? 265
    Where Is the Active Path? 270
        Identifying the Current Path Using the CLI 270
        Identifying the I/O (Current) Path Using the UI 271
    LUN Discovery and Path Enumeration 273
    Sample LUN Discovery and Path Enumeration Log Entries 275
    Factors Affecting Multipathing 279
        Accessing Advanced Options 280
    Failover Triggers 282
        SCSI Sense Codes 282
        Path Failover Triggers 284
    Path States 288
        Factors Affecting Paths States 289
    Path Selection Plug-ins 298
        VMW_PSP_FIXED 298
        VMW_PSP_MRU 299
        VMW_PSP_RR 299
    When and How to Change the Default PSP 299
        When Should You Change the Default PSP? 299
        How to Change the Default PSP 300
    PDL and APD 302
        Unmounting a VMFS Volume 304
        Detaching a Device Whose Datastore Was Unmounted 309
    Path Ranking 313
        Path Ranking for ALUA and Non-ALUA Storage 313
        How Does Path Ranking Work for ALUA Arrays? 313
        How Does Path Ranking Work for Non-ALUA Arrays? 315
        Confi guring Ranked Paths 316
    Summary 318
8 Third-Party Multipathing I/O Plug-ins 319
    MPIO Implementations on vSphere 6 319
    EMC PowerPath/VE 6 320
        Downloading PowerPath/VE 320
        PowerPath/VE Installation Overview 322
        What Gets Installed? 323
        Installing PowerPath/VE by Using the Local CLI 324
        Verifying the PowerPath/VE Installation 326
        Listing Devices Claimed by PowerPath/VE 330
        Managing PowerPath/VE 331
        Uninstalling PowerPath/VE 331
    Hitachi Dynamic Link Manager (HDLM) 333
        Obtaining Installation Files 334
        Installing HDLM 335
        Modifying HDLM PSP Assignments 341
        Locating Certifi ed Storage on the VMware HCL 344
    Dell EqualLogic PSP Routed 346
        Downloading Documentation 346
        Downloading the Installation File and the Setup Script 347
        How Does EQL MEM Work? 347
        Installing EQL MEM on vSphere 6 347
        Uninstalling the Dell PSP EQL Routed MEM 351
    Summary 352
9 Using Heterogeneous Storage Confi gurations 353
    What Is a “Heterogeneous” Storage Environment? 353
    Heterogeneous Storage Scenarios 354
    ESXi 6 View of Heterogeneous Storage 355
        Basic Rules of Using Heterogeneous Storage 355
        Naming Conventions 356
        How Does All This Fit Together? 357
    Summary 363
10 Using VMDirectPath I/O 365
    What Is VMDirectPath? 365
    Which I/O Devices Are Supported? 366
        Locating Hosts Supporting VMDirectPath I/O on the HCL 368
    Confi guring VMDirectPath I/O on vSphere 6.0 369
        What Gets Added to the VM’s Confi guration File? 378
    Toggling an I/O Adapter as Passthrough on vSphere 6.5 378
        Adding a Passthrough PCI Device to a VM by Using vSphere Web Client 380
        Adding a Passthrough PCI Device to a VM by Using the HTML5 ESXi Host Client 382
    Practical Examples of VM Design Scenarios Using VMDirectPath I/O 384
        Hyperconverged Infrastructure 384
        Passing Through Physical Tape Devices 384
        Supported VMDirectPath I/O Devices 385
        DirectPath I/O Example 385
    Troubleshooting VMDirectPath I/O 386
        Interrupt Handling and IRQ Sharing 386
        Device Sharing 387
    Summary 388
11 Storage Virtualization Devices (SVDs) 389
    The SVD Concept 389
        How Does It Work? 389
        Constraints 392
    SVD Design Decisions 393
        Front-End Design Choices 393
        Back-end Design Choices 396
        LUN Presentation Considerations 397
        RDM Considerations 397
    Summary 400
Part II File Systems
12 VMFS Architecture 401
    History of VMFS 401
        VMFS3 on-Disk Layout 404
        VMFS5 Layout 411
        Common Causes of Partition Table Problems 418
        Re-creating a Lost Partition Table for VMFS3 Datastores 418
        Re-creating a Lost Partition Table for VMFS5 Datastores 423
        More About VOMA 427
        Re-creating the Partition Table 429
        Preparing for the Worst: Recovering from a File System Corruption 432
        Span or Grow? 439
        Upgrading to VMFS5 449
        VMFS6 455
    Summary 460
13 Virtual Disks and RDMs 461
    The Big Picture 461
    Virtual Disks 462
        Virtual Disk Types 465
        Thin-on-Thin Confi guration 467
        Virtual Disk Modes 468
    Creating Virtual Disks Using the UI 468
        Creating Virtual Disks During VM Creation 468
        Creating a Virtual Disk After VM Creation 472
    Creating Virtual Disks Using vmkfstools 475
        Creating a Zeroed Thick Virtual Disk Using vmkfstools 476
        Creating an Eager Zeroed Thick Virtual Disk Using vmkfstools 477
        Creating a Thin Virtual Disk Using vmkfstools 478
        Cloning Virtual Disks Using vmkfstools 480
    Raw Device Mappings 484
        Creating Virtual Mode RDMs Using the UI 484
        Creating Physical Mode RDMs Using the UI 489
        Creating RDMs Using the Command-Line Interface 490
    Listing RDM Properties 491
        Listing RDM Properties Using vmkfstools 494
        Listing RDM Properties Using the UI 495
    Virtual Storage Adapters 498
        Selecting the Type of Virtual Storage Adapter 498
        VMware Paravirtual SCSI Controller 500
        SCSI Bus Sharing 502
    Virtual Machine Snapshots 502
        Creating the VM’s First Snapshot While the VM Is Powered Off 503
        Creating a VM Second Snapshot While Powered On 511
    Snapshot Operations 515
        Reverting to a Snapshot 515
        Deleting a Snapshot 518
        Consolidating Snapshots 520
    Reverting to the Latest Snapshot 525
    Linked Clones 527
    Summary 528
14 Distributed Locks 529
    Basic Locking 530
        Using ATS for the Heartbeat 531
        What Happens When a Host Crashes? 534
        Optimistic Locking 534
        Dynamic Resource Allocation 535
        SAN Aware Retries 535
        Optimistic I/O 537
        Operations That Require SCSI Reservations 537
        MSCS/WSFC-Related SCSI Reservations 538
        Perennial Reservations 540
        Under the Hood of Distributed Locks 546
    Summary 554
15 Snapshot Handling 555
    What Is a Snapshot? 555
    What Is a Replica? 556
    What Is a Mirror? 556
    VMFS Signature 557
        Listing Datastores’ UUIDs via the CLI 557
    Effects of Snapshots on VMFS Signatures 558
    How to Handle VMFS Datastores on Snapshot LUNs 558
    Resignaturing Datastores 559
        Resignaturing a VMFS Datastore Using vSphere 6 Web Client 559
        Resignaturing a VMFS Datastore Using ESXCLI 562
    Force-Mounting 566
        Force-Mounting a VMFS Snapshot Using vSphere 6 Web Client 566
        Force-Mounting a VMFS Snapshot Using ESXCLI 566
    Sample Script to Force-Mount All Snapshots on Hosts in a Cluster 568
    Summary 572
16 NFS 573
    History of NFS 573
        NFS on vSphere 6 574
        Confi guring an NFS Volume on NAS Servers 584
        NFS Exports on EMC Unity 584
        Confi guring Replication on EMC Unity 600
        Creating an NFSv3 Storage VM (SVM) on a NetApp Filer 603
        Creating an NFSv3 Datastore on a NetApp Filer 606
        Using Kerberos Authentication with NFSv4.1 Datastores 611
        ESXi Kerberos 5 Components 612
    Confi guring a NetApp Filer for NFSv4.1 614
        Creating an NFSv4.1 Storage VM (SVM) on a NetApp Filer by Using the CLI 614
        Creating an NFSv4.1 Storage VM (SVM) on a NetApp Filer by Using the UI 616
        Preparing ESXi Hosts for Kerberos Authentication 632
        Confi guring ESXi Hosts for Kerberos Authentication 634
        Preparing AD for NFSv4.1 Kerberos Authentication 636
        Confi guring AD for UNIX Identity Management 637
        Creating Kerberos-to-UNIX User Name Mapping 644
        Creating an NFSv4.1 Datastore on a NetApp Filer 651
        NFS Locking Mechanism 667
        NFS Client Advanced Confi gurations 669
        Useful Authentication-Related Commands 676
        Troubleshooting NFS Connectivity 678
    Summary 682
17 VAAI 683
    What Is VAAI? 683
    VAAI Primitives 684
        Hardware Acceleration APIs 684
        Thin Provisioning APIs 684
    Full Copy Primitive (XCOPY) 685
    Block Zeroing Primitive (WRITE _ SAME) 686
    Hardware Accelerated Locking Primitive (ATS) 687
        ATS Enhancements on VMFS5 687
    Thin Provisioned APIs 688
    NAS VAAI Primitives 689
    Enabling and Disabling VAAI Primitives 690
        Disabling Block Device Primitives by Using the UI 691
        Disabling Block Device VAAI Primitives by Using the CLI 693
        Disabling the UNMAP Primitive by Using the CLI 695
        Disabling NAS VAAI Primitives 696
    VAAI Filter and VAAI Plug-in 697
        Locating Supported VAAI-Capable Block Devices 698
        Locating Supported VAAI-Capable NAS Devices 700
    Listing Registered Filter and VAAI Plug-ins 702
    Listing the Confi gurations of VAAI Filters and Plug-ins 702
    XCOPY Custom Options 707
        Confi guring XCOPY Custom Options 708
    Listing VAAI VMkernel Modules 710
    Identifying VAAI Primitives Supported by a Device 711
        Listing Block Device VAAI Support Status by Using the CLI 711
        Listing NAS Device VAAI Support Status 714
        Listing VAAI Support Status by Using the UI 714
    Displaying Block Device VAAI I/O Stats Using esxtop 716
    The VAAI T10 Standard Commands 719
    Troubleshooting VAAI Primitives 720
        Example Using the VMFS Volume Label 721
        Example Using VMFS Volume UUID 721
        Sample Log Entries Related to VAAI 721
    Summary 723
18 Storage vMotion 725
    Storage vMotion History 725
        Storage vMotion Requirements 726
        Storage vMotion Within the Same Host 726
        Storage vMotion Between Hosts’ Local Datastores 730
        How Does SvMotion Actually Work? 731
    Summary 747
19 VisorFS 749
    What Is VisorFS? 749
    RAM Disks 752
    Stateful or Stateless? 753
    Boot Device Partitions 755
    Rolling Back to a Previous Version 758
    Summary 758
20 VASA 759
    What Is VASA? 759
        VASA Architecture 759
        VASA Versions 760
        Confi guring the vSphere Environment to Use VASA 761
        Locating VASA-Certifi ed Storage Arrays on vCG 761
        Installing VASA Providers 763
    Registering VASA Providers 767
    Summary 783
21 vSAN Core Features 785
    What Is vSAN? 785
        vSAN Requirements 786
        The File System on the vSAN Datastore 787
        vSAN Network 788
        Confi guring vDS 793
        Solid State Drives and Flash Storage 807
        I/O Adapters 810
        VMware Certifi cation Guide (vCG) 811
        Looking Up vSAN Ready Nodes on the vCG 812
        Ruby vSphere Console (RVC) 815
        vSAN Software Components 824
        Creating and Confi guring a vSAN Cluster 831
        vSAN Disk Groups 839
        vSAN Storage Policies 864
        vSAN Storage Policy Placement Rules 870
        Erasure Coding (RAID 5 and RAID 6) 888
        Reconfi guring Existing Virtual Disks’ Storage Policies 897
        HA on vSAN 904
        vSAN Easy Installation 907
    Summary 912
22 vSAN Advanced Features 913
    Fault Domains 913
        Creating vSAN Fault Domains 914
    Listing Fault Domains via the CLI 915
        Component Placement on Fault Domains 921
    Stretched Clusters 924
        A vSAN Stretched Cluster Inter-data Site Network 925
        Calculating a vSAN Network’s Minimum Bandwidth Requirement 925
        vSAN Stretched Cluster Witness 926
        vSAN Stretched Cluster Witness-to-Data-Sites Network 927
        Deploying a Witness Virtual Appliance 927
        Customizing the Nested ESXi Witness VM Network 930
        Adding a Nested ESXi Witness Node to a Datacenter 931
        Verifying Witness Network Confi guration 932
        Confi guring a Stretched Cluster 933
        Using RVC to List Stretched Cluster Witness Information 941
        vSAN Stretched Cluster I/O Path 941
        vSAN Stretched Cluster Failure Modes 941
        Sample UI and cmmds-tool Output from vSAN Stretched Cluster Failures 947
        RAID 1 Examples 949
        RAID 10 Examples 952
    Comparison Between Fault Domains and Stretched Clusters 957
    Nested vSAN Fault Domains 958
        Nested FD Storage Policies 958
        Confi guring a Nested FD Policy 959
        Nested RAID 1 Confi guration 962
        Nested RAID 10 Confi guration 970
        Nested Fault Domain I/O Flow 975
    Remote Offi ce/Branch Offi ce (ROBO) vSAN Cluster 976
        Confi guring a ROBO Cluster 978
    Deduplication and Compression 979
        Deduplication and Compression Licensing 980
        Confi guring Deduplication and Compression 980
        Displaying Dedupe and Compression Stats 984
    The vSAN Health Check UI 987
        Confi guring Health Check 987
        Accessing the Health Check UI 989
        Health Check Tests 990
        Running Burn-in Tests 1002
    vSAN Performance Monitoring 1007
        Enabling vSAN Performance Service 1007
        Accessing vSAN Performance Charts in the UI 1010
        Running vSAN Performance Diagnostics 1011
        Displaying Cluster Back-End Performance Charts 1012
        Displaying Cluster-Level VM Consumption Charts 1013
        Displaying Host-Level vSAN Performance Charts 1013
        Displaying VM-Level vSAN Performance Charts 1016
    vSAN Observer 1016
        Starting vSAN Observer for Live Monitoring 1016
        Collecting an Observer Bundle 1019
        Browsing vSAN Observer HTML Bundle Stats 1021
    vSAN iSCSI Targets (VITs) 1022
        vSAN iSCSI Targets Architecture 1022
        Confi guring a VIT Network 1022
        Enabling VITs 1024
        Creating VITs 1025
        Creating Initiator Groups 1028
        VIT Confi guration Under the Hood 1030
        Monitoring VIT 1038
        Using ESCXLI to List VITs details 1046
    Storage Policy Noncompliance 1052
        Monitoring Resyncing Components via the UI 1053
        Monitoring Resyncing Components via the RVC 1054
    Resync Throttling 1055
        Resync Throttling via the UI 1055
        Resync Throttling via the CLI 1056
        Sample Object Properties During Resync 1056
    vSAN Sparse Format 1059
    Common vSAN Basic Administrative Tasks 1062
        Removing a Host from a vSAN Cluster 1062
        Rejoining a Host to a vSAN Cluster 1065
        Using cmmds _ tool to List Various Types 1067
        Listing a Disk’s Stats by Using the CLI 1069
        vSphere 6.5 vSAN Health CLI Namespace 1070
    Summary 1075
23 Virtual Volumes (VVols) 1077
    What Are VVols? 1077
        VVols Architecture 1078
        Confi guring the vSphere Environment to Use VVols 1081
        Locating VASA-Certifi ed Storage Arrays on the vCG 1082
        Installing VASA Providers 1083
    Confi guring Block Storage Arrays for VVols 1083
        Confi guring Dell Virtual Storage Manager (VSM) and Registering the VASA Provider with vCenter 1083
    Allowing ESXi Hosts’ iSCSI Initiators to Access EQL VVols Protocol Endpoints 1085
    Confi guring a NAS Server for VVols 1086
        Creating Storage Pools for Use by EMC Unity VVols SCs 1086
        Confi guring the vSphere 6 Environment for VVols 1098
        Creating Virtual Machine Storage Policies for VVols 1131
        Deploying New VMs Using Storage Policies 1135
        I/O Flow to VVols 1142
        Listing Virtual Machine Files on a VVols Datastore via the CLI 1151
        OSFS and Its Role in VVols 1158
        Storage Array’s View of VVols 1162
        Native Snapshots on VVols 1164
        Putting It All Together 1170
        VVols Interoperability with Other vSphere Features 1175
        VVols Replication 1177
    Summary 1185
9780134268101, TOC, 6/26/2017