File Structures

Michael J. Folk manages the Scientific Data Technologies Group at the National Center for Supercomputing Applications at the University of Illinois in Urbana. He has been responsible for developing a general purpose scientific data file format called HDF and software for managing data in high-performance, high-volume computing environments. Prior to his work at Illinois, Dr. Folk was a professor of computer science for fifteen years at Oklahoma State and Drake Universities. Bill Zoellick is currently a partner in and founder of Fastwater LLP, a consultancy focusing on helping companies build effective web businesses. He frequently writes about the issues addressed in Web Engagement and speaks on them at user conferences such as Seybold and Internet World and at various user associations and seminars. He has been a software developer, business owner, executive in a $100 million software company, and, most recently, a management consultant and business analyst. Greg Riccardi is a professor of computer science at Florida State University and an associate of the Supercomputer Computations Research Institute. Professor Riccardiis research interests include scientific databases, object-oriented databases, and parallel computation. He is also affiliated with the Thomas Jefferson National Accelerator Facility where he works on the acquisition, management, and analysis of data for experimental physics. He received a University Teaching Award in 1997 from Florida State University. 0201874016AB05172001

INTRODUCTION TO FILE STRUCTURES.


The Heart of File Structure Design.



A Short History of File Structure Design.



A Conceptual Toolkit: File Structure Literacy.



Object-Oriented Programming in C++.



FUNDAMENTAL FILE PROCESSING OPERATIONS.


Physical Files and Logical Files.



Opening Files.



Closing Files.



Reading and Writing.



Seeking.



Special Characters in Files.



The UNIX Directory Structure.



Physical and Logical Files in UNIX.



File-related Header Files.



UNIX File System Commands.



SECONDARY STORAGE AND SYSTEM SOFTWARE.


Disks.



Magnetic Tape.



Disk versus Tape.



Storage as a Hierarchy.



A Journey of a Byte.



Buffer Management.



I/O in UNIX.



FUNDAMENTAL FILE STRUCTURE CONCEPTS.


Field and Record Organization.



Using Classes to Manipulate Buffers.



Using Inheritance for Record Buffer Classes.



Managing Fixed Length, Fixed Field Buffers.



An Object-Oriented Class for Record Files.



MANAGING FILES OF RECORDS.


Record Access.



More about Record Structures.



Encapsulating Record Operations in a Single Class.



File Access and File Organization.



Object-Oriented Approach to File Access.



Portability and Standardization.



ORGANIZING FILES FOR PERFORMANCE.


Data Compression.



Reclaiming Space in Files.



Finding Things Quickly: An Introduction to Internal Sorting and Binary Searching.



Keysorting.



INDEXING.


What Is an Index?



A Simple Index for Entry-Sequenced File.



Template Classes in C++.



Object-Oriented support for Indexed, Entry-Sequenced Files of Data Objects.



Indexes That Are Too Large to Hold in Memory.



Indexing to Provide Access by Multiple Keys.



Retrieval Using Combinations of Secondary Keys.



Improving the Secondary Index Structure: Inverted Lists.



Selective Indexes.



Binding.



COSEQUENTIAL PROCESSING AND THE SORTING OF LARGE FILES.


A Model for Implementing Cosequential Processes.



Application of the Model to a General Ledger Program.



Extension of the Model to Include Multiway Merging.



A Second Look at Sorting in Memory.



Merging as a Way of Sorting Large Files on Disk.



Sorting Files on Tape.



Sort-Merge Packages.



Sorting and Cosequential Processing in UNIX.



MULTI-LEVEL INDEXING AND B-TREES.


Introduction: The Invention of the B-Tree.



Statement of the Problem.



Binary Search Trees are not a Solution.



Multi-level Indexing, A Better Approach to Tree Indexes.



B-Trees: Working up from the Bottom.



Example of Creating a B-Tree.



An Object-Oriented Representation of B-Trees.



B-Tree MEthods Search, Insert, and Others.



B-Tree Nomenclature.



Formal Definition of B-Tree Properties.



Worst-case Search Depth.



Deletion, Merging, and Redistribution.



Redistribution during Insertion: A Way to Improve Storage Utilization.



B* Trees.



Buffering of Pages: Virtual B-Trees.



Variable-length Records and Keys.



INDEXED SEQUENTIAL FILE ACCESS AND PREFIX B+ TREES.


Indexed Sequential Access.



Maintaining a Sequence Set.



Adding a Simple Index to the Sequence Set.



The Content of the Index: Separators Instead of Keys.



The Simple Prefix B+ Tree.



Simple Prefix B+ Tree Maintenance.



Index Set Block Size.



Internal Structure of Index Set Blocks: A Variable-order B-Tree.



Loading a Simple Prefix B+ Tree.



B+ Trees.



B-Trees, B+ Trees, and Simple Prefix B+ Trees in Perspective.



HASHING.


Introduction.



A Simple Hashing Algorithm.



Hashing Functions and Record Distributions.



How Much Extra Memory Should Be Used?



Collision Resolution by Progressive Overflow.



Storing More Than One Record per Address: Buckets.



Making Deletions.



Other Collision Resolution Techniques.



Patterns of Record Access.



EXTENDIBLE HASHING.


Introduction.



How Extendible Hashing Works.



Implementation.



Deletion.



Extendible Hashing Performance.



Alternative Approaches.



APPENDIX A. FILE STRUCTURES ON CD-ROM.


Using this Appendix.



Introduction to CD-ROM.



Physical Organization of CD-ROM.



CD-ROM Strengths and Weaknesses.



Tree Structures on CD-ROM.



Hashed Files on CD-ROM.



The CD-ROM File System.



APPENDIX B. ASCII TABLE.


APPENDIX C. SIMPLE FILE INPUT/OUTPUT EXAMPLES.


List.c. C program to read and display the contents of a file.



List.cpp. C++ program to read and display the contents of a file.



Person.h. Definition for class Person, including code for constructor.



Writestr.cpp. Write Person objects into a stream file Readdel.cpp. Read Person objects with fields delimited by '|'.



Readvar.cpp. Read variable length records and break up into Person objects.



Writeper.cpp. Function to write a person to a text file.



Readper.cpp. Function to prompt user and read fields of a Person.



APPENDIX D. CLASSES FOR BUFFER MANIPULATION.


Person.h. Definition for class Person.



Person.cpp. Code for class Person.



Deltext.h. Definition for class DelimitedTextBuffer.



Deltext.cpp. Code for class DelimitedTextBuffer.



Lentext.h. Definition for class LengthTextBuffer.



Lentext.cpp. Code for class LengthTextBuffer.



Fixtext.h. Definition for class FixedTextBuffer.



Fixtext.cpp. Code for class FixedTextBuffer.



Test.cpp. Test program for all buffer classes.



APPENDIX E. A CLASS HIERARCHY FOR BUFFER INPUT/OUTPUT.


Person.h. Definition for class Person.



Person.cpp. Code for class Person.



Iobuffer.h. Definition for class IOBuffer.



Iobuffer.cpp. Code for class IOBuffer.



Varlen.h. Definition for class VariableLengthBuffer.



Varlen.cpp. Code for class VariableLengthBuffer.



Delim.h. Definition for class DelimFieldBuffer.



Delim.cpp. Code for class DelimFieldBuffer.



Length.h. Definition for class LengthFieldBuffer.



Length.cpp. Code for class LengthFieldBuffer.



Fixlen.h. Definition for class FixedLengthBuffer.



Fixlen.cpp. Code for class FixedLengthBuffer.



Fixfld.h. Definition for class FixedFieldBuffer.



Fixfld.cpp. Code for class FixedFieldBuffer.



Buffile.h. Definition for class BufferFile.



Buffile.cpp. Code for class BufferFile.



Recfile.h. Template class RecordFile.



Test.cpp. Test program for buffer classes and RecordFile including template function.



APPENDIX F. SIMPLE INDEXING AND TEMPLATE CLASSES.


APPENDIX G. MULTI-LEVEL INDEXING: B+ TREE CLASSES.


APPENDIX H. CLASSES TO SUPPORT HASHING. 0201874016T04062001