Agile Data Warehousing Project Management (eBook)

Front Cover 1
Agile Data Warehousing Project Management 4
Copyright Page 5
Contents 6
List of Figures 14
List of Tables 16
Preface 18
Answering the skeptics 19
Intended audience 19
Parts and chapters of the book 20
Invitation to join the agile warehousing community 21
Author’s Bio 22
1: An Introduction to Iterative Development 24
1 What Is Agile Data Warehousing? 26
A quick peek at an agile method 27
The “disappointment cycle” of many traditional projects 31
The waterfall method was, in fact, a mistake 35
Agile’s iterative and incremental delivery alternative 37
Agile as an answer to waterfall’s problems 38
Increments of small scope 38
Business centric 38
Colocation 39
Self-organized teams 39
Just in time 39
80-20 Specifications 39
Fail fast and fix quickly 40
Integrated quality assurance 41
Agile methods provide better results 41
Agile for data warehousing 42
Data warehousing entails a “breadth of complexity” 42
Adapted scrum handles the breadth of data warehousing well 43
Managing data warehousing’s “depth of complexity” 45
Guide to this book and other materials 49
Simplified treatment of data architecture for book 1 51
Companion web site 52
Where to be cautious with agile data warehousing 53
Summary 54
2 Iterative Development in a Nutshell 56
Starter concepts 57
Three nested cycles 58
The release cycle 59
Development and daily cycles 62
Shippable code and the definition of done 63
Time-boxed development 64
Caves and commons 65
Product owners and scrum masters 65
Product owner 66
Scrum master 67
Developers as “generalizing specialists” 67
Improved role for the project manager 68
Might a project manager serve as a scrum master? 69
User stories and backlogs 70
Estimating user stories in story points 71
Iteration phase 1: story conferences 73
Iteration phase 2: task planning 75
Basis of estimate cards to escape repeating hard thinking 75
Task planning doublechecks story planning 77
Iteration phase 3: development phase 78
Self-organization 79
Daily scrums 80
Accelerated programming 82
Test-driven development 85
Architectural compliance and “tech debt” 86
Iteration phase 4: user demo 88
Iteration phase 5: sprint retrospectives 90
Retrospectives are vital 93
Close collaboration is essential 95
Selecting the optimal iteration length 96
Nonstandard sprints 97
Sprint 0 98
Architectural sprints 98
Implementation sprints 99
“Spikes” 99
“Hardening” sprints 99
Where did scrum come from? 100
Distant history 100
Scrum emerges 101
Summary 102
3 Streamlining Project Management 104
Highly transparent task boards 105
Task boards amplify project quality 107
Task boards naturally integrate team efforts 108
Scrum masters must monitor the task board 109
Burndown charts reveal the team aggregate progress 110
Detecting trouble with burndown charts 112
Developers are not the burndown chart’s victims 114
Calculating velocity from burndown charts 115
Common variations on burndown charts 117
Setting capacity when the team delivers early 117
Managing tech debt 118
Managing miditeration scope creep 119
Diagnosing problems with burndown chart patterns 120
An early hill to climb 121
Shallow glide paths 122
Persistent inflation 123
Should you extend a sprint if running late? 125
Extending iterations is generally a bad idea 125
Two instances where a changing time box might help 126
Should teams track actual hours during a sprint? 127
Eliminating hour estimation altogether 128
Managing geographically distributed teams 129
Consider whether fully capable subteams are possible 131
Visualize the problem in terms of communication 131
Choose geographical divisions to minimize the challenge 132
Invest in a solid esprit de corp 132
Provide repeated booster shots of colocation for individuals 133
Invest in high-quality telepresence equipment 133
Provide agile team groupware 135
Summary 135
2: Defining Data Warehousing Projects for Iterative Development 138
4 Authoring Better User Stories 140
Traditional requirements gathering and its discontents 141
Big, careful requirements not a solution 143
A step in the right direction 143
Agile’s idea of “user stories” 145
Advantages of user stories 146
Identifying rather than documenting the requirements 147
User story definition fundamentals 148
Quick test for actionable user stories 149
How small is small? 150
Epics, themes, and stories 151
Common techniques for writing good user stories 153
Keep story writing simple 155
Use stories to manage uncertainty 156
Reverse story components 157
Focus on understanding “who” 157
Focus on understanding “what” 158
Focus on understanding “why” 160
Be wary of the remaining w’s 162
Add acceptance criteria to the story-writing conversations 163
Summary 164
5 Deriving Initial Project Backlogs 166
Value of the initial backlog 167
Sketch of the sample project 168
Fitting initial backlog work into a release cycle 169
The handoff between enterprise and project architects 171
Key observations 175
User role modeling results 177
Key persona definitions 178
Carla in corp strategy 178
Franklin in finance 179
An example of an initial backlog interview 180
Framing the project 185
Finance is upstream 187
Finance categorizes source data 188
Customer segmentation 188
Consolidated product hierarchies 189
Sales channel 189
Unit reporting 190
Geographies 191
Product usage 191
Observations regarding initial backlog sessions 193
Sometimes a lengthy process 193
Detecting backlog components 194
Managing user story components on the backlog 196
Prioritizing stories 196
Summary 197
6 Developer Stories for Data Integration 198
Why developer stories are needed 199
Introducing the “developer story” 201
Format of the developer story 202
Developer stories in the agile requirements management scheme 203
Agile purists do not like developer stories 204
Initial developer story workshops 205
Developers workshop within software engineering cycles 207
Data warehousing/business intelligence reference data architecture 208
Forming backlogs with developer stories 210
Evaluating good developer stories: DILBERT’S test 213
Demonstrable 213
Independent 215
Layered 215
Business valued 215
Estimable 217
Refinable 217
Testable 218
Small 218
Secondary techniques when developer stories are still too large 218
Decomposition by rows 219
Decomposition by column sets 221
Decomposition by column type 223
Decomposition by tables 224
Theoretical advantages of “small” 226
Summary 228
7 Estimating and Segmenting Projects 230
Failure of traditional estimation techniques 231
Traditional estimating strategies 232
Why waterfall teams underestimate 234
A single-pass effort 234
Insufficient feedback 234
Overoptimism rewarded 235
Few reality checks 235
Criteria for a better estimating approach 236
An agile estimation approach 238
Estimating within the iteration 238
Estimating the overall project 241
Quick story points via “estimation poker” 242
Story points and ideal time 246
Story points defined 247
Ideal time defined 247
The advantage of story points 248
Estimation accuracy as an indicator of team performance 250
Value pointing user stories 251
Packaging stories into iterations and project plans 252
Criteria for better story prioritization 254
Segmenting projects into business-valued releases 255
The data architectural process supporting project segmentation 256
Artifacts employed for project segmentation 257
Business target model 257
Dimensional model 257
Star schema 258
Tiered integration model 258
Categorized service model 258
Project segmentation technique 1: dividing the star schema 261
Project segmentation technique 2: dividing the tiered integration model 263
Project segmentation technique 3: grouping waypoints on the categorized services model 266
Embracing rework when it pays 269
Summary 270
3: Adapting Iterative Development for Data Warehousing Projects 272
8 Adapting Agile for Data Warehousing 274
The context as development begins 275
Data warehousing/business intelligence-specific team roles 278
Project architect 279
Data architect 285
Systems analyst 287
Systems tester 288
The leadership subteam 289
Resident and visiting “resources” 290
New agile characteristics required 291
Avoiding data churn within sprints 292
Pipeline delivery for a sustainable pace 296
New meaning for Iteration 0 and Iteration?1 299
Pipeline requires two-step user demos 301
Keeping pipelines from delaying defect correction 302
Resolving pipelining’s task board issues 303
Pipelining as a buffer-based process 306
Pipelining is controversial 307
Continuous and automated integration testing 308
High quality is a necessity 310
Agile warehousing testing requirements 311
Nominal data testing 312
Incoherent data 313
Missing data 313
Dirty data 314
Multiple time points 314
The need for automation 315
Requirements for a warehouse test engine 316
Automated testing for front-end applications 317
Evolutionary target schemas—the hard way 320
Summary 325
9 Starting and Scaling Agile Data Warehousing 326
Starting a scrum team 326
Stage 1: time box and story points 328
Stage 2: pipelined delivery 329
Stage 3: developer stories and current estimates 329
Stage 4: managed development data and test-driven development 330
Stage 5: automatic and continuous integration testing 330
Stage 6: pull-based collaboration 332
Scaling agile 332
Application complexity 333
Geographical distribution 334
Team size 334
Compliance requirements 334
Information technology governance 335
Organizational culture 335
Organizational distribution 336
Coordinating multiple scrum teams 337
Coordinating through scrum of scrums 338
Matching milestones 341
Balancing work between teams with earned-value reporting 342
What is agile data warehousing? 348
Communicating success 351
Handoff quality 352
Quality of estimates 353
Defects by iteration 353
Burn-up charts 354
Cross-method comparison projects 356
Cycle times and story point distribution 357
Moving to pull-driven systems 358
A glimpse at a pull-based approach 358
Kanban advantages 363
A more cautious view 364
1 How will a pull-based system respond when stories range too widely in size? 364
2 Can we really define workable units without keeping our estimating skills sharp? 364
3 Should we categorize our work and adapt the task board in great detail? 365
4 Is the underlying process stable enough to have only one optimal set of WIP limits? 365
5 When will we have enough data points to identify a dependable SLA? 365
6 Aren’t there other reasons for having iterations besides estimating? 365
Stages of scrumban 366
Summary 367
References 368
Index 376