Is Parallel Programming Hard, And, If So, What Can You Do About It?

1

Introduction

1.1 Historic Parallel Programming Difficulties . . . . . . . . . . . ...

1

Introduction

1.1 Historic Parallel Programming Difficulties . . . . . . . . . . . . . . . . .

1.2 Parallel Programming Goals . . . . . . . . . . . . . . . . . . . . . . . .

1.2.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.2.2 Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.2.3 Generality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.3 Alternatives to Parallel Programming . . . . . . . . . . . . . . . . . . . .

1.3.1 Multiple Instances of a Sequential Application . . . . . . . . . .

1.3.2 Make Use of Existing Parallel Software . . . . . . . . . . . . . .

1.3.3 Performance Optimization . . . . . . . . . . . . . . . . . . . . .

1.4 What Makes Parallel Programming Hard? . . . . . . . . . . . . . . . . .

1.4.1 Work Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . .

1.4.2 Parallel Access Control . . . . . . . . . . . . . . . . . . . . . . .

1.4.3 Resource Partitioning and Replication . . . . . . . . . . . . . . .

1.4.4 Interacting With Hardware . . . . . . . . . . . . . . . . . . . . .

1.4.5 Composite Capabilities . . . . . . . . . . . . . . . . . . . . . . .

1.4.6 How Do Languages and Environments Assist With These Tasks? .

1.5 Guide to This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.5.1 Quick Quizzes . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.5.2 Sample Source Code . . . . . . . . . . . . . . . . . . . . . . . .

2 Hardware and its Habits

2.1 Overview . . . . . . . . . . . . . . .

2.1.1 Pipelined CPUs . . . . . . . .

2.1.2 Memory References . . . . .

2.1.3 Atomic Operations . . . . . .

2.1.4 Memory Barriers . . . . . . .

2.1.5 Cache Misses . . . . . . . . .

2.1.6 I/O Operations . . . . . . . .

2.2 Overheads . . . . . . . . . . . . . . .

2.2.1 Hardware System Architecture

2.2.2 Costs of Operations . . . . . .

2.3 Hardware Free Lunch? . . . . . . . .

2.3.1 3D Integration . . . . . . . .

2.3.2 Novel Materials and Processes

2.3.3 Special-Purpose Accelerators

2.3.4 Existing Parallel Software . .

2.4 Software Design Implications . . . . .

...