Parallel Programming Survey
CS 441 Lecture, Dr. Lawlor
Pop quiz--so parallel computing is finally going mainstream. What will this event be most like?
- The new 2006 Democratic majority in the US Congress. (New names, lots of hype, but zero measureable effect.)
- The worst 1950's rumors about communist spies infiltrating honest
American institutions. (Reliable components slowly going bad
without anybody knowing... until it's too late!)
- The worst rumors about global warming coming true. (Plagues of locusts, drought, heat, and major lifestyle changes for everybody.)
- A meteorite similar to that which drove the dinosaurs extinct. (Slow suffocating starvation for 90%+ of earth's life.)
- The second coming. (Dead shall rise, seas shall boil, skies
shall burn, seven seals/seven bowls, battle of Armageddon, new
Jerusalem on a new Earth.)
Answer: all of the above!
- Whatever happens, there's still going to be C++-lookin' code, if
for no other reason than the huge quantity of the C++ that's out there
already! The names may change, and the way the code is called may change a lot, but deep down tons of sequential "variable=value;" code will not change. It's the code that calls your deep-down application code that's likely to change a lot.
- Multiple things happening at once means you have to worry about
"race conditions", where two (or more) pieces of code "race" to see
which one finishes first. You have to worry about three
(nondeterministic!) possibilities: A finishes first, B finishes first,
or a tie. Maybe two of these cases are really rare. But
either one might cause your code to blow up! (Crash, fail,
silently give the wrong answer to a crucial query...)
- Parallel programming is a new lifestyle. You've got to
worry about things you didn't have to worry about before.
Concurrency makes some formerly-moral stuff (like static variables)
immoral, and can make some formerly immoral stuff (like public members)
moral, because the situation has changed.
- Many perfectly innocent serial programs will slowly wither and
die, because they don't take advantage of parallelism. They will
have the embarrasingly limited, dated feeling of command-line DOS
applications (8 character limit),FAT32 partitions (4GB file limit), or software-rendered 3D games (ugly and slow). Few users will mourn their passing.
- Parallel programming really has the potential to change
everything. Just the hardware ranges from staid stuff like SSE to
multicore to distributed to GPU to FPGA. FPGA's don't even run
C++ code, dangit. It's the end of the world! (As we know it... and I feel fine.)
To get into the proper revolutionary mindset, read:
The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software
written by Herb Sutter, smart Microsoft guy on the C++ standards committe
Notable quotes:
- "Andy giveth, and Bill
taketh away." That is, Andrew Grove, Intel, keeps building faster
and faster CPUs. Bill Gates, Microsoft, keeps building slower and
slower (yet better and better!) software.
- "Concurrency is the next
major revolution in how we write software." That is, by 2015
purely sequential code will have the same retro feeling that COBOL has
today.
- "Probably the greatest cost of
concurrency is that concurrency really is hard: The programming model,
meaning the model in the programmer’s head that he needs to reason
reliably about his program, is much harder than it is for sequential
control flow." That is, human reasoning is sequential. Execution is parallel. Horrible things can crawl into our universe through this gap!
- "For example, Intel is
talking
about someday producing 100-core chips; a single-threaded application
can exploit at most 1/100 of such a chip’s potential throughput."
90% of commercial applications today are single-threaded. They
will either adapt or die out.
- "just because it takes one
woman nine months to produce a baby doesn’t imply that nine women could
produce one baby in one month." So if the problem is "make one
baby", parallelism is useless. But if you change the problem to
"make 1 million babies", suddenly you have million-way parallelism.
- "A few rare classes of
applications are naturally parallelizable, but most aren’t." In
other words, occasionally it's obvious how to run in parallel, and
actually running in parallel is easy. Usually, though, you have
to rethink and reformulate the basic algorithm used to solve the problem!