Students used the powerwall during the February 2007 MCM to view state-scale census data and develop a fully automated congressional redistricting algorithm.
MPIglut is our
small library for efficiently running sequential OpenGL applications
in parallel with MPI across the powerwall.
Xdmx is what we're using for a distributed
multiheaded X server. This splits up X requests from the head node,
and displays the results across all the nodes of the powerwall. Just
"export DISPLAY=:1", and you're using the 9000x4650 resolution DMX
server!
Home and scratch directories are mounted via NFS.
OpenSSH is the primary means to communicate within the cluster.
MPICH is what we use for MPI.
Blockbuster is a nice DMX-aware movie display program.
fvwm is our window manager.
We'd use a more modern manager, but most that we tried (along with all
GNOME applications) seem to crash under Xdmx.
The OS is 32-bit Ubuntu Linux 6.0.6 (Dapper). For fast 3D, we're using the proprietary nVidia
Linux GLX driver, which works great. Eventually we plan to move
to 64-bit Linux, although we've had enough software compatability
headaches as it is!
Hardware
Ten workstations. Seven are Dell Precision 390s with Intel Core2 Duo E6300 CPUs;
The remaining three are older Dell Precision 380s with Pentium D CPUs.
One gigE switch. A Dell PowerConnect 2724 24-port
gigabit ethernet switch. We've benchmarked the network at
110MB/s--that's 1 gigabyte in 9 seconds!
Twenty
monitors. Dell UltraSharp 2007FPW 20.1-inch DVI Flat Panel LCDs.
These run at 1680x1050, and each workstation drives two of them in TwinView mode.
The monitors are held onto the frame with 3/8" bolts, which (with
a suitably flat washer) just barely fit behind the monitor's metal VESA
bracket:
Three 20-amp 120VAC wall circuits. The monitors just
barely fit on one circuit; the other two circuits feed five
workstations each. We blew the breakers several times working out
how to divide up the load!
One homemade wooden frame to stack all the hardware in. It's
basically two vertical 4x4 uprights holding up four rows of monitors
bolted to horizontal 2x6 stringers, with a 2x12 frame around the
outside. It's heavy enough to be totally immobile. Gloss black paint
makes it look nifty!
We set up each machine by pulling its hard disk and copying its entire
configured Linux installation from an existing machine (using fdisk, mkfs.ext3, tar, and grub). This takes
about ten minutes per machine. We set up the entire cluster in less than half a day!
Ubuntu wouldn't recognize the BCM5751 motherboard network that ships with the Dell Precision 390; so we're using the Broadcom tg3 gigabit ethernet driver.
We were considering using synergy to broadcast the mouse and keyboard, but DMX nicely takes care of both input and display.
To work with DMX, your X servers have to be accessible on the
network. Edit /etc/X11/xinit/xserverrc to remove "-nolisten tcp", add
each cluster name to /etc/X1.hosts, and set "allowed_users" to
"anybody" in /etc/X11/Xwrapper.config. You may also need to poke a
hole in your firewall (we used firestarter) to let through traffic on
port 6000 from other powerwall machines.
If you'd like an account on the powerwall for a research project, just send an email to Dr. Lawlor!
