Through a strange series of backlinks I found a fresh blog about a Debian system set up completely without a GUI — console only, and some of starting points for making it work.
Before you pooh-pooh it as yet another wacky Pentium II fan devoted to making a 200Mhz machine “usable,” the test hardware is a Pentium 4 — 2Ghz, with commensurate hardware. I don’t see mention of a video card, so I’m assuming that board will run Intel graphics, if it matches this one.
If you’ve considered going X-less as an experiment, or if you are that wacky Pentium II fan, you’ll probably want to watch the progress. If it goes in the direction I expect, it’s going to be an incredibly fast, incredibly useful end product. (Added note: Ubuntu fans would do well to watch that site too, since much of how Debian behaves is the same in Ubuntu.)