Comments on: Compiling effortlessly … sort of

By: JP Senior

JP Senior — Tue, 08 Jun 2010 18:13:17 +0000

The completely fair scheduler is awesome; I do heavy visualization labs at home on qemu and virtualbox. Often my CPU is pegged at 100% on all four cores. The new scheduler is incredible when it comes to ‘user’ components with something just as simple as my mouse. IBM has a great report on CFQ at http://www.ibm.com/developerworks/linux/library/l-completely-fair-scheduler/ I’ll admit

With regards to compiling, most makefiles I’ve come across in the wild don’t attempt to use your neat-o multicore or hyperthreaded systems at all. Try make -j3 to run 3 jobs on a 4 processor system – huge difference. I usually leave one “cpu” or “job” available so I don’t end up pinning each of my CPUs, allowing me to do normal work while I wait for the compile to finish.

Here’s a comparison of compile times for pulseaudio 0.9.19.
w/o -j3

sudo time make
…snip…
95.67user 42.58system 2:20.17elapsed 98%CPU (0avgtext+0avgdata 222960maxresident)k
62192inputs+70096outputs (28major+13364466minor)pagefaults 0swaps

w -j4
sudo time make -j4
…snip…
96.60user 41.82system 0:38.79elapsed 356%CPU (0avgtext+0avgdata 222528maxresident)k
40inputs+66872outputs (0major+13299085minor)pagefaults 0swaps

2:20 vs 0:38! Incredible!

By: Michele Amato

Michele Amato — Thu, 03 Jun 2010 14:19:01 +0000

Thank you very much!!!

By: K.Mandla

K.Mandla — Thu, 03 Jun 2010 13:25:56 +0000

Not a problem.

http://pastebin.com/unqEFN3t

By: cthulhu

cthulhu — Thu, 03 Jun 2010 13:10:37 +0000

This is what it says on the distcc web-page:

“distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed. They can even have different processors or operating systems, if cross-compilers are installed.”

Not sure exactly what it means, but it sounds good.

By: Michele Amato

Michele Amato — Thu, 03 Jun 2010 12:00:09 +0000

Hi! Can you share your dot-config file for kernel 2.6.34, please? Thanks in advance and scuse me for my very bad english…

By: Luca

Luca — Thu, 03 Jun 2010 11:33:19 +0000

You can however it isn’t exactly effortless. You need to ensure the machines in your compile farm all have the same toolchain (note: architecture) and libraries installed. Yanking the drive is definitely the easier of the two.

By: cthulhu

cthulhu — Thu, 03 Jun 2010 07:48:48 +0000

I’m no wizard at networks and stuff, but it was my understanding that you could use distcc to distribute the compiling process on the other machines, thereby making it a lot faster.

By: K.Mandla

K.Mandla — Thu, 03 Jun 2010 03:02:50 +0000

I’m betting that’s the answer then. Before now I rarely did any compiling on this old machine because it seemed too taxing. Now, it seems to be able of doing both without suffering so much. I am likewise thankful for that improvement.

By: Reacocard

Reacocard — Thu, 03 Jun 2010 02:55:01 +0000

There was a lot of work done on the CPU scheduler over that interval – 2.6.32 in particular (http://x264dev.multimedia.cx/?p=185). The increase in responsiveness is indeed quite noticeable, even on my overpowered 2.5ghz core 2 duo. Software that gets faster as it ages is one of my favorite things about open-source.