Regarding live CDs, I meant to mention Quantian as well, but forgot to when dealing with that truncated post. It has a lot more software than ParallelKnoppix, which is focused on MPI-based clustering. In the specific case of Octave, PK has the latest version. Regarding access time, a live CD is a bit slower than a HD installation, but it's not at all a problem. A live CD booted into a virtual machine is also pretty quick, and a lot more convenient. Running PK from a VMware virtual machine, from the time I type "octave" to when I see the prompt takes about 4 seconds on my office computer. It takes about 3 seconds using my installed OS. Booting a live CD image in a virtual machine has the advantage that you are still running your ordinary OS, be it Windows or a flavor of Linux, so you can do your ordinary tasks as well as use Octave or whatever other software is on the live CD.
Live CDs booted into virtual machines are also decent at numerically intensive work. Using PK on a 4 node entirely virtualized cluster (VMware server, which is free to download) is about 75% as efficient as running natively, using an Octave task as a benchmark.
Finally, a virtual machine can be snapshotted and then restored at any time. In this way, it's available in seconds, with no need to go through the boot process every time.