If all you want to do is limit memory usage, process control groups can do that with less overhead than a VM.

kernel/Documentation/cgroups/memory.txt circa line 255