I've never really had much luck with stuff like this.
Whenever I have tried to write code in the C to be deliberately vulnerable and have tried to overwrite memory etc it never seems to work in the way I expect.
For example I will have two arrays that are contiguous in memory (checking this with gdb) and I will then write a big set of values into one array that should overflow into the other. Then try and do something like print the values that should have been overflowed into but I often find I either get a segfault or that the values that are printed are actually the correct values assigned the the array that should be overwritten.
I haven't tried these specific puzzles with the VMs though. I always do -fno-stack-protector when compiling but I don't know if there is some other security mechanism that could be stopping it from working?
-fno-stack-protector does a number of things that interfere with "standard" smashing of the stack, You'll learn a lot if you keep stack protections in place!
Also, be sure to turn off Write^Execute memory pages and ASLR as another commenter has suggested.
Alternatively, you could try your hand at defeating these protection mechanisms yourself with a number of (relatively) newfangled techniques.
One thing to note is that gdb screws with memory allocation. Just because it looked that way while running gdb does not mean it will look that way when run normally.
Linux 2.6.12 an up implements address space randomization (randomly varies the location of the stack over a 8MB range) which could explain what you're seeing.
Comments
I've never really had much luck with stuff like this.
Whenever I have tried to write code in the C to be deliberately vulnerable and have tried to overwrite memory etc it never seems to work in the way I expect.
For example I will have two arrays that are contiguous in memory (checking this with gdb) and I will then write a big set of values into one array that should overflow into the other. Then try and do something like print the values that should have been overflowed into but I often find I either get a segfault or that the values that are printed are actually the correct values assigned the the array that should be overwritten.
I haven't tried these specific puzzles with the VMs though. I always do -fno-stack-protector when compiling but I don't know if there is some other security mechanism that could be stopping it from working?
-fno-stack-protector does a number of things that interfere with "standard" smashing of the stack, You'll learn a lot if you keep stack protections in place!
Also, be sure to turn off Write^Execute memory pages and ASLR as another commenter has suggested.
Alternatively, you could try your hand at defeating these protection mechanisms yourself with a number of (relatively) newfangled techniques.
JITSpray - http://dsecrg.com/pages/pub/show.php?id=22 ROP - http://cseweb.ucsd.edu/~hovav/talks/blackhat08.html
and more!
One thing to note is that gdb screws with memory allocation. Just because it looked that way while running gdb does not mean it will look that way when run normally.
Linux 2.6.12 an up implements address space randomization (randomly varies the location of the stack over a 8MB range) which could explain what you're seeing.