The problem with the above approach, however, is that you can't create hundreds of thousands of threads while also transparently supporting the traditional C ABI. C ABIs aren't designed to dynamically grow the stack, and so any thread that needs to invoke C (or Objective-C) code must always create threads with very large stacks (on the order of hundreds of several hundred KB or even megabytes) if they want to support legacy code.
Where did this claim originate from? It gets tossed around all the time in greenthread discussions but it's completely false. When you create a real thread even though it has an 8MB stack or whatever it doesn't actually allocate 8MB. 8MB is not the allocation size, it's the growth limit. The stack grows dynamically allocating memory when necessary until it hits that limit. The C/C++/<insert any language here> ABI doesn't need to be compiled for this because it's just page faulting. Standard OS behavior for decades.
This is exactly what we do in Crystal: just mmap a new stack and let the OS deal with growing the stack through page faults. One thing I will say though is that you can't shrink the stack, which is a problem. You also may need to adjust vm.max_map_count to get more than 32k fibers.
One thing I will say though is that you can't shrink the stack
It'd be manual but you could madvise it to shrink. Unlikely to be worth the effort unless you have one code flow that has a particularly deep stack vs. the common case, though.
So it still consumes 8 MB of address space though? On a 32 bit system that would severely limit the number of threads you could allocate in each process wouldn't it? How do you allocate 8 MB of stack times a hundred thousand, in a 32 bit address space?
Yes, it breaks on 32bit but that's an acceptable tradeoff in most places because the vast majority of places where a modern language like swift would actually be used are 64bit.
The person I was replying to asked where the 'myth' came from. It came from 32 bit systems, where it isn't a myth. Even if it doesn't apply today (and administrating thousands of pages per thread still isn't free even if you have the address space, so I'm not sure it doesn't still apply really), that's where the 'myth' came from - that's the answer to their question.
Comments
Where did this claim originate from? It gets tossed around all the time in greenthread discussions but it's completely false. When you create a real thread even though it has an 8MB stack or whatever it doesn't actually allocate 8MB. 8MB is not the allocation size, it's the growth limit. The stack grows dynamically allocating memory when necessary until it hits that limit. The C/C++/<insert any language here> ABI doesn't need to be compiled for this because it's just page faulting. Standard OS behavior for decades.
This is exactly what we do in Crystal: just mmap a new stack and let the OS deal with growing the stack through page faults. One thing I will say though is that you can't shrink the stack, which is a problem. You also may need to adjust vm.max_map_count to get more than 32k fibers.
It'd be manual but you could madvise it to shrink. Unlikely to be worth the effort unless you have one code flow that has a particularly deep stack vs. the common case, though.
So it still consumes 8 MB of address space though? On a 32 bit system that would severely limit the number of threads you could allocate in each process wouldn't it? How do you allocate 8 MB of stack times a hundred thousand, in a 32 bit address space?
Yes, it breaks on 32bit but that's an acceptable tradeoff in most places because the vast majority of places where a modern language like swift would actually be used are 64bit.
The person I was replying to asked where the 'myth' came from. It came from 32 bit systems, where it isn't a myth. Even if it doesn't apply today (and administrating thousands of pages per thread still isn't free even if you have the address space, so I'm not sure it doesn't still apply really), that's where the 'myth' came from - that's the answer to their question.
What 32-bit systems even exist anymore? Much less 32-bit systems where you want to run thousands of threads in a single process?
The question was 'where did the myth come from' not 'does it still apply today'.