In the nested function, the variable x is passed in edi, and the pointer to the nested stack frame is passed in r10. In the lambda, the variable x is passed in esi, and the 'this' pointer for the lambda is passed in rdi. The function-level ABIs end up being quite different.
I can write the signature of the generated function of a C++ lambda as a C function. I cannot write the signature of a generated GCC nested function as a C function.
If your argument is that ABI doesn't matter, then by all means, propose a patch to GCC to change the ABI and see if it gets accepted.
I can not write the signature of a function that requires a static chain (which exist in many languages) in C, because we have not added such a feature to the language. But we could. And we should, because we now need a (type-unsafe) extension (__builtin_call_with_static_chain) to invoke such functions.
I do not want to change the ABI for nested functions as it is a useful ABI and a cross-language standard. ABI obviously matters, but it is not a fundamental difference in implementation that makes nested functions fundamentally different to C++ lambdas. If your point is merely that a compiler translating nested functions to lambdas would need to adapt the ABI in this case, I agree. This is not difficult though. And this question is relevant only if one allows taking the address of a nested function, because as long as it is called only locally, the compiler can use whatever ABI it wants.
Okay, so you accept that nested functions have a different ABI than C++ lambdas. And it looks like you accept that neither ABI is going to change. So long as the two features have different ABIs, they cannot be compatible with one another.
And this question is relevant only if one allows taking the address of a nested function
And that question is very relevant since taking the address of such functions (to pass to other functions, e.g., qsort) is one of the main use cases for their existence.
The ABI does not need to be compatible, because there is no way to call a C++ lambda directly from C.
If you take the address of a lambda function you get a pointer to an object of anonymous type, so you can not pass it to qsort, and qsort would also not know how to call this.
But if we added a feature similar to std::function_ref to C (i.e. a wide function pointer type), then such a type could be used to call both, nested functions and C++'s lambdas, and - in fact - many callable entities from other languages too. But for C++'s lambdas this would always involve a compiler generated thunk that adapts the ABI. This is also exactly what happens in C++ if you use std::function, because even in C++ you can not pass the address of lambda to a function without first erasing the type and creating the thunk.
Similar to how std::function_ref creates a thunk in C++ that calls the lambda so that it has a generic type-erased API that can be passed to non-templates, a conversion to a wide pointer would create a thunk that adapts the call from the nested function pointer ABI (that already exits for other languages also in LLVM whether we standardize the C feature or not) to whatever the lambda needs.
Showing things that are optimized by fully inlining into one function don't actually demonstrate equivalency, because you end up omitting anything that might actually evidence a difference in the semantics. And I get that, for your use cases, those differences might not matter. But as a compiler engineer, I can't say that only those use cases matter and therefore they're equivalent for all practical purposes.
It is also very unhelpful when you insist that this is "compatible" with other languages, where "compatible" actually means "compatible, if you put in a bunch of work in both languages to make something that makes them compatible, none of which I'm actually describing." Especially when there are competing proposals that do have compatibility in the sense of "I don't have to modify the C++ compiler to let it use this thing."
Equivalency does not mean that the everything has to be identical or even that the code has to be exactly identical for different implementations.
My point is that the implementation is structurally very similar: You synthesize a structure and put it on the stack and then pass a pointer to it around. It is so similar that you can certainly reuse your implementation of the lambda feature to implement this.
The wide pointer ABI question is also entirely orthogonal to other aspects, so we could decouple this discussion. The advantage of being compatible to other languages is because if we would use a common ABI that many other languages also use: Ada, Go, D, etc. In this case, no additional work has to be done for any of these languages. LLVM also supports this already.
The issue with C++ is that it does not use this common ABI and the closet thing it has even as a suitable API is std::function_ref. Where lambdas are not called locally, C++ already needs to create thunks anyway by going to some kind of these adaptors, so there is also no additional burden on the C++ side. One would simply have to implement this thunk in a slightly different way to adapt the calling convention.
I am not even sure that you need to do less adaption for other proposals, as many things the C++ semantics rely on do not exist in C (callable objects, templates), so you also need to adapt anyway at least in how you expose them in the language and in what other features you may need to make it work. In particular, JeanHeyds proposal does not even include the wide pointer part yet, which will also then be required at some point. The proposal also exposes far more features (different ways to capture), which makes it more work.
But I fully realize that the opposition for everything that looks different to C++ from the clang side comes from the perception that it is more work on your side. I can sympathize, but note that in GCC or other compiler that do not have a shared FE, we would essentially have to implement everything from scratch. So let's discuss this more if you want.
The advantage of being compatible to other languages is because if we would use a common ABI that many other languages also use: Ada, Go, D, etc.
I have looked it up and I can already tell you that Go is not using the ABI you would be proposing. I cannot speak for the other languages.
But I fully realize that the opposition for everything that looks different to C++ from the clang side comes from the perception that it is more work on your side.
That is not where the opposition comes from, and for as long as you continue to believe that, you will fail to understand the opposition at all.
"Closure calls follow the same conventions as static function and method calls, with one addition. Each architecture specifies a closure context pointer register and calls to closures store the address of the closure object in the closure context pointer register prior to the call."
In any case, the documented use of __builtin_call_with_static_chain in GCC and Clang is to be able to call closures of other languages, and for GCC Go is explicitly mentioned.
Comments
In the nested function, the variable x is passed in edi, and the pointer to the nested stack frame is passed in r10. In the lambda, the variable x is passed in esi, and the 'this' pointer for the lambda is passed in rdi. The function-level ABIs end up being quite different.
Thanks, but now you should explain why you think this slight (and well understood) difference in calling convention is a rather important difference.
I can write the signature of the generated function of a C++ lambda as a C function. I cannot write the signature of a generated GCC nested function as a C function.
If your argument is that ABI doesn't matter, then by all means, propose a patch to GCC to change the ABI and see if it gets accepted.
I can not write the signature of a function that requires a static chain (which exist in many languages) in C, because we have not added such a feature to the language. But we could. And we should, because we now need a (type-unsafe) extension (__builtin_call_with_static_chain) to invoke such functions.
I do not want to change the ABI for nested functions as it is a useful ABI and a cross-language standard. ABI obviously matters, but it is not a fundamental difference in implementation that makes nested functions fundamentally different to C++ lambdas. If your point is merely that a compiler translating nested functions to lambdas would need to adapt the ABI in this case, I agree. This is not difficult though. And this question is relevant only if one allows taking the address of a nested function, because as long as it is called only locally, the compiler can use whatever ABI it wants.
Okay, so you accept that nested functions have a different ABI than C++ lambdas. And it looks like you accept that neither ABI is going to change. So long as the two features have different ABIs, they cannot be compatible with one another.
And that question is very relevant since taking the address of such functions (to pass to other functions, e.g., qsort) is one of the main use cases for their existence.
The ABI does not need to be compatible, because there is no way to call a C++ lambda directly from C.
If you take the address of a lambda function you get a pointer to an object of anonymous type, so you can not pass it to qsort, and qsort would also not know how to call this.
But if we added a feature similar to std::function_ref to C (i.e. a wide function pointer type), then such a type could be used to call both, nested functions and C++'s lambdas, and - in fact - many callable entities from other languages too. But for C++'s lambdas this would always involve a compiler generated thunk that adapts the ABI. This is also exactly what happens in C++ if you use std::function, because even in C++ you can not pass the address of lambda to a function without first erasing the type and creating the thunk.
So there is no compatibility problem.
An example showing this equivalency is this:
https://godbolt.org/z/vEP5G9Pfr
Similar to how std::function_ref creates a thunk in C++ that calls the lambda so that it has a generic type-erased API that can be passed to non-templates, a conversion to a wide pointer would create a thunk that adapts the call from the nested function pointer ABI (that already exits for other languages also in LLVM whether we standardize the C feature or not) to whatever the lambda needs.
Edit: slightly updated example.
An example showing nonequivalency is this: https://godbolt.org/z/PxP4vrfM1
Showing things that are optimized by fully inlining into one function don't actually demonstrate equivalency, because you end up omitting anything that might actually evidence a difference in the semantics. And I get that, for your use cases, those differences might not matter. But as a compiler engineer, I can't say that only those use cases matter and therefore they're equivalent for all practical purposes.
It is also very unhelpful when you insist that this is "compatible" with other languages, where "compatible" actually means "compatible, if you put in a bunch of work in both languages to make something that makes them compatible, none of which I'm actually describing." Especially when there are competing proposals that do have compatibility in the sense of "I don't have to modify the C++ compiler to let it use this thing."
Equivalency does not mean that the everything has to be identical or even that the code has to be exactly identical for different implementations.
My point is that the implementation is structurally very similar: You synthesize a structure and put it on the stack and then pass a pointer to it around. It is so similar that you can certainly reuse your implementation of the lambda feature to implement this.
The wide pointer ABI question is also entirely orthogonal to other aspects, so we could decouple this discussion. The advantage of being compatible to other languages is because if we would use a common ABI that many other languages also use: Ada, Go, D, etc. In this case, no additional work has to be done for any of these languages. LLVM also supports this already.
The issue with C++ is that it does not use this common ABI and the closet thing it has even as a suitable API is std::function_ref. Where lambdas are not called locally, C++ already needs to create thunks anyway by going to some kind of these adaptors, so there is also no additional burden on the C++ side. One would simply have to implement this thunk in a slightly different way to adapt the calling convention.
I am not even sure that you need to do less adaption for other proposals, as many things the C++ semantics rely on do not exist in C (callable objects, templates), so you also need to adapt anyway at least in how you expose them in the language and in what other features you may need to make it work. In particular, JeanHeyds proposal does not even include the wide pointer part yet, which will also then be required at some point. The proposal also exposes far more features (different ways to capture), which makes it more work.
But I fully realize that the opposition for everything that looks different to C++ from the clang side comes from the perception that it is more work on your side. I can sympathize, but note that in GCC or other compiler that do not have a shared FE, we would essentially have to implement everything from scratch. So let's discuss this more if you want.
I have looked it up and I can already tell you that Go is not using the ABI you would be proposing. I cannot speak for the other languages.
That is not where the opposition comes from, and for as long as you continue to believe that, you will fail to understand the opposition at all.
Here seems to be the ABI, but I am not sure it is the right one and I am not sure if there are not different ABIs around. https://go.googlesource.com/go/+/refs/heads/dev.regabi/src/c...
"Closure calls follow the same conventions as static function and method calls, with one addition. Each architecture specifies a closure context pointer register and calls to closures store the address of the closure object in the closure context pointer register prior to the call."
In any case, the documented use of __builtin_call_with_static_chain in GCC and Clang is to be able to call closures of other languages, and for GCC Go is explicitly mentioned.
GCC: https://gcc.gnu.org/onlinedocs/gcc/Constructing-Calls.html
"This built-in can be used to call Go closures from C, .."
https://clang.llvm.org/docs/LanguageExtensions.html
"... as used by some language to implement closures or nested functions."
Yes, it is true that I completely fail to understand the opposition to this.
Doesn't this imply that the two functions have different semantics for capturing the environment?
No, why? It simply means that the arguments are in different registers.