I'd argue that code contests like Google Code Jam are the closest to 'controlled experiments' that we have in software engineering. There's a set of specs of what is to be implemented, and a limited time in which to do it. (With different contest structure/judging criteria to testing different things like, speed, adaptability/maintainability, etc.) By trying to entice as many programmer/programmer teams to enter, differences in individual programmer is hopefully lessened.
Even your question, "(What is) the effect of technology choices on programmer productivity?", tries to treat programmers as interchangeable when tech choice A is better for all programmers. Programmer productivity, like intelligence, is something that if you'd have a license to print money if you could accurately and precisely measure.
Comments
I'd argue that code contests like Google Code Jam are the closest to 'controlled experiments' that we have in software engineering. There's a set of specs of what is to be implemented, and a limited time in which to do it. (With different contest structure/judging criteria to testing different things like, speed, adaptability/maintainability, etc.) By trying to entice as many programmer/programmer teams to enter, differences in individual programmer is hopefully lessened.
Even your question, "(What is) the effect of technology choices on programmer productivity?", tries to treat programmers as interchangeable when tech choice A is better for all programmers. Programmer productivity, like intelligence, is something that if you'd have a license to print money if you could accurately and precisely measure.