The problems exposed by languages like Ruby are not necessarily problems with dynamic types. For example you can do reasoning about purity, side-effects and whatnot in languages of the LISP family as well. Clojure for example is much, much saner than Ruby in all the points listed by TFA - static typing is for example by definition anti-modularity and anti-adaptability and note that I prefer static typing over dynamic typing and Scala over Clojure. On the issue of static versus dynamic one has to view this as a different school of thought and to apply one versus the other depending on the needs of the project.
Uncontrolled side-effects are the real issue behind most of the accidental complexity that we are seeing. We all badly need to adopt more abstractions and techniques from functional programming.
Also, changing languages or idioms doesn't necessarily help with the exposed problems. We also need a change of mentality in how we are doing software development. Lets face it, when we need to do something right now, urgent, that should have been done yesterday - no matter the language, no matter the abstractions or idioms involved, we are bound to do stupid shit - because there's accidental complexity and then there's inherent complexity and nothing saves you from inherent complexity other than thinking really well about the problem at hand and splitting it into simpler, more manageable parts.
This is also why TDD is a failure and complete bullshit in how it is advertised. Tests don't save you from doing stupid shit. Tests don't tell you whether your architecture is any good, they only tell you if your architecture is testable. Tests don't prove the absence of bugs, they only prove their presence. Tests only tell if you reached a desired target, not what that target should be. And perhaps most importantly since this is touching the core of their purpose, when uncontrolled side-effects are happening in your system, tests are a poor safety net - anybody that had to deal with concurrency issues can attest to that.
Agile methodologies are also trying to paint a turd. Yes, we should deploy or publish as soon as we've got something to publish. We should pivot a lot. We should communicate more with the end-users or within the team. And so on and so forth. But it's an indisputable fact that some problems are hard enough that they can't be solved by puking code and tests in a matter of hours or days, or by adding more people to the team.
static typing is for example by definition anti-modularity and anti-adaptability
This is so painfully wrong. Static types are not in the way of modularity and type correct programs are not hard to adapt to another type correct program.
I would say it's the exact opposite. Types are perfectly modular and compose beautifully. Adapting your program's design in a type safe environment is a breeze, because your compiler/types more often than not tell you exactly what you can and cannot do.
And also, imagine trying to build a system that works like our body does, particularly fascinating is the process of wound healing: https://en.wikipedia.org/wiki/Wound_healing
You know, there's a reason for why Akka's actors, a library built for a fairly static and expressive language (Scala), are dynamically typed. Try finding out why that is.
The existence of the Expression Problem is orthogonal to whether or not types promote modularity or not. In fact it has a type-safe solution. I recommend reading through the original post Wadler wrote to get a better sense of what it is saying: http://homepages.inf.ed.ac.uk/wadler/papers/expression/expre...
(There is also a link at the bottom of the Wikipedia article you linked to.)
One can imagine the immune system being a kind of type checker for the body. Is it "static" or "dynamic?" Both! The "innate" immune system is basically fixed at DNA compile-time, while the evolutionarily later "adaptive" immune system is updated at runtime by exposure to pathogens.
I think it's unfortunate that we conflate strong explicit typing with static compilation. Common Lisp, for example, can be seen as statically typed; it can give you compile-time type errors. But you can also recompile at runtime; add new types at runtime; and so on.
Exactly - I did want to express that, forgetting to mention that it's not a black and white issue. I mentioned the healing process as fascinating, because the cells act as independent actors that when the wound happens, they first start arguing against each other, but then they start cooperating to reduce the wound, to provide cover for the new tissue and to form the new tissue. We are far from building systems that are as smart as the process of evolution could build.
Ruby is more dynamic than other languages and that's in a bad sense. I always get a kicker when thinking about the purpose of the "class" keyword, being to open the class context or create the class as a side-effect if it doesn't exist. Ruby is built for runtime mutation of types - which is good in certain contexts, but unfortunately you cannot scope those mutations, leading to the ultimate side-effecting hair ball if not careful about both your code and other people's libraries - it would be useful to say, modify the String type or import this library, but only for this block of code.
Notice how the very definition you give of the Expression Problem has the constraint of "retaining static type safety"?
It doesn't care about dynamic solutions, and they don't capture as much information anyway. Of course you can express anything in a dynamic fashion, in the end assembly is untyped too.
So, if those indeed were "people smarter than you", you didn't understand what they were saying.
I don't get the misunderstand we are having here. I was claiming that static typing has problems with modularity and adaptability because - one reason would be the expression problem.
You're then saying that the expression problem wouldn't exist if dynamic typing is allowed. Well, that was my whole point.
Akka will have typed actor support in future versions. There are obvious reasons why a direct port of code that was originally written for an untyped language (Erlang) would be written in an untyped way.
No, I'm right, because people smarter than me have said it.
Wow. Just... wow. Seriously? No! The expression problem can be solved in Haskell in a straightforward manner using type classes[0].
You know, there's a reason for why Akka's actors, a library built for a fairly static and expressive language (Scala), are dynamically typed. Try finding out why that is.
Because Scala is a crappy, broken language built out of compromises with the JVM? Take a look at Haskell's distributed-process[1] library. It solves the problem that Akka punted on (namely type-safe serialization of function closures).
Akka has to expose the actor registry, has to handle network connectivity issues and scalability issues, which means messages have to go through actors posing as proxies and routers; and actors may be dynamically created - and also because of the async model, actors can and must change/adapt their interface asynchronously, like when doing back-pressure and waiting for acknowledgement that a message was received on the other end, during which there can be no progress, as in "context.become", fundamental for the actor model - which actually renders static typing useless.
Saying that Haskell's distributed-process solves the problems that Akka is solving is short-sighted and wrong. But then again, the main reason for why Haskell hasn't attracted me has nothing to do with technicalities and everything to do with a vocal minority of Haskell users that I find totally repulsive.
But then again, the main reason for why Haskell hasn't attracted me has nothing to do with technicalities and everything to do with a vocal minority of Haskell users that I find totally repulsive.
This.
The expression problem can be solved in Haskell in a straightforward manner using type classes.
Lol, WUT? No.
Btw, I think that you are actually trying to lecture Scala people about typeclasses just shows how clueless you are.
This is also why TDD is a failure and complete bullshit in how it is advertised. Tests don't save you from doing stupid shit.
Why do people think tests are for that? Tests are basically same as free climbers safety lines. They won't protect you from everything and sure they are tedious to place, but once they save your tush, you'll be glad they were there, and you weren't splattered across the floor.
I've seen lack of tests in practice and it's not pretty. Nope. Not pretty at all. Basically lots of entangled undocumented, untested systems that you can't refactor because your refactor just broke some code somewhere.
To demonstrate some of the bugs, if you accidentally type your username wrong, the whole server crashes and resets. It was not pretty.
Tests are basically same as free climbers safety lines.
I never claimed that they aren't useful.
once they save your tush, you'll be glad they were there
Of course, I'm glad when I work on well tested codebases. But I was speaking about the advertisement that it received. And people really do think that because that's what TDD enthusiasts claimed - and note that I'm making a difference between testing and TDD.
The introduction is genius and I quote: "Ron Jeffries attempts to create a sudoku solver - here, here, here, here and here. (You really ought to read these articles. They are ummm...{cough} ...err.... enlightening.) ... Peter Norvig creates a Sudoku Solver."
I don't honestly know what people claim TDD does. But where the requirements are clearly specified (albeit changing) and problem space isn't well understood it has its uses.
The example given is one TDD is kinda one of worst case scenarios for it. If for example the goal was to write a novel game that is popular with some audience Norvigs analytical approach would falter. TDD wouldn't fare much better but I believe it would be better than purely analytical problem. I however have little doubts Norvig would adapt to the challenge.
Do note that by TDD I consider only red-green-refactor methodology on clearly specified parts and adding tests on encountered bugs. Rest I consider fluff.
Summary: Tests and test first design is a tool. It can't be used for everything.
Tests are basically same as free climbers safety lines.
Absolutely. So many people seem to aggressively argue for 'test all the things!' or 'test nothing!', where I imagine most jobbing programmers are practical enough to test the key things first and then expand from there. If I'm greenfielding something, I'll do TDD-first, but on new features on an old code-base or legacy apps it's initially way more just a catch in case I come off the mountain unexpectedly. I don't want to spend a day writing a feature and then find out I've broken something fundamental, I want to find that out after a couple of hours.
"you can't refactor because your refactor just broke some code somewhere" - picking a language that is not matched with first-class automated refactoring (ReSharper, IDEA) was their first mistake (rather than lack of unit testing).
You couldn't rename things because they used AspectJ to bind certain method names to some validation methods. Of course this was well documented on a dark side of planet orbiting Andromeda.
Also the Javascript used #id an suffixes to call JSP fragments. You change the name of the div in that JSP fragment and you have to change any mention of it in any JS or Java files.
On the other hand an expressive static type system (with emphasis on expressive, because Java's type system doesn't qualify) can save you from countless of accidental bugs happening - and a single bug that's caught by the compiler is a bug that won't reach production.
Plus, static typing approaches the problem from a different perspective than unit testing. Through testing we try to prove that a piece of code conforms to the business logic that is being solved and to guard ourselves against regressions. A static type system on the other hand actually proves that your code has certain characteristics - for example depending on the language we are talking about, it can prove that you won't get any null pointer exceptions, or it can prove that the interface of this component is still the one expected by this other component, etc... for example I have a component modeled as a fairly complex FSM and I fixed a difficult non-deterministic bug by eliminating the possibility of it happening through the type-system - it was quite the eye opener.
Static typing doesn't negate the need for testing of course since they serve a different purpose, but if you find yourself writing tests for things that a compiler could prove, then you probably picked the wrong language.
You'd be surprised what transformation modern IDEs can do that IS refactoring, not just renaming and pushing variables around.
Plus, you're wrong: a lot of refactoring is also about renaming stuff and pushing variables around. You don't always have to rewrite everything in new hierarchies and patterns in order to do a refactor.
Renaming, pushing variables, changing their types, stuff like that. It's still
editing work. Can any IDE split the class into two according to methods'
responsibilities? Can it abstract a set of functions to a single generic
function? Can it get rid of unnecessary boilerplate code scattered around in
various classes? Can it change data structures used to store data?
Because refactoring is about making the code simpler and more flexible.
Rearranging that IDEs do is only a method (and not the only one) to achieve
that.
Anyone who doubts this should watch Jim Weirich's talk "Adventures in Functional Programming" - the last (and most mind-blowing) quarter of the talk is almost entirely made up of automatic refactoring.
I think a large part of good software design involves making the implicit into the explicit (although there are always tradeoffs).
Instead of writing six different very similar functions that implicitly do the same thing but on different types, we write a single function with generic type parameters. Instead of writing a bunch of nested for loops working on a bunch of mutable state in an imperative way, we use map, filter, reduce, fold, etc to explicitly describe the transformations we are trying to accomplish. Instead of allowing our code to crash when bad inputs cause an error, we use asserts to declare pre- and post-conditions. And so forth.
TDD style tests have their place, but they don't do all that much to make the implicit into the explicit. If you use tests as a source of truth or a form of documentation, you are essentially saying your specification doesn't actually exist except as some weird emergent phenomenon - and then you have to take the time to disentangle the assumptions inherent in the tests from the assumptions inherent in the choices of test inputs and outputs. Tests can, by their nature, only exercise an infinitesimally small fraction of the possible combinations of inputs and state transitions that your code can go through. So tests, which serve to validate the code, need themselves to be validated. How can we even arrive at the intuition that our tests are covering some useful sample of this gigantic state-space? Metrics like code coverage and the size of a test suite are only meaningful if you assume the tests themselves are sound in the first place.
Part of the problem I see after 11 years in the business, is that now I can see problems like you describe. I try and push as much work to the database as possible, not for performance (though that is a consideration) but because it is declarative, so you get less side effects. Its not as natural to think in sets as it is to think imperatively. But do people want to pay me more for this knowledge? Not here (in Spain). They want to pay extra for people using the latest fad technology, not people who know how to avoid problems at the design stage.
On the other hand, people pay first of all for working solutions. If you keep building robust and simple solutions, your managers will see it eventually - even though many managers don't recognize design talent when they see it, for fear of having to increase your salary.
Or the incompetent ones aren't seeing it because their metric is in the lines of code written, whereas a good developer always avoids writing unneeded code - this can be extended to other areas as well - people really good at solving concurrency issues for example, avoid concurrency issues like the plague they are. If that's the case, then it's time to search for something better. I live in Romania, our situation is similar, but I discovered that I have no problems in finding interesting work remotely - I stopped doing that because it's boring for me to not have colleagues nearby and found a small local company that's pretty cool. But yeah, you don't have to stay within the local market if you can't find something you like.
How do you go about getting remote clients? I am keen to work remotely for a few reasons. Seems that you need a portfolio, but all my work is in house (though I am working on a small app to show I can code to a decent standard).
It's not natural to think imperatively, otherwise people would have no trouble with learning programming. It's just the way we were taught to write programs.
Its generally easier that thinking in sets. Certainly my not so talented team leader always describes database queries in terms of "if it has this then do this". I translate these into "give me the set where".
No, it's not generally easier. You can't generalize because people around you, who were taught to think imperatively, are thinking imperatively. You don't know how would it be if they were taught to think in terms of set theory. It might be the other way around or it could be not. That's the point: we don't have the data to generalize on which thinking is easier.
Ok, have you got any non anecdotal evidence that proves it is "being taught imperatively" that makes imperative programming the "mainstream"? In 11 years I have met very few people who program functionally, and usually those people are good enough to know multiple paradigms. As far as I know most universities teach both.
Most of the universities I have seen or heard about teach C, C++, Java, Python
or the like on their courses. Only a subset of them teach also functional
languages (which is still quite far from declarativeness, but let's ignore
this for the time being), but even those don't require functional programming
throughout the whole time and people fall back to imperative languages.
Let's go further. Most of the materials in the internet are about programming
in imperative languages. Those are the most popular, including popularity
among newcommers.
I have never heard about anybody who started learning programming by
learning Lisp, Haskell, OCaml, SML or any other functional language. Similarly
I haven't heard about somebody, who learned one of those as his second
language and got any proficiency in them.
Please provide any data that contradicts the common sense conclusion (at least
common sense for me) that vast majority of people who were taught programming,
were taught programming imperatively.
Comments
The problems exposed by languages like Ruby are not necessarily problems with dynamic types. For example you can do reasoning about purity, side-effects and whatnot in languages of the LISP family as well. Clojure for example is much, much saner than Ruby in all the points listed by TFA - static typing is for example by definition anti-modularity and anti-adaptability and note that I prefer static typing over dynamic typing and Scala over Clojure. On the issue of static versus dynamic one has to view this as a different school of thought and to apply one versus the other depending on the needs of the project.
Uncontrolled side-effects are the real issue behind most of the accidental complexity that we are seeing. We all badly need to adopt more abstractions and techniques from functional programming.
Also, changing languages or idioms doesn't necessarily help with the exposed problems. We also need a change of mentality in how we are doing software development. Lets face it, when we need to do something right now, urgent, that should have been done yesterday - no matter the language, no matter the abstractions or idioms involved, we are bound to do stupid shit - because there's accidental complexity and then there's inherent complexity and nothing saves you from inherent complexity other than thinking really well about the problem at hand and splitting it into simpler, more manageable parts.
This is also why TDD is a failure and complete bullshit in how it is advertised. Tests don't save you from doing stupid shit. Tests don't tell you whether your architecture is any good, they only tell you if your architecture is testable. Tests don't prove the absence of bugs, they only prove their presence. Tests only tell if you reached a desired target, not what that target should be. And perhaps most importantly since this is touching the core of their purpose, when uncontrolled side-effects are happening in your system, tests are a poor safety net - anybody that had to deal with concurrency issues can attest to that.
Agile methodologies are also trying to paint a turd. Yes, we should deploy or publish as soon as we've got something to publish. We should pivot a lot. We should communicate more with the end-users or within the team. And so on and so forth. But it's an indisputable fact that some problems are hard enough that they can't be solved by puking code and tests in a matter of hours or days, or by adding more people to the team.
This is so painfully wrong. Static types are not in the way of modularity and type correct programs are not hard to adapt to another type correct program.
I would say it's the exact opposite. Types are perfectly modular and compose beautifully. Adapting your program's design in a type safe environment is a breeze, because your compiler/types more often than not tell you exactly what you can and cannot do.
No, I'm right, because people smarter than me have said it. Here's the quintessence of this argument: https://en.wikipedia.org/wiki/Expression_problem
And also, imagine trying to build a system that works like our body does, particularly fascinating is the process of wound healing: https://en.wikipedia.org/wiki/Wound_healing
You know, there's a reason for why Akka's actors, a library built for a fairly static and expressive language (Scala), are dynamically typed. Try finding out why that is.
The existence of the Expression Problem is orthogonal to whether or not types promote modularity or not. In fact it has a type-safe solution. I recommend reading through the original post Wadler wrote to get a better sense of what it is saying: http://homepages.inf.ed.ac.uk/wadler/papers/expression/expre...
(There is also a link at the bottom of the Wikipedia article you linked to.)
http://en.wikipedia.org/wiki/Argument_from_authority
One can imagine the immune system being a kind of type checker for the body. Is it "static" or "dynamic?" Both! The "innate" immune system is basically fixed at DNA compile-time, while the evolutionarily later "adaptive" immune system is updated at runtime by exposure to pathogens.
I think it's unfortunate that we conflate strong explicit typing with static compilation. Common Lisp, for example, can be seen as statically typed; it can give you compile-time type errors. But you can also recompile at runtime; add new types at runtime; and so on.
Exactly - I did want to express that, forgetting to mention that it's not a black and white issue. I mentioned the healing process as fascinating, because the cells act as independent actors that when the wound happens, they first start arguing against each other, but then they start cooperating to reduce the wound, to provide cover for the new tissue and to form the new tissue. We are far from building systems that are as smart as the process of evolution could build.
Ruby is more dynamic than other languages and that's in a bad sense. I always get a kicker when thinking about the purpose of the "class" keyword, being to open the class context or create the class as a side-effect if it doesn't exist. Ruby is built for runtime mutation of types - which is good in certain contexts, but unfortunately you cannot scope those mutations, leading to the ultimate side-effecting hair ball if not careful about both your code and other people's libraries - it would be useful to say, modify the String type or import this library, but only for this block of code.
Notice how the very definition you give of the Expression Problem has the constraint of "retaining static type safety"?
It doesn't care about dynamic solutions, and they don't capture as much information anyway. Of course you can express anything in a dynamic fashion, in the end assembly is untyped too.
So, if those indeed were "people smarter than you", you didn't understand what they were saying.
I don't get the misunderstand we are having here. I was claiming that static typing has problems with modularity and adaptability because - one reason would be the expression problem.
You're then saying that the expression problem wouldn't exist if dynamic typing is allowed. Well, that was my whole point.
Akka will have typed actor support in future versions. There are obvious reasons why a direct port of code that was originally written for an untyped language (Erlang) would be written in an untyped way.
Akka has typed channels right now, as a compromise for people that do want static typing. But go ask its designers about it on its mailing list.
No, I'm right, because people smarter than me have said it.
Wow. Just... wow. Seriously? No! The expression problem can be solved in Haskell in a straightforward manner using type classes[0].
You know, there's a reason for why Akka's actors, a library built for a fairly static and expressive language (Scala), are dynamically typed. Try finding out why that is.
Because Scala is a crappy, broken language built out of compromises with the JVM? Take a look at Haskell's distributed-process[1] library. It solves the problem that Akka punted on (namely type-safe serialization of function closures).
[0] http://paulkoerbitz.de/posts/Solving-the-Expression-Problem-...
[1] http://hackage.haskell.org/package/distributed-process
Akka has to expose the actor registry, has to handle network connectivity issues and scalability issues, which means messages have to go through actors posing as proxies and routers; and actors may be dynamically created - and also because of the async model, actors can and must change/adapt their interface asynchronously, like when doing back-pressure and waiting for acknowledgement that a message was received on the other end, during which there can be no progress, as in "context.become", fundamental for the actor model - which actually renders static typing useless.
Saying that Haskell's distributed-process solves the problems that Akka is solving is short-sighted and wrong. But then again, the main reason for why Haskell hasn't attracted me has nothing to do with technicalities and everything to do with a vocal minority of Haskell users that I find totally repulsive.
This.
Lol, WUT? No.
Btw, I think that you are actually trying to lecture Scala people about typeclasses just shows how clueless you are.
Why do people think tests are for that? Tests are basically same as free climbers safety lines. They won't protect you from everything and sure they are tedious to place, but once they save your tush, you'll be glad they were there, and you weren't splattered across the floor.
I've seen lack of tests in practice and it's not pretty. Nope. Not pretty at all. Basically lots of entangled undocumented, untested systems that you can't refactor because your refactor just broke some code somewhere.
To demonstrate some of the bugs, if you accidentally type your username wrong, the whole server crashes and resets. It was not pretty.
I never claimed that they aren't useful.
Of course, I'm glad when I work on well tested codebases. But I was speaking about the advertisement that it received. And people really do think that because that's what TDD enthusiasts claimed - and note that I'm making a difference between testing and TDD.
Particularly funny is this story on Sudoku solvers: http://ravimohan.blogspot.ro/2007/04/learning-from-sudoku-so...
The introduction is genius and I quote: "Ron Jeffries attempts to create a sudoku solver - here, here, here, here and here. (You really ought to read these articles. They are ummm...{cough} ...err.... enlightening.) ... Peter Norvig creates a Sudoku Solver."
I don't honestly know what people claim TDD does. But where the requirements are clearly specified (albeit changing) and problem space isn't well understood it has its uses.
The example given is one TDD is kinda one of worst case scenarios for it. If for example the goal was to write a novel game that is popular with some audience Norvigs analytical approach would falter. TDD wouldn't fare much better but I believe it would be better than purely analytical problem. I however have little doubts Norvig would adapt to the challenge.
Do note that by TDD I consider only red-green-refactor methodology on clearly specified parts and adding tests on encountered bugs. Rest I consider fluff.
Summary: Tests and test first design is a tool. It can't be used for everything.
Absolutely. So many people seem to aggressively argue for 'test all the things!' or 'test nothing!', where I imagine most jobbing programmers are practical enough to test the key things first and then expand from there. If I'm greenfielding something, I'll do TDD-first, but on new features on an old code-base or legacy apps it's initially way more just a catch in case I come off the mountain unexpectedly. I don't want to spend a day writing a feature and then find out I've broken something fundamental, I want to find that out after a couple of hours.
"you can't refactor because your refactor just broke some code somewhere" - picking a language that is not matched with first-class automated refactoring (ReSharper, IDEA) was their first mistake (rather than lack of unit testing).
You couldn't rename things because they used AspectJ to bind certain method names to some validation methods. Of course this was well documented on a dark side of planet orbiting Andromeda.
Also the Javascript used #id an suffixes to call JSP fragments. You change the name of the div in that JSP fragment and you have to change any mention of it in any JS or Java files.
Renaming stuff and pushing variables around is not refactoring. It's just some editing work.
IDE refactoring is glorified grep/sed.
On the other hand an expressive static type system (with emphasis on expressive, because Java's type system doesn't qualify) can save you from countless of accidental bugs happening - and a single bug that's caught by the compiler is a bug that won't reach production.
Plus, static typing approaches the problem from a different perspective than unit testing. Through testing we try to prove that a piece of code conforms to the business logic that is being solved and to guard ourselves against regressions. A static type system on the other hand actually proves that your code has certain characteristics - for example depending on the language we are talking about, it can prove that you won't get any null pointer exceptions, or it can prove that the interface of this component is still the one expected by this other component, etc... for example I have a component modeled as a fairly complex FSM and I fixed a difficult non-deterministic bug by eliminating the possibility of it happening through the type-system - it was quite the eye opener.
Static typing doesn't negate the need for testing of course since they serve a different purpose, but if you find yourself writing tests for things that a compiler could prove, then you probably picked the wrong language.
You'd be surprised what transformation modern IDEs can do that IS refactoring, not just renaming and pushing variables around.
Plus, you're wrong: a lot of refactoring is also about renaming stuff and pushing variables around. You don't always have to rewrite everything in new hierarchies and patterns in order to do a refactor.
Renaming, pushing variables, changing their types, stuff like that. It's still editing work. Can any IDE split the class into two according to methods' responsibilities? Can it abstract a set of functions to a single generic function? Can it get rid of unnecessary boilerplate code scattered around in various classes? Can it change data structures used to store data?
Because refactoring is about making the code simpler and more flexible. Rearranging that IDEs do is only a method (and not the only one) to achieve that.
Anyone who doubts this should watch Jim Weirich's talk "Adventures in Functional Programming" - the last (and most mind-blowing) quarter of the talk is almost entirely made up of automatic refactoring.
[1] http://vimeo.com/45140590
http://www.jetbrains.com/resharper/features/code_refactoring...
http://www.jetbrains.com/idea/features/refactoring.html
I think a large part of good software design involves making the implicit into the explicit (although there are always tradeoffs).
Instead of writing six different very similar functions that implicitly do the same thing but on different types, we write a single function with generic type parameters. Instead of writing a bunch of nested for loops working on a bunch of mutable state in an imperative way, we use map, filter, reduce, fold, etc to explicitly describe the transformations we are trying to accomplish. Instead of allowing our code to crash when bad inputs cause an error, we use asserts to declare pre- and post-conditions. And so forth.
TDD style tests have their place, but they don't do all that much to make the implicit into the explicit. If you use tests as a source of truth or a form of documentation, you are essentially saying your specification doesn't actually exist except as some weird emergent phenomenon - and then you have to take the time to disentangle the assumptions inherent in the tests from the assumptions inherent in the choices of test inputs and outputs. Tests can, by their nature, only exercise an infinitesimally small fraction of the possible combinations of inputs and state transitions that your code can go through. So tests, which serve to validate the code, need themselves to be validated. How can we even arrive at the intuition that our tests are covering some useful sample of this gigantic state-space? Metrics like code coverage and the size of a test suite are only meaningful if you assume the tests themselves are sound in the first place.
Part of the problem I see after 11 years in the business, is that now I can see problems like you describe. I try and push as much work to the database as possible, not for performance (though that is a consideration) but because it is declarative, so you get less side effects. Its not as natural to think in sets as it is to think imperatively. But do people want to pay me more for this knowledge? Not here (in Spain). They want to pay extra for people using the latest fad technology, not people who know how to avoid problems at the design stage.
I can feel your pain :-)
On the other hand, people pay first of all for working solutions. If you keep building robust and simple solutions, your managers will see it eventually - even though many managers don't recognize design talent when they see it, for fear of having to increase your salary.
Or the incompetent ones aren't seeing it because their metric is in the lines of code written, whereas a good developer always avoids writing unneeded code - this can be extended to other areas as well - people really good at solving concurrency issues for example, avoid concurrency issues like the plague they are. If that's the case, then it's time to search for something better. I live in Romania, our situation is similar, but I discovered that I have no problems in finding interesting work remotely - I stopped doing that because it's boring for me to not have colleagues nearby and found a small local company that's pretty cool. But yeah, you don't have to stay within the local market if you can't find something you like.
How do you go about getting remote clients? I am keen to work remotely for a few reasons. Seems that you need a portfolio, but all my work is in house (though I am working on a small app to show I can code to a decent standard).
It's not natural to think imperatively, otherwise people would have no trouble with learning programming. It's just the way we were taught to write programs.
Its generally easier that thinking in sets. Certainly my not so talented team leader always describes database queries in terms of "if it has this then do this". I translate these into "give me the set where".
No, it's not generally easier. You can't generalize because people around you, who were taught to think imperatively, are thinking imperatively. You don't know how would it be if they were taught to think in terms of set theory. It might be the other way around or it could be not. That's the point: we don't have the data to generalize on which thinking is easier.
I was taught both. Who is generalising now?
And now you support your general claim by anecdotal evidence? Note that I don't make any claim apart from "it can't be generalized for now".
Ok, have you got any non anecdotal evidence that proves it is "being taught imperatively" that makes imperative programming the "mainstream"? In 11 years I have met very few people who program functionally, and usually those people are good enough to know multiple paradigms. As far as I know most universities teach both.
Most of the universities I have seen or heard about teach C, C++, Java, Python or the like on their courses. Only a subset of them teach also functional languages (which is still quite far from declarativeness, but let's ignore this for the time being), but even those don't require functional programming throughout the whole time and people fall back to imperative languages.
Let's go further. Most of the materials in the internet are about programming in imperative languages. Those are the most popular, including popularity among newcommers.
I have never heard about anybody who started learning programming by learning Lisp, Haskell, OCaml, SML or any other functional language. Similarly I haven't heard about somebody, who learned one of those as his second language and got any proficiency in them.
Please provide any data that contradicts the common sense conclusion (at least common sense for me) that vast majority of people who were taught programming, were taught programming imperatively.