Type-safety helps you ensure that your training script is not going to fail after a couple hours because of some simple type error.
This isn’t a failure mode that ever happens in DL... 2 hours into the job you will only be dealing with floats anyway no matter what language you are using. If you’re going to fail on anything typed it will be in the first 20 seconds probably, basically the instant you start your first epoch.
This is true, but it's also because Tensorflow is a typed language, but it uses the syntax of python. (At least the default way) in tensorflow, the graph is type checked on startup, and it'll fail if anything is wrong.
Contrast this with pytorch, chainer, or tensorflow's dynamic computation graphs and they're much more likely to have a bug that happens later, since their graphs aren't verified up front.
Unfortunately, typed languages won't help you much there. A big reason people use pytorch is because of its flexibility (i.e. they were bumping up against the constraints of a static graph system and wanted out)
I admit I don’t have much experience with those, I’m a Keras and CNTK guy but the principles will be the same: marshal your data into a huge matrix of floats/one-hot and hand it off to training where it will spend 99.9% of it’s time.
I am a fan of strong/static typing and was once very active in the OCaml community but that just struck me as a very odd thing for the OP to say... it’s just not something that people doing DL worry about. It could be valuable in the marshalling phase but that all happens before DL begins and (in my experience) in a separate program.
Comments
Type-safety helps you ensure that your training script is not going to fail after a couple hours because of some simple type error.
This isn’t a failure mode that ever happens in DL... 2 hours into the job you will only be dealing with floats anyway no matter what language you are using. If you’re going to fail on anything typed it will be in the first 20 seconds probably, basically the instant you start your first epoch.
This is true, but it's also because Tensorflow is a typed language, but it uses the syntax of python. (At least the default way) in tensorflow, the graph is type checked on startup, and it'll fail if anything is wrong.
Contrast this with pytorch, chainer, or tensorflow's dynamic computation graphs and they're much more likely to have a bug that happens later, since their graphs aren't verified up front.
Unfortunately, typed languages won't help you much there. A big reason people use pytorch is because of its flexibility (i.e. they were bumping up against the constraints of a static graph system and wanted out)
I admit I don’t have much experience with those, I’m a Keras and CNTK guy but the principles will be the same: marshal your data into a huge matrix of floats/one-hot and hand it off to training where it will spend 99.9% of it’s time.
I am a fan of strong/static typing and was once very active in the OCaml community but that just struck me as a very odd thing for the OP to say... it’s just not something that people doing DL worry about. It could be valuable in the marshalling phase but that all happens before DL begins and (in my experience) in a separate program.