The basic idea is to have an expressive programming language where all constructs are differentiable. Since the composition of diffeomorphisms is a diffeomorphism, large programs (like ray-tracers) will be differentiable as a result.
I don't think you mean to use a term as strong as diffeomorphism. A diffeomorphism is differentiable, sure, but it is also invertable with a differentiable inverse. They have a lot of properties that machine learning honestly does not want, like preservation of dimension.
The basic idea is to have an expressive programming language where all constructs are differentiable. Since the composition of diffeomorphisms is a diffeomorphism, (...)
No. It has nothing to do with diffeomorphisms (which are necessarily between spaces of the same dimension), but with piecewise differentiable functions.
Wow, I wish you'd written the article. Thank you. I can only imagine what you'd be able to explain if you had as much space as the author...
It sounds like you're just describing an evolution of functional programming; where each area of computation in the program is contextually generalizable, because its set of inputs and outputs is smooth and differentiable.
One followup question: are you sure the maps need to be (or are generally intended to be) isomorphic? That strikes me as very limiting. I follow your point about compositions of diffeomorphisms being diffeomorphisms themselves, but do you need that invertibility to make this paradigm work?
Differential programming is not tied to functional programming. The Python library PyTorch enables differential programming since you can use almost arbitrary python code and differentiate it, including if-then control flow, allowing you to use gradient descent to optimize the parameters of your model.
Traditionally deep learning just meant a sequence of functions applied compositionally (hence the "deep") where each function (termed a layer) is a matrix multiply followed by some well-behaved non-linear function ("activation function"). These were differentiable by design and optimized using gradient descent.
But now the models we want to build are more complex structurally than this merely sequential composition of functions. We want to be able to use control flow, accept multiple inputs, return multiple outputs, etc but we still want the model to be differentiable so we can use an iterative optimization procedure like gradient descent. So this extension from what deep learning traditionally meant (a fairly restrictive class of sequential function compositions) to complex, branching models are now termed differentiable programs.
For a concrete example, our recent paper RenderNet learns end to end differentiable ray tracing operations like ambient occlusion and normal maps which can be used in phong shading.
Since the composition of diffeomorphisms is a diffeomorphism, large programs (like ray-tracers) will be differentiable as a result.
That's just bs. If you just read the ray-tracer article it clearly states that f(scene parameters) -> image is not differentiable -- simply put because of sharp edges of objects being rendered. Also the word `diffeomorphism` makes no sense at all in this context.
Comments
The basic idea is to have an expressive programming language where all constructs are differentiable. Since the composition of diffeomorphisms is a diffeomorphism, large programs (like ray-tracers) will be differentiable as a result.
I don't think you mean to use a term as strong as diffeomorphism. A diffeomorphism is differentiable, sure, but it is also invertable with a differentiable inverse. They have a lot of properties that machine learning honestly does not want, like preservation of dimension.
No. It has nothing to do with diffeomorphisms (which are necessarily between spaces of the same dimension), but with piecewise differentiable functions.
Thanks, I was also a little confused about that in the parent comment.
Wow, I wish you'd written the article. Thank you. I can only imagine what you'd be able to explain if you had as much space as the author...
It sounds like you're just describing an evolution of functional programming; where each area of computation in the program is contextually generalizable, because its set of inputs and outputs is smooth and differentiable.
One followup question: are you sure the maps need to be (or are generally intended to be) isomorphic? That strikes me as very limiting. I follow your point about compositions of diffeomorphisms being diffeomorphisms themselves, but do you need that invertibility to make this paradigm work?
Differential programming is not tied to functional programming. The Python library PyTorch enables differential programming since you can use almost arbitrary python code and differentiate it, including if-then control flow, allowing you to use gradient descent to optimize the parameters of your model.
Traditionally deep learning just meant a sequence of functions applied compositionally (hence the "deep") where each function (termed a layer) is a matrix multiply followed by some well-behaved non-linear function ("activation function"). These were differentiable by design and optimized using gradient descent.
But now the models we want to build are more complex structurally than this merely sequential composition of functions. We want to be able to use control flow, accept multiple inputs, return multiple outputs, etc but we still want the model to be differentiable so we can use an iterative optimization procedure like gradient descent. So this extension from what deep learning traditionally meant (a fairly restrictive class of sequential function compositions) to complex, branching models are now termed differentiable programs.
Thank you, this added a lot more clarity.
For a concrete example, our recent paper RenderNet learns end to end differentiable ray tracing operations like ambient occlusion and normal maps which can be used in phong shading.
No “coding” required.
https://github.com/thunguyenphuoc/RenderNet https://papers.nips.cc/paper/8014-rendernet-a-deep-convoluti...
That's just bs. If you just read the ray-tracer article it clearly states that f(scene parameters) -> image is not differentiable -- simply put because of sharp edges of objects being rendered. Also the word `diffeomorphism` makes no sense at all in this context.