As a mathematician, I love how the word "differentiable" is changing meaning so fast for so many people!
If I had been told a few years ago that the mainstream definition of "differentiable function" would become so different during my lifetime I would not have believed it! Cannot wait to teach a course in "differentiable calculus" and introduce the new stuff.
As a non-mathematician, how did the word "differentiable" change its meaning? Is it not still that one can compute the derivative (so that Gradient Descent can be applied)? (honest question)
The really canonical non-differentiable function is the Takagi/Blancmange Function[0] if you like things deterministic, and the Wiener Process[1] if you like stochastic processes.
Both are everywhere-continuous, nowhere-differentiable.
But the definition of differentiable did not change, did it? RELU is not differentiable (or - in other words - is differentiable at all points except 0). Not nitpicking, just trying to improve my understanding.
> But the definition of differentiable did not change, did it?
It did! In the title of this article they are describing a blatantly non-differentiable function (the rasterizer) using the word "differentiable". This is indeed a new usage of the word, only seen since the advent of automatic differentiation a few years ago.
To be fair, AI researchers used strictly differentiable functions (which is required for back-propagation) until recently. For example lenet5 uses the logistic function.
Only in 2011, some smart-asses [1] :) experimented with rectifier units and discovered they're even better
[1] Xavier Glorot, Antoine Bordes and Yoshua Bengio - Deep sparse rectifier neural networks (2011)
They make the point in their presentation to say that their method replaces the non-differentiable step function with a differentiable sigmoid function to enable the rasterizer to be differentiable.
Add: ah I see what you mean. In uni differential calculus we learned about differentiable from above(1), from below(0), so I assumed RELU is considered "differentiable", as opposed to Cantor set or Dirichlet / Thomae's function...
> To other commenters: It’s important not to confuse floating point with mathematics. In floating point, both zeroes are considered to be exactly equal to zero, to each other. The redundant sign bit is “piggybacked” information which retains a sign across a series of multiplies or divisions, and it only affects downstream results in a few exceptional cases (eg dividing non-zero by zero, yielding inf or -inf). The behaviour has been standardized based on various pragmatic considerations, and will not always be consistent with evaluating infinitesimals in mathematics.
I can see the reasoning that log(-0.0) should give the same exceptional result as log(-1). But bear in mind it can arise from something like log( -(a+b)) where a+b evaluates to 0. Whereas ((-a)-b) would be +zero. I.e the sign of the zero is generally not meaningful when the zero originally arises from a sum or difference. Indeed, in such cases, if you consider a and b to carry “rounding fuzz”, then the proper sign of a+b is indeterminate, and not correlated to the sign bit generated by the add. When zeroes are generated by underflow of mul or div, at least the sign is “correct”.
Although it doesn't work for modern networks, where the forward pass is done with int8 (not float, thus no +0/-0), and only the backward pass is float16/32. (common for networks used in mobile phones)
Coauthor of the paper here: this tangent does not apply to this post.
The key thing about the actual work linked in this post is that it is pointing out and addressing an important case where discontinuous computations (polygon coverage at edges) are in fact meaningfully differentiable.
Thanks for re-highlighting this point, and for doing it with a pun!
While not directly related to the differentiability question, the idea that you can resolve spatial movement to well below the spatial resolution of pixels in an imager is relevant.
It is possible to resolve displacements that are e.g. 0.02 pixels wide [1] by relying on the "natural" anti-aliasing effect of the point-spread function [2] that arises from 1. the optics (where even if lenses were free of any aberration, the aperture cannot be infinitely small nor can the lens diameter be infinitely large), 2. the pixels don't sample light at a point.
This enables video motion amplification/magnification, which has been on the front page of HN before [3].
There, the rasterizer is a physical process that needs to be modeled. Here, the rasterizer is constructed to have the necessary property.
Note that if you had a perfect model of the physical process, and if it had no noise (e.g. no photon noise and no Johnson noise---impossible), and if the pixel intensities were given to infinite precision (as opposed to being discretized by the imager bit depth), then you could resolve arbitrarily small displacements of point light sources. I wonder if that points more to a continuity question than a differentiability question... but I felt compelled to make the connection in any case.
Returning back to your comment:
> case where discontinuous computations (polygon coverage at edges) are in fact meaningfully differentiable
It seems like the rasterizer ends up being a continuous computation, unless we disagree on what counts as the "rasterizer" (I'm including any anti-aliasing strategy, including one that underlyingly makes multiple calls to a discontinuous rasterizer.)
If I had been told a few years ago that the mainstream definition of "differentiable function" would become so different during my lifetime I would not have believed it! Cannot wait to teach a course in "differentiable calculus" and introduce the new stuff.