I could definitely see an argument that knowing gradient descent is a requisite just as much as knowing matrix multiplication.

Both of these mathematical concepts are being abstracted over and are being used by the author's neutral network implementation.

Now whether you consider using other people's libraries for this as being "from scratch" is up to you.