In a high level 3d engine like this you do not need advanced math. You basically only need to know about coordinate systems. Linear algebra is good if you want to make your own 3d engine from scratch. Just start with something simple and learn along the way.