You're describing theoretical computer science:

Theoretical computer science is a division or subset of general computer science and mathematics that focuses on more abstract or mathematical aspects of computing and includes the theory of computation.

https://en.wikipedia.org/wiki/Theoretical_computer_science