You need a few dozen bytes of storage per voice, like four bytes for the phase accumulator (uint32_t), another four for the current pitch offset that gets added, and so on.

It all adds up, especially if you want to implement a ladder filter properly.