Absolutely. It'd work with any sort of distributed filestorage. It wouldn't be too hard to wrap the S3 methods so that the API is consistent (the MogileFS API is basically a dict), so app code wouldn't even have to change.

We used Mogile over S3 mostly because I didn't realize there was no minimum monthly fee for S3.