Unless there is a bug in your remote desktop client that can be exploited by a compromised server...