I work with digital audio/MIDI and a latency of 15-20 ms between pressing a key and hearing the sound is easily audible making it near impossible to play expressive music. 10ms can be felt and is 'annoying', under that is acceptable. I usually keep latency down to 5ms, at most 7ms, only letting it increase, e.g. during mixing when responsiveness is not utterly essential.