You disassemble and analyze any software you have access to, and then you put Wireshark in between the software and the device. You watch the bytes being sent to the device and trace their path backwards through the software. Eventually patterns become visible and you can start making guesses about how things work.