The money quote, after several pages of did-you-try-submitting-a-ticket and no-the-other-ticket-system-via-the-help-pages:
Sorry for the inconvenience here, currently we do not have the ability to revoke the free Universal SSL certificates, however, after disabling Universal SSL, the certificates are removed from our edge. If you enable it again, new certificates are issued, since we do not have those old certificates anymore.
However, depending on the issuer you may be able to request their revocation directly with the CA:
Comments
The money quote, after several pages of did-you-try-submitting-a-ticket and no-the-other-ticket-system-via-the-help-pages:
...when the CA (DigiCert) requires the private key to perform the revokation which Cloudflare does not give away