Do you know if there are any modern, open recognizers that are not "massively deep learning", i.e. that one could tweak and retrain without Google-level resources, and that are not completely black box (not just wave to text)?
It would be interesting to build a more clever voice assistant. E.g. you say "computer, play ..." and it would immediately call a recognizer for game titles, and one for song titles. And the song title recognizer would know that certain song names are in English, other in Spanish, and so on. "OK Google" constantly gets tripped up on this case for me. It seems to transcribe everything to my language (German in this case) and only then interpret it. If I ask for an English language song I get silly results.
Kaldi hybrid models (mixture of DNNs and HMM acoustic modelling with WFST based decoding) are still competitive. They’re only really outperformed by the largest models (and even then, that may be a function of investment in man-hours in tuning and data cleaning, rather than the capacity of the underlying models).
You can tweak and retrain many existing models with transfer learning and that does not require you to have Google-level resources ... only the people that do the original model.
If you want good recognition quality, there's no way around massive deep learning models and getting access to 64x V100 for a month is just expensive, no matter how you approach that.
On the other hand, if you are interested in German and English, you might be in luck :)
In Schleswig-Holstein, there'll be local government voting in May and the chamber of commerce has pushed the topics towards "digitization" and "high tech research". But the problem is, even if you wanted to, most German companies cannot just use Google/Microsoft/Amazon for their speech recognition, due to data privacy and industry espionage concerns. We're in the process of setting up a collaboration between multiple small companies to promote local AI usage (https://deutscheki.de/ work in progress). As part of that effort, we're building mixed-language German-English acoustic models and language models with the focus on them being re-usable and a heavy focus on privacy. The financial approach is that tools will be accessible for free for non-commercial use and then companies can buy them for integration into their own products and/or to install it onto their own private infrastructure. Simultaneously, we'll be lobbying the local government to buy out the source code, which would turn the whole thing into a government-paid open source product. (Horizon Europe is the funding budget for that)
In our approach, the super-expensive acoustic model is going to output a real-time stream of phoneme- or character-level tokens. So if you say "Hallo" it'll output "H H A L L : L L O O O _" with : being the glottal stop and _ being the word end silence.
This data can then be easily transformed into "HALLO " and on average we see a 95% success rate on the character level.
For more advanced use-cases, the token sequence along with their likelihoods can be fed into a language model. That one would do stuff like turn "HALO" into "HALLO". Those language models are based on KenLM and you can train a good one with a 2TB SSD and 64 GB of RAM, so that'll be in the realm of regular people.
The full pipeline currently ranks worse than Microsoft Azure but better than Google Cloud in word recognition accuracy. Estimated training costs so far are €20k.
Users will hopefully receive the whole thing as a pre-compiled Windows DLL / Linux SO with SWIG language bindings. The idea behind splitting things into "wav to phonemes" and "phonemes to words" is that it allows participating companies to train specialized word models for their field and also to do hotword boosting, which is kind of what you want. If the AI recognizes that you said "computer, play ..." then it can switch to a language model which only allows song titles for the rest of the sentence. Similarly, the acoustic part will work for both German and English, so by switching the language model, you can switch the recognition language on the fly.
We are also preparing additional models like webcam virtual background replacement, spell- and grammar-checking, automated video subtitling, phone Q&A systems, etc.
Feel free to email us with the address on that page and I'll reply when we have a first release out.
Comments
Do you know if there are any modern, open recognizers that are not "massively deep learning", i.e. that one could tweak and retrain without Google-level resources, and that are not completely black box (not just wave to text)?
It would be interesting to build a more clever voice assistant. E.g. you say "computer, play ..." and it would immediately call a recognizer for game titles, and one for song titles. And the song title recognizer would know that certain song names are in English, other in Spanish, and so on. "OK Google" constantly gets tripped up on this case for me. It seems to transcribe everything to my language (German in this case) and only then interpret it. If I ask for an English language song I get silly results.
Kaldi hybrid models (mixture of DNNs and HMM acoustic modelling with WFST based decoding) are still competitive. They’re only really outperformed by the largest models (and even then, that may be a function of investment in man-hours in tuning and data cleaning, rather than the capacity of the underlying models).
You can tweak and retrain many existing models with transfer learning and that does not require you to have Google-level resources ... only the people that do the original model.
Sorry, but the short answer is: No.
If you want good recognition quality, there's no way around massive deep learning models and getting access to 64x V100 for a month is just expensive, no matter how you approach that.
On the other hand, if you are interested in German and English, you might be in luck :)
In Schleswig-Holstein, there'll be local government voting in May and the chamber of commerce has pushed the topics towards "digitization" and "high tech research". But the problem is, even if you wanted to, most German companies cannot just use Google/Microsoft/Amazon for their speech recognition, due to data privacy and industry espionage concerns. We're in the process of setting up a collaboration between multiple small companies to promote local AI usage (https://deutscheki.de/ work in progress). As part of that effort, we're building mixed-language German-English acoustic models and language models with the focus on them being re-usable and a heavy focus on privacy. The financial approach is that tools will be accessible for free for non-commercial use and then companies can buy them for integration into their own products and/or to install it onto their own private infrastructure. Simultaneously, we'll be lobbying the local government to buy out the source code, which would turn the whole thing into a government-paid open source product. (Horizon Europe is the funding budget for that)
In our approach, the super-expensive acoustic model is going to output a real-time stream of phoneme- or character-level tokens. So if you say "Hallo" it'll output "H H A L L : L L O O O _" with : being the glottal stop and _ being the word end silence.
This data can then be easily transformed into "HALLO " and on average we see a 95% success rate on the character level.
For more advanced use-cases, the token sequence along with their likelihoods can be fed into a language model. That one would do stuff like turn "HALO" into "HALLO". Those language models are based on KenLM and you can train a good one with a 2TB SSD and 64 GB of RAM, so that'll be in the realm of regular people.
The full pipeline currently ranks worse than Microsoft Azure but better than Google Cloud in word recognition accuracy. Estimated training costs so far are €20k.
Users will hopefully receive the whole thing as a pre-compiled Windows DLL / Linux SO with SWIG language bindings. The idea behind splitting things into "wav to phonemes" and "phonemes to words" is that it allows participating companies to train specialized word models for their field and also to do hotword boosting, which is kind of what you want. If the AI recognizes that you said "computer, play ..." then it can switch to a language model which only allows song titles for the rest of the sentence. Similarly, the acoustic part will work for both German and English, so by switching the language model, you can switch the recognition language on the fly.
We are also preparing additional models like webcam virtual background replacement, spell- and grammar-checking, automated video subtitling, phone Q&A systems, etc.
Feel free to email us with the address on that page and I'll reply when we have a first release out.