I'm after something that can transcribe medical notes and unfortunately it does not work well for that case. (almost nothing does though) There's quite a few people interested in something that doesn't turn "laparoscopic" into "leper as cop it".
Maybe the current progress will help though. Models adjusted by your own dictionary or from postprocessing fixes would be amazing.
I've been using Whisper Memos for some time now. Simple but useful app to quickly save an idea or memory when you don't have time to type. Speech recognition is much better than native one, especially with languages which are not widely supported.
This app has been my go-to solution for efficiently recording thoughts or memories without the need to spend time typing. Its proficiency in speech recognition is notably outstanding.
I had an application for radiology and whisper large-v2 with beam size five was essentially 100% across multiple different types of dictated radiology reports.
We are using Nuance Dragon Medical at work which is intuitive to use and surprisingly accurate even with very fast dictation. I have yet to come across a solution, that is as accurate, although I'm not sure if they offer solutions for end users directly.
I work at Rev.AI, we have a model tuned for medical and we are HIPAA compliant across the board. We do human and AI transcription and our ASR is #1 accuracy in the world right now
I took at a look at the Rev.AI website and didn't see any mention of a medical-specific model nor HIPAA compliance. It would be nice if this information was presented in your marketing!
Yup! Love the whisper.cpp (and llama.cpp) projects, they're incredible -- I mostly ended up using transformers.js for this just because the JS interface is a bit more straightforward/well-documented and its the same underlying model
When I try this on Firefox on Linux (not incognito) I get the following error:
Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1:2992
Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1
Media resource blob:https://www.ermine.ai/e762a6f1-f292-4b23-96e0-8059a7f9d635 could not be decoded. www.ermine.ai
Media resource blob:https://www.ermine.ai/e762a6f1-f292-4b23-96e0-8059a7f9d635 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_METADATA_ERR (0x806e0006)
(also, the weights json doesn't download at all in Firefox incognito).
Would be good if you could pop some kind of alert (literally alert() might do the trick) on an exception just so people don't wait for a couple of minutes before realizing something's gone wrong :)
I also get this error. Unfortunately nowadays most developers forget about Firefox, this includes big sites such as Microsoft Azure Console not working under Firefox and Google Cloud Console being slow on Firefox.
Thanks for the detailed error report! I was under the impression that it worked in FF, but I guess there's some additional testing needed -- I'll definitely look into it!
Very nice. Would it be easy to add other languages than English?
Also, as others have notes, I had to open it in Chrome to make it work, Firefox didn't work.
In theory, yes -- but tbh in practice, I tried using the multilingual version of whisper-tiny and saw significantly worse performance, so that's why it's english-only at the moment. Possible that there's some sort of parameter/tuning that could improve performance though, I'm looking in to it!
This is pretty cool! Just yesterday I finished a similar demo, also using transformers.js. I am currently in the process of adding real-time transcription, do you plan on adding that?
Thanks! I actually initially had it doing transcription in 5- and 10-second chunks for close-to-realtime results, but the CPU usage on my laptop (which admittedly doesn't have the best specs) was a bit higher than i wanted. 30-second blocks gave me the best balance of semi-real-time and good performance, especially since the whisper model is built for 30-second chunks. If you get real-time working smoothly though, i'd love to take a look!
Unfortunately not at the moment, but that feature (speaker diarization) is something I'm looking in to! AFAIK the Whisper model (which this uses currently) doesn't support that functionality, but I'm experimenting with adding an auxiliary model just for diarization.
Sorry, it looks like the modal background overlaps the logo so it only works once you've loaded the model -- here's a direct link: https://github.com/vishnumenon/ermine-ai
Not sure if it'll work on an Arduino, but maybe take a look at https://github.com/ggerganov/whisper.cpp -- it works on a Raspberry Pi at least, so resource requirements are fairly minimal
Doesn't work on Firefox Linux for me either ( error is Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1 )
Comments
I'm after something that can transcribe medical notes and unfortunately it does not work well for that case. (almost nothing does though) There's quite a few people interested in something that doesn't turn "laparoscopic" into "leper as cop it".
Maybe the current progress will help though. Models adjusted by your own dictionary or from postprocessing fixes would be amazing.
OpenAI Whisper is really good.
Here’s an iOS app to play with it: https://whispermemos.com
It even formats recording as paragraphs by running through GPT.
This site is using Whisper:
I've been using Whisper Memos for some time now. Simple but useful app to quickly save an idea or memory when you don't have time to type. Speech recognition is much better than native one, especially with languages which are not widely supported.
This app has been my go-to solution for efficiently recording thoughts or memories without the need to spend time typing. Its proficiency in speech recognition is notably outstanding.
I really like the accuracy of Whisper, but I feel like it operates at roughly real-time on my machine.
You can speed that up 16x with "faster whisper" https://github.com/guillaumekln/faster-whisper
How is the latency? This is whisper running on the iPhone?
I had an application for radiology and whisper large-v2 with beam size five was essentially 100% across multiple different types of dictated radiology reports.
We are using Nuance Dragon Medical at work which is intuitive to use and surprisingly accurate even with very fast dictation. I have yet to come across a solution, that is as accurate, although I'm not sure if they offer solutions for end users directly.
Have you tried Siri Dictation? The transcription process doesn’t leave iOS 14 and on - and I’ve been pleasantly surprised.
Laparoscopic - that worked fine ;)
Edit: if you have one available, mind sending over a sample deidentified note to my email in profile? I’m working on something.
No, there's no Apple hardware available in my scenario.
Also in medical context, if I can't tell where the data goes, the solution is not usable.
Have you tried https://speechmatics.com/ ? I think they have a specially tuned medical version, and quite a generous free allowance.
I have not, not will do tomorrow. Thanks for the link.
I work at Rev.AI, we have a model tuned for medical and we are HIPAA compliant across the board. We do human and AI transcription and our ASR is #1 accuracy in the world right now
I took at a look at the Rev.AI website and didn't see any mention of a medical-specific model nor HIPAA compliance. It would be nice if this information was presented in your marketing!
picovoice processes on the device and you can fine-tune the models https://picovoice.ai/platform/cat/
Have you tried Whisper and simply saying to GPT the context of conversation and to fix it. I think it should work
You'll probably need a custom model tailored to medical content.
Nice!
Are you aware that whisper.cpp has a WASM-version as well? See https://github.com/ggerganov/whisper.cpp/tree/master/example... - demo at https://whisper.ggerganov.com/
Yup! Love the whisper.cpp (and llama.cpp) projects, they're incredible -- I mostly ended up using transformers.js for this just because the JS interface is a bit more straightforward/well-documented and its the same underlying model
When I try this on Firefox on Linux (not incognito) I get the following error:
(also, the weights json doesn't download at all in Firefox incognito).Would be good if you could pop some kind of alert (literally alert() might do the trick) on an exception just so people don't wait for a couple of minutes before realizing something's gone wrong :)
I also get this error. Unfortunately nowadays most developers forget about Firefox, this includes big sites such as Microsoft Azure Console not working under Firefox and Google Cloud Console being slow on Firefox.
Thanks for the detailed error report! I was under the impression that it worked in FF, but I guess there's some additional testing needed -- I'll definitely look into it!
I released a similar web WASM transcription tool recently:
https://bigwav.app
Which languages does your tool support?
I need a Windows executable that takes a directory of audio files, and transcribes them to similarly named text files, so they can be searched.
Example: 20230406115923.mp3 ==> 20230406115923.txt 20230406083110.m4a ==> 20230406083110.txt
I wish someone would build one and sell it for $10 a copy.
Related : https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...
So this is all client-side and my speech is not sent antwhere or is ut "client-side ui" for some API?
I will have a look at the repository to find out, but maybe someone already looked into it.
It's all client side, it downloads a model and performs the inference using ONNX runtime (which uses WASM).
Very nice. Would it be easy to add other languages than English? Also, as others have notes, I had to open it in Chrome to make it work, Firefox didn't work.
In theory, yes -- but tbh in practice, I tried using the multilingual version of whisper-tiny and saw significantly worse performance, so that's why it's english-only at the moment. Possible that there's some sort of parameter/tuning that could improve performance though, I'm looking in to it!
I think so, yes. The website uses whisper-tiny.en, so I suppose all they need to do is switch to whisper-tiny instead.
Love this idea! Tried a 10 sec clip on Firefox for Android. App seems to be stuck on Transcribing... for few mins now...
Seems to fail on Desktop Firefox (I'm on Linux) as well.
Wow, it worked on an iPhone despite it saying it wouldn't work on Safari (needed Chrome).
The little "replay your audio capture" <audio> HTML element says "error" but the transcription actually worked.
This is pretty cool! Just yesterday I finished a similar demo, also using transformers.js. I am currently in the process of adding real-time transcription, do you plan on adding that?
Thanks! I actually initially had it doing transcription in 5- and 10-second chunks for close-to-realtime results, but the CPU usage on my laptop (which admittedly doesn't have the best specs) was a bit higher than i wanted. 30-second blocks gave me the best balance of semi-real-time and good performance, especially since the whisper model is built for 30-second chunks. If you get real-time working smoothly though, i'd love to take a look!
How are you currently distributing the audio download? Any interest in using a distributed CDN layer – happy to get help get it funded
There's actually no server-side component, the audio file is generated entirely client-side in JS!
Can it identify speakers? For examples
SPEAKER A: blah blah
SPEAKER B: blah blah
So it can be used for transcribing phone calls?
Unfortunately not at the moment, but that feature (speaker diarization) is something I'm looking in to! AFAIK the Whisper model (which this uses currently) doesn't support that functionality, but I'm experimenting with adding an auxiliary model just for diarization.
Is there a github link? I tried clicking the logo but it didn't work.
Sorry, it looks like the modal background overlaps the logo so it only works once you've loaded the model -- here's a direct link: https://github.com/vishnumenon/ermine-ai
Will it be released in an open source license?
Just added an MIT license!
does not work in Firefox, but works draw-droppingly well in Chrome
Same observation on Firefox 112.0b2 (64-bit) here.
I hope you didn't crack your draw when you were speaking your comment....
So nice, it worked.
Is there a version of this we can run on ESP32 (arduino) devices?
Thanks!
Not sure if it'll work on an Arduino, but maybe take a look at https://github.com/ggerganov/whisper.cpp -- it works on a Raspberry Pi at least, so resource requirements are fairly minimal
Keep up the good work!
You're only going to be able to do wake-words and command recognition locally on a microcontroller. The maximum external SPI RAM on an ESP32 is 4MB.
https://github.com/espressif/esp-skainet
https://github.com/tomzbj/dhrystone_score
Nice! Also enjoy the ffmpeg usage. Gonna give it a try!
Doesn't seem to work in Safari
Thanks for the feedback! I tested in Chrome and Firefox but unfortunately I run Fedora so I haven't been able to test in Safari. I'll look into it!
Doesn't work on Firefox Linux for me either ( error is Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1 )
Thanks, appreciate the error message, will look into it ASAP!
Safari was forked from Konqueror. While a lot has changed since then it might be worth it to try Konqueror.
Doesnt work in firefox for me (latest version on windows).
Worked on safari on iOS for me.
Normally I expect a lot of things that push the limit to not work on iOS, but this one did!
Worked for me on Safari
can we add a "new" button? I have to refresh to do it a second time
Definitely, I'll add one ASAP! Thanks for the suggestion
Works great, thanks
I want this but let me upload a file first.