Add Whisper API option
complete
Neil Chudleigh
complete
Shipped in v1.27
Check out the launch video here:
https://youtu.be/Bz4WnS_ApRE
Neil Chudleigh
in progress
GPT-3.5 access has been added for all Pro users
Janssen Manno
+1 option to add API, also for LLM, similar to iOS app 🙏
Neil Chudleigh
planned
Neil Chudleigh
v1.16.6 includes the huggingface distil models, which reduce the memory requirements of pro and ultra models by half (down to 800mb and 1.5gb - while also making them faster.
Should help with resource constraints.
D
Dev
Neil Chudleigh: Amazing, thanks. Not sure if a bug, but had to download the model myself as the app only downloads 27kb
Neil Chudleigh
Dev: Can you explain this in some more detail? What do you mean by download the model yourself and what do you mean it only downloaded 27 kb?
D
Dev
Neil Chudleigh: Stuck here like this. Manually I mean downloaded from hugging face and renamed it like yours.
M
Monica
Any update on this? My 16GB M2 cannot handle the 3GB+ of the large model :(. Love superwhisper though!
Neil Chudleigh
Monica: The Ultra English model is now 1.5 gigs instead of the 3 it was previously. This should allow it to work for you, Monica.
Neil Chudleigh
in progress
Neil Chudleigh
planned
Neil Chudleigh
Curious, if the RAM for Ultra was reduced to 1.5GB would this still be desired? Model quantization is possible to compress the size of the weights and results in a smaller memory footprint.
D
Dev
Neil Chudleigh: Probably not anymore. Would be a no brainer imo.
Neil Chudleigh
Dev: The drawback with quantization is the quality of results suffer a little bit. But definitely is more efficient on RAM, CPU and Power usage.
I had a release with them included primed up previously but decided to not go ahead given it made model selection more complex for the user.
D
Dev
Neil Chudleigh: Both options, API and quantization, would make sense for paying users. Even with 32GB I sometimes sadly have to quit superwhisper when I am running Docker and PyCharm with a big project.