Desert Ant Labs：本地快速模型，可在设备上运行

Domain: desertant.com Points: 261 | Comments: 69 [Top Comments] - at first i got very excited about a new fast transcription model (voz) but turns out its just parakeet v3 with some new inference code which is macOS/iOS specific - I was also thinking that this is almost too good to be true - It's an ANE optimized version of Parakeet, with our own inference, which enabled us to push performance to about 300x realtime speed on an iPhone 16/17. Our next gen Voz model is trained from scratch and will be at least twice as fast. Android and other platforms will land soon. - Nice, now I know what I actually need. - This is a cool idea. The most useful one for me would be something that can process pdf files into a json schema. Title and tag generation from a post would also be useful. I'm interested in web app though. - OCR on steroids. Our Schemer model will soon be available (free form text to structured JSON). Once that lands, we want to jump into image to JSON. - A lot of the models would be useful in a web context, to improve on the CMS we're making for clients. But they look like most of them are iOS only, few have a node package or something other, and all the benchmark are running it on modern iPhones so I doubt it would be that fast on a 20$ VPS. - Just a few are iOS first (pure practical timing/sequencing). We plan to make all models available cross-platform in the coming weeks. - Shiny layer of marketing and proprietary code on top of open models? Voz is Parakeet 0.6B v3 Clear is DeepFilterNet 3 Ear is the language predictor from whisper-tiny ... - seems to be… - AI beige theme and obnoxious AI writing signaled as much. - It is now also reasonably straightforward if you have access to frontier LLMs, a recent-ish mac and a recent-ish iPhone to point them at the job of porting a given model to run on the ANE - it's a reasonably easy task to hill-climb at this point! - I wonder why they only support Apple platforms, citing CoreML. Doesn't Android have a similar framework, ML Kit? - We plan to make most our models available for Android and web too. Some are a bit harder to port to the different platforms and will take a bit longer to properly land on Android or web. Mostly sequencing (Voz, Clips, Title). Soon! - This is a cool way of approaching AI models. I'm a big fan of local LLMs, local specific models like this should be even more powerful. > Every model is free up to 100k monthly active devices. No tokens, no logins. I dunno about the business model though. Cloud LLM billing makes sense: you're getting another computer to do work with each request, and using their compute via their gateway that they bill you. These local models are like old school software. They're producing the weights, and then giving them to people. If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers? The whole "but we need to keep it updated for your security" doesn't really work as well for software designed to run fully offline like these local models are. I'm not saying they shouldn't get paid, but I guess I feel a personal sadness that it's less obvious how to successfully monetise such a sincerely useful and beneficial approach towards AI models. - > why should you continue getting money off me and my customers? Welcome to the concept of fair market value. Less snarkily you're conflating the concepts of price and cost; theyre not the same thing and theyre not the same for you or the seller. - How is it not obvious and fair that they are asking you to pay them when you see success (defined as >100K MAU)? How more aligned can you and them be besides this? - > If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers? I definitely understand the appeal of the desire to “buy it once”, but I think there are a few issues: - almost no software is static. Look at a package like python Requests and even though it does the simplest thing and has barely changed from a user perspective, it gets updated all the time. This is true for most software. This is doubly true for something like local AI models where both the software and the hardware are changing constantly. Subscriptions motivate sellers to keep their software up to date. - If I’m an app developer, the idea that I can try something out for $X/month is very appealing versus making an upfront investment of (let’s say) $X\*20. This is doubly true for something like local models where I will almost certainly want something new when the models improve. - To add to the first point, I work at a startup. No one asks questions when I want $20/month licenses. But let’s say I want something that’s gonna be in the 5 figure range annually. If I go to my CFO and ask for $50k upfront and then we implement something and the project fails, I look like an idiot. If I ask for $2000/mo budget for something and then we try it for two months and it fails, no one cares. Subscriptions are just safer in this sense. - I love this idea and hope to see more on-device models. How do they make money, though? I tried out their demo for Clear, the audio quality improvement model.[0] I'm not sure if it's just I don't have refined enough an ear or their demo is broken, but the "raw" and "enhanced" versions sounded exactly the same to me. [0] - The underlying model (DFNet3) is not particularly great but it is very small and fast - imo the best commercially usable denoising model is MossFormer2 (no affiliation - it's just excellent) with one drawback in that it can't remove reverb. Nvidia's RE-USE model can do what MossFormer2 does \_and\_ can remove reverb, but it is non-commercial licensed. - The website looks amazing. - Thanks! - > opinionated on-device intelligence > Hate speech triage. On-device moderation that flags hateful, abusive and threatening text What could go wrong here? - I don't think you understood. It means these are specialized models. Their toxic model could be ideal for video game lobbies without investing a ton of money if you're an indie dev This also could be ideal if you want your child to play online to have auto-censorship - 1. Not having human in the loop to review it because humans are expensive. 2. Having human in the loop to review it and subject said human to the worst other humans produce. - I definitely think there's a lot to be done with small models dedicated to specific tasks. I've always thought the REAL value is in having large models be able to easily build small models for custom tasks (which I know is kind of a thing), but perhaps just providing the small models directly is the more accessible approach. >accessible via one SDK for Swift, Kotlin, and JavaScript Lol well let me know when there's a Python SDK and I'll give it a try then. Obviously this isn't a deal breaker if you have a real case, but as someone who is willing to spin something up and try it out if there's a quick "pip install" command, this is getting put back on the shelf for now. - The founders and website seems to talk a lot about mobile devices and how many are being sold/shipped, that's the "unused compute power" they're targeting. So obviously mobile-first platforms and SDKs seem to be the focus first, hence those languages. Understandable, given their target, but just like you I wish there was an easier way to give it a try on a desktop computer. - Same, just wish I could afford the computer to train my own ngl - I would love to use Voz and Ear, but I’d need a version that is competitive with other audio transcription LLMs for platform availability - meaning macOS, Windows and Linux, and supporting GPUs if available. - Cool! is there a local model for LLM command approval? - Awaiting web version... - I wish we could pop a tiny one into my phone so that when I type "Will see you" and swipe the word "later" it chooses that instead of "lasso" - Honestly this is a really cool idea and i am glad that there are companies being built in this space. This is closest to the vision of what i want to do next.