Last week, I posted SentrySearch, a CLI for semantic video search using Gemini's embedding API. The #1 request was local model support.
Turns out Qwen3-VL-Embedding can natively embed video into the same kind of vector space, no API, fully offline. Runs on Apple Silicon (MPS) and NVIDIA GPUs (CUDA). The 8B model needs ~18GB RAM, or use the 2B model on smaller machines.
sentrysearch index /path --backend local
Also added: similarity threshold to suppress weak matches, and a Tesla metadata overlay that renders speed/location onto matched clips.
yea, it's so events on a chunk boundary still get captured in at least one chunk. i haven't had the chance to do formal benchmarks on overlap vs. no-overlap yet. the 5s default is a pragmatic choice, long enough to catch most events that would otherwise be split, short enough to not add much cost (120 chunks/hr to ~138). also it's configurable via the --overlap flag.
dashcam is just one of the use cases and the one i tested on. but this could theoretically work with any kind of video footage like home security footage
Yeah, this is a great idea, I’ve actually been thinking about exactly this as the next logical step.
SentrySearch already returns precise in/out timestamps for any natural-language query and uses ffmpeg to auto-trim clips. Turning that into an EDL (or even a direct Premiere plugin that exports an editable cut list) feels natural.
I’m not a Premiere expert myself, but I’d love to see this happen. If you (or anyone) wants to sketch out a quick EDL exporter or plugin, I’ll happily review + merge a PR and help wherever I can. Just drop a GitHub issue if you start something!
I've found I have to be very specific to get the clip I'm searching for. For example, "car cuts me off" just returned a clip of a car driving past my blindspot. But, "car with bike rack on back cuts me off at night" gave me exactly the clip I was looking for.
Thanks! Yeah that would be pretty cool, but continuous indexing would be pretty expensive now, because the model's in public preview and there are no local alternatives afaik.
This very well might be a reality in a couple years though!
Totally valid concern. Right now the cost ($2.50/hr) and latency make continuous real-time indexing impractical, but that won't always be the case. This is one of the reasons I'd want to see open-weight local models for this, keeps the indexing on your own hardware with no footage leaving your machine. But you're right that the broader trajectory here is worth thinking carefully about.
It's 2.50 an hour because Google has margins. A nation state could do it at cost, and even if it's not a huge difference, the price of a year's worth of embeddings is just $21,900. That's a rounding error, especially considering it's a one time cost for footage.
Right? $2.50 an hour is trivial to a Government that can vote to invent a trillion dollars. Even just 1 million dollars is the cost of monitoring 45 real time feeds for a year. I'm sure just many very rich people would pay that for the safety of their compound.
From what I see the code downsamples video to 5 fps, so 1 hour of video is 3600 seconds * 5 fps = 18,000 frames. 18,000 frames * $0.00079/frame = $14.22. A couple dollars more with the overlap.
(The code also tries to skip "still" frames, but if your video is dynamic you're looking at the cost above.)
you're right that the code uses ffmpeg to downsample the chunks to 5fps before sending them, but that's only a local/bandwidth optimization, not what the api actually processes.
regardless of the file's frame rate, the gemini api natively extracts and tokenizes exactly 1 fps. the 5 fps downscaling just keeps the payload sizes small so the api requests are fast and don't timeout.
i'll update the readme to make this more clear. thanks for bringing this up.
Not aware of any that do native video-to-vector embedding the way Gemini Embedding 2 does. There are CLIP-based models (like VideoCLIP) that embed frames individually, but they don't process temporal video. you'd need to average frame embeddings which loses a lot.
Would love to see open-weight models with this capability since it would eliminate the API cost and the privacy concern of uploading footage.
Turns out Qwen3-VL-Embedding can natively embed video into the same kind of vector space, no API, fully offline. Runs on Apple Silicon (MPS) and NVIDIA GPUs (CUDA). The 8B model needs ~18GB RAM, or use the 2B model on smaller machines.
sentrysearch index /path --backend local
Also added: similarity threshold to suppress weak matches, and a Tesla metadata overlay that renders speed/location onto matched clips.
Details on the README.