2025-03-10 03:46:17
Re @groby @__gotcha Yeah, the threat model here can be as serious as nation state funded hacking groups
2025-03-10 02:43:05
Re @groby @__gotcha It has been interesting talking to people at the NICAR data journalism conference who have legitimate reasons to need off-line models - if you are dealing with sensitive leaked data you really can't risk it leaving your machine
2025-03-10 02:04:15
Re @RuneX_ai I'm hoping to get 10-50,000 tokens working comfortably
2025-03-10 02:03:26
Re @groby I want to do structured text extraction from longer documents, so I think I need to input between 10 and 50,000 tokens
2025-03-10 00:51:52
Anyone had much success running long context prompts through local LLMs? On my M2 64GB Mac I'm finding that longer prompts take an unreasonably long time to process, am I holding it wrong? Any models or serving platforms I should try out that might respond reasonably quickly?
2025-03-10 00:42:53
Re @burkov They're about the UI layer (and RAG tools and suchlike), not model customization - but since their stack runs against Ollama any custom model that you can turn into a GGUF should work with Open Web UI
