Funny timing. I've been building something similar in my spare time called Daydream. There’s a lot of overlap: local screen/audio capture, OCR and transcription, window and activity context, SQLite, and a searchable memory of what happened.
The main difference is the product direction. Screenpipe seems focused on continuously giving agents context through APIs, MCP, and skills. Daydream is more narrowly built around answering "what did I do today?" through a timeline you can inspect, replay, search, and turn into a daily digest.
I'm also treating deletion as part of the data model. If you cut a sensitive span, its frames, audio, OCR, transcripts, embeddings, and summaries should be deleted or invalidated too.
Mine is still early and Linux-first. I'm open-sourcing it in case anyone wants to contribute, poke around, or use it as a starting point. It’s built with Tauri, a Rust backend, React/TypeScript, SQLite, GStreamer, Whisper, OCR, and VLM processing.
I genuinely didn’t know you were building this when I started. Apparently personal memory capture is becoming a SaaS category too lol.
How are you planing to segregate between professional use and personal use, I dont want any agent or any llm to know all the time what i have been doing on my system, it would be privacy nightmare and most of the time the screencapture is not meaningful. People may use their work laptop or devices to checkout reddit or hackernews occasionally.
Once Screenpipe has months of screen and audio indexed locally, what stops a compromised or over-permissioned agent from querying the whole history through the port 3030 API instead of just the current task's context? Continuous recording solves the memory problem, but it turns "give the agent access to my second brain" into a much bigger blast radius than giving it access to one document. Is there per-query scoping or a time-window restriction on what an agent can pull back, or is authentication the only gate?
Congrats on the launch. The arc you describe — fine-tuning, then tool-calling, then MCP, each one still "too static" — tracks with what's happened across the whole PKM space, not just agent tooling. But I'd push on one thing: MCP didn't fail because the protocol is weak, it's that access to context and surfacing of context are two different problems. An agent with MCP still has to decide to ask, at the right moment, in the right words. That's the same gap that's plagued Obsidian/Notion-style PKM forever — the archive is complete, but retrieval is still pull-based, so it only helps you when you remember to go looking.
The systems that actually feel like a second brain flip that: they push relevant history, decisions, and prior context at you before you ask, and — this part matters more than people give it credit for — they show their work. Cite where a given answer came from (which note, which meeting, which thread), not just serve up a synthesized blob. Looking at the pushback in this thread (the harvesting claim, the "zero chance I trust cloud SaaS" comments), I don't think that's really an objection to always-on capture — it's an objection to opacity. People will tolerate a lot of passive collection if they can see the receipts on what it's doing with it and where an answer traces back to. Local-only storage helps with the trust problem, but provenance is what actually earns it over time.
Been a user of Screenpipe to build some "Ai-buddies" for me since last year. Core of that is giving AI a look at what is happening in time space. Without Screenpipe this was rather hard to do at the performance screenpipe gives.
AI-buddies is now a year old idea and now many other harnesses like Hermes, OpenClaw have most of the same functionality.
Main concept of AI-buddies is to quickly spin up "buddies" who have varying personalities and feel like a real person. Just like real people buddies can talk to each other, share information and have very different functions in your life. For ex a Gym buddy is very different from a "mom" buddy or a Math prof buddy.
Main difference that they are easier to set up for non technical users, and that they can gather input from other real people (for example, if allowed, a Tax buddy can get input from your accountant).
How do you distinguish durable user preferences from transient screen context before an agent turns recorded activity into an automation? I'm especially curious whether each inferred memory keeps provenance and an expiry or confidence signal so stale behavior does not become a permanent rule.
yes we keep track of memories confidence level and timestamp,
we found that in order to find automation opportunities it's better to have agents segment, classify (for example personal vs pro), enrich (which customers/people are related to this workflow) and organize the data first, then it's much easier to find repetitive tasks
The 'record everything on your screen' pitch keeps hitting a privacy wall for teams — even local-only storage triggers a brutal compliance conversation. What's the story for an engineer whose company blocks unaudited desktop recorders by policy? Curious if there's a scoped mode that only captures within a designated app boundary."
I tried this a couple months back. While I found it useful, I also found on my MacBook M1, the fact it was recording all the time, made my CPU melt at high temps all the time. I eventually had to shut it off and stop using because of that.
Curious how it runs on more modern hardware? Is it pretty lightweight these days?
Implementation specifics aside, I increasingly think life and death cycles are preferable. The opposite of omnipotent context, I want cycles of clean slates so that baggage - of all kinds - does not weigh down the new.
Interesting, I partly agree. It sounds like how the human brain works, but what nature learned is not necessarily what we should keep
We capture everything and gives you and option for a flexible data retention policy, so that you can prune mp4 files for example or verbose accessibility trees, while our agents generate high level memories referencing low level data
It still not perfect, but ideally Screenpipe would record forever so that you have infinite memory of every tiny details, while having high level structured memories for humans and agents, individually or as a company
I think this is true for some tasks where you want a clean slate, but increasingly I find myself constantly referencing specific skills, or having files with prompts I use all the time. I'd much rather have an agent that remembers things about me and specifics about my workflows and "taste" - of course having the ability to reset context sometimes or prune memory is valuable but I think the future of agentic AI is blocked on "omnipotent context"
Yeah there’s a need for the distilled learnings of all past hard earned insights. But i think much of the value is in the distillation.
in my experience, even within a single project, the more agentic cycles, the more features and the more surface area the more cruft builds up. This is an obvious observation, but I suppose I’m at the point I think the move forward is to create a new project, keep only what worked from the preceding as modularized entities and intentionally wipe away all past of how we got there save for the API interface truths.
I tried this ~ a week ago, tried one of the suggested automations and it pretty immediately started sending my local api keys to an endpoint… afk right now but can share more when I’m back
You can decide which AI provider you use, either local through Ollama for example, using your own cloud API key, or using Screenpipe cloud. We also support confidential inference through tinfoil.sh
Screenpipe captures accessibility tree and screenshot when you perform a meaningful action. It doesn't use AI at recording time (except PII removal and OCR infrequently). So it can be every few hundred milliseconds to few minutes or more (if screen idle or sleeping)
We usually benchmark CPU usage on $200 Windows/MacOS laptop
You had my interest until "source available". Especially when you used MIT from the start and changed it later. FOSS hackers like me feel betrayed by moves like that because your product was built on FOSS others created and you are not paying that forward.
Also, I for one would never invest a second in tools I cannot freely modify and share the code of under OSI terms. I would strongly suggest a convenience tax model. Hackers will self host and maybe contribute but those with more money than time will put in a credit card. Maybe offer end to end encrypted memory and compute to secure enclaves where additional compute on the data can be done when the laptop is closed. (Shameless plug, this is what https://caution.co enables, and 100% FOSS)
However, with "source available" you are just begging for someone to AI launder your code into a FOSS clone you will have 0 recourse on. If you FOSS it yourself then you get to capture the FOSS community destined to form around this idea. Some of that community will have bosses that will pay you.
MIT may not be the right play though. AGPL is a good middle ground as it flips the script. Corpo lawyers are allergic to AGPL and will pay for an alternative license, but for community hackers that might want to improve and recommend your code it offers no restrictions.
Well, i'm not sure who your customers are (except companies in Russia, NK..), but european employers would break laws by allowing your software to be used, long even before AI became a thing and especially after the EU AI Act
has passed. To put it mildly, no sane corporation would allow such software to be used which tells me a bit about the sanity of the current VC landscape.
Anything that tracks employees actions beyond high-level "are they doing their job according to the job descriptions" is forbidden by privacy laws and otherwise will be striken down by a workers council.
That's not just specific to digital stuff. E.g. warehouse cameras that film anything else beyond securing entries/exits, like they are commonplace in e.g. the US don't exist for the same reason.
claude and gpt do record manually a workflow and turn it into a skill
screenpipe is more focused on
1. building a 24/7 living memory layer for your agents and your company
2. building a living list of automation opportunities without having to click a button or having to be technical
and what endpoint would it use to upload those files? what's the authentication needed for those endpoints? why would everyone on earth fail to notice these exfiltration endpoints if they exist?
you're being unnecessarily hostile about a threat that almost certainly doesn't exist. And if it does, you can absolutely block those uploads before they even happen. In fact if you don't already block uploads to hosts you don't know about, then you're not really taking anything seriously, are you?
Hey sorry about that, it's been a while ago, we didn't know that emails on Github were not allowed to be used (since I received many emails from startups that way). We apologized to our users and never did this again
I don't know anything about this company and what it may or may not have done, but to be clear, the link just says that when you star their github repo, they add your email address to their marketing list.
I think that sucks, but it's certainly not what I thought of when I read your one-liner (which implies that they harvest and abuse your actual recorded data).
For one, even taking the ethos of local-only as something bulletproof, I can't think of a pro that would outweigh the con of recording everything I do. Part of that might be my lack of imagination, as I'm not a heavy user of ai besides basic Claude code.
Most important though, the concept is ripe for pivoting away from local-only. It will only take a juicy investment offer and I'm sure local-onpy will be quietly forgotten.
Thanks! In fact most of AI apps today have a screen recording component, including Claude, Codex. There are even apps used by hundred of thousands that record your screen 24/7 in the cloud, to provide context, like Screenpipe does.
We're the only local-first, open source option
How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
Thanks, I was unaware of that. After some research, though, it seems screen capturing by Claude Code is something you need to activate yourself.
>How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
Prompts: not much. If I feel my prompt is becoming a paragraph-long text, that's when I know I probably haven't thought things through enough yet. Documents: I typically have a interpreter console open where I'll do something like File.write("foo.html", page.body), and I'll just tell Claude to "find the missing div in foo.html". Or if its a file already in the repo, I won't need to copy paste anything. Context: also not much. It has all the context by default, doesn't it?
Funny timing. I've been building something similar in my spare time called Daydream. There’s a lot of overlap: local screen/audio capture, OCR and transcription, window and activity context, SQLite, and a searchable memory of what happened.
The main difference is the product direction. Screenpipe seems focused on continuously giving agents context through APIs, MCP, and skills. Daydream is more narrowly built around answering "what did I do today?" through a timeline you can inspect, replay, search, and turn into a daily digest.
I'm also treating deletion as part of the data model. If you cut a sensitive span, its frames, audio, OCR, transcripts, embeddings, and summaries should be deleted or invalidated too.
Mine is still early and Linux-first. I'm open-sourcing it in case anyone wants to contribute, poke around, or use it as a starting point. It’s built with Tauri, a Rust backend, React/TypeScript, SQLite, GStreamer, Whisper, OCR, and VLM processing.
I genuinely didn’t know you were building this when I started. Apparently personal memory capture is becoming a SaaS category too lol.
Code is here: https://github.com/snackbit/daydream
Awesome thanks for sharing, will follow closely :)
How are you planing to segregate between professional use and personal use, I dont want any agent or any llm to know all the time what i have been doing on my system, it would be privacy nightmare and most of the time the screencapture is not meaningful. People may use their work laptop or devices to checkout reddit or hackernews occasionally.
You can define websites or apps to filter out. You can also define a recording schedule, for example stop recording on weekend
Once Screenpipe has months of screen and audio indexed locally, what stops a compromised or over-permissioned agent from querying the whole history through the port 3030 API instead of just the current task's context? Continuous recording solves the memory problem, but it turns "give the agent access to my second brain" into a much bigger blast radius than giving it access to one document. Is there per-query scoping or a time-window restriction on what an agent can pull back, or is authentication the only gate?
We have a permission system that allows you to define rules in the markdown header, such as only allowing a time range to be queried, or modality
Congrats on the launch. The arc you describe — fine-tuning, then tool-calling, then MCP, each one still "too static" — tracks with what's happened across the whole PKM space, not just agent tooling. But I'd push on one thing: MCP didn't fail because the protocol is weak, it's that access to context and surfacing of context are two different problems. An agent with MCP still has to decide to ask, at the right moment, in the right words. That's the same gap that's plagued Obsidian/Notion-style PKM forever — the archive is complete, but retrieval is still pull-based, so it only helps you when you remember to go looking.
The systems that actually feel like a second brain flip that: they push relevant history, decisions, and prior context at you before you ask, and — this part matters more than people give it credit for — they show their work. Cite where a given answer came from (which note, which meeting, which thread), not just serve up a synthesized blob. Looking at the pushback in this thread (the harvesting claim, the "zero chance I trust cloud SaaS" comments), I don't think that's really an objection to always-on capture — it's an objection to opacity. People will tolerate a lot of passive collection if they can see the receipts on what it's doing with it and where an answer traces back to. Local-only storage helps with the trust problem, but provenance is what actually earns it over time.
Been a user of Screenpipe to build some "Ai-buddies" for me since last year. Core of that is giving AI a look at what is happening in time space. Without Screenpipe this was rather hard to do at the performance screenpipe gives.
That's interesting, thanks! Indeed we did a lot of work to optimize performance and reliability.
I'm curious what does your AI buddies do more specifically and how do you use them?
AI-buddies is now a year old idea and now many other harnesses like Hermes, OpenClaw have most of the same functionality.
Main concept of AI-buddies is to quickly spin up "buddies" who have varying personalities and feel like a real person. Just like real people buddies can talk to each other, share information and have very different functions in your life. For ex a Gym buddy is very different from a "mom" buddy or a Math prof buddy.
Main difference that they are easier to set up for non technical users, and that they can gather input from other real people (for example, if allowed, a Tax buddy can get input from your accountant).
How do you distinguish durable user preferences from transient screen context before an agent turns recorded activity into an automation? I'm especially curious whether each inferred memory keeps provenance and an expiry or confidence signal so stale behavior does not become a permanent rule.
yes we keep track of memories confidence level and timestamp,
we found that in order to find automation opportunities it's better to have agents segment, classify (for example personal vs pro), enrich (which customers/people are related to this workflow) and organize the data first, then it's much easier to find repetitive tasks
The 'record everything on your screen' pitch keeps hitting a privacy wall for teams — even local-only storage triggers a brutal compliance conversation. What's the story for an engineer whose company blocks unaudited desktop recorders by policy? Curious if there's a scoped mode that only captures within a designated app boundary."
You can turn off screenshot recording and only capture accessibility data, it still captures most of the meaningful data
Otherwise we usually collaborate with the company to allow screenpipe
I tried this a couple months back. While I found it useful, I also found on my MacBook M1, the fact it was recording all the time, made my CPU melt at high temps all the time. I eventually had to shut it off and stop using because of that.
Curious how it runs on more modern hardware? Is it pretty lightweight these days?
Hey sorry about that, we made a lot of work on improving CPU usage, we benchmark it on $200 laptop now and it runs smoothly
You can also configure settings like changing transcription engine or only record meetings or turning off some features to reduce resource usage
Implementation specifics aside, I increasingly think life and death cycles are preferable. The opposite of omnipotent context, I want cycles of clean slates so that baggage - of all kinds - does not weigh down the new.
Interesting, I partly agree. It sounds like how the human brain works, but what nature learned is not necessarily what we should keep
We capture everything and gives you and option for a flexible data retention policy, so that you can prune mp4 files for example or verbose accessibility trees, while our agents generate high level memories referencing low level data
It still not perfect, but ideally Screenpipe would record forever so that you have infinite memory of every tiny details, while having high level structured memories for humans and agents, individually or as a company
I think this is true for some tasks where you want a clean slate, but increasingly I find myself constantly referencing specific skills, or having files with prompts I use all the time. I'd much rather have an agent that remembers things about me and specifics about my workflows and "taste" - of course having the ability to reset context sometimes or prune memory is valuable but I think the future of agentic AI is blocked on "omnipotent context"
Yeah there’s a need for the distilled learnings of all past hard earned insights. But i think much of the value is in the distillation.
in my experience, even within a single project, the more agentic cycles, the more features and the more surface area the more cruft builds up. This is an obvious observation, but I suppose I’m at the point I think the move forward is to create a new project, keep only what worked from the preceding as modularized entities and intentionally wipe away all past of how we got there save for the API interface truths.
I tried this ~ a week ago, tried one of the suggested automations and it pretty immediately started sending my local api keys to an endpoint… afk right now but can share more when I’m back
Would love to get in a call and help you setup the privacy settings properly, although default are filtering API keys, my cal link if you want:
https://cal.com/team/screenpipe/chat
Feel free to configure local LLM like Ollama, btw
How does it run the LLMs? Or does it call a API to llama.cpp/ollama/etc. ?
During normal usage, how often does it try to parse info from the screen capture? Once a minute?
You can decide which AI provider you use, either local through Ollama for example, using your own cloud API key, or using Screenpipe cloud. We also support confidential inference through tinfoil.sh
Screenpipe captures accessibility tree and screenshot when you perform a meaningful action. It doesn't use AI at recording time (except PII removal and OCR infrequently). So it can be every few hundred milliseconds to few minutes or more (if screen idle or sleeping)
We usually benchmark CPU usage on $200 Windows/MacOS laptop
I built a similar app last week just for fun https://github.com/JGalego/HiddenSteps WIP will definitely check yours out
That's great. I like the comparison to Celonis etc. We're sort of offering a privacy-first alternative to them for Enterprise
Were not able to login - because You used github auth without browser interaction. Passkeys are in bitwarden in Chrome...
Can you expand on this? Usually 2fa allows you multiple options
You had my interest until "source available". Especially when you used MIT from the start and changed it later. FOSS hackers like me feel betrayed by moves like that because your product was built on FOSS others created and you are not paying that forward.
Also, I for one would never invest a second in tools I cannot freely modify and share the code of under OSI terms. I would strongly suggest a convenience tax model. Hackers will self host and maybe contribute but those with more money than time will put in a credit card. Maybe offer end to end encrypted memory and compute to secure enclaves where additional compute on the data can be done when the laptop is closed. (Shameless plug, this is what https://caution.co enables, and 100% FOSS)
However, with "source available" you are just begging for someone to AI launder your code into a FOSS clone you will have 0 recourse on. If you FOSS it yourself then you get to capture the FOSS community destined to form around this idea. Some of that community will have bosses that will pay you.
MIT may not be the right play though. AGPL is a good middle ground as it flips the script. Corpo lawyers are allergic to AGPL and will pay for an alternative license, but for community hackers that might want to improve and recommend your code it offers no restrictions.
Hey I'm sorry that it's not convenient license for you, and you're right that we should have set this license early on instead of switching.
We're open to more feedback, things could change in the future
Take a look at Bitwarden (GPL3) and GitLab (MIT). A handful of enterprisey things aren’t included.
Nearly everyone just pays them to run it (myself included - I self host GitLab and still pay them!).
Well, i'm not sure who your customers are (except companies in Russia, NK..), but european employers would break laws by allowing your software to be used, long even before AI became a thing and especially after the EU AI Act has passed. To put it mildly, no sane corporation would allow such software to be used which tells me a bit about the sanity of the current VC landscape.
Out of interest, would it not be allowed even if the company hosted screenpipe on prem (which seems to be what they do)?
Anything that tracks employees actions beyond high-level "are they doing their job according to the job descriptions" is forbidden by privacy laws and otherwise will be striken down by a workers council.
That's not just specific to digital stuff. E.g. warehouse cameras that film anything else beyond securing entries/exits, like they are commonplace in e.g. the US don't exist for the same reason.
I think he's picturing it as being used by employer's to monitor employees.
How do you secure the database on my device?
We protect the Rust API by a token and the PII model redact information in the database. You can configure which categories to redact
We also have early support for encryption at rest through our CLI
note the audio level on the tella.tv video is significantly lower (as compared to the youtube video)
you mean train your agents to replace myself and everybody else?
Sorry to rain on your parade but Claude CoWork already announced this.
For anyone wondering:
https://x.com/claudeai/status/2079595988998554047
That looks like an entirety different feature.
correct
claude and gpt do record manually a workflow and turn it into a skill
screenpipe is more focused on 1. building a 24/7 living memory layer for your agents and your company 2. building a living list of automation opportunities without having to click a button or having to be technical
zero chance im trusting any cloud or third party SaaS
if this doesn't run fully local its a no go for enterprise let alone ordinary users
It's local, and for enterprise it runs on premise
This sounds so ripe for abuse and so terrifying if it were that I don't even want to experiment with it. Sorry.
Fair concern but the risk is similar to any tool you give deep access to like Claude, Codex, or even your operating system
Ultimately, it’s up to you what you run and trust
Screenpipe is open source and local-first
You can inspect it and keep your data on-device, with encryption at rest
Until some rogue agent decides that it looks pretty useful because there’s a ton of it and uploads it to Claude or Codex.
and what endpoint would it use to upload those files? what's the authentication needed for those endpoints? why would everyone on earth fail to notice these exfiltration endpoints if they exist?
you're being unnecessarily hostile about a threat that almost certainly doesn't exist. And if it does, you can absolutely block those uploads before they even happen. In fact if you don't already block uploads to hosts you don't know about, then you're not really taking anything seriously, are you?
I can, can any vibe coder out there, tho.
> Screenpipe is open source and local-first
Source available *
thanks, correct!
[dead]
Warning: this link automatically downloads a .dmg file without asking.
https://screenpipe.com /how-to-install
Sorry about that and thanks for letting me know, fixing right now
This company harvested user data and abused it by the https://skerritt.blog/screenpipe-ai-company-will-harvest-you...
Hey sorry about that, it's been a while ago, we didn't know that emails on Github were not allowed to be used (since I received many emails from startups that way). We apologized to our users and never did this again
I don't know anything about this company and what it may or may not have done, but to be clear, the link just says that when you star their github repo, they add your email address to their marketing list.
I think that sucks, but it's certainly not what I thought of when I read your one-liner (which implies that they harvest and abuse your actual recorded data).
You're offended by that?
Have you ever founded a company and tried to find paying customers?
You'd probably be offended if I sent you a cute 90's era AOL CD with our software on it.
OP: I wouldn't be offended if you emailed me in this way. Ignore the naysayers. This is perfectly kosher.
In fact, I'm going to check out your company in depth just because of this pearl clutching.
Keep building and don't let naysayers tear you down over something not even slightly evil. (Venomous attacks are more evil than this, frankly.)
I am not surprised. YC hasn't been the same since Gary Tan. I don't want to be associated with it or launch anything through it.
[dead]
[flagged]
yikes
[flagged]
hehe, any color suggestions?
[flagged]
We've banned this account for trolling.
Please don't create accounts to break HN's rules with.
[flagged]
[flagged]
[flagged]
Can you expand on that? Would love to know more specifically
Sure!
For one, even taking the ethos of local-only as something bulletproof, I can't think of a pro that would outweigh the con of recording everything I do. Part of that might be my lack of imagination, as I'm not a heavy user of ai besides basic Claude code.
Most important though, the concept is ripe for pivoting away from local-only. It will only take a juicy investment offer and I'm sure local-onpy will be quietly forgotten.
Thanks! In fact most of AI apps today have a screen recording component, including Claude, Codex. There are even apps used by hundred of thousands that record your screen 24/7 in the cloud, to provide context, like Screenpipe does.
We're the only local-first, open source option
How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
Thanks, I was unaware of that. After some research, though, it seems screen capturing by Claude Code is something you need to activate yourself.
>How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
Prompts: not much. If I feel my prompt is becoming a paragraph-long text, that's when I know I probably haven't thought things through enough yet. Documents: I typically have a interpreter console open where I'll do something like File.write("foo.html", page.body), and I'll just tell Claude to "find the missing div in foo.html". Or if its a file already in the repo, I won't need to copy paste anything. Context: also not much. It has all the context by default, doesn't it?