So there’s an interesting question of “how broad was the business context that led to a line of code or a TODO.” And this can vary between businesses! Sometimes it’s sufficient to have a comment link back to a well maintained story tracker etc., or a motivating slack thread, which a modern LLM is incentivized to follow. But sometimes that motivating example is just part of a broad exploration of numerous experiences and insights, which might have gone through many colleagues exploring the space over time. A future iteration would over-index on the one example. We see this a lot in our curated marketplace, where a fix to an integration exists in the context of dozens of times listings were manually patched or overridden beforehand. And if I were fixing that fix, would I find everything that motivated the old code? It’s like Chesterton’s Fence meets the Dark Forest. I wouldn’t give up on your product just yet!
I concur that it's probably negative value for SWE, but it is *indispensable* for research work. Often in research you need to try something, step back and try again, etc. Having that "exploratory but not permanent" work done witch accompanying logic has saved my bacon on multiple occasions.
You manage that memory pretty tightly though, based on the researcher skillset (iirc). I'm talking less about storing in progress work and more about carrying around random notes across research projects (even if they are wrong)
yeah, it's tightly organized with a heirarchy of files, and the agents don't read all those transcripts as a matter of course. But every convo gets saved as a summary, and every implementation plan references the convo. Mostly it doesn't matter, the commit goes in at the end and the result (either a piece of analysis code, or a report) gets saved as well. But when I **do** have a question about "why the hell did I do this?" having those links back to the original decisions lets me recall the thought process in minutes, rather than having painfully reconstruct.
To be fair "Why the hell did I do this?" is a very, VERY common question in research, and not nearly as common when using SWE tooling like JIRA
I've tried having the AI write hints files containing discussion of why code is a certain way, lessons learned, and important considerations. It doesn't help. It turns out, there is no substitute for actually learning.
So there’s an interesting question of “how broad was the business context that led to a line of code or a TODO.” And this can vary between businesses! Sometimes it’s sufficient to have a comment link back to a well maintained story tracker etc., or a motivating slack thread, which a modern LLM is incentivized to follow. But sometimes that motivating example is just part of a broad exploration of numerous experiences and insights, which might have gone through many colleagues exploring the space over time. A future iteration would over-index on the one example. We see this a lot in our curated marketplace, where a fix to an integration exists in the context of dozens of times listings were manually patched or overridden beforehand. And if I were fixing that fix, would I find everything that motivated the old code? It’s like Chesterton’s Fence meets the Dark Forest. I wouldn’t give up on your product just yet!
In this way it is most human.
I concur that it's probably negative value for SWE, but it is *indispensable* for research work. Often in research you need to try something, step back and try again, etc. Having that "exploratory but not permanent" work done witch accompanying logic has saved my bacon on multiple occasions.
You manage that memory pretty tightly though, based on the researcher skillset (iirc). I'm talking less about storing in progress work and more about carrying around random notes across research projects (even if they are wrong)
yeah, it's tightly organized with a heirarchy of files, and the agents don't read all those transcripts as a matter of course. But every convo gets saved as a summary, and every implementation plan references the convo. Mostly it doesn't matter, the commit goes in at the end and the result (either a piece of analysis code, or a report) gets saved as well. But when I **do** have a question about "why the hell did I do this?" having those links back to the original decisions lets me recall the thought process in minutes, rather than having painfully reconstruct.
To be fair "Why the hell did I do this?" is a very, VERY common question in research, and not nearly as common when using SWE tooling like JIRA
I like the idea of storing the transcripts for the human way more than I do for the agent 😂
I've tried having the AI write hints files containing discussion of why code is a certain way, lessons learned, and important considerations. It doesn't help. It turns out, there is no substitute for actually learning.