英文带时间戳转写

[0:03 - 0:46] Hi, my name is Lauren. You might know me as Potato on X, and I work on Gropout at SpaceX AI. So last month I did something pretty crazy. I shipped 2,000 pull requests to production. A lot of how I'm able to do this is through trust, and I think a lot about trust in terms of how I can trust my agents to produce high quality work even when I'm not there. And my argument and thesis for this talk today is that if you set up your environment for your agents really really well, You can end up with something that looks more like a personal or even team software factory, where you're producing very high quality code at much greater rates than before. But I'm not a fan of the term software factory. I like the analogy of a Michelin kitchen better, where, you know, as technologists,

[0:46 - 1:31] we're not really producing, we're not mass producing a product on an assembly line, but the work that we do looks very creative. It's very, it was the act of building a product and it's art in some sense. So, you know, even though with agents, we're not cooking the individual components that go into the product anymore, we're still responsible for the final outcome and thinking about our kitchen setup, right? Because depending on how you set up your line cooks, your sous chefs, you know, the kind of equipment they have, the kind of training they have, you know, dishwashers and the ratio, I guess, of line cooks to dishwashers, all of these ingredients go into making the final product and I think this analogy is really up. So I wanted to talk a little bit about a story before we begin, where six months ago, when I first joined Cursor,

[1:31 - 2:15] before we were a part of SpaceX AI, I obviously had no agent skills to use, right? I just joined the company, it was a fresh code base and a fresh product that I was working on. So at the time, Cursor was building the replacement for the Cursor IDE, which is the new agents window. And before I joined, the Cursor agent window had quite a lot of performance issues. And my manager at the time asked if I was able to help them. And since I had spent some time on the React team before I joined Cursor, it seemed like a good fit. But when I first started, I quickly realized how manual this process was. Obviously, I have done performance work before, but at the rate at which pull requests were being landed, it felt almost an insurmountable wall of pull requests that just kept coming in. And I had no idea whether or not

[2:15 - 2:58] the performance of the app would be regressing. So a lot of my early time on the Cursor team was spent looking at Chrome DevTools and doing performance traces and taking keep snapshots, and it was extremely manual. It was so manual, it got to the point where I just got really frustrated and started to think about, wait, we have agents, what am I doing? And so I started thinking about verification skills, where what if my agent could actually run the application itself and take the traces for me, understand the traces and find the hotspots and basically he'll climb to better performance in our application automatically. And throughout the last six months of being a cursor in SpaceX AI, you can really see that my productivity has skyrocketed.

[2:58 - 3:43] And I never set out to ship 2000 pull requests a month. That was not a goal of mine at all. But I realized that all of the skills, all the tools and code-based changes I were making laddered up to this idea of trust. I didn't know it at the time, but I had this thought in the back of my head, head which was, I am the bottleneck and I need to be able to take all the knowledge that I have as an engineer and impart them into my team of agents so that I didn't need to be the blocker for everything. and you can clearly see that it's paid off. So I think it really comes down to trust, but how exactly do you build that trust? And where do you start if you're, you know, wherever you are in your journey of using agents? So when I started, obviously I was in this category,

[3:43 - 4:28] you know, the one to one to five range, where you still feel like you have to babysit every chat and conversation and you're just constantly course correcting, you know, you're intervening, you're correcting your agent so that it does the right thing. And if you're not there, basically nothing happens, and the agents do the wrong thing. Now I would actually argue that this phase of using agents is actually the hardest to get out of, because it's not always very clear how exactly you get out of it. And again, it just comes down to trust. The reason you're unable to go from one or one to five agents to something more, like 100, is because you don't have trust in your agent's work yet. So if you don't have trust and you try to spawn 100 sub-agents or client agents, you're gonna quickly find that you're just gonna get a ton of slot pull requests and a bunch of regressions

[4:28 - 5:11] and a bunch of bugs shipped and no one's gonna be very happy with them. So that begs the question, how do you trust your agents more? So for me, it really started when I, like I said, I joined the cursor team and I was starting to work on performance where I realized the need for verification. So when I say verification, there are sort of levels to that because on the, I guess, the lower end of the scale for verification, You have things like the verification skills that I talk about where you teach your agent how to run your application, you know, and use things like the Chrome DevTools protocol or whatever other protocol that you have for debugging and teach them how to, you know, debug the application, take performance traces, take heat snapshots and so on.

[5:11 - 5:55] And then on the opposite end of spectrum, or de-spectrum, which is much, much harder and so very much an open question is like more formal verification where, you know, maybe you rely on our formal methods or languages like Lean or TLA Plus to do verification so that you can check that your business level or business logic invariants are always true and that you can formally verify that your application is always in the correct state. But I would say that even if you don't have the ability to run formal methods and very few people really do, that with verification skills, you can get very far. So when I joined Cursor and started to work on the Cursor agents window, The first skill that I built was a skill called Control Glass, which is a verification skill that teaches the agent

[5:55 - 6:39] how to run the application and take traces, like I mentioned, and it does that through the Chrome DevTool protocol. Something interesting about that skill is, and I kind of iterated my way to this. It didn't start out this way, but the control or verification skill really has two components to it. The first part is obviously a CLI. So you want your agents to be able to reproducibly be able to run the application and collect traces and collect empirical evidence that code is working, that your performance bar is being met, and rather than have your agents create scripts every time that can differ between agent sessions, you can actually create a CLI that's within the skill directory, and then your agents

[6:39 - 7:24] will just use that every time. And then, of course, you need to actually invest in it and make it good so that it can handle all sorts of different use cases and be able to run the application correctly. Another thing that's also really important is this idea of a feature map. So a feature map really is something that I kind of coined, I guess, where we started using these control skills within cursor. Then we quickly realized that a Slack report would come in and a user would post a very big screenshot, right? Like a very small slice of the UI and just like three question marks. And the agents using our control skills are just no idea. Like it could run the application, but it would just be guessing at what exactly the user meant. And so I had this idea to create something called a feature map, which is, I guess, kind of inspired by a site map.

[7:24 - 8:07] It essentially is a form of materialized memory, like how exactly does your application work, what features does it have, how does a user reach it, in terms of keyboard shortcuts or what DOM elements to click on, things like that, and what all the different features do. And this feature map is stored into skill itself in the code base as part of the skill directory. And we have an automation that maintains this feature map as well. But when we combine the CLI and the feature map, we quickly realized that this combination was very, very powerful because now agents can not only reproducibly control the application and take traces, but it could also understand requests that came in from internal users as well as external users. And so we very quickly realized that the control

[8:07 - 8:52] verification skills were so useful that they've become more or less a critical infrastructure for our team, and we constantly maintain it. But the ability for an agent to verify its own work is extremely powerful. And we have spent a lot of time on this skill and it's very, very powerful for building that trust. In addition to verification, of course, you want, because verification is really about correctness. Correctness to me is really about does the thing, does the feature or code do the thing that you want it to do? Right, like does the check out button, does it actually check out the card? Verification is really important for that because you get empirical evidence that this feature actually works. But it doesn't tell you much about the performance or the code quality of that feature. And that's where you start thinking about skills

[8:52 - 9:35] that teach agents to work like real software engineers. So I've built a plugin called PSAC. I'm not gonna talk about the plugin too much today, but a lot of the inspiration for that plugin, which is a collection of skills that I've created, are inspired by the kind of workflows that I personally have used in my time doing software engineering for all sorts of different types of tasks. So debugging, feature development, prototyping, there's a whole bunch of different playbooks and skills that ship in PSNAC that teach your agents how to write code the way that you want them to. And this is where the more experienced engineers on your team can really contribute to set up a team repository of skills that just make your agents a lot smarter. And when you combine those skills

[9:35 - 10:19] with verification skills, then you're able to get to a point where your agents are able to not just verify that the work that you're doing is correct, but that it's also high quality. And again, because of verification, you can collect real performance metrics. You can get real numbers and statistics and telemetry on the performance of the application. So I think that's a really important part to invest in. Another thing I think is really important is refactoring and rewriting your architecture to be more agent-friendly. And I almost wanna say that this is one of the most important things you can do as a software engineering team, because if you really truly believe that agents are gonna be writing all the code in the future, then we need to design our code bases so that they do the right thing by default.

[10:19 - 11:04] And you'll find that there's this, I was gonna say like a scale or a continuum between these different pieces of building trust in your agents, where I have like five of these points here, where the code base is really like the best form of memory because agents love to extend existing patterns that they see. And I think this is just the nature of how LLMs work, where they're more likely to use whatever is in their complex window to make changes. And of course, the files that the agents read and opens are part of its context window. And therefore, the code base is a really important part because agents aren't going to just refactor your code in every single PR. They're going to just look at what's already there and just extend. The next level, I think, is about static analysis,

[11:04 - 11:45] where you have linters, you have compiler diagnostics, You have continuous integration. And these are guidelines and constraints that you can enforce in your code base so that whenever you correct your agent, you find that they just keep making the same mistake. You can add those as late rules, or even better, you can refactor your code base so that the mistake that the agent is making becomes categorically impossible. And then a step above that, right, and this is where we're starting to get more into less of like a hard constraint and enforcement and more into the realm of guidance. You have things like rules, you have bug bot, You have skills which your agents will obviously sometimes and mostly use when they're doing their work, but there's also a chance that it might, for various reasons, forget to read a rule

[11:45 - 12:30] or maybe the user that is piloting the agent ignores them. So these aren't quite as enforceable but also an important part of setting up your environment so that you can really trust what your agents are doing. And then finally, you have a style guide which is really only enforceable by humans in the code review. I guess you could put these in your rules and bug, bug and skills as well. But if you don't, then you have this big glaring hole in your review process. Or now humans have to look at every single line that's being changed and remember to comment. And with the rate of pull requests that are coming in, it just becomes impossible. So I definitely wouldn't recommend relying only on the style guide. I think the style guide or looking at human reviews is a good place to start in terms of what's missing. But you should really invest the time to think about the other four parts,

[12:30 - 13:14] your code base, making things categorically impossible through better data structures or algorithms, static analysis, and then, of course, you layer that with rules and bugbots and skills. On the code base front, and in the GROC code base, we actually have invested into setting something up that we call DUNE, which is our agent-friendly framework. So the inspiration for DUNE really came about from a lot of performance issues we were seeing in the cursor agent window. And so a lot of lessons came out of that inspiration, But the key principle that we landed on is really that agents love taking shortcuts. So what if we design a framework such that the shortcut, the easy path is the right path for agents? And also one that would be a code base

[13:14 - 13:58] that is maybe pretty annoying for humans to work in because it's so locked down in terms of what you can do and what you can't do. But it actually creates the perfect environment for agents, especially ones that have very minimal context because not every contributor to your code base is going to be an engineer anymore. You can have designers, you can have product product product managers, you can have CEOs, you know, going into the code base and checking features. So we wanna really think a lot about how we invest and set up our code bases so that, you know, even agents that are piloted by busy people with not a lot of context can do a good job by default. And like I mentioned before, your code base is really a form of memory for agents because they love to extend existing patterns that they see. And the reverse is actually also true, right? You can invest the time to set up your code base in a way that things are, you know, bad patterns are categorically impossible

[13:58 - 14:41] or you have like lit rules that prevent them. But the reverse is also true in the sense that if you have existing anti-patterns, you'll actually find that these will spread kind of like a virus where you have like one small workaround or a comment that explains the workaround. And you quickly find that agents just love to copy that. And then in a matter of a few days or a few weeks, you'll find that the workaround has spread everywhere and it has become like a de facto pattern for all agents. And that's a really, really bad place to be in. And that goes back to what I was saying about why it's really important, you know, whenever you're correcting your agents, that you invest the time into thinking about code-based changes and static analysis to, and of course, the layering them with rules, good rules and bug-bar and skills, because of that reason.

[14:41 - 15:24] And another analogy that I like in addition to the Michelin kitchen is this idea that, you know, your code base is kind of like a garden, where you have workarounds, you know, that seem kind of innocent at first, but then because of the nature of agents, you just copy that pattern over and over again, and you quickly end up with a very vibe-coded code base that is a pain in the butt to maintain and has a lot of performance issues. So in my opinion, the perfect agent code base is one that's so locked down that, again, it's really annoying for humans to write code in, but it's so conventional, it's so standardized that even innocent-looking patterns are just broken. And the best example of this I have is actually something that seems very, very innocent when you look at it, but when you think about it, it's actually really bad.

[15:24 - 16:07] And that pattern is agents leaving comments in the code. Now, you know, when I first saw agents starting to do this, I initially wasn't, well, I did think that a lot of them were slot, but I also thought that, you know, it actually doesn't, it's not a bad thing, right? I guess if agents are leaving comments in the code, because as humans, we left comments in the code whenever we saw, you know, edge cases or we needed to actually make a workaround or, you know, leave a note to ourselves or a colleague on a particularly tricky part of the code base. But what I quickly realized when we saw this happening in the cursor codebase was that agents were just using the comments around the code as justification for why it wasn't going to solve the actual problem. And it said paper over it with a Band-Aid or a short-term solution. So in Dune, which is, again, the framework

[16:07 - 16:49] that powers GraphBot, we made the choice to actually ban comments for that reason, where so that agents would not just copy that pattern and propagate it everywhere in the codebase. So my pitch here is that every team really needs something that, you know, a role that I'm calling a gardener. In the same way that, you know, with a real garden, you need someone who is thinking a lot about the, you know, things that can kind of creep in and grow in ways that you don't want. Like, you know, you have weeds, you have, you know, just other types of organic growth. I don't really know gardening that well. But you have things that, you know, unwanted pests and stuff like that that kind of creep in empirical base. And so you want to nip them in the bud

[16:49 - 17:32] as soon as possible before they start propagating everywhere. A lot of the principles behind Dune are really centered around these three things. First of all, we want to delete tech depth that we already have for reasons I just mentioned. We want to keep or enforce a single paved path for most blessed patterns. There should be one conventional way to do some things so that agents don't really need to guess. And there should be enough guidance in the code base, in CI, in Lint rules so that the agent are guided to follow that path. And then finally, whenever you see tech debt or bad patterns, your instinct should be, I need to write a lit rule against it. You don't always have to clean up immediately because if you write a lit rule, you can at least stop the bleeding and which doesn't solve the problem entirely, but it at least prevents it from growing.

[17:32 - 18:16] So I definitely recommend really thinking a lot about how you can guard against anti-patterns so that they don't spread like a virus and then also spend time to actually clean up agents to clean them up so that your code base is just constantly kept in a state where you would be happy if an agent were to copy it. that that's the kind of mindset that I would recommend having. And then I won't actually go through all the details of doing it self, but I'll just kind of gloss through some interesting parts. So again, as a reminder, doing this, the architecture, the client framework that we built to power GraphBot, we've invested a lot into all the things I was saying, where we have conventions. We have a lot of conventions about where a code should live and where and how a code should be imported between them. So in doing applications, there's different concepts where like, for example, features are all co-located in a single folder.

[18:16 - 19:00] have an entry point and of course you have your client, which powers the overall application. And we have a lot of strict boundaries between these things, where just as an example, things that run on the main process, the main thread in Electron, aren't allowed to be run on the render thread. And we keep that separation very intentionally because of lessons we learned from Cursor's agent window, where we would sometimes see code accidentally get imported into the render thread and, you know, slow code. And since on the render thread you want your UI to be very smooth and performant you need

[19:00 - 19:45] to make sure that you don't have any long tasks or you know things that take longer than 16 milliseconds or if you want like 60 frames per second or 8 milliseconds if you want 120 frames per second and so your render has to be constantly in a state where it is it can really kind of chunk up the work and not do them all at once and so we have code within doing that enforces this these boundaries through the import and dependency graph but yeah That's just an example of a pattern that we saw lead to really bad performance that we categorically eliminated through the architecture of Dune. And then all of these other pieces aren't that interesting, but again, the core theme here, it's not about Dune, but the idea that an agent-friendly framework

[19:45 - 20:27] of your own is actually very, very powerful and you can encode all of the learnings that you and your best engineers on your team have tribal knowledge of. And I think the lesson here is that how do you take that away from what used to be in the style guide process of reviewing code and engineers reviewing other engineers' work and leaving comments to extract, almost like extracting that knowledge and encoding that into the framework in the code base itself, so that the code base access the memory, right? It's the thing, I keep coming back to this idea that the code base is just the thing that it's the materialized snapshot of the state in which you want your agents to extend and you want that code base to be so pristine, so great that the next agent that comes along is just very likely to continue that pattern and keep it really good.

[20:27 - 21:12] And if you spend enough time on this process, like I mentioned, you can really set up a Michelin kitchen or a software factory where because you've spent so much time on all of these pieces that allow you to trust your agents, whether it's in the code base, whether it's lint rules, whether it is diagnostics or rules or bug or skills, these layers come together and provide you a lot of trust Because now, just imagine from when you're working in the Rockbot code base. It's super locked down. It's almost impossible to write bad code. So even an agent with very little context, even an agent with not a lot of reasoning, can come in and actually write code that's good. And going back to my example about the Michelin Factory, I think there's a lot here where we're setting up

[21:12 - 21:55] our agents, our bots with skills and tools. We're training them. We're setting up our kitchen in a way that makes sense for the agents and bots to do the right thing by default. Whenever we see, for example, in the kitchen example, if we notice that one of our cooks or dishwashers is constantly tripping over something, of course we need to fix that, right? We need to problem solve and ensure that others don't trip as well, because in the kitchen it's a very dangerous place and you don't want to hurt yourself. It's the same mindset I think that we should have with our co-paces. How do we set it up so that even agents without a lot of knowledge can do a good job? And I think with RockBot, RockBot and Cursor play an interesting role together, where GropBot is really great at providing what I call the autoloop because you can connect

[21:55 - 22:39] GropBot to lots of different connectors like Slack, to Datadog, Century, PlanetScale, whatever services that you use, and you can aggregate all of that information together and use that to make really good decisions for itself. Some people call this like a company brain. I personally don't think you need anything that sophisticated here because agents are really good at using tools, and so if you connect these tools to GropBot and you start having your GROCKBOTs auto kickoff things like cloud agents, you can actually find that it's really not, you don't really have to invest in a lot of infrastructure to build a software factory. In fact, I'm gonna cross out this term because I don't like this term. I think you can set up this personal Michelin kitchen for yourself through GROCKBOT, things like GROCKBOT routines, which let you subscribe to Slack events,

[22:39 - 23:23] to Century Alerts that you kick off things automatically. And when you combine all of these things that I've been mentioning, your code base, your rules, your skills, they all compound. And GrapBot will be able to automatically respond to events that come from the outer loop and then kick off Cloud Agents. And you can also set up cursor automations and use our SDK to set up additional bots as well that reuse a lot of these pieces of agent infra that you've set up and allow them to do much more complicated tasks. So if you've done all this, then I think you can get to a point where I have some screenshots here of some of our automations in our agents in the network on cursor, where we are automatically reproducing bug reports,

[23:23 - 24:07] we're automatically opening pull requests, we are essentially adding a lot of value to the entire team because all of these things compound. So if we kind of zoom out again and go back to this graph, I think that to kind of close off the talk, if you spend a lot of time thinking about all of the pieces that you need to be able to ascend the trust graph, you start getting to a place where you can really trust your agents more and parallelize your work and also empower your entire team to build on top of these pieces of infrastructure for your agents and empower everyone, every engineer on your team, every builder to be extremely productive and to be able to write high quality code. So the last thing I wanna leave you with is actually this piece.

[24:07 - 24:51] Sorry, not that piece, but this piece. I think if there's only one thing you take away from my talk, it should be this slide here, which is that these are the activities that will help you build up towards a high trust environment, you know, whenever you find yourself correcting and interviewing your agent, you really wanna think about it from these five pieces. And where is the most effective step in this sequence in order to make your agent much more trustworthy? And of course, I definitely recommend thinking about it in this order, where you either invest the time to make that pattern categorically impossible through your code base and architecture and data structures, or you start looking at things like static analysis and then you layer that on with rules and bogot and skills. If you do all of that, and you also spend some time

[24:51 - 25:23] thinking about your code quality in terms of skills, you get to a place where you trust the environment so much that your agents can just be free, right? And personally, I have spent a lot of time for this for GorgBot's code base, for example, and this is really the secret, right? Well, it's not really a secret, it's a lot of hard work, but I hope you found this talk useful And please reach out to me on X. My handle is potato with an E. And I hope that you'll have a lot of fun and successfully in your own mission in the kitchen. Thanks for watching.