Tracer bullets + working backwards: simple framework for solving problems - Caitlin Hudon

Transcript generated with OpenAI Whisper large-v2.

So in the spirit of NormCop, I thought that we could kick off with a good vintage tweet, which happens to be a favorite of mine. A wise man once pointed out that when you find yourself doing the same thing multiple times, you should move toward automation. I love Dave's second point here, which is when you find yourself answering the same questions, you should consider automating your advice. Today I want to talk about frameworks for efficiently conquering the unknowns that stand between us and getting things to production. So these are things that I've seen happen over and over again, and I'm trying to come up with some ways to combat some of the unknowns as we're starting new projects. For example, taking a machine learning model from idea to production often involves similar steps. So the data and the specific application might vary from project to project. But I know that at some point I'll have to get out, get some data, spin up a notebook and build something. And I can't fully automate that process, but I can find patterns in some of the thinking that drives each step and try to get more efficient that way by focusing on some of the patterns I'm seeing. The best way I've found to automate things that don't fit into a function or a package is via frameworks. And so that's why I want to talk about frameworks today. Frameworks are especially helpful for automating workflows around organizing my thinking and working through problems. They're boring, but they work really well and they make my life easier because I don't have to remember things. And so I feel like they're very norm-conf. I'm at Figma now, but previously worked at Online Med Ed, which is a medical education company, and they created hundreds of online videos on various topics to advance a flipped classroom model. In order to scale, we had to match our curriculum to that of med schools and governing bodies to make sure that our material covered all of the same points that they expected us to cover. This was a really fun problem because as Tolstoy foretold, each unique syllabus is unique in its own way, which in this case means that lessons are named very differently. They're parceled out differently in terms of how they're covered across different syllabi, and they definitely didn't match the lesson names that we had in our training data. So the manual version of matching syllabi with our curriculum involved a very knowledgeable medical subject matter expert reading a lesson and some of its key points on the syllabus, then searching on our site, which included keywords but not like the full text of all of the videos, and scrubbing through those videos that matched to see if it was really a match and how fully it covered each of the lesson sub points. You can imagine this is not ideal, and so we thought that maybe automating this matching process via machine learning would be really helpful. So here we've outlined this tool that would give the person the top end lessons to check first based on looking at the full content of each of the videos. And so this would significantly improve our ability to scale the process. And so this is what we proposed. This was the first machine learning model that we built at Online Med Ed, so there was a ton of uncertainty around the project. The biggest open question being like not knowing whether the type of classification we wanted to do would even work with our data, like was our data big enough and unique enough at the lesson level to actually do classification was a very open, important question. At the time, we were also doing a big rewrite of our code base, so there were also questions around where the model would eventually live and dependencies and whether we would be able to roll it out. Anything that I could do to reduce uncertainty, both for stakeholders and for myself, was really helpful here. So I wrote a lot of documentation around like inputs and outputs and what those would look like, what the architecture could look like, in order to build buy-in. Once we had buy-in, we decided that the best course of action would be to build a proof of concept. And so I decided to try a couple of frameworks out in that process to see if we could do it like quickly than I had, more quickly than I had moved on previous projects. So with any project model code base, you have a couple of kinds of unknowns. Known unknowns are the things that you know you don't know. If you started a list of things you don't know, you could slot those things into the list. These are more manageable than the other kind because being aware of these gaps means that you can plan for them. In machine learning, this could be knowing that you don't know which type of algorithm you're going to apply or knowing that you'll have to apply data transformations, but not being sure which ones. Unknown unknowns are the things that you don't know you don't know. So it would be impossible to list those things because your unknown unknowns are by definition unknown to you. You learn about these things as you're working through problems. And so a lot of them just kind of come up in hindsight. These are things that you didn't know. In a machine learning context, these could be things like dependencies that you definitely need to consider as you're implementing something, but you weren't aware of ahead of time. Today we'll focus on two frameworks that help speed up the process of getting ML to production or analyses or dashboards or new features. You can kind of insert your own thing here by efficiently conquering unknowns. So we'll start by tackling the known unknowns. How do we capture all of those things that we don't know and use that data to be more efficient? So we can borrow an idea from professional poker player and master decision maker, Annie Duke. Annie suggests that we start with a pre-mortem. So that's what we'll do. In order to tackle unknown unknowns, we want to make sure that we're not going to like trip over them. And so we'll dive into the second framework here, which is tracer bullets. We'll talk more about that in a minute. So we'll start with pre-mortems. What is a pre-mortem? A pre-mortem is an opportunity to predict possible outcomes for a project. Practically speaking, I think like the benefit you get here is a list of known reasons that a project could fail. This is a really good time to put all of the known unknowns into a list and figure out if anyone might be able to help you with answering some of those questions. If there are doom factors that would render a project like a total failure, missing a key deadline, for example, this is a really good place to discuss those things. I also want to note that I'm kind of framing this as a team exercise, but I found it really helpful to do pre-mortems, even as a solo exercise as I'm starting projects. I've also found that interviewing subject matter experts and using that as documentation is juice that's very worth the squeeze. So I'll often include pre-mortem-y questions about like where they think there might be areas that we need to be really careful about as I'm working with subject matter experts. And taking these things together and like building this list is a really good way to take all of the strands of the things that you know and like braid them into a constellation that will guide the work. So the idea of a pre-mortem is pretty simple. I wanted to show an example. This is Annie Duke's recommended format. She also adds sections around luck and skill. Coming from poker, she has this really interesting take on decision making and especially in her world you get feedback really quickly. So she's gotten really good at increasing the quality of her decisions while making really high quantities of decisions. And so I do like that framing. I didn't include it here, but you could. If you're thinking that this looks like a prioritization exercise, especially because of that column on the right, I would agree. Besides like getting the concerns into one place, it also gives you an idea of which places you might want to focus on first. I love a pre-mortem because I'm very good at guessing how things might fail and this is a rare opportunity to use that talent in a safe place where it is advancing the discussion. So they give people the opportunity to voice concerns without detracting. And I think that's super important because concerns are typically evidence of past experiences and learning from experience is one of the best ways for us to be better and more efficient in our work. So I love writing a pre-mortem. Now that we've accounted for known unknowns through the pre-mortem, we're aware of some things that could go wrong and we know where we might get tripped up. So armed with that knowledge, we can start building. To do that, we can move to our second framework, Tracer Bullets, for efficiently dealing with unknown knowns. So last year, our engineering department read The Pragmatic Programmer as a book club. This is a collection chock full of practical advice built on years of experience and it's solidified itself as an industry classic. I feel like it's very norm-comp. The stuff in there is not necessarily groundbreaking, but it's a really good read. It's something I wish I had read a lot sooner. Anyway, I loved it. I highly recommend it. And I've since adapted some of the ideas there for my own machine learning projects. And so specifically today, I want to talk about the idea of Tracer Bullets, which come from this book. So outside of a code base, in real life, Tracer Bullets are bullets that are loaded at intervals alongside regular ammunition. When they're fired, phosphorus ignites and leaves a pyrotechnic trail from the gun to whatever they hit. So you can see the path that each bullet takes and how close it gets to a given target. These are particularly useful for quickly refining your aim because they provide real-time feedback under actual conditions. So you're shooting these alongside anything else that you would be shooting. When you're building something that hasn't been built before, Tracer Bullets can visually illustrate the need for immediate feedback under actual conditions with a moving goal. So what does that look like in a code base? This is the visual that's in the Pragmatic Programmer. So each bar here represents a layer of logic, and you can see the Tracer Bullet goes through each layer of logic. When it intersects each layer, it carves out a small end-to-end path through all of the layers. The goal is to get through each layer with the absolute minimum amount of logic required before moving to the next one. So if we think like Peter in his earlier talk, we can say that we're building a bridge that's cheap enough that it barely stands, but it does stand. Tracer Bullet code operates in the same environment and under the same constraints that actual code would. And the focus on using Tracer Bullets in development is to get to the target quickly for more immediate feedback. Because Tracer Bullets don't require building a full project, they're a relatively cheap way to test an implementation. And so that's why we wanted to test them with the project I was working on. So I've translated this a bit to Tracer Bullets in machine learning. So we might start with a data pipeline, actually getting the data that we need to use, and then work through feature extraction. How do we make sure that we can get the data to a place where we can put it into a model or algo? And then we actually need to build the model or algo and deploy it, start to apply. In this case, it was doing classifications, so starting to actually do the classification on data and put it into an API or somewhere you can access that and then put like a front end on it. So how are we going to make sure that we're getting this stuff into the hands of users or whatever system is going to use the outputs? A couple of additional notes on Tracer Bullets and like how they're different from prototypes, because this is something I didn't understand until I read it a few times in the book. With a prototype, you're exploring specific aspects of a final system. And so you typically will throw that code away and recode properly once the lessons are learned. So if you've seen something like a design sprint, you kind of want to have something that you can show to stakeholders by the end of the week, and you might not be actually coding a whole thing. It's really just important that they can see like the UI or the front end or that last step. This is about answering questions along the way with the implementation. So you're doing like the minimum implementation possible so that you can see how your entire system works end to end. So Tracer code is lean, but it's complete. And it forms the parts of the skeleton of the final system. Once you have the skeleton built, I could choose to go work on like the tail for a while or the head for a while or flesh out the middle. But the entire system will continue to work with the Tracer code framework. So, this is where I needed to aim my Tracer Bullet. This is a quick architecture diagram from an early stage of the process and kind of what we were doing. So we have ETL logic, we have SRT files, these are like text files that are taking everything that's being said in a video and transcribing it. So that's all the text data that we're working with. We need to build logic to process that data and then to train the classifier. We need to actually get a classifier that works and is good enough for our use case. That was one of those open questions that we had. And then we need to be able to output the closest lesson. And so this is the example of the stuff that we wanted to deliver. A lot of this lived in a Jupyter notebook. We actually decided to put like a front end on it so that people could interact with it. But I think having this clear path for like, I need my Tracer Bullet to go through every single one of these layers was really helpful to answer questions that we had around architecture and like, would this thing even work? So firing the Tracer Bullet, we built the data pipeline, did feature extraction, built the classifier, started doing classifications, made sure all of that stuff was up to spec, and then added the front end. But something important here is like what's left after the Tracer Bullet. So if you look at like the pink area, that would be the part that was included or the part that was coded out as part of the Tracer Bullet. There is lots left to do at each stage here. So for example, the data pipeline, I didn't grab all of the videos or the files containing like the text transcriptions of the videos. Some of those were still left on the table. They weren't necessary for the implementation. Feature extraction, some of the work that we were doing around like tokenizing, stemming, etc. We could change in future iterations, but we had like enough to build a model that was a good proof of concept. The machine learning code, we had a good idea for a model that worked. Obviously, that's something we could go back and update, but having it as part of the pipeline was really important. And then the API and the user interface, we were able to get feedback on. And so if there's anything that we needed to change, we'd be able to do that pretty easily. And so this was really helpful to have an idea of just like we're leaving some things on the table to come back and do. But also, this doesn't mean that we're waiting, for example, for the entire data pipeline to be built before we're moving on to the next thing. So the advantage is that you can get to delivering something or shipping something faster, which is really helpful for stakeholder management too, especially if you're working with something where there's a lot of open questions. I built this slide about tracer bullet driven development. And so this would be like, let's say you had five checkpoints and you were working as part of a team. If you look at this first approach, this is if we're building sequentially. So we start with like the data sets and then the feature engineering and then the modeling and then serving the model and then having a front end. That takes a long time before we're actually seeing results and having an idea for whether the thing works or not. But if we take that same amount of work, if we assume it takes the same amount of work at each checkpoint, we can see that we're delivering like a V1, the first checkpoint, and then V2 and then V3 and V4 and V5. And so from a stakeholder perspective, this was really helpful to be able to say like, oh, look, we have this thing working end to end. We're going to improve the accuracy. That's no problem. It's just updating information and like steps one, two and three. So I really liked using the tracer bullets, especially for working with stakeholders and reducing uncertainty for them. And the other nice thing about using these two frameworks together is so the known unknowns, once you have those and you know the areas that could trip you up, you can build your system around them and make sure that they're included as part of the layers for the tracer bullets. So you kind of know based on those, like what are the layers you need to fire the tracer bullet through? And then you're able to fire the tracer bullet, make sure that you're accounting for at least everything you know about. That gives you the opportunity to find things that you didn't know about because you're actually building it. That always happens. You're always going to have surprises. But it gets you from the start of the project to the end of the project a lot faster. And with some check in points where you can offload information or share information. And I found that really helpful and really exciting. It's something I want to try more and more with future projects. And so I wanted to say thank you so much to the NormConf organizers for getting everyone together. This has been great. I've been watching all morning and loving it. For anyone who wants to learn more about these topics, there's a couple of books that I recommend. So I talked earlier about the Pragmatic Programmer. That's a classic. It's really good. Annie Dupes' Thinking in Bets is also fantastic. She has a couple of other frameworks that I've used. Things like backcasting. So if you have an idea of what success looks like, similar to like Amazon's PR approach, where they start with like, what does the press release look like? And then you work backwards. She basically applies the premortem but to success. So if we're successful, what are the points that we have to hit along the way? So those books are great. And if you want to learn more from me, I'm at Bianna Poesy on Twitter and And I also have a blog at