How to translate to PM speak and back - Katie Bauer

Transcript generated with OpenAI Whisper large-v2.

I'm Katie Bauer. I am currently the head of data at Gloss Genius, a company that is used by independent owner operators of beauty salons to run their businesses. Prior to that, I have run and built teams at companies like Twitter and Reddit, and I've also spent a lot of time attempting to communicate with PMs and sometimes even doing it successfully. That's kind of what we're going to get into here today. An original concept I had for this talk, by the way, was to conceptualize PMs as some kind of weird and vaguely hostile alien species. I could not think of how to extend that metaphor for 20 whole minutes, so unfortunately all of the Dali many generations that I created for it will go unused. Perhaps they can be released at another time. Instead of talking about PMs as aliens, I'm instead going to begin with a story, kind of a personal experience of my own, beginning many years ago. Originally, for a lot of my early career, I worked in sort of a help desk situation. Many of you are probably familiar with this kind of format for running a data team. You kind of get low-context requests from people all across the business, perhaps through JIRA, perhaps through some other ticketing system. This was really how I ended up collaborating with stakeholders for the first couple of years that I worked in data. It was fine, but I very rarely had context for what I was being asked to do. It was a lot of just go and execute. About the time that I was entering mid-career, I had gotten a new job where I officially had the title data scientist. I was very excited about this, and I was moving into more of an embedded structure. By that, I mean I was someone who, as a data scientist, sat on a product team. I attended their stand-ups. I would get tagged in on PRs and sometimes review code, and I was expected to have opinions about how the product should be built. This was very exciting to me. It was something that felt like science, as opposed to just kind of being asked to pull data for people. This was also notably the first time I worked closely with PMs. One of the most exciting things to me in particular about this role was that the company was starting to run A-B tests. When I say they were starting to run A-B tests, I really mean they were really, really, really just starting. They had a way of randomizing users into experiment buckets, but they didn't have really anything else. That meant that when they were trying to run an experiment and they weren't sure what to do, the answer was usually, ask Katie. For me, this felt like a great opportunity to have impact. An A-B test is an opportunity for me to use real statistics and do something that really, truly felt like science. I got very excited and I got very into the whole idea that I was going to help them run rigorous experiments. I came up with templates. I would ask them to pre-register their hypotheses. We would calculate rigorous sample sizes to make sure that we were really achieving statistical significance on all the important metrics. I even started developing data pipelines to pre-calculate experimental metrics that had an official name, the standard battery of metrics. So important, so official. I would get requests from the product team to go and calculate and help design an experiment. I would come back with extremely specific answers. I would come back and say, oh, well, you need exactly 11,567 users for this to work. They didn't like it. They were so overwhelmed by the level of detail that I was giving them that I may have been speaking a different language. It ended up being kind of this weird pattern where they would ask me to do something. I would run off and go try to get a really precise answer, design a really robust experiment. And then their response was, well, let's just keep things simple and use 5% of traffic. And this was extremely frustrating for me. I'm a data scientist. I have a deep capacity for being pedantic. And I wanted to put that pedantry to a productive end. But for the most part, I was just really not doing the right type of work to work well with these PMs. They would ask me to do something. I would go run off, try to get something very precise. And by the time I finished, I would come back and they had moved on. Or they would only use what I gave them about half of the time. And it ended up putting me in just kind of a really bad relationship with a lot of the PMs that I worked with. And it continued to be this way for many years. I just didn't understand what they were looking for. I could not comprehend the reasons why they would ask me to do something and then not use what I did. I could be doing so many other things with my time and with the company's time that it felt like a waste. And I started actually having a somewhat adversarial relationship with product managers that I worked with. Requests felt antagonistic. It felt like they were trying to get things out of me or work around me. And I just could not comprehend how to talk to them in a way that they would understand. Eventually, I did have a mentor pull me aside and give me a piece of advice that once I heard it the first time, I started hearing it all over the place. And that was that I should assume good intent. For those of you who haven't heard this advice, it's very good advice. It's basically thinking about the people that you work with and assuming that you are on the same team. Like you might disagree on particular things. You may not work at the same pace, but you should assume that they are trying to do the right thing, that they are trying to do the same thing as you. They're not trying to intentionally cut you out of the loop or do anything like that. And while it did not necessarily help me understand PMs better, it did help me start improving my relationship with a bunch of them. And over time, things became more cordial and more civil. But it was not really the end of the story for me. Things didn't really change for me in my relationships with PMs until I ended up working with a PM whose team was in trouble. And I don't think he would characterize his team as having been in trouble. But this product team was very much like the defense against the dark arts position in the Harry Potter books where there's always a new professor every year. It's kind of cursed for whatever reason. Something bad always happens to the people who run that department, and something always happened to people who led this product team. And there's often churn in their engineering and design leadership. And this product manager had just come into a really bad situation. He'd had departure of other leadership that he was working with. And as the data science lead working with him, I was the closest thing he had to appear. And it meant that we ended up talking all of the time. And we both felt pretty stressed about the situation we were in. We were really behind on a goal that had been set earlier in the year before either of us joined. And we didn't really feel like anyone understood how well this project or product worked. So most of our conversations were just brainstorming and trying to really understand it, iterate, and get to a place where we could tell a better story about how we were turning the product around. And the fact that I spent so much time talking to this PM actually ended up helping me get into his frame of mind and understand something different about him, which was not just his good intent, but also his incentives, which improved my relationship substantially. We talked all of the time. And eventually, I began to understand that for him, one of the biggest pressures he was under was that the head of product wanted to cut this product. He thought it was doomed to fail. And this product manager that I was partnered with, the thing that he wanted more than anything else was to prove that he was in control and could turn things around. And it led to me eventually amending that advice that I had received earlier in my career, which was not just to assume good intent, but consider incentives if you want to communicate with people better. It ultimately turned me into a place where I realized that if I wanted to speak the language of incentive and work better with PMs, I needed to realize that I had been really overly oriented on details, kind of missing the big picture, and learn that product managers frequently speak a language of progress. This image on the right, I couldn't find a good picture for progress. So you get this transcendent moment in time. So if you want to speak in terms of progress and work better with PMs, I have a couple of pieces of advice about how you can think about doing that. The first is to remember that most questions that PMs ask you are implicitly causal. It doesn't even necessarily have to be in the context of experimentation. PMs are always concerned about figuring out what buttons they can push and what happens when they do it. So when a PM asks you to do something for them, whether it's analysis, design an experiment, whatever you end up doing in partnership with them, the number one thing that you should remember is to focus your work on things that they can control. If they want to know how something works, the reason that they're asking you for that is because they want to change it and try and hit a goal or achieve some kind of outcome for the product that they're building. So you should try to do analysis specifically on those things. If you include exogenous factors or macroeconomic conditions, it's only to count them out, and you should really start telling stories around things that happen when they change things they can control. Similarly, you should prioritize logical consistency in your analysis over being technically correct. You don't necessarily need to get them the most precise answer. You need to get something that is in line with actions they would take, with the company's values, with the company's roadmap. Those are things that are actually going to help you say things that PMs will be able to contextualize and use and increase the chances that they will listen to you. And then a final way to kind of put a ribbon around this paradigm or this way of thinking that a lot of questions are implicitly causal and about actions that you would take is to describe your results specifically in terms of inputs and outputs. Regression coefficients are an interesting example of this. Talking about how if you increase one of the input variables by a certain amount, what likelihood that is about to have on the output variable, that's a way of turning something into a story. So if we say, for example, every one additional dollar we spend on marketing spend, according to a regression coefficient or an experiment coefficient, that would result in 10 additional sales. Something like that. Giving them a good example of an input and the output that they would be likely to receive on it is a great way to increase the chance that they will hear what you're saying and take it into account and be influenced by it. A second step is to plan your work according to the positioning of the PM. And by this, I really mean their positioning in the product organization. Product organizations, from the outside, it can be kind of hard to understand that they are all sort of in competition with each other, mostly for resources. They all report to the same people. They all end up getting things from the same budget. And it might not explicitly be cutthroat, but it is important for them to look good in front of their peers and to their bosses. And if you want to build alliances with product managers, you need to think about how you can help them tell stories associated with their products, specifically based on what kind of product that they are working on. In my earlier example, the product manager I was working with was working on a struggling product. And one of the things that we ended up doing was quantifying the type of loss that we were seeing for the company and actually figuring out what we could do based on things that we could control to try and staunch the bleeding. Telling that story helped us get buy-in from the head of product to better resource us and change the way that we were perceived at the company. Things might be different if you're working on a successful product. Hopefully, it's easier. There are a lot of different ways to tell good stories about successful products. Maybe it's talking about the quality of the users who are already using them and the good things that they do for the site. It's trying to find complementary opportunities related to that existing successful product and trying to position the product that you're already working on as key to moving towards those complementary opportunities. It's all about trying to make sure you figure out ways to expand the success. And as you're doing analysis and modeling and designing experiments, try to think about how you can position your work towards telling a story of success. Emerging products are probably the bread and butter, and there's way more of a playbook of how to support and tell good stories about them as data scientists. It's really all about product market fit. So for users that you are acquiring, like how long did they stick around? How much did it cost to acquire them? Helping product managers tell those stories when you're working on an emerging and new product, that is a great way to help gain their trust and communicate better with them. The final thing to know when trying to communicate with PMs is that in many cases, there is no perfect translation. And I mean this partly in a way where there's no English equivalent of the word hygge. There's no PM equivalent of the word statistical significance. Hygge is like an idea of like coziness or compiness from Danish culture. Like there's a lot of cultural baggage wrapped up into it that can't be easily communicated in one word, much in the same way that there is a bunch of technical knowledge and cultural like field practices, like wrapped up in statistical significance. And it is not important for PMs to understand this. It's something that you can still communicate well with them, even if they don't understand all the technical details. And this can be a really hard thing to balance when your job is to add rigor to a situation. So if it's helpful, think about some of your recommendations sometimes as being more guardrails. For example, on the opening anecdote that I shared about trying to calculate precise sample sizes, the PM ultimately did listen to my advice, even if they did not listen to it in a very specific way. Because the 5% of traffic was enough that we were actually able to meet the sample size that I recommended. And I was frustrated that they hadn't specifically listened to me, but they still got the gist in a way that helped them make a good choice and ultimately reach an outcome that was rigorous enough for what we were trying to do at that time. Really, what you need to do is make sure that you are careful to pick your battles of where you want to introduce rigor and insist on accurate translations when stakes are high. If the cost of a choice is very high when you're wrong, if it's really hard to reverse the choice that you're about to make, then that's a good time to apply a lot of rigor and move carefully and be slower. But you don't always have to do that. Sometimes a Google Translate level translation is all that you need to communicate well with a product manager, and that is okay. They don't have to always understand the specific details that you're getting into. So if you take time to carefully consider these things that we've talked about, you don't have to have an adversarial relationship with a product manager. In fact, you can build strong friendships with them, even if they're a little different from us and maybe can frustrate us sometimes. The alliances between data science and product managers are some of the strongest that I've ever seen, and some of my best working relationships have actually been with product managers that I've worked closely with. So with that, I will call it and be ready to take questions.