The Experimentation Edge - Nick Beyler
===
[00:00:00]
Nick GUEST: The holy grail for me of experimentation would be a North Star metric that accurately and reliably predicts long-term value. Because obviously as veteran experimenters know, we can measure the short term, but we can only predict the long term. So what actions and behaviors of our clients in the short term can help predict whether, they are gonna be long-term clients for us and satisfied with not just their first fix, but subsequent fixes.
Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.
This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest
Ashley HOST: Hello, and welcome to today's [00:01:00] episode. Today we have Nick Beyler, data science manager from Stitch Fix. Nick, welcome to the show.Thanks
Nick GUEST: for having me. Excited to be here.
Ashley HOST: Yeah, and Nick, maybe you could kick it off by telling us a little bit about what you do for Stitch Fix
Nick GUEST: Sure. And I'd be happy to talk just a little bit about Stitch Fix in general as well, just in case some of your listeners don't know what we do.
Ashley HOST: Yeah, absolutely. Let's start there
Nick GUEST: All right, I'll start with Stitch Fix. For those that don't know, Stitch Fix is a l- leading online personal styling service.
So what we do is we have expert stylists, and we also use best-in-class AI and recommendation algorithms to leverage a rich assortment of apparel and accessories brands to meet our clients' needs. So the idea is if a client comes to our website or comes to an app, to, to our app they fill out a quiz saying what they want in terms of their style, their fit, their preferences.
It's a comprehensive quiz. We wanna know a lot about what you're looking for in your fit. And then clients are matched with a human stylist. I feel like these days you have to say human stylist in the age of AI. But [00:02:00] yes, we have human stylists that review the quiz results account for any other request notes clients may have.
And then the stylist selects up to eight items to put in a Fix, and you get your Fix in a box at your door. And it's, for me, it's like opening up a Christmas present to see what's all in there, like the cool shirts or shorts or shoes that I get. And then, yeah, you can try the clothes on in the comfort of your home, keep what you want, return free of charge what you don't want.
And for those of you that, want to shop outside of Fix, we also have Freestyle. So this is where, you can do your online shopping pick out i- items, and this is also curated specifically for the client. So we, again, have our algorithms working in the background to make sure we're putting items in front of you that we think you wanna see.
So that's the Stitch Fix model in a nutshell.
Ashley HOST: Got it. I didn't realize that you could just go and shop just like in a e-commerce store
Nick GUEST: Yep. It's the freestyle experience at Stitch Fix. So I once you get your fix and you can't wait till your next fix, you can always just hop on our website and [00:03:00] do some freestyle shopping
Ashley HOST: I love it. I have to say that I tried out the service in advance of this episode and was just blown away. I got uh, six different items in the package. I'm keeping all six.
, This is one of them.
Nick GUEST: You are keeping our - keep rates up. That's a key metric we track in our experiments, so that's great
Ashley HOST: I'll bet it's very important for your business. And I'm even gonna wear this to the board meeting later today, so
Nick GUEST: All right. I hope you tell them it's from Stitch Fix
Ashley HOST: I will, for
sure. It, it definitely. So yeah, that's great. So it's interesting though just listening to you describe the business. I'm sure you're going through quite a bit of change with the application of AI to your business and just allows you to take personalization to a whole new level.
Nick GUEST: It does. And we always like to tout the balance we strike because Our human stylists are critical for our business model. And, our clients wanna be able to talk to the stylists and, send them request notes, tell them like, " Hey, I'm going to a wedding this weekend. I wanna... it's gonna be like a summer-themed wedding.
I, I need some [00:04:00] outfits." You know, And for all that AI can do that, it's maybe not gonna find the right outfit for the client in that respect. You need that human touch. But yes, AI, assistance is definitely embedded into our platform more and more, and it's a lot of the experiments we're running is like what kind of AI tools and features can improve the client experience and frankly the experience for our stylists too who, wanna match the right clothes and accessories to the client.
So yes, 100% we are focused a lot on generative AI and what it can provide our business
Ashley HOST: Yeah. And can you tell us a little bit about your role there?
Nick GUEST: Sure. So yeah I'm, like you said at the onset, I'm a data science manager at Stitch Fix. I lead our decision and insights team, and my background's actually more in traditional statistics. I actually... I'm gonna be dating myself here, but I went to grad school before data science was even a buzzword on LinkedIn, so that kinda speaks to my background.
And I'm actually a more recent convert to the tech AB testing space, if you will. I was doing a lot of experimentation more in the [00:05:00] public policy arena, so think, your bigger randomized control trials or quasi experimental designs where, it takes months, even years, to see a test through.
And recently when I transitioned to the AB testing space I was blown away. It's just such an adrenaline rush. Just being able to design your test and then roll it out and then, you learn oh, something's not working quite right. You iterate, you roll it out again, improvements, and it just yes, awesome.
It's working the way I thought it would. And, you still get to apply the same statistical methods and techniques that I'm so passionate about. So yeah, that's a bit about my background. My team we oversee our experimentation platform at Stitch Fix, so we have an internal platform that other teams, if they, wanna test a product or feature, they come to us.
They're saying, "Hey, let's allocate. Let's roll this out, on Monday. We wanna run it for four weeks. We're looking for this sample size." We consult not only on how to set up the experiment, but also on the design, how they wanna do their analyses, how they wanna make their decision criteria, any guardrails or stopping criteria, that kind of thing.
[00:06:00] So that's my team. We get to play in everyone's backyard. We get to see all the different kind of experiments that are going on across the company.
Ashley HOST: That's great. I have to come back to that point. It's such an adrenaline rush. I think you're the first guest to say that on the show,
Nick GUEST: It i- Oh I loved it. I can't say enough. "Where has this been my whole career?" Kind of thing. It's just I've fallen in love with A/B testing in the tech space
Ashley HOST: Got it. That's great. And so how many different teams are you working with?
Nick GUEST: Oh, man. I'd say it's almost every team across Stitch Fix. So we definitely have our client experience and marketing teams. As you can imagine, they're doing a lot of experiments, but, sometimes our merch team, our merchandise team, or even our people and culture team or ELT, like we're working a lot with our leadership team sometimes, talking directly to them about experimental results.
So again I get to play my team, we get to play in everyone's backyards and talk to everyone across company
Ashley HOST: Yeah. I was thinking also that's a pretty important connector type of role where you get to see learnings from different parts of the [00:07:00] company and hopefully, share those and help everybody benefit from them.
Nick GUEST: Exactly. No, we have maybe I might be jumping ahead here, but we have an experimentation review group. So this is a weekly forum where all the expert experimenters, data scientists, product managers, we meet weekly and talk about the upcoming launches. And it's just a great venue that everyone can say "Oh, I tried that experiment or that kind of experiment a couple months ago or a year ago.
Here's what I learned there that this may be helpful for you." And it's just a such a great venue that everyone's into experiments at Stitch Fix and, we're, we can come together in a weekly forum like that
Ashley HOST: Yeah. I particularly love the way you just described it, 'cause it's not "Oh, I tried that. Don't do it." It's like, "Here's what I learned," 'cause
Nick GUEST: so maybe your first run can be a little better than my experience
Ashley HOST: That's right. 'Cause yeah, there's so many different ways to implement
Nick GUEST: 100%. And of course the market and climate changes. So something you tried two years ago, three years ago may actually be worth trying again,
Ashley HOST: yeah, that's really interesting, especially if the rest of the [00:08:00] online experience at Stitch Fix is changing what might have...
Nick GUEST: how does this product or feature, how is that embedded in our new platform that is, relies not just on stylists, but AI tools? Yeah
Ashley HOST: Yeah. And are there certain areas in particular that you've been focusing on in particular?
Nick GUEST: In terms of specific types of experiments?
Ashley HOST: right, yeah
Nick GUEST: Yeah. I think we're talking a bit about AI So I think one interesting... I think there's a lot of different generative AI tools and features we're testing, and one of them is AI style assistant. So the idea here is a client can, t-type up their note their request note for their stylist, and then they can use an AI assistant to help them, help them work on the grammar or clarity of that note.
I'm sure a lot of us write an email or a Slack message and I'm like, "Oh, Nick, that's awful. . Your grammar's off your run on sentences," that kind of thing. And so same idea here. You can throw it into a chat bot and it helps you clean up the message a bit so it's clear.
So that's one tool we've definitely tested and experimented with. We've seen success there, but at the same [00:09:00] time, we've noticed, which is great, and this is one thing I love about experimentation too, the AB testing world is it's not just the quantitative results. We get feedback directly from our clients that are part of the experiment.
They're saying like, "Hey I like this tool, but I wish there was more flexibility. I wish I wasn't... I feel like I, I'm forced to use it," and that's, not the intent. So that is where we get a lot of our key learnings is in the how we roll out a product or feature and how the clients are reacting to it.
So that, with a lot of these AI tools, that's, I think, a key theme that's coming through
Ashley HOST: Yeah, that makes a lot of sense. And I guess with your business model, there's probably more opportunities to get that kind of qualitative feedback from your users.
Nick GUEST: 100%. Yeah, they're, they have a fix and we give our clients lots of opportunity to talk to our stylists and, or just leave open-ended feedback. Lots of rich data. They're-- we love getting the input from our clients and using that in our, in what we're developing.
Ashley HOST: That's terrific. Could you tell us about a time when you had an experiment where you got a lot of learnings?
Nick GUEST: [00:10:00] Sure. There's another one, it's interesting 'cause I knew you were gonna ask me this question, so I went back and polled some product managers and data scientists that have been around a while. "Hey I'm going on this podcast. He's gonna ask me, this...
It's all about learnings and experiments," which is awesome. And it's interesting, a common theme emerged from all the examples that were shared with me is the most impactful learnings aren't necessarily coming from a bug in the product or feature. Because usually by the time a product or feature gets to the AB testing phase, at least for a lot of our bigger ticket products and features we're pretty sure it's gonna be a winner in some ways 'cause, we've seen a demand or a need and we've done our research, done our homework.
Where a lot of the breakthrough learnings . is with the adoption friction, if you will. Is the messaging about this new product right, or are we introducing it in the right place in the funnel so that users don't feel overwhelmed with it or something like that. So that kind of is the overarching theme, I think, that where I took away from asking about what kind of learnings we have.
I mentioned the AI [00:11:00] assist example and I, I know that we talked offline a bit about the Stitch Fix vision feature that we have as well. That's where we're also getting a lot of learnings is, you have to upload a selfie to s- set up the vision tool and, some users aren't comfortable uploading a selfie or don't know how to do it, and that was a point of friction that, we're working on as well as an example
Ashley HOST: Yeah, let me just show the
Nick GUEST: Oh, great.
Ashley HOST: thought it was pretty interesting
Nick GUEST: Very cool. Yeah, there you go.
That's, are so into this feature, and,
um, It's just cool to see you in your outfit in a cool setting, maybe on vacation or something like that. ~That's~ that's so great that you were able to pull that up. That's
Ashley HOST: Yeah, I love it. And of course it encouraged me to like, "Okay, now let me load some more photos,"
Nick GUEST: exactly. There you go. That's intent. Yeah, no, once... With this feature, we found that once our clients are able to use it, they love it. So that's kinda what we're going for.
Ashley HOST: Yeah, that makes a lot of sense. And I thought it was interesting how you introduced the feature, right? 'Cause it's not a huge part of the [00:12:00] initial onboarding experience, but you do ask for a headshot.
And then as you draw people in, then you encourage people, "Hey, you wanna load a full body shot."
So I... it's clear you put a lot of thought into just, like, how to evolve that. Yeah.
Nick GUEST: 100%. No, we want the user to not feel like they're forced to do it, 'cause, it, it's, you have to a- upload a selfie and but it's completely optional, and yes we're still working through this. But it's a hit with people that are using it.
So no, that's... i'm glad as a f- fellow experimenter you noticed. You, yeah we've definitely done our homework about where to insert this into our into our
Ashley HOST: Yeah. And taking a headshot, like I just took a quick one of myself right then and there and uploaded it and didn't really think about it. And then, oh, now you want a body shot, just me. And it took me a while to find one on my
Nick GUEST: e-exactly. Yep. Yep
Ashley HOST: Yeah. So yeah, you've clearly thought about all that.
That's su- that's super interesting. But I wanna come back to your point 'cause I thought it was such, such an important one [00:13:00] on that, you have a high degree of confidence in the feature. The question is how do you drive the adoption and where does it live in the buyer's journey? That's been a real repeating theme across the show.
And I've heard it talked about in two different ways. One is just like understanding where people are in the buyer's journey, and then if you're gonna introduce a new element to the experience, where does it logically live that's gonna best support that user experience? And the other one is more okay, we've got something new, we believe in it, but we don't necessarily know, like, how it's gonna resonate best with users.
And so maybe the first time it fails and then you iterate. And so it sounds like you're doing a lot of that as well.
Nick GUEST: Yeah, we are for sure. My mantra if you will, is if you're only testing winners you're maybe not taking as m- enough risks, right? I
think . especially on the first iteration, you're gonna find losers. And of course there is always the worry of unintentionally harming our client's experience, right?
By [00:14:00] we're rolling something out that we don't think they're ready for or that we didn't maybe do our due diligence in making sure it's ready to go. But that's why you build in g- guardrails and you ensure that, when something's put out there you're ready to go in case something's not looking right.
So yes it's always, to me it's striking that balance with that second camp of experiments you're talking about where, you're like, "I'm not sure if this is gonna be a winner or not. But, I think we're at the stage where we need to test it." So yeah we're doing a lot of that kind of work too, for sure
Ashley HOST: Yeah. And shopping is such a intensely personal experience that I think that raises the bar even higher
Nick GUEST: 100%. We wanna get it right and, especially with new clients, we wanna make that great first impression like it sounds like we did with you, w- with your first fix. That's really important to us. So we take great time and care before we roll out any feature that would potentially degrade in any way the client's experience
Ashley HOST: Yeah. And how do you think about what are the right metrics for an experiment there, and what's your North Star and what are your guardrails and [00:15:00] things like that?
Nick GUEST: Sure. Yeah. I don't think we have a single north star metric necessarily because with our business model, like we have our kids' line of business and our women's line of business and our men's line of business, and we may be rolling out product or features that are trying to improve or us lift in certain areas of those different business lines.
And I think I've heard guest on your podcast talk about this. There's not necessarily a single north star metric for their businesses either. With The Fix we're definitely looking at things like keep rate so the number of items a client's keeping. Is that, some new product or feature encouraging them to keep more items?
We look at a successful fix, so this is where if a client keeps at least one item, and they also give us some intent that they're planning to get a second fix, a, a subsequent fix that's obviously a great signal for us as well. And then, we also our stylists, we run experiments that are focused on our stylists.
So is, does something help improve their satisfaction with their experience and, or how much time it takes them to style a fix, th-those kind of [00:16:00] metrics as well. So really depends on the experiment we're running kinda what our kinda key north star metric is.
Ashley HOST: Yeah. Super interesting. Yeah I couldn't help but thinking as I was signing up for it, like, how do they really know what I like and what I don't like? And just there's a lot that would go into styling a fix
Nick GUEST: 100%. The fact that you kept all your items is a signal to us. For, so if you're ever in a future experiment, that'll probably be a key signal that we're like, "All right, A- Ashley liked all her items in this fix for a subsequent fix, let's keep that in mind."
Ashley HOST: Yeah. Super interesting. And I would imagine that just the lifetime value of a Stitch Fix customer can be very high 'cause once you've got them and they're happy and it's a good fit for them, they're gonna stay with you a long time.
Nick GUEST: The holy grail for me of experimentation would be a North Star metric that accurately and reliably predicts long-term value. Because obviously as veteran experimenters know, we can measure the short term, but we can only predict the long term. So what [00:17:00] actions and behaviors of our clients in the short term can help predict whether, they are gonna be long-term clients for us and satisfied with not just their first fix, but subsequent fixes.
So definitely some causal inference modeling that our team works on to, to try to tease that apart. What are the key signals that we get from clients early on to help predict their long-term value? So yeah you definitely hit the nail on the head there.
Ashley HOST: Yeah. Yeah, that, that's super fascinating. We could have a whole episode just on that topic.
Nick GUEST: I'd love it, yeah.
Ashley HOST: So where do you see experimentation going at Stitch Fix?
Nick GUEST: Yeah. Right now we're working on a new in-house platform actually. So we're we're blowing up our user interface. It's worked well for us for the past 10 or so years, but we wanna bring more people in and feel comfortable with looking at experimental results.
So data scientists, product managers leadership team members, just your casual observer that's "Hey I know they did that experiment with generative AI. I wanna see how it went." So we're building a platform and user experience to allow all these different kinds of users in [00:18:00] based on, some have more technical understanding than others.
So that's front of mind for us right now. We're hoping to roll that out in the next couple of months and it's been a really fun beta testing process, just hearing folks say, "Oh, I love that you have this easier s- search feature that I can just go find past experiments or we embedded sequential testing."
So for listeners that may not know, this is a way you can monitor your experimental results before the end of the experiment without, inducing maybe false, higher false positive rates. So anyway that's something we're working on right now. Longer term, I think it's all about for me at least agentic AI.
I am really curious if AI agents can help improve n-not just like setting up an experiment with the design or the power analysis or summarizing the results, but can they help us make launch decisions? 'Cause I know any experimenter out there has had those kind of borderline cases where they're like, "Yeah, it wasn't quite stat sig, but some of our other diagnostic metrics suggested it would be good."
Is [00:19:00] there any other learnings like an AI agent can pull together for us from past experiments, from the macroeconomic climate? That's what I think agents are good at is they can act as humans, but multiple humans and pulling this information together and maybe giving us some insights we can't just catch looking at our dashboard.
So To be fair, I think the jury's still out in terms of if that could be a true value add to our tried and true statistical methods that we use. But that's what I'm super excited about now, and I've done a little research in this area, and I think it's a growing line of a l-line of research that folks are looking at other companies too.
Ashley HOST: Yeah. Yeah, super hot topic. It seems like experimentation and AI intersect in so many different ways.
Nick GUEST: It is blowing up. It is exploding. Yeah,
Ashley HOST: Yeah. Yeah. There's all sorts of interesting challenges around how do you A/B test an AI experience, especially if it, there's not like a clear outcome
Nick GUEST: 100%. No, I think that's exactly right. I think a lot of us have to, in the experiment world, have to [00:20:00] redefine maybe how we measure an impact in the age of AI. So yeah, that's a great point
Ashley HOST: Yeah. Nick, thank you so much for coming on today's episode. I feel like we covered a ton and such a fun user experience really enjoyed having you on the show
Nick GUEST: I really enjoyed being here. Appreciate the podcast and the forum to talk experiments. I love it. So thanks for having me
Ashley HOST: Thank you