Hey everyone, welcome to This Anthro Life as always, this is Adam Gamwell. Today I’m sitting down with the Super Cool LaiYee Ho and Alex Limpaecher. They are two co-founders of a software company called delve. Delve is a software suite that is used for qualitative research to help researchers work asynchronously or together for coding transcripts, finding insights, and pulling out actionable data from research. So we had a great conversation. I met with them when I was in New York for the intelligence piece conference in July and so we’re excited to bring you this conversation now where we get into a lot of really interesting conversations around the question of what research is, why qualitative and quantitative data are different in how they can be used to complement each other and provide a more robust way of knowing what’s happening in the world. Answering questions, both the what, the how and the why and some really interesting ideas about, you know, what does it mean to research researchers? You know, how do people do research? How do we understand what’s happening in the world of humans, in the weird technologies that we make and the dreams and designs that we bring into the world? What does it mean to research the people that are trying to make sense of that? So, this is kind of fun. It’s a bit of a Meta episode sometimes as well as just getting into how the software works. So, we recommend you check it out @delvetool.com as well as check in for the conversation. So, we’re going to turn it over to LaiYee Ho and Alex Limpaecher to introduce themselves, and then we will get started. Cheers.
I’m LaiYee. I’m a co-founder of Delve. I do UX design and research.
And I’m Alex Limpaecher. I’m the other co-founder of Delve Software Development and Machine Learning.
What kinds of formative educational experiences, or formative work experiences, brought you [to this point that] you’d say, this is what I do?
Sure, I studied information science, and then I started off my career as a UX designer. Not too far from where I am now, but my journey into research and learning more about ethnography and qualitative methods was that whenever I designed something, I would just ask so many questions that I realized I had to go out and answer those myself. I didn’t have the luxury of working with in house research teams. I eventually started spending more and more time answering those questions than designing and then also found that personally, I found that to be really enjoyable, just having an objective or a question and going out and learning about people. So that kind of brought me to building the research team at the company that we had, where we met, and then as we journeyed into Delve, it was a great way to combine both design and research, but really exciting because we get to research how researchers do research and then build a research tool for them. [I] kind of arrived at it through that journey of asking questions and arriving there.
Yeah. For me, I feel like I had just spent a lot of time in academia before I sort of went to the startup – where you [LaiYee] and I worked together for the first time, and it was both in Undergrad and in Grad school where I was doing computer science. I was always really interested in sort of measuring things that people usually considered unmeasurable, or even [that] maybe you shouldn’t measure it. Specifically, I did a lot in Grad school, of what I call art analytics. Where basically I was working together with artists to try to do these studies to try to understand why they drew, where they drew. And if there was a sort of a mathematical underpinning of that decision and to what extent you could say, oh yeah, your artists draw here because ‘this is an edge’ versus ‘this is like an artistic flourish.’ And separating those two. I was always really fascinated by the sort of overlap between computer science and the things that you would sort of maybe put more in the sort the human condition, like stories or art. So for me, sort of when LaiYee and I started working together at Delve, I had never seen someone take a very sort of structured and structured approach to both understanding people, but then also [to] designing the workshops and conversations around getting a group of people to understand an idea collectively, coming to consensus. And I would just, I was sort of fascinated by that and thought it was fantastic. And so, I think Delve in many ways was the combination of those two things where qualitative data is in many ways. Qualitative analysis is a sort of structured way to understand sort of humans. And so, it’s sort of fun to get to combine those two things together and build software that’s around something that’s very human.
Right on. Software as the human artifact, you know, tell, tell us a bit about like, this. Let’s, let’s dive into what Delve is. So walk us through, so we’re not looking at it. Obviously, we’re talking on a podcast. So like walk us through like what, what does Dell do?
So you’re a researcher, and you’ve spent a couple of weeks out in the field and collected a number of transcripts and what delve enables you to do is bring that data all into one place. So, it’s an online platform and then now you have all your transcripts within Delve and what you can do is code it. So, if you’re looking for themes or certain observations, you can structure that data in particular ways. You can create a hierarchy, merge those codes, but essentially it’s a structured way to break down those transcripts, organize them in ways to find your insights, export a report, and then you can share that with your stakeholders or clients. But essentially what it gives you is a way to trace a lot of your insights back to the original data. One of the critiques we had in the methods we use in our previous company was that oftentimes qualitative research was delivered in this glossy report with beautiful quotes. But it was difficult to look underneath that. Like where did it come from besides trusting the researcher’s narrative? So with Delve, what you can do is pick it apart, and you know, look at the entire process. It makes something that was previously opaque to non-researchers more visible — and really hoping that it’ll enable more people to understand that process of qualitative research.
Yeah, I think someone, I think it was on your podcast was almost talking about, you know, opening up this black box of qualitative work analysis. We saw, you know, at our own startups and lots of people that we’ve talked to is that, you know, oftentimes stake-holders they don’t, they either trust a quote blindly, and there’s no way of knowing if it’s just like a cherry-pick quote just to prove a point or it’s actually a representative quote that is actually, you know, from a really backed insight. Or there’s that one end, and then on the other end, there are stakeholders who don’t trust qualitative analysis at all. Cause I think it just sort of from the gut made up on the spot think that anyone really can do it. And, you know, just being able to see the sort of amount of work that goes into delivering those insights and the fact that your stakeholder can sort of stress test insights that have come out of Delve is like one of the major benefits that we’ve seen.
And just to add to that, a lot of the principles that we brought to Delve were very much inspired by different researchers we met in academia. So, I mean, neither of us have a social science like academic background, but we actually interviewed different people across sociology and anthropology, and that’s where we learned about some of the more rigorous methods or practices such as coding. I actually hadn’t coded qualitative research before building Delve, but we learned about that from sociologists. Think [the process of] writing a paper and having to explain that methodology. We found that there’s a lot that the industry could take from that. I think not the entire process, but I think there’s different ounces of it that really influenced it all because we felt that that level of transparency and explanation is really critical for having actual insights be accurate and things that people should take action on.
Right on, I mean, this, this may be like a deep, nerdy question, but I guess that’s the point, right? How do you stress test some of this stuff and deal, like how are you having, how does that enable like a, like a stakeholder to come look at it and say, let me check and make sure that these quotes are like backed up by, you know, the research process?
I think the first thing is that all the quotes are cataloged in terms of the codes. And then the second is that we have some high-level numbers in there. So the purpose is not to make it quantitative, but if you have a theme or code, then you can see the difference between a code that shows up across all 20 transcripts versus one that might lean really heavily on just one or two. Like you can look and say, this was really profound. It showed up only in Joe’s transcript. What does that mean? And then you can ask that question about how you might want to approach the insight. But before we had something like delve actually in our previous companies, sometimes we would actually just take the most evocative quote, post it all over the wall and maybe not necessarily question exactly how many people are represented. So this kind of count that we put in Delve really just gives you that level so you can ask more questions.
We were actually doing an internal study recently of our users. I had talked to a number of them, and I’d actually listened in on many of them, so I was already in some, you know, I was the stakeholder in this case, and I had at least had some context of what I had heard in the past or had listened to these. But basically she was able to send me over, you know, in Delve sort of like the high-level report and I could read through that and whenever I read something that I didn’t quite believe or I remembered it differently, I could click on that and get all the supporting quotes to see what was backing it. And there was one case where, being able to see who said what, I just noticed that like I found that two people had just sort of fallen from the high-level insights and I could point that out and then I LaiYee took a look and noticed that yeah, there were two.
There’s a whole different basically use case that we had completely missed and it wasn’t that, and it only would have impossible because I was able to sort of in a way very easily and quickly sort of see what analysis she had done and seen what conclusions she came to. And most of it was great, but being able to just have that transparency really was amazing and was really one of the first times you were like, wow, we, we don’t know how many sorts of people we’ve talked to the past that we just sort of accidentally forgot because of the methodology we’re using didn’t allow us to realize that we had sort of forgotten these participants.
I really appreciate that idea that you know, depending on the methodology that you’re using, if you’re doing kind of traditional analysis, you may totally just drop or not realize you missed something or like in the way you’re coding it. But so this is a, you can kind of Crawford Cross-reference both things that people say in the codes as well as these high-level numbers. That is that the high-level numbers are kind of how many times like a theme has shown up or how many times like an idea goes across even that like, I mean, yeah, the capacity to visualize that. I think it’s so important, you know, and that you know, again, you know, traditional ethnographic, even user experience research stuff too, you may not see that you’re not depending on how you’re putting the information in. So it seems really interesting that that’s, that’s kind of, you know, this is that that pain part.
Cause you mentioned earlier on that when people like research is one of the main pain points that a lot of social scientists or even UX or depending on you, how are you doing AI or machine learning stuff? Like it’s like that. How does that research happen? So one of the, again, I love the black box metaphor is always good, right? How do we open that and see that process? That’s a really good illustration of how that can happen. In a simple sense of like thinking across numbers and codes, right? Just like, let’s put these pieces together and let me be able to see what, how you got to your conclusions. And the second piece that, that struck me that that is with this too is that, so this then enables you to work at a different speed too, right?
I think part of it, and as the reason I think the power and the interesting thing parts of delve too is that you can work at kind of industry speed, right? As you probably know when you’re talking to sociologists and anthropologists, I don’t know if they mentioned how slow research could go in academia or are you just, you just know, right. But industry moves at a very different speed as you know. And so, you know, I guess like, I love to think about that with you and hear about your thoughts about speed and how do we, how do we move qualitative to this sort of industry speed?
It’s interesting you say that because when we were talking academics, even though in comparison it’s more time like they feel crunched for time too. So overall, I think all across the board, the biggest thing we found was that qualitative researchers don’t have enough time. And I think another layer to that is that especially in industry when it’s really rushed, I think there’s also kind of a nagging anxiety that you might rush something just to deliver to a client, but you might not be totally sure that it was right or accurate. And it really is the nature of that, just the constraint. But what we really like with delve is that because you can catalog and code, we’ve had studies that would normally take two weeks, just take two days. And then on top of that, since other people like I can send my study to Alex, he can also double-check and verify without looking through every individual post-it notes or every transcript. So being able to have other people on your team look at it super easy and then the coding itself is also a lot more streamlined.
At the anthro conference that we’re all at together, I think a common theme was that you know, anthropologists are struggling with sort of how to apply their methodologies, but how to take their methodologies from academia and bring them into the world while, you know, and doing that in a way that is fast enough. But also, you know, there’s a struggle for the, between the people who want to keep their methodologies very pure versus the more and more anyone goes more into industry sort of a certain amount of pragmatism starts seeping in and a lot of that’s good. But at the very least, you know, ideally, you know, we can keep a lot of those methodologies or what’s behind them as sort of pure as possible while you know, if it’s technology that’s holding you back from moving faster and that, that part we can solve, you know, it doesn’t, while, you know, I’d rather have the methodologies, make it as you know, intact as possible from the transition from academia to industry. And, and sort of remove all the technical barriers which are slowing people down right now.
Yeah, but let me just speak to that point, Alex I think is quite interesting. Is that like how, how is industry sort of think about qualitative and quantitative, right? Because it seems like there’s a love for big data and there are numbers. We want those. But then like there’s also the recognition that I think you’re both bringing to the table too, is that qual is like super important, right? It’s like, it’s the why behind the what sometimes, you know, and I don’t know. So part of it is like what kind of perceptions have you seen in industry for these two types of data, and you know, how are you trying to address those, that perhaps the gap therebetween them in industry?
Yeah, I mean sort of seen firsthand and for a lot of people that we, I think we live in a society right now where we overemphasize quant considerably. And it’s not that I want quant to go away. That’s entirely my background. But it is it, it’s, it’s trusted too much, and I think people don’t realize, I think what you said with like why versus what, like there are some real limitations that come with quant. In terms of like, quite literally data is not very intelligent. Even if you throw fancy machine learning words at it, at the end of the day, it can only tell you how the world is, which is great for like getting your bearings. But as soon as you sort of use numbers to sort of understand the world and try to draw conclusions from that, I think what often people don’t know when they’re looking at quantitative data is you’re actually sort of like applying a certain narrative structure to that quantitative data.
And it’s just gonna be because cause quantitative data and machine learning, even the majority of machine learning we can go into some exceptions. It can’t tell you how the world can be different than it is. It can’t tell you what the results of you taking action could be all of that, all that sort of like a counter positive or, or sort of hypothetical sort of predictions of what might happen if you act on data. All of that is being introduced. That sort of cause and effect is introduced by the person interpreting the quantitative data. And that’s where I really see that you know, qualitative playing a role there. If you don’t, if you’re not doing qualitative analysis to sort of introduce sort of stress testing, the narrative that you have about the world, you’re just going to be projecting.
You know, you are that at the end of the analyst is the executive is just gonna be projecting their own mental model. Which means you may be drawing completely the wrong conclusions and that problem doesn’t go away. When you start applying machine learning, it just turns [honestly] numbers into a black box. And now we, at that point everything’s a black box. And so I love qualitative data cause it is like you’re saying, it is the only way to answer why and knowing why is the only way we’re going to be able to know what actions we should take-off of our understanding of the world.
And in addition to quantitative giving them what, there’s so much that qualitative can contribute to the what as well. Like for example, in the previous company we were working at, we had a lot of information from the data analytics in terms of behavior patterns of how people were using a particular mobile app, but it wasn’t telling us what the family was doing around them, or you know, what was happening during that time of day. Like we really had to go in person and observe that and understand the what and the why together. So you really, in order to really answer the question, you often just need both of them.
I totally appreciate that idea that qual also gives us the what, because it’s almost like, you know, and then some have like because of the industry perception and bias towards like the big data is a big data, the big data, but as giving us the what, right? And then we kind of say, well call it gives us the why, but it’s true. Like qualitative also doesn’t just give us a “what”, it gives us an important one that’s, that can be different than the numbers too, right? They obviously can go, they work quite well together, I think. But I think that’s actually super valuable that it’s, I don’t think that that doesn’t get stressed enough to that there’s a what value also for qualitative data. You know, we’re not just saying like, well I know why you did it, but what are you doing also matters.
It does. I find that I think people underestimate how much time it takes a turn quantitative data into anything actionable. I think because it’s so easy, especially in this day and age with like the, you know all the, you know, the Internet and technology, it’s so easy to collect quantitative data that it seems like a win. But when you actually see what are they, you know, what questions you can actually answer. It’s actually quite limited. It’s really hard to get quantitative data to answer anything, but exactly what you’re, what you measured. So if you just measure how many people clicked on something, it’s going to tell you that. But like that often doesn’t really tell you all that much. And qualitative, I feel like really jumps to the point, you know it really allows you to sort of understand something.
Like I think people think qualitative is really expensive to do cause it is in many ways you have to put a, you know a human into the loop like recruit people, go to places, talk to people. But at the end of the day, I think the sort of it will actually, I feel like almost always gets you something actionable that you can be confident in and often at that point also even really increases the value of the quantitative data that you also have. I feel like it’s only that [way] because quantitative data is so easy to ‘passively’ collect. That feels like a win when it really often is quite difficult to turn it into insights.
And then I think you also talk, you talked about that perception of in quantitative data it depends on either the data scientist or the analyst to add a narrative to that. And what we found in the company we worked is that when you have quantitative data, it can be interpreted in multiple ways. Like if people aren’t clicking on a button and an app marketing person could say it is because they weren’t aware of it. A UX Person will say it is because they couldn’t figure out how to do it. Like so you almost organizationally you’ll get into these debates interpreting that data. But then if you bring in qualitative and you bring in users or whoever it is to really understand the why and the context that becomes almost undeniable. Like when you are faced with your participants, then you can actually see their reactions. It gives a lot more context to that number. And then qualitative data being actionable was a big theme. So yeah, it wasn’t until we added qualitative to the quantitative methods that Alex was doing that we, we could actually get everybody to align and figuring out what the roadmap was.
I love this idea of undeniable data when you put the pieces together. Cause that’s actually quite fascinating to think about. You know, and I think that is a wonderful way to capture that and to give an example of a UX or a marketer look at the same non-click of a button and think of it different ways. But then it’s like you put those together with a user narrative and then you actually have three different ways of interpreting the same action, and then you look at those think like what is what, what is the answer to that question? Then it’s like what’s depends who you ask. But then if you have all the answers that are like meta-information metadata, you know?
But you got another perspective in there.
Yeah, it’s great to be able to triangulate a single, you know you’ve got quantitative, you’ve got the user insights and then yeah, you do want to bring in the marketer and the designer to bring the expert insight to cause there they are going to have some really good interpretations on what you know that, that problem.
I’m totally thinking, I don’t know like Delve two or 3.0, or like depending on, maybe you already have this, but it’s just like the idea of if you’re bringing in transcript data it’s like are there ways to code in like, “oh we talked to the market or the market I like pointed out this thing on the UX person pointed out that thing,” but it’s like, but the instead of necessarily being, you know, focused on this like one word or something. Maybe that doesn’t like the idea of the phrase, but it’s like focused in on this idea that we found that we’re all looking at this button right in the topic is the button. Then you look at, and you click the button code and then it’s like all the people around it that said different things or different perceptions could be interesting. I mean it sounds like that’s kind of what you’re doing anyway. It’s just like this may be another way of visualizing it.
I think the great thing about coding too is that I think anything that happens or something that happens in the world can be interpreted with multiple lenses. And with the same set of data, you can have different sets of codes, and that’s almost how like the marketer versus the UX designer is looking at something. But when you can catalog that and trace it, then you can also triangulate those and maybe see what’s a theme amongst what all three people are seeing. And then also see those differences.
It is something though that that problem space is something we’re playing around with right now. Cause we’ve definitely seen that in many ways. Oftentimes the researcher is going to sort of create the world that everyone else can play in. You know, they collect the data, they organize it a little bit. Like the marketer isn’t gonna spend a full day coding anything, you know. So I do really, I’m just, I mean I’m fascinated by like how, yeah, how do you allow everyone to bring their level of, you know time and interests and expertise to, to these insights that are sort of that mainly the, you know, the user researchers is bring forth but everyone needs to have an input on. And so in terms of Delve 2.0 3.0, I think we’re really fascinated by that question.
My brain is cranking on this now. It’s like when you talk about like we’re bringing in transcripts, you know, like me as an anthropologist; I’m thinking, Oh I’ve gone out and done some interviews with people. Then I’m putting those in and then I’m using this to code things I’ve found like in these conversations, themes like high-level stuff, you know, like, LaiYee you said this and Alex that, and Adam said this then and Susan’s said that, you know, and like, okay we’ve seen this quote four times, this one, two times. But then, thinking about in terms of like coding the coders as part of this too, and this is what got me thinking about like is the market and the UX are going to see the same action slightly differently but also the same way as this, that question of triangulation I think is so valuable too because that’s really, I mean that’s part of letting this conversation of what qualitative and quantitative together do is they do help us triangulate these problems with trying to solve, you know, then on top of that like knowing who it is that’s asking you know, and what they’re like bringing to the table I think is quite interesting too.
You know, ’cause it’s, it’s funny. Not to harp on the same idea, but just the idea of like how the designer, you know, or a Coder, like a web Dev might bring in different language into the perspective of the same problem. You know, if the UX is like, well does the thing flow back and forth and the code is saying, well does it? Like is there a bug in between? You Click it, what’s it doing? You know, and they’re looking at the same thing, but like thinking about the levels of who the researchers are. So this is kind of the meta-question, Delve is kind of like this meta company, right? That you are researching the researchers, who do research. Talk to me about that. I like this idea. It’s just like turtles all the way down, you know?
Well, I mean from the beginning, even before Delve existed, we interviewed researchers on their pain points, and then we built Delve. But as we built it, it’s not like we just built a product and it’s good to go. We’re constantly doing research on those researchers and talking to our users, understanding the context of how Delve is helping them or not, and then taking those insights researching and analyzing that ourselves and then applying it to Delve. So it’s ultra meta because we’re just always researching researchers.
[We are] researching researchers to build research tools.
[Yes, we are] researching researchers on how they research.
Which can be a little, I don’t say nerve-wracking, but a little. It’s interesting when someone critiques how you’re researching them in some ways. Mainly that’s like a commute. You know, it’s like, oh, you’re gonna ask a bad question. Okay. That was a bit leading. You know?
There was a bit of a dance when I’m doing a method and asking questions to a researcher because I know they’re trying to figure out what I want to know about them always. And then, of course, there’s always critique like, “oh, I don’t think you should have asked the question that way.” Or like, “wait, who’s researching here?” Or when they turn it around and ask questions about me.
That’s funny. Do you find that that process helps? I totally get that it is nerve-wracking because you’re like, “oh my bad I meant to ask this,” you know? But do you find like if they ask questions back to you, is that, is that useful?
It’s actually really useful. Because another thing about researching researchers is that I ultimately learned about their methodologies, I mean there’s definitely a perspective, and how to analyze the research that exists at Delve and us researchers can sometimes challenge that and ask like, do you really need an account, et cetera. And then I can ask and learn about where did they learn their philosophy from. And then we’re constantly being influenced by different qualitative methods across different fields. And then we can slowly build them in to Delve.
It’s a really good stress test for what we are building. And Yeah, I know we’re really, I think, I think you know, how much, for example, how much quantitative we bring to Delve itself. Like how much do people want things to be numeric and I think it’s sort of a range. There are some people want, yeah, some people want a graph like charting, you know, how their codes are coming up versus a lot of people being very also sort of anti you know, don’t, you know, you’re, you’re flattening the qualitative data as soon as you like assign numbers to things.
I think it’s now way less nerve-wracking than it was like two years ago when we started just because I think we’ve gotten to the point where we’ve stress-tested the idea enough. I mean it’s always great when we hear it and a newer perspective on, on how we’re tackling the problem. Did we do six months of research or four months of research?
I mean, we continue.
We continue, yeah.
Years of research.
And I feel like we’re a bit more prepared for the questions now than we used to be, which is, yeah.
But I mean, I’m love nerding out with researchers on methodologies. So, these, whenever they ask the question, and then we could start debating or just geeking out on different ways to analyze qual. I mean, it’s the most fun type of qualitative research I’ve done.
Just a bunch of people who are constantly existing in a state of like abstract thought and trying to understand human beings. Like it’s, yeah, my favorite demos I gave or someone who’s like, “oh, can we just like chat like for like 20 minutes after I’ve seen demos?” I love that.
This is awesome. So thanks so much for chatting with us so far. So let us know how can the good people find you and you know, you on Instagram and Twitter website, how do we see what’s going on with Delve?
Yeah. So you can find Delve on delvetool.com. We’re also on Twitter and Instagram. Check it out.
Are you, are you on Twitter and Instagram @delvetool?
Awesome. LaiYee, Alex, thank you so much. This has been awesome. Super enlightening. I’m excited to share this conversation with the rest of the world.
Thank you so much.
Thanks so much. Let’s delve in. Sorry. Pun.
Thanks for hanging out and listening to the conversation I had with LaiYee Ho and Alex Limpaecher from Delve. We hope you enjoyed it and found some really interesting takeaways and tidbits too about of data and qualitative versus quantitative research and what it means to even again, study the world and use software to help us understand what’s happening out there. As always, as you’d like This Anthro Life, one of the best compliments you can give us is by sharing and letting someone else you know and love that you also know and Love TAL. And, pass the conversation onto them and so we can keep building this community of culturally curious listeners and thinkers and talkers and so that we can keep sharing and building this interesting world and conversation together also as well that you might give us a review, a couple of stars on iTunes, maybe five that’d be better and whatever else. If you want to pass this around on radio public, you can find us on Spotify all the place you’re listening, and you’re listening right now obviously at this point, so thanks for hanging out with us. It’s been fun. We can’t wait to bring you more content here in 2019 and we will catch you next time. I’m Adam getwell and this is This Anthro Life.