Let’s build a user research toolkit

Session facilitator(s): Emily Goligoski, Andrew Losowosky

Day & Time: Thursday, 4-5pm

Room: Innovation Studio

Andrew: Hello! , hello, hello, welcome to the session on hungry for audience insights? Hello, I’m Andrew L and I’m Emily. If you can’t hear, it’s because we have a microphone, if you can’t hear us we should definitely use it, so just raise your hand if you can’t hear it and say a thing and we will do that while we’re talking about voices and amplification, SRCCON is per experimenting if with some sessions and recording some of them. If anyone is uncomfortable with that, just raise your hand and we’ll switch it off. If that changes during the session, just let us know, that is totally fine and I think do we have a transcriber in this session?

And similarly for the transcriber, if there’s something you want off the record, not to be transcribed, then please start by saying can we make this off the record and then remember to say when you’re back on the record again. So that—our transcriber whose name is I’m sorry?

NORMA: Norma.

Norma can get it all. All right s we’ve going to be making a DIY user research toolkit so what that looks like and how that works will depend a little bit on who is in the room and what we want to work, but what we’re aiming towards is to get to a point where we have, collectively through our experiences and ideas, come up with a series of tools that we can then go and use to improve our user research and how we use it in our projects. We’re going to be using the etherpad for this, so if you have your laptop with you and you want to use it, go to the schedule and go to this session and you’ll see a link to the etherpad in there. We’ve got the URL up but it’s really long, so if you just go to the schedule it’s LinkedIn there. And in fact, when we’ll be moving to group discussion, so we’ll be prepared we’re going to ask each table to nominate a transcriber, who will be writing some notes about your. It doesn’t have to be line by line. Just notes about some of your conversation. We’re getting ahead of ourselves. Emily.

Emily: Thank you all for joining us in what feels like this treasure ship and we’re excited to that you’re here with us and excited to come up with a tool that is hackable and not hackable. Before we do that we want to have sort of a level set around what is user research anyway, and I’m curious how many people in the room have conducted or led or participated in conversations with news consumers to benefit their teams?

Great, OK, so about half of the people.

And then the way that I think of user research is conversations with people beyond your team, not just sort of navel-ging, but thinking about what you can learn from people beyond your organization that can help inform the products and the services that you offer. And when it comes to research, you know, a lot of in-person and remote technologies that we use to basically bring data, bring qualitative data back to our organizations that helped inform our decisionmaking and so that we’re not just operating from instinct or making decisions based on what we think is best, but really going out into the world, talking to people about that work before you ship, or to make continuous improvements to your product.

So what we would love to have you do in your small groups, go ahead and just introduce yourself. First name is totally fine. I you want to share any organizational affiliation and share with the people that you’ll be working with what it is you want to get out of this time together. Is it that you love to go back to work on Monday, with sort of a sense of how to get started with user research? Are you just looking to make good connections? Just sort of share quickly why it is that you came here at 4 p.m.

[laughter]

Andrew: Yeah, you don’t need a note-taker on this one.

[group activity]

Emily: All right, thank you for introducing yourselves. And just quickly, I would love to open it up to the whole group, sort of the question of what is user research good for and what types of questions is it really well suited to address? And I also realize that Andrew and I didn’t introduce ourselves. So my name is Emily. I work in user research in the New York Times newsroom very crucially finding out when they’re not with us, what are they spending time doing, really trying to build out a robust understanding of what their news and information diets look like. And Andrew.

Andrew: I’m Andrew, I’m the project lead on the Coral Project which is Mozilla collaboration with the New York Times and the Washington Post. And user research is really the core of the huge amount of research we did before we built any software, we talked to more than 330 people in 30 different countries, and people who don’t read the news and people who never comment and people who never comment and so on, just building a huge picture of what people think and what people want and what people need and at the same time, there were lots of things we of could have done better as we look back and understand the processes that we use. So yeah, that’s my connection to the topic.

Emily: I would say one of the things that user research doesn’t do a great job of addressing is should this button on our website be red or blue? Sometimes I think questions of taste are not the best types of questions to take to large or small groups of news consumers, but questions like, you know, what does it look like to use this during a busy day? [inaudible]

PARTICIPANT: Absolutely. I’m just curious and I know that this first group was starting to go into this sort of vein of conversation, which is, what types of projects are user research really well suited to.

Andrew: Raise a hand, what did you find it useful for.

PARTICIPANT: Just tracking do people click on these things, not what are we spending time doing? So it’s not so much should they be red, but do people ever find this?

Andrew: Could everybody hear? Could everyone hear? OK. So yes.

PARTICIPANT: My sense is that it’s offer helpful for fing out like the ways people actually use something, rather than what they think they do, or you even just watching the way somebody actually navigates a site rather than matches preferences.

Emily: Right, I think anyone who’s ever watched someone use a website without a demo, it can be a really humbling experience if you’ve been part of that team to all of a someone watch someone fumble and you’re thinking, I thought this was so intuitive. And that idea of what sometimes what people do and what they say they do are different things, so really trying to watch those processes as they happen. Thank you.

Andrew: Any more? Yes at the back.

PARTICIPANT: I’m from NBC news, we found user research very useful when we were developing our fire TV app. We found that the organization of the content, how stuff was getting pled into pockets of categories, were not really what people were looking for in that particular context, so we kind of rearranged and regrouped things based on the user feedback.

Andrew: And so one more hand, I think if.

PARTICIPANT: Identifying assumptions an counteracting biases.

Andrew: Identifying assumptions and counteracting biases.

Emily: To the first point about actually watching site behaviors, I wanted to just quickly take a moment to talk about different methods that sort of fall under a research or audience development umbrella, just so that we can sort of thinking about all the tools that are available at a news organize’s disposal, and really just sort of knowing what—when it is worth calling on one method over another, because there is so much that’s available?

I really think that user research is fantastic when you want to find out about information needs about preferences, about motivations, all of the things that you maybe can’t get through, you know, just understanding site traffic. However, what site traffic and data analytics are really good for are understanding patterns and propensities. What are things that a likely to happen again with certain types of visitors to your site? And I know a lot of organizations—and we just mentioned this with front line, also engaged in market research, so using third-party tools to really understand, you know, where do I fit within other organizations of my size or that report on stories similar to the ones that I * undertake.

so all along, the way, you know, saying that user research is especially good at helping us think about what are the things that I can’t get through data analytics and I can’t get just through market research or kind of like desk research? And so going out and talking to people really can help you get that—those motivations, those unmet needs, those really rich design spaces.

PARTICIPANT: We would love to have you get back in your groups and really understand, are there projects that you’ve worked on where you felt like user data was really valued? That the peopl on the team were particularly interested in it, that it helps shape decisions and on the other hand, are there projects where you worked on where you felt like it could have been more valuable? So please go ahead and take a few minutes. If the transcriber team doesn’t mind typing up a few short notes that will help with information capture and we’ll come back in a few moments.

Andrew: Start by deciing who’s doing transcribing at your table, and—it must notes. It doesn’t have to be full transcription. Do you need the question repeated? Can you see it in the back? Have you participated in a project where suggested product or design improvements were valued or not valued? Either one, pick one project that you worked on and just share with the others in your team that oh, yeah, they really valued input to improve the project or they really didn’t, and what was that like for you?

[group activitAndrew: OK, there were some really great stories that I heard going round. How about just getting a couple of tables just to share a story that came up on your table? How about let’s start in the back right where I’m pointing, Sasha, your table. Someone from your table.

PARTICIPANT: So we talked about the concept of data-driven versus data informed and we really like the latter, because it means we should all understand data. It doesn’t mean we’re in servitude of data, but it means we’re making more decisions and the user can then decide whether it matches your editorial or business decision and use it that way. And really [inaudible] we talked about in our group that some of—… talked about how she uses data a lot, she’s data journalist and uses it in some of the projects they work in in order to tweet and create nuanced experiences based on [inaudible] we talked about the power of qualitative and quantitative data, married together, that that’s asking why that, you know, that shouldn’t come to the assumption that 1 million: Is necessarily better than 20,000. You have to ask a question why and dig deeper to understand what the particular values are of those different audiences.

And we had experiences from Sybil, I’ sorry—Lynn, but they had this extra data and trying to validate how this type of data would be—[inaudible] interesting.

Andrew: That’s terrific, so we have data-driven versus data what was the word you ued? Informed, thank you, yes, and quantitative and qualitative. That’s great. I’m afraid we can’t pass the microphone around, because it gets feedback in this room, but yeah, there were some great pointers there. How about the table that is in front of them, towards—so yes? The people there?

PARTICIPANT: Let’s see, we talked about interviews we did with people who were online sleuths trying to solve murder cases, we did a project where we tried to help them use a government website that wasn’t very good. Partially because we had a narrow time to develop it, we told our staff that we were going to listen to these people and not to you, essentially, and because of the first time period, they actually listened to that and we got amaing feedback that we never would have expected on our designs just by talking to the people who use it. Let’s see.

Oh, we talked about how like every time—or that at Vox every time they tried to do some testing, it doesn’t happen a long time, but it doesn’t take that long if you actually get to it, and so it—a lot of us said that we wished we did it more.

Andrew: Great we’ll come back to that, that sesting doesn’t take long once you actually get started. So tips and tricks to make that happen. Yes we will get to that, for sure. And how about the table there that * has Emma on it? How about your grou

PARTICIPANT: Well, we had—we have a lot of stories of data being ignored for personal preference, basically trying to figure out you know, more about the buy-in and what’s driving this local either ignoring it or kind of willfully misinterpreting the data. Basically kind of—many anecdotes about that. Or actual think we had an instance of user—user feedback on it, in-house, you know, in a newsroom CNS change that was initially ignored, and then they went to the nonuers and then when they dealt with that, they went back to the user again. Who found out that that was probably a better way to go. To really, I guess, give it a second shot, talk to the people that were going to be using it, because they really did know stuff about how it should be.

Andrew: Who’d have thought the uers know something about how to use it. So data-driven and data informed and data ignored. And one thing that I heard on one table was about the difficulties that they faced around trying to get user research to happen in lots of projects. One project it worked and others it doesn’t. So one thing we have to think about is how to tell the stories of the work we have done and are doing within the newsrooms and outside the newsrooms to help fuel the opportunity to keep doing it.

Emily: One quick thing I’d love to do is just get a show of hands for meme who feel that their news organizations have all the resources they would like to use when it comes to research. If you really feel that way?

PARTICIPANT: I mean we should. And we do, we just don’t use it.

PARTICIPANT: Yeah, like we have lots of analytics.

PARTICIPANT: We have a data science team, I mean we’re pretty well tapped.

PARTICIPANT: Who feels like we’ve just begun to dip our toes in the water but I would like to see more of this work? OK, and who feels as though user research are two words that have never been paired within that you are organization to their knowledge. OK. Great. So one thing that Andrew and I are really firm bers in is the idea that user research is a set of skills that can be taught and really the idea of democratizing these practices, so that anyone can go out and collect data on behalf of their organization, and be—bring that back to make more powerful tools and more powerful processes. And to that point, one thing we’ve thought about a lot is that it shouldn’t just rest on design teams or marketing teams or customer service representatives to do this work, but really there might be ways to make it more of a level playing field so that developers can undertake this, and product managers, and people in finance, and copy writers, etc.

Andrew: I’d like to add to that, as well. That also everyone’s already doing it, like even if you’re not in user research, you go home and you share with your partner or you show it to your mom of hey, here’s this thing I’ve done and they’ll start giving you feedback, but if you have nowhere to put it or no way to get it back in the product or there’s no receptiveness on the team that it’s not their job. That all of this stuff seems to get lost and we’re trying to find ways where that doesn’t.

PARTICIPANT: In the interest of our time I’m going to skip this question and just quickly move to a conversation for you all to have within your groups which is what language and tools could we think about designing and developing together that would make this work easier? So where are places where you sort of sense education gaps that, you know, just putting some great education or some great training materials, or bringing in some expertise to your organization, would actually help begin to address gaps that you see and based on just a show of hands, it looks like about 95% of people feel like there are more opportunities in the places that they work for this work, so now we’d love to kind of roll up our sleeves together and really start thinking, you know, what could we design together that would help make user research more accessible?

Andrew: So yes, start with specifics about your jobs and where you are, and then we’ll move back into more applicable t th things.u.

[group activity]

Andrew: One more minute and then we’ll move on to the final piece.

Emily: All right. In the interest of time, we’re going to have you go ahead and just jump into the last question, which is: What do you wish more people knew about user research? And the challenge which I’ll give you here in just a second, really is the emphasis is what are things that you could design that could actually help you? So instead of thinking global, really think about the problems that your team is faing, so that we’re designing things that feel scrappy and kind of hackable. What we’d like to ask you to do is to come up with an exercise that you’d like to introduce to your team that could help them conduct some phase in the product or feature design process and you’ll pick amongst yourselves. We just threw out some of the areas that you could design exercises for are during concepting, so before there’s even a line of code is written, during development, when you’re actually building the thing you want to release. During product testing, when you’re actually going out and doing sort of a road show and collecting a lot of feedback, or post-release, when you’re trying to make improvements. So we’ve put paper and post-its in front of you. Please leave us with some kind of artifact that really can help us—have a little bit of a roadmap for what it is that you’d like this exercise to look like. We will be going around, and but the—again, the idea here is what is something that could be a very first component to a design research toolkit, an exercise that we could try to introduce, and polish and put out into the world?

PARTICIPANT: And this could be for what do you want for your team directly, so write down who it is it’s for, so it might be for developers, it might be for the journalists. It might also—I’ve heard a lot of people talking about their bosses an publishers and the trouble has been trying to get people to understand the value of doing research or data. So it might be for your publisher or for your boss so write down who it’s for and what is a step that you could take towards that and discuss it towards the group, so see if you can get other people’s feedback, think about it for yourself and then share it with the group and see if you could help improve each other’s OK? So is that clear? Who is it for? Your developers and your team, your boss, your designer, somebody else? And then what is this idea and keep it small and practical and just compact. OK? Great, go!

[group activity]

Emily: Last minute to get your ideas down …

Emily: All right, everyone. So just quickly, Andrew and I are planning to stay after this session, and read through the ideas and sort of begin to synthesize some of what you’ve shared. If you have a little time and you’re interested, please, you’re very welcome to grab an adult beverage and come and join us in that process. Before we break as a big group, I’d love to hear just one example or one exercise idea, per table, the thing that maybe is the waciest idea that you all came up with. Go ahead and take us through. First table?

PARTICIPANT: Well, I mean, like I think a lot of us are just trying to figure out what is that process. Like, what question should be asked, you know, really general.

Andrew: So the thing was just trying to figure out the questions to ask. So what is the process, so coming up with ways to analyze that big question, what is the process. This table here?

PARTICIPANT: We were talking about a toolkit for paper prototing. So just kind of like you could automatically get something that gives you a mobile screen, with like cut here marks with each screen and everybody would have kind of a something they could slide it into.

Andrew: Great, something that has mobile genes that you can cut out and just slide the paper through with the different screens. That sounds great. Yes, this table here?

PARTICIPANT: Well, we should, a lot of these are research. We didn’t get into the desig part, but we’re using a new data roll as data journalism roll at the Washington Post as a sort of study, case study, to think about sort of like what the kinds of things—what we want it and where the pipelines should be, between data research and—… Andrew: Great, what the pipelines would look like. The table in front of you.

PARTICIPANT: The only thing that we came up with was a sticky note that says clip me for user research. Like Microsoft Clippy.

PARTICIPANT: I’m going to repeat that because it’s so fantastic. Clippy for user research.

PARTICIPANT: We talked about maybe making templates for finding out how to get buy-in for leadership or management to for user research.

PARTICIPANT: Templates fo designing user research. Yes, the table here?

PARTICIPANT: We asked why five times, to try to get to the root cause of the root issue that kind of takes you back to you’re like, why did that happen? Why did that happen? Why did that happen? To identify what really caused this –

Andrew: Great, just asking people to ask why five times. And at the back?

PARTICIPANT: Uh, people who aren’t necessarily in the newsroom, but are in marketing can come up with a different part within the organizations. So we had journalists at the board, find a specific article or topic, and then document how many clicks it took. What platforms did they use. Was it Twitter, mobile, desktop? And then what were the search terms that they used? And we—…

Andrew: Great just have a nonjournallist user to get to the article and how would they do it and how many clicks would it take and what methods would they use?

PARTICIPANT: And based on that do a content and design audit.

Andrew: Yeah, fantastic, test the assumptions as David said. We’ve really just scratched a surface with another hour or so we would have gotten much deeper and much more depth in terms of that. I think this is exactly what we want from this. It’s a quick and dirty, some ideas to help you. We will then, what we’ll do is we’ll pull it all together and put a LinkedIn the etherpad and also circulate it around online, SRCCON will tweet it out and share it, too, if you want to leave your email address, too, inside the etherpad, we can send you link to it, if you’re interested in developing this further, let us know. This is a start of a conversation of how we can get more people in the newsroom, understanding the value of research and also participating in it, and this is just the beginning.

Emily: Thank you all for coming. Hope you’ll stay. Give yourselves a big round of applause.

[applause]