The Code For America Qualitative H&S § 11361.9 Research .
The Code for AmericaH&S § 11361.9QualitativeImplementation BlueprintResearchPracticeGuideA service delivered by Code for AmericaSpring 2020
WelcomeCode for America’s Qualitative Research Practice Guide is a statement from our qualitativeresearch team of how we approach qualitative research, why we believe research is critical to theeffective delivery of government services, and how you can engage with our research practice.Through our work in government service delivery, we have learned how to conduct research inways that can help everyone ensure that their products and services are as inclusive as possible.This guide touches on everything from Code for America’s core research philosophy, toour approach to ethics and trauma-informed research, to specific methods for conductingresearch and analysis. We have also included plenty of practical tips on planning andexecuting research, as well as suggestions on how to synthesize your findings into action.This is version 1 of the guide. These pages are a snapshot of our research team’s work and theapproaches we use to lift up the voices of people as we work towards a world where governmentis oriented around an understanding of the needs of those it serves. As we continue this work,we hope to keep learning and sharing what we’ve learned with the broader research community.As part of that learning process, we would love your feedback. Please share any thoughts orsuggestions you (or your team) have about this guide with us at research@codeforamerica.org.Who is this guide for?Government agenciesand service providersResearchersand designersPeople engaged in civic techand social impact workAt Code for America, we want toinspire public servants to drivechange by developing better waysto deliver government servicesand sharing our learnings. Webelieve that when governmentis committed to centeringpeople in the development andimplementation of technology,operations, and policy, it willbe best equipped to servethe American public.Qualitative researcher, userresearcher, UX designer, howeveryou identify, we are all part of agrowing and evolving field. AtCode for America, we want tolearn from our researchcolleagues across the industryfrom community organizers totech to academia.Our civic tech and social impactcolleagues across all levels ofgovernment in the US and abroad,and at nonprofits across theworld, are in partnership with usto make change and to advocatefor human-centered practices.We encourage all governmentworkers and agencies to learnabout how we work, to adoptmethods from our approach, andto hire qualitative researchersin their own offices.By being transparent about ourown approach and processeswe hope to contribute toconversations and growthin our field.Together we can build a movementto push the work farther and fasterthan we ever could on our own.2
ContentsWelcome 2Research methodology & principles 4Research philosophy 5Guiding principles 6A note about language 7Research operations 8Ethics & safety 9When to ask for research 14Planning research 15Incentives 19Researching with non-researchers 20Qualitative research methods 22In-depth interviews 23Observation/shadowing 27Journey mapping & service blueprinting 30Intercept testing 34Concept testing 36Usability testing 38Making sense of qualitative data 42Organizing & storing data 43Analyzing & synthesizing data 453
Researchmethodology& principlesCode for America’s research methodology andguiding principles frame why research is critical toour design process. They are the foundation for ouruse of the research methods and tools in this guide.4
Research philosophyCode for America’s mission is to develop toolsand services that help government work forthe people, by the people, in the digital age.The people we serve often belong to vulnerablecommunities, and their voices are not alwaysheard in the context of government servicedelivery. The goal of qualitative research atCode for America is to infuse the voices ofthe people we serve into our process and,ultimately, the products and services we create.Our research strives to see people as completehuman beings acting within a broadercommunity rather than just as isolated “users”of our products and services. Beyond productrefinement and testing, our research seeksa broad exploration of a group of peopleand their beliefs, values, and desires so thatthose findings can be a jumping off pointfor innovative and life-changing products.From interviews and in-person observations atthe start of a project, to creating journey mapsthat display needs and pain points, all the waythrough to usability testing, qualitative researchensures that we build the right thing at theright time for the people who need it most.We cultivate a “research mindset” in ourengagement with research participantsand in the analysis of their experiences.Our research mindset is humble,comfortable with uncertainty, and opento anything we might discover.When we conduct research we are nothunting for interesting anecdotes or seedingquotes for a slide deck—we are answeringreal questions in order to provide actionableguidance that will push projects forward.The point of research is to inspire action to buildbetter products that solve real problems. Thebest-planned, most insightful research is worthlittle if the problems it finds are never fixed,or the opportunities it reveals go unrealized.Ultimately, research is a tide that lifts allboats. It is fundamental to developinggovernment services that better and moreequitably meet the needs of communities.Raising the bar on quality of research raisesthe bar on quality and effectiveness foreverything that we seek to do for the world.5
Guiding principlesWe strive to be humble,trauma-informed, andexperience-ledWe practice deep listening in order to treat everyindividual’s lived experience with care and to ensurethey feel heard. We cultivate humility, knowingthat even if we have subject matter knowledge in aproblem space, we are not the experts on anyone’slife. We respond empathetically to a person’s needsand concerns without explaining why technologicalcapacity or government bureaucracy might limitour ability to implement a remedy right away.People lead the wayWe believe that the experiences and needsof the people we serve should drive productdecisions and organizational vision.People are not defined by theproducts and services they useWe believe that everyone we serve has needsand experiences that exist outside of just theproduct or service that we’re working on.Research gives us the unique opportunity tothink beyond the boundaries of organizationalstructure and government silos to discover newopportunities that serve our population best.We ask big questions bystarting smallWe believe in creating systems changethrough building an understandingof individual experiences.Research is a shared valueand a respected practiceWe believe that design, engineering, policy,marketing, development, operations, andleadership all benefit from proximity to thepeople we serve and their stories. We shouldall hear their voices often and firsthand.While everyone should regularly participatein research, our skilled research staff are bestequipped to plan and guide it. Rigorousresearch practices and protocols ensure validdata which help generate the most impact.We care for ourselves and eachother with the same respectwe give our participantsWe believe that community care for researchersis an ethical imperative. We care for ourselves andour fellow researchers in order to be able to bestcare for, serve, and respect our participants. Weacknowledge the challenges of this work by givingourselves breaks and protecting each other’stime to process. Creating an interdependentcommunity of researchers ensures we can showup fully for each other and for our participants.6
A note about languageWhen communicating with participants and representing our research findings, we intentionallyuse language that reflects our values. Here are some ways that we try to do this:Mirror the language thatthe person usesWhenever possible, it is important to mirrorthe language that people use to talk aboutthemselves and their situations. Whereverpossible, enable research participants totalk about and define themselves usingtheir own terms. This builds trust andcompassion and allows them to haveagency over their own experiences.Use “people” over “user”We believe in using the word “people” to talkabout those we are trying to serve. This ensuresthat we view people as more than those who useour services or products, but as multifacetedhuman beings that have myriad experiences.It allows us to think holistically about thepeople we serve. Often, in the servicesthat we work in, there is not one singular“user,” or our “user” is not the one directlyinteracting with the service, and using “people”ensures we acknowledge that distinction.Embrace people-first languagePeople-first language centers people ashuman beings rather than as their particularcircumstances or experiences. Thislanguage acknowledges that one experienceor situation does not define a person’swhole life. Some examples of people-firstlanguage include using “people experiencinghomelessness” rather than “homeless people”or “people living with convictions” ratherthan “convicted people,” “convicts,” or“former prisoners.” Language is tricky andalways evolving, so if you are unsure whatto say it’s okay to ask another researcheror, better yet, the person themself.Use a person’s voice when sharingtheir experiencesWhen we communicate research findings,we try not to paraphrase what we thoughtwe heard, especially when communicatingan individual’s stories. Instead, we usedirect quotes where possible, with explicitpermission from the participant aboutwhich quote is going to be used and how.7
ResearchoperationsResearch operations includes our commitmentand guidelines for ethics and safety, how to bringteammates into the field, data storage, writing aresearch plan, recruiting participants, and when toask for research.8
Ethics & safetyCode for America’s qualitative research practice is grounded in creating safe and ethicalenvironments both for ourselves and for our research participants. The following are generalsafety and ethics guidelines that we believe should be followed in all research engagements.We encourage you to discuss safety and ethics openly with your team at the beginning ofeach project and update or add to this list as necessary.Research ethicsUnderstand and address power dynamicsWhen conducting research, you should be conscious of the power dynamics betweenyourself as the researcher and research participants. In the role of researcher, you set the toneand assert power dominance, intentionally or not, by establishing a hierarchy where you askthe questions and the participant is compelled to answer them. We grapple with these powerdynamics both in our drive towards equity and inclusion and as to not bias research findings.While we can never truly erase the power dynamics we bring into the room, we do haveseveral practices designed to acknowledge them: We provide incentives at the beginning of a research session right after the participant hassigned a consent form so people feel comfortable leaving at any time. For more on ourincentives practices, see “Incentives” on page 19. We dress for research engagements to mirror our participants in casual,approachable clothes. We use non-threatening body language and room set-ups. We set the expectation with research participants that there is no right or wrong answer,that their stories are valid, and that we come to research humbly, to learn from them.When bringing prototypes or products to get feedback on, we find ways to demonstratethat these are works in progress and we encourage honest criticism. We talk openly and vulnerably about how power manifests in the work we do. To exploremore about power in design research, read this medium post by George Aye of GreaterGood Studio: Design Education’s Big Gap: Understanding the Role of Power.Set realistic expectationsWhen beginning any research with a participant, explicitly lay the groundwork of what willhappen during and after the conversation. Participants should understand what sorts of9
Ethics & safetyquestions they are expected to answer and why those questions are being asked. Researchersshould be clear about what the outcome of the study is and how the participant’s input willbe used to further the project.Obtain informed consentCode for America’s consent forms explicitly speak to the purpose of the study as well ashow information and data will be shared and used. When obtaining consent, we highlightthe specific aspects of the consent form that may be relevant and are explicit that theparticipant can stop the conversation at any point. We allow participants to opt in to differentdocumentation methods (e.g. audio recording, photos, or videos) and explain how thoseartifacts will be shared or used. Obtaining consent is asking for ownership of their words,stories, pictures. Therefore, we do not push participants to consent to something they areuncomfortable sharing and make sure they receive their incentive whether they complete theinterview or not.If your organization doesn’t have a consent form, download our starter template.Make best use of participant’s timeAs you recruit and screen research participants, confirm that each person is of the populationyou are trying to target to ensure that you are best using everyone’s time. Pick an interviewlength that optimizes for time. You want research conversations to be long enough to buildrapport, but not too long as to exhaust both the researcher and the participant. Additionally,always be conscious about what questions are most important to be answered during theinterviews to ensure you effectively manage everyone’s time.Acknowledge and notice biasesWhile it is impossible for us to be able to remove all biases, we need to notice andacknowledge when biases may be affecting our interactions with participants. This helps tounderstand how biases may change our research outcomes. Over time, using what we learnfrom those self-evaluations we can work to decrease the role personal biases play in ourwork. This process can take the form of having frank conversations with the team about theassumptions you have entering a research project and periodically checking in on how thosebiases might be manifesting in the work.Be trauma-informed when engaging with participantsParticipants that we work with may have experienced significant trauma, especially as itrelates to the services and programs that we are trying to redesign or reform. As such, it’simportant to be trained to recognize signs of trauma (e.g. body language, tone, etc.) andrespond (e.g. offering space, cold water, time to breathe) in a way that does not retraumatize10
Ethics & safetyparticipants. Being trauma-informed also means giving participants agency and choice overwhat activities they participate in and what topics they talk about. Make sure participantsunderstand that they have control to stop a research session at any time, for any reasonwithout any repercussions. We must also recognize what information is actually useful andnot ask participants to open up in ways that may be retraumatizing.Read more about Code for America’s approach to trauma-informed research in our criminaljustice work. For more in-depth guidance on accounting for trauma in research activities,see the Substance Abuse and Mental Health Services Administration (SAMHSA) manual onTrauma and Trauma-Informed Approaches.Listen and empathize, but don’t offer adviceThe research team acknowledges that we are not experts in providing services for every fieldthat we conduct research in. It is not our role to offer professional advice to participants, evenif we believe we know what the right approach is. The reality of our work is that we are oftentalking to people who are asking for assistance with their benefits application, their tax filing,or something else entirely. This may mean that they are seeking assistance with a governmentprogram that we are working within. In cases where it is applicable, we may refer participantsto service providers, but ultimately the researcher’s job is to listen, not to problem solve.Represent participants’ information honestlyWhen presenting or sharing out what you learned from your research, information andquotes gathered from participants should represent what participants actually said (i.e.direct quotes from people shared in their appropriate context, not cobbled together phrasesor broad quotes applied to different situations). When we speak about a participant’sexperience we are careful and thoughtful about placing that experience in context to ensurethat we represent people as accurately as possible. We don’t look through data for quotesthat confirm what we want to hear. We are open to what was said, even if it contradicts ourhypotheses or criticizes a product direction. In reports, we are intentional not to hide ormislead findings—reports should be as close to what we heard and observed as possible.Keep participants’ data safe and anonymousAfter research has been conducted, ensure that participants’ personal information and data iskept securely and anonymized so that it cannot be connected back to individuals. This doesnot just mean changing names, but also any other personal identifying information that mayallow someone to discern who a participant is from their unique data or story. For more onthis see “Organizing & storing data” on page 43.11
Ethics & safetyStaying safe as a researcherThis work can be exhausting and draining, and ensuring that both researchers andparticipants are cared for is especially important for the sustainability and longevity of aresearch team.Prepare researchers and participants for a comfortable andsafe conversationWhen scheduling an interview, consider safety and ethics for both the interviewer(s) and theparticipants. In-person interviews should always be conducted by staff in pairs (or more).Interviews should be conducted in a neutral location such as a library or a coffee shop and, ifpossible, in a private room, especially if sensitive information is going to be discussed.Protect researcher’s personal contact informationIt is often our job to text and call research participants with interview reminders or toconduct phone interviews. It is our practice to not distribute a researcher’s personalphone number and to conduct all participant communication through a third-partycommunications platform such as Google Voice (or Front, Intercom, ZenDesk, or similar).Choosing a location for in-person researchWhen scheduling in-person research, it is our practice to schedule around the researchparticipant’s needs, which often means being invited to a participant’s house or to acoffee shop or restaurant nearby. We do not want to force research participants to make apotentially difficult trip for the convenience of the research team, but we also don’t wantresearchers to put themselves in situations that feel unsafe or insecure.If you, as a researcher, have been invited and are comfortable visiting a participant’s homeyou must (1) bring another researcher, staff member, or partner along and (2) tell someoneelse where you’ll be and at what time.It is your prerogative as a researcher to end any research session at any time that you feeldiscomfort (whether that session is in a participant’s home or a McDonald’s or a library). Ifsomething about the interaction does not feel right, that’s okay. Politely end the interview,compensate the participant for their time, and leave.Cultivate self-care practicesWe care for ourselves and each other with the same respect we give our participants—webelieve in the importance of researcher care particularly during long rounds of on-theground research. It is natural to feel strong emotions while condu
Research Practice Guide. 2 Code for America’s Qualitative Research Practice Guide is a statement from our qualitative research team of how we approach qualitative research, why we believe research is critical to the effective delivery of government services, and how you can engage with our research practice.
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
Bruksanvisning för bilstereo . Bruksanvisning for bilstereo . Instrukcja obsługi samochodowego odtwarzacza stereo . Operating Instructions for Car Stereo . 610-104 . SV . Bruksanvisning i original
Chính Văn.- Còn đức Thế tôn thì tuệ giác cực kỳ trong sạch 8: hiện hành bất nhị 9, đạt đến vô tướng 10, đứng vào chỗ đứng của các đức Thế tôn 11, thể hiện tính bình đẳng của các Ngài, đến chỗ không còn chướng ngại 12, giáo pháp không thể khuynh đảo, tâm thức không bị cản trở, cái được
10 tips och tricks för att lyckas med ert sap-projekt 20 SAPSANYTT 2/2015 De flesta projektledare känner säkert till Cobb’s paradox. Martin Cobb verkade som CIO för sekretariatet för Treasury Board of Canada 1995 då han ställde frågan
