5. Methods – The CUTGroup Book

In our discussion of the components of the CUTGroup, we were careful not to be prescriptive, but we were able to lay down some markers around what we think are the minimum elements for a viable Civic User Testing Group.

In our chapter listing the tools of the CUTGroup, we showed that it took very little money to start a program, and much of the necessary material was laying around any civic tech operation.

As you are considering whether and how to implement a CUTGroup program near you, we urge you to consider using the suggested methods we describe below.

Recruitment

The most fundamental concept behind recruitment of testers is the idea of individual relationships. The power of the Internet is that 1:1 relationships that are endlessly formed and re-formed in our experience of the Web.

Our testers come from every ward in the city because we spend the time and make the effort it takes to do that. We’ve done flyering campaigns at all twelve City Colleges of Chicago. We analyzed our signups and did flyering campaigns at the 25 public libraries where our efforts were lagging. Then we narrowed down our efforts to just the 10 libraries in the two wards where we still couldn’t get signups. We did a mass email campaign to more than 10,000 subscribers of Nightlife magazine.

In community engagement, there are lots of high-quality organizations made up of hundreds of regular residents, and the temptation is to work with those groups to sign up CUTGroup testers in one swoop.

I urge you to take the long way around—recruit individual testers, on your own, to start off an independent relationship with each tester.

The power of the Visa gift card

Once a resident signs up to be part of the CUTGroup, we send them a $5 Visa gift card. If and when they are chosen to test a civic app, we give them a $20 Visa gift card.

The Visa gift card is the most essential element of recruitment. There are all sorts of reasons why people might join the CUTGroup—to get involved with civic hackers, give back to their community, or just plain get out of the house. But money is a good motivator, and respects people for their time.

We chose this incentive very deliberately. It is the most open, accessible, fungible incentive we could give—the one most akin to cash. Unlike an Amazon gift card, it is useful in the real world and can be used by walking outside and going into a corner store. It’s easily transferrable and acts like cash.

Also, the gift card gives us a good way to do a simple validation that the tester actually lives (or at least has access to a mailbox) in the area we’re interested in—Chicago.

When purchasing gift cards for your program, there are four main considerations: type of cards, cost and fees associated per card, quantity, and expiration date.

There are different types of gift cards out there: prepaid Visa or MasterCard gift cards, store-issued, bank-issued, online, etc.

Other gift card types also include specific store- or bank-issued gift cards. Sites like ScripSmart can provide comparisons between gift cards, and they can give you an idea of what you need to ask about before purchasing your own cards.

By purchasing these cards, we are spending more than face value on fees, and have to take time to mail them out, but the value of accessible gift cards is worthwhile for the goals of our organization and this program.

That said, we have done a lot to cut our costs in managing the gift cards. When Sonja Marziano joined the team in September 2013, she cut costs from $10.17 per card to $7.07 per card. Here’s a breakdown of costs with our current gift card vendor, Awards2Go Visa Award Card:

• Face value of card: $5.00
• Card processing fee: $1.75
• Credit card order fee (1% of total order): $0.05 • Shipping fee for 100: $27.00
• Total: $7.07 per card

Once you’ve got a start, build on the existing network

Last October, we learned that we still had a lot of gift cards that were about to expire and some were already expired. With the gift cards that expired, we lost out because the cost of the fees to “restock” the cards would be higher than the value of the cards we would receive.

We also had 118 $20 and 103 $5 gift cards that were going to expire at the end of November. If we sent back these cards to the vendor, we would only receive half the value of the card in return.

So we had to think up some creative ways to use the cards. The most successful campaign was “Refer a friend”. Here was the pitch:

Hi <<First Name>>,
We have some $5 Visa gift cards that are about to expire at the end of the month and we need your help! We want to make sure these gift cards are used, so if you refer a friend to join the Civic User Testing Group (CUTGroup) by the end of the day this Thursday, we will send you another $5 Visa gift card! Forward this email to your friends and have them complete this form.

How it works:

We will send you a $5 Visa gift card if your friend (maximum of 4 friends) completes this form and lists your email in the “Referred by” field. Your friends also get a $5 Visa gift card for signing up.
If more than 4 of your friends sign up, we can only provide up to a $20 Visa gift card, but everyone you refer will still get a $5 Visa gift card just for signing up.
It’s important everyone signs up by the end of the day of Thursday, November21. Once you get your gift card, be sure to use it right away!Thanks for being a member of the CUTGroup! As usual, call or write with questions.

We signed up 100 new users through this campaign.

We now purchase gift cards with long lives (8 years!), though the value decreases from the card at 13 months after purchase, so you still have to carefully manage your gift cards.

Recruiting developers

One of the central tenets of the CUTGroup is that developers are a part of the process from start to finish. That’s why we never start planning a CUTGroup test unless the developers of the product ask us to do so. This ensures that they are willing to learn from talking directly with users and that they will make changes to the system when the results are in.

We engage developers on an ongoing basis by organizing, attending, and presenting at civic hack nights, and being an engaged part of Chicago’s development community. For example, we are founding members of
1871, Chicago’s digital co-working space, and often have seats available for developers and other thinkers interested in civic innovation. Nothing can replace genuine engagement in your local development community, and a CUTGroup can be an important part (or start) of that engagement.

Design

Once we have a developer who wants to work with us, we set up a meeting to learn more about the product, decide how to segment the CUTGroup tester base so that we put together the best group for test night, and design the test itself.

Questions we ask at this stage:

What day do we want to do the test, and do we have a general part of town where we want to do it? This is based on availability of the developers and Smart Chicago staff. When it comes to location, we check to see where the most recent tests were conducted, and we endeavor to target a part of town where we haven’t been in a while. There are some instances where an app is more applicable to a particular area as well. (We once did a test on an app focused on the retail environment of a particular neighborhood, for example.)
What is the current stage of the app? Is it a paper prototype, is it in beta, or does it have thousands of users? This basic information gives us a start that helps the entire process. We’ve worked with apps in each of these states of being.
Who do we want to target? Do they have very specific requirements for the test? For instance, we needed to find parents whose children attended a Chicago Public School and who took personal responsibility for getting those children to school in the morning.
Potential screening questions to ask. For each test, we fashion some specific questions that help us screen testers to make sure we have a room of relevant people on test night. Examples are, “Do you write reviews on Yelp?” for a restaurant inspection site or, “Do you know your Alderman’s first name?” for a neighborhood news site. People who post on Twitter might be more likely to post on other forums.
What type of test? We employ a number of test types to get the information developers need: in-person test, remote test, and focus group are the main types.
Preliminary questions in the test itself: We also try to design questions for the test itself, starting with some framing questions. Sometimes it’s an open question (“How do you use maps?”). For others, it’s more specific (“Do you attend any of these types of meetings?”, with a list of six types).
Questions that are specific to the app: This is where we get to the heart of the user interface questions that the developer wants answered.
Additional goals from the developer: This is where we cover the full circle of digital skills for the developer—what else do they want to get out of this test?

Here’s a portion of an example test plan, done for Foodborne Chicago:

Who are current users of Foodborne?
http://blog.corynissen.com/2013/11/mapping-foodborne-chicago- reports.html
Who do we want to target for this CUTGroup test?
We will be targeting Twitter users (with at least 100 tweets)—both very heavy and not as heavy users
We are looking to get a mixed group from all areas of Chicago—we want to have approximately 2/3 of the group from the South Side, 1/3 of the group from the North and West sides

We will ask for Twitter handles (not a required field)
Types of screening questions to ask to gather more information:• Do you have a Twitter account? (required) • How often do you use Twitter?
• How often do you go out to eat in a week?
When creating our group, we will consider neighborhoods, Twitter usage, if they eat out, and if they have ever used 311 (mix of yes/no responses)What type of test?
- Three sessions in a focus group style with small groups (5 people). We will be able to bring up pages and views to show the groups and get their feedback
- This is a qualitative test, since there will be a quantitative survey coming out later
- We will record the sessionsWhat do we want to ask? (Please note that these are just a rough outline of the type of questions we’ll ask)
- How do you currently use Twitter? Do you tweet often? What types of things do you tweet about?
- If you got food poisoning, would you tweet about it? Would you tell people you’re sick? Why or why not?
- If you get food poisoning, what do you do? Do you tell someone about it? Do you submit a 311 request?
- Show: Foodborne Twitter account and gather feedback about the tweet- er and the tweet
- Show: Foodborne Chicago form page and gather general impression— What do you think this form does? How do you feel about this information?
- Discuss form information: Is this something you would fill out? Why or why not? Do you feel comfortable with giving this information? Do you trust this site?
- Do you know that this information is being sent to the Chicago Department of Public Health (CDPH)? How do you feel about that?

Show: Q&A page—see how people feel about the request being made into a 311 request and the service tracker—this information might need to be more prominent elsewhere, this is where we will get more information about 311
What would make you more willing to respond to a random tweet? Followup: Would a tweet from a person in the community be better?
What would make you more willing to fill out this form?
Do you like Foodborne Chicago? Why or why not?
Do you think Foodborne Chicago is an effective way to report food poisoning? Why or why not?What does Foodborne want to learn from the CUTGroup test?
Foodborne wants to learn why South Siders are not responding
How do residents feel about responding to a random tweet?
Is the form too much? What fields might not need to be required?
Would users feel better/worse if there were more CDPH or 311 visibility? (In terms of Twitter handles, domain names, and logos)SegmentingOnce we’ve got a rough plan for the test, it’s time to gather a relevant set of testers. We do this through MailChimp email campaigns integrated with Wu- foo surveys in a three-step process:
- A broad call-out to people giving them a general idea of the test type and asking testers their availability for a two-hour time period on the test date and any specific requirements of the test
- An email discussing the specifics of the test, including location and test type, and asking them to affirm their availability, along with a preferred 20-minute slot in the context of the entire test period
- Emails to the set of people who did not get segmented into the testThe final steps in segmenting are done through a regular email client (not through the MailChimp mass email tool). We hand-schedule the testers into slots based on the tester’s choice, the number of testers, and test type. This is a great example of a task we’d like to automate with our Patterns software— we just haven’t gotten around to it.

Here are some sample emails and surveys you can use in planning your own emails to testers:

Broad call-out, initial email

Subject line: Make $20 at an in-person test of a food poisoning app. Hi <<First Name>>
We’ve got a new opportunity for you to make money in the Civic User Testing Group (CUTGroup) by testing an app.
This app helps report food poisoning incidents to the Chicago Department of Public Health. We want to know if this app serves your needs, and how it can be improved.
Are you available for a 30-minute, in-person test on Monday, April 14, between 4:00 and 7:30 p.m.? If so, you qualify. Complete this form to start the process.
We are in the process of gathering responses, so we will be in touch to check availability and let you know if you have been chosen. We are look- ing for 15-20 testers for this test.
For your participation, you will receive a $20 Visa gift card. You’ll also help make better software for Chicago.
Thanks for being a member of the CUTGroup! As usual, call or write with questions.
Remember: if you want to be a part of this test, please complete this form.

Specifics of the test email

Hi <<First Name>>,
Thanks for your interest in our CUTGroup test of a food poisoning app. Now that we have our location nailed down, we’d like to know if you can still make it next Monday and, if so, what time you want.
Here are the details:
Monday, April 14, 2014
Between 4 and 7:30 p.m.
Blackstone Branch of the Chicago Public Library 4904 S. Lake Park Avenue
If you can still make it on this night, please let us know what time slot you want by completing this form. For your participation, you will receive a $20 Visa gift card.

There are limited slots available for this night, and it’s coming up fast, so please let us know as soon as you can. We will get back to you with a confirmation if you are chosen to do this test.
If this location doesn’t work for you, or if you just can’t make it on this night after all, no big deal. If you have any questions or comments, just hit “reply” and let me know what you think.
Remember: use this form to tell us about your availability.
Thanks for being a member of the CUTGroup!

Did not get segmented email

Hi <<First Name>>,
I wanted to follow up with you about testing a food poisoning app Foodborne Chicago. We really appreciate your response to our call out.
Foodborne Chicago searches Twitter for tweets about food poisoning and then responds to those tweets with a link to report it so that the Chicago

Methods 27

Department of Public Health can take any necessary action. Therefore, for this test, we were looking for heavy users of Twitter.
Since you indicated that you don’t have a Twitter account, this just was not the right test for you.
We will send you info on our next test (should be in May or June), and we’ll definitely try to spread the tests around so that everyone gets what they want out of the CUTGroup experience. We do keep track of these things!
Thanks for being a member of the CUTGroup. As usual, call or write with questions.

The more we ask of testers, the smarter we get about testing

Overall, our strategy is to engage testers over multiple emails and gather new information. We develop deeper relationships and have more information that allows us to segment for new tests.

Our open rate for the blast availability emails, which are sent to all
active CUTGroup users, regularly comes in at 45-55%, and we have a very
low unsubscribe rate. People in the CUTGroup look forward to getting emails from us.

Now we have testers, a developer and goals for our tests. Where should we hold the test? What do we do once we get there?

Scouting

An absolutely essential criteria is that CUTGroup tests take place all over town. To decide where to hold our next test, we start with the list of Connect Chicago locations. Chicago Public Library has great, accessible, and free meeting room space for community organizations.

Much of the test location planning is classic event planning—it’s important to make sure that librarians, patrons, security guards, and anyone else we encounter during the test are fully informed and comfortable about our presence.

Proctoring

We use a mix of test administration styles in the CUTGroup: direct test proctoring, focus group, remote, and self-driven/in-person.

Direct test proctoring

This is the most common test style we use in the CUTGroup. Most often, we pair every tester with a single proctor who works with the tester to complete the test. Sometimes, based on the vagaries of scheduling, a single proctor will work with 2 or more testers at the same time.

Focus Group

Sometimes we can gather more intelligence about an issue in a focus group, which is great for getting qualitative information about a topic. We first have testers answer a survey, so that they have a chance to form their own opinions before the group discussion. Next, we engage them as a group—they all share their answers and we ask additional questions.

Remote

We always value in-person tests and believe they are an opportunity to convene participants from all areas of Chicago. We gain valuable responses in person and are able to record a tester’s actions and reactions. But remote testing can give us the sheer quantity of answers that can put an exclamation point on test results.

Self-driven/in-person

These tests were created so that no proctors were needed. We designed this type of test before CUTGroup #4, EatSafe.co. It is an example of a test that had lots of testers, and some testers did not have a proctor with them at the time.

Analysis

After the test is done, we analyze the results and publish all of the raw test data as well. We export all of the results out of Wufoo into a spreadsheet,
then use a Microsoft Word macro to populate an easy-to-scan document of the raw results.

The analysis is done by hand by the main proctor of the test. Sonja Marziano has done the majority of analyses for our CUTGroup tests, and she designed the method for exporting and formatting all test results.

Since the person writing the analysis was also present at the test itself, she is able to take the quantitative information and add qualitative insights so as to get actionable recommendations for the developers.

Followup

The interface recommendations are published in the body of our analysis blog post and we almost always create Github issues as well. This ensures that we create concrete, direct recommendations—nothing squishy or weak.

What happens next is up to the developer!

Remember, a lot more can be found on our website and blog at smartchicagocollaborative.org. We’ve posted in-depth info and documentation on what we’ve learned.