Author: DXO

6. Examples

Post author By DXO
Post date

Based on how we’ve put together this book, we may have given you the idea that we’ve put our methods into practice in an utterly linear and structured way. This would be giving you the wrong impression.

In reality, the CUTGroup is just one of many programs we have going at any given time. As you can see from the Acknowledgements, we’ve worked with dozens of experts giving us bits of advice and helping us figure out what we’re doing.

The best way we can show you how we got to where we are is to directly show you how our methods met practice. We cover each of the tests we’ve done below and talk briefly about the impact each had on our methods.

For each of the examples, we have a complete set of results published on the Smart Chicago website. This includes links to every piece of communication we’ve sent to testers, details on all test objectives, and the raw results of every test.

#1 FreedomPop Router

The first test we ever did was also the one that was least like the others. We spent two months recruiting testers, and we were ready for our first test. By this time, the City of Chicago was looking to launch a pilot with FreedomPop, but they weren’t sure if the 4G routers they offered would be of acceptable quality. Since expanding access to the Internet is one of the core missions of Smart Chicago, we decided to come up with a process to use the CUTGroup to test the hardware, customer workflow, and coverage for this product.

Segmenting was important in this test, because the product was only available at this price in certain ZIP codes of the city. At the end of March and beginning of April 2013, two emails were sent to CUTGroup members in the twenty ZIP codes where the lowest-cost EveryoneOn option was available.

We had 94 possible testers, and 8 people signed up for the test through what would soon become our standard process for recruiting and segmenting. Testers indicated that unboxing the device was a breeze and that it worked well. We were able to determine that the service met expectations by engaging with people over email, home delivery, and a web form. The system worked, but we knew we had to get people together in a room to start delivering on the promise of community engagement.

We had to draw testers from just 20 Chicago ZIP codes.

The Freedom Pop 4G routers — The Freedom Pop 4G Routers

References

#2 – Go to School! User Application Test

This test centered around a website that promised “Four steps get to your CPS school on time.” It is a simple wayfinding tool with school start times and contact information baked in. We learned two things from our first in-person test: how to segment users for a complex requirement and the power of the “shared screen” in community engagement.

We had to do a significant amount of segmenting of our list of 368 CUTGroup members. We asked all of them two questions:

Are you responsible for taking one or more children to a Chicago public school this fall?
Are you available for testing on either May 28 or May 30?

There were 28 people who responded to these requirements. Based on the locations of these people, we decided to hold tests at two libraries: the Kelly Library in the Englewood neighborhood on Tuesday, May 28th, and at the Uptown Library in the Chicago Uptown neighborhood on Thursday, May 30th. There were 14 people who expressed interested in each location, so it worked it really well.

The testers were spot-on knowledgable about the topic of getting kids to school on time. In an effort to maintain privacy and to provide the widest pos- sible utility of our testing to civic developers everywhere, we developed some shorthand “personas” that helped in identifying their stance vis-a-vis the app.

This was the first time I personally came into contact with the idea that the CUTGroup was a great framework for discussing difficult topics in unemo- tional and (hopefully, ultimately) helpful ways. I proctored “Tester #2: Father responsible for three children (17, 16, and 10) Samsung Galaxy Kelly Library, Englewood.” The topic of school closings and “safe passage” was very active and contentious. But in the context of reviewing this site, he was able to express in everyday terms the choices he made in terms of danger associated with certain routes.

This is the joy and the promise of the CUTGroup. We end up using a “shared screen”—a concrete, tangible interface that everyone can point to, and talk about—to serve as a common ground for discussion of difficult topics that can affect policy.

On a more prosaic note, we were able to uncover trouble with the custom time/date picker screen. People didn’t understand how to work it. The developer, Tom Kompare, made changes to the picker based on the feedback.

This test was an early example of the kind of broad geographical range we can deliver in our tests.

Julie Harpring proctors a test while Chris Gansen takes notes.

References

#3 – Chicago Health Atlas

The Chicago Health Atlas is a place where you can view citywide information about health trends and take action near you to improve your own health. The test took place at the Erie Family Health Center, a Smart Health Center in Humboldt Park.

We wanted to see how everyday residents were using it and make sure that people could find what they were looking for on the site.

The test revealed a number of user interface issues, mostly revolving around a key feature: the ability to see healthcare resources near the user. We made some simple modifications to the site, mainly making text bigger, explaining the features more clearly, and showing links to resources as users were viewing the map. We’ve yet to re-test the site, but informal testing (watching people view the site) indicates that we’ve improved clarity.

The “View local resources” link was missed by many users.

References

#4 – EatSafe.co

Our fourth session was focused on EatSafe.co, a website that shows details of food inspections near you, which was developed by Hoy Publications. This in-person test took place at the Chicago Public Library’s Hall Branch at 4801 S. Michigan Ave in the Grand Boulevard neighborhood.

This was one of the more community-focused tests. We had overbooked the sessions, and lots of people showed up early. It was touchy for a while because we were struggling to match up proctors to testers. After we loosened up and accepted the fact that we had to double- and triple-up testers to proctors, people really enjoyed working together and talking about the website. Here’s what Hoy Managing Editor Fernando Diaz had to say:

“Partnering with the CUTGroup was the most effective research experience I’ve ever been a part of. We met real Chicagoans who were generous with their time and feedback. And among the highlights is that we have all of the results for further evaluation and incorporation into future iterations of our project.”

We got good, actionable feedback from testers. When clicking a violation, testers wanted an explanation of the violation. (The current function was that they were directed to a list of establishments with the same violation.) Testers were also interested in better explanations of the inspection results.

We had one particularly remarkable experience worth relaying, because it shows how civic apps fit into the real lives of residents. The tester was the mother of a child in a public school near her home. While browsing restau- rants, she saw that her child’s school had failed an inspection.

She said: “I didn’t hear anything about it. I didn’t know.” You could hear the frustration, and associate with it. We’ve all been in situations where a lack of information takes us aback, makes us feel powerless. She read the text of the violation, relating to washing facilities. “That’s not good. That’s not a good look at all,” she said.

I asked her what she wanted to do next, asking a classic UX testing nudge, drawing the tester back to the interface. She wanted to “contact the school, and find out what’s going on, because my son is eating lunch there.” There was no way to contact the school from the website.

Detail page for eatsafe.co — Detail page for inspection result. Testers had difficulties searching for a specific establishment and deciphering the results.

Daniel X. O’Neil observes a tester with piano and candy in background.

References

#5 – ChicagoWorksforYou.com

Our first remote test was for ChicagoWorksforYou.com, a Smart Chicago project. ChicagoWorksforYou provides citywide information about city service requests. Users are able to get a ward-by-ward view of service delivery in Chi- cago, learn about the top service requests made on a given day, view photos of requests, and learn more about the process of submitting service requests.

We did this test remotely because we wanted to get as many people as pos- sible, mainly because we were in a bind—we had to dispense gift cards that were about to expire, and the Thanksgiving holiday was coming up. But we also had a theory that the CUTGroup could serve as a form of advertising and marketing for a website and thereby increase the user base.

We started by sending out an email to all 565 CUTGroup participants, asking them if they would be available to provide feedback through a remote test. We received 116 responses in one day and asked 90 random respondents to do the test.

We asked more “yes” or “no” questions than we usually do, in order to gather quantitative results. We leavened this with open-ended questions to
see what users were interested in when visiting the site, and finally we asked users to click on specific links of the website, and discuss their experience. We were pleasantly surprised at the thoroughness of testers’ responses without a proctor being present.

In addition, we asked 5 willing testers to do a test via screen share. We randomly chose our respondents, compiling a group of testers from all areas of Chicago, and had a variety of responses to our questions. Due to technical problems, we were unable to do the screen share. We should have tested it more and provided more technical instructions on how to screen share. The other issue was more conceptual: Can we do the same kind of community engagement in a remote test, over a shared video connection, that we can do in a public computer center? We’re going to look into this more.

We promise in our release form that, “In any report we might publish, we will not include any information that will identify you.” So in order to keep track of testers, we ask them to provide a “tester profile”, which we describe as “a short phrase that describes yourself and/or your relationship to the software.” People have fun with the tester profiles.

Here are a couple of responses we heard from our testers specifically about the remote test:

“An online test is a better form of testing a website or app. It is done within the comfort of one’s home, with flexible times. It allows more people to participate and allows for a more natural environment.” –Tester #77, Elizabeth07
“I do like the remote survey better than the one I had to go to the library for. The particular public library I had to go to was in a very dangerous area and I didn’t know before I went… Other than that it has been a pretty good experience being a part of the CUTGroup thus far and I’m definitely willing to give my input on multiple websites.” Tester #46, 3rd Year Student

Here’s the complete text of our release form.

Cutgroup Consent To Participate In Usability Test
What is the purpose of this test?
We are asking you to participate in a usability test because we are trying to learn more about how people are likely to use the website you are being asked to test.
How much time will this take?
This study will take about 30 minutes of your time as follows: 5 minutes of discussion, 25 minutes for evaluating the site and 5 minutes of wrap-up.
What will I be asked to do if I agree to participate in this study?
If you agree to be in this study, you will be asked to perform basic interac- tions with the application under consideration.
The test session will be video recorded. Following the session, the video recordings may be analyzed by the CUTGroup in order to provide further feedback, and videos with voice and the website interface may be shared to educate others about website usability.

42 The CUTGroup

What are the risks involved in participating in this study?

Being in this study does not involve any risks other than what you would encounter in daily life interacting with a computer-based application. It is important that you understand that your information will remain confiden- tial during and after the testing session.

What are the benefits of my participation in this study?

You will not personally benefit from being in this study beyond the basic remuneration that has been offered. However, we hope that what we learn will help contribute to improving the quality of the applications being studied.

Can I decide not to participate? If so, are there other options?

Yes, you can choose not to participate. Even if you agree to be in the study now, you can change your mind later and leave the study. There will be no negative consequences if you decide not to participate or change your mind later.

How will the confidentiality of the research records be protected?

The records of this study will be kept confidential. In any report we might publish, we will not include any information that will identify you. Study records will be stored securely and only the CUTGroup will have access to the records that identify you by name. Some people might review our re- cords in order to make sure we are doing what we are supposed to. If they look at our records, they will keep your information confidential. Digital versions of all video recordings will be kept in password-protected files and will be destroyed within three years after start of the study.

Whom can I contact for more information?

CUTGroup: 312.565.2867
You will be given a copy of this information to keep for your records.

Statement of Consent:

I have read the above information. I have had all my questions answered. (Check one:)

I consent to be in this study. I DO NOT consent to be in this study. Signature: _______________________________ Date: ______________ Printed name: __________________________________________________

Examples 43

Personal Release for Filming:
I authorize the CUTGroup to take and use video recordings of me in connection with the usability study.
Signature: _______________________________ Date: ______________ Printed name: __________________________________________________

service request photo page of ChicagoWorksForYou — We discovered in the test that people loved seeing the service request photos, so we decided to bump that up in the interface and expand the section.

References

#6 – OpenStreetMap Editor

For our sixth test, we focused on the editor feature of OpenStreetMap.org. This in-person test took place at the Chicago Public Library’s Rogers Park Branch at 6907 N. Clark Street in the Rogers Park neighborhood.

OpenStreetMap (OSM) is a website that is built by a community of map- pers who contribute local knowledge and information to a map for everyone to use. Anyone can sign up, add information, and edit the map.

OSM is open data, and you are free to use it for any purpose as long as you credit OSM and its contributors. The main thing we tested was the ease of signing up and editing a map. This is of enormous interest to us at Smart Chicago, because we think that OSM is an opportunity for community mem- bers to describe their own streets, buildings, and assets with greater accuracy than anyone else. The fact that OSM runs millions of map experiences per day means that the more regular residents update OSM, the more communi- ties can put their best foot forward on the Web. The more comfortable people are in editing the map, the more accurately that communities can be shown on the map.

Sixteen testers provided their feedback regarding OSM, and we learned a lot. Not only about the functionality of the map editor, but also about peo- ple’s feelings on the concept of editing a map. Some testers liked the idea
of contributing knowledge for others to use, while other testers thought the concept of making live changes to a map was “scary” or “dangerous.” Here are a couple of telling comments we got:

“I believe in power of people and having a significant contribution to these things. Gives a sense of community and add value in the sense of belonging” – Tester #5, kirehernan
“I personally would but I would not want others to have the same access as I would due to the lack of restrictions.” – Tester #7, B

We also saw that the map scale posed challenges to the testers. Some became frustrated that when their search was not in the visible map area, they had to choose “Search Worldwide.”

When searching, testers sometimes received too many options that were irrelevant to what they were looking for, or they could not find their search term. Testers are interested in having a clearer way of searching for locations.

Fourteen testers (88%) said they liked the site, and 11 testers (69%) said they would use the map editor again. Here are some reasons why testers wanted to use OSM’s editor:

Update outdated data
Contribute to a neighborhood’s visibility
Feel the need to customize or make a map more personal • Add information about safe pedestrian and bike paths

Ma showing geographic range for a test — The geographic range for a test given in the harsh winter weather really surprised us.

Person proctoring a test — Ian Dees works with a tester at the Rogers Park Branch of the Chicago Public Library.

References

#7 – EveryBlock iPhone App

EveryBlock was coming back. The popular Chicago version of the neighbor- hood news and conversation website was relaunched in January 2014 by new owners, Comcast NBC Universal. They wanted to test their iPhone app—a vestige of the old site—to help them plan new features after their relaunch. I was part of the original team that launched EveryBlock in 2007 and had continued working in civic tech at Smart Chicago, so it was fun to get a chance to test a product I had worked on long ago but which still had relevance to me in civic tech.

This in-person test took place at the Chicago Public Library’s Mayfair Branch at 4400 W. Lawrence Avenue in the Mayfair neighborhood.

We sent out an email to 269 CUTGroup participants who said they had an iPhone as their primary or secondary device. We asked them if they would be willing to test a neighborhood app on February 10. We also asked some screening questions to gather extra information, and we chose our group of participants based on a diverse selection of answers.

We had 12 testers who came from neighborhoods across Chicago including Albany Park, Hermosa, Edgewater, Uptown, Logan Square, Auburn Gresham, and more. The farthest a tester traveled from their home location was 15.3 miles.

This in-person test was the first opportunity we had for every tester to be paired with a proctor. In previous tests, some testers were paired with proc- tors, while others would answer questions about the website or app through an online form.

One thing we tested was the propensity of testers to want to post via the Ev- eryBlock mobile experience, and 83% of testers said they would. The majority of testers thought this was a convenient option, and they wanted to comment on things while they are happening. We noted that posting from mobile was a far more common activity than even a year ago, when the site was shut down.

The biggest takeaway from this test is that users were interested in features which would allow them to have an experience that matched their experience on the EveryBlock website.

Screenshot of EveryBlock iphone app — A neighbor message used during the EveryBlock CUTGroup test.

EveryBlock promotional material — EveryBlock Chicago returned in January 2014

References

#8: Waitbot

For our eighth CUTGroup session, we tested the Waitbot app, where you
can find estimates for waiting times of all sorts of things, including transit, restaurants, airports, and more. This test had an in-person and a remote component to it. The in-person test took place at the Chicago Public Library’s Clearing Branch at 6423 W. 63rd Place in the Clearing neighborhood.

Through this test, we were interested in finding answers to these questions:

What makes users download an app? Delete an app
Do users want to use Waitbot on a daily basis? Why or why not?
What features do users want?
What other wait-related categories would users want to see?
Do users want to share wait-time information on social media?

On March 5, we sent out an email to all of our 749 CUTGroup participants. We asked them if they would be willing to test a wait-time estimates app on March 12, 2014. We asked some screening questions to gather information, and we chose our group of participants based on a diverse selection of answers and also device types.

We were interested in having about 15 participants from different Chicago neighborhoods, but we only had 6 testers come to test in-person. A lot of testers could not come due to a combination of weather and distance, so we reached out to 4 more testers to do the test remotely.

For the in-person test, proctors were able to work with testers one-on-one. Testers looked at the app on their own devices and provided feedback, while the proctors wrote down notes. After the test, we sent out additional, optional questions by email to see if testers were using the app and to see how they liked the app in their own neighborhood.

For the remote test, we asked testers to use the app on their own, and we provided questions to lead them through the test. In the end, we got great responses from both types of tests.

Most testers were not interested in sharing wait-time information on social media. One tester would share on Facebook only if it was automatically connected, while another tester said he would not do it unless there was an incentive. Only 3 out of 10 testers would share on social media.

When testing the Waitbot app, testers liked the transit page and the fact that it populated with nearby options. There was some confusion with col- or-coding, and testers wanted added features such as route display. However, testers overwhelmingly liked this page.

One tester, My eyes are dried out (#10), explains why he doesn’t like the Waitbot app in general, but thought that the transit page was the most useful:

“The Swiss army knife is useful and practical. Then the impostors ‘improved on it,’ making it bigger and more cluttered with useless features. Sometimes I feel app creators want to entice a large crown, instead of just perfecting one good thing.”

Chicago map showing distribution of testers — We had a mix of in-person and remote testers.

A tester reviews Waitbot on her own device.

References

#9 – Foodborne Chicago

We really feel like we’re hitting our stride with the last test we want to cover in this book—a test for Foodborne Chicago, an app that searches Twitter for tweets related to food poisoning and helps users report these incidents to the Chicago Department of Public Health. Joe Olson, one of the creators of Foodborne, received a grant, administered through Smart Chicago and the Chicago Community Trust, to build better communication strategies to engage all Chicago residents.

Cory Nissen, a statistician who wrote the Twitter classification code for this project, mapped out Foodborne Chicago reports and showed that there was less of a response from the South Side of Chicago. That is to say, of all the people who publish a tweet that includes the phrase “food poisoning” from inside the city of Chicago, those on the South Side are less likely to click on Foodborne’s prompt and complete a report. Through this test, we were interested in learning more about how people use Twitter, and whether there were differences among communities and networks.

This was also the test where we had the largest number of direct insights that could be turned into features and Github issues to improve the site. We made a number of changes, and the results are promising.

Here is a list of questions we wanted to answer through this test:

How do users feel about responding to a random tweet?
Does the form require too much information?
Would users feel better or worse if there was more Chicago Department of Public Health or 311 visibility?
Why might residents on the South Side of Chicago not respond as often as residents from other Chicago neighborhoods?

On April 9, we sent out an email to all of our 754 CUTGroup participants. We asked them if they would be willing to test an app on April 14, 2014, that helps report food poisoning incidents to the Chicago Department of Public Health via 311. We asked potential testers some screening questions and then chose a group of participants who were Twitter users, and who were primarily from South Side neighborhoods. This in-person test took place at the Chicago Public Library’s Blackstone Library at 4904 S. Lake Park Avenue in the Kenwood neighborhood.

We did a focus group-style test with three 30-45 minute sessions, with 5 people in each session. First, we had testers fill out a survey, prepping them to form their own opinions before the group discussion. Next, we asked testers to discuss their answers, and then we asked additional questions as part of a group discussion. We had some very interesting in-depth conversations, not only about Foodborne Chicago, but also about Twitter and social media use in general.

We learned that Twitter is used as a private communications network for the majority of our testers, even though their tweets are public. Twitter users tend to connect with people they know or who are in their networks. Most testers were not sure about responding to people they did not know, because strangers usually tended to be spammers.

Foodborne prompts people on Twitter to fill out a web-based complaint form that routes to 311. For our user test, we wondered whether the 311 complaint form had too many fields. Would users want to fill it out? We learned that users had little issue with the form, and they liked that it was simple and did not require too much information. The tester Bakunin thought that it was his “civic duty” to fill out the form to make sure others did not get sick. While we were not asking too much information on the form, testers still had questions about our privacy policy and the process.

Overall, testers were interested in seeing a stronger connection to the City of Chicago, the Chicago Department of Public Health and 311. Testers thought that seeing relevant logos on the website would emphasize these official, municipal connections. Testers also indicated that language about “The City” on the form was not descriptive, and they wanted more information about the process.

Based on feedback from the test, we immediately changed our tweet language to include compassion and an official component—the Chicago Department of Public Health. We also added a Twitter card (a way to attach rich photos, videos and media to tweets) in order to provide more detail about information being sent through the Chicago 311 service. Twitter cards weren’t available to small publishers like us when we first launched Foodborne, so that is a lesson in itself— we should be on top of new features and implement them as soon as we can.

We changed the URL to https://www.foodbornechicago.org/ from https:// foodborne.smartchicagoapps.org. People said they would be more likely to click that link.

Testers felt more comfortable the more they knew that there would be an official response. We changed the header information to include the Chicago Department of Public Health name and logo.

Testers also felt more comfortable with the form and the process of providing their information after reading the Q&A page. The final step is figuring out ways to be part of more people’s networks so that they feel comfortable clicking the link in a Foodborne tweet and submitting the form to report on their food poisoning.

Twitter cards were effective in showing the quasi-official nature of Foodborne Chicago.

Red Eye cover with skull and crossbones — Foodborne Chicago has been the subject of lots of press attention, which helps in overall success. The more familiarity people have with the site, the more likely they are to complete the form

References

5. Methods

Post author By DXO
Post date

In our discussion of the components of the CUTGroup, we were careful not to be prescriptive, but we were able to lay down some markers around what we think are the minimum elements for a viable Civic User Testing Group.

In our chapter listing the tools of the CUTGroup, we showed that it took very little money to start a program, and much of the necessary material was laying around any civic tech operation.

As you are considering whether and how to implement a CUTGroup program near you, we urge you to consider using the suggested methods we describe below.

Recruitment

The most fundamental concept behind recruitment of testers is the idea of individual relationships. The power of the Internet is that 1:1 relationships that are endlessly formed and re-formed in our experience of the Web.

Our testers come from every ward in the city because we spend the time and make the effort it takes to do that. We’ve done flyering campaigns at all twelve City Colleges of Chicago. We analyzed our signups and did flyering campaigns at the 25 public libraries where our efforts were lagging. Then we narrowed down our efforts to just the 10 libraries in the two wards where we still couldn’t get signups. We did a mass email campaign to more than 10,000 subscribers of Nightlife magazine.

In community engagement, there are lots of high-quality organizations made up of hundreds of regular residents, and the temptation is to work with those groups to sign up CUTGroup testers in one swoop.

I urge you to take the long way around—recruit individual testers, on your own, to start off an independent relationship with each tester.

The power of the Visa gift card

Once a resident signs up to be part of the CUTGroup, we send them a $5 Visa gift card. If and when they are chosen to test a civic app, we give them a $20 Visa gift card.

The Visa gift card is the most essential element of recruitment. There are all sorts of reasons why people might join the CUTGroup—to get involved with civic hackers, give back to their community, or just plain get out of the house. But money is a good motivator, and respects people for their time.

We chose this incentive very deliberately. It is the most open, accessible, fungible incentive we could give—the one most akin to cash. Unlike an Amazon gift card, it is useful in the real world and can be used by walking outside and going into a corner store. It’s easily transferrable and acts like cash.

Also, the gift card gives us a good way to do a simple validation that the tester actually lives (or at least has access to a mailbox) in the area we’re interested in—Chicago.

When purchasing gift cards for your program, there are four main considerations: type of cards, cost and fees associated per card, quantity, and expiration date.

There are different types of gift cards out there: prepaid Visa or MasterCard gift cards, store-issued, bank-issued, online, etc.

Other gift card types also include specific store- or bank-issued gift cards. Sites like ScripSmart can provide comparisons between gift cards, and they can give you an idea of what you need to ask about before purchasing your own cards.

By purchasing these cards, we are spending more than face value on fees, and have to take time to mail them out, but the value of accessible gift cards is worthwhile for the goals of our organization and this program.

That said, we have done a lot to cut our costs in managing the gift cards. When Sonja Marziano joined the team in September 2013, she cut costs from $10.17 per card to $7.07 per card. Here’s a breakdown of costs with our current gift card vendor, Awards2Go Visa Award Card:

• Face value of card: $5.00
• Card processing fee: $1.75
• Credit card order fee (1% of total order): $0.05 • Shipping fee for 100: $27.00
• Total: $7.07 per card

Once you’ve got a start, build on the existing network

Last October, we learned that we still had a lot of gift cards that were about to expire and some were already expired. With the gift cards that expired, we lost out because the cost of the fees to “restock” the cards would be higher than the value of the cards we would receive.

We also had 118 $20 and 103 $5 gift cards that were going to expire at the end of November. If we sent back these cards to the vendor, we would only receive half the value of the card in return.

So we had to think up some creative ways to use the cards. The most successful campaign was “Refer a friend”. Here was the pitch:

Hi <<First Name>>,
We have some $5 Visa gift cards that are about to expire at the end of the month and we need your help! We want to make sure these gift cards are used, so if you refer a friend to join the Civic User Testing Group (CUTGroup) by the end of the day this Thursday, we will send you another $5 Visa gift card! Forward this email to your friends and have them complete this form.

How it works:

We will send you a $5 Visa gift card if your friend (maximum of 4 friends) completes this form and lists your email in the “Referred by” field. Your friends also get a $5 Visa gift card for signing up.
If more than 4 of your friends sign up, we can only provide up to a $20 Visa gift card, but everyone you refer will still get a $5 Visa gift card just for signing up.
It’s important everyone signs up by the end of the day of Thursday, November21. Once you get your gift card, be sure to use it right away!Thanks for being a member of the CUTGroup! As usual, call or write with questions.

We signed up 100 new users through this campaign.

We now purchase gift cards with long lives (8 years!), though the value decreases from the card at 13 months after purchase, so you still have to carefully manage your gift cards.

Recruiting developers

One of the central tenets of the CUTGroup is that developers are a part of the process from start to finish. That’s why we never start planning a CUTGroup test unless the developers of the product ask us to do so. This ensures that they are willing to learn from talking directly with users and that they will make changes to the system when the results are in.

We engage developers on an ongoing basis by organizing, attending, and presenting at civic hack nights, and being an engaged part of Chicago’s development community. For example, we are founding members of
1871, Chicago’s digital co-working space, and often have seats available for developers and other thinkers interested in civic innovation. Nothing can replace genuine engagement in your local development community, and a CUTGroup can be an important part (or start) of that engagement.

Design

Once we have a developer who wants to work with us, we set up a meeting to learn more about the product, decide how to segment the CUTGroup tester base so that we put together the best group for test night, and design the test itself.

Questions we ask at this stage:

What day do we want to do the test, and do we have a general part of town where we want to do it? This is based on availability of the developers and Smart Chicago staff. When it comes to location, we check to see where the most recent tests were conducted, and we endeavor to target a part of town where we haven’t been in a while. There are some instances where an app is more applicable to a particular area as well. (We once did a test on an app focused on the retail environment of a particular neighborhood, for example.)
What is the current stage of the app? Is it a paper prototype, is it in beta, or does it have thousands of users? This basic information gives us a start that helps the entire process. We’ve worked with apps in each of these states of being.
Who do we want to target? Do they have very specific requirements for the test? For instance, we needed to find parents whose children attended a Chicago Public School and who took personal responsibility for getting those children to school in the morning.
Potential screening questions to ask. For each test, we fashion some specific questions that help us screen testers to make sure we have a room of relevant people on test night. Examples are, “Do you write reviews on Yelp?” for a restaurant inspection site or, “Do you know your Alderman’s first name?” for a neighborhood news site. People who post on Twitter might be more likely to post on other forums.
What type of test? We employ a number of test types to get the information developers need: in-person test, remote test, and focus group are the main types.
Preliminary questions in the test itself: We also try to design questions for the test itself, starting with some framing questions. Sometimes it’s an open question (“How do you use maps?”). For others, it’s more specific (“Do you attend any of these types of meetings?”, with a list of six types).
Questions that are specific to the app: This is where we get to the heart of the user interface questions that the developer wants answered.
Additional goals from the developer: This is where we cover the full circle of digital skills for the developer—what else do they want to get out of this test?

Here’s a portion of an example test plan, done for Foodborne Chicago:

Who are current users of Foodborne?
http://blog.corynissen.com/2013/11/mapping-foodborne-chicago- reports.html
Who do we want to target for this CUTGroup test?
We will be targeting Twitter users (with at least 100 tweets)—both very heavy and not as heavy users
We are looking to get a mixed group from all areas of Chicago—we want to have approximately 2/3 of the group from the South Side, 1/3 of the group from the North and West sides

We will ask for Twitter handles (not a required field)
Types of screening questions to ask to gather more information:• Do you have a Twitter account? (required) • How often do you use Twitter?
• How often do you go out to eat in a week?
When creating our group, we will consider neighborhoods, Twitter usage, if they eat out, and if they have ever used 311 (mix of yes/no responses)What type of test?
- Three sessions in a focus group style with small groups (5 people). We will be able to bring up pages and views to show the groups and get their feedback
- This is a qualitative test, since there will be a quantitative survey coming out later
- We will record the sessionsWhat do we want to ask? (Please note that these are just a rough outline of the type of questions we’ll ask)
- How do you currently use Twitter? Do you tweet often? What types of things do you tweet about?
- If you got food poisoning, would you tweet about it? Would you tell people you’re sick? Why or why not?
- If you get food poisoning, what do you do? Do you tell someone about it? Do you submit a 311 request?
- Show: Foodborne Twitter account and gather feedback about the tweet- er and the tweet
- Show: Foodborne Chicago form page and gather general impression— What do you think this form does? How do you feel about this information?
- Discuss form information: Is this something you would fill out? Why or why not? Do you feel comfortable with giving this information? Do you trust this site?
- Do you know that this information is being sent to the Chicago Department of Public Health (CDPH)? How do you feel about that?

Show: Q&A page—see how people feel about the request being made into a 311 request and the service tracker—this information might need to be more prominent elsewhere, this is where we will get more information about 311
What would make you more willing to respond to a random tweet? Followup: Would a tweet from a person in the community be better?
What would make you more willing to fill out this form?
Do you like Foodborne Chicago? Why or why not?
Do you think Foodborne Chicago is an effective way to report food poisoning? Why or why not?What does Foodborne want to learn from the CUTGroup test?
Foodborne wants to learn why South Siders are not responding
How do residents feel about responding to a random tweet?
Is the form too much? What fields might not need to be required?
Would users feel better/worse if there were more CDPH or 311 visibility? (In terms of Twitter handles, domain names, and logos)SegmentingOnce we’ve got a rough plan for the test, it’s time to gather a relevant set of testers. We do this through MailChimp email campaigns integrated with Wu- foo surveys in a three-step process:
- A broad call-out to people giving them a general idea of the test type and asking testers their availability for a two-hour time period on the test date and any specific requirements of the test
- An email discussing the specifics of the test, including location and test type, and asking them to affirm their availability, along with a preferred 20-minute slot in the context of the entire test period
- Emails to the set of people who did not get segmented into the testThe final steps in segmenting are done through a regular email client (not through the MailChimp mass email tool). We hand-schedule the testers into slots based on the tester’s choice, the number of testers, and test type. This is a great example of a task we’d like to automate with our Patterns software— we just haven’t gotten around to it.

Here are some sample emails and surveys you can use in planning your own emails to testers:

Broad call-out, initial email

Subject line: Make $20 at an in-person test of a food poisoning app. Hi <<First Name>>
We’ve got a new opportunity for you to make money in the Civic User Testing Group (CUTGroup) by testing an app.
This app helps report food poisoning incidents to the Chicago Department of Public Health. We want to know if this app serves your needs, and how it can be improved.
Are you available for a 30-minute, in-person test on Monday, April 14, between 4:00 and 7:30 p.m.? If so, you qualify. Complete this form to start the process.
We are in the process of gathering responses, so we will be in touch to check availability and let you know if you have been chosen. We are look- ing for 15-20 testers for this test.
For your participation, you will receive a $20 Visa gift card. You’ll also help make better software for Chicago.
Thanks for being a member of the CUTGroup! As usual, call or write with questions.
Remember: if you want to be a part of this test, please complete this form.

Specifics of the test email

Hi <<First Name>>,
Thanks for your interest in our CUTGroup test of a food poisoning app. Now that we have our location nailed down, we’d like to know if you can still make it next Monday and, if so, what time you want.
Here are the details:
Monday, April 14, 2014
Between 4 and 7:30 p.m.
Blackstone Branch of the Chicago Public Library 4904 S. Lake Park Avenue
If you can still make it on this night, please let us know what time slot you want by completing this form. For your participation, you will receive a $20 Visa gift card.

There are limited slots available for this night, and it’s coming up fast, so please let us know as soon as you can. We will get back to you with a confirmation if you are chosen to do this test.
If this location doesn’t work for you, or if you just can’t make it on this night after all, no big deal. If you have any questions or comments, just hit “reply” and let me know what you think.
Remember: use this form to tell us about your availability.
Thanks for being a member of the CUTGroup!

Did not get segmented email

Hi <<First Name>>,
I wanted to follow up with you about testing a food poisoning app Foodborne Chicago. We really appreciate your response to our call out.
Foodborne Chicago searches Twitter for tweets about food poisoning and then responds to those tweets with a link to report it so that the Chicago

Methods 27

Department of Public Health can take any necessary action. Therefore, for this test, we were looking for heavy users of Twitter.
Since you indicated that you don’t have a Twitter account, this just was not the right test for you.
We will send you info on our next test (should be in May or June), and we’ll definitely try to spread the tests around so that everyone gets what they want out of the CUTGroup experience. We do keep track of these things!
Thanks for being a member of the CUTGroup. As usual, call or write with questions.

The more we ask of testers, the smarter we get about testing

Overall, our strategy is to engage testers over multiple emails and gather new information. We develop deeper relationships and have more information that allows us to segment for new tests.

Our open rate for the blast availability emails, which are sent to all
active CUTGroup users, regularly comes in at 45-55%, and we have a very
low unsubscribe rate. People in the CUTGroup look forward to getting emails from us.

Now we have testers, a developer and goals for our tests. Where should we hold the test? What do we do once we get there?

Scouting

An absolutely essential criteria is that CUTGroup tests take place all over town. To decide where to hold our next test, we start with the list of Connect Chicago locations. Chicago Public Library has great, accessible, and free meeting room space for community organizations.

Much of the test location planning is classic event planning—it’s important to make sure that librarians, patrons, security guards, and anyone else we encounter during the test are fully informed and comfortable about our presence.

Proctoring

We use a mix of test administration styles in the CUTGroup: direct test proctoring, focus group, remote, and self-driven/in-person.

Direct test proctoring

This is the most common test style we use in the CUTGroup. Most often, we pair every tester with a single proctor who works with the tester to complete the test. Sometimes, based on the vagaries of scheduling, a single proctor will work with 2 or more testers at the same time.

Focus Group

Sometimes we can gather more intelligence about an issue in a focus group, which is great for getting qualitative information about a topic. We first have testers answer a survey, so that they have a chance to form their own opinions before the group discussion. Next, we engage them as a group—they all share their answers and we ask additional questions.

Remote

We always value in-person tests and believe they are an opportunity to convene participants from all areas of Chicago. We gain valuable responses in person and are able to record a tester’s actions and reactions. But remote testing can give us the sheer quantity of answers that can put an exclamation point on test results.

Self-driven/in-person

These tests were created so that no proctors were needed. We designed this type of test before CUTGroup #4, EatSafe.co. It is an example of a test that had lots of testers, and some testers did not have a proctor with them at the time.

Analysis

After the test is done, we analyze the results and publish all of the raw test data as well. We export all of the results out of Wufoo into a spreadsheet,
then use a Microsoft Word macro to populate an easy-to-scan document of the raw results.

The analysis is done by hand by the main proctor of the test. Sonja Marziano has done the majority of analyses for our CUTGroup tests, and she designed the method for exporting and formatting all test results.

Since the person writing the analysis was also present at the test itself, she is able to take the quantitative information and add qualitative insights so as to get actionable recommendations for the developers.

Followup

The interface recommendations are published in the body of our analysis blog post and we almost always create Github issues as well. This ensures that we create concrete, direct recommendations—nothing squishy or weak.

What happens next is up to the developer!

Remember, a lot more can be found on our website and blog at smartchicagocollaborative.org. We’ve posted in-depth info and documentation on what we’ve learned.

Money flyer — The “money flyer” was created by Emily Escaara and art directed by Kyla Williams. It drives home a key point about the CUTGroup.

References

Kyla Williams prepares to proctor a test.

4. Tools

Post author By DXO
Post date

Following is a list of tools we use in the CUTGroup program, along with a rudimentary take on the costs for each. Our focus in pricing is to determine the bare minimum it takes to start up a reputable CUTGroup program with materials on hand.

Website

We start with a public-facing website to present the program. The website itself is just a few files hosted on Github.

The CUTGroup website is just three pages of information about the program, a map of participants, and embedded Wufoo signup forms for the general public and developers. This website uses the Jekyll tool to generate static HTML pages that can be hosted in Amazon S3 for pennies a month.

CUTGroup Signups is a very small application that keeps the map of signups up-to-date. It is written in Ruby and runs on Heroku. After a CUTGroup participant completes the signup form, Wufoo sends the result to the CUTGroup Signups application. The application reads the ward of the participant and increments the number of signups for that ward.

This is certainly not a requirement, but we think it is effective in encour- aging a geographically diverse set of people to join. It’s always nice to see that people you know will be at a party.

If you’re not comfortable working with the files on Github, you can make the site any way you want, using any tools you are familiar with. WordPress is a very easy site creation tool—we highly recommend it.

If you have no way to set up a website, then just a Facebook page will do
as well. The main thing here is that you have to have some sort of wrapper
for the signup form—some URL that people can share with their friends and family that explains what your program is all about and displays or links to the signup form.

Costs: development time only.

Wufoo

We use Wufoo for information collection. Wufoo is a wonder. It is super easy to set up, it has powerful reporting tools, and it allows you to create sophisti- cated ways for sharing data among team members and between systems.

You are free to use any service you like, but given the features and pricing of Wufoo, we don’t see any reason to use anything else.

Costs: $14.95 per month, given the number of forms and fields we think you need to set up a good program. Free if you are just trying it out and want to collect less information from users.

MailChimp

We use MailChimp for all of our outbound communication with testers. It has great templates, it integrates really well with other systems like Wufoo, and it allows us to segment our testers in logical ways. There are many email management systems, but we’re impressed with MailChimp.

Costs: free at first. It took us a long time before we had to upgrade to an account costing $45 per month. Keep in mind that we also use MailChimp for other email marketing needs in our organization, so the costs of the tool are spread across the entire organization, not just this program.

Visa gift cards

We use Awards2Go Visa Award Card as our vendor for the incentives we use for recruiting and compensating testers. Gift cards are the central tool for tester recruitment.

Costs: $7.07 per card, including staff costs for managing card distribution.

Excel

From the very start, we have developed our own custom software to manage the CUTGroup. But Microsoft Excel has been an important make-ready tool for wrangling data. Excel was pre-installed on our office computers, but if you don’t have it, use an alternative like LibreOffice or Google Sheets.

Costs: free

Patterns

As part of this project, we’ve developed software and processes that allow us to manage such a large group of people testing a wide variety of hardware and software across the city. The main tool is Patterns (which we codenamed, “Kimball”, because that’s a street near Chris Gansen’s house), which allows us to segment our participants by lots of criteria (location, device, Internet connection, etc.).

Patterns uses data that we gather from other Web-based systems like MailChimp (for outbound email notifying CUTGroup members about testing opportunities) and Wufoo (for managing metadata about testers and their availability). Each of these systems has very strong APIs that allow us to
move data in and out of Patterns. It makes operating the CUTGroup a breeze. This kind of lightweight approach is at the heart of our philosophy here at Smart Chicago.

As you can see by looking at the Patterns code repository readme page on Github, we have a lot of work to do on this software:

Events
• Invite
• RSVP
• Attendance tracking • Reminder emails
Programs
• Associate results
People
• Add arbitrary fields
• Attach photograph
• Attach files
• Link with their social networks
• Show activity streams
• Track program status (e.g. has received Visa card)
• Show output from Tribune boundaries service on individual person pages
Backend
• Terms of service/privacy policy
• Managed access to anonymized data for research • Audit trails
• Comments on all objects

Costs: developer time.

Test Devices

We sometimes bring a number of devices to make our tests run smoothly. In other instances, it’s important that the users bring their own devices. Bring- ing extra iPads and laptops is definitely a good idea, because pretty much anything can and will go wrong when you are out in the real world engaging with real people.

We publish details on lots of the equipment we use for all of our different programs on our website. Keep tabs on our site and you can keep up with current devices, model numbers, prices, and configurations.

Costs: varied — use personal devices (free)

Recording Devices

Livestreaming opengov and civic engagement events is an important compo- nent of our overall mission at Smart Chicago, and it is an important com- ponent of traditional user testing. I also have a passion for photography and documentation, so it is a deep part of our culture.

Once again, the idea is to press existing devices into service. Any cellphone camera will suffice when it comes to documenting the tests, either by photo or video. Again, we publish specs on our own equipment to our website.

Costs: varied — use personal devices (free)

Wifi

If you’re doing mobile technology work, you’ve got to be in control of your own connection to the Internet. Even though most of our tests are conducted in public libraries, and they all have public wifi, it is essential to carry a back- up. We use Mobile Citizen.

Costs: $120 for signup, $10/month for service

Candy

Ya gotta have something to eat. Also: sharing is caring. We bring multiple types of candy (hard candy, mini chocolate bars, sour twists, etc.) Stay away from messy things (no Doritos) and, just like in elementary school, make sure you have enough for the whole class.

Propaganda

We do lots of work with computer centers, and this is an opportunity to get our message out. We leave information about our Connect Chicago program and also flyers for anything else we happen to have going on.

Here’s a look at a schematic of Smart Chicago systems. Diagram by Chris Gansen.

The ever-present candy.

References

3. Components

Post author By DXO
Post date

There are three components to what we do with the CUTGroup: UX testing (“user experience testing,” also called “usability testing,” a technique used in user-centered interaction design to evaluate a product by testing it on users); digital skills (which we define broadly as the human ability to get things done on computers); and community engagement (which, in our context, is defined as a process of building relationships for the purpose of collectively making lives better through technology).

One of the most important aspects of our work is that it does each of these three separate things pretty well, but none of them really well. The only thing we do really well is the CUTGroup itself.

We struggle at Smart Chicago with how prescriptive we should be with our program. Clearly, any kind of user testing is helpful to the technology developers. The teaching and learning of digital skills is a worthwhile act, regardless of context. And any time civic hackers can get with community members—in any setting, for any purpose—that’s a good thing.

As you’ll see in the Methods chapter and in the examples of tests we’ve conducted, different CUTGroup components take on higher or lower levels of importance, depending on the particular nuances of the need for any given project.

Sometimes, the app isn’t made yet, and we’re testing the relative value of a concept. In others, we’re testing a mature website that has some user interface issues, and just want to get bug reports. Since we make a lot of technology, and we are engaged with audiences and experts all the time, sometimes we just want to have a structured meeting that helps us think fresh and re-engage with some of the people who matter most.

But we’ve settled on the belief that the melding of these three components is what makes the CUTGroup the CUTGroup—its essence. It would therefore be impossible to say that you’re running a CUTGroup program near you without being devoted to building these components in some semblance of equality and strength.

So let’s take a look at each component, talk about how they fit together, and lay down some markers around what we think are the minimum elements for a viable CUTGroup.

UX testing

We have been careful to design the CUTGroup as legitimate UX testing. By that we mean the developers who work with us get specific, actionable feedback from relevantly situated users. We can obtain both quantitative and qualitative info, discovering how dozens of people answer the same question and drill into a deep conversation about any particular feature. We’ve found glaring bugs, discovered unique insights, and helped people plan new product releases.

We stray from UX test design principles, however, in a very key way: by requiring the developer to participate in the test. We deprecate this and other more social-science aspects of classic UX testing (never coaching the user, for instance) because it’s not conducive to the other two components.

There is an odd dynamic here, however. At Smart Chicago, we have a broad mission to help build the civic innovation sector of the technology industry.
By this, we mean to deliberately situate the work of civic hackers firmly in the broader market for consumer-focused technology.

By cutting against the grain of UX testing methods designed to align a software product with the needs and desires of people who might buy it, we risk keeping the sector in an immature state of development. We justify this by seeing it as a temporary condition of the field. There is currently so little engagement with regular users, and very little tradition around user testing, that we’ve developed this hybrid to account for that.

The minimum UX testing element that must be included in a CUTGroup program is the delivery of concrete, actionable direction to the developer that has been generated directly from testers who have been deliberately chosen for the test.

Digital skills

This is the component for which there is probably less precedence in the work of a typical civic hacking aficionado. But it’s a really important one to our work at Smart Chicago, which was formed out of conversations about the digital divide.

A CUTGroup test typically consists of a mix of 10-20 people of widely varying digital skills, all learning and teaching each other, in the same room, out in plain public view. It becomes something of a salon. People get to know each other, laugh, eat candy, and pick up digital skills that they didn’t expect to get when they first signed up. This goes for the developers, too—they are learning how to conduct UX tests and to make their sites better.

The more professional-grade digital-skills teachers would want to formalize this learning into a structure that allows the tester to move along a continuum of learning toward an established goal. We gave up on that idea because it would cost too much money (we can’t pay people a $20 Visa gift card every time they show up for a class), and because it would defeat the informal but focused nature of a test.

So the minimum digital-skills element that must be included in a CUTGroup program is simple: a spirit of learning and a commitment that all of the testing happen in the same room, with all of the messiness that goes along with that.

Community engagement

This is the component that has surprised us the most, and the one for which we did the least amount of planning.

We always knew we wanted to conduct the tests in public computer centers and mainly in libraries. We love the vibe there and, to us, these are the most accessible and open places we could imagine. There are so many structural elements of the library—their wide geographic range, onsite wifi, community rooms, open architecture, and accessibility compliance—that our program slots in well when we center it there.

But we were surprised to see how much people like the CUTGroup. We always ask the same last two questions on every CUTGroup test form. One is quantitative (“Did you like this CUTGroup test?”) and the other is qualitative (“Anything else to add?”). Out of the hundreds of testers, one has answered “no” the the first question, and we’ve received dozens of positive comments about the experience.

People often have to be shepherded out of the meeting room as the lights go off in the library. We all have a really good time, and sometimes we’re able to talk about difficult things (like how to get to school in an era of school closings) in very flat, constructive ways. The CUTGroup is a joy. To anyone who has ever done civic engagement (which can be a slog), you understand that this is a big deal.

From the start, we’ve been completely committed to drawing together as many people as we can from every part of town and all sorts of backgrounds into a single set of people on a common mission.

We differ from the practice of community organizers, however, because we don’t drill down into specific geographic areas or subject matters, and we don’t attempt to achieve specific social or policy outcomes. However, this leads to trust among our users because we stay focused on what we said we were interested in when they signed up: We give them money, and they give us feedback on technology.

In order to make the CUTGroup representative of the community, and to serve these larger goals around development, the minimum community engagement element that must be included in a CUTGroup program is the recruitment of a large and diverse tester base.

This map shows the wide geographic (and, by extension socio-economic) range of people who came to a test of OpenStreetMap. Some people came 20 miles on a January evening when it was 10 degrees Fahrenheit with light snow.

Hoy Managing Editor Fernando Diaz working with CUTGroup members - CUTGroup #5 - Eatsafe.co - Hall Library

Developer Fernando Diaz tests his food inspection site with three residents in the Kelly Library.

References

2. Origins

Post author By DXO
Post date

An immediate impetus

At Smart Chicago, all of our programs grow directly from our work. We think and learn by doing. In the fall and winter of 2012, we worked with our partner, the Illinois Science & Technology Coalition, on the Illinois Open Technology Challenge. Our mission was to bring “governments, developers and communities together in a common mission to use public data and create digital tools that will serve today’s civic needs and promote economic development”.

As part of our work, we did meetups up and down the state. We traveled 900 miles to conduct 8 meetups in 6 locations in 4 cities with 149 people. We worked with 12 government IT professionals to publish 138 new datasets (34 in Champaign, 15 in Rockford, 12 in Belleville, and 77 from the 42 municipalities South Suburban Mayors & Managers Association).

Here’s how we described the work:

This is the time for anyone with an idea to present it to the group in a more complete fashion and make a pitch for people to join them. We will have lots of materials that will help you express yourselves—easels, large Post-Its, markers, etc. You’ll want to talk about what data you’d like to use and what community issues you want to solve. We also ask you two questions when signing up for the Meetup—the more thinking you do before the event, the more you will get out of it.
Do you have an idea for an app that you’d like to submit? Let us know!
Do you have a community issue you’d like to address through data and technology? If so, can you describe?

We tried to communicate the idea that everyone was welcome, whether you had “an idea for an app” (which assumes some fluency with technology) or just a “community issue you’d like to address,” you could join a team and win money.

As you can see from the overall numbers, we had some success in getting people out for the meetings. We met plenty of local developers and were able to introduce them to city officials. Midway through, we realized we had to be more aggressive in outreach to community members, so we sought and received coverage on the nightly news in Rockford and afternoon radio in Champaign. Here’s a snip from the news coverage:

With countless amounts of data, the government is seeking a way for the public to access the information to solve common problems. The City of Rockford Information Technology Department invites the community to take the Illinois Open Technology Challenge. If you develop an app to solve a problem using Rockford data, you could win $15,000. Smart Chicago Collaborative Executive Director Dan O’Neil encourages anyone with an idea to check out illinoisopentech.org. “You can make technology that’s actually in the service of the people. That’s the idea. That we can change Rockford, change your city and change your world with technology.”

But we failed at getting regular residents to show up at these meetings. I realized that with a value proposition that starts with “If you develop an app,” there was no way we were going to get regular people to show up. We were offering $15,000 in prizes in four cities, but our program was too involved:

Come to a meeting on a weeknight
Develop/present an original idea for an “app”
Persuade one or more developers to build the idea
Follow the process through to completion
Submit the finished site/app
Prosper

When we got to Belleville—as far south as one could get in Illinois—we had the mayor, some developers from St. Louis, and zero members of the public. There had to be an easier way.

Some antecedents

By January 2013, I was definitely casting about for an easier way to get regular residents involved in civic tech. At the same time, I was pulling together thoughts around the limitations of civic hacking—the practice of local developers making technology with local data to serve local needs. I put those thoughts down in a blog post called “Turning Civic Hacking into Civic Innovation.” Here’s a snip:

There are lots of reasons why civic hacking works here in Chicago—a rich baseline of data and technology, an engaged developer community, real discussions with government about policy and data, and the support of institutions are all important factors.
But what we’re missing most is sustained engagement with the residents of the city of Chicago. That’s how we can turn mere hacking into real innovation. The magic combination of government, developers, and community members is what we’re after.

A rich baseline of public data projects, an engaged developer community, government that cares, and support from institutions. But we had to appeal to regular residents. I turned to my experience.

In the summer of 2003, I taught a course of 15 elementary and high school students in a weeklong “computer camp”
In 2004, I did bilingual website training for Spanish speakers in Lincoln Square. I created and led a number of bilingual computer-training sessions for the large (but mostly invisible to the Caucasian population at the Church) Spanish-speaking community at a parish in the North Side. I saw the need, designed tools to fill the need, and conducted the training myself
In 2006, I developed a custom 9-hour “Websites for Small Businesses” course taught in three-hour stretches over three days. This was open to the public, and I taught all comers. Almost everyone had very low skills. Many had never had an email account before

These experiences—setting up shop, in public computer centers, for all comers—were formative for me. My experience as a co-founder of EveryBlock was also important. Everyone on the team was expected to answer user emails, which consisted of bug submissions, feature requests, and comments of all manner.

With all of this in mind—our current inability to attract community members and my past experience in working directly with people around technology—I came up with the concept of the CUTGroup.

Our motto: “If it doesn’t work for you, it doesn’t work”.

Spanish language version of church bulletin.

Belleville City Hall

References

1. Introduction

Post author By DXO
Post date

A how-to on civic tech engagement

This is the CUTGroup book, an extensive how-to on the Civic User Testing Group—a set of regular Chicago residents who get paid to test civic apps. It began with a simple idea—that civic technologists should be in communion with the people they seek to serve—and it has grown to a community of more than 800 people who work together to make lives better through technology.

In this book, we cover in great detail how we do UX (or user experience) testing, digital skills, and community engagement in one civic tech system. We cover the hardware and software you need, methods for tester and developer recruitment, test design, location scouting, and results analysis. We show detailed budgets, exact website configurations, complete text of recruitment emails, the raw results of every test we’ve conducted, and all the other nuts and bolts it takes to make a CUTGroup in your city.

How it works

Here’s the call to action we use on the CUTGroup website:

Be a tester, get paid

The Civic User Testing Group (CUTGroup) is a set of regular Chicago residents who get paid to test out civic apps.

Fill out a CUTGroup profile and sign up to be a tester of civic apps, and we’ll send you a $5 Visa gift card
If and when you are chosen to test a civic app, you get paid a $20 Visa gift card

There is a large and growing community of “civic hackers” in Chicago—technology developers who make websites, mobile apps, and other tools that often have specific use in Chicago. The goal is to make software that helps make lives better in the city. What’s currently missing? The people.

That simple proposition has worked for us, giving us hours of great discussion and dozens of useful insights about what people really need out of civic technology in Chicago. This book shows you how it can work for you.

The CUTGroup is the work of Smart Chicago

The CUTGroup is an initiative of the Smart Chicago Collaborative, a civic organization devoted to improving lives in Chicago through technology. We work on increasing access to the Internet, improving skills for using the Internet, and developing meaningful products from data that measurably contribute to the quality of life of residents in our region and beyond.

The CUTGroup is a central program for Smart Chicago because it cuts across our three areas of focus:

Access: the majority of our tests are conducted in public computer centers. This allows us to celebrate and promote these important connectivity points, and it also helps draw together human networks
Skills: we focus on “on ramps”, and everyone who participates in a CUTGroup test is on one of these ramps. We deliver rudimentary digital-skills training to testers, as we’re often introducing them to new technology. We train developers how to design tests and engage residents in order to gather concrete feedback for their products. And we, as the Smart Chicago team, never leave a test without new skills and knowledge
Data: by helping improve existing websites and apps, and by encouraging the creation of more effective and popular products in our field, we help deliver on the goal of creating a strong civic technology sector of the technology industry

This unique framing—access, skills, and data lead us directly to the creation of this program. This framing existed in the founding of Smart Chicago, which was first conceived in the 2007 report “The City that NETWorks: Transforming Society and Economy Through Digital Excellence.” This book is our attempt to share our processes on this particular program, and we have a great desire that you will share our methods as well.

Map showing CUTGroup tester coverage across the city.

Sonja Marziano conducting a CUTGroup test at the Chicago Public Library Clearing Branch, 6423 W. 63rd Place.

References

Bookmark

The CUTGroup Book

Post author By DXO
Post date

Civic User Testing Group as a New Model for UX Testing, Digital Skills Development, and Community Engagement in Civic Tech

By Daniel X. O’Neil and the Smart Chicago Collaborative

In this book, we cover in great detail how we do UX (or user experience) testing, community engagement, and digital skills in one civic tech system. We cover the hardware and software you need, methods for tester and developer recruitment, test design, location scouting, and results analysis. We show detailed budgets, exact website configurations, complete text of recruitment emails, the raw results of every test we’ve conducted, and all the other nuts and bolts it takes to make a CUTGroup in your city.

Dedication

To the people of the City of Chicago.
If it doesn’t work for you, it doesn’t work.

Epigraphs

This is not a pipe.

—René Magritte, The Treachery of Images, 1929

I mean, we’re all stuck here for a while.
Let’s try to work it out.
Let’s try to beat it.
Let’s try and work it out.

—Rodney King, Speech on May Day, 1992

“Elaborate usability tests are a waste of resources.”

—Jakob Nielsen, 2000

Get the book:

It’s available in three ways:

As a print-on-demand paperback from Amazon for $4
Free download here
In complete text on a stand-alone website at cut.group