SIMS 213 Assignment 4

Lo-fi Prototype and Usability Tests

Introduction

We are devloping BriefBank for the Samuelson Law, Technology and Public Policy Cinic at Boalt Hall Clinic to facilitate the sharing of legal briefs within the technology and public policy community. Using our previously defined scenarios, we developed the following interaction flow which focuses on the searching, browsing, and submitting of legal briefs.

Note: The interaction flow does not include the following secondary pages included in the prototype: About Us, Terms & Conditions, and Privacy Policy.

Next, we developed our initial prototype of BriefBank's user interface and tested the interface with three partipants to determine the strengths and weaknesses of the interface. We developed a paper-based prototype for the following reasons:

• Low-fi prototypes (including paper-based prototypes) can be developed quickly.
• A developer can "play computer" and allow a test participant to "walk through" task scenarios even though the application is not functional.
• A developer can easily makes changes to the prototype based on feedback from the test participants.

Prototype

For our prototype we decided to represent each web page for our application on a single piece of letter size paper with a portrait layout whenever possible. With this approach the representation that we provide to the test participants approximates what the participants would see on a 15" monitor. We developed each page using Adoble Illustrator because this application allowed us to control the formating and appearance each page's information and to produce multiple pages with consistent formatting. See Appendices for a PDF file of all screens used in this prototype.

In creating the pages for the prototype, we intentionally refrained from using colors and minimized the use of graphics because we did not want the prototype to appear too polished or finished. We wanted the look of the prototype to encourage comments from the test participants.

Following are images of significant prototype pages, project member Tom Selsley "playing computer", and Tom's organization of the pages in preparation for a test.

Method

Demographics of Participants

Our three participants in the informal usability test were selected for their experience with legal issues and legal information system. Participant 1 is a first-year MIMS student who takes advanced classes at Boalt Hall School of Law, under the auspices of the our client, the Samuelson Clinic. Participant 1 is male, mid-twenties, and very advanced in computer use and system design. Participant 2 is a second-year law Student at Boalt Hall. He is male, mid-twenties, and passionate about techlaw issues, involved in a few active legal cases. He is moderately technically inclined, possesses a lot of experience with legal information systems for research, a basic understanding of HTML, and does not program or design systems. Participant 3 is a second-year MIMS student who works at the Law Library and was a paralegal for several years before starting at SIMS. She is in her late-thirties and avoids all hard-core technical tasks, though she is very adept at leading legal information systems and even elaborate interfaces used to build legal information systems.

Test Methods

Test Scenario 1: Find all the briefs in BriefBank that relate to Universal vs. Reimerdes.

This scenario gives BriefBank users a typical yet easy search task that helps one become familiar with the system. The underlying logic tested in this scenario is searching by a facet "case." This scenario was relatively successful, though it could be improved by changing some confusing labels on our search screens.

Test Scenario 2: Find a brief or briefs on broadcasting laws in the jurisdiction of the US Court of Appeals for the Fourth Circuit.

Scenario 2 is meant to be typical but more complex search task and introduces the facet "court" or supported by the system. We wanted to see whether legal researchers would map the idea of "jurisdiction" to "court." There is a second facet in the "subject" of the query suggested in Scenario 2. There are many approaches to modeling a "subject;" our team has grappled with several, but we did not model this well in our scenario. Scenario 2 was the least liked by and least successful with test participants. Our results show more work is needed in this area.

After Scenario 2, participants were given a one page evaluation form for Test Scenarios 1 & 2 (See Appendices). We chose to combine Scenarios 1 & 2 on one evaluation form to help reduce the amount of effort required on the part of participants. We also hypothesized that participants would view the tasks as similar, search-based tasks, though progressing in difficulty for the second one.

Test Scenario 3: Submit a brief to BriefBank with the following information:

Name: Wilma Donahue
Organization: Acme Law School
Email address: Wilma@acme.edu
URL: www.acme.edu/~wilma
Brief title: Educators' Amicus Brief for ACLU v. Ashcroft
Case Name: ACLU v. Ashcroft
Case Number: 6000-20
Moving Party: American Civil Liberties Union
Responding Party: John Ashcroft, in his capacity as attorney general of the United States
Court: Supreme Court
Brief Description: Amicus brief for ACLU v. Ashcroft; relevant issues: First Amendment, copyright.

Scenario 3 was designed to test our submission flow. We gave participants a detailed but highly organized task, and the prototype contained the most detailed page at this stage in our system. This task was most highly rated by and most successful with our particpants.

After Scenario 3, participants were given a one page evaluation form for Test Scenario 3 (See Appendices). We chose to have participants evaluate Scenario 3 on a separate form because the flow in Submitting a brief calls upon very different parts of the interface than the earlier search-based tasks.

Testing Procedure

Team members had three distinct roles: one person greeted and facilitated the test and was the only one who spoke during the session; one person "played computer," displaying the lo-fi paper "screens" in response to participants' actions; and one person took notes. Each participant was greeted, offered beverages and snacks, introduced to lo-fi testing and our reasoning for using a paper prototype, introduced to the team members, and told a description of the project. The test session began by the "computer" displaying a paper "home page" for BriefBank, and participants were encouraged to investigate or generally surf around the prototype. When participants were ready, the facilitator offered and explained scenario 1. Scenario 2 followed immediately, and after scenario 2, participants were given a one-page form to evaluate the first two task scenarios. Task Scenario 3 followed, and then its own one-page evaluation form. The three tests in this round took 20 to 30 minutes.

Results

Task Scenario 1 was fairly successful, Scenario 2 was problematic, and Scenario 3 was the most successful. All three participants provided candid suggestions for each of the three scenarios.

Raw Data Collected from Three Participants

Discussion

Problems with Basic Search Page

We designed the basic search page with only one textfield for the query terms to simplify the search process. We assumed users would often enter a party name or case name into the textfield and select the radio button for the appropriate search facet. Unfortunately, the combination of one textfield and a list of facets was confusing for two of the test participants. The presence of only one textfield led the participants to assume they could use any term to search. The participants struggled to reconcile this notion with the need to select a facet. For the first task scenario, all three participants used one or more party names to search and were able to find one or more satisfactory legal briefs. For the second task scenario, however, all three of the participants wanted to use a search topic to search. It was not clear to them whether they could use a subject term to perform a search. When the participants looked at the advanced search page, they understood which facets they could use for searching because their is a textfield or other input component for each facet. Likewise, they realized that subject was not a viable facet. Based on the experiences of our test participants, we learned that our basic search page must clearly illustrate which facets can be used for searching. For the interactive prototype, we will make the basic search page more like the current advanced search page.

Desire to Search by Subject

The participants expressed a desire to search by subject. At this time, it is not feasible to assign subjects to legal briefs as they are entered into BriefBank's database. Consequently, we need to make it clear to users that BriefBank does not support searching by subject.

Problems with Home Page Layout

All three participants commented that the text on the home page drew their attention away from the function links like "Search" and "Submit" on the top of the page. In addition, they expressed their expectation that a search capability should be included on the home page. For the interactive prototype, we will use icons to make the function links more prominent and replace the home page text with the basic search components. Consequently, the home page and the basic search page will be the same page.

Importance of Terminology

Two of the participants commented that the descriptions of the facets listed on the basic and advanced search page were not clear enough. For example, one of the participants did not know whether "brief title" referred to the name of a legal brief or a shortened version of the "case name". Furthermore, two of the participants were not clear on the meaning of "browse". For the interactive prototype, we will evaluate all of the key terms and modify them as necessary.

Preference for Submit Page

All three participants indicated that the layout of components on the submit page made it easy to complete the task of submitting a legal brief. Even though they did not have prior experience submitting a legal brief to a web site, they felt that the prototype's submit page clearly delineated the necessary steps.

What The Evaluation Didn't Tell Us

Only one of the test participants used the prototype's browse capability during the task scenarios. Consequently, we do not know whether this capability is important.

Appendices

Screens produced for lo-fi prototype (PDF, 5.53 MB)
Test Script (MS Word, 23 kB)
Evaluation forms for Scenarios 1&2, 3 (MS Word, 23 kB)
Incident logs for all three participants (MS Word, 71 kB)