BriefBank

SIMS 213
Spring 2002
John Fritch
Tom Selsley
Kaichi Sung
Mary Trombley
email whole group
 
Final Report
(Comprehensive)
Final Presentation [PPT]
 
Third Interactive Prototype
(Requires MSIE 5.x., Navigator 6.x or Mozilla 1.x)
Second Interactive Prototype
(Requires MSIE 5.x.)
First Interactive Prototype
(Requires MSIE 5.x.)
 
Assignment 1
(Project Proposal)
Assignment 2
(Project Personas, Goals, and Task Analysis)
Assignment 3
(Project Scenarios, Competitive Analysis, and Preliminary Design)
Assignment 4
(Lo-fi Prototype and Usability Tests)
Assignment 5
(First Interactive Prototype)
Assignment 6
(Project Heuristic Evaluation)
Assignment 7
(Second Interactive Prototype)
Assignment 8
(Pilot Usability Study)
 
Work Distribution
 
Heuristic Evaluation for MDTP Project


SIMS 213 Assignment 8

Pilot Usability Study


Introduction

Method

Test Measures

Results

Discussion

Formal Experiment Design

Appendices

Introduction

Using the second interactive prototype, we developed a pilot usability study involving four tasks (three search tasks and task of submitting a brief). We tested the prototype with six testers and learned that the third interactive prototype needs to include more metadata for briefs.

In addition, we discovered we would need to modify our method for recording task completion times to improve accuracy and we developed a hypothesis that providing additional brief metadata will make it easier for users to determine whether a brief satisfies a search.

Method


Participants

Tester Gender Age Education Legal Research Experience
1 Male 28 Masters - Civil Engineering 1 undergrad law class; some law library research
2 Female 37 Masters - Information Management & Systems Currently law librarian at Boalt; previously worked as paralegal
3 Male 37 J.D. Law school; externship; internship
4 Male 41 J.D. Practicing law; law school
5 Female 28 J.D. Practicing law; law school
6 Male 33 J.D. Practicing law; law school

We selected tester 1 because we wanted to make sure our usability study was designed properly. We included tester 1 in our results because his results were consistent with the other testers. We selected testers 2 - 6 because they have significant legal research experience.

Apparatus

We conducted tests 1 - 5 in the SIMS lab. We conducted test 6 in the tester's office. We used our second interactive prototype for all of the tests.

Tasks

Task Purpose
You've been assigned to research MPAA v. 2600 for your supervisor. Find all the amici briefs for the plaintiff that relate to the MPAA v. 2600 case. Assess ease of executing search using the parties to a case and assess whether brief metadata is sufficient to differentiate briefs.
Your boss at your law school's legal clinic wants you to write a brief on a case similar to Eldred v. Ashcroft. She remembers that she found a particularly good brief about Eldred v. Ashcroft at BriefBank several weeks ago, but she doesn't remember its name. Your professor wants you to find that brief. The brief discusses whether the D.C. Circuit Court made a mistake when it ruled that Congress has the power to retroactively extend copyrights? Assess how testers review brief documents to answer a specific question.
You've been assigned to write a brief on electronic trespass. Find all the briefs that relate to the case Intel Corp. v. Hamidi. Determine whether familiarity with BriefBank's search function decreases the execution time.
You've written a brief that recently has received cert from the Supreme Court. Your colleagues have been driving you crazy asking for copies. You decide to send your brief to BriefBank. That way, you can just refer other lawyers to that site when they want a look at your research. Assess ease of contributing a brief.

Procedure

  1. Introduce test group members.
  2. Have tester to review and sign informed consent.
  3. Have tester to complete pre-test survey.
  4. Explain purpose of test.
  5. Show tester the BriefBank home page and allow tester to explore web site for several minutes.
  6. Have tester attempt task 1.
  7. Have tester attempt task 2.
  8. Have tester attempt task 3.
  9. Have tester attempt task 4.
  10. Have tester complete the post-test evaluation.
  11. Ask tester for opinions regarding interface alternatives for third interactive prototype.

Test Measures

We tested the prototype using observations of the testers' actions & comments as they attempted the tasks and a written post-test evaluation that the testers completed at the conclusion of the four tasks.

Observations


During our observations, we focused on the following:
  • Page flow: Can testers navigage the system efficiently? Do they use the back button excessively? Can they make reasonable guesses about the consequences of their actions (i.e., pushing a button)?
  • Labels: Do the system's labels and vocabulary match the tester's conceptual model?
  • Relevance of the material: Does the metadata retrieved from search/browse match the tester's expectation of such? Does the metadata help the tester make relevance judgments?
  • Task completion time: Can testers complete tasks in a reasonable time? Does the time decrease as testers become from familiar with the prototype?

Post-test evaluation

At the conclusion of the tasks, testers stated their degree of agreement or disagreement with statements regarding the prototype. These statements concerned the ease or difficulty of completing the tasks. Additionally, testers provided comments regarding the easiest and most difficult aspects of the prototype.

Results

Observations

Page Flow: All of the testers grasped the page flow for the search and submit a brief functions. Several testers asked whether the textfields for describing a submitted brief could hold complete descriptions. One tester questioned the purpose of the "Other Court" textfield for submitting a brief.

Labels: For the most part, testers were comfortable with the prototype's labels and vocabulary. One tester was not sure whether the term "Brief Title" referred to the title of a brief or an abbreviated name for the case.

Relevance of retrieval: Several testers commented that they would like to have more metadata for each legal brief. Currently, the prototype provides extensive metadata for the cases, but it only provides a title for each brief.

Task completion time: We attempted to collect task completion times for testers 4, 5, and 6. In general, the times appeared to be acceptable, and testers needed less time to complete task 3 than task 1 (two similar searches). In addition, we observed that testers needed significantly more time to complete task 2 than either task 1 or task 3. The completion times for task 2 were longer because this task required testers to review the contents of briefs to determine whether a brief satisfied the task requirements.

Obtaining accurate times was difficult because testers discussed their approaches to evaluating legal briefs in addition to commenting on the tasks. Consequently, it was not always clear when a tester believed a task was completed.

Post-test evaluation scores (Statements evaluated on a scale of 1 [Disagree] to 5 [Agree])


Statements in which a low score is better Avg Score
It was hard for me to complete the tasks I was given today using BriefBank 1.17
It was difficult to find the briefs I needed to find 1.00
It was difficult to evaluate the relevance of individual briefs the system showed me in response to my searching or browsing activities 3.50
The search tool was difficult to use 1.00


Statements in which a high score is better Avg Score
The words and concepts in the system were familiar to me from my legal research 3.83
It was easy to submit a brief to BriefBank 5.00
The search gave me the kind of results I was looking for 4.83
I was confident that I found what I was looking for 4.17
I was satisfied with BriefBank as a tool to perform the kinds of tasks I was asked to do today 4.67
I would trust BriefBank enough to post my own briefs here 4.33
I was able to find briefs quickly on BriefBank 5.00

Post-test comments

What was easiest? All of the testers commented on the ease of searching.

What was most difficult? None of the testers raised the same issue. One tester mentioned that the label "brief title" was confusing. Another tester mentioned the inability to select several courts when searching.

Did you notice you could not search by subject? Only three of the testers noticed they could not search by subject. Those who noticed the lack of subject search did not consider it to be a significant problem.

Discussion


Changes to interface

According to our observations and the testers post-test feedback, the most important change is the inclusion of additional metadata for the briefs. In particular, when displaying briefs we intend to identify which party a brief supports and whether a brief is an amicus brief or not. To facilitate the gathering of this information, we will also ask individuals to provide this information when describing a brief that is going to be submitted to BriefBank.

Following are other potential changes
  • Allow users to select multiple courts when searching.
  • Change the label from "brief title" to "title of brief" to avoid confusion.
  • Change the mechanism for identifying a new court when submitting a brief to avoid confusion.
  • Make the links for BriefBank's contributors and developers more prominent on the home page to increase credibility.

Changes to "real" experiment

In conducting a "real" experiment we would change our approach to recording the task completion times. To increase time accuracy, we would ask testers to indicate when a task is complete and to hold their comments until after a task is completed. We would phrase this request carefully because we do not want to discourage testers from providing comments.

Formal Experiment Design

Hypothesis: Our hypothesis is that providing additional metadata for briefs will make it easier for users to determine whether a brief satisfies a search.

Factors and levels: The factor is the amount of metadata provided for briefs. The four levels are T (title of brief only), A (title of brief and whether brief is an amicus brief or not), S (title of brief and which party the brief supports), and Z (title of brief, whether brief is an amicus brief or not, and which party the brief supports). The response variable is the time to complete a search. Search completion time is an appropriate response variable because testers should complete searches more quickly if additional brief metadata allows them to include (or exclude) a brief.

Blocking and repetitions: The experiment will be based on a within groups design and involve 16 third year law students. Each law student will complete five searches. The first search will not be included in the test results because we do not want the completion times to be influenced by a learning affect. Searches 2 - 5 would involve one search for each level.

Appendices