By Deborah Hinderer Sova and Cory Knobel
Used with permission from the Society for Technical Communication, Arlington, VA U.S.A.
This article describes a case study of a challenging but successful rapid-development web project, which incorporated customer-centered design using multiple methodologies. Within ten weeks, our team conducted field studies and focus groups, produced paper prototypes of three navigational concepts, conducted a usability test using paper prototypes, and performed heuristic evaluation on the resulting design. Keys to our success include: assembling a top-notch team, running project phases concurrently, and employing good, ongoing team-client and intra-team communication.
Project Background
In Summer, 2000, General Electric Medical Systems—now known as GE Healthcare—in Waukesha, WI, engaged a usability team from the former Optavia Corporation (which was based in Madison, WI) to participate in the redesign of the GE Medical Systems web site.
Usability specialists Deborah Hinderer Sova (now senior user experience researcher at Tec-Ed in Milwaukee) and Cory Knobel (now University of Michigan Information Science doctoral student) were key contributors to the GE Healthcare web site redesign project.
The usability team’s primary role was to work directly with GE customers ultimately to deliver improved usability to the site. It was a daunting task because the diverse GE Healthcare site audience is very large, consisting of both internal and external customers. Internal customers include sizable, well-respected GE Healthcare sales and service staffs. External customers include medical technologists, clinicians, and purchasing decision-makers for a variety of medical facilities.
GE wanted the overall redesign to result in site navigation and information architecture that effectively supports all site customers and their tasks. Because implementation of the site would be a substantial investment for GE, and time-to-market a necessary consideration, GE chose to enlist usability professionals’ guidance to meet their rapid-development schedule while ensuring a customer-centered final product. In the web environment, “time-to-market” often is placed ahead of customers’ needs; however, GE recognized the criticality of selecting a final design based on objective customer data.
Using the complementary usability techniques described in this article, GE and the usability team gained a greater understanding of GE customer characteristics, needs, and typical tasks done on the GE site. Based on the results of this joint project, GE created a web site that effectively supports both its internal and external customers.
"GE wanted the overall redesign to result in site navigation and information architecture that effectively supports all site customers and their tasks. The usability team’s primary role was to work directly with GE customers ultimately to deliver improved usability to the site."
CONSECUTIVE VS. CONCURRENT METHODOLOGIES
In an optimal design schedule, user and task analysis, evaluation, testing, and prototyping occur consecutively, with each completed phase contributing to the design and implementation of the next phase. However, the reality of web design and development usually dictates shorter, more compressed schedules. With the time parameters of this project as tight as they were, we knew we needed to consider creative ways of accommodating GE’s need for fast turnaround without compromising good design practice.
We weighed the consequences of running project phases consecutively vs. concurrently as follows:
Consecutive. We understood we would have to cut back heavily on traditional user and task analysis activities if we wanted to do them consecutively. The advantage of running project phases consecutively, of course, was that we could more comfortably complete phases one at a time, learn from each as we went, and then build from each to implement the next. The disadvantage was that we ran the risk of compromising the quality of data gathered by “short-changing” the length of time we would normally spend on each phase.
Concurrent. To keep the integrity of the analysis tasks intact without compromising the schedule, concurrent activity seemed like the only solution. But how would we be able to build one component from the other with multiple phases taking place at the same time?
We solved this dilemma by assigning more usability specialists with a wide variety of skill sets to the project and designing a tightly woven schedule of concurrent activity that included regular, informal updates and formal information-sharing—with each other and with the GE development team.
To facilitate debriefs and enhance knowledge transfer, we also encouraged the GE team to observe as many usability sessions as possible.
MULTIDISCIPLINARY TEAM
To accommodate the multiple phases of the redesign project and adequately cover the skills needed to complete them, we assembled a large, multi-talented team of usability professionals and outside contractors, as described below.
The project manager (Cory Knobel) was responsible for keeping the “big picture” in full view. He drew on his experience as both usability specialist and project manager to act as the single point-of-contact and oversee all project phases through critical “user-centered eyes”. Cory also fulfilled the role of note-taking observer for the field study and usability test and was one of two reviewers for the heuristic evaluation of the subsequent redesign.
The senior usability specialist (Deborah Hinderer Sova) drew on her extensive practitioner, design, project management, and technical writing experience to conduct and manage the field studies and usability test, act as lead report writer, provide significant re-design recommendations, and present results to the GE team. Along with Cory Knobel, Deborah also was a reviewer for the heuristic assessment of the subsequent redesign.
An R & D director provided high-level human factors advice and design expertise throughout the project, especially during the paper prototyping phase. She also participated in the presentations of the evaluation results to GE.
A Ph. D. human factors specialist drew on his knowledge and expertise in both human factors and statistics to design and observe focus groups, design the majority of the usability test protocol, and analyze all quantitative evaluation data.
The graphic artist rendered paper prototypes for all three navigational models for the scenarios chosen for usability testing. She also acted as “computer” for the paper prototype usability tests.
Participant recruiter. The usability team understood that only truly representative users whose characteristics match those of the GE Healthcare site audience could provide us with the most reliable data for customer-centered redesign recommendations. Therefore, we engaged a professional recruiter with several established contacts in the geographical areas in which GE had key customers.
METHODOLOGIES USED
To ensure that we could gain as much knowledge as possible about the GE site customers, and use the data effectively, we employed several user-centered methodologies, as described below.
User and Task Analysis
To collect user and task data, we concurrently conducted six focus groups and three field-study sessions with both internal and external customers. Together, the focus groups and field studies produced both overlapping and distinct information about customer characteristics and tasks. The results helped confirm, and sometimes challenge, what GE believed about their customers and their customers’ tasks.
For example, the GE team were fairly sure, but wanted to confirm, that their customers generally followed a product-oriented navigational path when researching medical equipment online. Both the field study and focus groups confirmed that assumption, but the focus groups challenged another assumption. Discussion during the focus groups indicated that customers visit the site usually during budgeting and purchasing activities, which occur only a few times per year. Therefore, customers likely were not visiting the site as often as GE had previously thought.
As we completed focus-group and field-study sessions, we immediately fed critical customer-centered information into designing the usability test protocol and paper prototypes. The advantage, again, was time saved. The disadvantage was that unexpected results sometimes necessitated adjusting certain task scenarios for testing.
Focus Groups: We conducted focus groups with GE Healthcare sales and service staff, as well as external customers. The sessions gathered information about these groups’ characteristics and tasks and helped clarify customer requirements, preferences, and any difficulties they were having using medical systems sites (not necessarily just GE’s).
Field Studies: Representative participants were chosen from the external customer groups (clinicians, purchasing decision-makers, and technologists.) We scheduled two-hour sessions in the participants’ work places—the most common environments in which they access medical systems web sites.
We videotaped customers doing typical tasks on the GE Healthcare web site as they verbally described their actions and processes. We observed the environment and context within which the work was taking place, and asked clarifying questions to probe issues and further elicit customers’ site requirements.
Paper Prototyping
New Navigation Design Concepts. Before our usability team even came on board, GE already had developed preliminary concepts for three different navigation strategies. However, based on data acquired through the focus groups and field studies, we worked with GE in joint sessions to refine these navigation concepts, combining elements to reduce the number of all-new models we would develop to two.
Three Navigation Models. We worked with a graphic artist, who also had a background in web design, to refine sketches and navigation styles through the iterative design process. Reducing the number of all-new navigation concepts enabled us to include the current site’s navigation, in similar low-fidelity form, as the baseline for comparative usability testing. This process overlapped with development of the usability test task scenarios that ultimately dictated how much of the site to sketch for testing.
We created paper prototypes of the three models:
- “Model A” portrayed stable navigation, in which navigation bars remained constant on each page.
- “Model B” portrayed a “breadcrumb-trail” approach where the path taken by the customer displayed at the top of the page.
- “Model C” portrayed the current GE Healthcare site navigation, as a baseline.
We produced sketches of only those pages of the site needed for five unique usability test tasks, allowing up to two alternative paths. The resulting combined prototypes totaled 75 pages. All three models contained identical content, taken from the current site.
Comparative Usability Testing
Goal. The primary goal of the usability test was to provide findings and recommendations to help the GE team choose the most usable navigation structure and information architecture for site redesign. We designed a comparative test that evaluated the usability of the three navigation prototypes described above with a mix of 24 internal and external customers.
Process. The test included both performance and preference testing. For performance testing, we observed how easily and efficiently participants performed tasks and recorded time-on-task, errors, number of clicks, and success ratios. For preference testing, we administered Likert-scale questionnaires which quantified customers’ ratings of the model or models they preferred for navigation, ease of finding information, overall comfort, researching equipment, and purchasing equipment. In addition, we asked customers to list their favorite and least favorite features of each model.
Data. To collect qualitative data, we used a combination of protocol analysis (also called “thinking aloud” protocol) and direct observation. The qualitative inquiry focused on terminology and organizational problems encountered. We also solicited general feedback on home page layout. Unlike internal customer participants, external usability test participants were unaware of GE’s sponsorship of the study.
Users and Bias. The usability test included five distinct user groups, with six participants per group. Because each participant would be performing the same set of tasks on each of the three prototype models, we incorporated strict counterbalancing into the test protocol. Counterbalancing helped account for inter-model influence within the tests.
Protocol. In the counterbalanced protocol, each participant completed three representative tasks on the first model they were shown, and then filled in a feedback questionnaire for that model. We followed the same procedure for the second and third models, having participants perform the same tasks on all three models.
REPORTING
The success of this complex project was due in large part to the size and excellence of the usability team we assembled, as well as the multiple methodologies we employed. But we must give due credit to the extensive debriefing and reporting that occurred throughout the project, not just at the end.
Interim Debriefs. After each focus group and field study session, we emailed short “Quick Summaries” to the GE team, providing immediate results. However, because of the large number of test sessions, we phoned in reports at the end of each day of five days of testing.
Interim Results Presentations. At the conclusion of each phase of the project, we gave a PowerPoint presentation of key results and our preliminary recommendations. Because the field studies and focus groups were conducted concurrently, we presented the results for them in the same meeting. The usability test results were presented separately, as were the results of the follow-on heuristic evaluation, described near the end of this article.
Final Report. We prepared and presented a comprehensive report covering all phases of the project. Because of interim reporting and the GE team’s direct observation of many of the customer sessions, the final report was well received (no major surprises) and quickly understood by the GE team.
KEY RESULTS AND RECOMMENDATIONS
The customer studies resulted in GE moving forward with creating their new site based on the navigational concept most preferred of the three concepts tested by their customers. Even as we were completing usability testing and preparing the final written report, the GE team was redesigning their site based on information provided during debriefs and results presentations.
Key Results
Our studies showed that GE customers:
- Found Model B (“breadcrumb”) navigation the easiest to use as evidenced by the consistently low error rates, and the low time to complete the tasks.
- Chose a product-oriented navigational path for most tasks.
_ Were confused by inconsistent labeling and by links embedded in large text blocks.
- Would value a site that contains useful enough information to consider GE a “one-stop shop”; providing information that customers value beyond GE products could be very attractive.
- Felt security and confidentiality are very important.
- Would not purchase online; the role of the site, in their view, was complimentary to sales and service.
Recommendations
Because the primary goal of the engagement was to select one of the three navigation concepts, the report focused heavily on justification for our recommendations for the customers’ top navigational choice. In addition, because of the extensive anecdotal and incidental information gathered in the focus groups and field studies, we were able to suggest other interface enhancements, as well as communicate to GE some operational and CRM information that did not directly relate to the interface design.
Some of our recommendations to GE included:
- Because Model B outperformed the other models and customers preferred its inner-page and breadcrumb navigation, emulate Model B as much as possible in the redesign of the GE site.
- Make all areas of the site available through a product path.
- Make consistency a priority. Ensure that link labels and destination page titles match, placement of primary and secondary navigation is the same from page to page, and that the Home link is in the same place on every page.
- Avoid embedding links in bodies of text as the only or primary path to key information. Make the path to important information more direct.
- Provide the types of information customers say they want [here we went into great detail about what information to include].
FOLLOW-UP HEURISTIC EVALUATION
The GE graphic design team had begun to develop a high-fidelity prototype during the comparative usability testing phase of the project, based on early feedback from user and task analysis and frequent usability test-session debriefings.
One component of the plan for follow-up usability work we suggested was heuristic evaluation of the redesigned web site. GE engaged our team to conduct the evaluation.
In the spirit of earlier usability inquiry, we completed the heuristic evaluation in less than one week. We used a usability report format that would allow us to complete the evaluation and generate the report simultaneously.
Evaluation of ten high-level pages yielded new findings as well as some that echoed earlier evaluations.
With iterative usability evaluation, GE ensured that the customer data from the previous phases of the redesign project would be reflected in the new site. When we presented findings to the GE team, we continued to make a strong case for improving the overall consistency of the new site. Our recommendations supported and advanced the findings and recommendations borne out of the earlier project phases.
KEYS TO OUR SUCCESS
In retrospect, we acknowledge several lessons learned from conducting concurrent methodologies in a rapid development strategy. A number of factors contributed to our successful outcome, as summarized below.
Ensure that both the usability team and the client team understand the upcoming project challenges.
We were fortunate to work with a client team that actively coordinated with our own team and clearly understood that the project’s success would be contingent on adherence to the tight schedules we devised together. Some of the details of maintaining this critical component were:
- Strict timelines and schedules suggested in our original proposal then clearly defined in the planning phase, with timing contingencies written into the subsequent contract.
- An acknowledgement on both the client and the consultant sides that concurrently scheduling project components was a first for both parties.
- Designating individuals on both the client and consulting sides to specifically keep the “big picture” of the project in focus.
Assemble a multi-skilled, flexible team that embraces new situations.
A team such as the one we assembled would be a luxury for most projects. We were fortunate to be able to staff each project phase almost as if it were a separate project, with key overlapping staff to help dovetailing components proceed as smoothly as possible.
We benefited from assigning phase managers because it enabled us handle each phase as a mini project. From a staffing perspective, these assignments proved to be not only an efficient allocation of resources, but more humane to the project staff as a whole.
The rapid nature of the project called for specialists who were experts in their fields. The ability to complete activities quickly and accurately was critical in providing data that would fit into coordinated project components.
While deciding on staff for various phases, we mapped out the skills needed for completing each phase and matched them with consultant skills. Realizing that skills required for different phases sometimes resided within one consultant, we carefully constructed a schedule that would allow for overlap without compromising the quality of any one consultant’s performance.
Be willing to involve outside talent who can complement and enhance your own team’s strengths.
One of our team’s strong points was recognizing when to outsource for skills either not represented in our group, or not able to be covered simultaneously by current staff.
For example, the resources required to recruit participants for all focus group and usability test sessions in the short time frame was completely out of our purview. The person on our staff with the most recruiting expertise was already assigned to conduct field studies and usability tests and oversee paper prototyping. In addition, she was new to the geographical area in which we would be recruiting.
Therefore, we budgeted for and then engaged an outside recruiting firm with many contacts in the area. Although we had mostly excellent results, we subsequently decided to budget for internal recruiting training.
And, although all our team members had sketched web pages, again, we were assigned full time to other project components. In addition, we needed a highly professional level of artistic skill to run a comparative test using only paper prototypes. Therefore, we engaged a top-notch graphic designer who could handle the challenge of producing 75 pages of high-quality paper prototype web pages for this project.
Make strong communication a top priority.
Much of the successful relationship built between GE and our usability team was due to regular and frequent communication. The need for carefully timed coordination of resources on both sides of the engagement called for excellent communication.
Early on, we decided that meeting the challenge of quick turnaround times would be best served by project managers acting as single points of contact for the duration of the engagement. Because of this communication style, we could take efficient action on project decisions.
The GE and usability team project managers spoke on a nearly daily basis. They would then provide information and feedback to all their team members. The frequency of cross-team communication allowed interim reports and discussions to be brief, saving resources and eliminating the need to write lengthy status updates.
Despite our talented teams, diverse skills, and tight coordination of resources, we found that clear communication was absolutely essential for meeting our project schedule and helping the GE team design a more user-centered web site.
Comments