How We Tested Web Sites
Finally, there's some description of the methodology used in testing, which is important to consider in assessing the applicability of their tests to a given site.
Types of Questions
Examples are provided of the questions that they asked users of the various sites.
For example, some of the questions asked about Fidelity's site were:
- Simple Fact: How many mutual funds does fidelity offer?
- Judgment: Would [a specific] mutual fund be a good investment for retirement?
- Fact comparison: Which mutual fund has a better one-year market return?
- Judgment Comparison: Which international funds would be good for a conservative investor?
Questions were formulated by looking into the actual content of the site, then testing questions with more web-savvy users to ensure they could be answered with a "reasonable" amount of effort.
The example given suggests there may have been some bias not only in the user's familiarity with the Web, but also with the subject matter.
Sampling/Methodology
The group sought users who had a moderate degree of familiarity with web browser software, but it ranged greatly: from two weeks to two years of experience, from people who used the Internet less than an hour a week to those who used it more than four hours a day.
All users were set up on the same machine, with the same browser, and bookmarks set to the sties' home pages. The researchers acted as passive observers, not prompting users to "think aloud", but recording what they happened to say and do.
Additionally, a post-task questionnaire was administered after the completion of each task to collect scalar data. These numerical scores were tabulated to arrive at a comparison of the sites' ratings.
How Good Is Good Enough?
The study allowed the researchers to compare various sites and draw some general observations about user behavior on the Web. It was not to assess whether any given site was "good" in and of itself, merely the degree of ease with which a user could perform a specific task.
EN: I expect this note is to assuage any bent noses on the part of comapnies who feel this book is unfairly critical of their Web sites.