Measuring the User Experience on a Large Scale:
User-Centered Metrics for Web Applications
Kerry Rodden, Hilary Hutchinson, and Xin Fu
Google
1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
{krodden, hhutchinson, xfu}@google.com
ABSTRACT
More and more products and services are being deployed
on the web, and this presents new challenges and
opportunities for measurement of user experience on a large
scale. There is a strong need for user-centered metrics for
web applications, which can be used to measure progress
towards key goals, and drive product decisions. In this
note, we describe the HEART framework for user-centered
metrics, as well as a process for mapping product goals to
metrics. We include practical examples of how HEART
metrics have helped product teams make decisions that are
both data-driven and user-centered. The framework and
process have generalized to enough of our company’s own
products that we are confident that teams in other
organizations will be able to reuse or adapt them. We also
hope to encourage more research into metrics based on
large-scale behavioral data.
Author Keywords
Metrics, web analytics, web applications, log analysis.
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User
Interfaces—benchmarking, evaluation/methodology.
General Terms
Experimentation, Human Factors, Measurement.
INTRODUCTION
Advances in web technology have enabled more
applications and services to become web-based and
increasingly interactive. It is now possible for users to do a
wide range of common tasks “in the cloud”, including those
that were previously restricted to native client applications
(e.g. word processing, editing photos). For user experience
professionals, one of the key implications of this shift is the
ability to use web server log data to track product usage on
a large scale. With additional instrumentation, it is also
possible to run controlled experiments (A/B tests) that
compare interface alternatives. But on what criteria should
they be compared, from a user-centered perspective? How
should we scale up the familiar metrics of user experience,
and what new opportunities exist?
In the CHI community, there is already an established
practice of measuring attitudinal data (such as satisfaction)
on both a small scale (in the lab) and a large scale (via
surveys). However, in terms of behavioral data, the
established measurements are mostly small-scale, and
gathered with stopwatches and checklists as part of lab
experiments, e.g. effectiveness (task completion rate, error
rate) and efficiency (time-on-task) [13].
A key missing piece in CHI research is user experience
metrics based on large-scale behavioral data. The web
analytics community has been working to shift the focus
from simple page hit counts to key performance indicators.
However, the typical motivations in that community are
still largely business-centered rather than user-centered.
Web analytics packages provide off-the-shelf metrics
solutions that may be too generic to address user experience
questions, or too specific to the e-commerce context to be
useful for the wide range of applications and interactions
that are possible on the web.
We have created a framework and process for defining
large-scale user-centered metrics, both attitudinal and
behavioral. We generalized these from our experiences of
working at a large company whose products cover a wide
range of categories (both consumer-oriented and business-
oriented), are almost all web-based, and have millions of
users each. We have found that the framework and process
have been applicable to, and useful for, enough of our
company’s own products that we are confident that teams in
other organizations will be able to reuse or adapt them
successfully. We also hope to encourage more research into
metrics based on large-scale behavioral data, in particular.
RELATED WORK
Many tools have become available in recent years to help
with the tracking and analysis of metrics for web sites and
applications. Commercial and freely available analytics
(13 Seiten)
Manymanuals.com
Manymanuals.de
Manymanuals.fr
Manymanuals.it
Manymanuals.pl
Manymanuals.cz
Manymanuals.es
Manymanuals-pt.com
Kommentare zu diesen Handbüchern