WordPress.org

Ready to get started?Download WordPress

Contributing to WordPress, Part III: Usability Testing

Posted May 4, 2009 by Jen Mylo. Filed under Development, User Interface.

One of the reasons WordPress 2.7 was such a success is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in the application. That kind of thing? Takes a lot of time. :)

For readers who aren’t familiar with the process behind usability testing, here’s an overview. First, determine the scope of your test and create a test protocol/script. Determine the audience segments to be included in the test group(s), and begin recruiting. Recruiting may mean hiring an agency to find participants, but for testing WordPress, it makes more sense to recruit from within this community, so that means making a screening survey, reading all the responses, segmenting the respondents into categories and contacting people until you’ve filled your desired quotas (for whatever segments you’re seeking, such as newbie, experienced user, developer, CMS user, photoblogger, mobile user, etc. ). Then come the test sessions.

Depending on what is being tested, these last anywhere from half an hour to an hour and half apiece. Sessions are generally recorded using screen capture and web cam, with a video camera for backup. The moderator(s) generally take notes during sessions and/or (depending on what software is being used for the session capture) set markers in the video to indicate task completion, comments of interest, etc.  In some cases, auxiliary test methods such as eye-tracking may be included. When the sessions are complete, the results are analyzed. All the notes and videos are reviewed, patterns are identified, and ultimately a report is written and the feedback informs the next round of revisions.

Some people think it shouldn’t take much time to do all this. I’ve lost count of the number of people who cite an old article by Jakob Nielsen that says you don’t need to test with more than 5 users because usability issues become clear right away. While I’ve found that to be generally true, when your user base is as diverse in experience level, usage, platform  configuration, language (right to left languages have a pretty different experience) and demography as the WordPress community is, 5 users really isn’t enough to get a clear picture. We try to test with at least a dozen people each round, but then we are limited to a geographic region (test in NY, test in SF, or test wherever we can schedule enough people back to back to make it worthwhile), while WordPress users are located all over the globe.

To address this, we’re introducing a set of new contribution opportunities to expand our usability testing program. As with development and graphic design, we’re going to create an infrastructure to allow community participation in usability testing on a regular basis and in a much broader capacity than existed before, when it was limited to announcements that we needed participants in x city on y date. We’ll be looking for volunteer moderators as well as participants, hopefully from all over the world.

Moderators. Observational usability testing isn’t rocket science, but neither is it a simple task to reduce bias. Because of this, at first we’ll choose only moderators who have professional experience conducting usability tests. People who conduct testing for design agencies, software companies, usability consulting firms and the like will be our first round draft picks. In the future, when we have a group of regular volunteers and have ironed out any kinks in the process, we’ll ideally match up experienced testers with aspiring ones, using a mentorship model to increase the number of people who can contribute in this fashion.

Participants. If you use WordPress, chances are you could participant in a usability test at some point. In some cases we’re looking for particular behaviors (people who upload large video files, people who blog from their iPhone, people who manage more than 5 blogs, etc.), while other times the behaviors we’re looking for are much more common (do you have widgets in your sidebar, have you changed themes in the last 6 months, is there more than one author on your blog, etc.).

So how will these opportunities come into play, and how will it make WordPress better?

We’ll start with the moderators, and try to get volunteers with a decent geographic spread. Then, we’ll start signing up potential test participants in those areas (though we’ll also allow at-large registrations, since traveling testing will still be happening). We’re working on a registration process for potential participants. You’ll enter your basic info (location, contact info) and answer some questions about your WordPress usage to be entered in the database, and when there’s a testing opportunity coming up that’s appropriate for you, a local moderator will get in touch to see if you’re interested. Further down the road we may experiment with remote testing and other methods, but for now, this approach will broaden the geographic scope of our testing.

All moderators will follow the same test protocols and script, and their results/reports/video will be reviewed and collated, with a composite report (including the protocol/script that was used) published to the community. This will provide designers and developers with broader feedback during the dev cycle, and will allow the wider community to both understand and participate in the testing program.

If you’re interested in being a moderator during this initial phase (meaning you do it professionally), send me an email and introduce yourself. If you’re interested in signing up as a potential test participant, watch this space. We’ll post a link to the registration survey once it’s ready.

19 Pings

RSS feed for comments on this post.

  1. [...] community of super loyal fans to tap for testing, and this morning that’s just what the company announced it is going to do. WordPress usability testing is being opened up to the community of [...]

    Pingback from WordPress Wants Your Help With Usability Testing | GroupHelp.NET - Easy everything! on May 4, 2009

  2. [...] community of super loyal fans to tap for testing, and this morning that’s just what the company announced it is going to do. WordPress usability testing is being opened up to the community of [...]

    Pingback from web: WordPress Wants Your Help With Usability Testing | Alseek:The Academic Search Engine on May 4, 2009

  3. [...] 以下は、2009 年 5 月 4 日に書かれた WordPress.org 公式ブログの記事、「Contributing to WordPress, Part III: Usability Testing」を訳したものです。 [...]

    Pingback from WordPress | 日本語 » WordPress への貢献、その 3 : ユーザビリティテスト on May 4, 2009

  4. [...] WordPress is an good example of open source gone right. I have side-stepped numerous times to other comparable scaled CMS (Typepad, Drupal, Xoops, etc..) but always return to WordPress. As well as wearing semantic web on their sleeve, there is a host of plugins to extend your website. Calling WordPress a blog system is very underestimating its capability. And it is surprising how many web savvy people dont understand that a blog post, an article, and static web page are essentially the same thing, i.e. a template, i.e. data and design. One of the reasons WordPress 2.7 was such a success is the amount of usability testing that took place during the development cycle.Contributing to WordPress, Part III: Usability Testing [...]

    Pingback from Usability Testing 4 WordPress - jaffamonkey on May 4, 2009

  5. [...] equipa do WordPress (o software de blogging que uso aqui no meu blog) está a recrutar participantes e moderadores voluntários para conduzirem testes de usabilidade ao interface de administração do WordPress. Os [...]

    Pingback from WordPress recruta moderadores e participantes para testes de usabilidade | IvoGomes.com on May 4, 2009

  6. [...] of super loyal fans to tap for testing, and this morning that’s just what the companyannounced it is going to do. WordPress usability testing is being opened up to the community of [...]

    Pingback from Big Brass Blog on May 4, 2009

  7. [...] Contributing to WordPress, Part III: Usability Testing [...]

    Pingback from Daily Digest for 2009-05-04 | Education Inkwell on May 5, 2009

  8. [...] » Contributing to WordPress, Part III: Usability Testing [...]

    Pingback from Linkwertig: Petition gegen Internetfilter, simfy, produki, Google Goats » netzwertig.com on May 5, 2009

  9. [...] 如皋说社交软件有什么做的不够的话,那无疑是可用性测试。WordPress优势就在于它有一个全球化的忠实粉丝社区可以用来协助测试,而今早,公司就宣布了这样的计划(link)。WordPress的可用性测试将对社区用户敞开大门。 [...]

    Pingback from WordPress的可用性测试需要你的帮助 - 读写网唯一官方中文站 - 搜狐IT独立群体博客 on May 6, 2009

  10. [...] 如皋说社交软件有什么做的不够的话,那无疑是可用性测试。WordPress优势就在于它有一个全球化的忠实粉丝社区可以用来协助测试,而今早,公司就宣布了这样的计划(link)。WordPress的可用性测试将对社区用户敞开大门。 [...]

    Pingback from WordPress的可用性测试需要你的帮助 « 每日IT新闻,最新IT资讯,聚合多站点消息,保证你与世界同步 on May 7, 2009

  11. [...] Contributing to WordPress, Part III: Usability Testing May 4, 2009 One of the reasons WordPress 2.7 was such a success is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in the application. That kind of thing? Takes [...] [...]

    Pingback from jhjhkhjkhjlhj | :: HAJIMARI :: on May 7, 2009

  12. [...] Wells who was hired on to the Automattic team because of her usability specialty has published a post on the WordPress development blog initiating phase three, Usability Testing, in her ongoing series, how to contribute to WordPress. [...]

    Pingback from Your Chance To Make WordPress Better on May 7, 2009

  13. [...] Wells who was hired on to the Automattic team because of her usability specialty has published a post on the WordPress development blog initiating phase three, Usability Testing, in her ongoing series, how to contribute to WordPress. [...]

    Pingback from Your Chance To Make WordPress Better - Word Press Magazine on May 7, 2009

  14. [...] how the new interface for WordPress 2.8 is holding up and where some improvements can be made. In Contributing to WordPress, Part III: Usability Testing, she explains the process of how the usability testing works, and how WordPress is going to tap [...]

    Pingback from WordPress News: BuddyPress News, WordPress 2.8 News, WordCamp Toronto & Virginia, WordPress.tv Feeds, and More | The Blog Herald on May 8, 2009

  15. [...] you fit the bill, and have some time to give to help improve WordPress, check out this post for more [...]

    Pingback from Help WordPress Devs With Usability Testing « Lesley Harrison’s Blog on May 8, 2009

  16. [...] WordPress优势就在于它有一个全球化的忠实粉丝社区可以用来协助测试,而今早,公司就宣布了这样的计划(link)。WordPress的可用性测试将对社区用户敞开大门。 [...]

    Pingback from 爱上网 » 邀你参与WordPress的可用性测试 on May 12, 2009

  17. [...] WordPress › Blog » Contributing to WordPress, Part III: Usability TestingTo address this, we’re introducing a set of new contribution opportunities to expand our usability testing program. As with development and graphic design, we’re going to create an infrastructure to allow community participation in usability testing on a regular basis and in a much broader capacity than existed before, when it was limited to announcements that we needed participants in x city on y date. We’ll be looking for volunteer moderators as well as participants, hopefully from all over the world. [...]

    Pingback from Fábio Caparica » del.icio.us entre 02.05.2009 e 15.05.2009 on May 25, 2009

  18. [...] telling you about the Crowdsourcing Usability effort we had launched. Since then we’ve seen that WordPress have launched a similar campaign and they managed to get coverage in the New York Times no less, so we will be watching their [...]

    Pingback from Johnny Holland - It’s all about interaction » Blog Archive » Drupal 7 UX: Reflecting between Iteration Zero and Iteration #1 on June 15, 2009

  19. [...] how the new interface for WordPress 2.8 is holding up and where some improvements can be made. In Contributing to WordPress, Part III: Usability Testing, she explains the process of how the usability testing works, and how WordPress is going to tap [...]

    Pingback from WordPress News: BuddyPress News, WordPress 2.8 News, WordCamp Toronto & Virginia, WordPress.tv Feeds, and More | OpenABlog on July 30, 2009

See Also:

For more WordPress news, check out the WordPress Planet.

There’s also a development P2 blog.

To see how active the project is check out our Trac timeline, it often has 20–30 updates per day.

Categories

%d bloggers like this: