“Wait… if I read the title correctly… are you talking about how people should go about using a gorilla?!?” No silly… it’s a play on words. What I am talking about is a usability study that @michaelseidel and I conducted recently @AJBombers. I want to share how you can do usability studies without breaking the bank.
Traditionally, I have participated in usability studies that have been conducted in a lab. We recruit about 10-15 users and pay them about $100 each to come in and test a website in a lab while designers, developers and business owners sit behind 2-way mirrors and watch the subjects interact with a website. We double book each time slot to make sure we get a participant in every slot and we still pay the people we don’t use. We write scenarios for the users to walk though and we have a couple of usability specialists walking the users though the scenarios and taking copious notes. We analyse the results after each user and compile all the information into a recommendation document.
These type of sessions were the status quo in the past. Unfortunately, these sessions were expensive and time consuming. They were necessary at the time, but the 800 lb. Gorilla in the room (see what I did there) was the cost of these sessions. There may still be a very limited place for this type of testing; however, there is another way…
So here is what we did… We are working on a section of a website that has a search mechanism and refinement tools that help the user narrow the results based on criteria they select. The refinement tools could be placed in 2 different spots on the results page. We wanted to test what position gave the best user experience for these tools.
We didn’t have a lot of time to plan a full blown (gorilla) usability lab and we didn’t have the budget for it either. So what we did is turn to social media and offered free lunch.
We started kicking around ideas on how to get good user testing without breaking the bank when it hit us; we could turn to Twitter to recruit users for this and we need to set up the testing somewhere that is convenient for our local networks to meet. We also need some sort of compensation for these users time. We came up with the perfect spot – AJBombers.
I decided to contact Joe from AJbombers and bounce the idea off him. I told him that we were going to conduct a user study and test some options of a website. I also mentioned that we were planning on buying all our participants lunch at his place. He was more than happy to help us out, in fact, he was excited about it.
We started out by broadcasting via Twitter what we were planning to do and when. Joe helped us out by Re-tweeting it to his loyal following which spurred a good response. The great thing about this entire approach was we planned it on a Friday for the following Wednesday and we filled all the spots we were looking for with very little effort. A few tweets and some re-tweets and we were good to go.
On the day of the testing Joe gave us a prime location to set up so we could funnel users through during the time slots we had set aside and we had a great system for getting food and drink orders started with our awesome waitress “B the Tweetless.” while the testing was going on. I was completely amazed how smoothly the entire session went seeing as this was our first attempt at doing anything like this.
The test went great; we noticed that the atmosphere is more realistic in a setting like that compared to a lab environment. The lab comes off as clinical and stiff. This was relaxed and the users seemed more comfortable in this enviroment… less spotlighty. People were willing to have a conversation about what we were testing; it didn’t come across as a rigid question and answer session. The feedback we gathered was genuine and gave us insight into things about the site we weren’t even testing. It also seemed like people were more grateful with receiving lunch than paying them cash to come out to our lab. “This is it? This is all I need to do? I feel like I need to do more to receive a free lunch.”
The test was a major success. We were able to test 10 users over 3 hours and under $200 total. This method of testing is completely viable, portable, and cost effective. This will now become our preferred method of testing moving forward. Granted, there will still be times where we have to bring people into a lab because we are testing something bigger that takes longer to run users through, but I think those sessions will be fewer and further between.
Finally, I would like to extend a huge Thank You to Joe and his staff @AJBombers! They really helped us out and that place is truly the best usability lab on the planet.
Tags: Usability
+1.414.303.4413
Hi Mike,
Interesting post. Stupid question, but what exactly were you testing? And how did the results pan out?
I’m part of a team that’s developing a new online usability tool at intuitionhq.com – it’s also a really simple concept, although the concept is slightly different. Usually our users send the tests on to their clients so they can get feedback from that end, and help the clients understand and feel more involved in the design process, without actually over involving them, if you know what I mean.
If you’d like to have a try with out service, you can flick me an email, or hit us up on twitter @intuitionhq and I can give you some free tests to try it out.
Cheers,
Jacob.
We were testing some on-sale functionality for a travel website. We basically tested the search capability and the refinement tools for the results. Ultimately we were trying to see what the best user experience is for the refinement tools. We had two scenarios that tested the refinement tools together at the top of the results or running down the side of the results seen typically in retail shopping sites.
We found that it really came down to user preference. The designs we had were very usable either way and the users did not have issues, the just had preferences. Thanks Jacob, I will email you in a bit. I am very interested it what you are working on.