Conducted by Dwayne PearceINN530 – Online Information ServicesAssignment 2Usability and its use for AcademicLibraries: A case study of an onlinelearning tool
Presentation AgendaBackground to ProjectOverview Key ConceptsResearch Question, Objectives, and OutcomesData Collection and AnalysisFindingsDiscussion - What can we learn from this?Discussion – Limitations of ResearchRecommendations
Background to Project• This project is a part of a greater work that has beencommissioned by South Bank Institute of Technology toinvestigate the perceived effectiveness of SAIL. • The research used to help improve the product and provide userswith a more interactive and dynamic learning tool.• The importance of usability testing to the overall redesign andmaintenance of an online based resource.• The ways in which usability testing can improve the userexperience of an online resource.
Research Question, Objectives, andOutcomesResearch question –•How usable is the online tutorial tool SAIL from a student’s perspective?Research Objectives•The research project focuses on the following objectives:•To explore the student user experience of SAIL through conducting usability testing.•To examine the value of usability testing in libraries.•To develop an understanding of the ways usability testing can be used by libraries to improve theonline information literacy tools they deliver.•Research Outcomes•The researcher expects the following outcomes from the project:•To provide insight into the usability of the SAIL tutorial to inform further development of the site.•To raise awareness of the value of usability testing in libraries.•To contribute to further understanding about usability considerations for online information literacytools to the library and information studies research space.
Data Collection Participants were selected based on the following criteria which have been outlined by Southbank Institute of Technology: •The students must be a student of Southbank Institute of Technology.•The student must be studying a course that is certified as Certification III and above. A usability test was then conducted in which 5 participants will be observed using SAIL with a small list of tasks they must perform. Thistests all consisted of 3 main components: •Task based usability testing – The participant must complete a series of tasks, and data will collected concerning completion time,completion rate, and ease of use.•Think aloud protocol - The test users are asked to verbalise their thoughts, feelings, and opinions while interacting with the system(Nielsen, 1993).•Post Session Questionnaire – The test user will be asked a series of questions based on their interaction with the SAIL tool. The data was collected in two forms:•Quantitative data: Success rates, task time, error rates, and satisfaction questionnaire ratings.•Qualitative data: Observations about pathways participants took, problems experienced, comments, and answers to open-endedquestions. (U.S. Department of Health & Human Resources, n.d., para. 2)
Data Analysis Following the usability test, an analysis of the results was presented. This section contained an analysis of:•The percentage of participants who completed each task, and all tasks.•The average time taken to complete each task for those who completed the task.•The satisfaction results.•The tasks that had the highest and lowest completion rates.•A summary of the successful task completion rates by participant, task, and average success rate by task and show the data in a table.•An examination of the answers provided to open-ended questions.•A summary of the post session questionnaire results•A description of the data recorded during the usability test.•An analysis of the problems encountered during the usability test. (U.S. Department of Health & Human Resources, n.d., para. 2) The data analysis was presented in the following forms:•Severity ratings, which are comprised of:•The frequency with which the problem occurs.•The impact of the problem if it occurs.•The persistence of the problem.•(Nielsen, n.d., para. 2)•Descriptive statistics that identify user performance issues such as errors and difficulties, and tasks that do not meet acceptable criteria e.g. the number oferrors that occur on tasks and the number of users who successfully perform tasks (University of Texas, 2011, para. 2).•Thematic coding, identifying general trends or categories of users’ behaviour such as how well users can complete a task and where they are encounteringproblems, users’ general perceptions and opinions of the technology’s usability (University of Texas, 2011, para. 3).•Observation notes that describe and identify themes and trends in the data collected which will then be presented in a table format (University of Texas,2011, para. 4).
Findings – Quantitative ResultsTask Number and Task Description Percentage of participants who completed eachtaskTask 1: Use SAIL to reference a journal articleusing APA Referencing conventions.100%Task 2: Use SAIL to find an example of aninstruction word.0%Task 3: Use SAIL to find the section “How do youget copyright permission?”100%Task 4: Use SAIL to find the section “The RoughDraft.”100%Task 5: Use SAIL to find the section “Locatinginformation: Using databases.”60%All tasks 0% Table 1:The percentage of participants whocompleted each task, and all tasks.
Findings – Quantitative ResultsTask Number and Task Description Average time taken to complete task Task 1: Use SAIL to reference a journal articleusing APA Referencing conventions. 30 secondsTask 2: Use SAIL to find an example of aninstruction word. 1:30:00 (100% unsuccessful)Task 3: Use SAIL to find the section “How doyou get copyright permission?” 15 secondsTask 4: Use SAIL to find the section “The RoughDraft.” 25 secondsTask 5: Use SAIL to find the section “Locatinginformation: Using databases.”45 seconds Table 2:The average time taken to complete each taskfor those who completed the task.
Findings – Quantitative ResultsTask Number and Task Description Average Satisfaction result Task 1: Use SAIL to reference a journal articleusing APA Referencing conventions. 0Task 2: Use SAIL to find an example of aninstruction word. 3 (100% unsuccessful)Task 3: Use SAIL to find the section “How doyou get copyright permission?” 0Task 4: Use SAIL to find the section “The RoughDraft.” 2Task 5: Use SAIL to find the section “Locatinginformation: Using databases.”2 Table 3:Satisfaction resultsScale used –0 = User completed task with zero difficulty. (ZeroFrustration)1 = User completed task with only minor problem(s).(Little Frustration)2 = User completed task, but it required moreeffort/time/dead-ends than the user expected.(Medium/High Frustration)3 = User did not complete task. (Point of Failure)
Findings – Qualitative ResultsTask Number and Task Description Recurring words, phrases, and expressionsexpressed throughout the Think Aloud protocolTask 1: Use SAIL to reference a journal articleusing APA Referencing conventions.“Where is it?”, “Easy to find”, “Should beseparated from SAIL”, “Why is referencing partSAIL?”Task 2: Use SAIL to find an example of aninstruction word.“I have no idea where to look”, “It isn’t obvious”,“Do I click the boxes?” “I really have no clue” “Itseems important, so shouldn’t it be easier tofind?”Task 3: Use SAIL to find the section “How do youget copyright permission?”“Easy to find”, “This was a lot easier to find”, “Apop menu on the side would be nice, on thecopyright section”, “Found it easily”Task 4: Use SAIL to find the section “The RoughDraft.”“Again, this was easy to find”, “Pretty easy tofind”, “Is it near Assignment Calculator? It shouldbe”, “Again, pop out menus would work better”Task 5: Use SAIL to find the section “Locatinginformation: Using databases.”“Can I click on the pictures?”, “What do thepictures do?”, “Is it one of the pictures?” “Itshould be at the beginning, as one of thepictures” Table 4:Recurring words, phrases, and expressionsexpressed throughout the Think Aloud protocolfor those who completed the task.
Findings – Qualitative ResultsTask Number and Task Description Recurring words, phrases, and expressionsexpressed throughout the Post SessionQuestionnaire What is your overall impression of SAIL? Words:Clearer, bigger presence, hidden, reference tool, moreobvious, more accessiblePhrases:“An effective tool to help students reference”, “It couldbe more detailed”, “It is important to use, but it needs tobe a little clear” “Needs to be more obvious on the libraryhomepage, I didn’t know it was there”From your experience, what did you like bestabout SAIL?Words:Current, referencing help, assignment help, simple to usePhrases:“It provides a lot of information”, “Very simple, andreally helps provide an outline for assignments”, “ Themind maps and other activities help get your mindgoing”, “It is really good to return to freshen up onassignment writing if you haven’t studied for a long time” Table 5:Recurring answers provided during the Post SessionQuestionnaire
Findings – Qualitative ResultsFrom your experience, what did you like leastabout SAIL?Words:Lost, awkward, clumsy, hard to read, too“wordy”, too few picturesPhrases:“ I think that the colour scheme is too dated”,“It is too hard to read” “There is too much text”,“It is too hard to navigate”, “I don’t like theboxes”, “I find myself getting lost”If you were the website developer, what wouldbe the first thing you would do to improve thewebsite?Phrases:“Make it more of a feature on the library page”, “Make itmore visible”, “Make it fit the page better, lesslandscape”, “Create a more visually interesting homepage”, “Make the boxes easier to see”, “Put referencing asa separate tab on the library homepage”, “Make the sidebar more functional” Table 5 continued:Recurring answers provided during the Post SessionQuestionnaire
Findings – Qualitative ResultsIs there anything that you feel is missing on thissite?Phrases:“A title at the top of the page to stop me gettinglost,” “Something that tells me where I am,maybe at the top of the page so I don’t get lost”,“The boxes aren’t functional enough, they needto let you know where you are going”, “Popout menus”, “Better links on the sidebar” If you were to describe this site to a colleaguein a sentence or two, what would you say?Phrases:“Invaluable to students, but needs to be promotedbetter on the library site”, “Could be a useful tool if Iknew what was being offered in greater detail”, “Useful,but too difficult to navigate”, “A great place to go forreferencing, but too hard to find your way around”,“You might need to take a survival kit through if you getlost” Would you use SAIL again? Phrases:All of the respondents interviewed said yes they woulduse SAIL again, but would use it more frequently if thesite were changed in the ways they suggested. Table 5 continued:Recurring answers provided during the Post SessionQuestionnaire
Discussion – What can we learn fromthis?This report seeks to bolster the belief that the iterative designapproach is “the only evidenced-based way to improve the quality ofcomputer-based systems, including Web sites and web applications”(Usability.gov, 2005, para. 8).Although this report only contains one usability test, and does notdocument the redesign of the SAIL site that will follow this report,the aim is to provide findings that promote the method of iterativedesign usability testing, as the literature reviewed for this projectsuggests that “revising a system based solely on one lone usabilitystudy does not lead to better user performance or increased usersatisfaction” (Usability.gov, 2005, para. 8).
Discussion - LimitationsSelection bias on behalf of the researcherSelection bias of Think Aloud Protocol and PostSession Questionnaire Answers Research not reaffirming the role of IterativeTesting in the Usability Process
RecommendationsRecommendations for the future redesign of SAIL:Decrease the amount of clicks a user needs to open the main SAIL page.Make SAIL easier to navigate by developing side menus, pop out windows, and othernavigational tools that allow the user to see not only the page they are currentlyvisiting, but other pages that are associated with the module that they are viewing.Separate the referencing component of SAIL from the current SAIL page to make it asingular stand alone feature of the library site.As iterative design promotes design, testing, redesign, and retesting, it isrecommended SAIL be redesigned and retested using this project as a means to framethe next series of usability tests.