Usability Testing & Reflection

Where data and reflection meet.

Usability Testing

According to Steve Krug (2020), the usability testing process is a way to see how users interact with a product and where they struggle, rather than relying on their opinions. He also states that even simple, frequent "discount" tests with a small sample group of three to five users are valuable throughout the design process. The aim is to quickly uncover and fix any major problems before they become difficult or too costly to fix later.

Planning and Conducting Usability Testing

I selected 10 educators across several school districts, including a librarian, a CTE Director, two CTE Coordinators, a Math teacher, two English teachers, and three CTE teachers, and sent them an email asking them to participate in my survey. My usability testing lasted one week to give my stakeholders plenty of time to review my course material and explore each unit, and then to answer the survey once they had finished. The email looked like the image on the right.

Google Classroom Link

Class Code: jzbtqpp3

The email included a brief background section to help the stakeholder understand the purpose. Followed by a detailed section of instructions that stated the following:

Please join my personal Google Classroom
Review the material
Complete a short usability survey after looking through the course. The survey has 16 Likert-scale items (Strongly Disagree → Strongly Agree) and 4 open-ended questions. Once I receive feedback, I will compile results for my instructor, but will not include names due to FERPA/HIPAA.

Finally, it had the deadline I needed them to submit their feedback by in order for me to compile my data.

Once they assessed the Google Classroom material, the survey had five parts, including the open-ended questions. The first category was on course structure and content. The second part was on learning activities and assessments. The third section covered technology and resources. The fourth part asked about Instructor Effectiveness. Then there were 4 open-ended questions for them to provide deeper feedback. I even had one teacher discuss their thoughts in person after they finished their submission.

What worked:

Very organized and easy to navigate
From a CTE Coordinator, "The variety of career clusters that are offered."
From one English teacher, " I truly loved the real-world application and assessments that coincide with each lesson. It allows the students hands-on knowledge and experience to foster their learning and enhance their expertise."
From a CTE teacher in another school district, "The organizational and flow of the unit was fantastic and great for any school looking to implement this content! The overview of the calendar, with the TEKS, activities, and assignments listed with an overlook, was the best tool for me and could be the most effective, as you can see the year planned out with all the activities and modules covered."
Having Step-by-step instructions.

What didn't:

Initially, sent the wrong link to join Google Classroom. Users were unable to log in.
I had also drafted all my material and forgot to post it so my "students" could see it. So I had to post all my material.
Some videos seemed long.
Possibly overwhelming my stakeholders with the whole year vs. just the first two modules.
I quickly put the test together so they had time over Thanksgiving break to assess. Take a little more time to double-check my work.
I need to give more guidance on where to start, rather than letting them just join and browse.

Suggestions for improvement:

I received two pieces of feedback for improvement.

As mentioned, possibly implement a real-world guest speaker to talk with the students in real-time in order for them to acquire first-hand knowledge of the various careers. Otherwise, excellent job!
This is a job well done on this Google Classroom! I hope this course can be used by students one day! The only suggestion I could think of is a more real-world application thought, maybe under one of the topics, is to either visit a job or business with like-minded students for experience, if possible, or have a few guest speakers potentially come in from VR or any of the topics covered in the GC. Great job!

Learner Support:

To support students who are new to my Google Classroom, I’ve added the following:

A short video walkthrough of the LMS layout
I will create more instructional videos for students on assignments so they can work on them at home, if they are out sick.
Reword any instructions to be detailed and precise so students are able to work independently or able to learn at their own pace

Feedback Summary/ Results

As much as I would have liked to have screen or audio recordings of my users' feedback, that was not possible with this testing group. Therefore, I was happy to take the informational survey. Out of the 10 educators I emailed, I received 6 responses. I initially planned to ask my students to test the material as well, but I felt I would not get proper results from them. I felt my fellow educators would offer honest insight into what did and did not work and any changes needed, as this is the LMS platform we use within our district and how our organization operates.

Participants rated the first 16 questions on a 1-5 Likert scale (1 = lowest, 5 = highest). I averaged each question's scores, then computed the overall mean of those averages. The final rating for the class material was 4.6875 out of 5 possible points.

Instructional Design Overview

Quick Links

Instructional Design

Final Reflection:

This was a great process for bringing real-time feedback/data in a digital age and for utilizing a digital course design. I believe the group of stakeholders who responded to my survey provided valuable feedback. I also believe it was honest, and they shared their true feelings. When creating my Likert scale, I believe I included a good variety of questions to elicit diverse feedback across several topics.

This experience confirmed that my class design was well laid out and my instructions were pretty clear. For me, and how I operate, I see some minor things I would like to change. However, it might be one of those things that needs to work itself out as we go along. Just my personal quirks.

For my students, I will draft all my material and have it ready to go, but I will provide further details on the warm-ups and create more self-explanatory videos of our assignments. So, overall, I believe I got great feedback and also developed a great course design.

Transcript

References

Krug, S. (2020). Usability test demo. YouTube.

https://www.youtube.com/watch?v=1UCDUOB_aS8

Usability Testing & Reflection

Usability Testing

​Planning and Conducting Usability Testing

​​

​The email included a brief background section to help the stakeholder understand the purpose. Followed by a detailed section of instructions that stated the following:

​

Please join my personal Google Classroom

Review the material

Complete a short usability survey after looking through the course. The survey has 16 Likert-scale items (Strongly Disagree → Strongly Agree) and 4 open-ended questions. Once I receive feedback, I will compile results for my instructor, but will not include names due to FERPA/HIPAA.

​​

Finally, it had the deadline I needed them to submit their feedback by in order for me to compile my data.

​

What worked:

Very organized and easy to navigate

From a CTE Coordinator, "The variety of career clusters that are offered."

From one English teacher, " I truly loved the real-world application and assessments that coincide with each lesson. It allows the students hands-on knowledge and experience to foster their learning and enhance their expertise."

Having Step-by-step instructions.

​​

What didn't:

Initially, sent the wrong link to join Google Classroom. Users were unable to log in.

I had also drafted all my material and forgot to post it so my "students" could see it. So I had to post all my material.

Some videos seemed long.

Possibly overwhelming my stakeholders with the whole year vs. just the first two modules.

I quickly put the test together so they had time over Thanksgiving break to assess. Take a little more time to double-check my work.

I need to give more guidance on where to start, rather than letting them just join and browse.

​

Suggestions for improvement:

I received two pieces of feedback for improvement.

As mentioned, possibly implement a real-world guest speaker to talk with the students in real-time in order for them to acquire first-hand knowledge of the various careers. Otherwise, excellent job! ​

​

​​

Learner Support:

To support students who are new to my Google Classroom, I’ve added the following:

A short video walkthrough of the LMS layout​

I will create more instructional videos for students on assignments so they can work on them at home, if they are out sick.

Reword any instructions to be detailed and precise so students are able to work independently or able to learn at their own pace

Feedback Summary/ Results

​​​​

Participants rated the first 16 questions on a 1-5 Likert scale (1 = lowest, 5 = highest). I averaged each question's scores, then computed the overall mean of those averages. The final rating for the class material was 4.6875 out of 5 possible points.

Planning and Conducting Usability Testing

The email included a brief background section to help the stakeholder understand the purpose. Followed by a detailed section of instructions that stated the following:

As mentioned, possibly implement a real-world guest speaker to talk with the students in real-time in order for them to acquire first-hand knowledge of the various careers. Otherwise, excellent job!

A short video walkthrough of the LMS layout