Testing and Validation
To justify the feature of the guidebook and ensure it is needed for the game, user-tested was completed on the final result from sprint 2. The testing that occurred was observational testing, which meant that I let the users play the game and recorded what they did and where they got stuck. I tested a total of five people to help establish what is intuitive in the game and what is not.
I started the game for the users, and let them play through the game while I watched. I took notes of what they did, where they got stuck and what they said while playing the game to help establish emotion.
After they finished working through the game and said that they don't think there are any more aspects to explore, I explain how to actually use the game.
From testing five users, only two figured out how to make the main character move, and attack. No users figured out how to or even that you could place buildings. Many of the users just kept on clicking around the screen expecting something to happen. One user went into the shop five times trying to find something useful to help the player move.
From this many of the users did not figure out what the point of the game was, thus, the game is not intuitive. This means that the game does need instructions and a guide to know how to play and win the game.
To best present the information in a meaningful and helpful manner, we conducted user testing on our low fidelity guidebook screen wireframes to understand which format was the most legible, captivating and suitable for the target audience. The following testing plan outlines the tests undertaken and how we measured success to produce the best outcome for users. See this link to view low-fidelity prototypes and more about wireframes. From left to right --> Options 1, 2 and 3 respectively.
The testing undertaken was customer preference testing. Whereby users are given three options to choose from and must rank in order of which they like the most, with 1 being their first preference and 3 being their last preference. After scoring the options, the user must respond with an explanation for their ranking. Preference testing is fundamental to grasping users' dislikes and likes early on.
-
Preference: 2,3 & 1.
Reasoning: "I really like the idea of creating a book as the layout adds character to the aesthetic and ties in the idea of the oceanic adventures out on the sea with those old leather sailing books. So that’s why I chose 2 and 3 as my top preferences and made 1 last as it doesn’t have as much of a special effect." -
Preference: 2,1 & 3. Reasoning: "Just by looking at option 3, I have never seen anything done like that, and I’d say it would be because one it’s hard to produce and two people would prefer to read a flat book than a sentence or a paragraph that ascends upwards. Although it’s creative, I think for game instructions I just want it plainly laid out for me, like given directly. So that’s why I went with 2 as you still get that cool book design but, it’s easier to quickly read the instructions."
-
Preference: 2,1 & 3. Reasoning: "Option two appeals to me the most as it is similar to how I read on my iPad. Which is for those who don’t know a flat top-down view of the text. I guess it works the best as there is no curvature or effects that make it difficult to read, and that is probably why I put 3 last. I don’t mind option 1, but it is a bit bland and I personally appreciate games that make the effort to intertwine their game theme throughout all interfaces."
-
Preference: 1,2 & 3. Reasoning: "I strongly dislike the angled book layout (option 3) as it is hard to read text angled like that. If I were playing a game, I’d find it a pain and almost nauseating to have to tilt my head to read text. For that reason, I prefer the 2D format of both 1 and 2."
-
Preference: 2,1 & 3 Reasoning: "I like the middle option as the layout is the most intuitive. What I mean by this is that I can easily find the exit button, see how to flick through pages and the content on the right-hand side is presented neatly. I ranked the first option second as I also like the simple layout of that option. But I just didn’t enjoy the buttons “next and prev”, I prefer the arrows so I went with the middle one."
After gathering the scores, we found that there was a strong favour towards option 2. Option 2 has a flat view and is made to appear like an old-fashioned novel. The users leaned towards this option as they found it was the perfect mix of integration of the game and legibility. There was a common preference to be able to quickly read the information and return to the game, and not make the process a hassle. Additionally, the option still captured the theme nicely with a lot of the users after the testing saying it would also look great when produced with the pixel design. Therefore, customer preference testing was very beneficial in grasping how users would like the guidebook to be organised and depicted.
A big task for the guidebook was arranging the topics put forward by our studio members in a logical order that a user would expect to view the information. There is no better way than hearing what users themselves have to say by doing some testing. Therefore, we asked users to arrange the topics in how they'd like to view the content which is relative to their first experiences playing the game. Since the topics are subject to the studio's opinions, we gave the user the ability to transform the topics by adding or removing topics too.
Please list in order (1 being the first option, 2 being the second option, and so on…) the order you would like to view the game information from the guidebook. Some motives for ranking could be the importance you associate with the topic or logical timeline or sequence of the game. We also have the option to ‘not include’ a topic; do so by placing this topic name in the red cell. If excluded from the guidebook, please give a valid reason why. And if you identify any elements you wish to add to the topics, please add them in the green cell and provide a reason why and where you’d place them.
See the image below which outlines the feedback from user testing.
The user testing brought to light the irrelevance of certain topics and a common understanding of what users like to know about in a particular order. The majority of users didn't think cut scenes, camera-related adjustments, and NPCs were important enough to be outlined in the guidebook. More specifically, users were perplexed as to why cut scene controls exist and said just to give users the option to skip. And for camera-related adjustments, they should just be found in settings and don't really require a full-on explanation. In terms of the generally agreed-upon ordering of the table of contents for the guidebook, it seems as though users would like to know about the crystal and main character, then who they have to protect themselves from (enemies) when to expect the enemies (night cycle), what to do in the day cycle (buildings, shop etc). A final version of the list will be presented below. However, we have also manipulated the table of contents topics to include suggestions from users, which were generic how-to-play instructions, currency and the final boss level. These were added based on frustrations and pain points experienced by the users when playing the game and not having enough information.
Table of Contents - Revised:
- How to Play
- Crystal
- Main Player (their movements, health, lives, and objectives)
- Day and Night cycles
- Enemies (Generic Enemies and hint to boss level)
- Types of Buildings
- Resource Buildings
- Currency
- Shop resources
- Inventory
To ensure the guidebook contained the correct information and was in line with our fellow studio members, we created a draft guidebook google document which was shared among all teams. Teams were able to manipulate the draft and a change log was created to record the cross-team collaboration.
See the following google doc guidebook or the images below of the document:
To gauge how successful sprint three's end product we conducted user testing (navigational based) to measure how well the user interacts with the feature. The results are helpful in understanding how intuitive an action may be, and ultimately, measures the interface's success. For this sprint, a mixture between the original game and the high-fidelity wireframe was used to increase the realistic feeling of the finished game.
The user must complete the following tasks without any guidance or assistance. The condition that must be satisfied is that the users must not have any prior knowledge regarding the game.
- Try to play the game without a manual or guidance
- Find the help that is available on the Main Game screen
- Search for the game topic that they are confused
- Let users wander the game; tests whether the game is already playable without any need of further guidance.
- To know whether the Guidebook icon that is in the Main Game screen is recognizable as a place where users turn to when they are confused and are looking for clarification.
- To measure the clarity of the structured content of the Guidebook
The navigational testing was done on 5 people raging from age 20 to 25. The results of each task are as follows.
From this task, 1 out of 5 users are struggling to play the game as is without any further clarification needed. The rest of the test participants said they definitely need help in playing and operating the game.
The results from this task are interesting. All of the users know where to get help; they go directly pressing the Guidebook page. They agreed that the button to access the Guidebook is clear and straightforward.
The users pressed into the topics that are available on the "Table of Contents" page. They feel that it is so convenient for them to be redirected after pressing the button. The content of the Guidebook is well-organized and the fact there is a picture that clarifies the description helps tremendously.
From the results of the given tasks, we achieved 100% task completion rate when it comes to things related to the Guidebook. On the other hand, the game still needs a sort of tutorial in the beginning to guide user on what is the game, how to play it, and how to win it since the test subjects are mainly confused regarding those topics.