In late January, the Apps QA team was assigned to determine which apps in a certain list of apps sorted by business needs could be demoed at mobile world congress. The challenge of solving this problem is knowing that you have determine the quality of a list of apps on many different platforms, allow them to be compared against easily, and know what to do to improve the app experience for those apps. Improving the app experience includes asking developers of the Apps project to fix bugs and asking the creators of the app to fix problems related to their app specifically. Additionally, you need to pay attention to the underlying business needs behind each app, as the success of the app on your platform mutually benefits the platform as a whole by increasing the use of the platform and attracting more people toward the platform to create and use apps. The target problem to solve is the following:
How do you assess the quality of an app across different platforms and identify ways to improve app experience quality?
The solution I used to solve this problem was to develop a subjective measurement of the app’s quality overall on the phone, tablet, web, and the native desktop. The subjective measurement consisted of the following four different categories: Excellent, Good, Fair, and Poor. Below I summarize a description of each category I used for subjective app quality analysis.
- Cross-device support (works on all devices)
- Look and feel adapts to each device’s requirements (phone, tablet, desktop)
- App functionally works as expected (no functional errors on each device)
- Relatively usable across devices, even if look and feel doesn’t match each device’s requirements exactly (e.g. desktop only site, but functional on mobile, device screen-size)
- User responsiveness is good (pages load in a reasonable amount of time)
- App partially works or is only partially usable (some clicks on app work, others generate incorrect behavior, rendering makes app hard to use)
- User responsiveness is not that great
- App won’t render, won’t work entirely, unusable (doesn’t function correctly)
When the categories were defined for the app on each platform, I then provided a rationale behind why the quality level was specified. For example, one app with an excellent quality level was described to have an “excellent look and feel, easy to use, functional, no issues, app-like.” Another example with an app with a poor quality level had a rationale documenting that the app did not ever load a live stream to allow the app to be used.
With a rationale specified, the next steps was to figure out what could be done to improve the quality of the app. The two quality improvement strategies I employed looked at what the developers of the platform could do and what the developers of the app could do to improve the app experience. For the platform developers, this typically involved specifying certain bugs that needed to be fixed on respective platforms to increase the app experience quality. For app developers, this involved providing accurate, detailed feedback to the developer on what needs to be changed to improve the app experience on our platform.
With these actions items specified, I then moved forward to track these issues overtime to understand the status of each of the issues overtime. For platform issues, I flagged certain bugs as important to our demo to clearly tell the developers that these bugs are especially important to being fixed to allow the quality of the demo to improve with a certain set of apps. For app developer action items, I kept an ongoing backlog of issues specific to certain apps and their status while I was communicating back and forth with the developer to ensure that the quality of the app is improved for the platform. As the fixes to the issues came in for a particular app, I would then perform a re-evaluation of the app subjective score for the platforms affected.
After going through this process, I can now see that I saw success in subjectively analyzing the apps, as I did end up with a list of apps that could be effectively demoed. Going in the future, I plan on continuing to use this technique to evaluate app quality in future milestones after seeing the success of this technique. Seeing the success of this technique, I’m interested to hear what people think about subjective quality analysis approaches you have used to assess quality. What subjective quality approaches have you used? Was the approach successful? Why was it successful? What value did the subjective analysis provide on your project?