Subjective Quality Analysis of Apps on the Open Web Apps Project

In late January, the Apps QA team was assigned to determine which apps in a certain list of apps sorted by business needs could be demoed at mobile world congress. The challenge of solving this problem is knowing that you have determine the quality of a list of apps on many different platforms, allow them to be compared against easily, and know what to do to improve the app experience for those apps. Improving the app experience includes asking developers of the Apps project to fix bugs and asking the creators of the app to fix problems related to their app specifically. Additionally, you need to pay attention to the underlying business needs behind each app, as the success of the app on your platform mutually benefits the platform as a whole by increasing the use of the platform and attracting more people toward the platform to create and use apps. The target problem to solve is the following:

How do you assess the quality of an app across different platforms and identify ways to improve app experience quality?

The solution I used to solve this problem was to develop a subjective measurement of the app’s quality overall on the phone, tablet, web, and the native desktop. The subjective measurement consisted of the following four different categories: Excellent, Good, Fair, and Poor. Below I summarize a description of each category I used for subjective app quality analysis.


  • Cross-device support (works on all devices)
  • Look and feel adapts to each device’s requirements (phone, tablet, desktop)


  • App functionally works as expected (no functional errors on each device)
  • Relatively usable across devices, even if look and feel doesn’t match each device’s requirements exactly (e.g. desktop only site, but functional on mobile, device screen-size)
  • User responsiveness is good (pages load in a reasonable amount of time)


  • App partially works or is only partially usable (some clicks on app work, others generate incorrect behavior, rendering makes app hard to use)
  • User responsiveness is not that great


  • App won’t render, won’t work entirely, unusable (doesn’t function correctly)

When the categories were defined for the app on each platform, I then provided a rationale behind why the quality level was specified. For example, one app with an excellent quality level was described to have an “excellent look and feel, easy to use, functional, no issues, app-like.” Another example with an app with a poor quality level had a rationale documenting that the app did not ever load a live stream to allow the app to be used.

With a rationale specified, the next steps was to figure out what could be done to improve the quality of the app. The two quality improvement strategies I employed looked at what the developers of the platform could do and what the developers of the app could do to improve the app experience. For the platform developers, this typically involved specifying certain bugs that needed to be fixed on respective platforms to increase the app experience quality. For app developers, this involved providing accurate, detailed feedback to the developer on what needs to be changed to improve the app experience on our platform.

With these actions items specified, I then moved forward to track these issues overtime to understand the status of each of the issues overtime. For platform issues, I flagged certain bugs as important to our demo to clearly tell the developers that these bugs are especially important to being fixed to allow the quality of the demo to improve with a certain set of apps. For app developer action items, I kept an ongoing backlog of issues specific to certain apps and their status while I was communicating back and forth with the developer to ensure that the quality of the app is improved for the platform. As the fixes to the issues came in for a particular app, I would then perform a re-evaluation of the app subjective score for the platforms affected.

After going through this process, I can now see that I saw success in subjectively analyzing the apps, as I did end up with a list of apps that could be effectively demoed. Going in the future, I plan on continuing to use this technique to evaluate app quality in future milestones after seeing the success of this technique. Seeing the success of this technique, I’m interested to hear what people think about subjective quality analysis approaches you have used to assess quality. What subjective quality approaches have you used? Was the approach successful? Why was it successful? What value did the subjective analysis provide on your project?

Leave a comment


  1. Seems very interesting. Can you give a ballpark estimate of your sample size? (Were you considering 3 apps, a dozen, 100s?)

    Were there cases where the platform vs app split was ambiguous? (As in, the app *could* work around a problem, but wouldn’t have to in an ideal world?)

    Were there common themes on either the platform or app side? (eg “many apps screw up X”) If so, did any of those on the app side seem automatable? (“lint for apps”?)

  2. > Can you give a ballpark estimate of your sample size?

    24 apps were analyzed in total.

    > Were there cases where the platform vs app split was ambiguous?

    To clarify – Is this referring to if there were cases where there was an issue in the platform that the app could work-around, but would not need to if the platform didn’t have that issue (the ideal platform)? If that is what the question asking, I’d say definitely. Examples include:

    – Logout not working on the mobile platform for one app, but logging out of the webpage from the browser for that app would work. A user could work-around by closing and restarting the app, but that’s not a good user experience.
    – Certain links on the desktop platform causing it to fail to load the location of the link. A user could work-around this by choosing links that do work (only happens with specific links), but that’s also not a good user experience.

    > Were there common themes on either the platform or app side?

    Definitely. One common theme on the platform side for mobile was the lack of support for video. Without video, certain apps could not even be used (a few of them were video apps). Similarly, another common theme on the platform was the lack of plugin support (e.g. flash), so flash-based parts of an app could not be used (e.g. playing audio).

    For the apps side, a common theme was I saw was that the app’s starting URL would advertise installing the native app (e.g. Install app for android). Another theme I saw was there were definitely some apps that were not-optimized mobile. Instead, they were purely designed with a “desktop-oriented” view in mind. In certain cases with a phone, this made the app incredibly hard to use.

    > If so, did any of those on the app side seem automatable?

    Not sure. That’s definitely something I’ll want to look into. In the advertisement theme on the app side (i.e. advertising installing the native app), I guess the “linter” could involve crawling from the base app URL around its origin to find native app advertisements in a dom element (i.e. origin =, it would be anything under Capturing if a view is optimized for a particular device might be more challenging automate, but it would be interesting to look into, as there are rules to how we judge a website being optimized for a particular device.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: