This month, the Apps QA team has been working with the Mobile Firefox QA team to assess the compatibility of mobile apps on webkit (e.g. Android Stock Browser, Chrome for Android) vs. gecko (e.g. Firefox for Android). The underlying reason we have analyzed this is because in the past, the Mobile QA team has noted many web compatibility problems specific to gecko, such as a desktop site rendering instead of a mobile site. Knowing that this has been a past problem, the Apps QA team then knows it affects us too, as the platform mobile apps will run on gecko.
To analyze these apps, our Apps QA team has been looking at 130+ top sites designed to be web applications. In analyzing these 130+ top sites, our team used a funnel-based test approach to quickly assess a large amount of sites across webkit and gecko. This funnel test approach breaks down into three distinct steps. The first step is to look at screenshots of the mobile site on Firefox for Android vs. Chrome for Android for each app using an automated screenshot generation script Aaron Train created here. When we look at the screenshots, we are looking for issues such as the URL for the top app not loading anything, user agent sniffing, or a desktop site rendering. For user agent sniffing specifically, we detect this issue if we notice that webkit renders a completely different site than gecko, such as webkit rendering a mobile site on linkedin.com and gecko rendering a desktop site on linkedin.com. To confirm that user agent sniffing is occurring, we can use a Firefox for Android extension called phony to change the user agent to webkit on Firefox for Android to see if a different site renders. If a different site renders, then we know that user agent sniffing is evident with the web application. After checking for user agent sniffing and other issues through the screenshots, our team makes a judgment call if more analysis needs to take place such as noticing that there is more functionality within the app and the app is not broken.
If we determine more analysis is needed, our apps QA team then does a subjective quality analysis of the applications on the web on Chrome for Android and Firefox for android. To do this subjective quality analysis, our team would launch the app and use some of the underlying functionality of the app, such as logging in, recording music if it was a music app, viewing status updates if it was social networking app, etc. Upon using the app’s functionality on gecko and webkit, our team would classify the app into three different buckets below similar to what was used for the subjective apps quality analysis.
- Supported on phone and tablet
- Look and feel adapts to each device’s requirements, great looking for the platform
- App functionally works or mostly works as expected
- User responsiveness is okay to good
- Non-optimized mobile or sniff the user agent badly, but still relatively usable
- App won’t render, won’t work entirely, unusable
Upon classifying the apps for webkit vs. gecko, our team would then determine if there are still open questions left that are needed to judge the app’s quality. These questions would be addressed on a case by case basis. Then, knowing the app quality on gecko vs. webkit, we would then log bugs that are classified as either functional problems within gecko, evangelism problems for gecko, or general evangelism problem for the apps experience. A bug classified as a functional problem from gecko typically came from the team seeing a layout problem on gecko that was correctly implemented by the developer, but incorrectly implemented within gecko. A bug classified as an evangelism problem with gecko means that the team needs to evangelize to the developer of the app what they need to change to make their app mobile-friendly for Firefox for Android for users, such as supporting the Firefox for Android user agent, utilizing gecko-specific CSS prefixes, and more. Last, a bug classified as a general evangelism problem occurs when the app itself does not work in a mobile web environment, such as a desktop site rendering on mobile browsers, broken functionality in a mobile browser, and more. After bugs are filed, our team continues to track these issues overtime and re-assess the quality of the website on gecko vs. webkit.
I’m interested to hear from web developers or layout engine developers opinions on what we can do to assess mobile web compatibility, to evangelize to web developers, and to fix within existing layout engines such as gecko and webkit. What other techniques can provide a deeper assessment of mobile websites across different layout engines? What web development techniques should be evangelized to developers to allow their websites to be runnable on any layout engine? What should be fixed within gecko to allow mobile websites optimized for webkit to additionally be mobile-friendly for gecko?