Misconceptions concerning User-Agents are limiting the possibilities
Friday 4th March 2016 saw 180 web professionals all with an interest in progressive enhancement assemble in London for a one day conference organised by Simon McManus.
Nat Buckley set the scene with an emphasis on low bandwidth, pure HTML and CSS web experiences with reduced reliance on JavaScript. Nat was the first of several speakers to quote Douglas Crockford a leading developer of the JavaScript programming language before explaining 0.9% of web browsers with JavaScript enabled fail to receive all the required JavaScript.
The Web is the most hostile software engineering environment imaginable. Douglas Crockford
Anna Debenham introduced the audience to the world of games console web browsing explaining they're all too often overlooked and highlighted the issues of comparatively weak hardware and browsers. Stefan Tilkov reinforced the theme of simplicity by refreshing the audience with the cycle of computing trends using CORBA and SOAP for reference. Forbes Lindsay advocated the server side approach used by Facebook picking React.js to demonstrate the art of the possible on both server and client web apps.
Adam Silver used real eCommerce businesses to highlight how multi page checkouts resulted in more completed transactions than the single page JavaScript driven equivalents. Jen Simmons explained how CSS standards are now removing the need for more complex JavaScript alternatives providing the best quote of the day with "Don't use bootstrap anymore". Other speakers covered themes of good copy, accessibility, design vs development and emerging technologies.
For my part I was impressed that so many qualified and respected speakers were endorsing the engineering principle of "keeping it simple" either to reduce complexity, improve user experience and even to increase business profit. As someone who often briefs designers and developers to avoid using JavaScript and particularly heavy frameworks I felt in good company.
Maligned User-Agents
However, I continue to be amazed at how the humble User-Agent is misunderstood. The User-Agent is part of the HTTP protocol and is designed to enable a client to advise the web server what it is. Such knowledge then enables a web server to adapt the content it serves to support the capabilities of the client including the physical screen size, available input methods, GPS support or software and browser capability among hundreds of others.
Jeremy Keith who facilitated the Q&A summarised the negative sentiment by "crossing" himself at the mention of "User-Agent". The consensus being that User-Agents "lie", were misused during the "browser wars", and as such can't be relied upon.
It is therefore surprising to me at least that some of the speakers used data from Google Analytics and the browser information web site caniuse to support their arguments. These services all use User-Agents to determine browser versions, device type and the model of hardware.
Other than to highlight what a great job Simon McManus did arranging his first conference my purpose in writing this blog post is to reset some myths passed down from web professional to web professional over the years concerning User-Agents typified by debate and conversation at Enhance Conference.
Bad Experiences
The simple regular expressions and community maintained libraries which have been used to make sense of User-Agents aren't capable of providing reliable enough information for many business purposes. I believe the deficiencies of such solutions have biased most web professionals opinion on the subject.
Consider the following common User-Agent from Windows Phone 8.1 as an example.
Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 635) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537
Simply using a regular expression to search for the presence of iPhone or Android would lead to the conclusion that the device is an iPhone or Android device rather than a Lumia 635 running Windows Phone. The order in which the regular expressions are applied would impact the conclusion. Incidentally Microsoft have adopted this approach to User-Agent construction to counter web sites which use poor quality User-Agent detection solutions.
Feature detection, the process of querying a web browser using JavaScript to determine if it is capable of performing a certain function, provides more reliable results when compared to poor quality User-Agent sniffing.
Quality Device Detection
51Degrees, the business I started and run, maintains a device detection solution that provides a +99.9% reliable detection rate across a unique set of real world User-Agents and includes indicators concerning false positives. User-Agents are used alongside other HTTP headers and JavaScript when available to provide the most complete understanding of a device and its accompanying browser and operating system. When considered over a large enough volume of web requests it's more reliable than the JavaScript figure provided by Nat Buckley, or the anecdotal problems associated with JavaScript implementation and support presented during Enhance Conference.
Distraction
The previous example shows how the Lumia 635 User-Agent is intending to deceive inferior detection solutions. 51Degrees eliminates these distractors. Use the following link to see the results from 51Degrees purely based on this single User-Agent input.
White Lies
Of course the User-Agent may not always be correct. Rather than this resulting in a bad user experience it results in the user receiving the experience they requested. That's why mobile web browsers have the option to request a "desktop site" which among other things alters the User-Agent provided to the web server.
Damned Lies
Sometimes irresponsible device and software vendors do not set their User-Agent header correctly. Sony's Playstation Vita is a possible example highlighted by Anna Debenham in her excellent console analysis of February 2014. The browser is reported in the User-Agent as Amazon's Silk and considered incorrect.
Mozilla/5.0 (PlayStation Vita 1.69) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2
Note: Amazon elude to the fact they provide a browser for the Vita in their advice for detecting Kindle devices based on the User-Agent. Vita 3.57, the most recent version of the Vita software, continues to report the browser as Silk nearly 2 years after the original analysis. I've not yet been able to recheck the Vita but will aim to do so in the next few weeks.
Of importance is that the hardware is correct and should the browser be incorrect this fact can be updated in the database of devices, operating systems and browsers.
The use case for device detection is generally concerned with identifying the hardware's capabilities rather than the browsers. Feature detection should be used in addition to pure device detection for the most robust deployments.
Conclusion
I'm biased as I setup a company to catalgoue all web enabled devices, operating systems and browsers indexing the data by HTTP headers, native SDK values and TAC codes. 51Degrees now employs 22 people the majority of whom work with vendors to classify over 170 new devices each week with over 99.9% accuracy. Over 415,612 combinations are tracked. We don't screen scrape GSM Arena or use regular expressions, XML and JSON. The underlying source code is licenced under Mozilla Public Licence 2 and available on GitHub. We have free versions of the database updated monthly, and weekly and daily subscriptions which cost less than a typical developer spends on coffee.
When used in commercial businesses the results are clear. That's why Adform, Basis Technologies, Dynata (Research Now), EPAM, Open Text Corporation, Sitecore, and more use the data subscription to analysis and optimise their web presence.
Read Case Studies