TIL: Feature Detection in Windows using GetProcAddress

In JavaScript, if you want to use a function that was introduced only in certain versions of browsers, you use Feature Detection. For example, you can ask “Hey, browser, do you have a function called `includes` on Array?” If the browser has it, you use it; and if it doesn’t, you either get along without it or load your own implementation.

It turns out that this same concept can be (and, in Firefox, is) done with Windows APIs.

Firefox for Windows is built against the Windows 10 SDK. This means the compiler knows the API calls and type definitions for all sorts of wondrous modern features like toast notifications and enumerating graphics adapters in a specific order.

However, as of writing, Firefox for Windows supports Windows 7 and up. What would happen if Firefox tried to use those fancy new Windows 10 features when running on Windows 7?

Well, at compile time (when Mozilla builds Firefox), it knows everything it needs to about the sizes and names of things used in the new features thanks to the SDK. At runtime (when a user runs Firefox), it needs to ask Windows at what address exactly all of those fancy new features live so that it can use them.

If Firefox can’t find a feature it expects to be there, it won’t start. We want Firefox to start, though, and we want to use the new features when available. So how do we both use the new feature (if it’s there) and not (if it’s not)?

Windows provides an API called GetProcAddress that allows the running program to perform some Feature Detection. It is asking Windows “Hey, so I’m looking for the address of this fancy new feature named FancyNewAPI. Do you know where that is?”. Windows will either reply “No, sorry” at which point you work around it, or “Yes, it’s over at address X” at which point to convert address X into a function pointer that takes the same number and types of arguments that the documentation said it takes and then instruct your program to jump into it and start executing.

We use this in Firefox to detect gamepad input modules, cancelable synchronous IO, display density measurements, and a whole bunch of graphics and media acceleration stuff.

And today (well, yesterday at this point) I learned about it. And now so have you.


–edited to remove incorrect note that GetProcAddress started in WinXP– :aklotz noted that GetProcAddress has been around since ancient times, MSDN just periodically updates its “Minimum Supported Release” fields to drop older versions.


Firefox Telemetry Use Counters: Over-estimating usage, now fixed

Firefox Telemetry records the usage of certain web features via a mechanism called Use Counters. Essentially, for every document that Firefox loads, we record a “false” if the document didn’t use a counted feature, and a “true” if the document did use that counted feature.

(( We technically count it when the documents are destroyed, not loaded, since a document could use a feature at any time during its lifetime. We also count top-level documents (pages) separately from the count of all documents (including iframes), so we can see if it is the pages that users load that are using a feature or if it’s the subdocuments that the page author loads on the user’s behalf that are contributing the counts. ))

To save space, we decided to count the number of documents once, and the number of “true” values in each use counter. This saved users from having to tell us they didn’t use any of Feature 1, Feature 2, Feature 5, Feature 7, … the “no-use” use counters. They could just tell us which features they did see used, and we could work out the rest.

Only, we got it wrong.

The server-side adjustment of the counts took every use counter we were told about, and filled in the “false” values. A simple fix.

But it didn’t add in the “no-use” use counters. Users who didn’t see a feature used at all weren’t having their “false” values counted.

This led us to under-count the number of “false” values (since we only counted “falses” from users who had at least one “true”), which led us to overestimate the usage of features.

Of all the errors to have, this one was probably the more benign. In failing in the “overestimate” direction we didn’t incorrectly remove features that were being used more than measured… but we may have kept some features that we could have removed, costing mozilla time and energy for their maintenance.

Once we detected the fault, we started addressing it. First, we started educating people whenever the topic came up in email and bugzilla. Second, :gfritzsche added a fancy Use Counter Dashboard that did a client-side adjustment using the correct “true” and “false” values for a given population.

Third, and finally, we fixed the server-side aggregator service to serve the correct values for all data, current and historical.

And that brings us to today: Use Counters are fixed! Please use them, they’re kind of cool.


After (4B more samples)

Another Stay of Execution for Firefox Users on Windows XP


Firefox users who are on Windows XP now have until August 28, 2018 to upgrade their machines. In the grand Internet tradition I will explore this by pretending someone is asking me questions.


The last Firefox release supporting Windows XP is Firefox ESR 52. Previously Firefox ESR 52 was set to end-of-life on or around May of 2018 after the next ESR, Firefox ESR 59, had been released and stabilized. Now, with an email to the Enterprise, Dev Platform, Firefox Dev, and Mobile Firefox Dev mailing lists, :Sylvestre has announced that the next ESR will be Firefox ESR 60, which extends the Firefox ESR 52 end-of-life to August 28, 2018.

No, not “Why did it change,” Why should anyone still care about Windows XP? Hasn’t it been out-of-service for a decade or something?

Not quite a decade, but the last release of Windows XP was over nine years ago, and even Microsoft’s extended support tapped out nearly four years ago.

But as to why we should care… well, Windows XP is still a large-ish portion of the Firefox user base. I don’t have public numbers in front of me, but you can see the effect the Windows XP Firefox population numbers had on the Firefox Hardware Report when we diverted them to ESR this past March. At that time they were nearly 8.5% of all Firefox users. That was more than all versions of Mac Firefox users.

Also, it’s possible that these users may be some of the most vulnerable of the Internet’s users. They deserve our thought.

Oh, okay, fine. If they matter so much, why aren’t we supporting them forever?

As you can see from the same Firefox Hardware Report, the number of Windows XP Firefox users was in steady decline. At some point our desire and capability to support this population of users can no longer match up with our desire to ship the best experience to the most users.

Given the slope of the decline in the weeks leading up to when we migrated Windows XP Firefox users to Firefox ESR, we ought to be getting pretty close to zero. We hate to remove support from any users, but there was a real cost to supporting Windows XP.

For instance, the time between the ESR branch being cut and the first Windows XP-breaking change was a mere six days. And it wasn’t on purpose, we were just fixing something somewhere in Gecko in a way that Windows XP didn’t like.

So who are we going to drop support for next?

I don’t know of any plans to drop support for any Operating Systems in the near future. I expect we’ll drop support for older compilers in our usual manner, but not OSs.

That pretty much sums it up.

If you have any questions about Firefox ESR 60, please check out the Firefox ESR FAQ.


An Unofficial Guide to Unofficial Swag: Stickers

Mozillians like stickers.


However! Mozilla doesn’t print as many stickers as you might think it does. Firefox iconography, moz://a wordmarks, All Hands-specific rounds, and Mozilla office designs are the limit of official stickers I’ve seen come from official sources.

The vast majority of sticker designs are unofficial, made by humans like you! This guide contains tips that should help you create and share your own unofficial stickers.

Plan 9
(original poster by Tom Jung, modifications by :Yoric and myself. Use under CC-BY-SA 3.0)


I’m not a designer. Luckily for my most recent printing project I was simply updating the existing design you see above. If you are adapting someone else’s design, ensure you either have permission or are staying within the terms of the design’s license. Basic Firefox product identity assets are released under generous terms for remixing, for instance.


The bigger they are, the harder they are to fit in a pocket or on the back of a laptop screen. Or in carry-on. The most successful stickers I’ve encountered have been at most 7cm on the longest side (or in diameter, for rounds), and many have been much smaller. With regards to size, less may in fact be more, but you have to balance this with any included text which must be legible. The design I used wouldn’t work much smaller than 7cm in height, and the text is already a little hard to read.


How will you distribute these? If your design is team-specific, a work week is a good chance to hand them out individually. If the design is for a location, then pick a good gathering point in that location (lunchrooms are a traditional and effective choice), fan out some dozen or two stickers, and distribution should take care of itself. All Hands are another excellent opportunity for individual and bulk distribution. If the timing doesn’t work out well to align with a work week or an All Hands, you may have to resort to mailing them over the globe yourself. In this case, foster contacts in Mozilla spaces around the world to help your stickers make it the last mile into the hands and onto the laptops of your appreciative audience.


50 is a bare minimum both in what you’ll be permitted to purchase by the printer and in how many you’ll want to have on hand to give away. If your design is timeless (i.e. doesn’t have a year on it, doesn’t refer to a current event), consider making enough leftovers for the future. If your design is generic enough that there will be interest outside of your team, consider increasing supply for this demand. Generally the second 50 stickers cost a pittance compared to the first 50, so don’t be afraid to go for a few extra.


You’ll be paying for this yourself. If your design is team-specific and you have an amenable expense approver you might be able to gain reimbursement under team-building expenses… But don’t depend on this. Don’t spend any money you can’t afford. You’re looking at between 50 to 100 USD for just about any number of any kind of sticker, at current prices.


I’m in Canada. The sticker printer I chose most recently (stickermule) was in the US. Unsurprisingly, it was cheaper and faster to deliver the stickers to the US. Luckily, :kparlante was willing to mule the result to me at the San Francisco All Hands, so I was able to save both time and money. Consider these logistical challenges when planning your swag.


Two weeks before an All Hands is probably too late to start the process of generating stickers. I’ve made it happen, but I was lucky. Be more prepared than I was and start at least a month ahead. (As of publication time you ought to have time to take care of it all before Austin).


After putting a little thought into the above areas it’s simply a matter of choosing a printing company (local to your region, or near your distribution venue) and sending in the design. They will likely generate a proof which will show you what their interpretation of your design on their printing hardware will look like. You then approve the proof to start printing, or make changes to the design and have the printer regenerate a proof until you are satisfied. Then you arrange for delivery and payment. Expect this part of the process to take at least a week.

And that’s all I have for now. I’ll compile any feedback I receive into an edit or a Part 2, as I’ve no doubt forgotten something essential that some member of the Mozilla Sticker Royalty will only too happily point out to me soonish.

Oh, and consider following or occasionally perusing the mozsticker Instagram account to see a sample of the variety in the Mozilla sticker ecosystem.

Happy Stickering!


Data Science is Hard: Dangerous Data

I sit next to a developer at my coworking location (I’m one of the many Mozilla staff who work remotely) who recently installed the new Firefox Quantum Beta on his home and work machines. I showed him what I was working on at the time (that graph below showing how nicely our Nightly population has increased in the past six months), and we talked about how we count users.

Screenshot-2017-9-28 Desktop Nightly DAU MAU for the Last Six Months by Version

=> “But of course we’ll be counting you twice, since you started a fresh profile on each Beta you installed. Actually four times, since you used Nightly to download and install those builds.” This, among other reasons, is why counting users is hard.

<= “Well, you just have to link it to my Firefox Account and then I’ll only count as one.” He figured it’d be a quick join and then we’d have better numbers for some users.

=> “Are you nuts?! We don’t link your Firefox Account to Telemetry! Imagine what an attacker could do with that!”

In a world with adversarial trackers, advertising trackers, and ever more additional trackers, it was novel to this pseudo-coworker of mine that Mozilla would specifically not integrate its systems.

Wouldn’t it be helpful to ourselves and our partners to know more about our users? About their Firefox Accounts? About their browsing history…

Mozilla doesn’t play that game. And our mission, our policies, and our practices help keep us from accidentally providing “value” of this kind for anyone else.

We know the size of users’ history databases, but not what’s in them.

We know you’re the same user when you close and reopen Firefox, but not who you are.

We know whether users have a Firefox Account, but not which ones they are.

We know how many bookmarks users have, but not what they’re for.

We know how many tabs users have open, but not why. (And for those users reporting over 1000 tabs: WHY?!)

And even this much we only know when you let us:


Why? Why do we hamstring our revenue stream like this? Why do we compromise on the certainty that having complete information would provide? Why do we allow ourselves to wonder and move cautiously into the unknown when we could measure and react with surety?

Why do we make Data Science even harder by doing this?

Because we care about our users. We think about what a Bad Actor could do if they had access to the data we collect. Before we okay a new data collection we think of all the ways it could be abused: Can it identify the user? Does it link to another dataset? Might it reveal something sensitive?

Yes, we have confidence in our security, our defenses in depth, our privacy policies, and our motivations to work for users and their interests.

But we are also confident that others have motivations and processes and policies that don’t align with ours… and might be given either the authority or the opportunity to gain access in the future.

This is why Firefox Send doesn’t know your encryption key for the files you share with your friends. This is why Firefox Accounts only knows six things (two of them optional) about you, and why Firefox Sync cannot read the data it’s storing for you.

And this is why Telemetry doesn’t know your Firefox Account id.


The Photonization of about:telemetry

This summer I mentored :flyingrub for a Google Summer of Code project to redesign about:telemetry. You can read his Project Submission Document here.


Google Summer of Code is a program funded by Google to pay students worldwide to contribute in meaningful ways to open source projects.

about:telemetry is a piece of Firefox’s UI that allows users to inspect the anonymous usage data we collect to improve Firefox. For instance, we look at the maximum number of tabs our users have open during a session (someone or several someones have more than one thousand tabs open!). If you open up a tab in Firefox and type in about:telemetry (then press Enter), you’ll see the interface we provide for users to examine their own data.

Mozilla is committed to putting users in control of their data. about:telemetry is a part of that.


When :flyingrub started work on about:telemetry, it looked like this (Firefox 55):


It was… functional. Mostly it was intended to be used by developers to ensure that data collection changes to Firefox actually changed the data that was collected. It didn’t look like part of Firefox. It didn’t look like any other about: page (browse to about:about to see a list of about: pages). It didn’t look like much of anything.


After a few months of polishing and tweaking and input from UX, it looks like this (Firefox Nightly 57):


Well that’s different, isn’t it?

It has been redesigned to follow the Photon Design System so that it matches how Firefox 57 looks. It has been reorganized into more functional groups, has a new top-level search, and dozens of small tweaks to usability and visibility so you can see more of your data at once and get to it faster.



Just because Google Summer of Code is done doesn’t mean about:telemetry is done. Work on about:telemetry continues… and if you know some HTML, CSS, and JavaScript you can help out! Just pick a bug from the “Depends on” list here, and post a comment asking if you can help out. We’ll be right with you to help get you started. (Though you may wish to read this first, since it is more comprehensive than this blog post.)

Even if you can’t or don’t want to help out, you can take sneak a peek at the new design by downloading and using Firefox Nightly. It is blazing fast with a slick new design and comes with excellent new features to help be your agent on the Web.

We expect :flyingrub will continue to contribute to Firefox (as his studies allow, of course. He is a student, and his studies should be first priority now that GSoC is done), and we thank him very much for all of his good work this Summer.


Another Advantage of Decreasing Data Latency: Flatter Graphs

I’ve muttered before about how difficult it can be to measure application crashes. The most important lesson is that you can’t just count the number of crashes, you must normalize it by some “usage” value in order to determine whether a crashy day is because the application got crashier or because the application was just being used more.

Thus you have a numerator (number of crashes) and a denominator (some proxy of application usage) to determine the crash rate: crashes-per-use.

The current dominant denominator for Firefox is “thousand hours that Firefox is open,” or “kilo-usage-hours (kuh).”

The biggest problem we’ve been facing lately is how our numerator (number of crashes) comes in at a different rate and time than our denominator (kilo-usage-hours) due to the former being transmitted nearly-immediately via “crash” ping and the former being transmitted occasionally via “main” ping.

With pingsender now sending most “main” pings as soon as they’re created, our client submission delay for “main” pings is now roughly in line with the client submission delay of “crash” pings.

What does this mean? Well, look at this graph from https://telemetry.mozilla.org/crashes:

Screenshot-2017-7-25 Crash Rates (Telemetry)

This is the Firefox Beta Main Crash Rate (number of main process crashes on Firefox Beta divided by the number of thousands of hours users had Firefox Beta running) over the past three months or so. The spike in the middle is when we switched from Firefox Beta 54 to Firefox Beta 55. (Most of that spike is a measuring artefact due to a delay between a beta being available and people installing it. Feel free to ignore it for our purposes.)

On the left in the Beta 54 data there is a seven-day cycle where Sundays are the lowest point and Saturday is the highest point.

On the right in the Beta 55 data, there is no seven-day cycle. The rate is flat. (It is a little high, but flat. Feel free to ignore its height for our purposes.)

This is because sending “main” pings with pingsender is behaviour that ships in Firefox 55. Starting with 55, instead of having most of our denominator data (usage hours) coming in one day late due to “main” ping delay, we have that data in-sync with the numerator data (main crashes), resulting in a flat rate.

You can see it in the difference between Firefox ESR 52 (yellow) and Beta 55 (green) in the kusage_hours graph also on https://telemetry.mozilla.org/crashes:

Screenshot-2017-7-27 Crash Rates (Telemetry)

On the left, before Firefox Beta 55’s release, they were both in sync with each other, but one day behind the crash counts. On the right, after Beta 55’s release, notice that Beta 55’s cycle is now one day ahead of ESR 52’s.

This results in still more graphs that are quite satisfying. To me at least.

It also, somewhat more importantly, now makes the crash rate graph less time-variable. This reduces cognitive load on people looking at the graphs for explanations of what Firefox users experience in the wild. Decision-makers looking at these graphs no longer need to mentally subtract from the graph for Saturday numbers, adding that back in somehow for Sundays (and conducting more subtle adjustments through the week).

Now the rate is just the rate. And any change is much more likely to mean a change in crashiness, not some odd day-of-week measurement you can ignore.

I’m not making these graphs to have them ignored.

(many thanks to :philipp for noticing this effect and forcing me to explain it)