I now know, which is to say that I can state with some degree of certainty, that Windows XP Firefox users are only a little less engaged with their Firefox installations as the Firefox user base as a whole.
To get here I needed to sanitize dates with more than four digits in their years (paging the Long Now Foundation: someone’s reporting Telemetry from the year 29634), with timezones not of this Earth (Wikipedia seems to think UTC+14:00 is the highest time zone, but I have seen data from +14:30), and with clones reporting the same session over and over again from different locations (this one might be because a user’s client_id is stored in their profile. If that profile is reused, then we will get data reported from all the locations that use that profile).
I also needed a rigorous definition of what it means for a user population to be “engaged”.
We chose to define an Engagement Ratio for a given day which is basically the number of people who ran Firefox that day divided by the number of people who ran Firefox the previous month. In other words: what proportion of users who could possibly be active actually were active on that day?
Of course, if you read my previous post, you know that the day of the week you choose is going to change your result dramatically. So instead of counting each user who used Firefox that exact day, we average it out over the previous week: if a user was active each day, they’re a full Daily Active User (DAU). If they’re active only one day, they’re 1/7 of a Daily Active User.
To see this in action, I chose to measure Engagement on March the 10th of this year, which was a Thursday. The number of users who reported data to the 1% Longitudinal Dataset who were active that day was 1,119,335. If we use instead the average of daily active users for the week ending on March the 10, we get 1,051,011.86 which is lower by over 6%. This is consistent with data we’ve collected in other studies that showed only 20% of Firefox users use it 7 days a week. Another 25% use it only on weekdays. So it makes sense that a weekday’s DAU count would be higher than a weekend day’s.
If you’ve ever complained about having to remember how many days there are in a month, you know that the choice of “month” is going to change things as well. So in this case, we just chose 28 days: four weeks. That there is no month that is always 28 days long (lookin’ at you, Leap Years) is irrelevant because we’re selecting values to make things easier for ourselves. So if a user was active on any of the previous 28 days, they are a Monthly Active User (MAU).
So you count your DAU and you divide it by your MAU count and you get your Engagement Ratio (ER) whose units are… unitless. Divide users by users and you get… something that’s almost a percent, in that it’s a value from 0 to 1 that represents a proportion of a population.
This number we can track over time. We expect it to trend downwards when we ship something that negatively impacts users. We expect it to trend upwards when we ship something that users like.
So long as we can get this data quickly enough and reliably enough, we can start determining from the numbers (the silent majority), not noisy users (the vocal minority), what issues actually matter to the user base at large.