Data Science is Hard: Dangerous Data

I sit next to a developer at my coworking location (I’m one of the many Mozilla staff who work remotely) who recently installed the new Firefox Quantum Beta on his home and work machines. I showed him what I was working on at the time (that graph below showing how nicely our Nightly population has increased in the past six months), and we talked about how we count users.

Screenshot-2017-9-28 Desktop Nightly DAU MAU for the Last Six Months by Version

=> “But of course we’ll be counting you twice, since you started a fresh profile on each Beta you installed. Actually four times, since you used Nightly to download and install those builds.” This, among other reasons, is why counting users is hard.

<= “Well, you just have to link it to my Firefox Account and then I’ll only count as one.” He figured it’d be a quick join and then we’d have better numbers for some users.

=> “Are you nuts?! We don’t link your Firefox Account to Telemetry! Imagine what an attacker could do with that!”

In a world with adversarial trackers, advertising trackers, and ever more additional trackers, it was novel to this pseudo-coworker of mine that Mozilla would specifically not integrate its systems.

Wouldn’t it be helpful to ourselves and our partners to know more about our users? About their Firefox Accounts? About their browsing history…

Mozilla doesn’t play that game. And our mission, our policies, and our practices help keep us from accidentally providing “value” of this kind for anyone else.

We know the size of users’ history databases, but not what’s in them.

We know you’re the same user when you close and reopen Firefox, but not who you are.

We know whether users have a Firefox Account, but not which ones they are.

We know how many bookmarks users have, but not what they’re for.

We know how many tabs users have open, but not why. (And for those users reporting over 1000 tabs: WHY?!)

And even this much we only know when you let us:


Why? Why do we hamstring our revenue stream like this? Why do we compromise on the certainty that having complete information would provide? Why do we allow ourselves to wonder and move cautiously into the unknown when we could measure and react with surety?

Why do we make Data Science even harder by doing this?

Because we care about our users. We think about what a Bad Actor could do if they had access to the data we collect. Before we okay a new data collection we think of all the ways it could be abused: Can it identify the user? Does it link to another dataset? Might it reveal something sensitive?

Yes, we have confidence in our security, our defenses in depth, our privacy policies, and our motivations to work for users and their interests.

But we are also confident that others have motivations and processes and policies that don’t align with ours… and might be given either the authority or the opportunity to gain access in the future.

This is why Firefox Send doesn’t know your encryption key for the files you share with your friends. This is why Firefox Accounts only knows six things (two of them optional) about you, and why Firefox Sync cannot read the data it’s storing for you.

And this is why Telemetry doesn’t know your Firefox Account id.



So I’ve Finished Final Fantasy XV

(Spoilers may lie within for a game that’s a year old.)

For a game that obviously had a lot of time (10 years!), effort, and money dumped into it… it feels unfortunately uneven.

The story, though standard Final Fantasy fare, is told poorly enough that characters react emotionally to situations that haven’t been earned, the player is forced to have her character make decisions without knowledge that her character has, and injects a person from the tie-in movie in a prominent part of the endgame without having ever, in the plot or the story, meeting the player character.

I like a good mystery. I like plots that surface only about a tenth of a world’s lore. I’m happy to think about questions posed by the narrative, and intrigued by the choices made by writers and directors about what pieces to include and which to omit.

This isn’t that. I mean it is, in places. I don’t need and didn’t receive a cutscene and backing barks about how Ifrit was the one who wrought the Starscourge. That’s a fine piece of information to put in an in-game codex or tie-in novel or whatever.

At the very least you must give the player time with secondary characters before fridging them if you want an emotional response. I didn’t know who Jared was before you killed him. The only reason I knew he was important was because the main characters became mopey-faced when they heard of his off-screen demise. And for the relationship in the game’s own logo I only have the characters’ words to go by to determine how much Noctis and Luna loved each other despite never being in the same place for ten years. But boy howdy was her death rendered beautifully and with excellent scoring.

If it were just the story that was uneven, I’d still be upset. But this unevenness extends throughout the title.

Barks during the fishing minigame are timed to the wrong events; only half of the casual conversations have lip-syncing; you can only have one “Kill <some monster(s)> and get <some reward(s)>” quest active at once; the control schemes for Chocobo riding, car driving, and walking all have different buttons for jump; the map doesn’t zoom in far enough to discriminate icons in town; the fog of war on dungeon maps only shows on the full map not the minimap…

I work in software. I know how bugs creep into release. But the only reason I can think of to explain three different jump buttons and unmarkable maps with different sort orders on quest lists is that Squenix ignores their interaction designers.

Story and mechanics aren’t the whole of it either: Final Fantasy XV’s representation of healthy male relationships is above anything I can remember from any Final Fantasy title. They even cry together, our roadtrip boys… if you wait midway through the credits for it. Yet relationships with women are tropish, boring, and underwritten. Despite the backlash Square Enix received after Episode Duscae (the first of the playable demos) they declined to design Cindy some mechanics coveralls or exclude superfluous car washing scenes. Iris is a schoolgirl stereotype the game cannot decide whether I’m attracted to, embarrassed by, or protective of. Luna is a damsel no matter how much we’re told her actions drive the plot. Aranea is a spinny death machine that battles in heels and bared midriff (though she almost has a character arc)…

You might think in reading this that I didn’t have fun playing FFXV and didn’t enjoy the game. I did, really… It’s beautiful, the four main characters have acceptable chemistry, story actions have story consequences, the battle system is fast and reasonably fun, the minigames are diverting, they finally learned how to communicate enemy scale, and did I mention it’s beautiful?

But when Dragon Age: Inquisition can, two years and one console generation earlier, “Open World” better that a mainline Final Fantasy… I just wonder what went wrong.

Two Days, or How Long Until The Data Is In

Two days.

It doesn’t seem like long, but that is how long you need to wait before looking at a day’s Firefox data and being sure than 95% of it has been received.

There are some caveats, of course. This only applies to current versions of Firefox (55 and later). This will very occasionally be wrong (like, say, immediately after Labour Day when people finally get around to waking up their computers that have been sleeping for quite some time). And if you have a special case (like trying to count nearly everything instead of just 95% of it) you might want to wait a bit longer.

But for most cases: Two Days.

As part of my 2017 Q3 Deliverables I looked into how long it takes clients to send their anonymous usage statistics to us using Telemetry. This was a culmination of earlier ponderings on client delay, previous work in establishing Telemetry client health, and an eighteen-month (or more!) push to actually look at our data from a data perspective (meta-data).

This led to a meeting in San Francisco where :mreid, :kparlante, :frank, :gfritzsche, and I settled upon a list of metrics that we ought to measure to determine how healthy our Telemetry system is.

Number one on that list: latency.

It turns out there’s a delay between a user doing something (opening a tab, for instance) and them sending that information to us. This is client delay and is broken into two smaller pieces: recording delay (how long from when the user does something until when we’ve put it in a ping for transport), and submission delay (how long it takes that ready-for-transport ping to get to Mozilla).

If you want to know how many tabs were opened on Tuesday, September the 5th, 2017, you couldn’t tell on the day itself. All the tabs people open late at night won’t even be in pings, and anyone who puts their computer to sleep won’t send their pings until they wake their computer in the morning of the 6th.

This is where “Two Days” comes in: On Thursday the 7th you can be reasonably sure that we have received 95% of all pings containing data from the 5th. In fact, by the 7th, you should even have that data in some scheduled datasets like main_summary.

How do we know this? We measured it:

Screenshot-2017-9-12 Client "main" Ping Delay for Latest Version(1).png(Remember what I said about Labour Day? That’s the exceptional case on beta 56)

Most data, most days, comes in within a single day. Add a day to get it into your favourite dataset, and there you have it: Two Days.

Why is this such a big deal? Currently the only information circulating in Mozilla about how long you need to wait for data is received wisdom from a pre-Firefox-55 (pre-pingsender) world. Some teams wait up to ten full days (!!) before trusting that the data they see is complete enough to make decisions about.

This slows Mozilla down. If we are making decisions on data, our data needs to be fast and reliably so.

It just so happens that, since Firefox 55, it has been.

Now comes the hard part: communicating that it has changed and changing those long-held rules of thumb and idées fixes to adhere to our new, speedy reality.

Which brings us to this blog post. Consider this your notice that we have looked into the latency of Telemetry Data and is looks pretty darn quick these days. If you want to know about what happened on a particular day, you don’t need to wait for ten days any more.

Just Two Days. Then you can have your answers.


(Much thanks to :gsvelto and :Dexter’s work on pingsender and using it for shutdown pings, :Dexter’s analyses on ping delay that first showed these amazing improvements, and everyone in the data teams for keeping the data flowing while I poked at SQL and rearranged words in documents.)