So I’ve Finished The Legend of Zelda: Breath of the Wild

20180315_132531.jpg

Wow, what a game. It’s a game that had me talking with friends about video games again. You know, in that cagey way where you hint at something you aren’t sure they’ve seen yet until they nod and then you head into spoiler territory?

It’s a game that gives you every tool you need in the tutorial. Swords, shields, bows, bombs… and then you’re off. It doesn’t care what you do from then on, or how you do it. You simply go.

It’s a game that decides to not measure progress. Sure, it’ll count shrines and things… but it doesn’t care. Nothing changes, nothing matters except defeating Ganon. And yet, it is happy to let you take the long way around to get there, both in mechanics and in story. Who needs experience points, or levels, or priority missions, anyway?

It’s not perfect. The framerate drops in the Great Hyrule Forest. It leans a little hard on motion controls that ask you to point with the bottom of the controller instead of the point. Inside the Divine Beasts voices lead you around by the nose as though they didn’t get the “trust the player” memo the rest of the game hinges upon.

And the entire Gerudo segment comes across as tone deaf. Treating an ancient culture’s mores as a puzzle to “solve” because you know “better” is a little colonial for the 21st Century.

But then you hold these stumbles up to moments like when the Deku tree says “Next time I’ll let you kill yourself on that sword” and you believe it. The game has been honest with you so far, trusting you and being trusted almost to a fault. I believed that tree about that sword.

Speaking of that sword, pulling it was billed as a test of strength. “Strength?” There are no character sheets in Breath of the Wild, what is strength? Strength is the number of hearts you have. You are stronger when you have more heart. WHAT.

I’m especially pleased with this game coming straight off of Mass Effect: Andromeda. The cartoon characters you meet in Hyrule are different and recognizably so from their design. Even the hapless travellers you rescue time and time again are expressive in text and in facial expression.

ME:A also never wanted to let the player fail. Zelda is only too happy to have you fail. Too close to your own bomb when it goes off? Congratulations, you ragdolled down a cliff and into the river. Find a guardian? They’ll roflstomp you for hours. And you keep running across them! Even the main quest has you tromping up a hill in Zora’s Domain and finding something you cannot win against. Not because the game has decided it, but because you’re just not ready. And you can brickwall against it, like I did. Maybe if you jump over here, maybe if you use lightning arrows… Nope. You aren’t supposed to win this fight. You are the mouse, not the lion (or in this case, the Lynel). Scurry, little mouse, and try not to be seen.

Oh. No. You’ve been seen. Better run, little mouse. Run!

I’m sure there’s a wave of people who’ve seen this first in games like Dark Souls, but for me Zelda: Breath of the Wild was the first game in recent memory that made me afraid.

And how I love it for doing so.

Attacking Hyrule Castle, pushing deeper within. Everything the world taught you being put to the test, but upside down. On the overworld height helps you find your way, lets you see danger coming. In the Castle, height will get you killed. Better run, little mouse.

Wait. No. You’re not a mouse any more. You’ve fought, you’ve learned, you’ve grown.

But… you still remember being a mouse. So the tension ratchets up more than it ought to, creatures down the hall loom a little larger than they actually are, and victory… oh, victory tastes so much sweeter when you remember how impossible it used to be.

And then you reach Ganon. They really didn’t mind being a little disgusting with the creature design of the calamity bosses, and Ganon got a double dose. I feel a little cheap not having learned the perfect parry before fighting him, having to rely on my powers to defeat him… but that’s fitting. It works with the story. The champions were there for the assist.

Or so I tell myself.

The ending… was fine. The writing was a little weak, but everything else was lovely. I’m a little disappointed I don’t get to play in the world my adventure helped create, instead being dumped outside the castle gates, moments before the final confrontation, but it was an end.

And so I decided to put Zelda: Breath of the Wild down. I feel I could spend a lot more time in there. I feel I may have played it “wrong” by rushing too much to expand the map without exploring it enough (and my final map percentage of 40.03% certainly seems to reflect that). I can think of, right now, another couple of corners of the map I maybe should’ve gotten to.

But some things should end. Some things should leave you with that bittersweet hangover of a world your mind isn’t finished living within.

It’s been a long time since I’ve felt that from a video game. I missed it.

Advertisements

TIL: Feature Detection in Windows using GetProcAddress

In JavaScript, if you want to use a function that was introduced only in certain versions of browsers, you use Feature Detection. For example, you can ask “Hey, browser, do you have a function called `includes` on Array?” If the browser has it, you use it; and if it doesn’t, you either get along without it or load your own implementation.

It turns out that this same concept can be (and, in Firefox, is) done with Windows APIs.

Firefox for Windows is built against the Windows 10 SDK. This means the compiler knows the API calls and type definitions for all sorts of wondrous modern features like toast notifications and enumerating graphics adapters in a specific order.

However, as of writing, Firefox for Windows supports Windows 7 and up. What would happen if Firefox tried to use those fancy new Windows 10 features when running on Windows 7?

Well, at compile time (when Mozilla builds Firefox), it knows everything it needs to about the sizes and names of things used in the new features thanks to the SDK. At runtime (when a user runs Firefox), it needs to ask Windows at what address exactly all of those fancy new features live so that it can use them.

If Firefox can’t find a feature it expects to be there, it won’t start. We want Firefox to start, though, and we want to use the new features when available. So how do we both use the new feature (if it’s there) and not (if it’s not)?

Windows provides an API called GetProcAddress that allows the running program to perform some Feature Detection. It is asking Windows “Hey, so I’m looking for the address of this fancy new feature named FancyNewAPI. Do you know where that is?”. Windows will either reply “No, sorry” at which point you work around it, or “Yes, it’s over at address X” at which point to convert address X into a function pointer that takes the same number and types of arguments that the documentation said it takes and then instruct your program to jump into it and start executing.

We use this in Firefox to detect gamepad input modules, cancelable synchronous IO, display density measurements, and a whole bunch of graphics and media acceleration stuff.

And today (well, yesterday at this point) I learned about it. And now so have you.

:chutten

–edited to remove incorrect note that GetProcAddress started in WinXP– :aklotz noted that GetProcAddress has been around since ancient times, MSDN just periodically updates its “Minimum Supported Release” fields to drop older versions.

Perplexing Graphs: The Case of the 0KB Virtual Memory Allocations

Every Monday and Thursday around 3pm I check dev-telemetry-alerts to see if there have been any changes detected in the distribution of any of the 1500-or-so pieces of anonymous usage statistics we record in Firefox using Firefox Telemetry.

This past Monday there was one. It was a little odd.489b9ce7-84e6-4de0-b52d-e0179a9fdb1a

Generally, when you’re measuring continuous variables (timings, memory allocations…) you don’t see too many of the same value. Sure, there are common values (2GB of physical memory, for instance), but generally you don’t suddenly see a quarter of all reports become 0.

That was weird.

So I did what I always do when I find an alert that no one’s responded to, and triaged it. Mostly this involves looking at it on telemetry.mozilla.org to see if it was still happening, whether it was caused by a change in submission volumes (could be that we’re suddenly hearing from a lot more users, and they all report just “0”, for example), or whether it was limited to a single operating system or architecture:

windowsVSIZE

Hello, Windows.

windowsx64VSIZE

Specifically: hello Windows 64-bit.

With these clues, :erahm was able to highlight for me a bug that might have contributed to this sudden change: enabling Control Flow Guard on Windows builds.

Control Flow Guard (CFG) is a feature of Windows 8.1 (Update 3) and 10 that inserts some runtime checks into your binary to ensure you only make sensible jumps. This protects against certain exploits where attackers force a binary to jump into strange places in the running program, causing Bad Things to happen.

I had no idea how a control flow integrity feature would result in 0-size virtual memory allowances, but when :erahm gives you a hint, you take it. I commented on the bug.

Luckily, I was taken seriously, so a new bug was filed and :tjr looked into it almost immediately. The most important clue came from :dmajor who had the smartest money in the room, and crucial help from :ted who was able to reproduce the bug.

It turns out that turning CFG on made our Virtual Memory allowances jump above two terabytes.

Now, to head off “Firefox iz eatang ur RAM!!!!111eleven” commentary: this is CFG’s fault, not ours. (Also: Virtual Memory isn’t RAM.)

In order to determine what parts of a binary are valid “indirect jump targets”, Windows needs to keep track of them all, and do so performantly enough that the jumps can still happen at speed. Windows does this by maintaining a map with a bit per possible jump location. The bit is 1 if it is a valid location to jump to, and 0 if it is not. On each indirect jump, Windows checks the bit for the jump location and interrupts the process if it was about to jump to a forbidden place.

When running this on a 64-bit machine, this bitmap gets… big. Really big. Two Terabytes big. And that’s using an optimized way of storing data about the jump availability of up to 2^64 (18 quintillion) addresses. Windows puts this in the process’ storage allocations for its own recordkeeping reasons, which means that every 64-bit process with CFG enabled (on CFG-aware Windows versions (8.1 Update 3 and 10)) has a 2TB virtual memory allocation.

So. We have an abnormally-large value for Virtual Memory. How does that become 0?

Well, those of you with CS backgrounds (or who clicked on the “smartest money” link a few paragraphs back), will be thinking about the word “overflow”.

And you’d be wrong. Ish.

The raw number :ted was seeing was the number 2201166503936. This number is the number of bytes in his virtual memory allocation and is a few powers of two above what we can fit in 32 bits. However, we report the number of kilobytes. The number of kilobytes is 2149576664, well underneath the maximum number you can store in an unsigned 32-bit integer, which we all know (*eyeroll*) is 4294967296. So instead of a number about 512x too big to fit, we get one that can fit almost twice over.

Welll….

So we’re left with a number that should fit, being recorded as 0. So I tried some things and, sure enough, recording the number 2149576664 into any histogram did indeed record as 0. I filed a new bug.

Then I tried numbers plus or minus 1 around :ted’s magic number. They became zeros. I tried recording 2^31 + 1. Zero. I tried recording 2^32 – 1. Zero.

With a sinking feeling in my gut, I then tried recording 2^32 + 1. I got my overflow. It recorded as 1. 2^32 + 2 recorded as 2. And so on.

All numbers between 2^31 and 2^32 were being recorded as 0.

sensibleError

In a sensible language like Rust, assigning an unsigned value to a signed variable isn’t something you can do accidentally. You almost never want to do it, so why make it easy? And let’s make sure to warn the code author that they’re probably making a mistake while we’re at it.

In C++, however, you can silently convert from unsigned to signed. For values between 0 and 2^31 this doesn’t matter. For values between 2^31 and 2^32, this means you can turn a large positive number into a negative number somewhere between -2^31 and -1. Silently.

Telemetry Histograms don’t record negatives. We clamp them to 0. But something in our code was coercing our fancy unsigned 32-bit integer to a signed one before it was clamped to 0. And it was doing it silently. Because C++.

Now that we’ve found the problem, fixed the problem, and documented the problem we are collecting data about the data[citation] we may have lost because of the problem.

But to get there I had to receive an automated alert (which I had to manually check), split the data against available populations, become incredibly lucky and run it by :erahm who had an idea of what it might be, find a team willing to take me seriously, and then do battle with silent type coercion in a language that really should know better.

All in a day’s work, I guess?

:chutten

Software Ideas People Should Steal, Edition One

Here are five little ideas that I think every relevant software project should implement immediately.

1) WordPress has an excellent feature for linkifying text where pasting links over selected text will linkify the selected text to point to the link. All rich-text editing software needs to implement this on the double: if the clipboard you’re overpasting with starts with ‘http’, then linkify the text, don’t replace it.

2) My new Samsung Galaxy A5 has a little touch where it checks the ambient light level before turning on the screen. If it is dim where the user is, it gradually increases the brightness as you turn on the screen instead of immediately jumping to the current, adaptive screen brightness level. This saves my eyeballs from wincing. All phone manufacturers need to implement this.

3) Speaking of phones, when you’re about to go to sleep at night, you need to tell your phone to be quiet (except for the alarm, which should be loud). On BlackBerry 10 you could do this from the lock screen by drawing a shade down over the phone, putting it into Bedside Mode. Nearest I can figure, no other device allows you to do this without unlocking the phone. Lock screen Bedside Mode should’ve been copied by the other phone OSes years ago.

4) Speaking of BlackBerry 10, it still has the best text selection I’ve encountered in a phone. You want to select a paragraph of text. On Android or iPhone you press-hold until it selects a word, then you grab handles and labouriously drag them to where you want. On BB10 you press-hold until it selects a word, and then you keep holding. It selects a sentence. Keep holding. It selects a paragraph. Keep holding. It will visually start selecting further down the page until you finally release. “Expandable Text Selection” is discoverable, delightful, and useful. Phone OS developers, please implement this yesterday.

5) May as well round this off with yet another BlackBerry idea. This time, the BB10 Keyboard. You start typing a message but then realize halfway through that your wording reads as insensitive. The first half’s fine, but your phrasing went downhill six words ago. In the BB10 keyboard just swipe to the left (or right in RTL) six times. Each swipe deletes a word. Then you can start typing again. Near as I can figure, every other keyboard relies on mobile OS text selection to quickly replace more than a few letters at a time. Take this idea, keyboard developers. It’s wonderful.

That’s all for now, folks. If anyone’s surprised at how many of these are ideas from BlackBerry 10, I’d introduce you to the list I’m not writing about all of the ideas that current smartphones _already_ copied from that now-failed platform. It’s much longer.

:chutten

Leaving my BlackBerry Z10 for a Samsung Galaxy A5… Successfully¬†

android-authority-galaxy-a-7-2017

z10_5

 

An iPhone SE wouldn’t do. There were no DTEK50s left in Canada. And it was January, 2018.

Our Z10s were five years old. Happy Birthday.

I was frustrated. I had been researching how to solve the Phone Problem for about nine months. Over the past month and a half of concerted effort I had twice tried to buy new phones and get them to work. And I had twice failed.

Defeated, I gave up trying to game the system and reluctantly worked within it. I signed up for a two-year plan. I ordered a “0$” phone. I ordered us Samsung Galaxy A5s.

Not sure you’ve heard of those? Me neither. The trademark soup of Android phone names hasn’t improved in the decade since its release.

In this particular case you might not have heard of it because it wasn’t originally supposed to show up in North America. It was going to target EMEA and LatAm only… but then some Canadian carriers made it worth Samsung’s while to bring it to the Great White North.

So how is it. It is a tidy little phone with just enough nods to the now-past (SD Card slot, headphone jack), and just enough gee whiz features (fingerprint sensor for unlock, Always On OLED screen, USB Type-C connector) to bridge our way out of BlackBerry Land.

BBM Groups works, but notification settings are a joke. The Hub is here, but it is a pale shadow of its always-available BB10 edition.

I’ve managed to find a way to install the Android edition of the BlackBerry keyboard. It is familiar enough that I’ve composed all three of these voluminous posts about phones on it. It is strange enough in the details that I still can’t get capitalization correct all of the time.

It makes a difference to me that Samsung is trying to compete with Google in enough ways that many things don’t have to default to data collection. We’re still deep in the belly of the beast, but there are holes that I can see sunlight through so long as I can find the right settings to turn off.

It has enough nice touches that make me think that maybe the Samsung devs care about their phones as much as I care about the ones which bear my code. Like how, when it’s dark, the screen fades into brightness slower than when it’s light out. There’s also all these little false-starts on the phone as well that suggests investment in the R part of R&D, like swiping the phone with the edge of your hand to screenshot. I’d never do that, and have turned it off, but you have to try things to find things that work.

Also, I’ve found a replacement for BlackBerry Blend. It has its quirks, but so did Blend, and I’m looking forward to bending it to my will.

In short, it’s a mixed bag of features poured into a big slab of glass. It’ll do. Since it has to.

So that’s that. We have now left the land of Products of Five Years Past. It is a strange world, but it’s one with Firefox in it, so it can’t be all bad, right?

I hope you’ve found these rambling diatribes of Old Man Yelling at Phone to be entertaining, educating, or at least diverting.

Here’s to having to not have to do this for another five years! (he says, ignoring a creeping sense of dread)

Leaving my BlackBerry Z10 for a BlackBerry DTEK50

blackberry-dtek50

z10_5

 

Previously in my attempts to upgrade my and my wife’s nearly five-year-old smartphones, I tried the iPhone SE. It didn’t go well. But at least there was no hurry to move off of our venerable Z10s.

And then the December update of BBM landed.

The consumer portions of BlackBerry Messenger for Android and iPhone were licensed by BlackBerry Ltd to the Indonesian communications technology company Emtek in 2017. Indonesia has a huge BBM userbase, and a huge Android userbase, so this made a lot of sense for both companies.

This left the company-formerly-known-as-Research-in-Motion with BBM Enterprise and keeping the lights on for the consumer BBM infrastructure supporting BBOS and BB10.

What this means is fragmentation. In much the same way that languages will evolve dialects when isolated, so too will previously-compatible software become incompatible.

In this particular case, the December Android and iOS BBM updated so that pictures added to BBM Groups were no longer compatible with the BB10 version of the app. This shows as a “Feature not supported” message and a link to a page that says “Upgrade to the latest BBM” if you are on Android or iOS… and “Upgrade your phone to an Android or iOS device” if you are on BB10.

I found this out when one of my friends updated his phone to a BlackBerry KeyONE and could no longer send pictures of how tall his son had gotten. (6’5″ last I heard).

That’s right. A BlackBerry couldn’t communicate with another BlackBerry. With BBM.

The irony.

This was especially painful to us as we used BBM groups as a private, self-expiring social network. At this it was quite good: we could share photos and text with small groups made up of only those who might care, and the content would scroll off the top and eventually disappear without intervention. High signal, low noise.

Our friends stopped upgrading once they heard of our troubles, but things weren’t going to get any better if we waited. So on Boxing Day I ordered two BlackBerry DTEK50s from Amazon.

BlackBerry had stopped selling these in Canada, having licensed the device hardware business to Chinese manufacturer TCL under the name BlackBerry Mobile who is interested in selling BlackBerry KeyONE handsets and preorders for the BlackBerry Motion. As such, the only DTEK50s I could find were the EMEA editions, but they supported 2600MHz LTE on Band 4, which the towers throughout the province supported quite well.

I opened the phone and plopped in a SIM and suddenly realized just how BlackBerry an Android phone could feel. I don’t think I properly accounted for how much I would miss the BlackBerry keyboard, the blinking LED for notifications… and the Hub. Having all of my messages and stuff show in a single place really is the way I’ve become used to handling the volume of emails I receive.control

(( Of course this is anathema to the app model. Heaven forfend I control messages and data from aught but the app that owns the data. APIs, what’re those? ))

So I was mostly a happy camper. The notification settings weren’t as precise as BB10’s, I could tell within a day that I was going to miss Blend, and I had almost but not quite finished mourning the loss of “headers only” email download…

But there was a problem. The phone wouldn’t connect at anything higher than HSPA+. No LTE. Weird. I checked this before I bought it. Band 4 overlaps. Right?

After spending hours holding for carrier support and hours scouring the wide Web for help, I discovered no solution. There was nothing I could try. There were no leads to run down. The phone just stubbornly refused to connect to LTE, and likely never would.

This was a deal breaker. I wasn’t going to spend hundreds of dollars on phones that didn’t work, no matter how much I pretended that I wouldn’t mind since I’m on WiFi most of the time anyway.

So back they went to Amazon.

And back I went to a BBM that wouldn’t receive group pictures from a steadily increasing number of friends and family.

Something had to be done. And soon.

:chutten

Firefox Telemetry Use Counters: Over-estimating usage, now fixed

Firefox Telemetry records the usage of certain web features via a mechanism called Use Counters. Essentially, for every document that Firefox loads, we record a “false” if the document didn’t use a counted feature, and a “true” if the document did use that counted feature.

(( We technically count it when the documents are destroyed, not loaded, since a document could use a feature at any time during its lifetime. We also count top-level documents (pages) separately from the count of all documents (including iframes), so we can see if it is the pages that users load that are using a feature or if it’s the subdocuments that the page author loads on the user’s behalf that are contributing the counts. ))

To save space, we decided to count the number of documents once, and the number of “true” values in each use counter. This saved users from having to tell us they didn’t use any of Feature 1, Feature 2, Feature 5, Feature 7, … the “no-use” use counters. They could just tell us which features they did see used, and we could work out the rest.

Only, we got it wrong.

The server-side adjustment of the counts took every use counter we were told about, and filled in the “false” values. A simple fix.

But it didn’t add in the “no-use” use counters. Users who didn’t see a feature used at all weren’t having their “false” values counted.

This led us to under-count the number of “false” values (since we only counted “falses” from users who had at least one “true”), which led us to overestimate the usage of features.

Of all the errors to have, this one was probably the more benign. In failing in the “overestimate” direction we didn’t incorrectly remove features that were being used more than measured… but we may have kept some features that we could have removed, costing mozilla time and energy for their maintenance.

Once we detected the fault, we started addressing it. First, we started educating people whenever the topic came up in email and bugzilla. Second, :gfritzsche added a fancy Use Counter Dashboard that did a client-side adjustment using the correct “true” and “false” values for a given population.

Third, and finally, we fixed the server-side aggregator service to serve the correct values for all data, current and historical.

And that brings us to today: Use Counters are fixed! Please use them, they’re kind of cool.

:chutten

bfcbd97c-80cf-483b-8707-def6057474e6
Before
beb4afee-f937-4729-b210-f4e212da7504
After (4B more samples)