Distributed Teams: Not Just Working From Home

Technology companies taking curve-flattening exercises of late has resulted in me digging up my old 2017 talk about working as and working with remote employees. Though all of the advice in it holds up even these three years later, surprisingly little of it seemed all that relevant to the newly-working-from-home (WFH) multitudes.

Thinking about it, I reasoned that it’s because the talk (slides are here if you want ’em) is actually more about working on a distributed team than working from home. Though it contained the usual WFH gems of “have a commute”, “connect with people”, “overcommunicate”, etc etc (things that others have explained much better than I ever will); it also spent a significant amount of its time talking about things that are only relevant if your team isn’t working in the same place.

Aspects of distributed work that are unique not to my not being in the office but my being on a distributed team are things like timezones, cultural differences, personal schedules, presentation, watercooler chats, identity… things that you don’t have to think about or spend effort on if you work in the same place (and, not coincidentally, things I’ve written about in the past). If we’re all in Toronto you know not only that 12cm of snow fell since last night but also what that does to the city in the morning. If we’re all in Italy you know not to schedule any work in August. If we see each other all the time then I can use a picture I took of a glacier in Iceland for my avatar instead of using it as a rare opportunity to be able to show you my face.

So as much as I was hoping that all this sudden interest in WFH was going to result in a sea change in how working on a distributed team is viewed and operates, I’m coming to the conclusion that things probably will not change. Maybe we’ll get some better tools… but none that know anything about being on a distributed team (like how “working hours” aren’t always contiguous (looking at you, Google Calendar)).

At least maybe people will stop making the same seven jokes about how WFH means you’re not actually working.

:chutten

Jira, Bugzilla, and Tales of Issue Trackers Past

It seems as though Mozilla is never not in a period of transition. The distributed nature of the organization and community means that teams and offices and any informal or formal group is its own tiny experimental plot tended by gardeners with radically different tastes.

And if there’s one thing that unites gardeners and tech workers is that both have Feelings about their tools.

Tools are personal things: they’re the only thing that allows us to express ourselves in our craft. I can’t code without an editor. I can’t prune without shears. They’re the part of our work that we actually touch. The code lives Out There, the garden is Outside… but the tools are in our hands.

But tools can also be group things. A shed is a tool for everyone’s tools. A workshop is a tool that others share. An Issue Tracker is a tool that helps us all coordinate work.

And group things require cooperation, agreement, and compromise.

While I was on the Browser team at BlackBerry I used a variety of different Issue Trackers. We started with an outdated version of FogBugz, then we had a Bugzilla fork for the WebKit porting work and MKS Integrity for everything else across the entire company, and then we all standardized on Jira.

With minimal customization, Jira and MKS Integrity both seemed to be perfectly adequate Issue Tracking Software. They had id numbers, relationships, state, attachments, comments… all the things you need in an Issue Tracker. But they couldn’t just be “perfectly adequate”, they had to be better enough than what they were replacing to warrant the switch.

In other words, to make the switch the new thing needs to do something that the previous one couldn’t, wouldn’t, or just didn’t do (or you’ve forgotten that it did). And every time Jira or MKS is customized it seems to stop being Issue Tracking Software and start being Workflow Management Software.

Perhaps because the people in charge of the customization are interested more in workflows than in Issue Tracking?

Regardless of why, once they become Workflow Management Software they become incomparable with Issue Trackers. Apples and Oranges. You end up optimizing for similar but distinct use cases as it might become more important to report about issues than it is to file and fix and close them.

And that’s the state Mozilla might be finding itself in right now as a few teams here and there try to find the best tools for their garden and settle upon Jira. Maybe they tried building workflows in Bugzilla and didn’t make it work. Maybe they were using Github Issues for a while and found it lacking. We already had multiple places to file issues, but now some of the places are Workflow Management Software.

And the rumbling has begun. And it’s no wonder, as even tools that are group things are still personal. They’re still what we touch when we craft.

The GNU-minded part of me thinks that workflow management should be built above and separate from issue tracking by the skillful use of open and performant interfaces. Bugzilla lets you query whatever you want, however you want, so why not build reporting Over There and leave me my issue tracking Here where I Like It.

The practical-minded part of me thinks that it doesn’t matter what we choose, so long as we do it deliberately and consistently.

The schedule-minded part of me notices that I should probably be filing and fixing issues rather than writing on them. And I think now’s the time to let that part win.

:chutten

This Week in Glean: A Distributed Team Echoes Distributed Workflow

(“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find an index of all TWiG posts online.)

Last Week: Extending Glean: build re-usable types for new use-cases by Alessio


I was recently struck by a realization that the position of our data org’s team members around the globe mimics the path that data flows through the Glean Ecosystem.

Glean Data takes this five-fold path (corresponding to five teams):

  1. Data is collected in a client using the Glean SDK (Glean Team)
  2. Data is transmitted to the Structured Ingestion pipeline (Data Platform)
  3. Data is stored and maintained in our infrastructure (Data Operations)
  4. Data is presented in our tools (Data Tools)
  5. Data is analyzed and reported on (Data Science)

The geographical midpoint of the Glean Team is about halfway across the north atlantic. For Data Platform it’s on the continental US, anchored by three members in the midwestern US. Data Ops is further West still, with four members in the Pacific timezone and no Europeans. Data Tools breaks the trend by being a bit further East, with fewer westcoasters. Data Science (for Firefox) is centred farther west still, with only two members East of the Rocky Mountains.

Or, graphically:

gleanEcosystemTeamCentres
(approximate) Team Geocentres

Given the rotation of the Earth, the sun rises first on the Glean Team and the data collected by the Glean SDK. Then the data and the sun move West to the Data Platform where it is ingested. Data Tools gets the data from the Platform as morning breaks over Detroit. Data Operations keeps it all running from the midwest. And finally, the West Coast Centre of Firefox Data Science Excellence greets the data from a mountaintop, to try and make sense of it all.

(( Lying orthogonal to the data organization is the secret Sixth Glean Data “Team”: Data Stewardship. They ensure all Glean Data is collected in accordance with Mozilla’s Privacy Promise. The sun never sets on the Stewardship’s global coverage, and it’s a volunteer effort supplied from eight teams (and growing!), so I’ve omitted them from this narrative. ))

Bad metaphors about sunlight aside, I wonder whether this is random or whether this is some sort of emergent behaviour.

Conway’s Law suggests that our system architecture will tend to reflect our orgchart (well, the law is a bit more explicit about “communication structure” independent of organizational structure, but in the data org’s case they’re pretty close). Maybe this is a specific example of that: data architecture as a reflection of orgchart geography.

Or perhaps five dots on a globe that are only mostly in some order is too weak of a coincidence to even bear thinking about? Nah, where’s the blog post in that thinking…

If it’s emergent, it then becomes interesting to consider the “chicken and egg” point of view: did the organization beget the system or the system beget the organization? When I joined Mozilla some of these teams didn’t exist. Some of them only kinda existed within other parts of the company. So is the situation we’re in today a formalization by us of a structure that mirrors the system we’re building, or did we build the system in this way because of the structure we already had?

mindblown.gif

:chutten

Distributed Teams: Why I Don’t Go to the Office More Often

I was invited to a team dinner as part of a work week the Data Platform team was having in Toronto. I love working with these folks, and I like food, so I set about planning my logistics.

The plan was solid, but unimpressive. It takes three hours or so to get from my home to the Toronto office by transit, so I’d be relying on the train’s WiFi to allow me to work on the way to Toronto, and I’d be arriving home about 20min before midnight.

Here’s how it went:

  1. 0800 Begin
  2. 0816 Take the GRT city bus to Kitchener train station
  3. 0845 Try to find a way to get to the station (the pedestrian situation around the station is awful)
  4. 0855 Learn that my 0918 train is running 40min late.
  5. 0856 Purchase a PRESTO card for my return journey, being careful to not touch any of the blood stains on the vending machine. (Seriously. Someone had a Bad Time at Kitchener Station recently)
  6. 0857 Learn that they had removed WiFi from the train station, so the work I’ll be able to do is limited to what I can manage on my phone’s LTE
  7. 0900 Begin my work day (Slack and IRC only), and eat the breakfast I packed because I didn’t have time at home.
  8. 0943 Train arrives only 35min late. Goodie.
  9. 0945 Learn from the family occupying my seat that I actually bought a ticket for the wrong day. Applying a discount code didn’t keep the date and time I selected, and I didn’t notice until it was too late. Sit in a different seat and wonder what the fare inspector will think.
  10. 0950 Start working from my laptop. Fear of authority can build on its own time, I have emails to answer and bugs to shuffle.
  11. 1030 Fare inspector finally gets around to me as my nervousness peaks. Says they’ll call it in and there might be an adjustment charge to reschedule it.
  12. 1115 Well into Toronto, the fare inspector just drops my ticket into my lap on their way to somewhere else. I… guess everything’s fine?
  13. 1127 Train arrives at Toronto Union Station. Disconnect WiFi, disembark and start walking to the office. (Public transit would be slower, and I’m saving my TTC token for tonight’s trip)
  14. 1145 Arrive at MoTo just in time for lunch.

Total time to get to Mozilla Toronto: 3h45min. Total distance traveled: 95km Total cost: $26 for the Via rail ticket, $2.86 for the GRT city bus.

The way back wasn’t very much better. I had to duck out of dinner at 8pm to have a hope of getting home before the day turned into tomorrow:

  1. 2000 Leave the team dinner, say goodnights. Start walking to the subway
  2. 2012 At the TTC subway stop learn that the turnstiles don’t take tokens any more. Luckily there’s someone in the booth to take my fare.
  3. 2018 Arrive at Union station and get lost in the construction. I thought the construction was done (the construction is never done).
  4. 2025 Ask at the PRESTO counter how to use PRESTO properly. I knew it was pay-by-distance but I was taking a train _and_ a bus, so I wasn’t sure if I needed to tap in between the two modes (I do. Tap before the train, after the train, on the bus when you get on, and on the bus when you get off. Seems fragile, but whatever).
  5. 2047 Learn that the train’s been rescheduled 6min later. Looks like I can still make my bus connection in Bramalea.
  6. 2053 Tap on the thingy, walk up the flights of stairs to the train, find a seat.
  7. 2102 “Due to platform restrictions, the doors on car 3107 will not open at Bramalea”… what car am I on? There’s no way to tell from where I’m sitting.
  8. 2127 Arrive at Bramalea. I’m not on car 3107.
  9. 2130 Learn that there’s one correct way to leave the platform and I took the other one that leads to the parking lot. Retrace my steps.
  10. 2132 Tap the PRESTO on the thingy outside the station building (closed)
  11. 2135 Tap the PRESTO on the thingy inside the bus. BEEP BEEP. Bus driver says insufficient funds. That can’t be, I left myself plenty of room. Tick tock.
  12. 2136 Cold air aching in my lungs from running I load another $20 onto the PRESTO
  13. 2137 Completely out of breath, tap the PRESTO on the thingy inside the bus. Ding. Collapse in a seat. Bus pulls out just moments later.
  14. 2242 Arrive in Kitchener. Luckily the LRT, running at 30min headways, is 2min away. First good connection of the day.
  15. 2255 This is the closest the train can get me. There’s a 15min wait (5 of which I’ll have to walk in the cold to get to the stop) for a bus that’ll get me, in 7min, within a 10min walk from home. I decide to walk instead, as it’ll be faster.
  16. 2330 Arrive home.

Total time to get home: 3h30min. Total distance traveled: 103km. Total cost: $3.10 for the subway token, $46 PRESTO ($6 for the card, $20 for the fare, $20 for the surprise fare), $2.86 for the LRT.

At this point I’ve been awake for over 20 hours.

Is it worth it? Hard to say. Every time I plan one of these trips I look forward to it. Conversations with office folks, eating office lunch, absconding with office snacks… and this time I even got to go out to dinner with a bunch of data people I work with all the time!

But every time I do this, as I’m doing it, or as I’m recently back from doing it… I don’t feel great about it. It’s essentially a full work day (nearly eight full hours!) just in travel to spend 5 hours in the office, and (this time) a couple hours afterwards in a restaurant.

Ultimately this — the share of my brain I need to devote purely to logistics, the manifold ways things can go wrong, the sheer _time_ it all takes — is why I don’t go into the office more often.

And the people are the reason I do it at all.

:chutten

Distributed Teams: Regional Peculiarities Like Oktoberfest and Bagged Milk

It’s Oktoberfest! You know, that German holiday about beer and lederhosen?

No. As many Germans will tell you it’s not a German thing as much as it is a Bavarian thing. It’s like saying kilts are a British thing (it’s a Scottish thing). Or that milk in bags is a Canadian thing (in Canada it’s an Eastern Canada thing).

In researching what the heck I was talking about when I was making this comparison at a recent team meeting, Alessio found a lovely study on the efficiency of milk bags as milk packaging in Ontario published by The Environment and Plastics Industry Council in 1997.

I highly recommend you skim it for its graphs and the study conclusions. The best parts for me are how it highlights that the consumption of milk (by volume) increased 22% from 1968 to 1995 while at the same time the amount (by mass) of solid waste produced by milk packaging decreased by almost 20%.

I also liked Table 8 which showed the recycling rates of the various packaging types that we’d need to reach in order to match the small amount (by mass) of solid waste generation of the (100% unrecycled) milk bags. (Interestingly, in my region you can recycle milk bags if you first rinse and dry them).

I guess what I’m trying to say about this is three-fold:

  1. Don’t assume regional characteristics are national in your distributed team. Berliners might not look forward to Oktoberfest the way Münchner do, and it’s possible no one in the Vancouver office owns a milk jug or bag cutter.
  2. Milk Bags are kinda neat, and now I feel a little proud about living in a part of the world where they’re common. I’d be a little more confident about this if the data wasn’t presented by the plastics industry, but I’ll take what I can get (and I’ll start recycling my milk bags).
  3. Geez, my team can find data for _any topic_. What differences we have by being distributed around the world are eclipsed by how we’re universally a bunch of nerds.

:chutten

 

Distributed Teams: A Test Failing Because It’s Run West of Newfoundland and Labrador

(( Not quite 500 mile email-level of nonsense, but might be the closest I get. ))

A test was failing.

Not really unusual, that. Tests fail all the time. It’s how we know they’re good tests: protecting us developers from ourselves.

But this one was failing unusually. Y’see, it was failing on my machine.

(Yes, har har, it is a common-enough occurrence given my obvious lack of quality as a developer how did you guess.)

The unusual part was that it was failing only for me… and I hadn’t even touched anything yet. It wasn’t failing on our test infrastructure “try”, and it wasn’t failing on the machine of :raphael, the fellow responsible for the integration test harness itself. We were building Firefox the same way, running telemetry-tests-client the same way… but I was getting different results.

I fairly quickly locked down the problem to be an extra “main” ping with reason “environment-change” being sent during the initial phases of the test. By dumping some logging into Firefox, rebuilding it, and then routing its logs to console with --gecko-log "-" I learned that we were sending a ping because a monitored user preference had been changed: browser.search.region.

When Firefox starts up the first time, it doesn’t know where it is. And it needs to know where it is to properly make a first guess at what language you want and what search engines would work best. Google’s results are pretty bad in Canada unless you use “google.ca”, after all.

But while Firefox doesn’t know where it is, it does know is what timezone it’s in from the settings in your OS’s clock. On top of that it knows what language your OS is set to. So we make a first guess at which search region we’re in based on whether or not the timezone overlaps a US timezone and if your OS’ locale is `en-US` (United States English).

What this fails to take into account is that United States English is the “default” locale reported by many OSes even if you aren’t in the US. And how even if you are in a timezone that overlaps with the US, you might not be there.

So to account for that, Mozilla operates a location service to double-check that the search region is appropriately set. This takes a little time to get back with the correct information, if it gets back to us at all. So if you happen to be in a US-overlapping timezone with an English-language OS Firefox assumes you’re in the US. Then if the location service request gets back with something that isn’t “US”, browser.search.region has to be updated.

And when it updates, it changes the Telemetry Environment.

And when the Environment changes, we send a “main” ping.

And when we send a “main” ping, the test breaks.

…all because my timezone overlaps the OS and my language is “Default” English.

I feel a bit persecuted, but this shows another strength of Distributed Teams. No one else on my team would be able to find this failure. They’re in Germany, Italy, and the US. None of them have that combination of “Not in the US, but in a US timezone” needed to manifest the bug.

So on one hand this sucks. I’m going to have to find a way around this.

But on the other hand… I feel like my Canadianness is a bit of a bug-finding superpower. I’m no Brok Windsor or Captain Canuck, but I can get the job done in a way no one else on my team can.

Not too bad, eh?

:chutten

Distributed Teams: Regional Holidays

Today is German Unity Day, Germany’s National Day. Half of my team live in Berlin, so I vaguely knew they wouldn’t be around… but I’d likely have forgotten if not for a lovely tradition of “Holiday Inbound” emails at Mozilla.

Mozilla is a broadly-distributed organization with employees in dozens of countries worldwide. Each of these countries have multiple days off to rest or celebrate. It’s tough to know across so many nations and religions and cultures exactly who will be unable to respond to emails on exactly which days.

So on the cusp of a holiday it is tradition in Mozilla to send a Holiday Inbound email to all Mozilla employees noting that the country you’re trying to reach can’t come to the phone right now, so please leave a message at the tone.

More than just being a bland notification some Mozillians take the opportunity to explain the history and current significance of the event being celebrated. I’ve taken a crack at explaining the peculiarly-Canadian holiday of Christmas (pronounced [kris-muhs]) in the past.

Sometimes you even get some wonderful piece of alternate history like :mhoye’s delightful, 50% factual exploration of the origins of Canadian Labour Day 2016.

I delight in getting these notifications from our remotees and offices worldwide. It really brings us closer together through understanding, while simultaneously highlighting just how different we all are.

Maybe I should pen a Holiday Inbound email about Holiday Inbound emails. It would detail the long and fraught history of the tradition in a narrative full of villains and heroes and misspellings and misunderstandings…

Or maybe I should just try to get some work done while my German colleagues are out.

:chutten

Distributed Teams: On the non-Universality of “Not it!”

I’ve surprisingly not written a lot over here about working on a distributed team in a distributed organization. Mozilla is about 60% people who work in MoLos (office workers) and 40% people who don’t (remotees). My team is 50/50: I’m remote near Toronto, one works from his home in Italy, and the other two sit in the Berlin office most days.

If I expand to encompass one extra level of organization, I work with people in Toronto, San Francisco, Poland, Iowa, Illinois, more Berlin, Nova Scotia… well, you get the idea. For the past two and a half years I’ve been working with people from all over the world and I have been learning how that’s different from the rather-more-monocultural experience I had working in offices for the previous 8-10.

So today when I shouted “Not it!” into the IRC channel in response to the dawning realization that someone would have to investigate and take ownership of some test bustage, I followed it up within the minute with a cultural note:

09:35 <chutten> (Actually, that's a cultural thing that may need
 explanation. As kids, usually at summer sleep-away camp, if there
 is an undesirable thing that needs to be done by one person in
 the cabin the last person to say "Not it" is "It" and thus, has
 to do the undesirable thing.)

“Not it” is cultural. I think. I’ve been able to find surprisingly little about its origins in the usual places. It seems to share some heritage with the game of Tag. It seems to be an evolution of the game “Nose Goes,” but it’s hard to tell exactly where any of it started. Wikipedia can’t find an origin earlier than the 1979 Canadian film “Meatballs” where the nose game was already assumed to be a part of camp life.

Regardless of origin, I can’t assume it’s shared amongst my team. Thus my little note. Lucky for me, they seem to enjoy learning things like this. Even luckier, they enjoy sharing back. For instance, :gfritzsche once said his thumbs were pressed that we’d get something done by week’s end… what?

There were at least two things I didn’t understand about that: the first was what he meant, the second was how one pressed one’s thumbs. I mean, do you put them in your fist and squeeze, or do you press them on the outside of your fist and pretend you’re having a Thumb War (yet another cultural artefact)?

First, it means hoping for good luck. Second, it’s with thumbs inside your fist, not outside. I’m very lucky there’s a similar behaviour and expression that I’m already familiar with (“fingers crossed”). This will not always be the case, and it won’t ever be an even exchange…

All four of my team members speak the language I spoke at home while I was growing up. A lot of my culture is exported by the US via Hollywood, embedding it into the brains of the people with whom I work. I have a huge head-start on understanding and being understood, and I need to remain mindful of it.

Hopefully I’ll be able to catch some of my more egregious references before I need to explain camp songs, cringe-worthy 90s slang, or just how many hours I spent in a minivan with my siblings looking for the letter X on a license plate.

Or, then again, maybe those explanations are just part of being a distributed team?

:chutten