Jira, Bugzilla, and Tales of Issue Trackers Past

It seems as though Mozilla is never not in a period of transition. The distributed nature of the organization and community means that teams and offices and any informal or formal group is its own tiny experimental plot tended by gardeners with radically different tastes.

And if there’s one thing that unites gardeners and tech workers is that both have Feelings about their tools.

Tools are personal things: they’re the only thing that allows us to express ourselves in our craft. I can’t code without an editor. I can’t prune without shears. They’re the part of our work that we actually touch. The code lives Out There, the garden is Outside… but the tools are in our hands.

But tools can also be group things. A shed is a tool for everyone’s tools. A workshop is a tool that others share. An Issue Tracker is a tool that helps us all coordinate work.

And group things require cooperation, agreement, and compromise.

While I was on the Browser team at BlackBerry I used a variety of different Issue Trackers. We started with an outdated version of FogBugz, then we had a Bugzilla fork for the WebKit porting work and MKS Integrity for everything else across the entire company, and then we all standardized on Jira.

With minimal customization, Jira and MKS Integrity both seemed to be perfectly adequate Issue Tracking Software. They had id numbers, relationships, state, attachments, comments… all the things you need in an Issue Tracker. But they couldn’t just be “perfectly adequate”, they had to be better enough than what they were replacing to warrant the switch.

In other words, to make the switch the new thing needs to do something that the previous one couldn’t, wouldn’t, or just didn’t do (or you’ve forgotten that it did). And every time Jira or MKS is customized it seems to stop being Issue Tracking Software and start being Workflow Management Software.

Perhaps because the people in charge of the customization are interested more in workflows than in Issue Tracking?

Regardless of why, once they become Workflow Management Software they become incomparable with Issue Trackers. Apples and Oranges. You end up optimizing for similar but distinct use cases as it might become more important to report about issues than it is to file and fix and close them.

And that’s the state Mozilla might be finding itself in right now as a few teams here and there try to find the best tools for their garden and settle upon Jira. Maybe they tried building workflows in Bugzilla and didn’t make it work. Maybe they were using Github Issues for a while and found it lacking. We already had multiple places to file issues, but now some of the places are Workflow Management Software.

And the rumbling has begun. And it’s no wonder, as even tools that are group things are still personal. They’re still what we touch when we craft.

The GNU-minded part of me thinks that workflow management should be built above and separate from issue tracking by the skillful use of open and performant interfaces. Bugzilla lets you query whatever you want, however you want, so why not build reporting Over There and leave me my issue tracking Here where I Like It.

The practical-minded part of me thinks that it doesn’t matter what we choose, so long as we do it deliberately and consistently.

The schedule-minded part of me notices that I should probably be filing and fixing issues rather than writing on them. And I think now’s the time to let that part win.

:chutten

This Week in Glean: A Distributed Team Echoes Distributed Workflow

(“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find an index of all TWiG posts online.)

Last Week: Extending Glean: build re-usable types for new use-cases by Alessio


I was recently struck by a realization that the position of our data org’s team members around the globe mimics the path that data flows through the Glean Ecosystem.

Glean Data takes this five-fold path (corresponding to five teams):

  1. Data is collected in a client using the Glean SDK (Glean Team)
  2. Data is transmitted to the Structured Ingestion pipeline (Data Platform)
  3. Data is stored and maintained in our infrastructure (Data Operations)
  4. Data is presented in our tools (Data Tools)
  5. Data is analyzed and reported on (Data Science)

The geographical midpoint of the Glean Team is about halfway across the north atlantic. For Data Platform it’s on the continental US, anchored by three members in the midwestern US. Data Ops is further West still, with four members in the Pacific timezone and no Europeans. Data Tools breaks the trend by being a bit further East, with fewer westcoasters. Data Science (for Firefox) is centred farther west still, with only two members East of the Rocky Mountains.

Or, graphically:

gleanEcosystemTeamCentres
(approximate) Team Geocentres

Given the rotation of the Earth, the sun rises first on the Glean Team and the data collected by the Glean SDK. Then the data and the sun move West to the Data Platform where it is ingested. Data Tools gets the data from the Platform as morning breaks over Detroit. Data Operations keeps it all running from the midwest. And finally, the West Coast Centre of Firefox Data Science Excellence greets the data from a mountaintop, to try and make sense of it all.

(( Lying orthogonal to the data organization is the secret Sixth Glean Data “Team”: Data Stewardship. They ensure all Glean Data is collected in accordance with Mozilla’s Privacy Promise. The sun never sets on the Stewardship’s global coverage, and it’s a volunteer effort supplied from eight teams (and growing!), so I’ve omitted them from this narrative. ))

Bad metaphors about sunlight aside, I wonder whether this is random or whether this is some sort of emergent behaviour.

Conway’s Law suggests that our system architecture will tend to reflect our orgchart (well, the law is a bit more explicit about “communication structure” independent of organizational structure, but in the data org’s case they’re pretty close). Maybe this is a specific example of that: data architecture as a reflection of orgchart geography.

Or perhaps five dots on a globe that are only mostly in some order is too weak of a coincidence to even bear thinking about? Nah, where’s the blog post in that thinking…

If it’s emergent, it then becomes interesting to consider the “chicken and egg” point of view: did the organization beget the system or the system beget the organization? When I joined Mozilla some of these teams didn’t exist. Some of them only kinda existed within other parts of the company. So is the situation we’re in today a formalization by us of a structure that mirrors the system we’re building, or did we build the system in this way because of the structure we already had?

mindblown.gif

:chutten

Controlling a Linux Laptop’s Internet Access

I fear the Internet. It’s powerful and full of awesome and awful things. That might be why, in my house, there are no smart devices.

This fear is now warring with my duties as a parent for ensuring my child has the best chance at succeeding in whatever she chooses to do with her life. No matter what she chooses, the Internet is likely to be a part of it because the Internet underpins everything these days. So like swimming, riding a bike, and driving a car (skills I see as necessities in Canada), she needs to learn how to handle herself on the Internet.

And like swimming, I need a shallow end. Like cycling, training wheels. Like driving, a learner’s permit. I need something to get both her and I used to the idea of the whole thing… but safer. More restrictive.

So to act as a stepping stone between always-on Full Internet Access and the “ask your parents to look something up for you”, my wife and I determined we’d repurpose an ancient and underpowered laptop (with the battery removed) as an Email and Scratch (MIT’s visual programming language that she’s using to make a Star Wars video game at the moment) machine.

I thought this’d be easy. There’s Scratch for Linux, and I could adjust the firewall to only pass IMAP (for email receipt) and SMTP (for email sending). Alas, her email is hosted by GMail, and all Google properties have standardized on OAuth2 for authentication. OAuth2 means HTTP requests. Letting HTTP access into the Laptop opens it up to always-on Full Internet Access, so what to do, what to do.

Luckily there is precisely one Google OAuth2 server with a well-known name that we need to reach over HTTP. Unluckily it can be at an arbitrary number of different IP addresses. Luckily I just learned how to use `dnsmasq` to configure how name resolution works on the laptop.

So this is how to adjust Linux Mint 19 to allow DNS queries to resolve exactly the servers I want to allow for email… and no others.

1) Tell `NetworkManager` (the network manager) to use its `dnsmasq` plugin by editing `/etc/NetworkManager/conf.d/00-use-dnsmasq.conf` to contain

[main]
dns=dnsmasq

2) Configure `dnsmasq` to our specifications by editing `/etc/NetworkManager/dnsmasq.d/00-urlfilter.conf` to contain

log-queries # Log all DNS queries for debugging purposes
log-facility=/var/log/dnsmasq.log # Log things here

no-resolv # don't use resolv.conf
interface=wlp12s0 # Bind requests on this interfaced (maybe I should bind eth0 too in case she plugs it in?)
listen-address=127.0.0.53 # This is where the system's expecting to find systemd-resolve's DNS. Take it over.

address=/#/127.0.0.1 # Resolve all hosts to localhost. Excepting the below
server=/imap.gmail.com/8.8.8.8 # Use Google's DNS to resolve incoming mail server
server=/smtp.gmail.com/8.8.8.8 # Use Google's DNS to resolve outgoing mail server
server=/googleapis.com/8.8.8.8 # Use Google's DNS to resolve OAuth2 server

3) Tell `systemd-resolved` (our current DNS resolver) not to get in our way by editing `/etc/systemd/resolved.conf` to have the line

DNSStubListener=no # Don't start the local `resolved` DNS cache/server (conflicts with dnsmasq)

4) Restart `​systemd-resolved` so it stops its DNS listener

sudo systemctl restart systemd-resolved.service

5) Restart `NetworkManager` so that `dnsmasq` can take over

sudo systemctl restart NetworkManager

Caveats:

Though this works for now insofar as loading up Firefox and trying to go to example.com will fail, it doesn’t stop the laptop from accessing the Internet. If you have an IP Address in hand, you can still get to where you want to be (though most third-party-hosted resources will fail (and since that’s how trackers work on the web, maybe this is a feature not a bug)).

Also this prevents even beneficial services from running. To update packages I need to bypass the filter (by commenting out `address=/#/127.0.0.1` and putting in `server=/#/8.8.8.8`).

Also also this effectively forbids access to local services like my network-attached storage and the printer: both things that my daughter should be able to use. (( there might be exactly one `server` line I need to add to make LAN-local shortnames resolve using my router, but I haven’t looked that up yet ))

But this was enough to get it up and running (though the laptop is ancient enough that even legacy gpu drivers have dropped support for it), and that’s the important thing. Now she’s sending emails to her relatives and fielding their responses regularly… and can crack open her Star Wars videogame and do a little recreational programming on the side.

I even had her type up her Science Fair documentation on it the other day (though she still hasn’t learned how to manually Save documents, so the ancient laptop’s instability caused some friction).

We’ll see how long this lasts. Not bad for a couple hour’s work and available parts. I wonder how soon I’ll be convinced to upgrade her to more of the Internet or a less-decrepit machine.

:chutten