Teaching the Feds to Use GitHub

As I mentioned in my last post on VoT, I helped work on NIST’s new version of Special Publication 800–63, the Digital Identity Guidelines for US Federal Government Agencies. That document is now in its public review period, which means that it’s very close to being finalized. Considering that the previous edition of the guidelines was in effect for years — or a veritable eternity in internet time — this publication has a very good chance of sticking around for a bit. Looking now at the near-final results, I’m proud of this document and my involvement with it. It was a monumental undertaking, and in my opinion the team did a great job with it.

But there’s something else I’m really proud of with this document: its development process was radically different from how most government organizations (including NIST) traditionally manage things. Normally, the federal government document creation process is very linear and consists of something like this:

• Get the stakeholders that you can think of (and find) together to talk about requirements
• Develop those requirements into a deliverable document
• Publish and circulate that requirements document to other key stakeholders and experts that you can think of (and find)
• Lock the nerds you found in a back room of a featureless government building for a few months and tell them to write something
• Take the results of this process and pass it around to everyone in the agency that might have a say in its contents, so they can put their own spin on it
• Iterate on the last two steps a few times, with special note to all changes being made behind closed doors
• Publish the completed version and ask for comments from the public
• Take what comments you can, but generally tell commenters that it’s too late to change anything because it’s already published and you’ve been working so hard on it anyway that you don’t have the ability to change anything anymore (since the work happened out of sight of most people, this can seem like a brush off)
• Begin the process over when the friction against the published document is great enough to warrant starting the machine again

The traditional system is set up to produce finished documents for public consumption, and the final results appear to outsiders as if the proclamations of the government spring fully formed from the foreheads-of-state. This isn’t the intent, of course, but few people get to see such sausage being made; and with the old model, the only real record of the process is the final document. This process made more sense back when document authorship and review were largely done on paper passed around between offices. Even small changes in such an environment are expensive, so it’s best to bundle everything into batches and try to get them all in one go.

But the fourth version of 800–63 (known as 800–63–3 because of zero-indexing fun) was intended to live and breathe in a world where the immediacy and transparency of social media would grate against the languid pace of the traditional government approach. Thankfully for all of us involved, a better and more dynamic pattern had been well established by the open source software movement. Development tools like GitHub have made it so that software developers can collaborate rapidly in an open environment, where speed and transparency are valued by all in the community. The history of all changes is clearly visible (including authorship), issue trackers allow people to report bugs and request enhancements, and active contributors can fork the repository and offer changes back as pull requests.

When I was pulled in to help author the next version of 800–63, it was clear that our team wanted to embrace a more open model. The heads of the project at NIST were very clear that they wanted to have a more transparent and dynamic process, and they fought hard to get the approval to do just that. The question was how to adapt the requirements of the government process to take advantage of the capabilities of the new tools. There has been growing adoption of such processes in standards definition organizations around the world, but we still ran into challenges as this approach is relatively novel within government spaces. Here’s what we did about them.

Staying See-Through

First was the problem of transparency. As the final versions of these documents tend to stand around for a very long time and have long reaching effects in the world, there’s a lot of pressure to make them as good as possible when the public first sees them, in order to avoid the embarrassment of publishing something with errors. There’s a similar pressure in software development to release code with as few bugs as possible. The open source development model lays flaws open to the public, but for good reason: many eyes make all bugs shallow.

First we needed to make sure the documents were in a format that was easy for everyone to consume. The previous edition of the 800–63 guidelines had been written in Microsoft Word and published as a PDF — neither format is particularly conducive to collaboration or tracking changes of a large number of authors over time. We quickly settled on Markdown, a text-based markup language that would let us make full use of Git’s version control features while still being able to be translated into a number of publishable formats like HTML and PDF for consumption of the final document.

Now we had the starter document, and we put it into a public GitHub repository. We then let everybody know that we were starting this rewrite, and doing so in plain sight. Pretty soon, the eyeballs we hoped for started to roll in.

The resistance to this approach comes from the fact that all those eyes are scary to those being looked at. People would see the authors make typos, use bad grammar, have mislabeled diagrams, and make other mistakes that serious document writers would really rather hide from the prying eyes of the public. This isn’t an issue just for the government; few enjoy their unfinished hard work being put out on display like that. Why go through this scrutiny? The document and all its flaws were not put out for criticism alone: we were asking everyone to help.

There’s simply no way that a single organization — no matter how brilliant those nerds in the back room are — could collect all of the best experts in every area that a document would touch. Even if an agency could get the best experts together and have them focus, they can’t get all the experts together. Peer review during the development and publication process is immensely useful as it encourages more people to engage in the areas that they are most concerned with. All of the internal experts can now interact with the other engaged and interested people in the outside world. By laying out the document as we were working on it, we were telling people “this is what we’ve got so far, what can you do to help us make it better?”

Crowdsourced Improvements

And make it better they did. As of this writing, over 550 issues have been filed against the document, and many of them have led to substantive improvements to the final results. Some people have offered direct wording changes, others raised new concerns we hadn’t considered directly, and still others helped us clarify text that was otherwise confusing.

As we addressed issues, we could link the issue reported to the exact text change that addressed it. Anybody coming along and browsing through the history would be able to see these links as well. Found a typo here? Fixed it there! This wording wasn’t clear? We rewrote that sentence!

This process was continual as well. Sometimes we’d decide to address an issue, but our solution didn’t quite do what people were looking for. The submitter would have a chance to see what we did and comment on it, suggesting a way to improve it. Even if we closed the issue on GitHub, the conversation could continue around it. Issues could be re-opened, or even re-reported as a new issue all together.

For the most part, we also treated changes from editing team as separate contributions. In other words, even though everybody on the editing team could write directly to the repository, we instead packaged our changes up as independent pull requests and submitted them. While we were working on this project, GitHub released a new feature called reviews that allowed the team to have an explicit review and notification of all important incoming changes. In practice, one author would create a set of changes and submit them as a pull request. Another author would then review and tag the request, assigning anyone on the larger team that might have a particular stake in it. The pull request could not be incorporated until these assigned parties had signed off on the changes. Additionally, anybody on the team could review and comment on the changes as they went in, sometimes triggering further changes. Finally, once everyone was satisfied, the pull request would be approved and merged into the document.

Noted

In spite of the best of intentions, not every submission to the process was helpful. Some people wanted the document to cater to their specific use case in a very narrow fashion. Some people wanted the guidelines to be something else entirely. A number of companies wanted us to ensconce their product as an approved solution. Some people even raised issues and made comments just to prove that they were able to make smart comments on important works — a strange but remarkably common tactic of career development in many companies.

However, we always read and considered the comments that we didn’t incorporate directly. We read through everything that came in and decided as a team whether or not it was something that would help us. If we decided that it wasn’t something that would help the document, that decision was made with specific reasons in mind that the team would write down. Since the comments all came in through a public issue tracker, we could communicate the reasoning for our decision in a record where everyone — the original commenters as well as anyone else who might bring up a similar issue in the future — could see.

In the traditional government process, all comments would be collected through official channels and gone over in a big meeting with everybody, then official answers would be written, collated, and eventually released back to the public. The open source model thrives on more direct interaction, as issues can be found and filed at anytime by anybody. This was one area where we ran into the government’s desire to remain professional in its interactions with the public. If a terrible idea came in, it wouldn’t do for one of the authors to simply comment “That’s a terrible idea, no way” and close the issue — no matter how much we wanted to on several occasions. Instead, the editing team built in a process for dealing with all incoming comments, especially ones that were to be rejected from the document, based on what large scale open source development teams already do. To accommodate these two worlds, we needed a way for our team to coordinate.

Back-Channel Chatter

While this process was much more transparent than the usual government method, full transparency is an illusion with any project, government or not. There are always back-channel communications between the main contributors of a project outside the public view. These channels are very important for fast communication between team members, half-baked ideas, debate, and even just blowing off steam about the project and life. Email served as a major channel for vehicle for our team, just as it does among many open source projects, and the team was quick to pull people in and out of the relevant threads. We also held phone calls for discussion and even a few traditional get-all-the-nerds-in-a-back-room in-person edition sessions.

Notably, we didn’t take this approach with every issue, just the potentially contentious or confusing ones. Team members were free to comment and act on issues as they came in, with the idea that an individual team member’s statements wouldn’t necessarily reflect the final decision made by the team but provide input into it. While nobody wants to see all the messy, internal bickering of a development team, it’s the sign of a healthy and vibrant team dynamic that some of that interaction was still shown out in the world of GitHub.

Wait, you might be thinking, isn’t this all of the same back-room chatter that made the old process opaque? To avoid those same pitfalls, it was important that the ideas and information built in the back-channel not stay there. Once we had an idea of which direction we wanted to go in, we would publicly comment on the issue, bringing things to light much faster and more transparently than was possible before. Once the results of these discussions were made available publicly on GitHub, they were open for further discussion. Where the legacy development process consisted almost entirely of back-channel discussion, the new process makes use of back channels where they’re efficient and moves as quickly to transparent and public interaction as possible. Traditionally, government agencies and other large organizations are hesitant to show any such interactions, but I believe our team skated a delicate balance with this project to great success.

We had another tool at our disposal beyond email threads: in addition to the public repository where all the “real” work happened, the editors all had access to a private repository. This private team space allowed us to work on harder issues in a shared team space without putting all of the guts on display. This isn’t much different from how a normal git-based open source project works, either: contributors work on private forks and branches, and push to the public repository when it’s good enough for other team members to review and incorporate.

This repository had both a copy of the document itself as well as its own issue tracker for internal team use. This issue tracker turned out to be vital for bridging the legacy government process and newer open source style process: government stakeholders could write long documents and emails with all of their comments in one go, like they used to do. These comments were usually not meant for public consumption (since the legacy system happened mostly behind closed doors), but we didn’t want them to live simply in someone’s email inbox. What we did instead was import these comments to the team’s internal tracker and address them internally just like we did the public comments.

As time went on with the project, we used the private tracker less and less. Now that the document is in its final public comment period, all comments coming in through email are pushed directly to the public issue tracker. Or people could bypass the official email process and simply file an issue directly. Since we’d been using the public issue tracker for input all along, this has been embraced by much of the community at large.

Real Time Feedback

One of the best artifacts of our approach was the ability to see the document as it took shape. Using the NIST Pages site allowed us to create the new 800–63 documents in Markdown and have them published automatically to upon check in to a publicly available website. That way, as soon as an issue was resolved by a commit, it was visible publicly and in context. It was a kind of continuous integration for standards documents.

Overall, this process helped shrink the turnaround time for responses to issues from months to days (and occasionally, minutes). Instead of people sending in a comment, then months later seeing a new version of the document and wondering if their change was included in a satisfactory manner, people could file an issue as they found it. If someone from the editing team happened to catch the issue notification, and it was a quick and obvious fix, the change could be made and pushed to the repository without additional oversight or discussion. The person who filed the issue would be able to see both that the change that was made as well as the results of that change in context of the overall document.

But what if such changes were made incorrectly, or needed to be amended? No problem here either: since edits are cheap in the open source model, more changes could be pushed right on top of that and things would be fixed again. When changes are cheap, you can afford to iterate quickly to arrive at the right answer.

A Good Model

I want to be clear that NIST didn’t ask me to write this article and tout the success of the development process. I honestly believe that this has been a very good model for the project, and one I think this team would recommend going forward. The transparency and speed were fantastic advantages, and we were able to workaround the issues we ran into with simple project management practices like assigning issues to individual members and embracing the back channel when necessary. The inertia was certainly difficult to overcome, but I believe the positive results of the process speak for themselves and contend that they can be replicated elsewhere.

And finally, as of the time of this article’s publication you’ve still got time to get your comments in on the 800–63 document before it’s finalized. We’d love your feedback — no, really! Even at this late stage of the document’s development, we want people to keep reading and reviewing this and make this the best that it can be, for everyone. Here’s what we’ve done. How can you help us make it even better?