EFF: Gopher: When Adversarial Interoperability Burrowed Under the Gatekeepers' Fortresses

Gopher: When Adversarial Interoperability Burrowed Under the Gatekeepers' Fortresses

When Apple's App Store launched in 2008, it was widely hailed as a breakthrough in computing, a "curated experience" that would transform the chaos of locating and assessing software and replace it with a reliable one-stop-shop where every app would come pre-tested and with a trusted seal of approval.

But app stores are as old as consumer computing. From the moment that timeshare computers started to appear in research institutions, college campuses, and large corporations, the systems' administrators saw the "curation" of software choices as a key part of their duties.

And from the very start, users chafed against these limitations, and sought out ways to express their desire for technological self-determination. That self-determination was hard to express in the locked-down days of the mainframe, but as personal computers started to appear in university labs, and then in students' dorm rooms, there was a revolution.

The revolution began in 1991, in the very birthplace of the supercomputer: Minneapolis-St Paul. It was named after the University of Minnesota's (UMN) mascot, the gopher.

#

In the early 1990s, personal computers did not arrive in an "Internet-ready" state. Before students could connect their systems to UMN's network, they needed to install basic networking software that allowed their computers to communicate over TCP/IP, as well as dial-up software for protocols like PPP or SLIP. Some computers needed network cards or modems, and their associated drivers.

That was just for starters. Once the students' systems were ready to connect to the Internet, they still needed the basic tools for accessing distant servers: FTP software, a Usenet reader, a terminal emulator, and an email client, all crammed onto a floppy disk (or two). The task of marshalling, distributing, and supporting these tools fell to the university's Microcomputer Center.

For the university, the need to get students these basic tools was a blessing and a curse. It was labor-intensive work, sure, but it also meant that the Microcomputer Center could ensure that the students' newly Internet-ready computers were also configured to access the campus network and its resources, saving the Microcomputer Center thousands of hours talking students through the configuration process. It also meant that the Microcomputer Center could act like a mini App Store, starting students out on their online journeys with a curated collection of up-to-date, reliable tools.

That's where Gopher comes in. While the campus mainframe administrators had plans to selectively connect their systems to the Internet through specialized software, the Microcomputer Center had different ideas. Years before the public had heard of the World Wide Web, the Gopher team sought to fill the same niche, by connecting disparate systems to the Internet and making them available to those with little-to-no technical expertise—with or without the cooperation of the systems they were connecting.

Gopher used text-based menus to navigate "Gopherspace" (all the world's public Gopher servers). The Microcomputer Center team created Gopher clients that ran on Macs, DOS, and in Unix-based terminals. The original Gopher servers were a motley assortment of used Macintosh IIci systems running A/UX, Apple's flavor of Unix. The team also had access to several NeXT workstations.

Gopher had everything a student needed to navigate complex information spaces—except for information! The Gopher team cast about for resources that they could connect to their Gopher servers and thus make available to the entire network. They hit on Apple's Tech Info Library (AKA the "Knowledgebase"), a technical documentation database that came on CD-ROMs that could only be accessed by programmers who physically traveled to the lab where they were kept (or who paid for subscriptions to Apple's Applelink service). The Gopher team also answered student support questions and used Apple's Tech Info Library to do their jobs. Why not make it self-serve? They loaded the Knowledgebase into some NeXT workstations, and realized that they could use NeXT's built-in full-text indexing to make the complete set of documentation both accessible and searchable by anyone connected to a Gopher server.

Full-text indexing via NeXT workstations turned out to be one of Gopher's superpowers: soon, Gopherspace included fully indexed and searchable Usenet feeds, competing with WAIS to bring much-needed search to the Internet's largest, busiest social space. Gopher used the NeXT indexer to ingest massive quantities of recipes, creating the first-ever full-text search for cook-books.

But there were many other tricks up Gopher's sleeve. Many of the Internet's resources were available via text-based terminal connections that could only be accessed if you could remember their addresses and the quirky syntax required by each of these services. The Gopher team brought these resources into Gopherspace through the magic of terminal automation, whereby a terminal program could be programmed to login to a service, execute a command or series of commands, capture the output, format it, and put it in a distant user's Gopher client.

An early case for terminal automation was the Weather Underground service, which would give users who knew its address and syntax a realtime weather report for any place on Earth. The Gopher team created a Weather Underground gateway that used terminal automation to simplify weather retrieval and it quickly became so popular that it overwhelmed the Weather Underground's servers. However, the collegial spirit that prevailed online in those days meant that the Weather Underground's administrators could settle the matter by contacting the Gopher team. (Later on, the Weather Underground's administrators at the University of Michigan asked the Gopher team for usage data so they could include it in their application to renew the NSF grant that funded the project!)

Terminal automation allowed the Gopher team to rip the doors off of every information silo on campus and beyond. Libraries had put their card catalogs online, but few of the library vendors supported Z39.50, the standard for interconnecting these catalogs. Terminal scripting brought all the library catalogs into one searchable interface, and as Gopherspace proliferated to other campuses, it was possible for the first time to search collections of research libraries around the world.

The Gopher team consolidated many of these one-off hacks and bodges into a unified Gopher gateway server, with pre-assembled software ready to be customized and connected to the network by people running their own Gopher servers. These were popping up all over the world by this point, being run by children, universities, hobbyists, corporations, and even MTV's most tech-savvy VJ, Adam Curry. The team called their ethic "Internet duct-tape": a rough-and-ready way to connect all the 'Net's services together.

The expanded universe of Gopher hackers brought even more resources to Gopherspace. Soon, Gopher could be used to search Archie, a tool that indexed the world's public FTP servers, home to all the world's shareware, text-files, free and open source software, and digital miscellanea.

The FTP-Gopher gateway was a godsend for Internet newbies, who struggled with FTP's own obscure syntax. Some FTP servers were so overwhelmed by inbound connections from FTP-Gopher gateways that they scrapped their FTP servers and installed Gopher servers instead!

Soon, researchers at the University of Nevada at Reno had made their own search tool for Gopherspace, called Veronica (Very Easy Rodent-Oriented Net-wide Index to Computer Archives), which crawled every menu of every known Gopher server and allowed users to search all of Gopherspace. Veronica spawned a competing search tool from the University of Utah called Jughead (Jonzy's Universal Gopher Hierarchy Excavation And Display), later changed to "Jugtail" after a trademark scare.

The Gopher team made some tentative moves to commercialize their invention, asking for payments from commercial users of the Gopher server software (in practice, these payments were often waived, as they were for Adam Curry after he agreed to wear a Gopher t-shirt during an MTV broadcast).

#

The Gopher story is a perfect case history for Adversarial Interoperability. The pre-Gopher information landscape was dominated by companies, departments, and individuals who were disinterested in giving users control over their own computing experience and who viewed computing as something that took place in a shared lab space, not in your home or dorm room.

Rather than pursuing an argument with these self-appointed Lords of Computing, the Gopher team simply went around them, interconnecting to their services without asking for permission. They didn't take data they weren't supposed to have—but they did make it much easier for the services' nominal users to actually access them.

And since the Gopher team was working in the early years of the networked world, they had a steady supply of new services to integrate into Gopherspace—so many that other people came and did an Adversarial Interoperability number on them, building multiple, competing search tools to make users' lives easier still.

A modern Gopher project would face innumerable—and possibly insurmountable—legal hurdles. In the early 1990s, violations of terms of service led to friendly negotiations with the likes of Weather Underground. Try to do that today with a big interactive service and you might find yourself charged with multiple felonies. Big, proprietary databases often use "access controls" that can't be bypassed without risking criminal and civil charges, and that goes double for distributing a "gateway server" to make it easier for others to connect their own proprietary resources to an open network.

Today's tech giants—and both their apologists and their critics—insist that their dominance is the inevitable consequence of "network effects," and so nothing we do will recapture the diversity that once defined the Internet. But adversarial interoperability is judo for network effects.

Armed with tools that relied on adversarial interoperability, the Gopher team was able to turn the installed bases of users for each of the services they interconnected into an advantage, merging these constituencies in an ever-larger pool, until Gopher became the most exciting thing on the net, the killer app that every newscast about the exciting new digital realm featured.

Gopher was born before the rise of severe penalties for crossing invisible legal lines, and it meant that Gopher could experiment with new ways of making information available without worrying that a single misstep would result in their utter ruination.

For example, the Gopher team put added support for a protocol called websterd, which would allow remote users to reach into the team's NeXT workstations to query the "DigitalWebster" edition of the Ninth Webster's Dictionary that came bundled with the systems, so that anyone on the Internet could look up English-language dictionary definitions. This led to a complaint from the dictionary's copyright holders, and Webster was modified to access alternative dictionaries, and it served language-learners and students for years afterward.

Ironically, perhapsm, adversarial interoperability was also Gopher's downfall. Even as Gopher was rising to prominence, an English physicist at the CERN research institute in Switzerland named Tim Berners-Lee was inventing something called the "World Wide Web," and with it, the first browser. With browsers came URLs, identifiers that could be used to retrieve any document on any Web server in the world. The Gopher team quickly integrated URLs into Gopherspace, adding more flexibility and ease to their service.

But it wasn't enough. The Web proved to be more popular—and doubly so, once browser vendors began to build Gopher clients into the browser themselves, so you could link from any Web page to any Gopher resource and vice-versa. Adversarial Interoperability allowed the Web to treat Gopherspace as a conveniently organized pool of resources to absorb into Webspace, and it made the Web unstoppable.

The Gopher team tried many things to revitalize their service, including a valiant attempt to remake Gopherspace as a low-resolution VR environment, but the writing was on the wall. Gopher, it turned out, was an intermediate phase of our networked world, a booster rocket that the Web used to attain a higher orbit than Gopher could have ever reached on its own.

Today's Web giants want us to believe that they and they alone are suited to take us to wherever we end up next. Having used Adversarial Interoperability as a ladder to attain their rarefied heights, they now use laws to kick the ladder away and prevent the next Microcomputer Center or Tim Berners-Lee from doing to them what the Web did to Gopher, and what Gopher did to mainframes.

Legislation to stem the tide of Big Tech companies' abuses, and laws—such as a national consumer privacy bill, an interoperability bill, or a bill making firms liable for data-breaches—would go a long way toward improving the lives of the Internet users held hostage inside the companies' walled gardens.

But far more important than fixing Big Tech is fixing the Internet: restoring the kind of dynamism that made tech firms responsive to their users for fear of losing them, restoring the dynamic that let tinkerers, co-ops, and nonprofits give every person the power of technological self-determination.

(Many thanks to Gopher co-inventor Paul Lindner for invaluable assistance in the research and drafting of this article)