Archive for the ‘unix’ Category

How a Nonstandard OS Led to Unix Standards

October 11, 2019

From 1988 through 1992, I worked as a system administrator for the Environmental Defense Fund in Washington, DC. As a nonprofit, it was too expensive for them to provide each employee with a PC, and Windows and PC networking were primitive back then, so their offices around the US used Wyse 50 terminals connected to a single computer in the copier room. When I arrived, that computer was a 68020-based Charles River Data Systems Universe 68, running their UNOS operating system.

1988-04 044-13 CRDS Universe 68-35T edfdc-Dfine-close

 

Command-Line Utilities on UNOS

UNOS was sort of like Unix System V if you squinted, but it changed many of the details, so porting software to it ran into all kinds of incompatibilities. The command line utilities were maddeningly deficient and different from the 4.2 and 4.3BSD Unix I had used at college. Apparently CRDS was trying to make Unix more user-friendly or more like some other operating systems; their version of cp was named copy, dd was called debe (a pun on girls’ names?), etc. and the command line options were multiple characters long. Completely useless for running shell scripts written for Unix.

I started looking around for a way to replace them. I found the source code for half-finished versions of some utilities in the GNU source tree at MIT, where I had access as a volunteer thanks to my friendship with Mike Haertel, a former roommate who wrote GNU grep and diff. I finished them and wrote some more missing utilities, and thus started the GNU fileutils, textutils, and shellutils, which were later all rolled up into the coreutils. I also ported GNU Emacs to UNOS, including its tricky “undump” method of restoring a RAM image from a file for faster startup. EDF paid me to spend some of my time on this work, because they wanted more usable computer systems themselves.

So a good deal of GNU/Linux code was written to make up for the deficiencies of UNOS. After a few years, EDF switched to timeshared 486 computers running SCO Xenix, then SCO Unix, and we ran the GNU utilities on them, too, because they were better than the SCO versions. EDF finally went to networked Windows PCs for everyone, but they’re still using GNU/Linux for their web presence, at least, and using descendants of the utilities they funded.

Kermit on UNOS

Long before SSH, Kermit was a popular file transfer protocol which worked over both serial connections, like modems, and network connections, like TCP/IP. It was often paired with terminal emulators. It was implemented for nearly every type of computer made in the 1980s and 1990s, in many programming languages. Yes, it was named after the muppet, with permission.

EDF needed a good way to transfer files to and from its UNOS systems (besides UUCP), so I ported C-Kermit (the version for systems that use the C programming language) to UNOS. For that, I had to deal with many quirks in UNOS system calls.

Standards Because of UNOS

Mike O’Dell, Internet pioneer, related on an email list that the incompatibilities of UNOS led to the creation of the UNIFORUM association and the POSIX standards, so the US government wouldn’t have a sole source for Unix products that were interoperable. Thus, the obscure and quirky OS from CRDS contributed to creating both the official standards for Unix and what is now the most-used implementation of them.

Pubnix Access Systems: A Pioneering Web Hosting Service

October 11, 2019

The Context

It was the early 1990s.

The Internet was beginning its transition from supporting only U.S. government and university research to supporting ordinary businesses. The main applications on the Internet so far were email, Usenet news, and anonymous FTP. The World Wide Web was still a primitive research project. Gopher and WAIS were obscure and of little consequence.

Most homes had nothing faster than a 28.8kbps dial-up modem with which to access the Internet or other online services like bulletin boards or AOL.

Microsoft Windows was still only a fragile shell over MS-DOS. Linux was still an obscure and immature hobby project. Neither was suitable for running a server on. MacOS X was almost a decade away. But BSD Unix, a favorite operating system at universities, had recently become available on the PC platform.

The Beginning

A group of friends who had worked together for the University of Maryland, College Park as student programmers and system administrators were trickling out into the real world. Many of them reunited in Fairfax, Virginia at UUNET Technologies, the first commercial Internet service provider.

When Kurt Lidl left UMCP for UUNET (with a stop along the way at SURAnet), he had an inspiration. Most dialup services of the time, such as AOL, were oriented toward home computers and were limited to their own little world, with no connection to the larger, growing world of the Internet. Kurt wanted to create a public-access Unix system that people could dial up to, login to a shell account, and connect to the Internet and use email. He came up with the name “Pubnix” for this public Unix service. Although Pubnix started out as a separate venture, Kurt soon sold its assets to UUNET, and it became a small semi-autonomous division of UUNET with Kurt managing it.

Soon Josh Osborne and Dave MacKenzie joined Kurt at UUNET and helped him start developing Pubnix. They wanted it to be as automated as possible, so they created a system to generate configuration files with Perl programs using an SQL database, University Ingres (the only DBMS at the time that ran on BSD Unix). Hours were spent at whiteboards designing the database schema and provisioning systems for users and servers. Within a few months, the service was up and running on a few DEC Pentium PCs running BSDI BSD/OS in UUNET’s offices at Fairview Park in Fairfax. The BSDI company had started in the UUNET offices, so there was a close connection from the beginning.

There were jokes about creating public terminal rooms and combining them with a microbrewery to create a “brewpubnix.”

Additions

As they were designing Pubnix and building the technology to run it, the Internet was changing. The World Wide Web gained enough capabilities to be a useful tool for companies to use, not just researchers. Kurt realized that they needed to change directions, as shell account systems were becoming obsolete, and that the Pubnix shell accounts technology could form the foundation of a shared web hosting platform, one of the first in the world. So they adapted their configuration system to provision a version of the NCSA httpd customized to support virtual hosts, and Josh modified the kernel of the BSD/OS operating system to support IP rate limiting by virtual IP address, so each customer could have a certain amount of bandwidth. By late 1994, UUNET’s virtual web hosting product had customers. Unlike GeoCities, which also started in 1994, each customer’s web site had its own domain name. Around that time, Chris Ross came to UUNET from UMCP and joined the group.

As customers were added, the Pubnix platform grew in capabilities. University Ingres was replaced by a port of the XDB SQL database which Kurt commissioned. The NCSA httpd was replaced by the more flexible Apache web server. SSL connections using the Netscape web server and Real Media hosting on Linux were added.

Dave MacKenzie had the idea to add virtual FTP hosting, and in a few days he modified the WU FTPD to support virtual hosting, added database and Perl script support, and presented the completed product to UUNET’s marketing group to add to the product price list and start selling. In those days as a small pioneering company, it was that easy to come up with a new product. This may have been the world’s first virtual FTP hosting service. It gradually took over from UUNET’s original scheme of selling customers a subdirectory of ftp.uu.net. Now they could have their own FTP host name.

UUNET started a Windows NT based web hosting service as well, which was independent of the Pubnix effort and seemed to have learned none of the lessons from it. Provisioning and monitoring were done manually, as NT was hard to configure programmatically, so the staff to customer ratio for the NT service was much higher. The independence from being an independent division of UUNET also proved to have a downside, in that the rest of the company began to marginalize and ignore the Pubnix product line, even though it was probably the most profitable product line the company had.

Multiplication

More people joined the Pubnix group, including Andy Crerar, Russell Street, Joey McDonald, Li Glover, and Peter Davis. Eventually, as UUNET grew, Kurt left the Pubnix group to start an R&D team. It was also based on the BSD/OS platform, and became a technology supplier to the Pubnix group in a symbiotic relationship, as the R&D systems were configured by the same SQL database. New technologies were incorporated as they became available, including Kerberos v5, Apache Stronghold for SSL,  ProFTPd, and Postgres. The service was running hundreds of web sites on a mixture of dedicated and shared servers.

UUNET was bought by MFS (Metropolitan Fiber Systems), who were soon bought by WorldCom, who then also bought MCI, and Pubnix became a tiny part of a huge conglomerate. Along the way, the former AOL web hosting division was acquired, and the Pubnix staff was moved into their offices in Reston, Virginia. The AOL web hosting ran on customized Solaris servers with little centralized or automated management. With input from Dave MacKenzie, they developed an Oracle database called FARMS to drive configuring their systems.

The End

As MCI WorldCom was going bankrupt in a financial scandal, most of the Pubnix and R&D staff left or volunteered to be laid off. At the end of 2002, the product line was sunsetted. Kurt Lidl tried to buy back the BSD/OS hosting business from them, but got stonewalled by MCI WorldCom’s legal department until it was too late.

For more context, see this history of UUNET’s product lines, which mentions Unix-based web hosting.

From Novice to Master, and Back Again

January 14, 2013

In 1985, I was a freshman at St. Olaf College in Minnesota. The college had a VAX 11/780 running 4.2BSD and a PDP-11/70 running v7 with some Berkeley and local code hacked in. It was my first experience with multi-user systems other than dialing into an MS-DOS BBS or two.

The college’s Academic Computing Center had printouts of the 4.2BSD manuals, plus some home-grown documentation, available for sale so students could learn how to use UNIX. One week I sat in the Science Center terminal room and started going through the alphabetical list of the commands available on the VAX, trying each one and reading its man page to learn what it did.

Eventually I got to “su”. “Become the super-user”? What’s that? Does it involve wearing a cape? Sounds interesting, so I tried it. To my disappointment, it just asked for a password, and wouldn’t do anything.

Shortly thereafter, someone came running into the room and asked, “Are you David MacKenzie? Did you just run ‘su’?” “Yeah… what does it do?” “Uh, don’t do that.” My failed “su” attempt had been logged on the system console and one of the sysadmins was worried about an attempted breakin.

Within a year, I did have root access on the VAX, as I learned enough to be hired as a student system programmer. I contributed to upgrading the machine to 4.3BSD when that was released.

Recently I was working on a CentOS Linux virtual machine and needed to look up the command-line options to “su”. I had worked for the past several years mostly on Macs where “sudo” is preferred, so my “su” skills were rusty. I ran “man su” and got the information I needed. Then at the bottom of the screen I sheepishly read “Written by David MacKenzie.”

In the 1990s, while filling in gaps in the GNU toolset, I wrote the GNU “su”, and I had forgotten about it. It’s still what Red Hat and other distributions are shipping.

At least I know what it does now.