If you have an Internet connection, eventually you are going to want to access USENET and its newsgroups. USENET is one of the most dynamic (and often controversial) aspects of the Internet. With access to the Internet, you can set up, access, and work with all kinds of newsgroups, but most Linux users will be interested in using USENET specifically. This chapter looks at the background of USENET and news services for UNIX in particular, as well as how the Linux news programs handle the news.
USENET is one of the most misunderstood aspects of the Internet. At the same time, it is one of the most popular and frequently used aspects of the Internet (with the possible exception of e-mail). To many users, especially those who don't use Internet's mail facilities, USENET is Internet, and vice-versa.
USENET was originally developed to facilitate discussion groups (called newsgroups in USENET jargon). A newsgroup lets any user with access to the system participate in a public dialogue with everyone else. By the end of 1995, USENET carried over 9,000 different newsgroups totaling well over 100M of information every day. USENET is supported in millions of networks in hundreds of countries and reaches hundreds of millions of users.
Despite what most people think, USENET is not a formal network or entity. Instead, it is a number of networked machines that exchange electronic mail (articles) tagged with predetermined subject headers for specific areas of interest (newsgroups). The articles are handled as electronic mail messages by most network machines; articles are processed as news information only by the applications called newsreaders that send and receive the messages.
Any machine that can attach itself to the Internet either directly, through a gateway, or through a forwarding service (such as an online service provider) can become part of USENET. All that is required to use USENET is the software that downloads and uploads the newsgroup mail and a reader package that lets users read and write articles.
The software that implements the passing of USENET messages over local area networks from one machine to another is the Network News Transfer Protocol (NNTP). Using NNTP, your Linux machine can interact with any other machines that handle the news. NNTP software is an integral part of most Linux versions, so you don't need to purchase or look for additional software. Indeed, many people establish Linux machines just to access Internet services like USENET, e-mail, and the World Wide Web.
USENET was developed out of a UNIX release known as UNIX V7, which implemented UUCP (UNIX to UNIX CoPy) for the first time. As UUCP became popular for communications between machines, it was expanded with program extensions and supplementary programs. USENET began at the University of North Carolina, where Steve Bellovin used shell scripts to write the first version of news software. UNC and Duke used this software to pass messages and commentary between the two universities. Interest in the news software spread when the UNC system was described at a Usenix conference in 1980. Steve Daniel was the first to implement the news software in the C programming language. This version eventually became the first general release of the news software, which was called release A.
To cope with the increasing volume of messages as new news sites were added to the expanding informal network, two University of California students, Mark Horton and Matt Glickman, rewrote the software and added new functionality. After a further revision of their release B, the news software was generally released in 1982 as version 2.1. From there, the Center for Seismic Studies' Rick Adams took over maintenance of the software in 1984, at which point it was up to release 2.10.2. One of Rick's first additions was the capability for moderated newsgroups, resulting in release 3.11 in 1986.
Since then, several contributors have added features to the software, the most important of which was a complete rewrite of the software undertaken in 1987 by the University of Toronto's Geoff Collyer and Henry Spencer. Their rewrite greatly increased the speed with which message mail could be processed and was generally released under the name C News (from Release C). Over the next few years, the basic news package went through some minor revisions but has remained true to Collyer and Spencer's version. Important changes were made to the way machines transferred news messages, and a daemon was added to process incoming and outgoing postings.
All the versions of news software developed to this point had used UUCP as the transport. To allow transfer of messages over a network, a protocol called the Network News Transfer Protocol (NNTP) was developed in 1986. NNTP-based software began to be refined, and a widely used version was implemented in software written by Brian Barber and Phil Lapsley, called nntpd. An alternative NNTP system that is widely available is INN (Internet News), which provides a complete news package (user interface and underlying software).
Apart from the underlying mechanics for transferring messages for newsgroups, developments also were continuing in the user interface area, where the newsreader exists. Newsreader software lets you read articles in newsgroups as they arrive. The original reader was called readnews, and it remains one of the most widely used newsreader packages, primarily because is it easy to use and is available on practically every UNIX system.
Several alternate newsreaders were developed, expanding on the features offered by readnews. Software such as rn (a more flexible version of readnews), trn (threaded readnews), and vnews (visual newsreader) are freely distributed now. All are character-based systems originally developed for UNIX and ported to many other operating systems. With the popularity of graphical user interfaces, newsreaders were also ported to these environments, resulting in software such as xrn (X Windows-based readnews). Most of the readnews variants share a basic command set, although each adds features that may appeal to some users.
Two types of software are involved in making a news service work on a Linux machine. The transport software (usually C News for UUCP connections or NNTP for TCP connections) gets the newsgroups to your machine. The newsreader then assembles and presents the articles to the user. Newsreaders are only involved in the actual user interface; they simply pass and receive news articles from the underlying software. For that reason, you don't need to look at the mechanics of a newsreader to understand how Linux processes news. The original news system relied completely on UUCP, so much of the news software was designed for UUCP and then modified later to accommodate alternate methods.
To transfer news from one machine to another, a technique called flooding is used. One machine calls another and transfers all the news articles. The machine that just received the news calls another and transfers the articles again. The news articles flow across the networks by moving from machine to machine instead of all the machines polling a single main news source. Each machine maintains a list of other sites it can contact to transfer mail. Each connection to another machine is called a newsfeed.
Each machine can generate new articles as the system's users interact with newsgroups. When new articles are created, the machine checks its list of newsfeeds and calls them to transfer the new mail. Because each article generated by a newsreader has a list of the machines that it has passed through (called the Path), the local machine knows whether the remote sites on its newsfeed list have already seen the article. As articles move from machine to machine, each machine adds its own identifier to the article's Path field, using the UUCP bang-style notation.
An entry in the Distribution field of the header may place a restriction on the machines that can be sent an article. For example, if you write an article that you want to stay within your local area network, you can specify this in the Distribution field of the message when you write it. Then when a newsfeed to a machine outside the local area network is created, the Distribution field prevents the article from being sent.
To help prevent duplicates of articles moving around USENET, each article has a unique identifying number called a message ID (which sits in the Message-Id field in the article header). The message ID is a combination of a unique number and the name of the machine that the article was originally posted on. Machines use these message ID numbers when a connection to a newsfeed is established. A history file on each system contains a list of all message ID numbers that the local system has. When the two machines communicate with each other, they can check the history file to find out whether the message should be sent. This process is part of a news transfer protocol called ihave/sendme.
With the ihave/sendme protocol, one machine sends a list of all the message ID numbers it currently has and waits for the other machine to identify the ones it wants. These numbers are transferred one at a time in response to sendme messages. Then the process can be reversed to update the other machine. This type of protocol works well, but it does involve a lot of overhead in the communications process. For that reason (coupled with the generally slow lines used by UUCP modem links), ihave/sendme protocols are not often used when a very large newsgroup transfer has to take place at regular intervals. You wouldn't want to use ihave/sendme to transfer 100M of articles every day, for example.
An alternative method used for large transfers is batching of articles. In this method, one machine sends everything it has to another machine. The receiving machine then performs a check of the newly arrived articles to see whether it already has them. By looking at the message ID number, the machine can discard duplicates. This method tends to be a faster for transferring, although it does have more processing overhead for the receiving machine when the machine deals with the newly arrived batch of articles.
For network-based news access, there are three ways to get articles from another machine. Using NNTP, your machine can download articles you want using a technique called pushing the news, which is similar to the ihave/sendme protocol. Your machine can also request specific newsgroups or articles from the remote based on the date of arrival, which is called pulling the news. Alternatively, you can interact on an article-by-article basis with the remote, never downloading the articles to your local machine. This process is called interactive newsreading, and it works only when you have a newsfeed you can log in to (which is common these days).
This chapter looked at the basics of USENET and news. This information provides a foundation for the next two chapters, which look at NNTP (for network-based access to news) and C News (for UUCP and network access to news). Entire books are written on the subject of USENET and the protocols it uses. Check out one of them if you need more information on this subject.
