Implementing A Complex Internetwork: A Case Study

Implementing A Complex Internetwork: A Case Study
Roy A. Standing and Anton Dahbura, Jr.*
National Library of Medicine
Bethesda, MD
Abstract
This paper concerns the implementation of a complex
internetwork in late 1989 through early 1990. The purpose is to discuss the methods and approaches employed.
The context is that as technology improves and computer
science advances, the same problems are re-solved over and
over for slightly different environments. The perspective
is that the methods and approaches will remain viable
while the hardware and software changes.
Introduction
The problem to be solved was to develop and provide
network services to a group of library personnel, contractors and consultants located throughout the National
Library of Medicine (NLM) and at multiple remote sites.
The solution had to integrate with the existing network,
support multiple vendors' hardware, and provide maximal
reliability with minimal maintenance. The workstations
were Apple Macintoshes, IBM PCs and compatibles.
The complex intemetwork we developed is represented
in Figure 1 with the preexisting portion shaded. While all
the information displayed is accurate, six networks have
been reduced to stubs and only a portion of the attached
equipment is represented.
Definitions [2,12]
Terminology
AppleTalk: A set of LAN communication protocols
defined by Apple. There are two types: Phase I and II.
Bridge: A device linking two network segments using
the same protocol into one larger segment. It transfers
non-local traffic from each segment. NLM uses DEC
LAN Bridges to join Ethernet segments and Shiva
NetBridges to join LocalTalk segments.
Broadband: A broad frequency carrier system. The
frequency band is subdivided into audiovideo and data
channels. NLM uses 1Mbps data channels.
Ethernet: A set of LAN communication protocols
defined by Xerox and standardized by the IEEE (802.3).
Ethernet uses a transmission speed of 10Mbps (ten
million bits per second). There are two media standards:
Ethernet (Thick Ethernet) and Thinnet (Thin Ethernet).
Gateway: A router linking two networks using different
protocols. NLM uses Shiva EtherGate, Kinetics
Fastpath 4, Cayman Gatorbox, Apple Internet Router
(software), and Retix gateways.
Hops: Distance on an internetwork is measured in hops,
the number of routers network transmissions must
l
Mr. Dahbura is now with Apple Computer, Inc.
0195-4210/91/$5.00 © 1992 AMIA, Inc.
545
travel through to reach their destination. Zero hops
means originator and recipient are on the same network.
Internetwork: A collection of individual networks linked
by routers/gateways.
LocalTalk: AppleTalk networks implemented using
shielded twisted pair wiring. LocalTalk uses a
transmission speed of 230Kbps. NLM uses LocalTalk.
Network (a.k.a. LAN): A group of computers and
devices that intercommunicate over the same medium.
Repeater: A device which extends the length of a
network segment and transfers all traffic. NLM uses
DESPRs (DEC Ethernet Single-Port Repeater) to join
Ethernet to Thinnet.
Router: A device linking two networks using the same
protocol to form an internetwork. Routers maintain
tables describing the internetwork, calculate the shortest
path (in routers) for non-local traffic to take to its
destination, and route traffic to the destination network.
Routers isolate and define network boundaries. NLM
uses Cisco routers to link Ethernet networks.
112 Router: A pair link two remote networks, each one
constitutes one-half of the router. NLM uses Shiva
TeleBridges to link remote LocalTalk networks.
Zone: A zone is a named group of networks. NLM has
assigned a one-to-one correspondence between physical
network and zone.
Environments [3,11]
NLM is a two building facility located on the National
Institutes of Health campus in Bethesda, Maryland.
Broadband connections have been installed in every office
and room of both buildings. Ethernet has been installed
throughout most of the second building. Internet access is
available through the Ethernet. At the time this project
began, the broadband system provided data channels supporting Sytek and PC Net LANs.
Our contractor's off-site office is a remote environment
requiring both Ethernet and LocalTalk LANs.
A consultant or home-based system is our typical single
protocol remote environment.
Planning
Design Philosophy
We began by establishing three basic tenets to guide our
efforts: transparency, seamlessness and low maintenance.
Transparency means the user's interface to network
services does not change regardless of where or how the
user is connected to the network. Hardware and software
specifics may preclude identity across vendor's platforms;
ir1-
-----------
'r'':
Legend
G
Apple File Server
M
El IEJ
Apple Macintosh
B
IBM PC
Rc')'u''"t'e'
.............
..........
......
Printer
is
...............
......
....
.1 ......
.....
.........
Figure 1: AppleTalk Network Layout at NLM (Simplified,
the acid test is whether the user interface to all services is
identical from any workstation of the same make anywhere in the internetwork.
Transparency had three primary implications for us.
First, the network administrator must take all responsibility for managing the network complexity. Second, all
users will learn and use the same procedures, and have the
same perception and understanding of the network. This
minimizes training, promotes user confidence, encourages
collegial support, and enables users to physically relocate
without loss of network accessibility. Thirdly, ongoing
network user support requirements are reduced due to the
elimination of special cases.
Seamlessness means the user's interface to network
services is integrated into the methods normally used to
control the user's workstation. If a service is equivalent,
the same control method is used. New functions use
natural extensions to existing control methods. Novel
controls may improve on local workstation control
methods, but should be provided in additional to seamless
controls. The acid test of seamlessness is whether the
interfaces use natural extensions of existing methods or
require novel user skills, knowledge and abilities.
Low Maintenande means hardware and software are
self-configuring, reliable, easy to use and administer from
anywhere in the internetwork. Technically networks may
have to be complicated, but administratively they don't.
546
represents original internetwork)
Where early operating systems provided complex control
to highly trained operators, later operating systems
embedded automatic control mechanisms. Similarly,
while many network administration systems provide
complex control to highly trained administrators, some
have already taken the latter approach. They function like
'black boxes', creating unique network addresses, connecting gracefully to the network, and managing themselves.
'Black box' systems are designed for people, offices and
companies wanting network benefits without training or
administrative responsibilities. They minimize the staff
resources required for ongoing network administration.
Implementation Methodology
We based our methodology on three technical decisions:
gateway isolation, Ethernet as 'glue', and AppleShare.
Gateway Isolation Gateways are commercially available to link AppleTalk, broadband and Ethernet and are
discussed below. Since gateways only route non-local
outbound traffic to another router they completely isolate
the local network; the 'firewall' effect. This minimizes
internetwork traffic, optimizing total internetwork performance and prevents problems in one part of the internetwork from effecting any other part.
Gateways also ensure that internetwork access does not
deprive the user of local network facilities. This can be
illustrated using an Ethernet network joined to a LocalTalk network through a gateway plus a Macintosh work-
station with an Ethernet card and a LocalTalk connection.
While the workstation can be physically connected to both
networks simultaneously, the workstation can only use
one set of network software drivers at a time. If the gateway is off the workstation only has access to the services
of its local network, i.e., determined by the active set of
network software drivers. However, if the gateway is on
the workstation has full access to the services of both networks regardless of which network is local.
Ethernet as 'Glue' We decided to use Ethernet as the
'glue' to connect our LANs. Ethernet's 10Mbps transmission speed ensured that internetwork communications
would not be slower than local network communications
for most users and activities. While geographic barriers
sometimes require multiple hops, all LANs ultimately
link to the Ethernet backbone as shown in Figure 1.
AppleShare AppleShare was designed to make powerful network facilities simple to use and administer. It integrated transparency, seamlessness, and low maintenance.
The project PCs use AppleShare PC version 2.0.1. An
initial disadvantage was the need to run AppleTalk Phase
II. Prior to the integration of PCs we had been a strict
Phase I environment. A complete transition to AppleTalk
Phase II could not be implemented due to Phase I only
services running elsewhere on the internetwork. Our
intermediate solution was to use the Apple Internet Router
software running the Phase II upgrade utility as a background process on a file server. This allowed our intemetwork to handle both Phase I and Phase II.
Installation [4,6,9]
The internetwork was installed and tested network by
network with only minor adjustments to the original plan.
Pivotal to success was the ability of the network devices
to perform different functions according to traffic needs.
Naming Conventions
We had to establish rules for naming zones and devices.
Our zone names have two parts; one ascribes organizational ownership and the other physical location. For example, zone MMS-4th means that the network belongs to
the Medlars Management Services (MMS) section and the
network is physically located on the 4th floor. Similarly,
device naming combines device type and zone name.
Our implementation added sixteen new zones, ten types
of network devices from six vendors, plus workstations,
printers, and special equipment turning the preexisting
NLM internetwork into a complex internetwork.
Gateways
Ethernet and LocalTalk LocalTalk LAN interfaces
are built-in to every Macintosh and network product from
Apple Computer. As previously noted, all AppleTalk
LANs at NLM are LocalTalk implementations.
Where Ethernet was available we used Shiva EtherGates
to connect LocalTalk to Ethernet, e.g., the MMS-4th zone
in Figure 1. The EtherGate V1.50 provides full support
for AppleTalk Phase I, II and TCP/IP protocols. The
EtherGate autoconfigures and uses non-volatile RAM for
its programming. The EtherGate will automatically
reboot and resume operation after a power failure. These
features provided reliability and low maintenance.
LocalTalk and Broadband When only broadband was
available we used a specially modified Kinetics Fastpath
4** to connect LocalTalk to broadband, see the GM-RR
zone in Figure 1. The Cactus Computer modified Fastpath actually contains a LocalTalk to Ethernet gateway
which is plugged into an Ethernet to broadband modem!
The Fastpath is complicated to program and must be manually configured but has been robust and reliable.
Getting to broadband only answers half of the problem;
a gateway from broadband to Ethernet is still required.
Broadband and Ethernet The Retix gateway shares a
broadband communications channel and one Retix was
already in operational use. Adding an additional unit was
a simple, solution completing the pathway for a LocalTalk zone gateway to Thinnet, see the GM-Mezz zone in
Figure 1. The only device on this Thinnet segment is a
Shiva EtherGate supporting the GM-Mezz zone.
Linking Remote Networks Connecting remote sites
is primarily a tradeoff between cost and communications
speed. Modest usage projections led us to choose the
cheapest but slowest solution, standard modem-controlled
dial-up phone lines. We held the option to upgrade to a
faster solution if performance requirements could not be
met; see the OCCS-MR zone in Figure 1.
Shiva TeleBridges with high speed modems attached
were placed at NLM and the remote site. The Telebridge
acts as a half bridge, linking the two site's networks with
full network services. Ignoring network and modem
overhead and just comparing the transmission speeds of
each pathway, i.e., Ethernet 1OMbs, LocalTalk 23OKbs,
and modems 19.2 Kbs, says roughly that transmissions
across the modem link are almost 12 times slower than
transmissions on LocalTalk and over 520 times slower
than Ethernet! Thus, while the remote network connection is a vital capability, activities involving large
amounts of data are scheduled. Most internetwork activities involve small transmission bursts and the speed of the
internetwork link is not a limitation.
We tested three modems on the remote links. The US
Robotics Courier HST Dual Standard in HST mode uses
asynchronous channel speeds (9600/300) and had the poorest data transfer rates because of the channel turnaround
speed. The Telebit Trailblazer T1000 was very fast but
unreliable with other vendor's modems. The Microcom
QX3296c gave the most impressive transfer rates, reliable
connection with other modem brands and became our
modem of choice for linking to remote networks.
Linking Remote Workstations Individual work-
**The Fastpath is now a product of Shiva Corporation.
547
stations can link to the internetwork using Shiva Dial-in
software and a modem. Dial-in lines are provided using
Shiva TeleBridges at the NLM site, see the GM-B 1 zone
in Figure 1. The Telebridges support both network to
network connections and single workstation connections.
The workstations receive full network services subject to
the speed penalty of the modem link.
Network Services (Hardware)
File servers, printers and modems are distributed
throughout the zones within the internetwork for the
convenience of the users. In addition, placing the equipment in the same zone as its primary users localizes
network traffic and permits uninterrupted services if the
zone is temporarily severed from the internetwork.
File Servers are 'headless', meaning they have no keyboards or monitors. They do contain video boards to support local servicing when keyboards and monitors would
be attached. The common file servers are located in the
main computer room for physical space, security, and
electrical power considerations. They are high performance systems connected to Thinnet communicating at
Ethernet speed; see the Thinnet segment in the B1 computer room in Figure 1. The Thinnet links to Ethernet
through a DESPR and hence the main Ethernet backbone
through a DEC LAN bridge. The file servers are isolated
zero hops from the main Ethernet backbone and centrally
located to workstations throughout the internetwork.
Timbuktu, further described below, permits administration from any Macintosh workstation in the internetwork.
Printers and Modems for public use are located on the
GM-B 1 zone; see Figure 1. The printers include a color
Postscript laserprinter and a special label printer. These
devices are physically accessible to all users.
Network Services (Software)
Each product was selected after being evaluated for sufficiency, transparency, seamlessness, and low maintenance.
Electronic Mail [1,5,7] The email system had to be
able to work with SMTP email, IBM Profs, ccMail,
AppleLink, a special purpose bulletin board email system
and allow email to be delivered to, and sent from, the
email system of the user's choice.
QuickMail, from CE Software, a server-based mail
system, integrates realtime electronic teleconferencing,
supports file enclosures and voice mail. QuickMail
executes as a Desk Accessory on the Macintosh and a
Terminate and Stay Resident on the PC allowing email
facilities to be available regardless of the user's activity.
QuickMail permits remote administration via Timbuktu.
Gatormail, a Simple Mail Transfer Protocol (SMTP)
bridge, connects QuickMail to Unix, NeXT and the
worldwide Internet email. QM-Link connects to AppleLink during off peak hours. A Message Handling System
(MHS) bridge to link to ccMail proved unreliable, but has
since been resolved. An X400 bridge by Touch Com-
548
munications was later installed with bridges to ccMail
and Proffs. Special arrangements have been made to
develop a bridge for the bulletin board email system.
FAXGATE and MailMaker, in association with a fax
modem, permit files to be transmitted as facsimile
documents. Any user with a QuickMail account can
generate a facsimile transmission from any Macintosh
workstation using any application that supports printing.
QM Concierge allows users to reroute selected email,
e.g., while on business travel. It also supports pseudoaccounts, e.g., System Administrator, that map to real
users. The user gets a single point of contact and the
internetwork manager can reassign responsibilities freely.
Electronic Teleconferencing links two workstations
for direct keyboard communication. Part of QuickMail, it
permits online communication with other users; essential
when a telephone is inconvenient or unavailable.
Centralized File Storage Facilities AppleShare
uses Macintoshes as file servers. The file servers appear
as additional disk devices. The network interface is
transparent and the user interface seamless. Multiple
groups can share a file server for independent purposes
with complete separation, security and privacy.
File Interchange Files may be placed on a file server
in a commonly accessible account. Files may be enclosed
in QuickMail email messages. The most powerful method uses Timbuktu, further discussed below, to connect to
the target workstation and uses several transfer methods.
Remote Monitoring and Control [8,10,13] is
possible between any two Macintosh workstations. The
client's display is transmitted to a window in the master's
display. It is not necessary for the client to have a keyboard, monitor or even a video card! Timbuktu is used to
administer file servers, monitor public workstations,
demonstrate problems to remote support staff, and effect
repairs remotely. Timbuktu's security system manages
access privileges and helps avoid misuse.
The Internet Manager from Shiva Corporation is used to
administer all Shiva network products. The Internet Manager can be run from any workstation on the internetwork,
password protection guards against misuse.
InterPoll and Peek are diagnostic utilities that monitor
and report network traffic. They are used to isolate and
diagnose network problems.
Remote Backup and Restore for Macintosh workstations are provided across the network. NetStream software produces streaming or file oriented backups to 8mm
data cartridges. Data transfer speeds up to 1OMBps are
supported and the data cartridges store up to 2.2GB.
Software Development Management for the Macintosh came from the Macintosh Programmers Workshop
(MPW). IBM support came from Polytron Version Control System (PVCS).
Evaluation
The project was completed over eight months meeting
or exceeding virtually all of its goals. User acceptance has
been excellent and the number of users has doubled.
Reliability and performance have been outstanding with
one hardware failure, a manufacturing defect found at
installation. Options to tune performance have not been
necessary. The internetwork operates continuously as an
evolving facility, it is no longer special -- it's expected.
What Worked
Complex internetworks are realizable today using existing commercial products. Transparent and seamless to
users, these networks can be delivered and maintained at a
minimal cost in capital and support personnel.
What Didn't
There were numerous small failures; voice mail seemed
promising but without compression the files are too large
and cross the internetwork too slowly. Simultaneous
monitoring of file servers, mail centers, and workstations
is impractical. The data load exacts a high internetwork
performance penalty and is too slow to be useful.
What Needs Improvement
Apple originally envisioned LocalTalk networks to be
small with few zones. Zone selection works well in
small networks but is cumbersome with large internetworks. Some evolutionary improvements are obvious,
but some revolutionary new solutions are needed.
What It Cost
Developing an intemetwork involves materials, hardware, software and staff. Generally materials and hardware
costs decrease over time while software and staff costs
increase. The single most important cost is maintenance
since that is a continuing requirement. The hallmark of
our internetwork is its extremely low maintenance cost.
Materials costs for thick Ethernet were $1.10/ft plus
$250/node; Thinnet was $.60/ft plus $250/node; and
LocalTalk was $.50/ft plus $35/node. Miscellaneous
costs were under $500 dollars.
Hardware The gateways cost $1200 to $4000; bridges
about $350; repeaters about $200; and modems $800.
Software Maintenance software was bundled with the
hardware. Network user software costs were about
$32/user for QuickMail and $100/user for Timbuktu.
Staff costs were divided between planning, installation
and maintenance. Planning consumed about .5 FrE (Full
Time Equivalent). Installation consumed .3 FTE since we
used concealed cable runs, special wall plates, etc. Maintenance requires about .1 FTE annually.
Summary
In a perfect world you analyze your needs, design an optimal solution and implement it. In our case, a complex
environment already existed and we had to adapt available
solutions to it. The most important decisions were the
first ones establishing our design philosophy. Looking
549
back now, the planning decisions seem clear and obvious
but at the time, much of the hardware and software was
just entering the marketplace. The project was an enjoyable challenge, the internetwork is reliable and robust, and
the design philosophy continues to direct its evolution.
Trademarks
PostScript is a registered trademark of Adobe Corporation.
Apple, AppleShare, AppleTalk, Interpoll, LaserWriter, Macintosh, and MPW are registered trademarks of Apple Co., Inc.
LocalTalk is a trademark of Apple Computer, Inc.
Gatormail is a trademark of Cayman Systems.
QuickMail and QM-Link are trademarks of CE Software.
Timbuktu"m is a trademark of Farallon Computing.
Proffs is a trademark of IBM.
QM Concierge is a trademark of Information Electronics.
ccMail is a trademark of Lotus Corporation.
Microcom and QX3296c are trademarks of Microcom, Inc.
NetStream is a trademark of PCPC Corporation.
PVCS is a trademark of Polytron, Inc.
Dial-in, EtherGate, FastPath, Internet Manager, NetBridge,
NetModem, NetSerial, and TeleBridge are trademarks of Shiva
Corporation.
FAXGATE and Mailmaker are trademarks of Solutions, Inc.
Telebit and Trailblazer are trademarks of Telebit Corporation.
X400 is a trademark of Touch Communications.
Courier HST Dual Standard is a trademark of US Robotics, Inc.
References
The articles and product reviews have been published
since the end of the project. They discuss the hardware
and software we used and competitor's products.
1. Deborah Branscum. Conspicuous Consumer.
MacWorld March, 1991:63-83.
2. Daniel P. Dern. Multifunction Routers. SunExpert
March, 1991 2(3):78-82.
3. Jim Heid. How To/Getting Started with Networks.
MacWorld December, 1990:291-295.
4. David Kosiur. On the Ethernet Highway. MacWorld
March, 1990:132-137.
5. op cit. E-Mail: A postal Inspection. MacWorld
June, 1990:164-171.
6. op cit. EtherGate 1.45 (review). MacWorld August,
1990:196-197.
7. op cit. High-Speed Modems. MacWorld November,
1990:191-197.
8. op cit. Managing Networks. MacWorld February,
1991:152-159.
9. op cit. Going the Ethernet Route. MacWorld April,
1991:130-137.
10. Brita Meng. Timbuktu 3.1 (review). MacWorld
October, 1990:237-239.
11. op cit. Networking for the Novice. MacWorld
December, 1990:202-209.
12. Steven Sooby. Shiva EtherGate User Manual. Shiva
Corporation.
13. Ed Tittel. Timbuktu/Remote 1.0.1 (review).
MacWorld March, 1990:195.