Integrating Enterprise Information on a Global Scale:
Building a Large-Scale Information System for Today's Demanding Business Environment
Abstract
In today's challenging business environment, companies are encountering levels of growth and change that can quickly make their business information systems obsolete. These enterprises are meeting this challenge by implementing real-time transaction processing systems to reduce cycle time, cut operation costs, and improve responsiveness to corporate users, customers, and vendors alike.This paper highlights the key factors that an organization must address when building a globally integrated business system. It also describes how 3Com Corporation implemented its own state-of-the-art enterprise resource planning (ERP) system using SAP R/3 business application solutions, the Informix OnLine Dynamic Server relational database, and 3Com's own networking systems and products. In addition, the paper touches on the advantages that the partnership between 3Com and Informix offers networking customers.
Thriving in a Volatile Business Climate
Many industries today are characterized by fierce competition, intense time-to-market pressure, and consolidation through mergers and acquisitions. Companies depend on their transaction processing systems to maintain a competitive edge, and to sustain business models that must constantly adapt to changing market conditions.An effective business system is a marriage between information system (IS) and business process. When laying out a strategic plan and defining the underlying architecture for a new corporate transaction processing system, most organizations seek to achieve these primary objectives:
- Make business operations more responsive to customers and the needs of the enterprise by integrating logistical data into one global system
- Implement real-time transaction processing to provide online information access anytime, anywhere
- Develop processes that reduce cycle time for order fulfillment and minimize inventory and distribution costs
- Design a future-proof solution that can scale easily to accommodate growth
- Leverage technology to reduce long-term IS costs
Information technology has progressed dramatically in the last few years. It is now possible to integrate diverse functions more fully using software that offers better price/performance as well as plug-and-play modularity. The latest technology also makes it possible to combine data in a scalable, high-performance relational database, and to transmit the information globally over a reliable high-speed network.
A business information system may be divided into three major elements: the ERP applications, the database, and the network configuration. Each of these elements is equally important, and all of them must mesh smoothly to ensure a successful implementation.
The ERP Implementation
Business system applications — ranging from accounting and sales support to manufacturing and distribution — that have served large organizations well in the past are often not sufficiently integrated to support today's real-time global business operations and centralized planning and decision making.
Moreover, the ongoing customization required to bring legacy applications in line with changing business needs makes them labor-intensive and costly to maintain. When companies merge, diverse and incompatible legacy applications further exacerbate the integration problem.
Even if a company reengineers its business processes to optimize efficiency before implementing a new business information system, these various business operations must be mapped to specific ERP applications. It is often desirable to contract with an experienced systems integrator to ensure that the project goes smoothly, and that any business processes that are not supported by ERP applications are integrated into the overall system by means of custom programs or third-party solutions.
Since R/3 is the ERP system of choice for many enterprises today, it serves as a reference point for the following discussion. There are many benefits to implementing a fully integrated ERP system such as R/3. Among the chief ones are:
- Real-time data access that provides the information necessary for corporate finance to perform accurate capital budgeting, financial planning, cash management, and treasury management
- Marketing and sales support that allows the company to effectively implement marketing programs and coordinate pricing and promotions worldwide
- Links between far-flung supply centers that permit up-to-the-minute production planning and inventory control of materials and finished products, reducing time to market and cutting costs
- Streamlined distribution procedures to move product quickly into multiple distribution channels
- More efficient credit and collection procedures to serve customers better and improve cash flow
- Improvements to the company's management reporting capabilities, including the elimination of hardcopy reports
- Year-2000 compliance
As one example of how a state-of-the-art business system can improve efficiency and reduce costs, consider the order-fulfillment cycle. Real-time access to manufacturing and logistical information allows order-entry clerks to check credit and stock availability, authorizing product shipments and pricing at order time. When fully implemented, this process allows product in stock to be shipped in a single day.
Because the ERP system is an amalgam of information technology and business operations, representatives from both the IS group and major business units need to be involved in the implementation from the outset. Furthermore, it is critical that network managers be part of this effort, since poor connectivity and insufficient bandwidth can severely impact quality of service. Just as the database resources for the ERP installation must be carefully optimized for performance and scalability, so should the network.
Whichever type of business system is selected, the company should institute an in-depth training program early in order to make sure productivity stays at a constant level throughout the process. Ideally, both the ERP vendor and the systems integrator can help with onsite instruction and procedures documentation.
The Database
The open R/3 environment allows customers the flexibility to select the database that best meets their specific needs. An organization implementing the system can choose from a variety of databases, taking into account the characteristics and features that will operate within the database itself. Many of the criteria for evaluating databases are similar to those used for evaluating business systems in general: performance, scalability, and manageability. The performance dimension is particularly important.
The database should be robust and scalable to take full advantage of the R/3 three-tier client/server platform. The database selected should be flexible enough to accommodate future enhancements. And like the R/3 applications, it should also be year-2000 compliant.
The SAP R/3 architecture can be based on the three-tier client server model: the presentation layer, the application layer, and the database layer. In this architecture, the users request and send information from the presentation servers to the application servers, which in turn submit requests to the database server. A key element of R/3 performance is the ability of the database server to internally "parallelize" the requests from the application servers. This parallel operation must be transparent to the R/3 system.
Because R/3 is integrated — that is, all business processes are handled on a single SAP suite or SAP system (which includes all the applications servers and the database servers) — poor database performance can cause the system as a whole to suffer. The basic R/3 three-tier architecture is depicted in Figure 1. As the diagram shows, applications servers may be added in tier 2 to improve R/3 application performance.
Since the database server in this implementation is a single device, performance and scalability on that server are critical.
Database performance. One standard method of evaluation is the benchmark, a controlled series of tests that determine the number of users that a specific hardware platform can accommodate on a database running under R/3. This is one of the methods 3Com used in its database evaluation. Just as significant as the benchmark, however, is the ability of the database to increase performance in the future to keep pace with increases in hardware performance.
In an online transaction processing (OLTP) environment, taking full advantage of the server's processor speed is paramount. This is particularly true of the R/3 environment, where system optimization is accomplished mostly by database internals.
Database scalability. Scalability becomes important when performance declines due to lack of system resources, most commonly as the result of overall system growth. There are two basic kinds of database scalability: horizontal and vertical. Horizontal scalability involves the capability of several servers to interoperate transparently and share the workload. Vertical scalability refers to increasing or decreasing the overall power of the system by adding new components.
Within the client/server architecture of R/3, it is possible to achieve horizontal scalability by adding new application servers to obtain more power on the application side. Due to the fact that R/3 supports a centralized database, new hardware architectures such as Symmetric Multi-Processing (SMP) or Massively Parallel Systems (MPP) clusters and loosely coupled systems are required to achieve vertical scalability within the database.
The Network
Traditional networks were a collection of relatively autonomous internetworked local area networks (LANs) largely defined by geographical location. A tacit assumption in such a configuration was that 80 percent of the data traffic would remain within each workgroup LAN. In the past, internetworking among the workgroups primarily supported e-mail, file transfer, and remote log-on to host applications.
But with the proliferation of wide area and any-to-any intranet traffic in today's networks, the 80/20 formula no longer holds true. Client/server, enterprise-wide computing architectures are an improvement on traditional LAN-based networking when it comes to matching the way today's companies actually do business. But workgroup and project-oriented information sharing make it more difficult to solve performance problems occurring "behind" a backbone bridge or router. In the R/3 environment, workgroup servers require their own support for adding disk space and performing maintenance and troubleshooting.
Adherence to the following general principles when designing the business network will go a long way toward ensuring a high level of performance and resiliency:
Prior to implementing the new business system, network management should create a baseline of the existing network to fully understand the network's traffic types, levels, and patterns. Managers should stress-test the network to determine whether the wide area network (WAN) links will perform well enough to handle the additional traffic generated by a larger constellation of users and real-time data transfers. Some of today's network devices feature built-in monitoring software and other software tools that can help with this analysis.
When designing networks, companies are increasingly thinking in terms of policy-based networking. This involves building intelligence into the network so that it can automatically handle applications according to preset policies that reflect the company's specific business needs. Rather than concentrating all the network's intelligence in core devices such as routers, which add latency and are costly to administer, policy-based networking makes use of end-to-end software intelligence that extends all the way to the desktops, where the applications originate.
Finally, a global network needs a global network management system, one that allows managers not only to monitor devices, but also to track business applications from source to destination. In addition to providing views of device status, alarms, troubleshooting tools, and other key management features, the management software should be able to make use of standards-based monitoring capabilities such as Remote Monitoring (RMON) and RMON2 embedded in devices across the network.
Business applications: SAP R/3
Hardware platform: Hewlett-Packard HP/UX
Database: Informix OnLine Dynamic Server
Network: 3Com NETBuilder II® routers, AccessBuilder® remote access devices, CoreBuilder™ switches, SuperStack® II hubs and switches, EtherLink® and TokenLink® family network interface cards, Transcend® Enterprise Manager and other TranscendWare™ software
Users: approximately 1200 by July 1997
Functional Areas: Order entry, order fulfillment, accounting, materials management, manufacturing, distribution, marketing, sales, and decision support
Company Background
3Com also began an aggressive program of acquiring companies and technologies to enhance its product portfolio with the goal of moving toward an integrated systems approach to networking. 3Com's complex, multichannel distribution system, which handled orders for as few as 10 or as many as 20,000 products using procedures ranging from sophisticated electronic data interchange (EDI) to paper-based forms, presented additional challenges.
By 1993 it was clear that 3Com's existing electronic business system could no longer keep pace with the increasing demands being placed upon it. The system was made up of a number of individual ASK/ManMan applications managed locally. These systems were not integrated at a level that allowed them to support global business operations in real time, or to give decision makers timely information. They were also expensive to maintain and upgrade.
A Phased Implementation
Because of the scale of the project, 3Com invited the participation of all the business units that would eventually be using the system. Price Waterhouse and SAP AG provided training onsite to smooth the transition from the legacy systems to SAP R/3, and to prevent conflicts and mishaps that might affect business operations.
The Database: Informix OnLine Dynamic Server
Processor Performance. Informix ODS optimizes processor performance by dividing tasks into smaller, more manageable increments and feeding them to the processor in the most efficient order. This function is accomplished independent of the R/3 application, occurring after R/3 hands off the operation to the database. By breaking tasks down into more manageable parts and executing these tasks in parallel, Informix ODS cuts down on processing cycles and idle time. This in turn increases throughput in the processor.
In the Informix environment, the database manages the distribution of tasks in a single processor through the use of virtual processors that carry out specific jobs within the database according to their designated class, or task group. The various tasks in the database — for example, writes, sorts, joins, and scans — are divided up and acted upon by lightweight context-switching programs, or threads, in the virtual processor. The resulting performance gain can be augmented further by adding more virtual processors and distributing the threads across them. Physical processors may be added for another level of parallelism to further increase performance. This is illustrated in Figure 2. Also noteworthy is the fact that virtual processors may be bound to a physical processor or may migrate from physical processor to physical processor.
In 3Com's business system, when tasks are submitted by the application servers to the database server the Informix database breaks them into smaller subtasks and coordinates execution of these tasks across all the processors in the Hewlett-Packard HP/UX Symmetric Multi-Processing hardware environment. These processes are managed by the Informix database rather than by the HP/UX computer operating system, removing overhead and further increasing performance.
Memory Management. Database operations in an R/3 solution can be extremely memory intensive. Memory utilization becomes an issue as the R/3 application competes for resources with the server operating system.
Informix ODS optimizes memory management through the Dynamic Shared Memory feature. Dynamic Shared Memory is a pool of memory that can be used and reused by the Informix threads. Pages of data that are pulled from disks into buffers that reside in a pool can be accessed again by subsequent threads needing data on that page. Informix ODS can automatically increase these memory pools as the need arises. Conversely, as threads finish with their use of memory, the memory is returned to the pool for reuse by the database. This memory can also be reclaimed by the operating system.
It is important to note that Dynamic Shared Memory allocation occurs dynamically while the system is up and running, making it ideal for 3Com's around-the-clock global information needs.
Disk Management. Yet another area of database performance optimization is disk management. Informix uses its own unique disk management system to optimize disk access for database usage and decrease the overhead of the UNIX-based operating system.
The slowest component of any database operation is generally input/output (I/O). Poor I/O management can cause stacking bottlenecks that prevent an R/3 solution from achieving maximum performance. Informix ODS uses a combination of parallelism and asynchronous I/O (or kernel asynchronous I/O) to increase processing throughput.
Parallelism in the Informix database also extends to disk access. In combined SAP and Informix implementations, tables can be divided across disks in such a way that the database engine "knows" where the data resides. This spreading of data across multiple spindles, called fragmentation, decreases seek time on a read from disk by allowing for simultaneous parallel reads across spindles, and by decreasing the amount of time it takes to find the data on an individual disk.
In large R/3 installations such as 3Com's, it is typical to have several extremely large tables. Partitioning these tables enables better performance, recoverability, archiving, and administration. As a table grows in size, Informix ODS can add more partitions or repartition the table dynamically in order to administer the table within a fixed time limit.
Mirroring. Mirroring is another method of increasing not only the security of critical tables in the database but also the throughput of the reads of the data in those tables. In 3Com's implementation the critical DBspaces are mirrored, allowing for simultaneous reads to those DBspaces and decreasing the time it takes to pull the pages of data into memory.
Scalability. Informix ODS has a scalable architecture that allows for performance increases proportionate to performance increases in the hardware as a result of adding more disk space, memory, and central processing unit (CPU) power. However, a system administrator can also reconfigure Informix ODS by adding virtual processes to alleviate many performance problems. In many cases, simply adding virtual processes in a CPU or I/O class is enough to increase the throughput. Equally important is that this can be done while the system is online.
When hardware limitations prevent the R/3 solution from performing optimally, sometimes the only recourse is to add processors to increase the throughput of the processes.
Informix's DSA core parallel architecture accomplishes this transparent internal parallelism by means of its multithreaded design plus a feature called Parallel Data Query (PDQ). After breaking a task into its basic components through specialized processes, the system can fan the threads across physical processors which again use the physical processors available. The resulting linear scalability enables an R/3 database server to utilize between 85 and 95 percent of each CPU added to a multiprocessor system.
Since PDQ is intended to harness the full power of all the CPUs on a single task, it is possible to see performance increase linearly with the number of CPUs on a system. Therefore, a four-way system could complete a query in one-fourth the time, an eight-way system in one-eighth the time, and so on. While the PDQ feature is most often used in batch mode processing and DSSA, the leverage it provides in these operations is significant. This scalability is illustrated in Figure 3.
Manageability. A primary factor in 3Com's decision to move forward with the business system project was the need to simplify system management by implementing a single, integrated business system solution. Such a solution provides the ability to manage database servers and applications from one console, located centrally or in a remote location. SAP AG, in collaboration with Informix, provides several management interfaces with the SAP Database Administration (SAPDBA) and Computing Center Management System (CCMS) tools.
SAPDBA allows comprehensive administration of the database, including disk space management, reorgs, and starting and stopping database functions. In addition to giving a clear view of the disk pool, SAPDBA issues warnings concerning space shortages, provides space checks prior to specific database operations, manages logs for database operations, and provides tools for other monitoring and management operations.
Through CCMS, all backups can be completed using a single interface. CCMS also provides a standardized window into a multitude of logs and alerts. Along with the CCMS feature's ability to complete these tasks is the ability to calendarize them. This enables administrators to review logs in order to check on performance and troubleshoot failures. Actions such as planning, scheduling, immediate execution, and resource allocation can all be completed by means of the calendar functions.
Year-2000 Compliance. Another issue driving the implementation of an integrated business system is year-2000 compliance. As the millennium nears, 3Com's widely dispersed systems had to be replaced with a solution able to handle the shift from 19xx to 20xx dates. R/3 and all Informix products are year-2000 compliant, solving the date problem in both present and future database activities, including such future implementations as data warehousing.
The Network: 3Com Systems and Products
Buildings on the main Santa Clara campus were connected using 100 Mbps Fiber Distributed Data Interface (FDDI) fiber links, with satellite offices linked using 56/64 Kbps lines. U.S. and European field offices employed small LANs installed at each office and connected to the rest of the network with Frame Relay (Figure 4).
Before implementing the new business system, 3Com created a baseline of the existing network. Using network analysis tools, the 3Com's Global Information System (GIS) group stress-tested the WAN to be sure it could handle the anticipated traffic. (The legacy system relied on once-a-day batch transfers.) GIS personnel checked utilization levels and measured peak periods. Findings included the facts that latency between international traffic and domestic traffic could vary by as much as 50 percent, and utilization peaked at no more than 40 percent.
The data traffic across the existing 3Com WAN was replicate in nature. That is, data was stored at the U.S. or U.K. site first, then replicated to the other site once a day. In this type of batch environment, response time and reliability is generally not an issue. In the event of a transmission problem, data could be resent without affecting business processes. While the existing 56 Kbps leased lines could handle the real-time R/3 traffic, unpredictability and lack of backup links presented a problem.
In addition, the existing network had not been designed for centralized management, and maintaining service levels across a network of this scope was at the time virtually unprecedented. 3Com's goal was to build in scalability, and to simplify the network architecture from end to end to provide a robust, manageable infrastructure for the R/3 solution.
The new network design achieved this goal using TCP/IP as the predominant protocol running on two 128 Kbps private virtual circuit (PVC) links, as shown in Figure 5. (These links have subsequently been replaced by a 512 Kbps Point-to-Point circuit, with a Frame Relay circuit providing resilience.) Each link is capable of carrying all the traffic in an emergency. Under normal operation, however, one link is designated a "clear" channel to ensure performance for the TCP/IP traffic, while the other link carries all the other data, including Novell IPX and AppleTalk traffic. In this way, the network enhances performance and also provides an automatic backup capability.
The 3Com network as retrofitted to support R/3 is shown in Figure 6. The advanced technologies described below were integrated into the design.
Link-state protocols. Link-state routing protocols, advanced inter-area routing protocols that are part of the TranscendWare™ software feature set provided by 3Com's NETBuilder II® routers, make decisions about the most efficient routes for all the traffic on the network except AppleTalk, which does not support link-state routing. The link-state routing protocol for TCP/IP is Open Shortest Path First (OSPF); for Novell IPX, the protocol is NetWare Link Services Protocol (NLSP).
Protocol reservation. Certain mission-critical applications may require a guaranteed percentage of WAN bandwidth. The 3Com NETBuilder® router's protocol reservation feature lets the network administrator guarantee that a portion of a WAN interface's bandwidth will be reserved for a specific protocol or application. Protocol reservation provides improved service for interactive and transaction-oriented network applications. A minimum percentage of a link's bandwidth is statically configured for each protocol that operates over the WAN link. For example, if 10 percent of the pipe is reserved for HTTP traffic, that bandwidth will be there—even if the pipe is otherwise full. If the HTTP traffic falls below 10 percent, the reserved bandwidth is made available for use by other protocols and applications.
Manageability. The original 3Com business network was managed from separate platforms using disparate, and in some cases nonstandard, network management systems from a variety of vendors. But the new enterprise network could not be effectively managed in that way. A new management architecture had to be implemented, one that was comprehensive, open, scalable, and homogeneous in terms of platforms and interfaces. And since a large part of the expense of a network is administrative costs, the management system also needed to consolidate management power at central sites to save time and labor.
3Com's TranscendWare software provides end-to-end intelligence that forms the foundation for a comprehensive management infrastructure. Software intelligence in 3Com network interface cards makes end systems active components of network management. Intelligence in 3Com's switches, hubs, routers, and probes collects data cross the network, analyzes it, and reports it. 3Com management applications act on this data to deliver end-to-end visibility and control of network and application performance. Managers can not only monitor devices, they can also track R/3 applications from source to destination.
Elements of 3Com end-to-end network intelligence include:
Since 3Com TranscendWare software is modular, 3Com's GIS group can add intelligence incrementally as necessary. New capabilities may be integrated into installed software by means of upgrades.
The Importance of Partnerships
Another key relationship in the implementation process was the one between the Informix and SAP organizations. To make sure 3Com received the most up-to-date releases in a timely fashion, and to ensure that these releases would port smoothly to 3Com's database, 3Com wanted the testing and porting processes in place to support a fast turnaround. A close relationship between the database vendor and SAP was crucial not only from the point of view of timely product information dissemination, but also with regard to tight product integration.
Informix currently has 14 engineers working side by side with SAP engineers at a development center in Walldorf, Germany, where SAP AG has its headquarters. This helps ensure fast turnaround for new releases and strong ties between the R&D sides of both organizations. Significant efforts have been made to keep Informix at the forefront of SAP technology. Informix has also added a testing environment in the U.S. for product development in conjunction with the SAP product suite.
In the area of customer support, Informix has recently staffed the SAP hotline with a regency support manager. This provides 3Com and other customers with level-two support at the hotline for expedient problem resolution or escalation. In addition, Informix has an escalation manager in Menlo Park, California, who works closely with the regency support manager at the hotline, the Informix technical support staff, and Informix's own Follow the Sun support.
For the future, 3Com is also looking at Informix's data warehousing and decision support capabilities, which can be implemented on Informix ODS and XPS, as well as on the new member of the Informix suite, IUS (Informix Universal Server). IUS is an extensible relational database that handles rich, complex data types through the combination of DSA power and scalability and the Informix DataBlade technology. This will allow for the management of complex data types as true objects, instead of as binary large objects. Plans are in the works to leverage this technology for use with SAP business application program interfaces.
Success Factors
Figure 1: Three-Tier R/3 Architecture
As enterprises increasingly rely on the ability of their data networks to provide support for sophisticated, bandwidth-intensive applications such as R/3, the network infrastructure can become overburdened with both data and management traffic. This not only affects response time and quality of service, it also makes the network less reliable and less responsive to business needs.
3Com's Business System in Brief
3Com's ERP Implementation: A Case Study
3Com joined with SAP AG and Informix Software, Inc. to build a state-of-the-art business system designed to meet 3Com's needs now and well into the next century. The installation is a good example of how a large organization can improve operations by establishing vendor partnerships that lead to a successful implementation of an integrated business system, one that is based on a sophisticated database structure and a high-performance, scalable network.
At the beginning of the 1990s, 3Com launched a strategic initiative aimed at growing sales and earnings by capitalizing on global manufacturing and marketing capabilities, and expanding market leadership into new product areas. The company opened a new manufacturing and design facility in the U.K. as well as a number of sales offices around the world, and forged a number of new partnerships.
To ensure a well-run ERP implementation, 3Com contracted with Price Waterhouse to provide system integration services that included a comprehensive implementation strategy. The project was divided into two phases. The first phase, completed in January 1997, focused on finance, sales, distribution, and some facets of materials management and production planning. The second phase, scheduled for completion in July 1998, involves manufacturing, cost accounting, accounts payable, and fixed assets. Both the database and network designs commenced at the beginning of the project so that each could be smoothly integrated with the R/3 environment when the time came.
Choosing the right database to run under R/3 is critical. 3Com chose Informix OnLine Dynamic Server (ODS) based on its performance, scalability, and manageability characteristics. 3Com determined that the Informix product matched SAP's three-tier client-server architecture well, and would also accommodate possible future enhancements such as data warehousing and parallel processing. With 3Com's recent acquisition of U.S. Robotics Corporation, database growth has been a major concern.
Figure 2: Virtual Processors Improve System Performance
Figure 3: DSA Architecture Increases System Scalability
At the beginning of the ERP implementation project, 3Com's corporate network consisted of four physical networks comprising one logical network. Network technologies included mostly 10 Mbps Ethernet LANs with some Token Ring LANs. The network supported approximately 400 servers running TCP/IP and Novell IPX protocols. Hewlett-Packard HP/3000 servers running business applications were located in 3Com's Santa Clara, California, and Marlow, England data centers, combined with an IBM ES/9000 mainframe in Santa Clara. These computers were linked via 56 Kbps leased lines.
Figure 4: 3Com Network Before ERP Implementation
Figure 5: Redundant Design Improves Performance and Resiliency
Figure 6: 3Com Network As Retrofitted to Support ERP
Vendor relationships can make a big difference when it comes to achieving a successful ERP implementation. The 3Com/Informix combination illustrates how a close collaboration between networking and database vendors resulted in a highly integrated enterprise business system with a global reach. 3Com and Informix customers can take advantage of this strong partnership when implementing their own systems.
Here are some key factors that can help ensure successful implementation of a large-scale business system:
Literature Stock Number: 500643-001
Copyright © 1998, 3Com Corporation. All Rights Reserved.
Link to 3Com
Used by Permission.
