8 Network Troubleshooting

8.0 Chapter Introduction

8.0.1 Chapter Introduction

Page 1:
Once a network is operational, administrators have to monitor its performance for the sake of the organization's productivity. From time-to-time network outages can occur. Sometimes they are planned and their impact on the organization easily managed. Sometimes they are not planned and their impact on the organization can be severe. In the event of an unexpected network outage administrators must be able to troubleshoot and bring the network back into full production. In this chapter you will learn a systematic process for troubleshooting network outages.

8.0.1 - Chapter Introduction
The diagram depicts the chapter objectives:
- Establish and document a network baseline.
- Describe the various troubleshooting methodologies and troubleshooting tools.
- Describe the common issues that occur during WAN implementation.
- Identify and troubleshoot common enterprise network implementation issues using a layered model approach.

8.1 Establishing the Network Performance Baseline

8.1.1 Documenting Your Network

Page 1:
Documenting Your Network

To efficiently diagnose and correct network problems, a network engineer needs to know how a network has been designed and what the expected performance for this network should be under normal operating conditions. This information is called the network baseline and is captured in documentation such as configuration tables and topology diagrams.

Network configuration documentation provides a logical diagram of the network and detailed information about each component. This information should be kept in a single location, either as hard copy or on the network on a protected website. Network documentation should include these components:

• Network configuration table
• End-system configuration table
• Network topology diagram

Network Configuration Table

Contains accurate, up-to-date records of the hardware and software used in a network. The network configuration table should provide the network engineer with all the information necessary to identify and correct the network fault.

Click the Router and Switch Documentation button in the figure.

The table in the figure illustrates the data set that should be included for all components:

• Type of device, model designation
• IOS image name
• Device network hostname
• Location of the device (building, floor, room, rack, panel)
• If it is a modular device, include all module types and in which module slot they are located
• Data Link layer addresses
• Network layer addresses
• Any additional important information about physical aspects of the device

Click the End-System Documentation button in the figure.

End-system Configuration Table

Contains baseline records of the hardware and software used in end-system devices such as servers, network management consoles, and desktop workstations. An incorrectly configured end system can have a negative impact on the overall performance of a network.

For troubleshooting purposes, the following information should be documented:

• Device name (purpose)
• Operating system and version
• IP address
• Subnet mask
• Default gateway, DNS server, and WINS server addresses
• Any high-bandwidth network applications that the end-system runs

Click the Network Topology Diagram button in the figure.

Network Topology Diagram

Graphical representation of a network, which illustrates how each device in a network is connected and its logical architecture. A topology diagram shares many of the same components as the network configuration table. Each network device should be represented on the diagram with consistent notation or a graphical symbol. Also, each logical and physical connection should be represented using a simple line or other appropriate symbol. Routing protocols can also be shown.

At a minimum, the topology diagram should include:

• Symbols for all devices and how they are connected
• Interface types and numbers
• IP addresses
• Subnet masks

8.1.1 - Documenting Your Network
The diagram depicts ways of documenting your network, which include router and switch documentation, end-system documentation, and a network topology diagram.

Router and Switch Documentation:
The diagram shows a router information table that documents the following. A sample entry is provided:
- Device Name and Model: R1, Cisco 2611XM
- Interface Name: FA0/0
- MAC Address: 0007.8580.a159
- IP Address/Subnet Mask: 192.168.10.1/24
- IP Routing Protocol: EIGRP 10

The diagram also shows a switch information table that documents the following. A sample entry is provided:
- Switch Name, Model and Management IP Address: S1, Cisco WS-C3550-24-SMI, 192.168.10.2/24
- Port Name: FA0/1
- Speed: 100
- Duplex: Auto
- STP State (Forward/Block): Forward
- Port Fast (Yes/No): No
- Trunk Status: On
- Ether Channel (L2 or L3): L2
- V LAN's: 1
- Key: Connects to R1

End-System Documentation:
The diagram shows an end-system information table that documents the following. A sample entry is provided:
- Device Name (Purpose): SRV1 (Web/TFTP Server)
- Operating System/ Version: UNIX
- IP Address/ Subnet Mask: 192.168.20.254/24
- Default Gateway Address: 192.168.20.1/24
- DNS Server Address: 192.168.20.1/24
- WINS Server Address:
- Network Applications: HTTP FTP
- High Bandwidth Applications: V o IP

Network Topology Diagram:
The diagram shows a graphical representation of a network, which illustrates how each device in a network is connected and its logical architecture. Each network device is represented on the diagram by an icon. These include PC's, switches, routers, servers, and network clouds, and connection links. Device names, interface names, IP address and subnet mask labels are applied.

8.1.2 Documenting Your Network

Page 1:
Network Documentation Process

The figure shows the network documentation process.

Roll over each stage in the figure to learn more about the process.

When you document your network, you may have to gather information directly from routers and switches. Commands that are useful to the network documentation process include:

• The ping command is used to test connectivity with neighboring devices before logging in to them. Pinging to other PCs in the network also initiates the MAC address auto-discovery process.
• The telnet command is used to log in remotely to a device for accessing configuration information.
• The show ip interface brief command is used to display the up or down status and IP address of all interfaces on a device.
• The show ip route command is used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols that have been configured.
• The show cdp neighbor detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

8.1.2 - Documenting Your Network
The diagram depicts a flowchart that describes the network documentation process for devices:
- Login enables you to log in to an undocumented device. Login leads you to interface discovery. Interface discovery allows you to discover relevant information about the device. Interface discovery leads you to a document allowing you to record device information in the network configuration table. This leads to a series of questions.
- Is it a component of the topology diagram? In other words, is this device information important to add to the topology diagram?
- If yes, diagram it and transfer the relevant information.
- If no, has the device been completely documented?
- If yes, perform device discovery to determine what other devices are connected to this one.
- If no, continue with interface discovery.
- Were any devices determined to be undocumented?
- If yes, continue with interface discovery.
- If no, the network documentation is complete.

Page 2:
This activity covers the steps to take to discover a network using primarily the telnet, show cdp neighbors detail, and show ip route commands. This is Part I of a two-part activity.

The topology you see when you open the Packet Tracer activity does not reveal all of the details of the network. The details have been hidden using the cluster function of Packet Tracer. The network infrastructure has been collapsed, and the topology in the file shows only the end devices. Your task is to use your knowledge of networking and discovery commands to learn about the full network topology and document it.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.1.2 - Documenting Your Network
Link to Packet Tracer Exploration: Network Discovery and Documentation

8.1.3 Why is Establishing a Network Baseline Important?

Page 1:
Establishing a network performance baseline requires collecting key performance data from the ports and devices that are essential to network operation. This information helps to determine the "personality" of the network and provides answers to the following questions:

• How does the network perform during a normal or average day?
• Where are the underutilized and over-utilized areas?
• Where are the most errors occurring?
• What thresholds should be set for the devices that need to be monitored?
• Can the network deliver the identified policies?

Measuring the initial performance and availability of critical network devices and links allows a network administrator to determine the difference between abnormal behavior and proper network performance as the network grows or traffic patterns change. The baseline also provides insight into whether the current network design can deliver the required policies. Without a baseline, no standard exists to measure the optimum nature of network traffic and congestion levels.

In addition, analysis after an initial baseline tends to reveal hidden problems. The collected data reveals the true nature of congestion or potential congestion in a network. It may also reveal areas in the network that are underutilized and quite often can lead to network redesign efforts based on quality and capacity observations.

8.1.3 - Why Is Establishing a Network Performance Baseline Important?
The diagram depicts a group of questions surrounding a network cloud. The text on the cloud states: The Network Baseline determines the "personality" of a network under normal conditions.

Questions regarding the Network Baseline:
-How does the network perform during a normal or average day?
-What part of the network is least used?
-What part of the network is most heavily used?
-Where are the most errors occurring?
-What alert thresholds should be set for the devices that need to be monitored?
-Can the network deliver the identified policies?

8.1.4 Steps for Establishing a Network Baseline

Page 1:
Planning for the First Baseline

Because the initial network performance baseline sets the stage for measuring the effects of network changes and subsequent troubleshooting efforts, it is important to plan for it carefully. Here are the recommended steps for planning the first baseline:

Step 1. Determine what types of data to collect

When conducting the initial baseline, start by selecting a few variables that represent the defined policies. If too many data points are selected, the amount of data can be overwhelming, making analysis of the collected data difficult. Start out simply and fine-tune along the way. Generally, some good starting measures are interface utilization and CPU utilization. The figure shows some screenshots of interface and CPU utilization data, as displayed by a network management system called WhatsUp Gold.

Click the Devices and Ports of Interest button in the figure.

Step 2. Identify devices and ports of interest

The next step is to identify those key devices and ports for which performance data should be measured. Devices and ports of interest include:

• Network device ports that connect to other network devices
• Servers
• Key users
• Anything else considered critical to operations

In topology shown in the figure, the network administrator has highlighted the devices and ports of interest to monitor during the baseline test. The devices of interest include routers R1, R2, and R3, PC1 (the Admin terminal), and SRV1 (the Web/TFTP server). The ports of interest include those ports on R1, R2, and R3 that connect to the other routers or to switches, and on router R2, the port that connects to SRV1 (Fa0/0).

By narrowing the ports polled, the results are concise, and network management load is minimized. Remember that an interface on a router or switch can be a virtual interface, such as a switch virtual interface (SVI).

This step is easier if you have configured the device port description fields to indicate what connects to the port. For example, for a router port that connects to the distribution switch in the Engineering workgroup, you might configure the description, "Engineering LAN distribution switch."

Click the Determine Baseline Duration button in the figure.

Step 3. Determine the baseline duration

It is important that the length of time and the baseline information being gathered are sufficient to establish a typical picture of the network. This period should be at least seven days to capture any daily or weekly trends. Weekly trends are just as important as daily or hourly trends.

The figure shows examples of several screenshots of CPU utilization trends captured over a daily, weekly, monthly, and yearly period. The work week trends are too short to accurately reveal the recurring nature of the utilization surge that occurs every weekend on Saturday evening when a major database backup operation consumes network bandwidth. This recurring pattern is revealed in the monthly trend. The yearly trend shown in the example is too long a duration to provide meaningful baseline performance details. A baseline needs to last no more than six weeks, unless specific long-term trends need to be measured. Generally, a two-to-four-week baseline is adequate.

You should not perform a baseline measurement during times of unique traffic patterns because the data would provide an inaccurate picture of normal network operations. You would get an inaccurate measure of network performance if you performed a baseline measurement on a holiday or during a month when most of the company is on vacation.

Baseline analysis of the network should be conducted on a regular basis. Perform an annual analysis of the entire network or baseline different sections of the network on a rotating basis. Analysis must be conducted regularly to understand how the network is affected by growth and other changes.

8.1.4 - Planning for the First Baseline
The diagram depicts considerations when planning for the first baseline. These include selecting which data to measure, identifying the devices and ports of interest, and determining the baseline duration.

Selecting Which Data to Measure:
The diagram shows a screenshot of server interface, CPU, and disk utilization data as displayed by a Fluke Networks network management system.

Devices and Ports of Interest:
The diagram shows a network with PC's, switches, routers, servers, network clouds, and connection links.

Devices and Ports of Interest include the following:
- Router to router ports
- Router to switch ports
- Router port to Web/TFTP server
- PC1 (Admin User)
- SRV1 (Web/TFTP server)

Determining Baseline Duration:
The diagram shows several screenshots of CPU utilization trends captured over various time periods. Examples include:
- Daily Graph (5 minute average)
- Weekly Graph (30 minute average)
- Monthly Graph (2 hour average)
- Yearly Graph (1 day average)

In each graph, max load, average load, and current load are listed.

Page 2:
Measuring Network Performance Data

Sophisticated network management software is often used to baseline large and complex networks. For example, the Fluke Network SuperAgent module enables administrators to automatically create and review reports using its Intelligent Baselines feature. This feature compares current performance levels with historical observations and can automatically identify performance problems and applications that do not provide expected levels of service.

Click the Manual Commands button in the figure.

In simpler networks, the baseline tasks may require a combination of manual data collection and simple network protocol inspectors. Establishing an initial baseline or conducting a performance-monitoring analysis may require many hours or days to accurately reflect network performance. Network management software or protocol inspectors and sniffers may run continuously over the course of the data collection process. Hand collection using show commands on individual network devices is extremely time consuming and should be limited to mission-critical network devices.

8.1.4 - Planning for the First Baseline
The diagram depicts methods of measuring network performance data, including automated data collection and manual commands using Cisco I O S.

Automated Data Collection:
The diagram shows a screenshot of a baseline detail report for analysis of application performance based on transaction time.

Manual commands:
The diagram shows Cisco I O S commands that can be used to manually collect device data.

Command: show version
Description: Shows uptime and version information for device software and hardware.

Command: show i p interface [brief]
Description: Shows all the configuration options that are set on an interface. Use the brief keyword to show only the up and down status of the IP interfaces and the IP address of each interface.

Command: show interface [interface_type interface_num]
Description: Shows detailed output for each interface. To show detailed output for only a single interface, include the interface type and number (for example, ethernet 0/0).

Command: show i p route
Description: Shows the contents of the routing table.

Command: show arp
Description: Shows the contents of the ARP table.

Command: show running-config
Description: Shows the current configuration.

Command: show port
Description: Shows the status of ports on a switch.

Command: show v lan
Description: Shows the status of V LAN's on a switch.

Command: show tech-support
Description: Runs other show commands and provides many pages of detailed output, designed to be sent to technical support. Also useful for other purposes.

8.2 Troubleshooting Methodologies and Tools

8.2.1 A General Approach to Troubleshooting

Page 1:
Network engineers, administrators, and support personnel realize that troubleshooting is a process that takes the greatest percentage their time. Using efficient troubleshooting techniques shortens overall troubleshooting time when working in a production environment.

Two extreme approaches to troubleshooting almost always result in disappointment, delay, or failure. At one extreme is the theorist, or rocket scientist, approach. At the other extreme is the impractical, or caveman, approach.

The rocket scientist analyzes and reanalyzes the situation until the exact cause at the root of the problem has been identified and corrected with surgical precision. While this process is fairly reliable, few companies can afford to have their networks down for the hours or days that it can take for this exhaustive analysis.

The caveman's first instinct is to start swapping cards, cables, hardware, and software until miraculously the network begins operating again. This does not mean that the network is working properly, just that it is operating. While this approach may achieve a change in symptoms faster, it is not very reliable, and the root cause of the problem may still be present.

Since both of these approaches are extremes, the better approach is somewhere in the middle using elements of both. It is important to analyze the network as a whole rather than in a piecemeal fashion. A systematic approach minimizes confusion and cuts down on time otherwise wasted with trial and error.

8.2.1 - A General Approach to Troubleshooting
The diagram depicts three approaches to troubleshooting:
- Caveman Approach (Brute Force)
- Rocket Scientist Approach (Theorist)
- Systematic Approach (Best)

8.2.2 Using Layered Models for Troubleshooting

Page 1:
OSI Versus TCP/IP Layered Models

Logical networking models, such as the OSI and TCP/IP models, separate network functionality into modular layers. When troubleshooting, these layered models can be applied to the physical network to isolate network problems. For example, if the symptoms suggest a physical connection problem, the network technician can focus on troubleshooting the circuit that operates at the Physical layer. If that circuit functions properly, the technician looks at areas in another layer that could be causing the problem.

OSI Reference Model

The OSI model provides a common language for network engineers and is commonly used in troubleshooting networks. Problems are typically described in terms of a given OSI model layer.

The OSI reference model describes how information from a software application in one computer moves through a network medium to a software application in another computer.

The upper layers (5-7) of the OSI model deal with application issues and generally are implemented only in software. The Application layer is closest to the end user. Both users and Application layer processes interact with software applications that contain a communications component.

The lower layers (1-4) of the OSI model handle data-transport issues. Layers 3 and 4 are generally implemented only in software. The Physical layer (Layer 1) and Data Link layer (Layer 2) are implemented in hardware and software. The Physical layer is closest to the physical network medium, such as the network cabling, and is responsible for actually placing information on the medium.

TCP/IP Model

Similar to the OSI networking model, the TCP/IP networking model also divides networking architecture into modular layers. The figure shows how the TCP/IP networking model maps to the layers of the OSI networking model. It is this close mapping that allows the TCP/IP suite of protocols to successfully communicate with so many networking technologies.

The Application layer in the TCP/IP suite actually combines the functions of the three OSI model layers: Session, Presentation, and Application. The Application layer provides communication between applications such as FTP, HTTP, and SMTP on separate hosts.

The Transport layers of TCP/IP and OSI directly correspond in function. The Transport layer is responsible for exchanging segments between devices on a TCP/IP network.

The TCP/IP Internet layer relates to the OSI Network layer. The Internet layer is responsible for placing messages in a fixed format that allows devices to handle them.

The TCP/IP network access layer corresponds to the OSI physical and Data Link layers. The network access layer communicates directly with the network media and provides an interface between the architecture of the network and the Internet layer.

Click the Devices at the OSI layers button in the figure.

Roll over each device to show which OSI Layers you commonly need to troubleshoot for that type of device.

8.2.2 - Using Layered Models for Troubleshooting
The diagram depicts O S I versus TCP/IP layered models and the O S I layers at which various network devices operate.

Layered Models:
O S I Layers:
Layer 7. Application
Layer 6. Presentation
Layer 5. Session
Layer 4. Transport
Layer 3. Network
Layer 2. Data Link
Layer 1. Physical

TCP/IP Layers:
Layer 4. Application
Layer 3. Transport
Layer 2. Internetwork
Layer 1. Network Access

The O S I model Layer 4, the Transport Layer, corresponds directly with the TCP/IP model Layer 3, the Transport Layer. Also, the O S I Layer 3, the Network Layer, corresponds directly with the TCP/IP Layer 3, the Internet Layer.

Devices at the O S I Layers:
The diagram shows at which layers of the O S I model devices operate.
Router: Layers 1 through 4
Firewall: Layers 1 through 4
Multilayer switch: Layers 1 through 4
Standard switch: Layers 1 and 2
Hub: Layer 1
End system: Layers 1 through 7

8.2.3 General Troubleshooting Procedures

Page 1:
The stages of the general troubleshooting process are:

• Stage 1 Gather symptoms - Troubleshooting begins with the process of gathering and documenting symptoms from the network, end systems, and users. In addition, the network administrator determines which network components have been affected and how the functionality of the network has changed compared to the baseline. Symptoms may appear in many different forms, including alerts from the network management system, console messages, and user complaints. While gathering symptoms, questions should be used as a method of localizing the problem to a smaller range of possibilities.
• Stage 2 Isolate the problem - The problem is not truly isolated until a single problem, or a set of related problems, is identified. To do this, the network administrator examines the characteristics of the problems at the logical layers of the network so that the most likely cause can be selected. At this stage, the network administrator may gather and document more symptoms depending on the problem characteristics that are identified.
• Stage 3 Correct the problem - Having isolated and identified the cause of the problem, the network administrator works to correct the problem by implementing, testing, and documenting a solution. If the network administrator determines that the corrective action has created another problem, the attempted solution is documented, the changes are removed, and the network administrator returns to gathering symptoms and isolating the problem.

These stages are not mutually exclusive. At any point in the process, it may be necessary to return to previous stages. For instance, it may be required to gather more symptoms while isolating a problem. Additionally, when attempting to correct a problem, another unidentified problem could be created. As a result, it would be necessary to gather the symptoms, isolate, and correct the new problem.

A troubleshooting policy should be established for each stage. A policy provides a consistent manner in which to perform each stage. Part of the policy should include documenting every important piece of information.

8.2.3 - Using Layered Models for Troubleshooting
The diagram depicts the stages of the general troubleshooting process.

Stage 1. Gather symptoms.
Stage 2. Isolate the problem.
Stage 3. Correct the problem.

Stages 1 and 2 are repeated until the problem is isolated. After the problem is corrected in Stage 3, if another problem arises as a result of the correction, the process returns back to Stage 1.

8.2.4 Troubleshooting Methods

Page 1:
Troubleshooting Methods

There are three main methods for troubleshooting networks:

• Bottom up
• Top down
• Divide and conquer

Each approach has its advantages and disadvantages. This topic describes the three methods and provides guidelines for choosing the best method for a specific situation.

Bottom-Up Troubleshooting Method

In bottom-up troubleshooting you start with the physical components of the network and move up through the layers of the OSI model until the cause of the problem is identified. Bottom-up troubleshooting is a good approach to use when the problem is suspected to be a physical one. Most networking problems reside at the lower levels, so implementing the bottom-up approach often results in effective results. The figure shows the bottom-up approach to troubleshooting.

The disadvantage with the bottom-up troubleshooting approach is it requires that you check every device and interface on the network until the possible cause of the problem is found. Remember that each conclusion and possibility must be documented so there can be a lot of paper work associated with this approach. A further challenge is to determine which devices to start examining first.

Click the Top-Down Method button in the figure.

Top-Down Troubleshooting Method

In top-down troubleshooting your start with the end-user applications and move down through the layers of the OSI model until the cause of the problem has been identified. End-user applications of an end system are tested before tackling the more specific networking pieces. Use this approach for simpler problems or when you think the problem is with a piece of software.

The disadvantage with the top-down approach is it requires checking every network application until the possible cause of the problem is found. Each conclusion and possibility must be documented. and the challenge is to determine which application to start examining first.

Click the Divide-and-Conquer Method button in the figure.

Divide-and-Conquer Troubleshooting Method

When you apply the divide-and-conquer approach toward troubleshooting a networking problem, you select a layer and test in both directions from the starting layer.

In divide-and-conquer troubleshooting you start by collecting user experience of the problem, document the symptoms and then, using that information, make an informed guess as to which OSI layer to start your investigation. Once you verify that a layer is functioning properly, assume that the layers below it are functioning and work up the OSI layers. If an OSI layer is not functioning properly, work your way down the OSI layer model.

For example, if users can't access the web server and you can ping the server, then you know that the problem is above Layer 3. If you can't ping the server, then you know the problem is likely at a lower OSI layer.

8.2.4 - Troubleshooting Methods
The diagram depicts the three main methods for troubleshooting networks, which are based on the O S I model:

Bottom up - Starts at the O S I Physical Layer

Top down - Starts at the O S I Application Layer

Divide and conquer - Can start at either the O S I Data Link, Network, or Transport Layer.

Page 2:
Guidelines for Selecting a Troubleshooting Method

To quickly resolve network problems, take the time to select the most effective network troubleshooting method. Examine the figure. Use the process shown in the figure to help you select the most efficient troubleshooting method.

Here is an example of how you would choose a troubleshooting method for a specific problem. Two IP routers are not exchanging routing information. The last time this type of problem occurred it was a protocol issue. So you choose the divide-and-conquer troubleshooting method. Your analysis reveals that there is connectivity between the routers so you start your troubleshooting efforts at the physical or Data Link layer, confirm connectivity and begin testing the TCP/IP-related functions at next layer up in the OSI model, the Network layer.

8.2.4 - Troubleshooting Methods
The diagram depicts guidelines for selecting a troubleshooting method.

A block chart is shown describing how to select a method. Start by analyzing the symptoms. Then determine the scope of the problem and/or apply previous experiences. If the scope is limited, use the top-down approach. If it is more complex, use the bottom-up approach.

If you choose to apply previous experiences and you have experienced the problem before, you may use the divide and conquer approach. If it is a new problem, use the bottom-up approach.

8.2.5 Gathering Symptoms

Page 1:
Gathering Symptoms

To determine the scope of the problem gather (document) the symptoms. The figure shows the a flow chart of this process. Each step in this process is briefly described here:

Step 1. Analyze existing symptoms - Analyze symptoms gathered from the trouble ticket, users, or end systems affected by the problem to form a definition of the problem.

Step 2. Determine ownership - If the problem is within your system, you can move onto the next stage. If the problem is outside the boundary of your control, for example, lost Internet connectivity outside of the autonomous system, you need to contact an administrator for the external system before gathering additional network symptoms.

Step 3. Narrow the scope - Determine if the problem is at the core, distribution, or access layer of the network. At the identified layer, analyze the existing symptoms and use your knowledge of the network topology to determine which pieces of equipment are the most likely cause.

Step 4. Gather symptoms from suspect devices - Using a layered troubleshooting approach, gather hardware and software symptoms from the suspect devices. Start with the most likely possibility, and use knowledge and experience to determine if the problem is more likely a hardware or software configuration problem.

Step 5. Document symptoms - Sometimes the problem can be solved using the documented symptoms. If not, begin the isolating phase of the general troubleshooting process.

Click the Commands button in the figure.

Use the Cisco IOS commands to gather symptoms about the network. The table in the figure describes the common Cisco IOS commands you can use to help you gather the systems of a network problem.

Although the debug command is an important tool for gathering symptoms it generates a large amount of console message traffic and the performance of a network device can be noticeably affected. Make sure you warn network users that a troubleshooting effort is underway and that network performance may be affected. Remember to disable debugging when you are done.

8.2.5 - Gathering Symptoms
The diagram depicts a flow chart describing the steps for gathering symptoms for network problems. Commands are also provided.

Steps for Gathering Symptoms:
Step 1. Analyze existing symptoms.
Step 2. Determine ownership. Is the problem within the autonomous system? If so, go to Step 3. Otherwise, contact the external administrator.
Step 3. Narrow the scope.
Step 4. Determine symptoms.
Step 5. Document symptoms. If the problem can be solved using the documented symptoms, solve the problem and document the solution. Otherwise, begin the isolating phase of the general troubleshooting process.

Commands:
Command: ping {host | i p-address}
Description: Sends an echo request packet to an address, then waits for a reply. The host | i p-address variable is the IP alias or IP address of the target system.

Command: trace route {destination}
Description: Identifies the path a packet takes through the networks. The destination variable is the hostname or IP address of the target system.

Command: telnet {host | i p-address}
Description: Connects to an IP address using the Telnet application.

Command: show i p interface brief
Description: Displays a summary of the status of all interfaces on a device.

Command: show i p route
Description: Displays the current state of the IP routing table.

Command: show running-config interface
Description: Displays contents of the currently running configuration file for a particular interface.

Command: [no] debug ?
Description: Displays a list of options for enabling or disabling debugging events on a device.

Command: show protocols
Description: Displays the configured protocols and shows the global and interface-specific status of any configured Layer 3 protocol.

Page 2:
Questioning End Users

When you question end users about a network problem they may be experiencing, use effective questioning techniques. This way you will get the information you need to effectively document the symptoms of a problem. The table in the figure provides some guidelines and end-user example questions.

8.2.5 - Gathering Symptoms
The diagram depicts guidelines for questioning end users.

Guideline: Ask questions that are pertinent to the problem.
Example Question: What does not work?

Guideline: Use each question as a means to either eliminate or discover possible problems.
Example Question: Are the things that do work and the things that do not work related?

Guideline: Speak at a technical level that the user can understand.
Example Question: Has the thing that does not work ever worked?

Guideline: Ask the user when the problem was first noticed.
Example Question: When was the problem first noticed?

Guideline: Did anything unusual happen since the last time it worked?
Example Question: What has changed since the last time it did work?

Guideline: Ask the user to recreate the problem, if possible.
Example Question: Can you reproduce the problem?

Guideline: Determine the sequence of events that took place before the problem happened.
Example Question: When exactly did the problem occur?

8.2.6 Troubleshooting Tools

Page 1:
Software Troubleshooting Tools

A wide variety of software and hardware tools are available to make troubleshooting easier. These tools may be used to gather and analyze symptoms of network problems and often provide monitoring and reporting functions that can be used to establish the network baseline.

NMS Tools

Network management system (NMS) tools include device-level monitoring, configuration, and fault management tools. The figure shows an example display from the What's Up Gold NMS software. These tools can be used to investigate and correct network problems. Network monitoring software graphically displays a physical view of network devices, allowing network managers to monitor remote devices without actually physically checking them. Device management software provides dynamic status, statistics, and configuration information for switched products. Examples of commonly used network management tools are CiscoView, HP Openview, Solar Winds, and What's Up Gold.

Click the Knowledge Base button in the figure to see an example of a knowledge base website.

Knowledge Bases

On-line network device vendor knowledge bases have become indispensable sources of information. When vendor-based knowledge bases are combined with Internet search engines like Google, a network administrator has access to a vast pool of experience-based information.

The figure shows the Cisco Tools & Resources page found at http://www.cisco.com. This is a free tool providing information on Cisco-related hardware and software. It contains troubleshooting procedures, implementation guides, and original white papers on most aspects of networking technology.

Click the Baselining Tools button in the figure to see some examples of baselining tools.

Baselining Tools

Many tools for automating the network documentation and baselining process are available. These tools are available for Windows, Linux, AUX operating systems. The figure shows a screen capture of the SolarWinds LAN surveyor and CyberGauge software. Baselining tools help you with common baseling documentation tasks. For example they can help you draw network diagrams, help you to keep network software and hardware documentation up-to-date and help you to cost-effectively measure baseline network bandwidth use.

Click the Protocol Analyzer button in the figure to see an example of a typical protocol analyzer application.

Protocol Analyzers

A protocol analyzer decodes the various protocol layers in a recorded frame and presents this information in a relatively easy to use format. The figure shows a screen capture of the Wireshark protocol analyzer. The information displayed by a protocol analyzer includes, the physical, data link, protocol and descriptions for each frame. Most protocol analyzers can filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured.

8.2.6 - Gathering Symptoms
The diagram depicts various software troubleshooting tools. These include network management system (NMS) tools, knowledge bases, baselining tools, and protocol analyzers.

Network Management System (NMS):
The diagram shows a screenshot from the What's Up Gold NMS software.

Knowledge Bases:
The diagram shows a screenshot of the Cisco Tools & Resources page found at http://www.cisco.com.

Baselining Tools:
The diagram shows a screenshot of the SolarWinds LAN surveyor and CyberGauge software.

Protocol Analyzers:
The diagram shows a screenshot of the Wireshark protocol analyzer.

Page 2:
Hardware Troubleshooting Tools

Click the buttons in the figure to see examples of various hardware troubleshooting tools.

Network Analysis Module

A network analysis module (NAM) can be installed in Cisco Catalyst 6500 series switches and Cisco 7600 series routers to provide a graphical representation of traffic from local and remote switches and routers. The NAM is a embedded browser-based interface that generates reports on the traffic that consumes critical network resources. In addition, the NAM can capture and decode packets and track response times to pinpoint an application problem to the network or the server.

Digital Multimeters

Digital multimeters (DMMs) are test instruments that are used to directly measure electrical values of voltage, current, and resistance. In network troubleshooting, most of the multimedia tests involve checking power-supply voltage levels and verifying that network devices are receiving power.

Cable Testers

Cable testers are specialized, handheld devices designed for testing the various types of data communication cabling. Cabling testers can be used to detect broken wires, crossed-over wiring, shorted connections, and improperly paired connections. These devices can be inexpensive continuity testers, moderately priced data cabling testers, or expensive time-domain reflectometers (TDRs).

TDRs are used to pinpoint the distance to a break in a cable. These devices send signals along the cable and wait for them to be reflected. The time between sending the signal and receiving it back is converted into a distance measurement. The TDR function is normally packaged with data cabling testers. TDRs used to test fiber optic cables are known as optical time-domain reflectometers (OTDRs).

Cable Analyzers

Cable analyzers are multifunctional handheld devices that are used to test and certify copper and fiber cables for different services and standards. The more sophisticated tools include advanced troubleshooting diagnostics that measure distance to performance defect (NEXT, RL), identify corrective actions, and graphically display crosstalk and impedance behavior. Cable analyzers also typically include PC-based software. Once field data is collected the handheld device can upload its data and up-to-date and accurate reports can be created.

Portable Network Analyzers

Portable devices that are used for troubleshooting switched networks and VLANs. By plugging the network analyzer in anywhere on the network, a network engineer can see the switch port to which the device is connected and the average and peak utilization. The analyzer can also be used to discover VLAN configuration, identify top network talkers, analyze network traffic, and view interface details. The device can typically output to a PC that has network monitoring software installed for further analysis and troubleshooting.

8.2.6 - Gathering Symptoms
The diagram depicts various hardware troubleshooting tools. These include Network Analysis Module (NAM), digital multimeters (DMM's), cable testers, cable analyzers, and network analyzers.

Network Analysis Module (NAM):
The diagram shows a screenshot of the browser-based Cisco Catalyst 6000 NAM Traffic Analyzer application. Also shown is the NAM hardware module for the Catalyst 6500 switch.

DMM's:
The diagram shows a photograph of the Fluke Networks 179 Digital Multimeter.

Cable Testers:
The diagram shows photographs of the Fluke Networks LinkRunner Pro CableIQ Qualification testers.

Cable Analyzers:
The diagram shows a photograph of the Fluke Networks DTX Cable Analyzer.

Network Analyzers:
The diagram shows a photograph of the Fluke Networks OptiView Series III Integrated Network Analyzer.

Page 3:
Research Activity

The following are links to various troubleshooting tools.

Software Tools

Network Management Systems:

http://www.ipswitch.com/products/whatsup/index.asp?t=demo

http://www.solarwinds.com/products/network_tools.aspx

Baselining Tools:

http://www.networkuptime.com/tools/enterprise/

Knowledge Bases:

http://www.cisco.com

Protocol Analyzers:

http://www.flukenetworks.com/fnet/en-us/products/OptiView+Protocol+Expert/

Hardware Tools

Cisco Network Analyzer Module (NAM):

http://www.cisco.com/en/US/docs/net_mgmt/network_analysis_module_software/3.5/user/guide/user.html

Cable Testers:

http://www.flukenetworks.com/fnet/en-us/products/CableIQ+Qualification+Tester/Demo.htm

Cable Analyzers:

http://www.flukenetworks.com/fnet/en-us/products/DTX+CableAnalyzer+Series/Demo.htm

Network Analyzers:

http://www.flukenetworks.com/fnet/en-us/products/OptiView+Series+III+Integrated+Network+Analyzer/Demos.htm

8.2.6 - Gathering Symptoms
The diagram shows a screenshot of the SolarWinds LAN surveyor network mapping application.

8.3 Common WAN Implementation Issues

8.3.1 WAN Communications

Page 1:
A communications provider or a common carrier normally owns the data links that make up a WAN. The links are made available to subscribers for a fee and are used to interconnect LANs or connect to remote networks. WAN data transfer speed (bandwidth) is considerably slower than the common LAN bandwidth. The charges for link provision are the major cost element, therefore the WAN implementation must aim to provide maximum bandwidth at acceptable cost. With user pressure to provide more service access at higher speeds and management pressure to contain cost, determining the optimal WAN configuration is not an easy task.

WANs carry a variety of traffic types, such as data, voice, and video. The design selected must provide adequate capacity and transit times to meet the requirements of the enterprise. Among other specifications, the design must consider the topology of the connections between the various sites, the nature of those connections, and bandwidth capacity.

Older WANs often consisted of data links directly connecting remote mainframe computers. Today's WANs connect geographically separated LANs. WAN technologies function at the lower three layers of the OSI reference model. End-user stations, servers, and routers communicate across LANs, and the WAN data links terminate at local routers.

Routers determine the most appropriate path to the destination of the data from the Network layer headers and transfer the packets to the appropriate data link connection for delivery on the physical connection. Routers can also provide quality of service (QoS) management, which allots priorities to the different traffic streams.

8.3.1 - WAN Communications
The diagram depicts WAN communications. Two CSU/DSU's are connected to a WAN cloud. A protocol stack attached to each CSU/DSU is labeled Router and provides a bridge that goes from the WAN O S I Layer 1 to Layer 2 over Layer 3 and back down to Layer 2 and then to Layer 1 on the local LAN. Caption text: WAN technologies operate at the lower three layers of the O S I model.

8.3.2 Steps in WAN Design

Page 1:
Businesses install WAN connectivity to meet the strategic business requirement of moving data between external branches. Because WAN connectivity is important to the business and expensive, you need to design the WAN in a systematic manner. This figure shows the WAN design steps.

Each time a modification to an existing WAN is considered, these steps should be followed. However, because many WANs have evolved over time, some of the guidelines discussed here may not have been considered. WAN modifications may arise from expanding the enterprise WAN servers or accommodating new work practices and business methods.

These are the steps for designing or modifying a WAN:

Step 1. Locate LANs - Establish the source and destination endpoints that will connect through the WAN.

Step 2. Analyze traffic - Know what data traffic must be carried, its origin, and its destination. WANs carry a variety of traffic types with varying requirements for bandwidth, latency, and jitter. For each pair of endpoints and for each traffic type, information is needed on the various traffic characteristics.

Step 3. Plan the topology - The topology is influenced by geographic considerations but also by requirements such as availability. A high requirement for availability requires extra links that provide alternative data paths for redundancy and load balancing.

Step 4. Estimate the required bandwidth - Traffic on the links may have varying requirements for latency and jitter.

Step 5. Choose the WAN technology - Suitable link technologies must be selected.

Step 6. Evaluate costs - When all the requirements are established, installation and operational costs for the WAN can be determined and compared with the business need driving the WAN implementation.

As shown in the figure, the design steps describe here are not a linear process. Several iterations of these steps may be necessary before a design is finalized. To maintain optimal performance of the WAN, continued monitoring and re-evaluation is required.

8.3.2 - Steps in WAN Design
The diagram depicts the steps in WAN design using a top-to-bottom flowchart.

Step 1. Locate LAN's.

Step 2. Analyze traffic.

Step 3. Plan the topology.

Step 4. Plan the bandwidth.

Step 5. Choose technology.

Step 6. Cost and evaluate.

An arrow labeled "Iterate" goes from Step 5 to Step 3. Another arrow labeled "Review" goes from Step 6 to Step 1.

8.3.3 WAN Traffic Considerations

Page 1:
The table in the figure shows the wide variety of traffic types and their varying requirements of bandwidth, latency, and jitter that WAN links are required to carry.

To determine traffic flow conditions and timing of a WAN link, you need to analyze the traffic characteristics specific to each LAN that is connected to the WAN. Determining traffic characteristics may involve consulting the network users and evaluating their needs.

8.3.3 - WAN Traffic Considerations
The diagram lists traffic types and traffic considerations.

Traffic Types:
Traffic: Voice
Latency: Low
Jitter: Low
Bandwidth: Medium

Traffic: Transaction data (for example, S N A)
Latency: Medium
Jitter: Medium
Bandwidth: Medium

Traffic: Messaging (e-mail)
Latency: High
Jitter: High
Bandwidth: High

Traffic: File transfer
Latency: High
Jitter: High
Bandwidth: High

Traffic: Batch data
Latency: High
Jitter: High
Bandwidth: High

Traffic: Network management
Latency: High
Jitter: High
Bandwidth: Low

Traffic: Videoconferencing
Latency: Low
Jitter: Low
Bandwidth: High

Traffic Characteristics:

Characteristic: Connectivity and volume flows
Description: Where does this traffic flow and how much traffic flows there?

Characteristic: Client/server data
Description: What kind of traffic flows between the client and server?

Characteristic: Latency tolerance, including length and variability
Description: Can the users tolerate delays? How much and how often?

Characteristic: Network availability tolerance
Description: How critical is network availability to the users of this LAN?
Can they tolerate WAN outages or would their work grind to a halt?

Characteristic: Error-rate tolerance
Description: Is this noisy traffic?

Characteristic: Priority
Description: Does this traffic have priority over other traffic?
For example, network management messages should have a higher priority than e-mail.

Characteristic: Protocol type
Description: What types of protocols operate within the network?

Characteristic: Average packet length
Description: What is the average size of packets being transmitted?

8.3.4 WAN Topology Considerations

Page 1:
After establishing LAN endpoints and traffic characteristics, the next step in implementing a WAN is to design a suitable topology. Designing a WAN topology essentially consists of the following:

Selecting an interconnection pattern or layout for the links between the various locations

Selecting the technologies for those links to meet the enterprise requirements at an acceptable cost

Click the buttons in the figure to view an example of each type of WAN topology.

Many WANs use a star topology. As the enterprise grows and new branches are added, the branches are connected back to the head office, producing a traditional star topology. Star endpoints are sometimes cross-connected, creating a mesh or partial mesh topology. This provides for many possible combinations for interconnections. When designing, re-evaluating, or modifying a WAN, a topology that meets the design requirements must be selected.

In selecting a layout, there are several factors to consider. More links increase the cost of the network services, but having multiple paths between destinations increases reliability. Adding more network devices to the data path increase latency and decreases reliability. Generally, each packet must be completely received at one node before it can be passed to the next.

Click the Hierarchical button in the figure.

When many locations must be joined, a hierarchical solution is recommended. For example, imagine an enterprise that is operational in every country of the European Union and has a branch in every town with a population over 10,000. Each branch has a LAN, and it has been decided to interconnect the branches. A mesh network is clearly not feasible because there would be hundreds of thousands of links.

The answer is to implement a hierarchical topology. Group the LANs in each area and interconnected them to form a region, interconnect the regions to form the core of the WAN. The area could be based on the number of locations to be connected with an upper limit of between 30 and 50. The area would have a star topology, with the hubs of the stars linked to form the region. Regions could be geographic, connecting between three and 10 areas, and the hub of each region could be linked point-to-point.

A three-layer hierarchy is often useful when the network traffic mirrors the enterprise branch structure and is divided into regions, areas, and branches. It is also useful when there is a central service to which all branches must have access but traffic levels are insufficient to justify direct connection of a branch to the service.

The LAN at the center of the area may have servers providing area-based as well as local service. Depending on the traffic volumes and types, the access connections may be dialup, leased, or frame relay. Frame Relay facilitates some meshing for redundancy without requiring additional physical connections. Distribution links could be Frame Relay or ATM, and the network core could be ATM or leased line.

When planning simpler networks, a hierarchical topology should still be considered because it may provide for better network scalability. The hub at the center of a two-layer model is also a core, but with no other core routers connected to it. Likewise, in a single-layer solution, the area hub serves as the regional hub and the core hub. This allows easy and rapid future growth because the basic design can be replicated to add new service areas.

8.3.4 - WAN Topology Considerations
The diagram depicts various common WAN topologies, such as star, full mesh, partial mesh, and hierarchical.

Star Topology:
The diagram shows a central router, labeled Paris, with four other routers connected to it labeled Brussels, Berlin, Vienna, and Rome.

Full Mesh Topology:
The diagram shows five routers labeled Paris, Brussels, Berlin, Vienna, and Rome. Each router has a WAN connection to every other router.

Partial Mesh Topology:
The diagram shows five routers labeled Paris, Brussels, Berlin, Vienna, and Rome. Each router has a WAN connection to at least two other routers, but only Paris has a WAN connection to every other router.

Hierarchical Topology:
The diagram shows a central router connected to a WAN backbone. Four other routers are connected to it by WAN links. Each of these four routers has six routers connected to it with WAN links.

Page 2:
WAN Connection Technologies

A typical private WAN uses a combination of technologies that are usually chosen based on traffic type and volume. ISDN, DSL, Frame Relay, or leased lines are used to connect individual branches into an area. Frame Relay, ATM, or leased lines are used to connect external areas back to the backbone. ATM or leased lines form the WAN backbone. Technologies that require the establishment of a connection before data can be transmitted, such as basic telephone, ISDN, or X.25, are not suitable for WANs that require rapid response time or low latency.

Different parts of an enterprise may be directly connected with leased lines, or they may be connected with an access link to the nearest point-of-presence (POP) of a shared network. Frame Relay and ATM are examples of shared networks. Leased lines are typically more expensive than access links but are available at virtually any bandwidth and provide very low latency and jitter.

ATM and Frame Relay networks carry traffic from several customers over the same internal links. The enterprise has no control over the number of links or hops that data must traverse in the shared network. It cannot control the time data must wait at each node before moving to the next link. This uncertainty in latency and jitter makes these technologies unsuitable for some types of network traffic. However, the disadvantages of a shared network may often be outweighed by the reduced cost. Because several customers are sharing the link, the cost to each is generally less than the cost of a direct link of the same capacity.

Although ATM is a shared network, it has been designed to produce minimal latency and jitter through high-speed internal links sending easily manageable units of data, called cells. ATM cells have a fixed length of 53 bytes, 48 bytes for data and 5 bytes for the header. ATM is widely used for carrying delay-sensitive traffic.

Frame Relay may also be used for delay-sensitive traffic, often using QoS mechanisms to give priority to the more sensitive data.

8.3.4 - WAN Topology Considerations
The diagram lists WAN connection technologies.

Technology: Leased Line
Charge: Distance, capacity
Typical Bit Rate: up to 45 Mbps (E3/T3)
Other: Permanent fixed capacity

Technology: Basic Telephone
Charge: Distance, time
Typical Bit Rate: 33 to 56 kbps
Other: Dialed, slow connection

Technology: ISDN
Charge: Distance, time
Typical Bit Rate: 64 or 128 kbps (BR I) up to 2 Mbps (PRI)
Other: Dialed, slow connection

Technology: X.25
Charge: Volume
Typical Bit Rate: up to 48 kbps
Other: Switched fixed capacity

Technology: ATM
Charge: Capacity
Typical Bit Rate: up to 155 Mbps
Other: Permanent variable capacity

Technology: Frame Relay
Charge: Capacity
Typical Bit Rate: up to 1.5 Mbps
Other: Permanent variable capacity

Technology: DSL
Charge: Monthly subscription
Typical Bit Rate: up to 3 Mbps
Other: Always on shared Internet

Technology: Metro Ethernet
Charge: Monthly subscription
Typical Bit Rate: up to 500 Mbps
Other: Limited geographical scope

Page 3:
Many enterprise WANs have connections to the Internet. Although the Internet may pose a security problem it does provides an alternative for inter-branch traffic. Part of the traffic that must be considered during design is going to or coming from the Internet. Common implementations are to have each network in the company connect to a different ISP, or to have all company networks connect to a single ISP from a core layer connection.

8.3.4 - WAN Topology Considerations
The diagram depicts using the Internet as a WAN.

Star Topology:
The diagram shows a central cloud labeled Internet, with five other routers labeled Brussels, Berlin, Vienna, Rome, and Paris connected to it with WAN links.

8.3.5 WAN Bandwidth Considerations

Page 1:
Recall that a network supports the business needs of a company. Many companies rely on the high-speed transfer of data between remote locations. Consequently, higher bandwidth is crucial because it allows more data to be transmitted in a given time. When bandwidth is inadequate, competition between various types of traffic causes response times to increase, which reduces employee productivity and slows down critical web-based business processes.

The figure shows how WAN links are typically classified either high or low speed.

8.3.5 - WAN Bandwidth Considerations
The diagram depicts a high-bandwidth WAN link that is very fast and a low-bandwidth WAN kink that is slow.

Bandwidth = Available bit transfer rate (bit rate capacity)

8.3.6 Common WAN Implementation Issues

Page 1:
The figure summarizes the common WAN implement issues and the questions you need to answer before you can effectively implement a WAN.

8.3.6 - Common WAN Implementation issues
The diagram depicts common WAN implementation issues. These include the following:

Reliability? Our branch depends on the WAN. Reliability is essential.

Private or Public? Which infrastructure should I use?

Latency? Delays can be a problem for real-time traffic.

Confidentiality? We need to send sensitive company information to our branches across the WAN.

Security? How do we protect ourselves from security threats over the WAN?

Q o S? End-to-end Quality of Service (Q o S) may be hard to obtain across the Internet.

8.3.7 Case Study: WAN Troubleshooting from an ISP's Perspective

Page 1:
The graphic illustrates the typical questions that the technical support desk of an ISP should ask a customer that is calling for support.

A significant proportion of the support calls received by an ISP refer to slowness of the Network. To troubleshoot this effectively, you have to isolate the individual components and test each one as follows:

Individual PC host - A large number of user applications open on the PC at the same time may be responsible for the slowness that is being attributed to the Network. Tools like the Task Manager in a Windows PC can help determine CPU utilization.

LAN - If the customer has network monitoring software on their LAN, the network manager should be able to tell them whether the bandwidth on the LAN is frequently reaching 100 percent utilization. This is a problem that the customer company would need to solve internally. This is why a network baseline and an ongoing monitoring is so important.

Link from the edge of the user network to the edge of the ISP - Test the link from the customer edge router to the edge router of the ISP by asking the customer to log in to their router and send a hundred 1500 byte pings (stress pings) to the IP address of the ISP edge router. This problem is not something the customer can fix it is the ISP's responsibility to engage the link provider to fix this.

Backbone of the ISP - The ISP customer service representative can run stress pings from the ISP edge router to the edge router of the customer. They can also run stress pings across each link that customer traffic traverses. By isolating and testing each link, the ISP can determine which link is causing the problem.

Server being accessed - In some cases the slowness, being attributed to the network, may be caused by server congestion. This problem is the hardest to diagnose and it should be the last option pursued after all other options have been eliminated.

8.3.7 - Case Study: WAN Troubleshooting from an ISP's Perspective
The diagram depicts a case study: WAN Troubleshooting from an ISP's Perspective. Questions to ask the customer include:

- What, if anything, has changed since before you started seeing this problem?
- Have you power-cycled (turned off and back on; rebooted) the router, switch, PC, server? Would you be willing to do it again while I stay on the phone with you?
- Has there been a power outage, lightening strike, or a power brown-out in your area recently?
- Do you have up-to-date virus software on your PC's?

Also do the following:
- Ask customers to fax or e-mail to you their network diagram.
- Help customers isolate the different parts of the Internet.

Page 2:
In this activity, you and another student will build the network displayed in the topology diagram. You will configure NAT, DHCP, and OSPF, and then verify connectivity. When the network is fully operational, one student will introduce several errors. Then the other student will use troubleshooting skills to isolate and solve the problem. Then the students will reverse roles and repeat the process. This activity can be done on real equipment or with Packet Tracer.

8.3.7 - Case Study: WAN Troubleshooting from an ISP's Perspective
Link to Activity: Troubleshooting Role Play

8.4 Network Troubleshooting

8.4.1 Interpreting Network Diagrams to Identify Problems

Page 1:
It is nearly impossible to troubleshoot any type of network connectivity issue without a network diagram that depicts IP addresses, IP routes, devices such as firewalls and switches, and so on. Generally, both logical and physical topologies aid in troubleshooting.

Physical Network Diagram

A physical network diagram shows the physical layout of the devices connected to the network. Knowing how devices are physically connected is necessary for troubleshooting problems at the Physical layer, such as cabling or hardware problems. Information recorded on the diagram typically includes:

• Device type
• Model and manufacturer
• Operating system version
• Cable type and identifier
• Cable specification
• Connector type
• Cabling endpoints

The figure shows an example of a physical network diagram that provides information about the physical location of the network devices, the types of cabling between them, and the cable identification numbers. This information would be primarily used for troubleshooting physical problems with devices or cabling. In addition to the physical network diagram, some administrators also include actual photographs of their wiring closets as part of their network documentation.

Logical Network Diagram

A logical network diagram shows how data is transferred on the network. Symbols are used to represent network elements such as routers, servers, hubs, hosts, VPN concentrators, and security devices. Information recorded on a logical network diagram may include:

• Device identifiers
• IP address and subnet
• Interface identifiers
• Connection type
• DLCI for virtual circuits
• Site-to-site VPNs
• Routing protocols
• Static routes
• Data-link protocols
• WAN technologies used

Click the Logical button in the figure to see an example of a logical network diagram.

The figure shows the same network but this time provides logical information such as specific device IP addresses, network numbers, port numbers, signal types, and DCE assignments for serial links. This information could be used for troubleshooting problems at all OSI layers.

8.4.1 - Interpreting Network Diagrams to Identify Problems
The diagram depicts a physical network diagram and a logical network diagram. The diagram shows a network with PC's, switches, routers, servers, network clouds, and connection links.

Physical Network Diagram:
Equipment locations are identified based on building, room number, wiring closet, rack, and shelf. Physical device names are listed as well as the name and type of WAN circuits.

Logical Network Diagram:
The same equipment is shown but the focus is on logical IP addressing and the interfaces used, not on physical location.

8.4.2 Physical Layer Troubleshooting

Page 1:
Symptoms of Physical Layer Problems

The Physical layer transmits bits from one computer to another and regulates the transmission of a stream of bits over the physical medium. The Physical layer is the only layer with physically tangible properties, such as wires, cards, and antennas.

Failures and suboptimal conditions at the Physical layer not only inconvenience users but could impact the productivity of the entire company. Networks that experience these kinds of conditions usually come to a grinding halt. Because the upper layers of the OSI model depend on the Physical layer to function, a network technician must have the ability to effectively isolate and correct problems at this layer.

A Physical layer problem occurs when the physical properties of the connection are substandard, causing data to be transferred at a rate that is consistently less than the rate of data flow established in the baseline. If there is a problem with suboptimal operation at the Physical layer, the network may be operational, but performance is consistently or intermittently lower than the level specified in the baseline.

Common symptoms of network problems at the Physical layer include:

• Performance lower than baseline - If performance is unsatisfactory all the time, the problem is probably related to a poor configuration, inadequate capacity somewhere, or some other systemic problem. If performance varies and is not always unsatisfactory, the problem is probably related to an error condition or is being affected by traffic from other sources. The most common reasons for slow or poor performance include overloaded or underpowered servers, unsuitable switch or router configurations, traffic congestion on a low-capacity link, and chronic frame loss.
• Loss of connectivity - If a cable or device fails, the most obvious symptom is a loss of connectivity between the devices that communicate over that link or with the failed device or interface, as indicated by a simple ping test. Intermittent loss of connectivity could indicate a loose or oxidized connection.
• High collision counts - Collision domain problems affect the local medium and disrupt communications to Layer 2 or Layer 3 infrastructure devices, local servers, or services. Collisions are normally a more significant problem on shared media than on switch ports. Average collision counts on shared media should generally be below 5 percent, although that number is conservative. Be sure that judgments are based on the average and not a peak or spike in collisions. Collision-based problems may often be traced back to a single source. It may be a bad cable to a single station, a bad uplink cable on a hub or port on a hub, or a link that is exposed to external electrical noise. A noise source near a cable or hub can cause collisions even when there is no apparent traffic to cause them. If collisions get worse in direct proportion to the level of traffic, if the amount of collisions approaches 100 percent, or if there is no good traffic at all, the cable system may have failed.
• Network bottlenecks or congestion - If a router, interface, or cable fails, routing protocols may redirect traffic to other routes that are not designed to carry the extra capacity. This can result in congestion or bottlenecks in those parts of the network.
• High CPU utilization rates - High CPU utilization rates are a symptom that a device, such as a router, switch, or server, is operating at or exceeding its design limits. If not addressed quickly, CPU overloading can cause a device to shut down or fail.
• Console error messages - Error messages reported on the device console indicate a Physical layer problem.

8.4.2 - Physical Layer Troubleshooting
The diagram depicts symptoms of physical layer problems. The seven layers of the O S I model are shown with symptoms expanding from the Physical Layer.

Symptoms
- Performance lower than baseline
- Loss of connectivity
- High collision
- Network bottlenecks or congestion
- High CPU utilization rates
- Console error messages

Page 2:
Causes of Physical Layer Problems

Issues that commonly cause network problems at the Physical layer include:

Power-related

Power-related issues are the most fundamental reason for network failure. The main AC power flows into either an external or internal AC to DC transformer module within a device. The transformer provides correctly modulated DC current, which acts to power device circuits, connectors, ports, and the fans used for device cooling. If a power-related issue is suspected, a physical inspection of the power module is often carried out. Check the operation of the fans, and ensure that the chassis intake and exhaust vents are clear. If other nearby units have also powered down, suspect a power failure at the main power supply.

Hardware faults

Faulty network interface cards (NICs) can be the cause of network transmission errors due to late collisions, short frames, and jabber. Jabber is often defined as the condition in which a network device continually transmits random, meaningless data onto the network. Other likely causes of jabber are faulty or corrupt NIC driver files, bad cabling, or grounding problems.

Cabling faults

Many problems can be corrected by simply reseating cables that have become partially disconnected. When performing a physical inspection, look for damaged cables, improper cable types, and poorly crimped RJ-45s. Suspect cables should be tested or exchanged with a known functioning cable.

Check for connections between devices which incorrectly use crossover cables, or hub and switch ports that are incorrectly cabled using crossover cables. Split-pair cables either operate poorly or not at all, depending on the Ethernet speed used, the length of the split segment, and how far it is located from either end.

Problems with fiber-optic cables may be caused by dirty connectors, excessively tight bends, and swapped RX/TX connections when polarized.

Problems with coaxial cable often occur at the connectors. When the center conductor on the coaxial cable end is not straight and of the correct length, a good connection is not achieved.

Attenuation

An attenuated data bitstream is when the amplitude of the bits is reduced while traveling across a cable. If attenuation is severe, the receiving device cannot always successfully distinguish the component bits of the stream from each other. This ends in a garbled transmission and results in a request from the receiving device for retransmission of the missed traffic by the sender. Attenuation can be caused if a cable length exceeds the design limit for the media (for example, an Ethernet cable is limited to 100 meters (328 feet) for good performance), or when there is a poor connection resulting from a loose cable or dirty or oxidized contacts.

Noise

Local electromagnetic interference (EMI) is commonly known as noise. There are four types of noise that are most significant to data networks:

• Impulse noise that is caused by voltage fluctuations or current spikes induced on the cabling.
• Random (white) noise that is generated by many sources, such as FM radio stations, police radio, building security, and avionics for automated landing.
• Alien crosstalk, which is noise induced by other cables in the same pathway.
• Near end crosstalk (NEXT), which is noise originating from crosstalk from other adjacent cables or noise from nearby electric cables, devices with large electric motors, or anything that includes a transmitter more powerful than a cell phone.

Interface configuration errors

Many things can be misconfigured on an interface to cause it to go down, causing a loss of connectivity with attached network segments. Examples of configuration errors that affect the Physical layer include:

• Serial links reconfigured as asynchronous instead of synchronous
• Incorrect clock rate
• Incorrect clock source
• Interface not turned on

Exceeding design limits

A component may be operating suboptimally at the Physical layer because it is being utilized at a higher average rate than it is configured to operate. When troubleshooting this type of problem, it becomes evident that resources for the device are operating at or near the maximum capacity and there is an increase in the number of interface errors.

CPU overload

Symptoms include processes with high CPU utilization percentages, input queue drops, slow performance, router services such as Telnet and ping are slow or fail to respond, or there are no routing updates. One of the causes of CPU overload in a router is high traffic. If some interfaces are regularly overloaded with traffic, consider redesigning the traffic flow in the network or upgrading the hardware.

8.4.2 - Physical Layer Troubleshooting
The diagram depicts causes of physical layer problems. The seven layers of the O S I model are shown with causes pointing to the Physical Layer.
Causes
- Power-related problems
- Hardware faults
- Cabling faults
- Attenuation
- Noise (EMI)
- Interface configuration errors
- Exceeding design limits
- CPU overload

Page 3:
To isolate problems at the Physical layers do the following:

Check for bad cables or connections

Verify that the cable from the source interface is properly connected and is in good condition. Your cable tester might reveal an open wire. For example, in the figure, the Fluke CableIQ tester has revealed that wires 7 and 8 are displaying a fault. When doubting the integrity of a cable, swap suspect cables with a known working cable. If in doubt that the connection is good, remove the cable, do a physical inspection of both the cable and the interface, and then reseat the cable. Use a cable tester with suspect wall jacks to ensure that the jack is properly wired.

Check that the correct cabling standard is adhered to throughout the network

Verify that the proper cable is being used. A crossover cable may be required for direct connections between some devices. Ensure that the cable is correctly wired. For example, in the figure, the Fluke CableIQ meter has detected that although a cable was good for Fast Ethernet, it is not qualified to support 1000BASE-T because wires 7 and 8 were not correctly connected. These wires are not required for Fast Ethernet, but are required in Gigabit Ethernet.

Check that devices are cabled correctly

Check to make sure that all cables are connected to their correct ports or interfaces. Make sure that any cross-connects are properly patched to the correct location. This is where having a neat and organized wiring closet saves you a great deal of time.

Verify proper interface configurations

Check that all switch ports are set in the correct VLAN and that spanning-tree, speed, and duplex settings are correctly configured. Confirm that any active ports or interfaces are not shut down.

Check operational statistics and data error rates

Use Cisco show commands to check for statistics such as collisions and input and output errors. The characteristics of these statistics vary depending on the protocols used on the network.

8.4.2 - Physical Layer Troubleshooting
The diagram depicts the process of troubleshooting Layer 1 problems. A series of checks is provided and screenshots of the Fluke Networks CableIQ Qualification Tester is shown. One screenshot shows an Ethernet cable with wires 7 and 8 improperly connected, and the other screenshot shows failure of the 1000 Base-T test. The 100 Base-TX, 10 Base-T, and V o IP tests are successful.

Checks to perform:
- Check for bad cables or connections.
- Check if the correct cabling standard has been used.
- Check if devices are cabled incorrectly.
- Verify proper interface configurations.
- Check operational statistics and data error rates.

8.4.3 Data Link Layer Troubleshooting

Page 1:
Symptoms of Data Link Layer Problems

Troubleshooting Layer 2 problems can be a challenging process. The configuration and operation of these protocols are critical to creating a functional, well-tuned network.

Data Link layer problems cause common symptoms that assist in identifying Layer 2 issues. Recognizing these symptoms helps narrow down the number of possible causes. Common symptoms of network problems at the Data Link layer include:

No functionality or connectivity at the Network layer or above

Some Layer 2 problems can stop the exchange of frames across a link, while others only cause network performance to degrade.

Network is operating below baseline performance levels

There are two distinct types of suboptimal Layer 2 operation that can occur in a network:

• Frames take an illogical path to their destination but do arrive. An example of a problem which could cause frames to take a suboptimal path is a poorly designed Layer 2 spanning-tree topology. In this case, the network might experience high-bandwidth usage on links that should not have that level of traffic.
• Some frames are dropped. These problems can be identified through error counter statistics and console error messages that appear on the switch or router. In an Ethernet environment, an extended or continuous ping also reveals if frames are being dropped.

Excessive broadcasts

Modern operating systems use broadcasts extensively to discover network services and other hosts. Where excessive broadcasts are observed, it is important to identify the source of the broadcasts. Generally, excessive broadcasts result from one of the following situations:

• Poorly programmed or configured applications
• Large Layer 2 broadcast domains
• Underlying network problems, such as STP loops or route flapping.

Console messages

In some instances, a router recognizes that a Layer 2 problem has occurred and sends alert messages to the console. Typically, a router does this when it detects a problem with interpreting incoming frames (encapsulation or framing problems) or when keepalives are expected but do not arrive. The most common console message that indicates a Layer 2 problem is a line protocol down message.

8.4.3 - Data Link Layer Troubleshooting
The diagram depicts symptoms of Data Link Layer problems. The seven layers of the O S I model are shown with symptoms expanding from the Data Link Layer.
Symptoms:
- No connectivity at the Network Layer or above.
- No functionality at the Network Layer or above.
- Network performance below baseline.
- Excessive broadcasts.
- Console error messages.

Page 2:
Causes of Data Link Layer Problems

Issues at the Data Link layer that commonly result in network connectivity or performance problems include:

Encapsulation errors

An encapsulation error occurs because the bits placed in a particular field by the sender are not what the receiver expects to see. This condition occurs when the encapsulation at one end of a WAN link is configured differently from the encapsulation used at the other end.

Address mapping errors

In topologies such as point-to-multipoint, Frame Relay, or broadcast Ethernet, it is essential that an appropriate Layer 2 destination address be given to the frame. This ensures its arrival at the correct destination. To achieve this, the network device must match a destination Layer 3 address with the correct Layer 2 address using either static or dynamic maps.

When using static maps in Frame Relay, an incorrect map is a common mistake. Simple configuration errors can result in a mismatch of Layer 2 and Layer 3 addressing information.

In a dynamic environment, the mapping of Layer 2 and Layer 3 information can fail for the following reasons:

• Devices may have been specifically configured not to respond to ARP or Inverse-ARP requests.
• The Layer 2 or Layer 3 information that is cached may have physically changed.
• Invalid ARP replies are received because of a misconfiguration or a security attack.

Framing errors

Frames usually work in groups of 8 bit bytes. A framing error occurs when a frame does not end on an 8-bit byte boundary. When this happens, the receiver may have problems determining where one frame ends and another frame starts. Depending on the severity of the framing problem, the interface may be able to interpret some of the frames. Too many invalid frames may prevent valid keepalives from being exchanged.

Framing errors can be caused by a noisy serial line, an improperly designed cable (too long or not properly shielded), or an incorrectly configured channel service unit (CSU) line clock.

STP failures or loops

The purpose of Spanning Tree Protocol (STP) is to resolve a redundant physical topology into a tree-like topology by blocking redundant ports. Most STP problems revolve around these issues:

• Forwarding loops that occur when no port in a redundant topology is blocked and traffic is forwarded in circles indefinitely. When the forwarding loop starts, it usually congests the lowest bandwidth links along its path. If all the links are of the same bandwidth, all links are congested. This congestion causes packet loss and leads to a downed network in the affected L2 domain.
• Excessive flooding because of a high rate of STP topology changes. The role of the topology change mechanism is to correct Layer 2 forwarding tables after the forwarding topology has changed. This is necessary to avoid a connectivity outage because, after a topology change, some MAC addresses previously accessible through particular ports might become accessible through different ports. A topology change should be a rare event in a well-configured network. When a link on a switch port goes up or down, there is eventually a topology change when the STP state of the port is changing to or from forwarding. However, when a port is flapping (oscillating between up and down states), this causes repetitive topology changes and flooding.
• Slow STP convergence or reconvergence, which can be caused by a mismatch between the real and documented topology, a configuration error, such as an inconsistent configuration of STP timers, an overloaded switch CPU during convergence, or a software defect.

8.4.3 - Data Link Layer Troubleshooting
The diagram depicts causes of Data Link Layer problems. The seven layers of the O S I model are shown with causes pointing to the Data Link Layer.
Causes:
- Encapsulation errors
- Address mapping errors
- Framing errors
- STP failures or loops

Page 3:
Troubleshooting Layer 2 - PPP

The difficulty in troubleshooting Layer 2 technologies, such as PPP and Frame Relay, is the unavailability of common Layer 3 troubleshooting tools, such as ping, to assist with anything but the identification that the network is down. It is only through a thorough understanding of the protocols and their operation that a network technician is able to choose the appropriate troubleshooting methodology and Cisco IOS commands to solve the problem in an efficient manner.

Most of the problems that occur with PPP involve link negotiation. The steps for troubleshooting PPP are as follows:

Step 1. Check that the appropriate encapsulation is in use at both ends, using the show interfaces serial command. In the figure for Step 1, the command output reveals that R2 has been incorrectly configured to use HDLC encapsulation.

Step 2. Confirm that the Link Control Protocol (LCP) negotiations have succeeded by checking the output for the LCP Open message.

Click the Step 2 button in the figure.

In the figure, the encapsulation on R2 has been changed to PPP. The output of the show interfaces serial command shows the LCP Open message, which indicates that the LCP negotiations have succeeded.

Step 3. Verify authentication on both sides of the link using the debug ppp authentication command.

Click the Step 3 button in the figure.

In the figure, the output of the debug ppp authentication command shows that R1 is unable to authenticate R2 using CHAP, because the username and password for R2 have not been configured on R1.

Refer to Chapter 2, "PPP" for further details on troubleshooting PPP implementations.

8.4.3 - Data Link Layer Troubleshooting
The diagram depicts the process of troubleshooting Layer 2 P P P problems. A series of steps is provided with a network topology, and output from show commands is displayed.

Network Topology:
Router R1 interface S0/0/0 is connected to a CSU/DSU that is connected to a P P P cloud. On the other side of the diagram, router R2 interface S0/0/0 is connected to a CSU/DSU that is connected to the same P P P cloud as R1.

Step 1. Check that the appropriate encapsulation is in use at both ends. Output from the show interfaces serial 0/0/0 command on R2 indicates HDLC encapsulation, which is a problem because it should be P P P.

Step 2. Confirm that the Link Control Protocol (LCP) negotiations have succeeded. After correcting the S0/0/0 encapsulation to P P P, the output from the show interfaces serial 0/0/0 command on R2 indicates P P P encapsulation and that LCP is now open and negotiations have succeeded.

Step 3. Verify authentication on both sides of the link using the debug p p p authentication command. Output from this command on R1 indicates that username R2 is not found, and that R1 is unable to authenticate R2 using CHAP, because the username and password for R2 have not been configured on R1.

Page 4:
Troubleshooting Layer 2 - Frame Relay

Troubleshooting Frame Relay network issues can be broken down into four steps:

Step 1. Verify the physical connection between the CSU/data service unit (DSU) and the router. In the figure, the physical connections between the routers R2 and R3 and their corresponding CSU/DSU can be verified using a cable tester and by verifying that all status LEDs on the CSU/DSU unit are green. In the figure, some of the status lights for the CSU/DSU at R3 are red, indicating a potential connectivity problem between the CSU/DSU and router R3.

Step 2. Verify that the router and Frame Relay provider are properly exchanging LMI information by using the show frame-relay lmi command.

Click the Step 2 button in the figure.

In the figure, the output of the show frame-relay lmi command at R2 shows no errors or lost messages. This indicates that R2 and the Frame Relay provider switch are properly exchanging LMI information.

Step 3. Verify that the PVC status is active by using the show frame-relay pvc command.

Click the Step 3 button in the figure.

In the figure, the output of the show frame-relay pvc command at R2 verifies that the PVC status is active.

Step 4. Verify that the Frame Relay encapsulation matches on both routers with the show interfaces serial command.

Click the Step 4 button in the figure.

In the figure, the output of the show interfaces serial command at routers R2 and R3 shows that there is an encapsulation mismatch between them. R3 has been incorrectly configured to use HDLC encapsulation instead of Frame Relay.

For further details on troubleshooting Frame Relay implementations, see Chapter 3, "Frame Relay".

8.4.3 - Data Link Layer Troubleshooting
The diagram depicts the process of troubleshooting Layer 2 Frame Relay problems. A series of steps is provided with a network topology, and output from show commands is displayed.

Network Topology:
Router R2 interface S0/0/1 is connected to a CSU/DSU, which is connected to a Frame Relay provider cloud. On the other side of the diagram, router R3 interface S0/0/1 is connected to a CSU/DSU that is connected to the same Frame Relay cloud as R1.

Step 1. Verify the physical connection between the CSU/DSU and the router. In the diagram, the physical connections between routers R2 and R3 and their corresponding CSU/DSU can be verified using a cable tester and by verifying that all status L E D's on the CSU/DSU unit are green. Some of the status lights for the CSU/DSU at R3 are red, indicating a potential connectivity problem between the CSU/DSU and router R3.

Step 2. Verify that the router and Frame Relay provider are properly exchanging LMI information by using the show frame-relay lmi command. Output from this command on R2 indicates no errors and that R2 and the Frame Relay provider switch are properly exchanging LMI information. The number of status requests is equal to the number of status messages received.

Step 3. Verify that the PVC status is active by using the show frame-relay pvc command. Output from this command on R2 indicates PVC status is active.

Step 4. Verify that the Frame Relay encapsulation matches on both routers with the show interfaces serial command. Output from this command on R2 indicates that encapsulation is Frame Relay, but encapsulation on R3 is HDLC, which is a mismatch.

Page 5:
Troubleshooting Layer 2 - STP Loops

If you suspect that an STP loop is causing a Layer 2 problem, verify if the Spanning Tree Protocol is running on each of the switches. A switch should only have STP disabled if it is not part of a physically looped topology. To verify STP operation, use the show spanning-tree command on each switch. If you discover that STP is not operating, you can enable it using the spanning-tree vlan ID command.

Use these steps to troubleshoot forwarding loops:

Step 1. Identify that an STP loop is occurring.

When a forwarding loop has developed in the network, these are the usual symptoms:

• Loss of connectivity to, from, and through the affected network regions
• High CPU utilization on routers connected to affected segments or VLANs
• High link utilization (often 100 percent)
• High switch backplane utilization (compared to the baseline utilization)
• Syslog messages that indicate packet looping in the network (for example, Hot Standby Router Protocol duplicate IP address messages)
• Syslog messages that indicate constant address relearning or MAC address flapping messages
• Increasing number of output drops on many interfaces

Step 2. Discover the topology (scope) of the loop.

The highest priority is to stop the loop and restore network operation. To stop the loop, you must know which ports are involved. Look at the ports with the highest link utilization (packets per second). The show interface command displays the utilization for each interface. Make sure that you record this information before proceeding to the next step. Otherwise, it could be difficult later on to determine the cause of the loop.

Step 3. Break the loop.

Shut down or disconnect the involved ports one at a time. After you disable or disconnect each port, check whether the switch backplane utilization is back to a normal level. Document your findings. Keep in mind that some ports may not be sustaining the loop but rather are flooding the traffic arriving with the loop. When you shut down such flooding ports, you only reduce backplane utilization a small amount, but you do not stop the loop.

Step 4. Find and fix the cause of the loop.

Determining why the loop began is often the most difficult part of the process, because the reasons can vary. It is also difficult to formalize an exact procedure that works in every case. First, investigate the topology diagram to find a redundant path.

For every switch on the redundant path, check for these issues:

• Does the switch know the correct STP root?
• Is the root port identified correctly?
• Are Bridge Protocol Data Units (BPDUs) received regularly on the root port and on ports that are supposed to be blocking?
• Are BPDUs sent regularly on non-root, designated ports?

Step 5. Restore the redundancy.

After the device or link that is causing the loop has been found and the problem has been resolved, restore the redundant links that were disconnected.

We have only touched lightly on the subject of troubleshooting STP loops. Troubleshooting loops and other STP problems is complex, and a detailed discussion is beyond the scope of this course. However, if you want to learn more about troubleshooting STP problems, an excellent techinical note is available at: http://cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml#troubleshoot.

8.4.3 - Data Link Layer Troubleshooting
The diagram depicts the process of troubleshooting Layer 2 STP loop problems between switches. A series of steps is provided with a network topology.

Network Topology:
Three switches, S1, S2, and S3, are connected in a full mesh with redundant trunk links between each pair of switches. Three PC's, PC1, PC2, and PC3, are connected to switch S2.

Steps for Troubleshooting STP Loops:
Step 1. Identify that an STP loop is occurring.
Step 2. Discover the topology (scope) of the loop.
Step 3. Break the loop.
Step 4. Find and fix the cause of the loop.
Step 5. Restore the redundancy.

8.4.4 Network Layer Troubleshooting

Page 1:
Symptoms of Network Layer Problems

Network layer problems include any problem that involves a Layer 3 protocol, both routed protocols and routing protocols. This topic focuses primarily on IP routing protocols.

Problems at the Network layer can cause network failure or suboptimal performance. Network failure is when the network is nearly or completely nonfunctional, affecting all users and applications using the network. These failures are usually noticed quickly by users and network administrators, and are obviously critical to the productivity of a company. Network optimization problems usually involve a subset of users, applications, destinations, or a particular type of traffic. Optimization issues in general can be more difficult to detect and even harder to isolate and diagnose because they usually involve multiple layers or even the host computer itself. Determining that the problem is a Network layer problem can take time.

8.4.4 - Network Layer Troubleshooting
The diagram depicts symptoms of Network Layer problems. The seven layers of the O S I model are shown with symptoms expanding from the Network Layer.
Symptoms
- Network failure
- Network performance below baseline

Page 2:
Troubleshooting Layer 3 Problems

In most networks, static routes are used in combination with dynamic routing protocols. Improper configuration of static routes can lead to less than optimal routing and, in some cases, create routing loops or parts of the network to become unreachable.

Troubleshooting dynamic routing protocols requires a thorough understanding of how the specific routing protocol functions. Some problems are common to all routing protocols, while other problems are particular to the individual routing protocol.

There is no single template for solving Layer 3 problems. Routing problems are solved with a methodical process, using a series of commands to isolate and diagnose the problem.

Here are some areas to explore when diagnosing a possible problem involving routing protocols:

General network issues

Often a change in the topology, such as a down link, may have other affects on other areas of the network that might not be obvious at the time. This may include the installation of new routes, static or dynamic, removal of other routes, and so on.

Some of the things to consider include:

• Has anything in the network changed recently?
• Is there anyone currently working on the network infrastructure?

Connectivity issues

Check for any equipment and connectivity problems, including power problems such as outages and environmental problems such as overheating. Also check for Layer 1 problems, such as cabling problems, bad ports, and ISP problems.

Neighbor issues

If the routing protocol establishes an adjacency with a neighbor, check to see if there are any problems with the routers forming neighbor relationships.

Topology database

If the routing protocol uses a topology table or database, check the table for anything unexpected, such as missing entries or unexpected entries.

Routing table

Check the routing table for anything unexpected, such as missing routes or unexpected routes. Use debug commands to view routing updates and routing table maintenance.

8.4.4 - Network Layer Troubleshooting
The diagram depicts the process of troubleshooting Layer 3 problems as they relate to routing problems. A series of checks is provided.

Step 1. Check for network topology changes.
Step 2. Check for equipment and connectivity problems.
Step 3. Check routing neighbor relationships.
Step 4. Check for topology database issues.
Step 5. Check for routing table issues.

8.4.5 Transport Layer Troubleshooting

Page 1:
Common Access List Issues

Network problems can arise from Transport layer problems on the router, particularly at the edge of the network where security technologies are examining and modifying the traffic. This topic discusses two of the most commonly implemented Transport layer security technologies. They are access control lists (ACLs) and Network Address Translation (NAT).

Click the Access List Issues button in the figure.

The most common issues with ACLs are caused by improper configuration. There are eight areas where misconfigurations commonly occur:

Selection of traffic flow

The most common router misconfiguration is applying the ACL to incorrect traffic. Traffic is defined by both the router interface through which the traffic is traveling and the direction in which this traffic is traveling. An ACL must be applied to the correct interface, and the correct traffic direction must be selected to function properly. If the router is running both ACLs and NAT, the order in which each of these technologies is applied to a traffic flow is important:

• Inbound traffic is processed by the inbound ACL before being processed by outside-to-inside NAT.
• Outbound traffic is processed by the outbound ACL after being processed by inside-to-outside NAT.

Order of access control elements

The elements in an ACL should be from specific to general. Although, an ACL may have an element to specifically permit a particular traffic flow, packets will never match that element if they are being denied by another element earlier in the list.

Implicit deny all

In a situation where high security is not required on the ACL, forgetting about this implicit access control element may be the cause of an ACL misconfiguration.

Addresses and wildcard masks

Complex wildcard masks provide significant improvements in efficiency, but are more subject to configuration errors. An example of a complex wildcard mask is using the address 10.0.32.0 and wildcard mask 0.0.32.15 to select the first 15 host addresses in either the 10.0.0.0 network or the 10.0.32.0 network.

Selection of Transport layer protocol

When configuring ACLs, it is important that only the correct Transport layer protocols be specified. Many network engineers, when unsure if a particular traffic flow uses a TCP port or a UDP port, configure both. Specifying both opens a hole through the firewall, possibly giving intruders an avenue into the network. It also introduces an extra element into the ACL, so the ACL takes longer to process, introducing more latency into network communications.

Source and destination ports

Properly controlling the traffic between two hosts requires symmetric access control elements for inbound and outbound ACLs. Address and port information for traffic generated by a replying host is the mirror image of address and port information for traffic generated by the initiating host.

Use of the established keyword

The established keyword increases the security provided by an ACL. However, if the keyword is applied to an outbound ACL, unexpected results may occur.

Uncommon protocols

Misconfigured ACLs often cause problems for less common protocols than TCP and UDP. Uncommon protocols that are gaining popularity are VPN and encryption protocols.

Troubleshooting Access Control Lists

A useful command for viewing ACL operation is the log keyword on ACL entries. This keyword instructs the router to place an entry in the system log whenever that entry condition is matched. The logged event includes details of the packet that matched the ACL element.

The log keyword is especially useful for troubleshooting and also provides information on intrusion attempts being blocked by the ACL.

8.4.5 - Transport Layer Troubleshooting
The diagram depicts symptoms of Transport Layer problems. The seven layers of the O S I model are shown with symptoms expanding from the Transport Layer.
Symptoms
-Intermittent network problems
-Security problems
-Address translation problems
-Problems with specific traffic types

The diagram depicts Transport Layer problems as they relate to common access list issues. A series of checks is provided.
Common ACL Issues
-Applied to incorrect traffic
-Incorrect control element order
-Implicit "deny any any"
-Addresses and wildcard masks
-TCP/UDP selection
-Source and destination ports
-Use of the established keyword
-Uncommon protocols

Page 2:
Common NAT Issues

The biggest problem with all NAT technologies is interoperability with other network technologies, especially those that contain or derive information from host network addressing in the packet. Some of these technologies include:

• BOOTP and DHCP - Both protocols manage the automatic assignment of IP addresses to clients. Recall that the first packet that a new client sends is a DHCP-Request broadcast IP packet. The DHCP-Request packet has a source IP address of 0.0.0.0. Because NAT requires both a valid destination and source IP address, BOOTP and DHCP can have difficulty operating over a router running either static or dynamic NAT. Configuring the IP helper feature can help solve this problem.
• DNS and WINS - Because a router running dynamic NAT is changing the relationship between inside and outside addresses regularly as table entries expire and are recreated, a DNS or WINS server outside the NAT router does not have an accurate representation of the network inside the router. Configuring the IP helper feature can help solve this problem.
• SNMP - Similar to DNS packets, NAT is not able to alter the addressing information stored in the data payload of the packet. Because of this, an SNMP management station on one side of a NAT router may not be able to contact SNMP agents on the other side of the NAT router. Configuring the IP helper feature can help solve this problem.
• Tunneling and encryption protocols - Encryption and tunneling protocols often require that traffic be sourced from a specific UDP or TCP port, or use a protocol at the Transport layer that cannot be processed by NAT. For example, IPsec tunneling protocols and generic routing encapsulation protocols used by VPN implementations cannot be processed by NAT. If encryption or tunneling protocols must be run through a NAT router, the network administrator can create a static NAT entry for the required port for a single IP address on the inside of the NAT router.

If encryption or tunneling protocols must be run through a NAT router, the network administrator can create a static NAT entry for the required port for a single IP address on the inside of the NAT router.

One of the more common NAT configuration errors is forgetting that NAT affects both inbound and outbound traffic. An inexperienced network administrator might configure a static NAT entry to redirect inbound traffic to a specific inside backup host. This static NAT statement also changes the source address of traffic from that host, possibly resulting in undesirable and unexpected behaviors or in suboptimal operation.

Improperly configured timers can also result in unexpected network behavior and suboptimal operation of dynamic NAT. If NAT timers are too short, entries in the NAT table may expire before replies are received, so packets are discarded. The loss of packets generates retransmissions, consuming more bandwidth. If timers are too long, entries may stay in the NAT table longer than necessary, consuming the available connection pool. In busy networks, this may lead to memory problems on the router, and hosts may be unable to establish connections if the dynamic NAT table is full.

Refer to Chapter 7, "IP Addressing Services" for further details on troubleshooting NAT configuration.

8.4.5 - Transport Layer Troubleshooting
The diagram depicts Transport Layer problems as they relate to common NAT issues, which include:
- Interoperability issues
- Incorrect static NAT
- Improperly configured NAT timers

8.4.6 Application Layer Troubleshooting

Page 1:
Application Layer Overview

Most of the Application layer protocols provide user services. Application layer protocols are typically used for network management, file transfer, distributed file services, terminal emulation, and e-mail. However, new user services are often added, such as VPNs, VoIP, and so on.

The most widely known and implemented TCP/IP Application layer protocols include:

• Telnet - Enables users to establish terminal session connections with remote hosts.
• HTTP - Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web.
• FTP - Performs interactive file transfers between hosts.
• TFTP - Performs basic interactive file transfers typically between hosts and networking devices.
• SMTP - Supports basic message delivery services.
• POP - Connects to mail servers and downloads e-mail.
• Simple Network Management Protocol (SNMP) - Collects management information from network devices.
• DNS - Maps IP addresses to the names assigned to network devices.
• Network File System (NFS) - Enables computers to mount drives on remote hosts and operate them as if they were local drives. Originally developed by Sun Microsystems, it combines with two other Application layer protocols, external data representation (XDR) and remote-procedure call (RPC), to allow transparent access to remote network resources.

Click the Application Protocols and Ports button in the figure to view a list of application protocols and their associated ports.

8.4.6 - Application Layer Troubleshooting
The diagram depicts an Application Layer overview. The seven layers of the O S I model are shown with the top three layers highlighted: Application, Presentation, and Session. The TCP/IP model is shown next to it. The TCP/IP layers are mapped to their corresponding O S I layers. The TCP/IP protocols that operate at each layer in the TCP/IP model are identified. Also shown is a tabular listing of application protocols and ports.

Application Layer Overview:
TCP Application Layer: Corresponds to O S I layers 5, 6, and 7. Applications and protocols include HTTP, Telnet, FTP, TFTP, SMTP, POP, I MAP, SNMP, NTP, DNS, and NNTP. NFS, XDR, and RPC are also shown at this layer.

TCP Transport Layer: Corresponds to O S I Layer 4. Protocols include TCP and UDP.

TCP Internet Layer: Corresponds to O S I Layer 3. Protocols include routing protocols, IP, and ICMP.

TCP Network Access Layer: Corresponds to O S I layers 1 and 2. Protocols include ARP and RARP. The Physical Layer is not specified with the TCP/IP model.

Application Protocols and Ports

Application: WWW browser
Protocol and Port: HTTP (TCP port 80)
Description: HTTP is used by Web browsers and servers to transfer the files that make up web pages.

Application: File transfer
Protocol and Port: FTP (TCP ports 20 and 21)
Description: FTP provides a way to move files between computer systems.

Application: Terminal emulation
Protocol and Port: Telnet (TCP port 23)
Description: Telnet provides terminal emulation services over a reliable TCP stream.

Application: Electronic mail
Protocol and Port: POP3 (TCP port 110)
SMTP (TCP port 25)
I MAP 4 (TCP port 143)
Description: Simple Mail Transfer Protocol (SMTP) transfers electronic mail between mail servers. It is used by mail clients to send mail. Mail clients use either the Post Office Protocol version 3 (POP3) or the Internet Message Access Protocol (I MAP) to receive mail.

Application: Network management
Protocol and Port: SNMP (UDP port 161)
Description: The Simple Network Management Protocol (SNMP) reports anomalous network conditions and sets network threshold values.

Application: Distributed File Service
Protocol and Port: X Windows (UDP ports 6000-6063)
NFS, XDR, RPC (UDP port 111)
Description: X Windows is a popular protocol that permits intelligent terminals to communicate with remote computers as if they were directly attached. NFS, external data representation (XDR), and remote-procedure call (RPC) combine to allow transparent access to remote network resources.

Page 2:
Symptoms of Application Layer Problems

Application layer problems prevent services from being provided to application programs. A problem at the Application layer can result in unreachable or unusable resources when the Physical, Data Link, Network, and Transport layers are functional. It is possible to have full network connectivity, but the application simply cannot provide data.

Another type of problem at the Application layer occurs when the Physical, Data Link, Network, and Transport layers are functional, but the data transfer and requests for network services from a single network service or application do not meet the normal expectations of a user.

A problem at the Application layer may cause users to complain that the network or the particular application that they are working with is sluggish or slower than usual when transferring data or requesting network services.

The figure shows some of the possible symptoms of Application layer problems.

8.4.6 - Application Layer Troubleshooting
The diagram depicts symptoms of Application Layer problems. The seven layers of the O S I model are shown with symptoms expanding from the top three O S I layers: Application, Presentation, and Session.
Symptoms
- User complaints about slow application performance.
- Application error messages.
- Console error messages.
- System log file messages.
- Network management system alarms.

Page 3:
Troubleshooting Application Layer Problems

The same general troubleshooting process that is used to isolate problems at the lower layers can also be used to isolate problems at the Application layer. The concepts are the same, but the technological focus has shifted to involve things such as refused or timed out connections, access lists, and DNS issues.

The steps for troubleshooting Application layer problems are as follows:

Step 1. Ping the default gateway.

If successful, Layer 1 and Layer 2 services are functioning properly.

Step 2. Verify end-to-end connectivity.

Use an extended ping if attempting the ping from a Cisco router. If successful, Layer 3 is operating correctly. If Layers 1-3 are functioning properly, the issue must exist at a higher layer.

Step 3. Verify access list and NAT operation.

To troubleshoot access control lists, use the following steps:

• Use the show access-list command. Are there any ACLs that could be stopping traffic? Notice which access lists have matches.
• Clear the access-list counters with the clear access-list counters command and try to establish a connection again.
• Verify the access-list counters. Have any increased? Should they increase?

To troubleshoot NAT, use the following steps:

• Use the show ip nat translations command. Are there any translations? Are the translations as expected?
• Clear the NAT translations with the clear ip nat translation * command and try to access the external resource again.
• Use the debug ip nat command and examine the output.
• Look at the running configuration file. Are the ip nat inside and ip nat outside commands located on the right interfaces? Is the NAT pool correctly configured? Is the ACL correctly identifying the hosts?

If the ACLs and NAT are functioning as expected, the problem must lie in a higher layer.

Step 4. Troubleshoot upper layer protocol connectivity.

Even though there may be IP connectivity between a source and a destination, problems may still exist for a specific upper layer protocol, such as FTP, HTTP, or Telnet. These protocols ride on top of the basic IP transport but are subject to protocol-specific problems relating to packet filters and firewalls. It is possible that everything except mail works between a given source and destination.

Troubleshooting an upper layer protocol connectivity problem requires understanding the process of the protocol. This information is usually found in the latest RFC for the protocol or on the developer web page.

8.4.6 - Application Layer Troubleshooting
The diagram depicts the process of troubleshooting Application Layer problems. A series of checks is provided.

Step 1. Ping the default gateway.
Step 2. Verify end-to-end connectivity.
Step 3. Verify access list and NAT operation.
Step 4. Troubleshoot upper layer protocol connectivity.

Page 4:
Correcting Application Layer Problems

The steps for correcting Application layer problems are as follows:

Step 1: Make a backup. Before proceeding, ensure that a valid configuration has been saved for any device on which the configuration may be modified. This provides for recovery to a known initial state.

Step 2: Make an initial hardware or software configuration change. If the correction requires more than one change, make only one change at a time.

Step 3: Evaluate and document each change and its results. If the results of any problem-solving steps are unsuccessful, immediately undo the changes. If the problem is intermittent, wait to see if the problem occurs again before evaluating the effect of any change.

Step 4: Determine if the change solves the problem. Verify that the change actually resolves the problem without introducing any new problems. The network should be returned to the baseline operation, and no new or old symptoms should be present. If the problem is not solved, undo all the changes. If new or additional problems are discovered, modify the correction plan.

Step 5: Stop when the problem is solved. Stop making changes when the original problem appears to be solved.

Step 6: If necessary, get assistance from outside resources. This may be a co-worker, a consultant, or Cisco Technical Assistance Center (TAC). On rare occasions, a core dump may be necessary, which creates output that a specialist at Cisco Systems can analyze.

Step 7: Document. Once the problem is resolved, document the solution.

8.4.6 - Application Layer Troubleshooting
The diagram depicts a flowchart describing the process of correcting Application Layer problems.

Step 1: Make a backup.

Step 2: Make only one change at a time.

Step 3: Evaluate and document the results of the change.

Step 4: Undo changes not resulting in success.

Step 5: If the problem is solved, document the resolution. Otherwise, re-evaluate the plan or seek a second opinion.

Page 5:
To successfully complete this activity, you need your final documentation for the PT Activity 8.1.2: Network Discovery and Documentation you completed previously in this chapter. This documentation should have an accurate topology diagram and addressing table. If you do not have this documentation, then ask your Instructor for accurate versions.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.4.6 - Application Layer Troubleshooting
Link to Packet Tracer Exploration: Troubleshooting Network Problems

8.5 Chapter Labs

8.5.1 Troubleshooting Enterprise Networks 1

Page 1:
You have been asked to correct configuration errors in the company network. For this lab, do not use login or password protection on any console lines to prevent accidental lockout. Use ciscoccna for all passwords in this scenario.

Note: Because this lab is cumulative, you will be using all the knowledge and troubleshooting techniques that you have acquired from the previous material to successfully complete this lab.

8.5.1 - Troubleshooting Enterprise Networks 1
Link to Hands-on Lab: Troubleshooting Enterprise Networks 1

Page 2:
This activity is a variation of Lab 8.5.1. Packet Tracer may not support all the tasks specified in the hands-on lab. This activity should not be considered equivalent to completing the hands-on lab. Packet Tracer is not a substitute for a hands-on lab experience with real equipment.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.5.1 - Troubleshooting Enterprise Networks 1
Link to Packet Tracer Exploration: Troubleshooting Enterprise Networks 1

8.5.2 Troubleshooting Enterprise Networks 2

Page 1:
For this lab, do not use login or password protection on any console lines to prevent accidental lockout. Use ciscoccna for all passwords in this lab.

Note: Because this lab is cumulative, you will be using all the knowledge and troubleshooting techniques that you have acquired from the previous material to successfully complete this lab.

8.5.2 - Troubleshooting Enterprise Networks 2
Link to Hands-on Lab: Troubleshooting Enterprise Networks 2

Page 2:
This activity is a variation of Lab 8.5.2. Packet Tracer may not support all the tasks specified in the hands-on lab. This activity should not be considered equivalent to completing the hands-on lab. Packet Tracer is not a substitute for a hands-on lab experience with real equipment.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.5.2 - Troubleshooting Enterprise Networks 2
Link to Packet Tracer Exploration: Troubleshooting Enterprise Networks 2

8.5.3 Troubleshooting Enterprise Networks 3

Page 1:
For this lab do not use login or password protection on any console lines to prevent accidental lockout. Use ciscoccna for all passwords in this scenario.

Note: Because this lab is cumulative, you will be using all the knowledge and troubleshooting techniques that you have acquired from the previous material to successfully complete this lab.

8.5.3 - Troubleshooting Enterprise Networks 3
Link to Hands-on Lab: Troubleshooting Enterprise Networks 3

Page 2:
This activity is a variation of Lab 8.5.3. Packet Tracer may not support all the tasks specified in the hands-on lab. This activity should not be considered equivalent to completing the hands-on lab. Packet Tracer is not a substitute for a hands-on lab experience with real equipment.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.5.3 - Troubleshooting Enterprise Networks 3
Link to Packet Tracer Exploration: Troubleshooting Enterprise Networks 3

8.6 Chapter Summary

8.6.1 Chapter Summary

Page 1:
In this chapter, you learned that a network baseline is required for effective troubleshooting. Creating a baseline begins with ensuring that network documentation is up to date and accurate. Proper network documentation includes a network configuration table for all devices and a topology diagram that reflects the current state of the network. When the network has been fully documented, a baseline measurement of network performance should be carried out over a period of several weeks to a month to establish the personality of the network. The first baseline is created during a time of stable and normal operation.

The most effective way to troubleshoot is with a systematic approach using a layered model, such as the OSI model or the TCP/IP model. Three methods commonly used to troubleshoot include bottom up, top down, and divide and conquer. Each method has its advantages and disadvantages, and you learned the guidelines for choosing which method to apply. You also learned about the various software and hardware tools that are used by network professionals to gather symptoms and troubleshoot network problems.

Although they operate primarily at the first three OSI layers, WANs have implementation issues that can affect the operation of the rest of the network. You learned about some of the considerations for implementing WANs and common problems that WANs introduce into networks, such as security threats, bandwidth problems, latency, and QoS issues.

Finally, you explored the symptoms and causes of common problems at each of the OSI layers and the steps for troubleshooting them.

8.6.1 - Summary and Review
In this chapter, you have learned to:
- Establish and document a network baseline.
- Describe the various troubleshooting methodologies and troubleshooting tools.
- Describe the common issues that occur during WAN implementation.
- Identify and troubleshoot common enterprise network implementation issues using a layered model approach.

Page 2:

8.6.1 - Summary and Review
This is a review and is not a quiz. Questions and answers are provided.
Question One. Explain the function and contents of network documentation, including router, switch, and end-user documentation, as well as network topology diagrams.
Answer:
Router Documentation:
- The router documentation should include the router names, model designation, location in the enterprise (building, floor, room, rack, and panel), configured interfaces, Data Link Layer addresses, Network Layer addresses, routing protocols configured, and any additional important information about the device.

Switch Documentation:
- The switch documentation should include the switch names, model designation, location in the enterprise (building, floor, room, rack, and panel), management IP address, port names and status, speed, duplex, STP state, PortFast setting, trunk status, L2 or L3 EtherChannel, V LAN ID's, and any additional important information about the device.

End-user Documentation:
- The end-user documentation should include the server names and function, OS version, IP address, gateways, DNS server, network application, and any additional important information about the device.

Logical Network Topology Diagram:
- Graphical representation that uses symbols to identify each network device and how it is interconnected.
- Also details the logical architecture, including interface types and numbers, IP addresses, subnet masks, routing protocols, A S domains, and any additional important information, such as DLCI numbers and Layer 2 protocol.

Question Two. Explain the recommended steps for planning the first network baseline.
Answer:
Step 1: Determine which types of data to collect.
- Start out simply by selecting a few variables that represent the defined policies and fine-tune along the way.
- Some good starting measures are interface utilization and CPU utilization.

Step 2: Identify devices and ports of interest.
- Devices and ports of interest must be identified, such as network device ports that connect to other network devices, servers, key users, and anything else considered critical to the operation.

Step 3: Determine the baseline duration.
- This period should be at least seven days to capture daily or weekly trends and should last two to four weeks.
- Do not perform a baseline measurement during times of unique traffic patterns.
- Should be conducted on a regular basis, such as an annual analysis of the entire network, or baseline different sections of the network on a rotating basis.

Question Three. Explain the three stages of the general troubleshooting process.
Answer:
Stage 1: Gather symptoms:
- Symptoms may appear in many different forms, including alerts from the network management system, console messages, and user complaints.
- Gather and document symptoms from the network, end systems, and users.
- Determine which network components have been affected and how the functionality of the network has changed compared to the baseline.

Stage 2: Isolate the problem:
- The problem is not truly isolated until a single problem, or a set of related problems, is identified.
- Examine the characteristics of problems at the logical layers of the network so that the most likely cause can be selected.
- Depending on the problem characteristics identified, gather and document more symptoms.

Stage 3: Correct the problem:
- Work to correct the problem by implementing, testing, and documenting a solution.

Question Four. Explain the three main methods for troubleshooting.
Answer:
Bottom-Up Troubleshooting Method:
- Start with the physical components of the network and then work up through the layers of the O S I model until the cause of the problem is identified.
- Good approach to use when the problem is suspected to be physical.

Top-Down Troubleshooting Method:
- Examine the end-user application first.
- Analysis continues downward from the upper layers of the O S I model until the cause of the problem has been identified.

Divide-And-Conquer Troubleshooting Method:
- Select a layer and test in both directions from the layer.
- If you can verify that a layer is functioning properly, it is typically a safe assumption that the layers below it are functioning.
- If a layer is not functioning properly, gather symptoms of the problem at that layer and work your way down.

Question Five. Explain the six steps for designing or modifying a WAN.
Answer:
Step 1 - Locate LAN's:
- Establish the source and destination endpoints that connect via the WAN.

Step 2 - Analyze Traffic:
- Find out which data traffic must be carried, its origin, its destination, bandwidth requirements, latency, and jitter tolerance.

Step 3 - Plan the Topology:
- Identify the various endpoints, geographic considerations, and the requirement for availability.

Step 4 - Estimate the Required Bandwidth:
- In consideration of the endpoints and the links chosen, the necessary bandwidth can be estimated.

Step 5 - Choose the WAN Technology:
- When the bandwidth availability is determined, suitable link technologies must be selected.

Step 6 - Evaluate Costs:
- Determine installation and operational costs for the WAN and compare them with the business need driving the WAN implementation.

Question Six. List three things to check when troubleshooting Layer 1 problems.
Answer:
Check for bad cables or connections:
- Use a cable tester to verify that the cable from the source interface is properly connected and is in good condition.
- When doubting the integrity of a cable, swap suspect cables with a known working cable.

Check that the correct cabling standard is adhered to throughout the network:
- Verify that the proper cable is being used for the connection.

Check that devices are cabled correctly:
- Verify that all cables are connected to their correct ports or interfaces.

Verify proper interface configurations:
- Check that all switch ports are set in the correct V LAN, and that spanning tree, speed, and duplex settings are correctly configured.
- Confirm that any active ports or interfaces are not shutdown.

Check operations statistics and data error rates:
- Use Cisco show commands to check for statistics such as collisions and input and output errors.
- The characteristics of these statistics vary depending on the protocols used on the network.

Question Seven. List three causes of Layer 2 problems.
Answer:

- Address mapping errors
- Framing errors
- STP failures or loops

Page 3:
In this comprehensive CCNA skills activity, the XYZ Corporation uses a combination of Frame Relay and PPP for WAN connections. The HQ router provides access to the server farm and the Internet through NAT. HQ also uses a basic firewall ACL to filter inbound traffic. Each Branch router is configured for inter-VLAN routing and DHCP. Routing is achieved through EIGRP as well as static and default routes. The VLANs, VTP, and STP are configured on each of the switched networks. Port security is enabled and wireless access is provided. Your job is to successfully implement all of these technologies, leveraging what you have learned over the four Exploration courses leading up to this culminating activity.

Detailed instructions are provided within the activity as well as in the PDF link below.

Activity Instructions (PDF)

Click the Packet Tracer icon for more details.

8.6.1 - Summary and Review
Link to Packet Tracer Exploration: CCNA Skills Integration Challenge

8.7 Chapter Quiz

8.7.1 Chapter Quiz

Page 1:

8.7.1 - Chapter Quiz
1.Match the items to the appropriate diagram type.
Items:
Cable type
IP address and subnet
Connection type
Device ID
Operating system version
Device type and model
Routing protocols
Connector type

Diagrams:
Physical diagram
Logical diagram

2.What is one symptom of a Physical Layer problem?
A.High CPU utilization
B.Excessive broadcasts
C.Slow STP convergence
D.Routing loops

3.A network administrator has received the output "Serial0 is up, line protocol is down" from the show interface s0 command. At what layer is this problem most likely being caused?
A.Physical Layer
B.Data Link Layer
C.Network Layer
D.Transport Layer

4.Which statement is true concerning network models?
A.While similar to the O S I model in construction, the TCP/IP model has more layers.
B.The Network Access layer in the O S I model incorporates both Physical and Data Link layers in the TCP/IP model.
C.Both users and Application Layer processes interact with software applications that contain a communications component in the O S I model.
D.TCP/IP communications only relate to the TCP/IP model.

5.Which three protocols could be involved in Network Layer problems? (Choose three.)
A.DNS
B.EIGRP
C.IP
D.RIP
E.TCP
F.UDP

6.Match the Application Layer protocol with the port number it is commonly associated with.
Application Layer Protocols:
FTP
HTTP
POP3
SMTP
SNMP
Telnet

Port Numbers:
20 and 21
23
25
80
110
161

7.A technician has been asked to troubleshoot a simple network problem that seems to be caused by software. Which troubleshooting approach should be suggested?
A.Bottom-up
B.Top-down
C.Divide and conquer
D.Middle-out

8.Which three questions are appropriate to ask when gathering information from a user? (Choose three.)
A.What does work?
B.Who did you call after the problem appeared?
C.When was the problem first noticed?
D.When does the problem occur?
E.What is your password?
F.What did you do after the problem occurred?

9.Which network troubleshooting tool can be used to test the physical medium for defects, such as near-end crosstalk?
A.Cable analyzer
B.Cable tester
C.Digital multimeter
D.Baselining tool

10.Which three documents are needed to efficiently diagnose and correct network problems? (Choose three.)
A.Network management command reference
B.Network configuration tables
C.Network device installation guide
D.Network topology diagrams
E.End-system configuration tables
F.Service provider documentation

11.What are three steps for establishing a network baseline? (Choose three.)
A.Determine the type of network management traffic to be collected and evaluated.
B.Determine the types of data to be collected and evaluated.
C.Identify devices and ports to be monitored.
D.Identify the virtual interfaces, V LAN's, and virtual routing tables to be monitored.
E.Determine the number of baseline tests to establish a typical picture of the network.
F.Determine the duration for baseline testing to establish a typical picture of the network.

12.What is associated with the first step in correcting Application Layer problems?
A.Analyzing existing symptoms.
B.Making a backup of configurations.
C.Making the initial hardware or software changes.
D.Pinging the default gateway to verify Layer 1 to Layer 3 functionality.

source: www.cisco.com

Read More!