Foglight APM for Real User Experience 5.9.11 - Installation and Setup Guide

Table 15. Issues detected for physical appliances
Issue Detected	Recommended Action
No link detected	Check cables. Verify that the appliance monitoring ports are connected to a network tap.
Low link speed	The link speed is showing 100 Mbps, which is unusually low for monitoring high-end servers. Check if the switch port is configured for gigabit.
No packets found	Check cables and port connections between the appliance and the network tap. On the tap, verify packet flows to the appliance.
No TCP packets found	Check cables and port connections between the appliance and the network tap. On the tap, verify packet flows to the appliance.
Ingress error	Ingress traffic is missing for some servers. Log in to Foglight, navigate to APM > Configure > Traffic Capture > Monitored IP Addresses, and click Discover. The results show which servers are missing ingress traffic. Verify that the network tap's ingress and egress ports are connected to the appliance.
Egress error	Egress traffic is missing for some servers. Log in to Foglight, navigate to APM > Configure > Traffic Capture > Monitored IP Addresses, and click Discover. The results show which servers are missing egress traffic. Verify that the network tap's ingress and egress ports are connected to the appliance.
VIP detected	A virtual IP addressing scheme is being used to mask IP addresses. Configure web servers to send an HTTP header containing the real server IP address. Configure this HTTP header name as a Server Identifier in Foglight. For instructions, see “Managing identifiers for virtual addressing schemes” in the Foglight APM Administration and Configuration Guide.
Duplicate packets	Examine traffic collection points to ensure that the appliance is not receiving duplicate packets. Check if the appliance is monitoring both sides of a load balancer. Check if the appliance is monitoring redundant links that are both sending the same data. If using an intelligent tap, consider enabling packet deduplication.
Unsupported SSL	Web servers appear to be using an unsupported key exchange algorithm (Diffie-Hellman). On each web server or other SSL termination point, disable all use of Diffie-Hellman (DH/ADH/DHE).
NIC packet drops	The capture system may be overloaded. Scale back the amount of traffic going to the appliance or split the load between multiple appliances. Consider using pre-filtering capabilities in an intelligent tap or switch.
Missing segments	Holes are detected in the TCP streams. If you are using a SPAN port for monitoring purposes, use a full-duplex network tap instead. If you are using a network tap, check the tap for overload. Make sure ingress and egress traffic are fed into two separate monitoring NICs on the appliance. Do not aggregate ingress and egress traffic into one monitoring port. For more information, see Understanding packet drops with SPAN ports and aggregation network taps.
Checksum errors	Check if switches are reporting similar TCP errors. High percentages indicate that something in the network path is causing bit errors. Check cabling and connections.
Jumbo frames	This is for information purposes only; typically no action is required. Sniffers support standard jumbo frame sizes by default (16110 bytes).
Duplicate client ip headers	More than one X-Forwarded-For header has been detected in the traffic. This could result in incorrect geographic location identification. Check proxies or reverse proxies. Some proxies incorrectly add a new X-Forwarded-For header instead of appending to an existing header. For more information about proxies, see “Managing identifiers for virtual addressing schemes” in the Foglight APM Administration and Configuration Guide.
VLAN tagging	Some types of VLAN tagging can prevent Sniffers from capturing traffic when the Sniffer is configured to do traffic filtering in the kernel. In this case, log in to Foglight, navigate to the APM > Capture > Traffic Capture > Sniffers, edit the Sniffer, and clear the Perform Traffic Filtering in Kernel check box.

Table 16. Issues detected for virtual appliances

Issue Detected

Recommended Action

No link detected

Try the following actions:

•

Verify that the appliance monitoring ports have been attached to a VM port group inside of a virtual switch.

•

Verify that the network ports on the physical host have cables attached to a switch or tap.

No packets found

Try the following actions:

•

Verify that the Virtual Machine Port Group is set to Accept Promiscuous Mode.

•

Verify that the Sniffer is located on the same ESXi® host as the VMs to be monitored.

•

Verify that the Sniffer monitoring ports are assigned to a Virtual Machine Port Group configured for promiscuous mode. Alternatively, verify that all VMs on the ESXi host are attached to same port group as the Sniffer. This second deployment option is necessary to handle cases where VMs are attached to a VM Network that is not attached to a physical NIC, which may occur for multitiered applications where the back end tiers only communicate with VMs within the ESXi host.

No TCP packets found

When a Sniffer’s monitoring ports are connected to a regular vSwitch port that is part of a VLAN, the Sniffer receives broadcast/multicast packets for the VLAN, but no end-to-end TCP traffic. Verify that the Sniffer’s monitoring ports are attached to a port group with promiscuous mode enabled.

VIP detected

A virtual IP addressing scheme is being used to mask IP addresses. Configure web servers to send an HTTP header containing the real server IP address. Configure this HTTP header name as a Server Identifier in Foglight. For instructions, see “Managing identifiers for virtual addressing schemes” in the Foglight APM Administration and Configuration Guide.

Duplicate packets

Try the folowing actions:

•

Examine traffic collection points to ensure that the appliance is not receiving duplicate packets.

•

Check if the appliance is monitoring redundant networks that are both sending the same data.

•

Verify that the Sniffer’s monitoring NICs are not attached to the same VM Network.

Unsupported SSL

Web servers appear to be using an unsupported key exchange algorithm (Diffie-Hellman). On each web server or other SSL termination point, disable all use of Diffie-Hellman (DH/ADH/DHE).

NIC packet drops

The capture system may be overloaded. Scale back the amount of traffic going to the appliance or split the load between multiple

appliances. Try reserving CPU capacity for the Sniffer VM.

Missing segments

Check the following:

•

Can be caused by using VMware® RSPAN (Remote SPAN) or ERSPAN (Encapsulated Remote SPAN). Try deploying Sniffers on each ESX® host instead of using RSPAN.

•

The capture system may be overloaded. Scale back the amount of traffic going to the appliance or split the load between multiple appliances. Try reserving CPU capacity for the Sniffer VM.

Checksum errors

Check if switches are reporting similar TCP errors. High percentages indicate that something in the network path is causing bit errors. Check cabling and connections.

Jumbo frames

This is for information purposes only; typically no action is required. Sniffers support standard jumbo frame sizes by default (16110 bytes).

Duplicate client ip headers

More than one X-Forwarded-For header has been detected in the traffic. This could result in incorrect geographic location identification. Check proxies or reverse proxies. Some proxies incorrectly add a new X-Forwarded-For header instead of appending to an existing header. For more information about proxies, see “Managing identifiers for virtual addressing schemes” in the Foglight APM Administration and Configuration Guide.

VLAN tagging

Some types of VLAN tagging can prevent Sniffers from capturing traffic when the Sniffer is configured to do traffic filtering in the kernel. In this case, log in to Foglight, navigate to the APM > Configure > Traffic Capture > Sniffers, edit the Sniffer, and clear the Perform Traffic Filtering in Kernel check box.

Understanding packet drops with SPAN ports and aggregation network taps

SPAN ports and aggregation network taps are often used for small-scale monitoring purposes. While you can use these devices with Foglight APM, in general they are not reliable enough to support large-scale, enterprise-critical monitoring solutions. The following sections explain why these devices are not recommended.

SPAN ports — SPAN ports (also known as diagnostic ports) can be found on most switches and routers. Network traffic flowing through a switch can be mirrored to the SPAN port, which in turn connects to a monitoring solution. Switch providers do not guarantee that 100% of the traffic reaches the SPAN port. Spikes in traffic through the switch can result in a significant number of dropped packets (from 5% to as much as 20% of all packets). Dropped packets translate to poor data quality in Foglight APM.

Aggregation network taps — Aggregation network taps take a full-duplex link and merge the ingress and egress streams into a one half-duplex stream. While they drop substantially fewer packets than SPAN ports (due to the use of buffering), aggregation taps are not guaranteed to capture 100% of the traffic routed through them. Aggregation taps generally drop less than 1% of packets, but a 1% drop rate (potentially higher in some instances) has a significant impact on the quality of the data in Foglight APM.

For information about the recommended full-duplex network tap, see Network taps.

Expanding full partitions on virtual appliances

Foglight Management Server, Archivers, and Sniffers run on separate virtual devices, each with pre-configured virtual disk sizes and partition sizes. By default, the disk is partitioned using all the available space, so if the disk size is 200 GB, the partition is also 200 GB. If a partition fills during use, you can increase the size of the disk and expand the size of the partition.


	IMPORTANT: If you are changing the virtual disk size before powering on the appliance for the first time, you only need to complete the steps in Expanding the size of virtual disks.The partition size is automatically set to the same size as the disk.