WireShark Network and Replication Troubleshooting Guide (4035207)

This article describes the network testing technique and steps necessary to perform a network test using the WireShark network protocol analyzer.

You may receive the following error during replication/transfers.

ERROR 2013-02-26T21:31:26 [495] – Replay.Core.Implementation.Replication.Jobs.ReplicationJob ()
Failed to sync replication with remote slave core.
System.AggregateException: One or more errors occurred. —> System.AggregateException: One or more errors occurred. —> System.AggregateException: One or more errors occurred. —> System.AggregateException: One or more errors occurred. —> System.AggregateException: One or more errors occurred. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Performing a complete network diagnostics test can help to detect the cause of the problem and provide an adequate solution.

You can perform a network analysis using free WireShark software. To download this software, click here.

To perform the network diagnosis, please follow the below steps.

1. Download and install WireShark.
2. Launch WireShark. (WireShark Legacy if you have a newer version)
3. Select Capture and then click Interfaces.
   • Place a check in the box next to the interface that is being used for communication to the target Core.
   • Click Options.
     • Place a check in the box next to Use promiscuous mode on all interfaces.
     • In the Display Options menu, ensure that all boxes are checked.
     • In the Name Resolution menu, ensure that all boxes are checked.
     • If you are only capturing replication traffic, enter "tcp port 8006" in the capture filter section.
     • Click Start to begin the capture.
4. Force replication (or force a snapshot for the replicated agent and wait until replication starts).
5. Discontinue working in WireShark for approximately 2 hours.
6. After 2 hours, select Capture and click Stop.
7. Click File and select Save As.
8. Name the file BITC.pcap and click Save.
9. Take a new AAinfo and add the BITC.pcap in the customer logs section.
10. Send this link to your support engineer.

If you have have not used a capture filter and would like to view the details of the capture as it pertains to AppAssure Replication, filter the results as described below.

1. Open the capture file that was just created.
   Replication between Cores is performed through the TCP protocol and uses the 8006 port. We will adjust the display filter to only show TCP activity on port 8006.
2. In the Filter field , expand TCP – Transmission Control Protocol and select tcp.port – Source or Destination Port.
3. Select the “==“ operator in the Relation column.
4. Specify 8006 in the Value field.
5. Select OK.
6. To apply this filter, click Apply.

• TCP Dup ACK (tcp.analysis.duplicate_ack)
  • A packet is duplicated somewhere on the network and received twice at the receiving host. It is very often not desireable to get these duplicates, as the receiving application might think that's "fresh" data (which it isn't).If a sending host thinks a packet is not transmitted correctly because of a PacketLoss, it might Retransmit that packet. The receiving host might already got the first packet, and will receive a second one, which is a duplicated packet.
    • If the Duplicate ACK count is very low (Ex: TCP Dup ACK #1), this may indicate an Out-of-Order packet.
    • If the Duplicate ACK count is high, this typically indicates packet loss.
• TCP Out-of-order packets (tcp.analysis.out_of_order)
  • Indicate that the packet was received out of sequence.  This means that the packet will be held in the buffer of the receiver until the proper packets to complete the sequence are received.  Once received then the sequence can be committed.  The more out of order packets that occur the more likely the buffer will fill up.  When the buffer is full, the receiver will then start dropping the out of order packets.  When it does that it will have to start requesting retransmission of those packets. This process increases the chances for timeouts of the data streams and failures of replication.
• TCP retransmissions (tcp.analysis.retransmission)
  • Indicate that there is a packet that is incomplete, out of order, corrupt, timed out or lost.  When seen with many TCP Out-of-order packets it can indicate that the problem is being caused by the flow of the packets and the fact that they are not coming through in sequence.
• TCP Spurious retransmissions (tcp.analysis.spurious_retransmission)
  • Indicate the sender did not get the acknowledgement and so the syn is being retransmitted. This can happen for two main reasons:
  1. The initial ack was not sent for the entire stream or not acked correctly.  
  2. The stream disconnected and needs to be reconnected.
  • Windows uses MTCP and multiple streams
    • MTCP creates a major connection between 2 points.  Within this major connection are multiple minor connections called streams (think highway with multiple lanes).  Each stream is a lane in the highway. If one lane gets cut off then the lane needs to be reopened. To do this additional syn packets must be sent (spurious retransmits).