BadTimeout/ConnectionWarningWatchdogTimeout problems + Wireshark question

[stenil]

Hi,
we have problems with intermittent BadTimeout (on WriteRequest with a 3 s timeout, normal response time from server is 60-80 ms) and ConnectionWarningWatchdogTimeout (using the default 5 s timeout) in our client.
The client uses C++ SDK 1.7.7, the server uses some UA SDK from 2020 or earlier.

1. Do you think there is any chance that these timeouts could be caused by some client side locking/hickup, and not by the server actually responding slowly? We send a WriteRequest every 10 s or so, and do not receive a lot of updates on the subscription, so it is not a lot of OPC UA traffic.

2. I have a pcap file from one occurence, but the problem is that the traffic is encrypted so I can't view the OPC UA messages, or so I thought , until I found:
https://www.unified-automation.com/news/news-details/analyzing-encrypted-opc-ua-traffic-with-wireshark.html
https://www.youtube.com/watch?v=GCRpvHWdKkU (a tip: skip the background music for your next videos, made it harder to hear the speaker)

Can the Wireshark support for the OPC UA key log file be used in some way, even when we do not have the C++ SDK version 1.8.3 and above that includes "Key Logging" support? Can the keys be retrieved with some SDK method available in 1.7.7 and then we could create the key log file manually in the correct format?

If we upgrade to the latest C++ SDK in our client, would the actual logging to file need to be implemented in our code, triggered by UaClientSdk::UaSessionCallback::newSecurityKeySet ? I could not find info in the docs about that for the client.

BR,
Stefan

[Support Team]

Stefan,

the export of keys is available in 1.8.7 but must be compiled in and activated. However being part of UaExpert you may use also on older servers (but requires to reproduce the issue with UaExpert), and hence get the pcap decrypted by exporting the keys with UaExpert.

From your description it seems that the complete server is blocked (not responding on the watchdog read, and also not responding on the write service). The watchdog-warning comes after 5 sec and the watchdog-error after 15sec, indicating the connection being lost (or server being blocked/hang/unresponsive). Blocking the server for so long time sounds like an overload scenario.

Is the server becoming responsive again? e.g. if you wait and try again on the same connection?

Is the server responsive on a second (different) connection? e.g. if you connect in parallel with UaExpert?

Besides from that is there any timing regularity when this happens? e.g. like every 20 minutes or every 45 minutes? Or is that just happening sporadically without any timing regularity?

[stenil]

Hi,
we have around 100 OPC UA connections, and we see a few timeouts in total every day, and not evenly distributed among the 100 connections.
So, for most connections, they stay up for weeks before this happens.
I have gathered more info and the below is a translation of some info that I sent to the server supplier (that uses your SDK), I think it answers your questions:
"Our clients run in containers in a Linux VM.
On this VM we are now running tcpdump with full capture, which is saved when we detect a communication problem in our logs.

Since the OPCA UA traffic is encrypted, you can't see in Wireshark which messages are sent exactly, but by comparing with unencrypted examples (where the problem hasn't happened yet, as only few connections are unencrypted) I can now realize which messages are being sent.
What we see is that we sometimes get ConnectionWarningWatchdogTimeout – we use the default settings -> every 5 seconds the client sends a ReadRequest with a timeout of 5 s.

We send a WriteRequest about every 10 seconds, and for some of these we have time to get a BadTimeout (the timeout is 3 seconds) just before the ConnectionWarningWatchdogTimeout.

We have recently changed our code so that we now restart the client immediately (if configured so) on ConnectionWarningWatchdogTimeout, previously it took longer before we gave up.

What we can see in Wireshark is that there are missing OPC UA response messages from the server to these ReadRequest/WriteRequest.

The next message from the server (received before we close the connection based on ConnectionWarningWatchdogTimeout) is a PublishResponse (i.e. subscription notification, with or without data) and Wireshark marks it as "TCP Previous Segment not captured" as the message does not have the expected sequence number, and you can then see how one or more "TCP Duplicate ACKS" is sent from our side.
Sometimes one can also see 1-3 more OPCUA messages in both directions, before we teardown.

Whether the server does not send the missing packets, or if they are lost/delayed along the way is unclear to us.

"

Thanks,
Stefan

[stenil]

Duplicate post removed.