Uploading big files to opcua server

[C.Thilo]

Hey!

I need to upload big files, up to 20GB or so, to the opcua-server. My current upload method uses a UaByteString which is then written to a file via UaFile.

But as this depends on memory, 20 GB is way to much to be uploaded with this way, right?
Is there a better way to get this done? Any example in best case?

Of course there is ftp, samba or something like that. But that would need another network port to be opened. So not a prefered method too.

Any hint is appreciated!

Cheers,
Thilo

[Lafond]

Hi Thilo

You should have a look to the File object.
This object has read / write methods that allows you to chunk the amount of data you want to transfer.
This avoids the maximum string and message size limit.

BR

Claude

[Lafond]

Hi Thilo

See OPC UA Part 5 Annex C for more details.

Claude

[C.Thilo]

Hi Lafond,

thank you for your answers.

So I'm trying to use the Object of Type "FileType" for uploading a file.
In my server I set the filepath to a temporary file (which I create before starting the server).
Then I can call open on that object via UaExpert and get an file handle. But when I want to upload a file with e.g. 300MB via a call of the write Method,
I get an BadEncodingLimitsExceeded or BadDisconnect status back and the client needs to reconnect, however the server is still running with no error.

What am I doing wrong? Do I need to program a client which uploads only a subset of the file? So uploading the file junk by junk?

Cheers,
Thilo

[C.Thilo]

Thanks for your hint, again.

I built a simple python script with python-opcua to test my implementation. So uploading junk by junk with my python opcua client to my opcua server.

Works perfectly!

Cheers,
Thilo

[C.Thilo]

When there is a network disconnect or any other external problem,
how am I supposed to close the open file handle?

I'll never get the same session object again and so, will never be able to close this filehandle, except restarting the server which is very unnice.

Any ideas?

Cheers,
Thilo

[Support Team]

Hello Thilo,

if you use the OpcUa::FileType class that will be handled automatically.
So when the file is opened the SDK creates a FileHandleData class that implements the SessionCallback. When the session is closed (either by a client or because of the session timeout) the file is closed automatically.

[C.Thilo]

Hey!

I digged a bit more. My problem is, that the upload stream is detected to be to old and the secure channel is deleted, altough I don't feel like waiting 5 secs. between two write calls.
So IMHO the 5000ms never got reached. This happens with very big files like 10GB mostly at about 75% or more.

Some trace output:

|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelManager_TimerCallback: Input stream older than 5000 ms! Closing connection.
....
|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelManager_TimerCallback: deleting 1 SecureChannel!

|0x2DF04700| 11:35:36.426Z OpcUa_SecureListener_ChannelManager_TimerCallback: Checking Channels for lifetime expiration!
|0x2DF04700| 11:35:36.426Z OpcUa_SecureListener_ChannelManager_TimerCallback: Checking 2 channels!
|0x2C2F6700| 11:35:36.430Z OpcUa_SecureStream_AppendInput: appended chunk with SN 120957, RID 2957
|0x2C2F6700| 11:35:36.430Z ReleaseSecurityKeyset: Keyset for token 1 released.
|0x2C2F6700| 11:35:36.430Z OpcUa_SecureListener_ProcessSessionCallRequest: Waiting for more chunks!
|0x2C2F6700| 11:35:36.430Z * OpcUa_TcpListener_EventCallback: Event Handler returned with status 0x00000000.
|0x2C2F6700| 11:35:36.431Z * OpcUa_TcpListener_EventCallback: Socket(0x25dff28), Port(0), Data(0x25d4f40), Event(OPCUA_SOCKET_READ_EVENT)
|0x2C2F6700| 11:35:36.431Z OpcUa_TcpListener_ReadEventHandler: Connection 0x25db568.
|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelManager_TimerCallback: Input stream older than 5000 ms! Closing connection.
|0x2C2F6700| 11:35:36.431Z OpcUa_TcpStream_DataReady: Read 8 bytes.
|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelManager_TimerCallback: Checking 2 channels done!
|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelManager_TimerCallback: deleting 1 SecureChannel!
|0x2DF04700| 11:35:36.431Z OpcUa_SecureListener_ChannelRemovedCallback: SecureChannel 121440742 timed out!

After this, I can't reopen the file to continue uploading the data.
My guess is, that the session isn't closed by this Secure Channel delete and so the file can't be reopened again....
Or something like that.

[Support Team]

Hello Thilo,

did you change any of the encoding settings (MaxMessageSize, MaxArraySize, MaxByteStringLength,...)?

[C.Thilo]

Not that I know of. Where could this be changed? In the model?

From my config xml:

Code: Select all

        <Serializer>
            <!--The largest size for a memory block the serializer can do when deserializing a message-->
            <MaxAlloc>16777216</MaxAlloc>
            <!--The largest string accepted by the serializer-->
            <MaxStringLength>16776192</MaxStringLength>
            <!--The largest byte string accepted by the serializer-->
            <MaxByteStringLength>16776192</MaxByteStringLength>
            <!--Maximum number of elements in an array accepted by the serializer-->
            <MaxArrayLength>65536</MaxArrayLength>
            <!--The maximum number of bytes per message in total-->
            <MaxMessageSize>16777216</MaxMessageSize>
        </Serializer>

[Support Team]

Hello Thilo,

there's a check in the server side communication stack that transmitting a single message can not take longer than a reasonable amount of time.
This is a protection for the server to prevent clients from blocking a resource (socket) for an infinite time.
If you did not change the encoder settings (it is recommended not to change those settings) the maximum message size is 16Mb. So if it takes more than 5 seconds for the client to send one WriteRequest you will see the error message "OpcUa_SecureListener_ChannelManager_TimerCallback: Input stream older than 5000 ms! Closing connection".
That is the intended behaviour. You could use smaller chunks in your client application when writing the file.

[C.Thilo]

Thank you for the explanation! Thats good to know.

Cheers,
Thilo