Hello team, I will need some help here. We have 1 client who made a project with 2 main servers + 42 workstations + 1 dev server + 1 reporting server (total 46 servers). The setup is something like this

• Reporting server is connected only to server 1 and server2
• Server1 and server 2 are connected to each of the workstation
• Each workstation does not connect to each other, so they only connect to server 1 and server 2,and also dev server
• Each workstation can query realtime and historical data of the other workstation, through realtime association and historical association that server1 and server 2 is part of. To achieve this, we have modified individual station.dat on each machine, so each station have slightly different file.
• Dev server is connected to all station, but only turned on when needed

Server01 and Server02 is running on Windows Server 2016, so we are sure that it should not encounter TCP/IP limit, which is usually 20 for Windows 10 platform. Now we have a problem, whereby from Server 1, I can see constant connection and disconnection to all the workstations. Sample error from server01 to 1 workstation as below. It happens to all workstation. You can see that every 40s it will get disconnected, and then after 30s it will reconnect again. I don't see timeout on watchdog message in the server.

2020/04/16,16:39:38.909,3,W,,3074,ADMINISTRATOR,1,Server->client connection abort, SERVER010S -> WORKSTN330C
2020/04/16,16:39:38.909,3,I,,3188,ADMINISTRATOR,1,WinSock error = 10054 WSAECONNRESET
2020/04/16,16:39:38.909,3,I,,3240,ADMINISTRATOR,1,Connection reset by peer


2020/04/16,16:40:08.913,3,I,,3070,ADMINISTRATOR,1,Server->client connection OK, SERVER010S -> WORKSTN330C
2020/04/16,16:40:08.959,3,I,,9008,ADMINISTRATOR,1,Networking message version of station WORKSTN33 = 120001

2020/04/16,16:40:48.965,3,W,,3074,ADMINISTRATOR,1,Server->client connection abort, SERVER010S -> WORKSTN330C
2020/04/16,16:40:48.965,3,I,,3188,ADMINISTRATOR,1,WinSock error = 10054 WSAECONNRESET
2020/04/16,16:40:48.965,3,I,,3240,ADMINISTRATOR,1,Connection reset by peer

2020/04/16,16:41:18.953,3,I,,3070,ADMINISTRATOR,1,Server->client connection OK, SERVER010S -> WORKSTN330C
2020/04/16,16:41:19.000,3,I,,9008,ADMINISTRATOR,1,Networking message version of station WORKSTN33 = 120001

2020/04/16,16:41:59.005,3,W,,3074,ADMINISTRATOR,1,Server->client connection abort, SERVER010S -> WORKSTN330C
2020/04/16,16:41:59.005,3,I,,3188,ADMINISTRATOR,1,WinSock error = 10054 WSAECONNRESET
2020/04/16,16:41:59.005,3,I,,3240,ADMINISTRATOR,1,Connection reset by peer

2020/04/16,16:42:29.009,3,I,,3070,ADMINISTRATOR,1,Server->client connection OK, SERVER010S -> WORKSTN330C
2020/04/16,16:42:29.056,3,I,,9008,ADMINISTRATOR,1,Networking message version of station WORKSTN33 = 120001

2020/04/16,16:43:09.061,3,W,,3074,ADMINISTRATOR,1,Server->client connection abort, SERVER010S -> WORKSTN330C
2020/04/16,16:43:09.061,3,I,,3188,ADMINISTRATOR,1,WinSock error = 10054 WSAECONNRESET
2020/04/16,16:43:09.061,3,I,,3240,ADMINISTRATOR,1,Connection reset by peer

2020/04/16,16:43:39.049,3,I,,3070,ADMINISTRATOR,1,Server->client connection OK, SERVER010S -> WORKSTN330C
2020/04/16,16:43:39.096,3,I,,9008,ADMINISTRATOR,1,Networking message version of station WORKSTN33 = 120001

I have refered to article on KB article KB1025KB and used nmap tool to check if there is any issue with destination server and also port, but all looks good. I also found something that I feel is not right from the audit, whereby I can see the amount of pending is very high and NOT reducing. I assume this is what causes the constant connection and disconnection, but anybody know how to fix this?

Your help is much appreciated. I can provide more information if you request. Also, is there anything else that I should check? Thanks, Kantha