Master Loss of Network Connectivity

jabramson · September 2013

NI-700 with lots of connectivity to other systems and out to the internet to grab scheduling & weather data. I've been having a problem where the unit will lose network connectivity. I originally thought it was related to a master hardware that died and I've since installed new hardware. I've also rebuilt my entire program adding in a section at a time and monitoring. Got a bunch of bugs worked out and all was working well for a few weeks, but the issue has returned. Sometimes it doesn't happen, other times it takes several hours, and other times within an hour. I'm not really sure where to check next. I suspect it may be running out of IP ports, but not completely sure. As far as I can tell, the master doesn't actually lock up, it just drops the network connectivity.

There are 10 TPs directly connected to this master
5 M2M connections. Other other Masters are running their own source.
This master just monitors the other masters, also updates the 10 TPs with schedule and weather data.

I was able to run the debug until it crashed and here are the last lines:

Line  54909 (00:23:50)::  SendString to socket-local port (16) invalid
Line  54910 (00:23:50)::  CIpEvent::OnError 0:16:3
Line  54911 (00:23:51)::  SendString to socket-local port (16) invalid
Line  54912 (00:23:51)::  CIpEvent::OnError 0:16:3
Line  54913 (00:23:57)::  CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer
Line  54914 (00:23:57)::  ICSPTCPRx15::CloseSocket: Closing Socket
Line  54915 (00:24:06)::  SendString to socket-local port (15) invalid
Line  54916 (00:24:06)::  CIpEvent::OnError 0:15:3
Line  54917 (00:24:09)::  SendString to socket-local port (15) invalid
Line  54918 (00:24:09)::  CIpEvent::OnError 0:15:3
Line  54919 (00:24:11)::  SendString to socket-local port (15) invalid
Line  54920 (00:24:11)::  CIpEvent::OnError 0:15:3
Line  54921 (00:24:12)::  SendString to socket-local port (16) invalid
Line  54922 (00:24:12)::  CIpEvent::OnError 0:16:3
Line  54923 (00:24:15)::  SendString to socket-local port (16) invalid
Line  54924 (00:24:15)::  CIpEvent::OnError 0:16:3
Line  54925 (00:24:16)::  SendString to socket-local port (16) invalid
Line  54926 (00:24:16)::  CIpEvent::OnError 0:16:3
Line  54927 (00:24:29)::  CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer
Line  54928 (00:24:29)::  ICSPTCPRx15::CloseSocket: Closing Socket
Line  54929 (00:24:31)::  SendString to socket-local port (15) invalid
Line  54930 (00:24:31)::  CIpEvent::OnError 0:15:3
Line  54931 (00:24:35)::  SendString to socket-local port (15) invalid
Line  54932 (00:24:35)::  CIpEvent::OnError 0:15:3
Line  54933 (00:24:36)::  SendString to socket-local port (15) invalid
Line  54934 (00:24:36)::  CIpEvent::OnError 0:15:3
Line  54935 (00:24:37)::  SendString to socket-local port (16) invalid
Line  54936 (00:24:37)::  CIpEvent::OnError 0:16:3
Line  54937 (00:24:40)::  SendString to socket-local port (16) invalid
Line  54938 (00:24:40)::  CIpEvent::OnError 0:16:3
Line  54939 (00:24:41)::  SendString to socket-local port (16) invalid
Line  54940 (00:24:41)::  CIpEvent::OnError 0:16:3
Line  54941 (00:24:52)::  SSH connection accepted 10.44.2.200:46899 socket=4125
Line  54942 (00:24:52)::  Accepted Telnet connection:socket=75 addr=10.44.2.200 port=46900
Line  54943 (00:24:52)::  SSH_closeConnection instance=4125
Line  54944 (00:24:57)::  SendString to socket-local port (15) invalid
Line  54945 (00:24:57)::  CIpEvent::OnError 0:15:3
Line  54946 (00:25:00)::  SendString to socket-local port (15) invalid
Line  54947 (00:25:00)::  CIpEvent::OnError 0:15:3
Line  54948 (00:25:01)::  SendString to socket-local port (15) invalid
Line  54949 (00:25:01)::  CIpEvent::OnError 0:15:3
Line  54950 (00:25:01)::  CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer
Line  54951 (00:25:01)::  ICSPTCPRx15::CloseSocket: Closing Socket
Line  54952 (00:25:02)::  SendString to socket-local port (16) invalid
Line  54953 (00:25:02)::  CIpEvent::OnError 0:16:3
Line  54954 (00:25:03)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46902
Line  54955 (00:25:05)::  SendString to socket-local port (16) invalid
Line  54956 (00:25:05)::  CIpEvent::OnError 0:16:3
Line  54957 (00:25:06)::  SendString to socket-local port (16) invalid
Line  54958 (00:25:06)::  CIpEvent::OnError 0:16:3
Line  54959 (00:25:08)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46903
Line  54960 (00:25:13)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46905
Line  54961 (00:25:13)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46906
Line  54962 (00:25:18)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46907
Line  54963 (00:25:18)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46908
Line  54964 (00:25:18)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46909
Line  54965 (00:25:18)::  Error setting TCP socket to blocking
Line  54966 (00:25:22)::  SendString to socket-local port (15) invalid
Line  54967 (00:25:22)::  CIpEvent::OnError 0:15:3
Line  54968 (00:25:23)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46913
Line  54969 (00:25:23)::  Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46914

10.44.11.207, 10.44.2.200 are another masters (M2M)
Ports 15 & 16 are IP devices which have gone offline. It attemps to reconnect every few mintues.

Any direction is appreciated

John Nagy · September 2013

Why is a master logging in repeatedly on telnet? Master-to-master comms don't use telnet...

You can try telnet diagnostics to see what is up on IP and if there are a lot of dead connections building up.

Show TCP List

The following TCP connections exist(ed):
1: IP=192.168.1.180:35284 Socket=23 (Active)
Last[MsgsInQ=0 RecvTime=0006892814 SendTime=0006892802]
2: IP=192.168.1.129:1271 Socket=24 (Active)
Last[MsgsInQ=0 RecvTime=0006892805 SendTime=0006892802]

IP Status

NetLinx IP Connections
TCP Server Listener on IP Port 5750 (backlog=1) (socket=34)
TCP Server waiting on IP Port 5750 LocalPort=21

jabramson · September 2013

Looks normal to me:

Welcome to NetLinx v3.60.453 Copyright AMX LLC 2010
>show tcp
Show TCP List
-------------
The following TCP connections exist(ed):
1: IP=10.44.11.200:2070 Socket=26 (Active)
   Last[MsgsInQ=0 RecvTime=0011144066 SendTime=0011144066]
2: IP=10.44.23.201:3544 Socket=27 (Active)
   Last[MsgsInQ=1 RecvTime=0011143222 SendTime=0011144517]
3: IP=10.44.7.202:3365 Socket=28 (Active)
   Last[MsgsInQ=1 RecvTime=0011143239 SendTime=0011144517]
4: IP=10.44.27.202:3520 Socket=29 (Active)
   Last[MsgsInQ=1 RecvTime=0011143217 SendTime=0011144518]
5: IP=10.44.23.200:3525 Socket=31 (Active)
   Last[MsgsInQ=1 RecvTime=0011143225 SendTime=0011144518]
6: IP=10.44.19.204:3705 Socket=32 (Active)
   Last[MsgsInQ=1 RecvTime=0011143228 SendTime=0011144518]
7: IP=10.44.3.202:1051 Socket=39 (Active)
   Last[MsgsInQ=1 RecvTime=0011143211 SendTime=0011144518]
8: IP=10.44.3.200:1319 Socket=36 (Active)
   Last[MsgsInQ=0 RecvTime=0011145560 SendTime=0011145560]
9: IP=10.44.19.203:1614 Socket=42 (Active)
   Last[MsgsInQ=1 RecvTime=0011143234 SendTime=0011144519]
10: IP=10.44.19.202:1043 Socket=43 (Active)
   Last[MsgsInQ=1 RecvTime=0011143214 SendTime=0011144519]
11: IP=10.44.7.203:3519 Socket=44 (Active)
   Last[MsgsInQ=1 RecvTime=0011143237 SendTime=0011144519]
12: IP=10.44.7.204:3774 Socket=45 (Active)
   Last[MsgsInQ=1 RecvTime=0011143231 SendTime=0011144519]
13: IP=10.44.35.202:1042 Socket=46 (Active)
   Last[MsgsInQ=1 RecvTime=0011143219 SendTime=0011144519]
14: IP=10.41.204.61:4588 Socket=47 (Active)
   Last[MsgsInQ=1 RecvTime=0011143272 SendTime=0011144520]
15: IP=10.44.11.207:1319 Socket=30 (Active)
   Last[MsgsInQ=0 RecvTime=0011147413 SendTime=0011146625]
16: IP=10.40.12.58:63907 Socket=0 (Dead) Last[MsgsInQ=0]

>ip status
IP Status
---------
NetLinx IP Connections
TCP Client connected on IP Port 4095 LocalPort=15 IP=10.44.3.136 (socket=37)

I'll check it again after it's been running for a while to see if anything builds up.

The telnet commands are a strange one. That is not a piece of my equipment (non AMX or related). I've reached out to our IT group to see if they know what it is.

Master Loss of Network Connectivity

Comments

Leave a Comment

Categories