Pernicious Winsock Errors (Empower)

[DR]

Typically, this error is associated w/ the death of a switch or pulling of a plug on the switch between the eSATIN & LAC/E. I'm not sure about IP addresses, I think the LAC/Es are all named and a DNS sorts out the IP addresses (but I'm not sure - IPs could be static).

The normal config is:
eSATIN--->switch--->LAC/E
LAC/E---->LAN hub, gigaman line--->Empower server

The errors are thrown when a LAC/E loses track of an eSATIN, they are thrown pretty much as a group, all at the same time.

As far as I can tell, there's no particular time when the errors happen.

Nodes, MTU ? don't know.

Waters has asked lots of questions, had me try some config. changes, different hardware pieces etc. They have a set of evt & log files...

[HbJ]

This points towards the switch.

Did you _completely_ change _every_ connection between eSATIN and LAC/E? This includes _all_, really _all_ network cables, the switch itself and the power supply of the switch?

Are you able to reconnect with the eSATIN immediately after the failure? Maybe the switch crashes and reboots. Does the switch have a syslog (or at least an uptime counter?)

[DR]

I did not replace all cables, I guess it would make sense to try this. I have tried replacing the switch and power supply (which didn't help, again pointing to cables...).

This is a tiny MiLAN 8 port switch. No software associated w/ it AFAIK.

I think I'll round up fresh cables and have another set run on the LC.

Thanks for all your suggestions (I'll let you know if this helps).

[DR]

I replaced all cables that touch the MiLAN switch, and I replaced the cable from LAC/E to LAN. It promptly failed again (~1 hour later), WinSock errors.
netstat -an >protocol.txt:

Active Connections

Proto Local Address Foreign Address State
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1271 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1272 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING
TCP 0.0.0.0:13482 0.0.0.0:0 LISTENING
TCP 10.96.22.51:135 10.97.20.11:3687 ESTABLISHED
TCP 10.96.22.51:139 0.0.0.0:0 LISTENING
TCP 10.96.22.51:1279 10.96.71.28:3857 ESTABLISHED
TCP 10.96.22.51:1288 10.96.71.28:1181 ESTABLISHED
TCP 10.96.22.51:1457 10.96.71.28:135 TIME_WAIT
TCP 10.96.22.51:1460 10.97.20.74:135 TIME_WAIT
TCP 10.96.22.51:1461 10.97.20.74:1025 TIME_WAIT
TCP 10.96.22.51:1464 10.97.20.80:445 TIME_WAIT
TCP 10.96.22.51:1469 10.97.20.80:445 TIME_WAIT
TCP 10.96.22.51:1472 10.97.20.80:135 TIME_WAIT
TCP 10.96.22.51:1473 10.97.20.80:1025 TIME_WAIT
TCP 10.96.22.51:1475 10.97.20.80:135 TIME_WAIT
TCP 10.96.22.51:1476 10.97.20.80:1025 TIME_WAIT
TCP 10.96.22.51:1477 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1479 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1480 10.97.20.76:445 TIME_WAIT
TCP 10.96.22.51:1483 10.97.20.78:445 TIME_WAIT
TCP 10.96.22.51:1492 10.97.20.80:445 TIME_WAIT
TCP 10.96.22.51:1498 10.97.20.80:135 TIME_WAIT
TCP 10.96.22.51:1499 10.97.20.80:1025 TIME_WAIT
TCP 10.96.22.51:1501 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1503 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1504 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1505 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1506 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1507 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1508 10.97.20.242:1433 TIME_WAIT
TCP 10.96.22.51:1509 10.97.20.80:445 TIME_WAIT
TCP 10.96.22.51:1513 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1514 10.97.20.172:1433 TIME_WAIT
TCP 10.96.22.51:1519 10.97.20.76:445 TIME_WAIT
TCP 10.96.22.51:1522 10.97.20.142:445 TIME_WAIT
TCP 10.96.22.51:1526 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1527 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1528 10.97.21.125:445 TIME_WAIT
TCP 10.96.22.51:1531 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1532 10.97.20.80:389 TIME_WAIT
TCP 10.96.22.51:1533 10.96.71.28:135 ESTABLISHED
TCP 10.96.22.51:3389 10.96.22.177:2405 ESTABLISHED
TCP 127.0.0.1:1056 0.0.0.0:0 LISTENING
TCP 192.168.0.1:139 0.0.0.0:0 LISTENING
TCP 192.168.0.1:1431 192.168.0.3:80 ESTABLISHED
UDP 0.0.0.0:69 *:*
UDP 0.0.0.0:445 *:*
UDP 0.0.0.0:500 *:*
UDP 0.0.0.0:1025 *:*
UDP 0.0.0.0:1026 *:*
UDP 0.0.0.0:1043 *:*
UDP 0.0.0.0:1487 *:*
UDP 0.0.0.0:4500 *:*
UDP 0.0.0.0:13482 *:*
UDP 0.0.0.0:13483 *:*
UDP 10.96.22.51:123 *:*
UDP 10.96.22.51:137 *:*
UDP 10.96.22.51:138 *:*
UDP 64.1.1.1:123 *:*
UDP 127.0.0.1:123 *:*
UDP 127.0.0.1:1028 *:*
UDP 127.0.0.1:1048 *:*
UDP 127.0.0.1:1139 *:*
UDP 192.168.0.1:67 *:*
UDP 192.168.0.1:123 *:*
UDP 192.168.0.1:137 *:*
UDP 192.168.0.1:138 *:*

[HbJ]

Thanks for the log but I can't figure it out right now

How long are the cables between the instrument, the switch and the PC? If they are too long 'late collisions' may happen.

What else lives on your LAN? Some data intensive applications which can congest the network?

And your switch should have some kind of configuration interface. Please consult the manual (standard are telnet (port 23), http (port 80), https (port 443) and SSH). Your switch should have a syslog or an uptime counter. This way you can check if the switch crashes (it is a piece of hardware which contains a piece of software, the firmware).

All in all this looks like a network problem but I can't quite pin it down.

Please check your instrument logs for the first time the error occurred. Maybe you can correlate that with an update on your box, your network or the instrument. And check if there is heavy network traffic (your switch should be able to tell you).

And what is Waters stance of the problem?

[DR]

Cables are short (all 7m or less)

switch I'm talking about is a small, dumb 8 port switch w/ LEDs on it

I can't track initial event because those reports have cycled out of the message center (was set to 3 days).

I have an open ticket w/ Waters. They're reviewing evt files, looking into it (slowly).

[HbJ]

Reading your report two things come to my mind:

- The PC your DS in running on may has become a data server within your network and/or the associated network is very busy. That might be true as your PC has several source IPs which are not 0.0.0.0 or 127.0.0.1. Furthermore some are 192.168.x.x (private C-Class), others 10.x.x.x (private A-class). Does your box have more than one NIC?
- An update to one of your software components (DS, Windows XP, ...) may have caused a slight, subtle configuration/behavior change which interferes with your DS.

All in all, I'm quite clueless right now.

But please check for a manual of your switch. A switch has to have a configuration interface (I've never seen one without). If it doesn't have one it might be a hub what could serve as an explanation.

Sorry I wasn't more useful

But you could try one last thing: Get yourself another PC, install Windows XP SP2 without any patches, install your original DS software and couple it with (at least) one instrument.
If this works, try to install one patch after the other until it fails.

This is my last, desperate measure if anything else goes wrong. That way, I've tracked down the strangest bugs.

But I know this is a real lab and I feel you won't have the time and the consent of your management.

Good luck!

[EDIT] Your animated Felix the Cat is a precise description how I'm trying to solve problems.

[DR]

Thanks for all of your input.

Yes, the multiple IPs are a function of having 2 NICs (one for the LAN, oen for the eSATINs (192.168...).

While your approach would almost certainly work (going back to a fresh XP isntallation, add patch, test, repeat...), I don't have time for that. I guess it's time to remind the vendor what we've paid them for...

[HbJ]

As long as you're sure your system hasn't become a server within the LAN...

I hope Waters can solve that problem. Please keep us informed

[bartjoosen]

Maybe you can go back to a fresh XP installation and don't install the patches for the lifetime of the systems?

Did you really need those patches?

It hasn't something to do with the new IExplorer? Empower isn't tested with IE7!


Bart

[DR]

Maybe you can go back to a fresh XP installation and don't install the patches for the lifetime of the systems?

Did you really need those patches?

It hasn't something to do with the new IExplorer? Empower isn't tested with IE7!

That's what was being discussed - and no, I don't have that kind of time. Unfortunately, the patches are pretty well required (PCs w/o patches don't get to play on the LAN).

IE7 isn't on anything yet, so it isn't that...