Empower 3 Injection Data Disruption

[beyondtheprize]

Folks,

I am at a loss for this disruption in Data Acquisition. Have see single injections with 50min run times stop at ~10mins and ~40mins. Nothing in Message Center except for user prompted abort. Nothing in Audit Trail.

This is E3FR2 in a citrix environment and in house built laces. The instrument is an Agilent 1260 (G4212B)

I have checked the instrument server and the recover log with no issues in there.

One thing I have noticed is in the "Node Properties" a similar HPLC 1260 lists the "Details" on the instrument tab, but the troubled Agilent doesn't have the details listed.

[shaun78]

I would be curious to see if the Waters DHCP server log on the lace the 1260 is connected to reported any IP address problems.

Information not being populated for the 1260 in the lace config is an indication that the lace is having trouble maintaining a connection with the instrument.

[beyondtheprize]

How do I access this?

[shaun78]

Login to empower (preferably as an admin) and select Configure System.

Click Nodes and select the LAC/E that your 1260 is connected to.

Right click that LAC/E and choose Properties.

Once the node configuration opens, click the Configure DHCP tab and click the Configure DHCP button on the window of that same tab.

You will be asked for an audit trail reason and password. I usually enter "check DHCP logs" as my reason.

A window will open showing IP, MAC, Type, and name of all the instruments connected to the LAC/E. On the top of that window is a Server menu item. Click Server and select Errors.

This will open a window that shows all the network errors on the instrument network card of the LAC/E since installation or when ever you last cleared them.

Check for error messages around the same times that your 1260 seems to randomly stop running.

[beyondtheprize]

the most recent drop out was on 16MAY and there was also a generic "Instrument Failure" on 13JUN.

Both well after the most recent DHCP error below

This is the most recent:
2016-03-08 14:56:56 - The network card is not enabled, check that the correct cables are properly connected and that the instruments are turned on.

[DR]

Search your Antivirus settings or the Registry on a LACE to be certain that your Empower folders are not being scanned. Dropped runs with few or no Message Center entries and no errors in Windows Logs are common when someone forgets about the Empower exclusion.

If searching the registry, search data for :\empower|*
which, if you're using Trend Micro, for example, shows up in hkey_local_machine/software/wow6432node/TrendMicro/PC-CillinNTCorp/CurrentVersion/ManualScanConfiguration

Excluded Folder

[beyondtheprize]

Yeah, I have seen this. We run Symantec and Empower is listed in our enterprise exclusions. We also are not seeing this on any other agilents and definitely not on any Waters instruments.

This is localized to 1260s running 2.13.

Will follow up on the exclusions, but very confident as I have review the scan logs on antivirus and don't see any quarantined items related to Empower

[DR]

Dropped runs are not the result of quarantined Empower items - AV scanning while data is being collected can cause the node to foul up the data collection process and drop the run.

[ender6574]

Are you sure that ALL power saver options on your LACE boxes are turned off? Especially for the network adapters, but also the advanced power options and any other power saver options.

[beyondtheprize]

Well I'll be damned...

I wasn't crazy. Not anti-virus, not power settings. We had Agilent engineer who specializes in Empower on-site and unbridged the instrument LAN.

We kept getting data drop outs. In addition we started seeing pressure spikes to the tone of 10^35. This is both not feasible for the pump and furthermore the pressure trace caught these without any message center error.

The engineer went back to the developers and supposedly they found some error in the code. I will update upon notice.

Empower 3 FR2 SR2 HF2
Agilent LC 2.1.13.0404
ICF 2.3.99
LC Driver A.2.11 SP1 HF2

[beyondtheprize]

Furthermore, they mentioned they had seen this in 4 other Agilent customers. One in Switzerland.

I asked if this was embedded to confirm upgrade could work, but they have not guided us one way or another.

Cheers

[DR]

Did un-bridging the instrument NIC ports help?
Why would this be needed?

I rather enjoy having them bridged when there are several available as it saves me having to install a hub...

[beyondtheprize]

The un-bridging of the network connections was recommended by an Agilent Engineer with very strong Empower skills. I would say, in my experience in a rather large pharma, you can get away with the bridging of connections but not doing so almost guarantees a robust data communication

Additionally, we are in a situation where we are forced to use Chemstation until Waters comes on site to do testing, as they (Waters and Agilent development staff) believe there is a coding issue with the ICF (Agilent LC 2.1, A.02.03 DU1 HF2).

We have been pretty disappointed with Agilent's support, as they have wrung their hangs of us. Because its not a "Hardware issue", they won't replace either instrument (which is fair), but it becomes red herring to the fact that one of the problem 1260s has hardware issues simultaneously.

[fifthelement]

The un-bridging of the network connections was recommended by an Agilent Engineer with very strong Empower skills. I would say, in my experience in a rather large pharma, you can get away with the bridging of connections but not doing so almost guarantees a robust data communication

Additionally, we are in a situation where we are forced to use Chemstation until Waters comes on site to do testing, as they (Waters and Agilent development staff) believe there is a coding issue with the ICF (Agilent LC 2.1, A.02.03 DU1 HF2).

We have been pretty disappointed with Agilent's support, as they have wrung their hangs of us. Because its not a "Hardware issue", they won't replace either instrument (which is fair), but it becomes red herring to the fact that one of the problem 1260s has hardware issues simultaneously.

Hi, I work in an environment very similar to yours and have experienced some wonky issues with Agilent HPLCs in Empower, especially newer ones (1260s). We've had missing data occur somewhat regularly, but ONLY on the first injection of a run, and this seems to coincide with the absorbance the detector is outputting going crazy (e.g. 10^80 AU, not possible). Less often we've had runs just stop in the middle with no indication, and once or twice we had missing data in the middle of an injection that wasn't the first injection.


Anyway, a couple things I'm wondering:

Any updates?

Any idea how to reliably reproduce any of your issues? We've had a heck of a time troubleshooting since we can't make things happen regularly.

Do you have a Waters PCS number or something we could use to look into our issues further?

Any help would be appreciated. Thanks!

[beyondtheprize]

So....

I would be much happier had I not been the impetus for a bug fix, but from now on it will be my claim to fame. In combination with another customer in Europe, there was found within the Agilent LC 2.1 through 2.2 ICF a bug.

Our business has already purchased OpenLab as a consequence of this wack-a-mole. Our Waters Sales Rep has been told by Product Management that they worked with Agilent separately and have determined the root cause of the problems and will be fixed in the Waters support for ICF 3.0, and because it's understood, there is the ability today to work-around it for those customers experiencing the issue.

Release date unkown...

https://www.waters.com/waters/support.h ... cale=en_US