Waters Alliance Baseline "Ripple"

[sdegrace]

I have a bizarre problem that is so far defying all my efforts to troubleshoot. I thought I would put it out there to see if anyone has any ideas. First, here's what the baselines look like:



Now for the weirdness. I have two Alliance systems, 2695 Separations Modules with 2996 PDAs. One also has a 2487 DWD. They are both experiencing the same problem on the PDAs. That chromatogram is at 252 nm, but it looks the same at practically any wavelength. similar oscillation is seen on the DWD, but with a lower amplitude and a slightly different frequency at any given wavelength.

The problem started for all around the same time - digging back in the data, after a certain day all baselines look like above, but before that day they are perfect.

The pressure is extremely steady with a good delta (about 8 psi with a column attached) and does not track the oscillation. Same issue is seen whether a column or a union is is place, and with isocratic 100% methanol at 1 mL/min or any combination of gradient and solvent.

With a union in place, 100% methanol, 1 mL/min, pressure is 200 psi, delta 4 psi, and the chromatogram looks much like the one above. That particular chromatogram was acquired with a 100x4.6 mm 3.5 um C18 in place running a water/ACN gradient.

PDA diagnostics all pass. PM was completed by the vendor on Nov. 20 and was working fine for a while after the PM was completed. Reading the PDA calibration tool with 21 ms exposure time, I get a response of a little less than 10000 at 230 nm on the PDA for one instrument, which is a little low, but a very healthy response on the other PDA. However, both instruments have close to identical behavior.

Looks to me like an environmental factor affecting the detectors...

Temperature in the room is very stable. Problem started in the middle of winter so it doesn't seem like a change in the building's enviromental settings is a likely culprit, any major change should alreay be done. Instruments have been in the same exact location for a couple years with no problems.

The instruments are in a small room with several other LCs, which are not experiencing any problems.

Each instrument is on its own surge protector bar connected to the same wall plate which is in turn connected to a UPS. In the same room as several other LCs including two UPLCs which are experiencing no issues. I tried plugging the Alliance system into the same wall plate as the Acquities (different UPS) but no change. I tried swapping the power bars but no change.

I've considered the flow cells... perhaps something was run on the instruments at the same time which contaminated the flow cells, although I can find no evidence of it. I tried flushing with THF, and I executed a protocol from Waters to clean with 30% phosphoric acid, but no change in the issue.

I thought of whether there might have been a power surge, but the UPS logs don't record anything significant around that date.

We have a service contract, and Waters is sending me a new flow cell and lamp to try. But I'm increasingly skeptical as to what that's going to accomplish.

I wonder if anyone has ever seen anything like this before and can give me any ideas.

[danko]

First of all, you’ll need to encircle the problem to a single instrument (i.e. is it the pump or the detector?). For that purpose you’ll need to alter the instrument method so that all other parameters are the same but the flow rate (should be set to zero). Then start a baseline and monitor the signal – if it’s flat, you have a pumping problem. If the ripple continues you have a detector problem and most probably disturbance by a nearby situated electrical equipment. It might be closest to the system/s that exhibit the problem and not close enough to the systems that perform trouble-free.
Maybe you’d report back, when you’ve pinpointed the “troublemakerâ€

[lcguy1]

Danko is going in the right direction. We had an alliance system that performed similarly. We spent months trying to figure it out. The thing was completely gutted and all parts replaced, thank God for service contracts, and we ended up giving it to the development group anyway. We were never able to get it to go away. The noise was not exactly like yours, but hauntingly familiar.

[Bluesman75]

I have had something similar, but on an Agilent system. It looked like a pump problem as the noise stopped when the pump flow was stopped, but ended up as a faulty seal around the cell window in the detector (possible a micro leak?). The cell/lamp tests had all passed.

If it is possible to bypass parts of the instrument such as the autosampler and the column oven so you are only trying to find a fault in the minimum number of modules that will help a lot. Also if you can plumb in a different detector that will help as well.

With regards the environmental factors, it does not look like typical air-conditioning interference as in my experience this tends to be more a sine wave with a wavelength of ~10 minutes. I would be looking for maybe a powerful electric motor.

[juddc]

I have seen rapid variations in relative humidity do this. I had a very similar situation with my 2695 last year which I tracked to rapid swings in RH. The period was slightly longer, but all other symptoms were nearly spot-on. The temperature in the room was also quite stable but the humidity was not. I borrowed a small electronic temperature / RH monitor (made by Dickson) from our micro group, placed it upon my detector and once the temp / RH data were downloaded and superimposed over a baseline trace, the cuplrit was immediately clear. The issue was traced to a malfunction in the HVAC system that had otherwise gone undetected. I'm not saying that this is definitely your problem, but it can indeed happen.

[Uwe Neue]

Considering that you observe similar patterns independent of the instrument settings (gradient, isocratic etc.) and a common onset on two instruments, I would look for an environmental factor: power lines connected to to a common source and external cyclic power consumption, air flow around both instruments from the same source (room heating cycles). Maybe around the time of the onset, somebody repaired or adjusted air flow into the room.

[sdegrace]

I watched the output of the PDA and DWD with zero flow, and there was no change in the oscillation (actually, it was if anything more perfect-looking and symmetrical).

I spoke with one of our engineers, and he clarifed that the UPS panels actually all go to only one large UPS unit, so I tried plugging the system in to an outlet which is not on the UPS. However, there was no change in the oscillation. The side of the room where the two Alliance LCs sit is next to the outer wall of the building, facing the parking lot... the engineer said there has been no work or any changes to the electrical system or new equipment running on that side of the building.

The RH suggestion is intriguing and I'll follow that up.

I could try physically moving the systems, but that would strictly be a desperation move. I will wait until the new lamp and flow cell come, supposedly tomorrow, and see what effect if any that has. But anyway, I think I'm out of ideas again. I am thinking Uwe is probably right, but frustratingly, I can't get anything from engineering or maintenance that would explain it.

[HW Mueller]

What about the A/C converter and PC? Are the effected instruments connected to the same PC via the same A/C? The other instruments use different setups? What surprises me is that there is no constant periodicity (so one can´t really talk about frequency), and that the two detectors give different patterns.

[sdegrace]

The two system are running workstation Empower on the same PC. The PC and monitor are connected to the same power bar as one of the Alliance systems and in turn connected to the same power supply.

One thing I have not tried but might be interesting is to run the two 2996 PDAs at the same time and see if they are in phase. I will do that and see what it looks like. When you look at the DWD and PDA side by side it seems like they are close to being at opposite phase to one another. The period, if not the same, is very, very similar.

In the same room, there are two Acquity systems connected to a different wall panel to the same UPS. They are also sharing a (different) work station running Empower. These systems are apparently unaffected. There is also a Gilson HPLC system connected to an RI detector. I don't often use it, but my colleague who does has not reported any problems. The Gilson is not on the UPS. In the next room there are two Varian HPLCs (on the UPS), two Agilent 6890 GCs and a Thermo Trace GC (not on the UPS) all apparently functioning normally.

[sdegrace]

I haven't downloaded the data yet to look at the trace, but just watching the Dickson data logger while the trace is acquired is eerie. The RH tracks the baseline almost perfectly. The highest RH I've seen so far is about 30.2% and the lowest 26.4%, and the period and even the fact it rises faster than it falls matches the baseline almost exactly.

So, humidity is suddenly Suspect Number One (unless it's not the humidity per se but an electrical effect from the system trying to control the humidity), I need to have a little talk with maintenance and also think how I'm going to conclusively prove if that's the problem and then how I'm going to properly remediate it. But this is the best lead I've had so far - thanks, juddc!

A big thanks to everyone who has weighed in on this so far. I will definitely follow up with any progress I make.

Also: Looking at the two PDAs side by side they match very closely, maybe the slightest bit out of step but maybe that is just my inability to start the acquisitions at precisely the same time.

[Hollow]

just some thoughts

What about some electrostatic charge somewhere there?
What's the humidity in the room?

Maybe the humidity is just below some value and enables to accumulate some charge on the detectors.
Did you try to ground the detector surfaces and/or the capillary tubing?
(and try to place the detector in a faraday cage?)

I'm thinking of the awkward discharge of electrostatic charge by touching some metal surface, which happens to me more often in the winter with dry room climate.

Another effect I sometimes observe, is that my lamp which has a touching dimmer is playing mad. In such moments, the lamp is oscillating without anyone is touching it (e.g. in the middle of the night). Last time there was a thunder storm going on outside, but it seems not to be a prerequisite...

[juddc]

Hmmmm...I wouldn't think that those levels of humidity themselves should bother your machines. The changes that I was seeing were of much greater magnitude AND beyond the manufacturer's specifications. During summer months, I was seeing oscillation between 60->90% RH (on rainy days) every few minutes. I also saw the sawtooth pattern on both the Dickson unit and the LC, though... It appears that you do have a correlation, though I'm suprised that the relatively small magnintude can cause an issue.

[sdegrace]

Hmmmm...I wouldn't think that those levels of humidity themselves should bother your machines. The changes that I was seeing were of much greater magnitude AND beyond the manufacturer's specifications. During summer months, I was seeing oscillation between 60->90% RH (on rainy days) every few minutes. I also saw the sawtooth pattern on both the Dickson unit and the LC, though... It appears that you do have a correlation, though I'm suprised that the relatively small magnintude can cause an issue.

The downloaded trace is an eerily perfect match to the output of the PDA. I too am very surprised at how tiny the fluctuation is, and yet it matches the baseline and thus must be related to the problem in some way.

I will do more experiments with the atmosphere in the room tomorrow to attempt to confirm and to try and find an effective remediation. I'll keep you all posted.

Thanks for your help,

Stephen

[sdegrace]

OK, I think I solved it, or at least I have a working solution.

Here's the structure of the lab... there is a wet lab which is connected to an instrument room by a door which usually remains shut. The instrument room is two rooms, an outer lab and an inner lab, connected by a doorway where the door has been physically removed. The Alliance systems are on the back wall of the inner instrument room.

We're in a very cold climate and theindoor RH can be extremely low in the winter, creating problems with static in weighing. Apparently some time in the last year, in order to address this problem, two industrial-strength humidifier units were installed in the ceiling of the wet lab and connected to the ventillation systems. One is for the wet lab and one is for the instrument room.

I got maintenance to shut off the one going to the instrument room and watched the baseline and the RH. As soon as the unit turned off, the baseline and the RH started to drop, with the baseline continuing to follow the RH in lockstep. The roughly three minute oscillation is completely gone, but the baseline continues to gently meander with the RH. You could acutally use the baseline to follow the RH in the room with a sensitivity of about 0.1%. It's uncanny.

So the question is, was the problem actually caused by humidity or was it caused by an electrical feedback from that particular humidifier? My engineer feels very strongly that it must be an electrical problem, feeling that it defies all sense for such a small humidity fluctuation to have usch a pronounced effect on the output of the detector. I have to say I would love to be able to agree with him, and certainly I wanted to try and determine which it was.

So, to try an differentiate, I held open the door that connects the instrument room and the wet lab, where a humidifier is still running. Unfortunately or fortunately, however you want to look at it, the air between those rooms doesn't seem to exhange much, because there was no change in the humidity and no change in the baseline. Meaning I can't really prove anything for now without doing further work. With the help of an electrician or electrical engineer I might be able to prove or eliminate the electrical feedback hypothesis, also, I could think of other ways to manipulate the humidity.

Personally, I actually think that (somehow) it is really the humidity that is directly causing the problem. The humidity in the room continues to meander slightly, and the baseline continues to follow the meander. Furthermore, when the humidifier was shut off, the flutuation didn't stop instantly... the baseline fell quite a lot as the humdity at first fell quite far and fast before stabilising. To my mind, that is not normal behavior for an electrical feedback. It's too coincidental - it seems to be that it has to be the humidity, however preposterous that sounds.

In terms of remediation, I think we're just going to have to keep that humidifier off and hope that the humidity and hence the baselines become basically stable. It is absolutely not an option to permit that baseline fluctuation to continue, and we cannot function without those two systems. Turning off the humidifer is the only thing I have done that has had any impact on the problem, so that is what it will have to be.

So I have to say for the record, I believe I have found that the Waters 2996 PDA and 2487 DWD are capable of being enormously sensitive in their output to the tiniest fluctuations in RH, certainly at low RH. To me, seeing this, it would actually have an impact on any future purchasing decisions I might make.

This has been a huge setback for me for a very busy schedule, so I may not be able to devote much more time to this issue now that the instruments are back to being usable. When I am able to find time to do more experiments and get the problem nailed down further I'll let you know what I find. In the meantime, I would like to thank everyone for their helpful comments and feedback - you guys totally saved me

Stephen

[DR]

Can you sparge the optical path of the detector with a low flow of bottled N2? Doing so should lock in a constant RH.

Old Beckman detectors used to come w/ barbed fittings on either end of a sealed monochromator chamber for just this purpose (though the need was different - to keep O2 from absorbing <200nm).