Page 1 of 1

Opening .d files in MATLAB

Posted: Fri Nov 19, 2010 2:09 pm
by marina
I have several .d files I would like to open and process in MATLAB.
Does anyone know what's the structure of a .d file? Is there some kind of Agilent document explaining the file format?
I don't have Chemstation myself, so I can't just save it into another format and all the translators I find are commercial..

I'd appreciate your help!

Posted: Fri Nov 19, 2010 2:51 pm
by DR
You would probably be better off exporting delimited text or ANDI files and pulling them into MATLAB.

Posted: Fri Nov 19, 2010 3:08 pm
by Peter Apps
How about this ?

viewtopic.php?t=14192

Seek and ye shall find.

Peter

Posted: Fri Nov 19, 2010 4:19 pm
by marina
I tried, but for some reason I can't get to install the converter...

Posted: Fri Nov 19, 2010 5:00 pm
by GasMan
Marina,

I hope that you realise that the ChemStation .d "files" are not really files. They are folders that contain files and other folders. If you are looking for the raw data file, this will have the .ch extension and you will find these in the .d folder. The .ch files are I believe in a binary format.

Gasman

Posted: Mon Nov 22, 2010 7:10 am
by Peter Apps
I tried, but for some reason I can't get to install the converter...
You can also try Adron: http://www.adronsystems.com/index.html

They have downloadable demo software. It turned raw Varian MSWS data into an ASCI file for me, and Chemstation is one of its source file options.

Peter

Posted: Mon Nov 22, 2010 9:04 am
by marina
Thanks, I'll try the demo software.

I know .d "files" are actually folders, and I suspect that I should extract the data from the .ms file. However, it seems to contain a lot of header information and I am unsure about were real data starts.

Posted: Mon Nov 22, 2010 2:29 pm
by lmh
I would really, really recommend you find someone with Chemstation and ask them to copy your data from a disk and export it in .aia format (i.e. cdf). It's so easy to do. There are several flavours of Chemstation file, depending on what version of the software you have (Chemstation's 16-bit version has a totally different file-format to the 32-bit version, but Agilent have hidden this seamlessly such that you won't notice unless you try to open 32-bit on a 16-bit Chemstation).

If you run into problems with the exported file, check it's big. If it's stupidly small, find someone with a more up-to-date version of Chemstation. The MS export thing in 32-bit didn't work properly for several versions (from the file-size I suspect it exported the TIC only, and not full MS data). It now does work properly.

Incidentally, congratulations to eselmeister for producing some free software to look at a wide range of different MS raw file formats.

BUT, I have slight concerns, though, about his comment on decoding file formats. I've also tried this in the past. The hideous worry is that if you do it by inspection, there's always some byte somewhere whose purpose remains unclear; you never know exactly what options the original programmer had in mind. And when you come across some piece of data that always has one of several values, you never know whether you've seen the entire list of values it can have, and what the other values might mean, and how this meaning alters the interpretation of the data whose meaning you think you know...It means you're always skating on thin ice, and have no guarantee that you're interpreting the data correctly.

If any instrument-manufacturer ever reads this, let them beware. If you don't publish your file-format, and someone decodes it and writes freebie software, then when it all goes pear-shaped because they were ignorant of something you later rely on, YOU will share the blame with them in the eyes of your (angry) customers. Good freebie software adds value to instruments; make life difficult for freebie programers, and you are alienating customers.

Posted: Mon Nov 22, 2010 3:44 pm
by Peter Apps
Incidentally, congratulations to eselmeister for producing some free software to look at a wide range of different MS raw file formats.

If any instrument-manufacturer ever reads this, let them beware. If you don't publish your file-format, and someone decodes it and writes freebie software, then when it all goes pear-shaped because they were ignorant of something you later rely on, YOU will share the blame with them in the eyes of your (angry) customers. Good freebie software adds value to instruments; make life difficult for freebie programers, and you are alienating customers.
Well said lmh !

And the same goes for manufacturers who make it difficult to add in-house hardware, and to integrate it with the GC controls. The exciting developments will get done on other manufacturer's intruments.

Peter

Posted: Tue Nov 23, 2010 10:38 am
by macgyver
If it's the MSD Chemstation .ms file format you're after, see below (sorry to you non-programmers!). It hasn't changed since the days of the Pascal Chemstation (although I could be stand corrected). It was originally listed in an appendix of the HP 59970 MS Chemstation Handbook which is long out of print. I managed to salvage a copy from the trash many years ago, so I can scan the relevant pages if you need it. There's not much more useful information than the listing below though.

I didn't have a scanner handy, so the listing below is a direct copy from an old usenet post: http://groups.google.com/group/sci.tech ... ce26fb6331


typedef struct HPUX_Header_Record {
char N_File_Number; /* No. of chars in File_Number */
char File_Number[3];
char N_File_String; /* No. of chars in File_String */
char File_String[19];
char N_Data_Name; /* No. of chars in Data_Name */
char Data_Name[61];
char N_Misc_Info; /* No. of chars in Misc_Info */
char Misc_Info[61];
char N_Operator; /* No. of chars in Operator */
char Operator[29];
char N_Date_Time; /* No. of chars in Date_Time */
char Date_Time[29]; /* Tue Jan 06 98 12:24:50 PM */
char N_Instr_Model; /* No. of chars in Instr_Model */
char Instr_Model[9];
char N_Inlet; /* No. of chars in Inlet */
char Inlet[9];
char N_Method_File; /* No. of chars in Method_File */
char Method_File[19];
unsigned long File_Type;
int Seq_Index;
int ALS_Bottle;
int Replicate;
int Dir_Entry_Type;
unsigned long Dir_Offset;
unsigned long Data_Offset;
unsigned long Run_Table_Offset;
unsigned long Normalization_Records_Offset;
unsigned int Extra_Records;
unsigned long Num_Data_Records;
unsigned long Start_Ret_Time;
unsigned long End_Ret_Time;
unsigned long Max_Signal;
unsigned long Min_Signal;
char Unused[214];
};

typedef struct HPUX_Norm_Record {
double Norm_Mass;
double Slope;
double Intercept;
};

typedef struct HPUX_Spectral_Record {
unsigned int N_Words1;
unsigned long Ret_Time;
unsigned int N_Words2;
unsigned int Data_Type;
unsigned int Status_Word;
unsigned int Num_Peaks;
unsigned int Base_Peak;
unsigned int B_P_Abundance;
unsigned int Mass[1000];
/* Abundance = (Value in Bits 0 to 13) * 8^(Value in Bits 14 to 15) */
unsigned int Abundance[1000];
unsigned int N_Words3;
unsigned int N_Words4;
unsigned int Unknown; /* Status word? */
unsigned long Total_Signal;
};

typedef struct HPUX_Dir_Record {
unsigned long Spectrum_Offset; /* in words */
unsigned long Ret_Time; /* milliseconds */
unsigned long Total_Signal;
};

Posted: Tue Nov 23, 2010 12:51 pm
by lmh
In LC Chemstation world at least, the .ms format has changed since the days of Pascal.

It had to. The old version saved masses as a 16-bit word, which meant it couldn't handle high resolution and high mass-range at the same time.

Posted: Fri Nov 26, 2010 1:55 pm
by marina
Thanks macgyver! I'll try to see if I can manage to open the files with this info.... I had been trying to find that Handbook for a while, but I think this is more than enough...

Posted: Sat Nov 27, 2010 6:47 pm
by eselmeister
Hi Marina,

what has gone wrong when you tried to install the Agilent converter? I may can give you assistance to install it, if you'd like to.

http://www.openchrom.net/main/content/contact.php

Cheers,
eselmeister

Re:

Posted: Tue Jul 19, 2011 6:00 pm
by JSen
If it's the MSD Chemstation .ms file format you're after, see below (sorry to you non-programmers!). It hasn't changed since the days of the Pascal Chemstation (although I could be stand corrected). It was originally listed in an appendix of the HP 59970 MS Chemstation Handbook which is long out of print. I managed to salvage a copy from the trash many years ago, so I can scan the relevant pages if you need it. There's not much more useful information than the listing below though.
This is very useful information and I'm extremely grateful for it. If you are still able to produce those scans, I'd be even more grateful. :lol:
I would really, really recommend you find someone with Chemstation and ask them to copy your data from a disk and export it in .aia format (i.e. cdf). It's so easy to do. There are several flavours of Chemstation file, depending on what version of the software you have (Chemstation's 16-bit version has a totally different file-format to the 32-bit version, but Agilent have hidden this seamlessly such that you won't notice unless you try to open 32-bit on a 16-bit Chemstation).
Comparing the two, the format hasn't changed that much. I've not managed to get a 32-bit file onto a working copy of the 16-bit ChemStation to check the results myself, but if the old ChemStation can't read it, it's because they didn't code the loading routine properly and hard-coded some of the offsets! Normally an older version of ChemStation should be able to read everything except the newer header information.

Jon

Re: Opening .d files in MATLAB

Posted: Tue Jul 19, 2011 8:53 pm
by sepscientologist
Hi Marina,

We don't know what kind of data you have. Is it HPLC or GC or GCMS or ???

If it is LC and you just have a few files I could convert them into CSV which you can read into Matlab
or just find someone with chemstation and export them as CSV on somebody elses machine.