Management of large data sets

[sassman]

Just curious how those of you who deal with very large data sets deal with them. I have spent the last six months putting a bunch of data into an MS Access database, but now my boss has decided that she doesn't want to use Access. I told her from the beginning that Access was not the best relational database, but she wanted to use it. So what do you use for large datasets? I'm talking too big for excel times 10. I'm guessing many of you are blessed with LIMS.

[Don_Hilton]

I keep them as "flat files" and suck them into SAS as needed. I have very large peak tables from GCxGC-TOFMS and a growing collection of them.

[varossf]

@Sassman
Depending on the budget, it might be worthwhile to look at Agilents OpenLAB ECM. It van sucks in the raw data and printed PDF reports, extract meta data (=results) from the files and store these automatically in the SQLServer or Oracle database (your choice).
Once int the databse, you can make views on top of it and select your reporting engine, like Excel; Crystal reports; whatever.
It also helps you to go paperless and manage the large amounts of data you rpoduce properly.

[aceto_81]

Or you could take a look at postgreSQL.

You still can use MS Access as a front end, while the postgres is the workhorse.

Ace

[sassman]

I am definitely hesitant about sinking money into something like openlab without knowing how well it will work for our application. We need something flexible enough to allow us to build custom data structures. I am always wary of software designed for "ease of use" (microsoft products come to mind). My experience is that these products lack flexibility.

I was also thinking postgresql or mysql. It shouldn't be too hard to export the access database to postgresql. Then I have the flexibility that I want and my boss gets an easy frontend to the data. I will probably recommend this course of action in our meeting, but I'm probably going to be overruled by all of my non-techie colleagues who are in love with excel and don't know anything more than flat files. Wish me luck!

[aceto_81]

I am definitely hesitant about sinking money into something like openlab without knowing how well it will work for our application. We need something flexible enough to allow us to build custom data structures. I am always wary of software designed for "ease of use" (microsoft products come to mind). My experience is that these products lack flexibility.

I was also thinking postgresql or mysql. It shouldn't be too hard to export the access database to postgresql. Then I have the flexibility that I want and my boss gets an easy frontend to the data. I will probably recommend this course of action in our meeting, but I'm probably going to be overruled by all of my non-techie colleagues who are in love with excel and don't know anything more than flat files. Wish me luck!

Porting your access db to postgresql is relative easy altough there are some traps (like the use of yes/no type of fields). I did this a while ago, so I've found a lot of workarounds. If you're stuck, just ask, maybe I can help you out.

Your non-techie colleagues are maybe happy with the datasheet view of Access forms?

Anyway, good luck

Ace