EBME Forums Forums Maintenance Biomed Databases Data Migration

OK ... excitement is mounting as you fire up your new database system - (pause) - but what about all your legacy data?

These days I suspect it is quite rare for a database to start off with a "clean sheet", as it were. In most cases there will be perhaps thousands of records - probably representing years of work - that need to be preserved, and "brought across" to, and integrated into, the new system.

But ... how's it done? And how much time (and effort) is involved?

For what it's worth, I believe that this rather important step in commissioning a new database system may sometimes be a bit, shall we say, overlooked!

As mentioned elsewhere, my own approach is basically to get the legacy data into an Excel spreadsheet, spend some time (time well spent, in fact) cleaning it up then saving as a .csv file, before importing into a native datafile (.dbf) for cleaning up and further processing by "the system".

How do commercially available systems go about this, I wonder? Can it be done simply "at the touch of a button"?

By the way (like most other interesting topics), this has been touched on before.

Let's have another go at this one ...

I can't imagine that any system could convert all other systems at the touch of a button.

Whenever I do a conversion, I write a script according to the job in hand. For the most part, it's just a case of recognising delimiters.

I only ever seem to use MySQL as the database these days and its command set is so powerful in terms of conversions and imports that I don't need to look elsewhere.


I don't know if you've ever come across it, but CSVed is an excellent and free CSV manipulator.

No ... but I imagine that some commercial systems may claim (and indeed be able) to import data readily from other well-known systems. Or, at least, earlier versions of the same system.

It would be possible, just as long as the developer of the receiving system had full information about the (datafile structures of the) donor system. Or at least had some example datafiles available for "studying".

And ... if a guy had such information about all possible systems, then he could most likely work up a general interfacing routine ... using an intermediate datafile, maybe.

Yes, doing a migration on site (with all tools to hand) should be easy enough. A one-off, as it were. But my own "problem" is that many users of TM are put off using the thing because they can't get their existing data ported across easily. That is, data that I have no knowledge of. So I've been trying to improve that aspect of the program; to help them to "funnel their data in", as it were. As long as the "legacy" data can be saved to a .csv file (deliberately chosen for its popularity as a "save as" format), then that's the first step squared away, as far as I'm concerned.

The next step is not so easy:- having the program try to make sense of the data being presented! I'm trying filters, templates ... and stuff like that; but can't see a way without at least some decisions being asked of the (hopefully intelligent) user. Basically, I just try and provide the tools sufficient to carry out the task.

Yeah, you can get the system to recognise (the) delimiters, but it takes a person (generally a biomed) to be able to work out which field name each column of data needs to end up at! Sometimes they may match up straight away (or some of them, at least) ... but as you know there seems to be (for example) a fair number of ways to "spell" the word Equipment. And then ... does that actually mean equipment type (or class as I prefer to call it) ... or what?

In short (and having probably seen more than my fair share of equipment data sets) - the possibilities (and SNAFU opportunities) are endless!

But thanks for the link to CSVed. I hadn't come across it before, but it's nice to be able to view good old .csv files (and carry out various manipulations) like that, almost as if they were databases. I have always liked .csv ... and sometimes (often) "simple" is best.

Having had a quick "play" I have noticed some very nice features. Like being able to export to quite a few other formats (including many variants of .dbf, for example). So here's a big cheer for Sam Francke (the developer).

About MySQL ... yes, if I ever update my machines, and get hold of some modern software etc. then I probably should concentrate on that. But, to be honest, I can't see that happening any time soon. I'm really looking to get the next revision of TM done and dusted, then giving it all a rest for a while (a summer, or so).

Version 2.4 of CSVed is now available.

Version 2.5.3 of CSVed is now available.

Meanwhile (and I realise this is a bit of long shot, as I have never seen a medTester in the UK) but I need some information - and ideally a sample file - about the file format used by medTester MTIMPORT.MUP files. I need this for (software) testing puposes!

These are (were) the test results files generated by medTester (for uploading to a PC etc. via the medCheck module). I believe these files were simple .csv types, but I need the structure (fields, and the order in which they appear).

I have read that the .MUP files were named by medTester in the following way (month, day, time):- 01262233.mup.

Further clues are that the MTIMPORT.MUP format could also be recognised by the Sentinel - and, I think, MediMizer - CMMS, as well as some of the later Datrend "apps". So I'm hoping that someone may be able to help me out here.