Data Migration Frequently Asked Questions

Over the last 30 years we have carried out thousands of data migrations accumulating a wealth of experience which we now apply to all data migration projects.

Below is a list of frequently asked questions to help alleviate any fears you may have when considering migrating to a new system.

Question: There are many library systems in the marketplace, where do I start?

The starting point in any analysis of a new system, begins with a determination of needs.

Who is your user community and what are their needs?
Are you servicing those users in the best possible way and if not what could you be doing differently?
How does your library system software support that activity? what is missing? what is holding you back?
Is the data in your system right now in the best shape that it could be?
Does this impact service to your users?
Is the user experience easy and intuitive or does it prevent some users from participating in the library service?
Can you easily control, amend and manage your data?
Are processes efficient and does the library system allow your staff to do more without sacrificing quality and service?
Are you getting the right amount of management information to make informed decisions about budgets and services?

Question: What is data migration? How is it performed?

Most libraries and archives today have data in a database of some form or another.

When the Soutron system is selected to manage the library or archive, we undertake to migrate the data into Soutron. It is part of our System Implementation and is carried out within a formal project.

The first task involves the extraction of data from the client’s original system in an electronic format.

We ask the client organisation to do this extraction for a number of reasons. We do not have a licence to use their existing software and do not wish to contravene your licence terms. Doing it this way ensures that the data that is is handed over to us as a complete set of data, avoiding any issues of possible missing data.

The client knows their own data and as such can check and confirm that the data being provided is complete and accurate. The data can be in a variety of formats. Much depends on how data can be extracted from the existing software and database. We prefer XML but often receive data in MARC21 format or CSV. It is possible that there are multiple data sets if part of the project is bring these together to establish one database

The transfer of data is then arranged via a secure FTP or SFTP account from the client organisation to our servers. We adhere to very strict security procedures when it comes to handling your data and as such never accept USB sticks or portable media devices. The data files are then checked and moved into a Test Server that is set up for the project.
We download the data from the FTP site and start to analyse the data files using a variety of data checking tools. We check the volume of records, we look at the file and field structures, the way that the data has been tagged and structured. We perform a variety of tests and checks to establish how consistent the data is, whether fields are missing data etc. This provides us with a clear picture of the existing data and we can start to assemble ideas about how we might better structure the data for our system and what tasks we might need to perform before we can load it into the test database. Once the data is loaded onto our test database, this can be seen by everyone involved in the project and ideas can be demonstrated to improve efficiency of the structure of the data.
We then design a schema for the data and prepare the test database ready to accommodate the data load. It may be necessary to perform a number of loads into the database depending on the data that we have been provided. All data loads are performed from XML files that we prepare in advance from the data supplied. This allows us to then write scripts to perform actions on the data so that it is presented in the database in the best shape possible.
Feedback is sought from the client library staff on the data that is loaded into the Soutron Test Instance. This takes the form of comments and observations that may require script changes to reshape or reformat the data and for it to be then loaded again (and again), checking the data conversion for accuracy and completeness so that when ready there can be a final data extraction and data load and the new system can Go Live.
As part of the project implementation some training and instruction on the system is provided. This is not the formal training that is given later in the project but sufficient to allow library staff to become more familiar with the system and to examine data and feedback on the data loaded into the database.

The most important part of any new system implementation is migrating the data and ensuring that the data structures and the way in which the data is held and presented in the database is ideally suited for the library and its users.

Question: Can I use all of the data from my current archive / library management system?

Normally yes! This is certainly our goal providing that the data can be exported from the original legacy system and is structured data. The Soutron system accommodates different types of data and is flexible to accommodate all types of data in the one system – both library and archive data.

Question: Can you import and host all of my record attachments, images, PDFs?

Yes, but much of the loading of files such as PDFs, Office Documents and images depends on the quality of the data files and the way in which files are named and formatted. Image files differ considerably from document files and can have a variety of additional metadata and may involve resizing and clean up before they are deemed good enough to load into the database.

Question: How long does a typical migration take?

The timescale for the migration varies depending on a number of factors. Much of the time is dependent on the number of data loads needed and the extent of the data checking that is needed.

If the original data is particularly poor in quality or structure or both, it may take considerable time to reach a point where it is deemed suitable for live operation. The time taken to reach this stage can also depend on library staff availability, since the burden falls on them to check data and feedback to the project team.

No one else is in a position to determine how data should be structured and if it is ready. Some projects take only one data load, if the data is straightforward. Others can take 4 or 5 months especially where data is not provided at the outset but appears late in the project or where the data provided is very poorly structured and needs multiple loads.

Question: Can I run both systems (the old one and the new one) until I am happy to make the switchover?

Yes! We advise that the original system continues to be used and is operational until the data migration is signed off as being correct. This means that there is no disruption to services and only when the data is ready and everyone is happy is a final copy of the data taken from the original system and converted. (Remember that the conversion routine is automated using XML schemas as transforms so that it is prepared according to the needs explored in the test loads. This makes it possible to perform repeated loads and checks on the data without manual effort to make changes in the data.

Once the new system is deemed acceptable and everyone is happy using the new system, the old one can be decommissioned. We recommend that the old system is kept in place for a period of a few weeks just to check back on any item in the early weeks of use of the new system. Some clients, knowing the data loads are good and having conducted repeated tests during the conversion turn off the old system at the point of Go Live. Such is the confidence that the migration and conversion process provides.

Question: Will I lose any data in the migration process?

Data is never “lost” however if the data provided in electronic format is unstructured or in a corrupt format it is unlikely to be correctly interpreted or understood programmatically and as such it may be impossible to load into a field structure in the database.

In over 30 years of business, only on a very few occasions have we faced a situation where data could not be converted. We take great care to minimise data loss.

Often we are able to recover, clean and tidy up data, increase accuracy and consistency and present it far better than when it came out of the original host system.

Question: What happens on go-live day?

The URL for the Live system is activated and the database is loaded with the latest copy of data from the test system. The user file has been populated at this point and tested for Single Sign On. From this point on the library staff can point their web browser to the new URL and perform the tasks that they have been trained to do using the new software. The Search Portal(s) will be active and end users will now link to the URL for the Search Portal.

Preparation and attention to detail makes the Go Live event a natural and straightforward activity, one that everyone is prepared for in terms of training in how to use the application and in having the data loaded and available via the Search Portal.

The data from the test system that has been used throughout the implementation process will be loaded into the final database that has been configured and made ready in advance. Remember this will have been tested beforehand with the data, the only difference is that the data loaded is the very latest set of data to have been extracted from the original system which has been frozen and is only used for Search purposes. No new data will have been added since the last extraction of data.

Question: How do I explain the process to my managers?

The implementation of a new system is the same as any IT project that is undertaken by your organisation.

The project follows a proven methodology with a dedicated project manager in charge who communicates all tasks and activities via a project plan and Gannt chart to all project members and meets regularly with the team assigned to implement the system. A project coordinator from the client organisation is assigned to the project to provide decision making and assist the project manager. This is preferably someone with detailed knowledge of the needs of the organisation in respect of the library or archive.

A set of milestones are agreed to measure progress and a timeline established for the project to be implemented. There will be a need to consult with the internal IT team about matters relating to security of data and systems, set up of users for Single Sign On, and possibly any integration with the web site or other systems.

The Finance and Procurement teams need to sign off against any budgets and be involved in the purchase and contractual aspects of the project.

In Summary

We hope the above helps to answer your questions regarding the whole process of data migration when moving data between systems.

At first it may all appear to be a mission impossible, a task not for the faint of heart. However, with our expertise and guidance, we can assure you that the process will be carried out professionally, securely and to the very highest standards in the industry today.