IDM's data migration methodology diagram above illustrates our incremental and iterative approach. For each increment / area the inner circles of the diagram are effectively repeatable stages which ensure risks are identified early. For instance;
Insurance Data Migration Planning & Strategy
At Insurance Data Migrations we understand the importance of a carefully planned and tailored approach to each migration project. The Migration Strategy depends entirely on the legacy and target systems involved and the type, quantity and quality of the data being migrated. Good communication between the project team and the business is necessary to install confidence, motivation and coordination between team members, business users and the wider network of key stakeholders.
Data Migration Project Planning
Both legacy and target system data are owned by the client and the migration process must be driven by the business decisions and processes. Engagement with the business right at the start of the project is vital in order to gather information on data stores and requirements and establish business and mapping rules.
You don't want to encumber a new system's performance with useless and incorrect data. A data migration can offer an ideal opportunity to have a good cleanse and tidy-up. Detailed extraction scripts based on your criteria can filter out information i.e. on or after a certain date for instance.
Insurance Data Migrations can offer advice on the amount of historical data which is typically migrated and in answering questions such as:
- Should we bring all or part of the data?
- If just part, which parts? - based on renewal date of policy or date the claim was created?
- Should we filter specific data?
- How should we access and extract the data from the legacy database?
- Are the desired fields of data importable into the new system, what must be modified? etc..
- What level of data quality is required?
These questions regarding the data and its requirement are business decisions. However, this is where IDM's professional knowledge of insurance and vast experience in data migration comes into play. Using this experience IDM will support and direct you towards the right answers to maximise the value of your data.
"The results of the initial data analysis are invaluable during the planning stage when deciding what to migrate and whether to migrate all or part of your data sets." Managing Director, Paul Johnson
At this planning stage we will also help you consider the different options available in terms of:
- data profiling tools and techniques,
- cleansing tools and data manipulation methods,
- mapping rules and techniques,
- planning resources for testing,
- live migration types,
- managing the transition.
Working as Part of a Larger Transition Project
A data migration project is traditionally part of a larger transition project. IDM understand that migration projects nearly always fit around other aspects of the project. So, after the bulk of the data migration development has been completed IDM are quite comfortable in maintaining the migration modules up until live day. It’s your call! IDM's aim is to be flexible and to pace your data migration at the rate that suits your business requirements at all times.
Insurance Data Migration Strategy Document
Once the project planning has been carried out and decisions have been made in terms of what and when the data is to be migrated a migration strategy will developed and documented.
Which strategy is recommended will depend on factors like budget constraints, resources available, the amount of data to be migrated and project deadlines.
This documentation will support a Comprehensive Audit Trail for the migration process and will give you full peace of mind that your migration is complete.
The Migration Strategy will also include;
- A detailed plan containing all the agreed steps of the migration.
- businesses rules and expectations including data security procedures,
- findings from the initial data and system analysis,
- necessary cleansing requirements from the Data Quality Report,
- techniques and tools to be used and mapping rules,
- resource profiling and testing and checking technique to be used.
Insurance Data Analysis & Cleansing
Data analysis and cleansing are on-going and iterative activities. Any new items of data that appear during the iterative phases must comply and be up to standard for the new system to run properly.
Your Data Integrity is Assured - IDM understand the complexities of insurance business rules & data standards required, and the different insurance systems you operate your business on and the continuous communication required with other systems i.e. EDI transactions, bordereaux, ELTO, Catastrophe Modelling etc.
IDM will carry out an independent unbiased assessment of your data which can include client information, high level policy information, risk information, documents, accounts, claims, diaries and notes. All data formats including relational databases, spread sheets, text files, scanned images and paper documents can be analysed and migrated to fit into a new system depending on your exact requirements.
Relevant data will be analysed and reported on. The reports will detail its accuracy and completeness i.e. where there are data quality issues, gaps and redundant questions in the data, duplication issues and validation problems etc. We can either use advanced data profiling software or do the analysis using SQL scripting or a mixture of both. This process will highlight data that does not meet the criteria set out in the new target system and what needs to be cleansed and modified prior to a migration.
Data Quality Assessment & Report
An initial data analysis and data quality assessment at the planning stage is invaluable when identifying: whether to migrate all or part of your data sets?, and what data fixes and data enrichment is required to ensure your data meets the required standards set by the business and performs correctly in the new system?
A Data Quality Report is produced off the back of the initial data analysis which includes information on the age/value and quality of your data, details of each issue along with IDM's recommendations as to how best to rectify them i.e. sometimes it can only be achieved through an automated process or in other instances it may be better for business users to rectify the data manually.
Data cleansing is done off the back of the data analysis and the data quality report. Migrating data from one system to another almost always requires a large amount of data manipulation. There will always be gaps in the data and parts of the data which require transformation.
All the factors which can render the data unfit for migration in its present state will need to be fixed. We can instantly use our expertise to improve the quality of your data through data fixes and enrichment.
Data quality management is an on-going activity and any new items of data that appear during the iterative migration phases must comply with data standards and business rules. Filters are also used to ensure the required level of data quality. Data quality perfection is impossible. However, IDM ensure that all critical data is up to standard for the new system to run properly. During extract, transform and load (ETL) stages data correction and cleansing usually takes place alongside the mapping stages.
Common issues found include:
- Duplication of information,
- Inaccurate or invalid data (such as postcodes, future inception dates),
- Data that doesn’t make sense within its own set of records, i.e. a policy which begins in 2001, but the client wasn’t put on until 2004),
- Data that fails business rules (i.e. a 14 year old driver or a 21 year old that passed his test ten years ago) etc.
Our detailed knowledge of insurance data ensures that these common issues are easily found and rectified quickly. When IDM update your data we will always provide a Comprehensive Audit Trail of what, when and how it has been updated.
Insurance Data Mapping
Data Mapping is a process which specifically identifies values from the legacy system which need mapping to new values in the target system. Migrating large volumes of insurance data correctly requires a good understanding of the complex and densely interconnected manual and system processes involved. A great deal of data stored in legacy applications is bound up with interdependencies and the mapping and merging of this data requires great care. It is critical to perform careful mapping to ensure continued communication with other systems (i.e. EDI, MID, ELTO) etc. IDM's experience and tried and tested techniques in data mapping ensures the data fits, makes sense and works correctly in the new system.
Examples are things like;
- With insurer legacy systems the risk classes and brokers / intermediaries will need mapping to the new values held on the target system.
- Often in broker legacy systems insurer descriptions are stored using abbreviations which new systems don’t recognise so as part of the mapping process, the old insurer descriptions get mapped to the correct values in the new system.
- The legacy claims may have descriptions which include the words ‘fault’ or ‘non-fault’, the new system may ask the fault / non-fault question separately. A mapping process could map the old description to the new question.
- Old legacy systems often contain less validation regarding user input than newer systems and so any poorly stored definitions that previously existed can have their data quality improved in a consistent way through the mapping process. (i.e. A country name that was abbreviated in several different ways will be mapped to one consistent value).
- More complex mapping is required where the legacy system held the policy sections as a free format text string, the migration may need to take this string for each policy and create the relevant sections underneath the policy on the target system.
Insurance Migration Development
The data migration development stages is when the extraction, transformation and loading of data takes place and all the data manipulations are carried out in accordance with business rules. IDM have experience in using ETL (Extract-Transform-Load) tools such as SSIS, and also in writing custom modules using technologies such as SQL scripting or CLR’s in C# in addition to using traditional languages such as Dibol and Cobol. Each development method has its own benefits depending on the demands of your particular migration. Quite often a migration may be made up of both ETL tools and custom modules. Whichever method is chosen the migration modules will include the business, mapping and cleansing rules that where defined in the earlier stages.
Naturally the users of your legacy system may be resistant to change. During the development stage IDM use a phased approach showing your users through early data conversions incremental results which help in setting their minds at rest.
For example once the development for client details is complete IDM will migrate and showcase to business users the data successfully working in the target system. This can be done through workshops and/or on a one to one basis with key stakeholders. These early conversions are then repeated for policies, claims, accounts, documents etc.
This phased approach helps fine tune each migration and delivers more accurate results in a quicker timeframe due to the data being seen early. Considering one area at a time reduces risk. Delivering value early provides opportunities for feedback and if necessary reworking so that the final release meets the business users exact requirements.
Month End Mini Migrations
Through-out a migration project, IDM will refresh the legacy data on a monthly basis and carry out mini-migrations of all the data. This approach ensures that any new values or trends in the legacy data can be identified and any new issues can be resolved early. It also usually means that come the live migration you have comfort in knowing that numerous end to end migrations have been completed successfully.
Go Live Migration
IDM have experience of delivering different types of data migrations to ensure the least amount of disruption to the business. It may be that all data is migrated overnight or over a long weekend and the old system is switched off except for viewing. Alternatively, the legacy system and target system may run in parallel for a period of time.
We also have experience in performing incremental data migrations where data may be migrated over a few months or even a year, usually using the renewal date by line of business or something similar. Each option has both advantages and disadvantages in terms of complexities and costs.
Insurance Migration Testing & Reconciliation
Testing and checking are often not given enough time and attention in mainstream projects and data migration projects are no different. To deliver a successful migration, testing requires an iterative approach throughout the migration process which is critical to reducing risk.
Data Migration Testing
Using a test environment which contains a read only copy of the source database and a clean copy of the target database. IDM test using a number of methods including focus testing using SQL and comparison of source and target system with migrated data.
Quite often it is the data that you can’t see on screen such as flags and counts that can causes problems in migrations. Comparison testing may not pick up certain data for instance in policies that have specific details that perhaps do not affect a large number. IDM’s experience and focus testing techniques ensure rare attributes to policies are identified and migrated completely. For instance, looking specifically at car insurance policies where additional drivers have convictions and checking these values are coming through in the target system correctly.
IDM will also do a certain degree of UAT system testing, For example completing tests on migrated policies ensuring that they can endeavour the end to end policy process. Testing to ensure migrated policies can be renewed, adjusted, cancelled and re-quoted.
IDM carry out tests after every release of the migration to ensure:
- The data migration process performs relatively fast and without any major bottleneck.
- All the required entities, e.g. customer records, policies, risks, accounts, etc. have loaded into the target database and there is no duplication or corruption.
- All data related to a particular entity is loaded into the relevant table in the target database and none of the data has been rejected by the target database.
- Every required attribute of every entity (present in the legacy database) has loaded into the target database.
- Previously identified bugs found either by IDM or the client in UAT can be tested by IDM after a fix has been applied.
- Mappings between the sources and target database are accurate and do not need changing.
- Each required business rule is implemented correctly in the ETL tool.
Reconciliation / Validation Reports
In order to reconcile the migrated data, IDM have a detailed suite of reconciliation scripts, which will extract total counts and summations of critical data for each of the business areas from the legacy, interim and target databases. Any discrepancies will be reported and must be understood and within an agreed tolerance limit. Typically reported in a spread sheet form these figures are broken down by business area to independently validate the end to end migration process. This ensures the mitigation of any differences and secures confidence in the migration.
As a simple example reconciliation can identify the number open and closed claims in the source system with the number of open and closed claims in the target system ensuring we are not missing off any critical data. The documented results of the Reconciliation will be an element of IDM's Comprehensive Audit Trail.
User Acceptance Testing (UAT)
Testing is very important it should not be compromised. As your company migrates from one platform or operating system to another, you need assurance that your data has been kept intact throughout the migration. Therefore, the results of each migration release need to be systematically tested. User Acceptance Tests (UAT) carried out by the business users will ensure the target system performs as expected once the migration data has been loaded. UAT by the business is critical in ensuring the client gets the best end product possible as even though IDM have many robust testing techniques in place, it is the client who knows their data the best!
Insurance Data Migrations will assist in identifying and correcting any problems that might occur in testing the data that has been migrated.
The exact type of testing and how the UAT testing is performed will depend on your particular project and environment. Here is a list of testing steps and tasks to consider which IDM can assist you with:
- developing a test strategy;
- planning the tests;
- setting up a secure test environment;
- identifying the best test team to analyse each area;
- creating different types of tests including data quality tests and testing the target systems functionality with migrated data;
- debugging, correcting, retesting, and regression testing;
- electronic documentation of results.
IDM's testing team will work alongside your testing resource to provide the information to thoroughly test and check the system giving the testers support throughout the process.
IDM's Comprehensive Audit Trail
Once all the analysis, development and testing stages are complete IDM will hand over all sign off documentation which details the data migration at both a business and technical level.
1). Data Migration Strategy Document 2). Data Quality Report 3). Data Migration Specification Document
This document is created during the data and system analysis stages and is used as a technical reference guide which explains what has been migrated at a field by field level. It gives a list of source fields from the legacy system against corresponding fields of the target system. Where there is no comparable source field it may just show a default value. Mapping may be included in this document or held in separate excel spreadsheets and referred to.
4). Data Migration Reconciliation Documents
These contain control totals for each part of the migration. They detail simple things like the number of clients and policies and gives evidence of more complex things like the value of your accounts by risk on a year by year basis.
5). Sign Off Data Migration Document
This document is aimed at the business user and there is no technical detail. It contains a section for each area of the migration including all details of transformations that were used, any data quality rules and details of any records that were excluded. It also contains a description of the logic used (i.e. business rules) to get the data from the old to the new system. It should be signed off by each team in the business prior to go-live. (i.e. Policy Administration, Stats, Claims etc.)
The above documents will provide a comprehensive audit trail of the migration process and will give you piece of mind that your migration is complete fully auditable and compliant.
IDM's Post Migration Aftercare Support
12 Months Warranty - Giving Our Customers Total Peace of Mind
IDM provide a single point of contact and 24x7x365 telephone support services to assist clients with any critical questions at any point in time.
IDM are on hand to resolve any post go-live migration issues for the full renewal period. However, from experience IDM’s iterative approach ensure the chances of any major issues arising are minimal.
You can expect to receive on-going comprehensive support.
Insurance Data Solutions Above & Beyond Data Migrations
IDM maintain excellent long term relationships with its clients and our collaborative approach and our understanding of insurance data and systems provides invaluable support above and beyond their data migration requirements.
IDM's ability to problem solve with flexible and speedy data solutions means our clients often go on to utilise our skills in ever more critical projects within the business.