What is Data Reconciliation

Data Reconciliation Definition

Data Reconciliation is a process of verifying data during data migration, where the target data undergoes a comparison with the original source data to make sure that the migration architecture has transferred the data accurately. There are high chances of mistakes to be present in the mapping and transformation logic. Along with this, failures like broken transformation and runtime failures can lead to data loss or invalid data. This can further lead to a range of predicaments similar to incorrect values, missing records, and values, innocently formatted values, duplicate records, broken relationships across tables or systems and so on. 

Data reconciliation and gross error detection are the commonly used methods to improve the accuracy of data which results in a satisfying the plant model of an industry. These methods were widely applied to petrochemical-plants, mineral industries, refineries, and so forth to deliver accurate plat-wide accounting. Deploying data reconciliation, the industries also aim to receive a higher profitability in the plant operations.

In a modern chemical plant or refinery, numerous variables such as temperatures, flowcharts, pressures, levels, compositions and so on are regularly measured and recorded for further process control, process economic evaluation, and online optimization. Data acquisition systems and computers facilitate the gathering and processing of these massive volumes of data. The use of computers to collect data enables to obtain data at a greater frequency and aids to eliminate errors that occur during manual recording. The increasing amount of bulk data can be broken down for further improving the reliability and accuracy of processed data through a systematic checking and treatment. The Processed measurements are corrupted by minor errors during the processing, measuring, and transmission of the measured signals.

The total error identified in a measurement that is derived from the difference between the measured value and the original value of a variable is considered as two different types of errors namely, gross error and random error. Gross errors occur due to nonrandom events such as failure of instrumental functionality, corrosion of sensors, miscalibration and so on.  Random errors occur when inherent fluctuations occur unpredictably in the readings of measurement apparatus or in the interpretation of the experimenter’s reading on the instrument.

The Process of Data reconciliation 

Data reconciliation process targets at correcting measurement errors that occur due to random errors. The principal contradiction between other techniques and data reconciliation is that it clearly makes use of process model constraints and acquires estimates of variables by altering and adjusting process measurements so that the estimates satisfy the constraint. For data reconciliation to be efficient, gross errors should also be nil.

Conventional approaches to data reconciliation have frequently relied on simple record counts to keep track whether the expected number of records have migrated. This was mainly due to the importance of processing essential data to execute field-by-field validation. Today’s data migration solution delivers comparative reconciliation capabilities and data prototyping functionality that allows full volume data reconciliation testing.