Data Quality: A Heavy Burden on Every Transformation

You are the new Product Enabler for Natuvion SOPHIA, our comprehensive analysis tool. For someone unfamiliar with Natuvion SOPHIA, can you briefly describe what it is and what makes it special?

Natuvion SOPHIA is an exceptionally powerful data analysis tool. It offers system-level, customized analyses that are user-friendly and quickly executable by anyone. With over 500 analyses across five different areas, it performs a holistic system scan. The results form the basis for estimating the project costs and duration of a migration. This comprehensive analysis toolset plays a crucial role, especially in the data discovery phase of a transformation project. Whether you need a basic overview, information on data volume, or detailed insights into data quality, Natuvion SOPHIA has predefined and tested analysis profiles for common migration scenarios.

In the latest edition of the 2024 Transformation Study, it is notable that almost one-third of companies are surprised by poor data quality during their transformation. What is your experience?

This is always a delicate issue. A thorough data analysis is essential before a transformation. My experience is that 50% of companies claim their data is well-maintained and in good condition, while the other 50% suspect they have weaknesses in this area. When we use our analysis tool Natuvion SOPHIA to examine the data, we uncover deficiencies in every customer’s data.

Why does data maintenance still seem to be lacking?

It’s not that companies take data quality lightly. The problem lies more in responsibilities: who takes care of which data, when, and how. Defining who is responsible, and how they are held accountable. And how data quality is measured. Another issue is data history. If data is already incomplete or incorrect, carrying it over during migration will inevitably lead to errors—an onerous burden for any transformation.

But shouldn't it be straightforward?

Yes, if the data fields are correctly set up and the business areas entered their data in the right places with the correct parameters, the data set is properly created and can be accurately read and analyzed by the system. Once this is correctly done, the time effort is manageable. The real problem is that the data entry person may not even know that they have made an error. No one does this intentionally. But such errors propagate through the entire data landscape, causing numerous follow-up errors. Data entry involves preliminary checks that assess technical correctness before archiving. These findings can be used to adjust business processes to meet the correct technical requirements. In this context, NTT DATA Business Solutions' product GLASSWING is worth a look. GLASSWING prevents data from being incorrectly entered across the company.

Are there additional problems with data?

We often discover deficiencies in technical and business implementation, especially during our workshops. These issues arise when business users are asked technical details. A typical question might be: Where are the tables needed for migration? How are they structured? How should they be populated? We often receive puzzled looks in response. These are knowledge gaps leading to inadequate and erroneous data maintenance.

But how can this be? Aren't people properly trained?

Often, it boils down to a lack of know-how. Nothing is more persistent than a faulty entry! If something has always been done incorrectly, every new person learns the "wrong" process. That’s why it’s crucial to fundamentally question and review all processes before a transformation. Are the setups correct? Are the information and processes relevant and accurate? Who needs this data, and how is it processed? Natuvion SOPHIA analyzes these details and identifies errors. It reviews which modules need to be used, how master data should be maintained and created, and more, ensuring that only error-free or cleaned data is migrated.

Does this mean that migrating unchecked data means you’re no further ahead, despite the new system?

Exactly. You must thoroughly review and analyze your data. Often, data maintenance is overlooked due to time constraints. Some systems can immediately detect incorrect entries, allowing for prompt corrections. However, these systems are rarely used, and the consequences eventually catch up. The 2024 Transformation Study shows that a significant motivation for IT system transformation is the use of new technologies like AI. But if you don’t get your data in tip-top shape, you won’t be able to fully benefit from AI’s many advantages!

When customers say their data is fine and they don’t need additional analysis, what do you do differently in your analyses?

We offer customers a comprehensive data assessment. When they use this service, we examine the structure and data in great detail, diving deeper, in a more targeted way, and more systematically than typical customer checks. Our extensive experience with complex, large-scale, and international transformations guides us in identifying crucial details. Customers usually check data only for completeness and correctness, while we also review processes and mapping. During the assessment, we identify which mappings will continue as they are, where requirements will change, or what may no longer meet current standards. Our customers receive extensive analysis and advice to prepare and execute the desired migration scenario, or we may suggest a different, potentially more effective migration strategy.

With the introduction of GDPR, there is a significant risk of penalties for incorrect data usage. Have all companies implemented deletion and archiving concepts?

Most companies now have a deletion concept. However, many still use traditional archiving methods, which are no longer sufficient. Data must also be removed from archives when no longer needed. System transformations have recently prioritized this issue, partly due to the significant fines reported in the media. Other companies face capacity limits and need to delete or archive data accordingly. This aligns with the findings from our 2024 Transformation Study, where "Review & Enhancement of Data Quality" was the top technical measure in Germany at 55%, followed by "Data Volume Reduction" at about 35%. Reducing data volumes and removing obsolete data is critical, especially when transitioning to SAP S/4HANA. Knowing which data to delete or keep is essential, and Natuvion SOPHIA is an invaluable aid in this process.

Thank you for the conversation.