IT firm optimizes data ops with LLM-automated validation

During the migration to Databricks, the client’s data department faced several challenges related to data quality. Migration processes flagged around 10% of all their data as needing further validation, as SQL results mismatched between the previous and new databases.

Manual validation would have required significant capacity commitment, so they were looking into automated solutions.
 

Taking their existing architecture as the basis, we built an LLM-powered validation layer that identifies, flags, and processes it to fit the new Databricks ecosystem.

Data is being migrated in 3-week waves. The system analyzes each wave, checks flagged data for errors, groups them by type, and issues proposed corrections based on source data. As data migration happens globally, and each migration wave is different, the solution is capable of identifying specific issues with specific use cases, and proposes solutions accordingly.