Skip to content
TabREM
← All case studies
Data reliabilityCase 07

Error Reprocessing Tool (ERT)

Error Reprocessing Tool data pipeline

A Kafka- and Azure-based observability and recovery pipeline orchestrating zero-loss migration of millions of legacy on-premise records to the cloud each day.

01 / 03Error recovery built beside the migration pipeline.

Contribution

Backend and data-platform engineering · observability · recovery

Domain

Enterprise data migration · event streaming · operational tooling

Availability

Internal Walmart product · no public implementation link

The problem

High-volume cloud migrations stalled when pipeline errors could not be inspected by domain and replayed safely. Limited observability produced manual debugging overhead and increased the risk of incomplete migrations.

What we built

  • Scalable synchronization engine: Consumed Azure Event Hubs/Kafka error streams in containerized Spring Boot services and partitioned records by domain into Azure SQL.
  • Reprocessing API: Exposed filtering and replay endpoints for granular or batch recovery to native Kafka topics.
  • High-observability dashboard: Centralized migration status, domain-level inspection, and manual replay controls.
  • Automated retries: Scheduled persistence-layer scans and initiated recovery according to configurable rules.
  • Containerized Azure architecture: Standardized deployments across Container Instances, Event Hubs, and Azure SQL for reliable multi-market scaling.
Next case studyUnified Data Platform big-data infrastructure

Have a system like this in mind?

Let’s scope it together.