The Data Migration Graveyard: Why 60% of ERP Projects Die in Data Migration — And How to Be the 40% That Survive
There is a cemetery I visit sometimes.
Not the kind with headstones and flowers. The kind with server rooms and regret.
It is where ERP projects go to die.
And if you look closely at the graves — if you read the autopsy reports, the post-mortem analyses, the quiet confessions of project managers over late-night coffee — you will find the same cause of death written again and again:
Data Migration.
Not the software. Not the budget. Not even the resistance to change.
Data. The silent killer.
The Lie We Tell Ourselves
Every ERP project begins with the same beautiful lie:
“We’ll clean up the data during migration.”
I have heard this sentence a hundred times. Maybe a thousand. It is spoken with such confidence, such certainty, that everyone in the room nods along.
And then, six months later, the same people are sitting in the same room, but now their faces are different. Now there is fear. Now there is blame. Now there is the slow realization that they are standing in quicksand and every movement pulls them deeper.
Here is the truth that nobody wants to hear:
You cannot clean a river by pouring it into a new glass.
If your data is dirty, migrating it to a new system doesn’t clean it. It just gives your dirt a new address.
The Three Ghosts of Data Migration
In my years of implementing ERP systems, I have met three ghosts. They haunt every migration project. And until you learn to see them, they will haunt yours too.
Ghost #1: The Ghost of Data Past
This ghost lives in your legacy system.
It is the customer record created in 2007 that nobody has touched since. It is the item code that was “temporary” and is now embedded in fifty reports. It is the supplier that went bankrupt five years ago but still appears in your vendor list because nobody dared to delete it.
The Ghost of Data Past whispers: “Keep everything. You might need it someday.”
And so companies migrate millions of records. Dead customers. Obsolete products. Transactions from a decade ago that serve no purpose except to slow down queries and confuse users.
The exorcism: Before you migrate anything, ask one question — “If this data disappeared tomorrow, would anyone notice?” If the answer is no, do not migrate it. Archive it. Let it rest.
Ghost #2: The Ghost of Data Present
This ghost is trickier. It hides in plain sight.
It is the duplicate customer records — one spelled “Al-Rashid Trading,” another “AlRashid Trading,” a third “Al Rashid Trading LLC.” Your old system treated these as three different customers. Your users knew they were the same. Nobody fixed it because the system “worked.”
It is the inventory that shows 100 pieces in the system but only 73 on the shelf. Everyone knows the number is wrong. Everyone has workarounds. Nobody has time to fix it.
The Ghost of Data Present whispers: “We’ll fix it in the new system.”
No. You won’t. Because the new system will not magically know that three records are one customer. It will not count your physical inventory for you. It will not correct the sins of years of neglect.
The exorcism: Data cleansing happens before migration, not during. Not after. Before. This is non-negotiable.
Ghost #3: The Ghost of Data Future
This ghost is the most dangerous because it doesn’t exist yet.
It is the field that your new system requires but your old system never captured. It is the classification scheme that makes perfect sense on paper but has never been applied to your 50,000 items. It is the cost structure that the new system needs but nobody has ever calculated.
The Ghost of Data Future whispers: “We’ll figure it out during go-live.”
And then go-live comes. And nobody has figured it out. And the system that promised efficiency becomes a prison of mandatory fields that nobody knows how to fill.
The exorcism: Map your new system’s requirements first. Identify every required field, every mandatory classification, every piece of data the new system expects. Then — and only then — assess whether you have that data today or need to create it.
The Migration Sins
Let me tell you about the seven sins I have witnessed. Each one has killed at least one project. Some have killed many.
Sin #1: Starting Too Late
Data migration is not a phase. It is a parallel workstream that begins on Day One and ends after go-live.
The companies that fail treat data migration as something to worry about “later.” Later becomes too late. The timeline compresses. Shortcuts are taken. Garbage flows into the new system.
The righteous path: Start data assessment in Week One. Not Week Ten. Not “after we finalize requirements.” Week One.
Sin #2: Delegating to IT Alone
Data migration is not a technical project. It is a business project with technical components.
IT can write the scripts. IT can move the files. But IT cannot tell you whether “WIDGET-A” and “WIDGET-A-NEW” are the same product. Only the business knows. Only the people who live with the data every day.
The righteous path: Every data domain needs a business owner. Items need someone from operations. Customers need someone from sales. Suppliers need someone from procurement. These people must own the cleansing, the mapping, the validation.
Sin #3: The Big Bang Approach
Some companies try to migrate everything at once. One massive extraction. One massive transformation. One massive load.
And when it fails — when the customer records don’t match, when the inventory is wrong, when the open orders are corrupted — there is no way to know where it went wrong. Everything is broken. Nothing is traceable.
The righteous path: Migrate in waves. Masters first — items, customers, suppliers, chart of accounts. Then balances — inventory, open AR, open AP. Then historical transactions if needed. Each wave validated before the next begins.
Sin #4: Skipping Reconciliation
I have seen companies migrate data and never check if it arrived correctly.
They assume that because the script ran, the data is right. They trust the technology. They skip the boring, tedious work of comparing source to target, record by record, field by field.
And then, three months after go-live, someone discovers that 2,000 customers are missing. Or that inventory values are off by a million dirhams. Or that ten years of transaction history simply… vanished.
The righteous path: Reconcile everything. Total record counts. Sum of values. Hash comparisons. Whatever it takes to prove — mathematically, irrefutably — that what left the old system arrived in the new one.
Sin #5: No Dry Runs
The first time you run your migration should not be go-live weekend.
Yet I have seen this. Companies that “didn’t have time” for trial migrations. Companies that tested their scripts on sample data but never on the full dataset. Companies that discovered, at 2 AM on go-live Sunday, that their migration takes 47 hours — not the 8 hours they planned.
The righteous path: Minimum three dry runs. Full data. Full validation. Each run teaches you something. Each run makes the real migration smoother.
Sin #6: Ignoring Data Dependencies
Data has relationships. Customers connect to invoices. Items connect to BOMs. BOMs connect to work orders.
If you migrate items before the unit of measure table, your items have no UOM. If you migrate invoices before customers, your invoices have no customer. If you migrate work orders before BOMs, your work orders reference nothing.
The righteous path: Map dependencies before you write a single line of migration code. Build a sequence. Follow it religiously.
Sin #7: No Rollback Plan
What happens if migration fails?
I ask this question in every project. The answer is usually silence. Or worse: “It won’t fail.”
It might fail. Systems crash. Scripts have bugs. Data has surprises. And if you have no plan for failure, failure becomes catastrophe.
The righteous path: Define your rollback before you define your migration. Know exactly how you will restore the old system if the new one doesn’t work. Test the rollback. Make sure it works.
The Migration Framework That Works
Enough about what goes wrong. Let me show you what goes right.
Phase 1: Discovery (Weeks 1-4)
Objective: Understand what you have.
- Inventory all data sources (not just the main ERP — spreadsheets, Access databases, paper records, tribal knowledge)
- Profile data quality (completeness, accuracy, consistency, duplication)
- Identify data owners for each domain
- Document known data issues
Deliverable: Data Assessment Report — a brutally honest inventory of your data reality.
Phase 2: Strategy (Weeks 5-6)
Objective: Decide what to migrate and how.
- Define migration scope (what moves, what archives, what dies)
- Choose migration approach by data type (automated, semi-automated, manual)
- Establish cleansing rules and transformation logic
- Create migration sequence based on dependencies
- Define validation and reconciliation procedures
Deliverable: Data Migration Strategy Document — your battle plan.
Phase 3: Cleansing (Weeks 7-14)
Objective: Fix the data before you move it.
- Execute cleansing activities by data domain
- Merge duplicates
- Fill missing required fields
- Standardize formats (names, addresses, codes)
- Validate against business rules
- Get sign-off from data owners
Deliverable: Cleansed data ready for migration — your data owners have certified that it’s correct.
Phase 4: Development (Weeks 8-16)
Objective: Build the migration machinery.
- Develop extraction routines
- Develop transformation logic
- Develop load procedures
- Build reconciliation reports
- Create error handling and logging
Deliverable: Migration toolkit — tested on sample data, documented, version-controlled.
Phase 5: Testing (Weeks 14-20)
Objective: Prove it works.
- Dry Run 1: Full extraction, transformation, load. Expect failures. Learn from them.
- Dry Run 2: Full migration with reconciliation. Measure timing. Identify bottlenecks.
- Dry Run 3: Dress rehearsal. Full migration following exact go-live procedures. Validate completely.
Deliverable: Proven migration process with documented timing, validated reconciliation, resolved issues.
Phase 6: Cutover (Go-Live Weekend)
Objective: Execute flawlessly.
- Freeze source system at designated time
- Extract final data
- Transform and cleanse any last-minute changes
- Load into production
- Reconcile everything
- Validate with business users
- Obtain sign-off
- Open new system
Deliverable: Migrated data — complete, accurate, reconciled, signed off.
Phase 7: Hypercare (Weeks 1-4 Post-Go-Live)
Objective: Catch what you missed.
- Monitor for data anomalies
- Address user-reported data issues
- Reconcile ongoing transactions against expected behavior
- Document lessons learned
Deliverable: Stable data environment — and wisdom for the next migration.
The Numbers That Matter
You want to know if your migration is healthy? Track these metrics:
The Conversation You Need to Have
Before your next project steering committee, before your next status report, have this conversation with your team:
“If we went live tomorrow with the data we have today, what would break?”
Write down the answers. All of them. Do not filter. Do not minimize.
Now you have your risk register. Now you have your cleansing priorities. Now you have the truth.
And the truth, however painful, is the only foundation for a successful migration.
A Final Story
There was a man who inherited a library from his grandfather.
Thousands of books. Decades of collection. The man was moving to a new house with a beautiful, modern library — shelves of glass and light, organized by a perfect system.
He had two choices.
He could move every book, exactly as they were — dusty, disordered, some damaged, some duplicates, some in languages he could not read.
Or he could spend time before the move. Cleaning each book. Deciding which deserved a place in the new library. Repairing what was damaged. Discarding what was beyond repair.
The first choice was faster. The second choice was wiser.
He chose wisely.
When he finally placed his books on those glass shelves, each one belonged there. Each one was known. Each one was loved.
His new library was not just a new location. It was a new beginning.
Your data is your library.
Your new ERP is your new house.
The choice of how you migrate is the choice of what kind of future you’re building.
Choose wisely.
What’s your biggest data migration fear? The one that keeps you up at night? Tell me in the comments. I’ve probably faced it — and survived.
