Enable Dark Mode!
how-to-perform-mass-data-migration-using-csv.jpg
By: Sayed Mahir Abdulla KK

How to Perform Mass Data Migration Using CSV

Technical Odoo 19 Odoo Enterprises Odoo Community

When migrating to Odoo 19 — whether from a legacy ERP, a spreadsheet-based system, or another Odoo instance — CSV remains the most practical and universally supported format for bulk data transfer. Unlike one-off API calls or real-time webhooks, a CSV-based migration gives you full visibility and control: you can inspect, validate, and correct the data before it ever touches your production database.

Done right, a CSV migration script handles thousands of records in minutes, applies upsert logic to avoid duplicates, logs every failure with enough context to fix it, and leaves your Odoo database in a clean, consistent state. This guide walks you through building exactly that—from a simple partner import to a full FTP-based automated pipeline.

Advantages:

CSV files are the lowest-common-denominator export format for virtually every business system — accounting software, HR platforms, e-commerce stores, and legacy ERPs all support it. For mass migration, this matters because:

  • No API credentials or custom connectors needed from the source system, just a plain file export.
  • Easy to inspect and clean in Excel or any text editor before running the import.
  • Batch-friendly: you can split large datasets into chunks and process them incrementally.
  • Replayable: If something goes wrong, fix the file and re-run. Upsert logic ensures no duplicates.

Basic CSV Import Script (Manual Trigger)

This method reads a CSV file and imports partner records into Odoo using the ORM's `create` and `write` methods. It includes upsert logic (update if exists, create if not), row-level error handling, and full logging.

File: your_module/models/import_helper.py

import csv
import logging
import os
from odoo import models, api
from odoo.exceptions import UserError
_logger = logging.getLogger(__name__)

class ImportHelper(models.TransientModel):
   _name = 'import.helper'
   _description = 'CSV Import Helper'
   @api.model
   def import_partners_from_csv(self, filepath='/home/odoo/imports/partners.csv'):
       """
       Reads a CSV file and creates or updates res.partner records.
       Triggered manually or via a button/scheduled action.
       CSV expected columns: name, email, phone, street, city, country_code
       """
       if not os.path.exists(filepath):
           raise UserError(f"File not found: {filepath}")
       created_count = 0
       updated_count = 0
       error_count   = 0
       with open(filepath, mode='r', encoding='utf-8') as csvfile:
           reader = csv.DictReader(csvfile)
           for row_number, row in enumerate(reader, start=2):  # Row 1 is the header
               try:
                   # Validate required fields
                   if not row.get('name') or not row.get('email'):
                       _logger.warning(
                           "Row %s skipped: 'name' and 'email' are required. Data: %s",
                           row_number, row
                       )
                       error_count += 1
                       continue
                   # Resolve country from code (e.g., 'US', 'IN', 'GB')
                   country = self.env['res.country'].search(
                       [('code', '=', row.get('country_code', '').upper())], limit=1
                   )
                   partner_vals = {
                       'name':       row['name'].strip(),
                       'email':      row['email'].strip(),
                       'phone':      row.get('phone', '').strip(),
                       'street':     row.get('street', '').strip(),
                       'city':       row.get('city', '').strip(),
                       'country_id': country.id if country else False,
                   }
                   # Upsert logic: update if email already exists, otherwise create
                   existing = self.env['res.partner'].search(
                       [('email', '=', partner_vals['email'])], limit=1
                   )
                   if existing:
                       existing.write(partner_vals)
                       updated_count += 1
                       _logger.info("Updated partner: %s", partner_vals['email'])
                   else:
                       self.env['res.partner'].create(partner_vals)
                       created_count += 1
                       _logger.info("Created partner: %s", partner_vals['email'])
               except Exception as e:
                   _logger.error("Error on row %s: %s | Data: %s", row_number, str(e), row)
                   error_count += 1
                   continue  # Don't let one bad row stop the whole import
       _logger.info(
           "CSV Import complete — Created: %d | Updated: %d | Errors: %d",
           created_count, updated_count, error_count
       )
       return {
           'created': created_count,
           'updated': updated_count,
           'errors':  error_count,
       }

CSV File Format Sample:

Your import file should follow this column structure:

name,email,phone,street,city,country_code

  • Alice Johnson,alice@example.com,+1-555-0101,123 Main St,New York,US
  • Bob Smith,bob@example.com,+44-20-7946-0958,10 Baker St,London,GB
  • Riya Nair,riya@example.com,+91-9876543210,MG Road,Kochi,IN

Best Practices for CSV Migrations:

Validate before importing: Run a dry-pass that logs issues without writing to the database. Fix the CSV, then run the real import. This avoids partial migrations that are painful to unpick.

Use external IDs: For any record that needs to be referenced elsewhere (e.g., a customer appearing on multiple orders), add an `external_id` column and map it to a field like `ref` or a custom `x_external_id`. This makes re-runs and cross-model imports reliable.

Commit in batches: For very large files (100k+ rows), use `self.env.cr.commit()` every N record to avoid a single massive transaction that locks the database and risks a full rollback on error.

Archive, don't delete:  After processing an FTP file, move it to an archive folder rather than deleting it. This gives you an audit trail and lets you replay the import if something went wrong.

Log everything: Each row should produce a log entry on success, warning, or failure. A final summary line (`Created: X | Updated: Y | Errors: Z`) gives you an instant health check after each run.

Store credentials in System Parameters: Never hardcode FTP passwords or API keys in the source file. Use `self.env['ir.config_parameter'].sudo().get_param(...)` so credentials can be rotated without a code deployment.

CSV-based mass migration is the most reliable path to populating a fresh Odoo 19 instance — or keeping it in sync with external systems on an ongoing basis. The patterns shown here — upsert logic, row-level error isolation, FTP automation, and scheduled cron triggers — cover the majority of real-world migration scenarios. Start with the simple partner import to validate your approach, then layer in the FTP pipeline for any recurring data feeds. With proper logging and error handling in place, your migrations will run predictably and leave a clean audit trail for every record that moves through the system.

To read more about How to Handle CSV File Operations in Odoo 18, refer to our blog How to Handle CSV File Operations in Odoo 18.


Frequently Asked Questions

What's the best field to use as the upsert key?

It depends on your data. `email` works well for contacts, `default_code` (internal reference) for products, and `vat` for companies. If your source system has its own unique ID, map it to `ref` or a custom `x_external_id` field and use that — it's the most future-proof approach.

How do I handle related fields like Many2one during a CSV import?

Resolve them before writing. Search for the related record by a natural key (name, code, email) and pass its `.id` in the `vals` dict. If the related record might not exist yet, use `get_or_create` logic, search first, create if not found, as shown in the `_get_or_create_partner` pattern above.

Can I run a CSV migration without writing a custom module?

For one-off imports, yes. Use `odoo-bin shell` to run a Python script directly. But for anything recurring or production-grade, a proper module is strongly recommended: it integrates with Odoo's permission system, can be version-controlled, and lets you attach a UI button or cron job to trigger the import safely.

What happens if the import fails halfway through?

Because each row is wrapped in its own `try/except`, a failure on row 500 won't roll back rows 1–499. Those records are already committed. Rows that fail are logged with full context so you can correct them and re-run — upsert logic ensures successfully imported rows are simply updated rather than duplicated.

If you need any assistance in odoo, we are online, please chat with us.



0
Comments



Leave a comment



Recent Posts

whatsapp_icon
location

Calicut

Cybrosys Technologies Pvt. Ltd.
Neospace, KINFRA Techno Park
Kakkanchery, Calicut
Kerala, India - 673635

location

Kochi

Cybrosys Technologies Pvt. Ltd.
1st Floor, Thapasya Building,
Infopark, Kakkanad,
Kochi, India - 682030.

location

Bangalore

Cybrosys Techno Solutions
The Estate, 8th Floor,
Dickenson Road,
Bangalore, India - 560042

Send Us A Message