Optimization of Inhouse Data Flow – Migration to Amazon SP API and Airflow

As part of the update and optimization of the Inhouse Data Flow, our client faced the following challenges:

• Integration of Amazon Vendor API to obtain data for the first-party model (1P).
• Migration of the outdated Amazon MWS to the new version of Amazon SP API.
• Optimization of cloud infrastructure and consolidation of all data processing scripts in one place.

To acquire data from Amazon SP API for Amazon Vendor, the following tasks were completed:

• Retrieval of Amazon Vendor Purchase Orders.
• Retrieval of Amazon Vendor Sales data.
• Retrieval of Amazon Vendor Inventory data.

The following data retrievals were successfully completed:

• Amazon Orders/Items.
• Amazon FBA Fees.
• Amazon Shipments.
• Amazon Restock Inventory.

As part of optimizing the cloud infrastructure on Amazon Web Services, all data processing scripts were migrated to Apache Airflow — a platform for orchestrating and managing data workflows.

We utilize Airflow to execute scripts for data retrieval, transformation, and processing (ETL = Extract, Transform, Load) for our clients, ensuring reliability and effective monitoring of process execution.

As a result of these optimizations and migrations, the client's data processing workflows became significantly more streamlined and efficient. The migration to Amazon SP API and Airflow led to:

• Improved data accuracy and consistency, thanks to the modernized data retrieval mechanisms.
• Enhanced process automation, reducing the need for manual interventions.
• Greater scalability and flexibility in handling increasing data volumes and complexities.
• A unified platform for managing data workflows, leading to better oversight and control over data processes.