AWS data pipeline
Bulk product create & update
A serverless AWS system that syncs product catalogs from external sources into Shopify — at
scale, with verification and full observability.
Runs daily in production. First put to work on a live catalog of over 500,000 products.
What it does
Keeping a large Shopify catalog in sync with an external source of truth is more than a one-time import — products are created, updated, re-priced, and restocked continuously. This system runs that sync as a reliable, observable pipeline.
- Ingests products from multiple sources — CSV, JSON, S3, or API — and creates or updates them in Shopify.
- Manages inventory, shipping, and images, including automatic placeholder-image assignment.
- Orchestrates sync: scans for products that need updating and queues only those.
- Maintains a SKU-to-Shopify metadata index for fast, reliable lookups.
- Verifies data integrity after every write, so problems surface immediately.
How it's built
- Serverless on AWS (Serverless Framework v4): a pipeline of Lambdas — import → create/update → verification — coordinated by SQS queues.
- DynamoDB for product storage and the SKU index; S3 for bulk-import files and placeholder images; Secrets Manager for credentials.
- Built for scale and trust: asynchronous queue-based processing, a dead-letter queue for failed messages, and CloudWatch alarms on queue depth, Lambda errors, and DLQ activity.
- Shopify Admin API; Node.js and Python.