AWS data pipeline
Taxonomy import enricher
Running in production on a live daily import pipeline.
What it does
In an ERP-to-Shopify pipeline, the importer writes whatever categories arrive on the batch — deciding those categories is a separate job. When that step isn't part of the pipeline, categorization becomes standing manual work that has to be repeated as new products keep arriving.
This service adds the missing step, in line. A raw batch lands in storage, the enricher classifies and stamps each product, and writes the enriched batch to the location the importer already watches. Categorization happens automatically as part of the daily run, and the importer is unchanged.
- Classifies each product against a reviewed, client-specific rule set.
- Stamps the taxonomy fields the importer expects, computed so its own write gate accepts them.
- Reports anything that needs a human decision as a short, explicit daily list.
- Adding a new company is a new configuration file, not a code change — enforced by a test.
How it's built
- Serverless on AWS: an event-driven function triggered by object creation, deployed with the Serverless Framework; Python.
- A standalone service. Nothing at runtime reaches into the importer or into the classification tool, and neither can break it by changing. The entire contract with the importer is one file landing in one place.
- It reuses the proven classification engine from the taxonomy & category mapper rather than reimplementing it — carried as version-pinned copies with recorded provenance. The tradeoff is drift, which the pins and a set of artifact tests exist to catch.
- Structured JSON logging for every outcome, with alarms on errors, dead-letter activity, and the pass-through path specifically.
- A separate scheduled function posts the daily summary to chat, deliberately off the import critical path so it can never delay a batch.
The parts that mattered
Three design decisions did most of the work, and each one is about staying predictable under conditions the happy path doesn't cover:
- Classification is conservative. A product matching no approved rule passes through unstamped and is surfaced for review rather than assigned a best guess. A category left visibly blank is cheap to resolve; a confident wrong one is expensive to find later.
- The import always wins. If classification can't complete, the batch passes through unchanged rather than holding up a time-sensitive daily run. Categorization is picked up on a later pass, and a partial result is never written.
- The risky step was made reversible. Deploying into an established pipeline meant amending shared configuration that other live services also rely on — and the standard tooling for that replaces the whole document rather than editing part of it. So the cutover does it differently: it reads the current configuration, saves a copy, applies only the intended changes, confirms nothing else moved, and runs in dry-run mode by default. Rollback is a single command against the saved copy.
How it was verified
The test suite runs against the real rule set and the real taxonomy rather than mocks, so drift in either fails a test. The load-bearing one feeds this service's output directly into the importer's own acceptance check: if that test passes, the category writes. That turned the integration question — "will the other system accept this?" — from something to discover during a release into something the suite answers on every run.