Cefalo × Aqua Robotics — Platform & ML Infrastructure

Engineering the cost out of the pipeline.

Between January and early August 2026, I rebuilt the platform's three most expensive systems — Airflow, the EMR/Spark inference pipeline, and SageMaker training — around scale-to-zero, spot capacity, and Fargate, while pushing Claude Code from an editor plugin into the team's default way of shipping: Jira-to-PRD automation, automated PR review, and standardized testing across nine repositories.

96%↓ Airflow idle-cost reduction after EKS → Fargate

74%↓ Per-task-hour cost after Fargate Spot migration

4 Cost-driven infrastructure rebuilds shipped

10+ Claude Code skills authored & adopted team-wide

Financial impact

Four separate levers, four separate units — a monthly infra bill, a per-run pipeline cost, a per-task-hour rate, and a risk eliminated from a single training job. Each figure below is sourced from a repo README, a written cost-audit plan, or a specific commit — not a blended estimate. Confidence is labeled per card.

airflow-pipeline
NDSW-503 / 475 / 522 / 539

Airflow platform — EKS → ECS Fargate + Aurora Serverless v2

$125/mo → $3–5/mo

idle cost, always-on cluster vs. scale-to-zero

Retired the always-on EKS deployment for a green-field rebuild on ECS Fargate + Aurora Serverless v2. An EventBridge-scheduled wakeup/shutdown chain scales all four services (api-server, scheduler, dag-processor, triggerer) to zero between DAG runs.

Documented — README.md

spark-application
NDSW-438 / 513

GPU image-inference pipeline (EMR + Spark)

$3.71/run → $3.26/run

~$111/mo baseline, five-point optimization plan

Audited the 3-stage pipeline: GPU inference (Stage 2) was already 84% efficient, but Stages 1 and 3 were burning full EMR clusters on sub-3-minute jobs. Plan: move both to Fargate, checkpoint Stage 2 against spot interruption, and shift the run to off-peak spot pricing.

Documented — optimization-plan.md

image_preparation
dataset-quality-analyzer · NDSW-505

Analyzer shard-worker — Fargate → Fargate Spot

$0.148/task-hr → $0.044/task-hr

~74% reduction, on top of a prior 12% right-size

Right-sized the task (16 GB → 12 GB) before moving it to Fargate Spot. Added a SIGTERM handler that flushes partial CSV progress to S3 within the 2-minute spot-interruption window, so the discount doesn't cost re-run risk.

Documented — migration plan

aws-sagemaker
NDSW-487

SageMaker training — runaway-job cost

$153.29/run → ~$15/run

actual AWS bill for one stuck run, vs. the post-fix baseline

A dataloader-iteration bug let distributed training jobs run for days instead of ~24 hours. One run (4× ml.g5.2xlarge spot, Dec 3–6, 2025) actually billed $153.29; an earlier one (2 instances, manually killed after 96 hours) billed $115.17. Fixed the iteration logic and capped max_run/max_wait to 86400s (24hr), bounding the cost of any future stall.

Documented — AWS Cost Explorer

Timeline

A month-by-month record, drawn from ~530 commits across nine repositories and the NDSW Jira project.

January

Jan 01 Confirmed and baselined the SageMaker dataloader/cost fix — 30-epoch run settling at ~24hr / ~$15.
aws-sagemaker · NDSW-487

February

Feb 05 Began the Python 3.10 → 3.12 runtime migration across the Lambda fleet.
infra-module · NDSW-528 / 541

March – April

Mar–Apr Refactored image inference onto PySpark and hardened the GPU EMR cluster — YARN GPU scheduling, batch-size tuning, per-image failure isolation.
spark-application · NDSW-438 / 513

Apr 29 Fixed Stage-2 GPU spot capacity by diversifying instance types — cut spot-unavailability stalls on the inference cluster.
spark-application · NDSW-513

May

May 13 Deployed the rebuilt Airflow stack on ECS Fargate.
airflow-pipeline · NDSW-503

May 14 P1 fix: shut down ECS on DAG failure instead of leaving services running idle.
airflow-pipeline

May 16 Added ECS Service Connect, failure-aware shutdown, and Slack failure notifiers.
airflow-pipeline

May 19 Migrated the metadata DB to Aurora Serverless v2, simplified the wakeup chain, and removed the always-on ALB.
airflow-pipeline

May 20 Tuned the EventBridge wakeup schedule (01:30 → 01:45 Europe/Oslo) and prefixed schedule/IAM names per environment.
airflow-pipeline

June

Jun 15–16 Built an ECS Fargate export pipeline converting trained PyTorch models to TorchScript.
aws-sagemaker

Jun 19–24 Wired EventBridge Scheduler → Lambda → GitHub Actions for on-demand and scheduled training runs; switched CI from static AWS keys to OIDC.
aws-sagemaker

Jun 2026 Shipped automated Claude Code PR review, gated on CI passing and re-triggerable via an @claude comment.
infra-module · spark-application · NDSW-540

July

Jul 2026 Documented a full cost/latency audit of the image-inference pipeline with five ranked optimizations.
airflow-pipeline · NDSW-513

Jul 2026 Removed the Python 3.10 leg from the CI test matrix — runtime migration complete across the fleet.
infra-module · NDSW-541

August

Aug 06 Planned and began the Fargate → Fargate Spot migration; added SIGTERM checkpoint-flush hardening so spot interruption never loses progress.
image_preparation · NDSW-505

Aug 06 Extracted the data-quality analyzer into its own repository; migration of its orchestration from EventBridge/Lambda onto Airflow is in progress.
dataset-quality-analyzer · NDSW-545 — in progress

Architecture decisions

The same four initiatives, expanded — what was wrong, what changed, and what it bought.

Airflow: off EKS, onto scale-to-zero Fargate airflow-pipeline +

Problem: Airflow ran on EKS as an always-on cluster, billed 24/7 for a DAG that runs a few hours a day.
Decision: Rebuilt on ECS Fargate (api-server, scheduler, dag-processor, triggerer as four services on one cluster) with Aurora Serverless v2 for metadata, discovered via Cloud Map instead of an always-on ALB. An EventBridge Scheduler → Lambda chain wakes the services before the scheduled DAG run and a shutdown Lambda, invoked as the DAG's last task, scales back to zero. UI access moved to on-demand SSM port-forwarding.
Impact: ~$125/mo → ~$3–5/mo idle; a full DAG run costs ~$3–5. Per-service breakdown in the scale-to-zero plan: webserver $18→$2.25, scheduler $12→$1.50, triggerer $10→$1.25, ALB $18→~$0.

GPU inference: spot capacity without spot risk spark-application +

Problem: The 3-stage image-inference pipeline ran every stage on full EMR clusters, even the two short (<3 min) stages where 68–84% of wall-clock was cluster bootstrap, not work.
Decision: Keep EMR only where it earns its keep: the GPU inference stage, on a g5.2xlarge fleet with a 100%-spot task fleet (price-capacity-optimized allocation, diversified instance types after an April spot-capacity stall). Move the short frame-extraction and package-export stages to Fargate. Checkpoint the GPU stage at the strip level so a spot interruption resumes instead of restarting, and shift the run to off-peak hours for cheaper spot pricing.
Impact: Documented baseline ~$3.71/run (~$111/mo) → projected ~$3.26/run, ~11 minutes faster, with each spot interruption now costing a resume instead of a re-run (avoids doubling a day's cost on failure).

Data-quality analyzer: right-size, then go spot image_preparation · dataset-quality-analyzer +

Problem: The analyzer shard-worker ran on-demand Fargate, over-provisioned at 16 GB per task.
Decision: Right-sized the task to 12 GB first (a safe ~12% cut with no capacity risk), then migrated to Fargate Spot. Since spot gives a 2-minute interruption warning, added a SIGTERM handler that flushes partial progress to S3 before the task dies, so the cheaper capacity doesn't cost re-processing. Also extracted the analyzer into its own repo and is moving its orchestration off ad-hoc EventBridge/Lambda triggers onto Airflow DAGs (in progress).
Impact: ~$0.148/task-hr → ~$0.044/task-hr, a combined ~74% reduction from the original on-demand, over-provisioned baseline.

SageMaker: capping the cost of a stuck job aws-sagemaker +

Problem: A dataloader-iteration bug meant a stalled distributed training run had nowhere to stop except SageMaker's configured ceiling — set to 345,600 seconds (96 hours). Pulling the actual AWS bill: experiment-50-attempt-7 (4× ml.g5.2xlarge spot) ran 55 hours before finishing on its own and billed $153.29; a prior attempt (attempt-5, 2 instances) had to be killed manually after running 96 hours straight into the ceiling, billing $115.17.
Decision: Fixed the dataloader iteration logic (commit 32d495d) and cut max_run/max_wait to 86,400 seconds (24 hours) — the actual time a healthy run needs. Backed training with spot GPU instances and DynamoDB-based checkpoint resumption so spot interruption doesn't cost progress either. Later added an EventBridge Scheduler → Lambda → GitHub Actions chain for scheduled/on-demand runs, and an ECS Fargate step to export completed models from PyTorch to TorchScript.
Impact: Confirmed post-fix baseline: 30-epoch run ≈ 24hr ≈ $15 (Jira, Jan 1 2026). Against the $153.29 stuck run that motivated the fix, that's roughly a 10x reduction per incident — and the 96→24hr cap now bounds the cost of any future stall outright.

Engineering practice

Cost work is one axis; the other is how the team ships. Made Claude Code the default path from a Jira ticket to a merged, tested, reviewed PR across nine repositories.

Skill	Repo	What it does
jira-plan	infra-module	Reads an NDSW ticket, writes a scoped `<TICKET>-PRD.md` against the actual codebase — the Jira-to-implementation-plan step.
jira-create	infra-module	Drafts and files new NDSW tickets from a described bug or feature.
promote-lambda	infra-module	Promotes a Lambda through qa → stage → main via infra-live merges, and comments on the originating Jira ticket.
audit-lambda	infra-module	Code-quality audit of a Lambda against team conventions.
write-lambda-tests	infra-module	Backfills pytest coverage for a Lambda.
find-layer-usage	infra-module	Traces shared-layer usage across the Lambda fleet before a refactor.
generate-readme	infra-module	Generates/refreshes a Lambda's README from its actual code.
update-lambda	infra-module	Implements a scoped code/config/layer change to a Lambda.
execute-api	infra-module	Executes an API implementation plan end to end.
emr-debug	spark-application	Diagnoses an EMR step/cluster failure from its ID by pulling and root-causing AWS logs.
session-handoff	repo-wide (9 repos)	Standardized end-of-session handoff summary — adopted as a convention across every active repo.

Guardrail hooks

block-dangerous-commands.sh blocks destructive Bash before it runs; pre-bash.sh blocks a prod deploy push unless explicitly overridden; lint-python.sh auto-lints on every edit. Deployed across infra-module and spark-application.

Automated review & testing in CI

Shipped claude-pr-review.yml (NDSW-540) — every PR gets an automated Claude Code review once CI passes, re-triggerable with an @claude comment, running Sonnet 5 at xhigh effort. Paired with the Python 3.10 → 3.12 runtime migration (NDSW-528, NDSW-541) across the Lambda fleet, backfilling pytest coverage per family as each migrated.

infra-module spark-application NDSW-540 · DONE

FinOps visibility

Operate an AWS FinOps tool that posts daily cost visibility straight to Slack, and opened NDSW-543 to formally audit ECS task costs — the natural next line item after the Fargate and EMR work above.

—

Notes

Confidence key

Documented — figure is stated in a repo README, a written cost-audit / migration plan, or pulled directly from AWS billing data.

Sources

Git history across 9 repositories (airflow-pipeline, spark-application, infra-module, infra-live, aws-sagemaker, image_preparation, dataset-quality-analyzer, lambda-layers, model-validation) — ~530 commits authored Jan 1–Aug 7, 2026.
Jira project NDSW ("Net Documentation System Workflow") — issues referenced above cross-checked against ticket status and comments.
Repo documentation: airflow-pipeline/README.md, airflow-pipeline/plans/airflow-ecs-scale-to-zero.md, airflow-pipeline/plans/optimization-plan.md, image_preparation/FARGATE_SPOT_MIGRATION_PLAN.md.
AWS Cost Explorer (SageMaker training-job billing, Nov 2025–Jan 2026) and sagemaker:DescribeTrainingJob — cross-referenced against instance count and billable seconds per run to attribute the $153.29 and $115.17 figures to specific training jobs.

Scope

Limited to Cefalo client work authored under moniruzzaman@cefalo.com. A personal side project under a separate identity was excluded as out of scope for a client-impact report.