The Ingredients of the Cake: Recipes for Sustainable AI Delivery

Flour – CI/CD Pipelines

Flour is boring, but without it there is no cake. Continuous integration and delivery pipelines are the flour of AI delivery, invisible when they work, catastrophic when they don’t.

Zillow Offers learned this the hard way. Their iBuying program leaned on AI to price homes, but without disciplined CI/CD for machine learning, flawed models were deployed unchecked. The result: mass overpricing, 7,000 unsellable homes, and nearly half a billion dollars in losses. The company shut down its home-buying business entirely (CNN coverage). Flour matters. CI/CD for both data and models, with validation gates and rollback mechanisms, is what turns AI experiments into systems you can actually serve.

Tools: Jenkins, Tekton, Spinnaker, GitHub Actions, Argo CD.
Fix: CI/CD for both data and models, with validation gates and rollback mechanisms.

If flour is the base, then eggs are the binding agent, the next ingredient we can’t forget.

Eggs – Observability

Eggs bind the batter; without them, the cake crumbles. Observability does the same for AI delivery: logs, metrics, traces, and drift dashboards bind complex systems together. Without them, the cracks show quickly.

Knight Capital discovered this in 2012 when a dormant code path reactivated with no monitoring, evaporating  $440 million in just 45 minutes (SEC report). In AI, the danger is subtler but just as real: Amazon Alexa’s “creepy laughing bug” in 2018, caused by misfired intents and absent observability, forced weeks of incident management and eroded trust (BBC coverage). Observability is not a dashboard you tack on; it is an ingredient you fold in at the very start of baking.

Tools: Prometheus, OpenTelemetry, Grafana, Arize AI, Fiddler.
Fix: Real-time anomaly detection, observability baked into CI/CD, and drift dashboards for ML.

Where observability binds, the next ingredient regulates: baking powder and salt.

Baking Powder & Salt – Governance & Policy

Small ingredients can have big consequences. Too little baking powder and the cake is flat; too much salt and it’s bitter. Governance works the same way: too little and you get chaos, too much and you create bottlenecks.

Cambridge Analytica in 2018 exposed the cost of weak governance at Facebook: 87 million profiles harvested and misused in elections, leading to a $5 billion FTC fine (FTC ruling). Microsoft’s Tay chatbot collapse in 2016 was another example of governance vacuum, where a bot turned toxic in less than 24 hours. Since then, frameworks have emerged: the Unified Control Framework (arXiv), the ResAI Governance Model (arXiv), and Microsoft’s 2025 Enterprise AI Code of Conduct (coverage). Weak oversight let models expand without control, leaving regulators to pick up the crumbs afterward. Governance is no longer optional; it is the ingredient that decides whether your cake rises or collapses. You cannot improvise leavening once the batter is in the oven.

Tools: Open Policy Agent (OPA), Kyverno, Sigstore, Wiz, Datadog Cloud Security.
Fix: Clear AI governance frameworks like the Unified Control Framework,  ResAI Governance Model, and Microsoft’s 2025 Enterprise AI Code of Conduct.

Governance prevents collapse, but provenance ensures you know what went in.

Shopping Lists & Receipts – Provenance

Receipts prove what you bought, shopping lists tell you what you planned. In AI, provenance is both: what data you used, and how it was sourced. Provenance means SBOMs, attestations, and receipts.

SolarWinds in 2020 proved this point: attackers injected malicious code into Orion updates, and without receipts, thousands of organizations unknowingly installed a backdoor (CISA alert). IBM Watson for Oncology, meanwhile, showed how poor training data provenance led to unsafe cancer treatment advice and $5 billion wasted (STAT coverage). More recently, Salesforce OAuth token thefts in 2025 hit Allianz, Pandora, and even Google CRM (TechRadar coverage). Without receipts, no one could prove what was clean or contaminated. You are feeding guests mystery cake.

Tools: CycloneDX, SPDX, in-toto, SLSA, Anchore.
Fix: Mandatory SBOMs under U.S. Executive Order 14028, OAuth hygiene, provenance attestation frameworks like SLSA, and strict monitoring of third-party integrations.

Once you know your ingredients, the question becomes: who’s in the kitchen?

Chefs – Delegation Logic

Who measures the sugar, who stirs the batter? Delegation is task routing and authority. If roles are unclear, disaster follows.

Uber’s 2018 self-driving crash showed what happens when delegation between AI and humans is unclear. The AI misclassified a pedestrian; the human assumed the AI was in control. The result: a fatal accident and a $100 million settlement (NTSB report). The program was paused for years, costing Uber over $100 million in settlements and lost momentum. Delegation must be explicit: clear handoff rules, redundancy, and role-based access control (RBAC). A kitchen runs on clarity, not guesswork. Make it clear who or what is responsible at every step.

Tools: Model Context Protocol (MCP), Apache Airflow, Prefect, RBAC, Kubernetes Operators.
Fix: Clear hand-off rules, explicit delegation logic, redundancy, and authority boundaries.

Even the best chefs can’t cook without ovens — and in AI, ovens mean infrastructure.

Ovens – Infrastructure

You can’t bake a wedding cake in an Easy-Bake oven. Infrastructure is the oven, and no AI workload should try to run on underprovisioned infrastructure.

Robinhood learned this in 2020 when trading surges melted its infra during market peaks. Users were locked out, lawsuits followed, and millions in trades were lost (CNCB coverage). Hours of downtime during peak market activity triggered lawsuits and millions in losses. Their oven was too small for the feast. Infrastructure determines not just whether you can bake but whether you can serve at scale.

Tools: Kubernetes, Docker, AWS Elastic Kubernetes Service, GCP Anthos, serverless functions.
Fix: Kubernetes scaling, hybrid cloud fallback with a mix of on-prem + cloud to handle peaks, autoscaling, chaos-tested environments, regional disaster recovery, and stress-tested infra pipelines.

Infrastructure bakes the cake, but without electricity, the oven never turns on.

Electricity – Compute Power

No electricity, no baking. Electricity is the raw compute juice of AI. Without enough power, nothing runs. With too much, you risk overprovisioning and spiraling costs.

GPU shortages in 2023 delayed countless AI projects, while underpowered clusters forced engineering teams into painful trade-offs. When OpenAI’s ChatGPT faced outages, the culprit was often GPU scarcity and scaling failures across data centers. Salesforce discovered this in 2019 when a faulty database script consumed massive compute cycles and knocked out access for thousands of customers worldwide. (TechTarget coverage) Hours of lost productivity turned into millions in downstream costs. Compute budgeting, auto-scaling, and efficiency audits are the equivalent of paying the electric bill. AI projects that cannot predict power draw or respond to fluctuating grid conditions face delays, blackouts, and wasted cycles.

Forget it, and the lights go out mid-bake and you are baking in the dark.

Tools: NVIDIA DGX, AMD Instinct, Google TPUs, Slurm schedulers, Ray.
Fix: Balanced capacity planning, elastic dynamic CPU/GPU/TPU provisioning, diversified compute sourcing, carbon-aware scheduling and energy-aware scheduling.

And electricity only works if you can keep systems cool — which brings us to water.

Water – Cooling and Resource Use

Water cools the oven and keeps the cake from burning. Scope 3 water is invisible until it runs out. Cooling towers and data center water use are now front-page issues.

Microsoft’s Iowa data center consumed nearly 12 million gallons in one month in 2023, sparking community protests. AI training workloads amplify this impact, drawing scrutiny in regions already facing droughts. Google disclosed in 2023 that its U.S. data centers consumed billions of gallons of fresh water for cooling, raising concerns in drought-prone regions. (CNBC coverage). In towns where water is scarce, AI becomes a public liability rather than a shared good. Cooling strategy is not an afterthought; it is part of the recipe. AI delivery is water-intensive, even if invisible to most users. If water is overlooked, sustainability collapses.

Tools: Immersion cooling, liquid-to-chip cooling, ASHRAE water efficiency standards, Cloud Carbon Footprint.
Fix: Transparent water usage reporting, WUE, cooling optimization, closed-loop cooling, recycled water systems, and siting data centers in water-resilient geographies with renewable hydro resources.

When electricity fails, operators often turn to the dirtiest fix: natural gas.

Natural Gas – Backup Turbines and On-Site Generation

Bakeries that never plan for outages throw away half-finished cakes. In AI, outages happen, and backup power determines resilience. When the grid falters, natural gas turbines and diesel generators are often the fallback.

xAI’s Memphis data center (Project Colossus) shows how dangerous this can be. Reports found 35 gas turbines, some unpermitted, pumping nitrogen dioxide into nearby communities. NO2 spikes near the site rose 79 percent above baseline, fueling lawsuits and environmental justice protests. Time writes: “Peak nitrogen dioxide concentrations in South Memphis increased ~3 % over baseline periods, but spikes closer to the facility rose ~79 %.” (Time Magazine, “Inside Memphis’ Battle Against Elon Musk’s xAI Data Center”). Backup power is an ingredient few want to admit is in the mix, but it is critical. The trade-off is stark: uninterrupted compute at the cost of air and public health. Backup is not luxury. It is survival.

Tools: On-site renewables, demand-shift scheduling, microgrids, energy storage (Tesla Megapacks, Fluence).
Fix: Shift away from gas turbines toward renewables and transparent emissions audits.

Without a plan, the cake burns dirty. The cake also needs fair-trade ingredients.

Fair-Trade Ingredients – Sustainability & Ethics

It’s not just what you bake with, but how it’s sourced. AI supply chains are under fire for both carbon cost and ethical shortcuts.

Stable Diffusion and the LAION dataset controversy showed what happens when ingredients are taken without consent. Artists filed lawsuits against Stability AI, MidJourney, and DeviantArt in 2023, arguing that copyrighted images were scraped and used without permission. (Verge coverage) Models were paused and retraining loomed. Fair-trade ingredients make sure the cake is not only edible but just.

Tools: Carbon-aware scheduling (Google), Cloud Carbon Footprint, Responsible AI audits, Pymetrics Fairness Toolkit.
Fix: Opt-in datasets, transparent consent frameworks, carbon-aware compute scheduling.

Fair trade adds integrity, but flavor gives character.

Flavourings – Data, Vectors, Agents

Vanilla and spices don’t form the cake but define its taste and personality. In AI, flavorings are your data: embeddings, raw inputs, and lightweight agents that bring uniqueness to the model. But flavorings are dangerous when mismeasured.

Apple’s 2019 credit card rollout with Goldman Sachs exposed bias in feature vectors, leading to discriminatory credit limits and regulatory probes (BBC coverage). On the technical side, raw vectors can leak sensitive information. Researchers have shown that embedding spaces can be reverse-engineered to recover training data, exposing secrets like passwords or health info. (arXiv preprint) Without balance, the cake may look beautiful but taste bitter.

Tools: Pinecone, Weaviate, Milvus, Databricks, Snowflake, LangChain, LlamaIndex.
Fix: Bias audits, anonymization, consent, fairness constraints, and feature provenance tracking.

And once all that is in place, the frosting goes on top.

Frosting – AI Agents

Frosting looks great, but it slides off without structure. In AI, frosting is the agent layer: chatbots, copilots, autonomous decision-makers.

Samsung learned this in 2023 when engineers pasted sensitive code into ChatGPT, exposing IP and prompting a ban (Forbes coverage). Agents amplify risk because they chain together actions, magnifying small mistakes into critical failures.

Tools: LangChain Agents, AutoGen, Guardrails AI, Haystack.
Fix: Private GPT deployments, contextual access controls, and agentic guardrails.

Frosting can delight, but only if the cake underneath can hold it.