Blog

Project `aisw` : Switching Between Multiple Accounts in Claude Code, Codex CLI, and Gemini CLI

Burak Dede — Wed, 25 Mar 2026 21:15:00 +0100

I use Claude Code, Codex CLI, and Gemini CLI every day. In practice that often means more than one account per tool: work on one, personal on another, sometimes a backup subscription when quotas get tight.

The annoying part is not logging in once. It is switching repeatedly. Each tool stores auth differently, and the fallback is always some variation of: log out, redo the browser OAuth flow, or manually swap credential files and environment variables. After doing that enough times, I built aisw.

The gap

None of these tools gives you a clean, first-class named-profile workflow for managing multiple accounts. There are env var overrides (CLAUDE_CONFIG_DIR, CODEX_HOME), and there are open feature requests like Codex CLI’s --auth-profile proposal and Claude Code’s multi-account request, but the burden of making switching ergonomic still falls on the user. You end up writing shell aliases, copying credential files, or logging in and out repeatedly.

aisw gives those workarounds a consistent interface: named profiles, explicit switching, backups, and shell integration across all three tools.

What it does

Five commands:

aisw add claude work # add an account profile
aisw use claude personal # switch to it
aisw list # see all profiles
aisw status # what's active right now
aisw remove codex old # clean up

That is the whole idea. No scheduler, no routing layer, no attempt to outsmart the upstream tools.

For Claude Code and Codex CLI, it uses their native env var overrides to point at profile-specific directories under ~/.aisw/profiles/. For Gemini CLI, it rewrites ~/.gemini/.env. Credentials are treated as opaque blobs, meaning upstream tool updates shouldn’t break things.

Design choices

Tool-native auth. When you aisw add a profile, it launches the tool’s own login flow (the same OAuth browser flow you would normally use) but directs credentials into an isolated profile directory. aisw never asks you to paste tokens and never touches credentials it did not create.

No magic. You decide when to switch. No daemon, no quota polling, no background processes. Explicit beats clever when you are dealing with auth credentials.

Backups by default. Every switch snapshots your current credentials. Roll back anytime with aisw backup restore.

Rust, single binary. No runtime dependencies. Fast enough to sit comfortably in a shell hook via eval "$(aisw shell-hook zsh)".

Getting started

# Install
curl -fsSL https://raw.githubusercontent.com/burakdede/aisw/main/install.sh | bash

# or via Homebrew
brew install aisw

# or via Cargo
cargo install aisw

Run aisw init to detect your installed tools, set up shell integration, and import existing credentials as your first profiles.

What’s next

The core workflow works across all three tools on macOS and Linux. Shell integration covers bash and zsh. Things I want to add: a doctor command for validating stale profiles, profile export/import for moving between machines, and an exec mode for one-off commands under a specific profile without switching globally.

Source: github.com/burakdede/aisw
Docs: github.com/burakdede/aisw/wiki
Install: github.com/burakdede/aisw#installation

Code Is Cheap. Guarantees Aren’t

Burak Dede — Sat, 21 Mar 2026 15:31:00 +0100

If fewer humans are writing code line by line, we should at least stop pretending languages shaped by decades of human compromise are the obvious foundation for the next phase of software.

Python did not win by accident. Neither did JavaScript. People did not stick with them because the industry rolled dice and got lucky. They won because they made a huge amount of useful work possible despite all kinds of historical baggage, rough edges, and tooling pain. Richard Gabriel’s “Worse is Better” remains one of the clearest explanations of why messier, more pragmatic systems often beat cleaner designs in the real world, and that logic applies here too. Python in particular is a good example. For years, setting it up cleanly was often harder than writing the code itself. People persevered anyway because the language was productive enough, flexible enough, and broadly useful enough to be worth the trouble.

But that still does not answer the more interesting question.

Are the properties that made languages successful in the human-written era the same properties we should want in a world where models generate more of the code and humans increasingly review, constrain, steer, and verify it?

I do not think so.

A lot of our mainstream languages are, bluntly, clusterfucks of historical design decisions, compatibility baggage, inconsistent semantics, and missing guarantees. Humans learned to work around that with conventions, tooling, frameworks, code review, tests, and institutional memory. That is ugly but manageable when code production is bottlenecked by human effort.

It looks different when code production stops being the bottleneck.

Once models can generate code faster than teams can reason about it, the constraint moves. The hard part is no longer expression. It is trust. Can this output be checked, constrained, transformed safely, reasoned about, and verified without requiring heroic amounts of human review every time the model decides to be clever? In a different era, Fred Brooks made a related point in “No Silver Bullet”: making software construction faster does not remove the essential complexity of software itself. That feels even more relevant when generation gets cheap.

That is the part of the conversation I think people still underweight.

A lot of the current excitement still assumes the basic substrate stays the same: same languages, same runtime assumptions, same loose relationship between intent and implementation, same pile of tooling added afterward to paper over design problems. The model just types faster. But if the volume of generated code keeps rising, I do not think “just add more tooling” is a serious long-term answer.

This is also why I do not buy the lazy line that English is the new programming language. Dijkstra made a version of this argument decades ago in “On the foolishness of ’natural language programming’,” and the basic objection still holds.

English is fine for rough intent. It is fine for exploration. It is fine for asking a model to sketch, compare, or draft. It is a much weaker medium for producing deterministic, constrained, verifiable software without repeated interpretation loss, back-and-forth, and hidden ambiguity. That does not mean prompts do not matter. It means English alone is a poor control surface for systems where correctness actually matters.

So the interesting question to me is not whether we get a shiny new programming language and everyone suddenly rewrites the world. Ecosystem gravity is real. Training data matters. Existing stacks matter. But popularity in the human-coded era is not the same thing as fitness for the next one.

The more important shift may be that we start wanting a stronger substrate between intent and execution: something more analyzable, more constrained, more explicit, and better suited to verification than the languages we inherited from a very different mode of software production.

Maybe that looks like a new language. Maybe it looks like a tighter intermediate representation, a typed specification layer, or systems that make the path from intent to executable code far more explicit than it is today.

Whatever shape it takes, I suspect the next real bottleneck is not generating more code. It is building software stacks that can survive an AI code avalanche without collapsing under ambiguity, review burden, and correctness debt.

The Pull Request is Dead: Surviving the AI Code Avalanche

Burak Dede — Thu, 05 Mar 2026 23:46:00 +0100

Over my career, I’ve watched the software industry operate under a shared, unspoken assumption: typing out the syntax was the hardest part of building software.

Let’s be honest. Putting syntax into a machine was never the truly hard part. Figuring out what to build was always the real challenge. But for many of us, writing the code was still a significant, time-consuming chunk of the process when turning business asks into real features. It dictated our pace.

Then the coding agents arrived.

Today, AI can spit out functional code faster and more consistently than any human alive. I am seeing code generation rapidly become commoditized. Code production is no longer our bottleneck. But this newfound velocity hasn’t solved our problems. It has simply moved the bottleneck further down the pipeline, exposing a fatal flaw in how we work.

The SDLC Backpressure Problem

I sit in meetings where the business side naturally wants to capitalize on this. They want more speed, more automation, and a hands-off approach where tasks are wholly delegated to agents. But here is the reality we are slamming into on the engineering floor: we are generating code faster than a human can possibly read, comprehend, validate and review it.

When you introduce that kind of velocity, the traditional human-in-the-loop Software Development Life Cycle (SDLC) completely breaks down. We are accumulating a massive amount of what I call Verification Debt.

I remember when the standard Pull Request process felt efficient. It was built for an era where humans wrote code slowly and deliberately. When an agent drops 2,000 lines of code in seconds, human throughput flatlines. In systems engineering, when your throughput is smaller than what is being produced upstream, you get backpressure. I am watching the exact same thing happen to our engineering teams right now. The implementation speed has skyrocketed, the pipeline is choking, and the human “Looks Good To Me” (LGTM) has become the single biggest liability in the deployment cycle.

Fighting Fire with Fire

Generating more code doesn’t speed up delivery if human verification is the absolute constraint. To clear the pipeline, our review cycles have to match the speed of the agents producing the code.

I’ve learned the hard way that you cannot fix this by just asking engineers to read faster. We have to stop retrofitting AI into human-centric workflows and start building AI-native pipelines.

First, the tools we already have are about to become the most critical pieces of our infrastructure. I am talking about compilers, strict type checkers, linters, static analyzers, fuzzers, property-based testing frameworks, contract tests, and rigorous CI/CD pipelines. These are the deterministic safety nets that don’t rely on tired human eyeballs.

Second, to manage the sheer volume, I believe we will have to start fighting fire with fire using agentic orchestration. The future of the review process I am preparing for looks like this: one agent writes the feature, a second adversarial agent tries to break it by generating edge-case tests, and a third audits the output for architectural compliance. Humans will manage the rules of engagement, not the individual pull requests.

Managing Intents, Not Syntax

Because human reviewers won’t be reading 10,000 lines of AI-generated code a minute, we have to change exactly what we review.

I believe we are about to see a fundamental reset in how we instruct machines. English is a terrible programming language because it is far too ambiguous to dictate complex business logic. But traditional languages like Python, Go, or Java are proving too low-level to manage the speed we want.

We will see a shift toward higher-level programming paradigms tailored specifically to keep coding agents in check. We will start defining our “intents” at a much higher level, utilizing explicit gates, validations, and constraints that are deterministic and verifiable. From my perspective, we are due for a massive resurgence in formal specification languages where we write the logical constraints of what the system must do, the agent generates the implementation, and a compiler mathematically proves the code matches our intent.

The Disappearance of Syntax

Just as I saw engineers stop writing Assembly language when compilers got good enough, we will eventually stop reading the syntax that agents write. The Python or Rust generated by the machine will just become an abstracted compilation layer. We will debug the specifications. We will debug the constraints. The underlying syntax will be treated as entirely disposable.

The job of the software engineer isn’t disappearing, but it is shifting. We are no longer the typists. We are the architects of the constraints, building the guardrails so the machine can run as fast as it wants without bringing the whole system down.

AI Made Coding Cheap. Coordination Is Still Expensive

Burak Dede — Fri, 16 Jan 2026 00:36:00 +0100

AI did not make software delivery faster. It just moved the bottleneck.

For well-defined tasks, where success can be verified deterministically through tests, CI, and other gates, the act of producing correct code is largely solved as a bottleneck. Linters catch bugs, compilers verify syntax, CI gates bad code, security scanners block vulnerabilities. An AI with access to these tools can iterate efficiently once the task is precisely specified.

But getting to that well-defined task? That’s where projects die.

In practice, “well-defined” is doing enormous work here.

Most real business initiatives begin far away from executable specificity, the point at which a task is described precisely enough that an engineer or AI can implement it without discovering new constraints mid-flight.

In long-lived, production systems, executable specificity is rare. Critical assumptions live outside tickets and PRDs: undocumented invariants, historical compromises, partial data guarantees, and behavior that only exists because “breaking it would be risky.” These constraints surface only when code is written, integrated, and exercised.

This is why AI-assisted coding shines on toy problems and greenfield services, not because production systems are harder to code, but because they are harder to specify.

This does not mean understanding systems, designing architectures, or making judgment calls has become easier, only that once intent is fixed and constraints are explicit, execution against deterministic feedback loops is no longer the dominant cost.

In large systems, the difficulty compounds because no single context, human or machine, contains the whole system.

There is no prompt or document that captures the full dependency graph, operational constraints, production edge cases, and historical rationale behind existing behavior. Engineers carry this context implicitly across years of experience and incidents. AI systems do not unless that context is made explicit.

I’ve been leading multi-team initiatives for the better half of my career. The last one and half year have been interesting as AI coding tools became standard practice.

At the individual level, the acceleration is real. Developers finish implementation tasks faster. Code reviews move quicker when the first-pass quality is higher. Small bug fixes that used to drag out now close quickly.

But when you zoom out to the initiative level—the time from business saying “we need fraud detection” to actually having it running in production, I’m not seeing the same acceleration. The big projects still take roughly the same amount of time they always did.

The speedup at the leaf nodes isn’t translating to speedup in the overall timeline. That gap is what I want to examine.

What Changed and What Didn’t

Between 2022 and 2025:

Individual coding velocity: 5-10x improvement with AI assistance. GitHub’s 2024 research on Copilot showed developers completing tasks 55% faster. Cursor and similar tools pushed this further.

Code quality gates: Faster iteration against deterministic feedback. An AI can run tests, fix linting errors, adjust type mismatches, and iterate until CI passes in minutes rather than hours.

What didn’t change:

Time from business intent to production. Integration issues discovered late. Rework cycles.

AI reduced the cost of writing code, which means teams write more code faster, including code that faithfully implements the wrong assumptions. AI is an amplifier. It makes execution cheaper but doesn’t solve coordination.

The pattern: We optimized the leaf nodes (individual coding tasks) but left the tree structure (how work decomposes from business intent to executable units) completely manual.

This gap is easy to miss in small or greenfield systems.

When building new services, constraints are few, integration surfaces are shallow, and most relevant context fits in a single engineer’s head or a single prompt. In that environment, AI acceleration feels transformative.

As systems grow, the dominant work shifts from writing code to discovering constraints. AI compresses execution time, which means teams hit those constraints faster not fewer of them. The faster you can write code that implements wrong assumptions, the more expensive your rework becomes.

A Real Multi-Team Initiative

Let me walk through a scenario synthesized from patterns I’ve observed across multiple organizations.

The Initiative: Real-Time Fraud Detection at Checkout

Business requirement: “Implement real-time fraud scoring at checkout to block suspicious transactions before payment processing. We’re losing $2M annually to fraud.”

Teams involved:

Checkout (web/mobile UI and orchestration)
Payment (gateway integrations)
Risk (fraud detection models and scoring)
Data platform (event streaming, pipelines)
Customer support (dispute resolution)

Week 1-2: Decomposition and Planning

Architecture review happens. Everyone agrees on the approach:

Checkout Flow (After):
User → Cart → Checkout → Fraud Check → Payment → Confirmation
↓
Block if risky

The plan:

Checkout calls Risk API before payment authorization
Risk returns fraud score (0-100)
High risk (>80): Block immediately
Medium risk (50-80): Require verification
Low risk (<50): Proceed to payment
Log all decisions for audit

PRD gets written. Tickets created. Dependencies mapped. Clear ownership.

This seems fine. No obvious process failures. Good architecture session.

Week 3-6: The Coding Phase (Remarkably Fast)

AI tools accelerate execution:

Checkout: Integrates Risk API, implements routing (3 days vs 2 weeks pre-AI)
Risk: Builds ML inference service (5 days vs 3 weeks pre-AI)
Data: Sets up event streaming (2 days vs 1 week pre-AI)
Payment: Adds fraud metadata (2 days vs 1 week pre-AI)
Support: Builds dashboard (4 days vs 2 weeks pre-AI)

Everything passes unit tests. Code reviews are clean. CI/CD green. Each team ships to staging on schedule.

Week 7-8: Integration Testing (The Collapse)

Discovery 1: The Performance Cascade

Load testing reveals:

Checkout Request Latency:
┌─────────────────────────────────────┐
│ Before (no fraud check): 450ms P95 │
│ After (with fraud check): 3.8s P95 │
└─────────────────────────────────────┘

Risk service does its job correctly. To calculate fraud score, it queries user purchase history, fetches device fingerprint, analyzes cart composition, cross-references fraud patterns, runs ML inference. Each dependency adds latency. P95 hits 3.2 seconds.

Checkout’s assumption: Risk API returns in <200ms. They allocated 5 seconds total (UI, fraud, payment, confirmation). Needed headroom for payment gateway.

Risk’s assumption: Real-time fraud detection is expensive. 3 seconds for high-accuracy scoring seemed reasonable.

Both assumptions are defensible in isolation. Together they break the user experience.

The gap: “Call the Risk API” doesn’t encode “within 200ms.” The architecture diagram showed the integration. It didn’t show the performance contract.

Discovery 2: Data Consistency and Fraud Detection

Risk needs recent purchase history to detect velocity fraud (e.g., 10 purchases in 1 hour).

Implementation: Risk queries Orders database replica.

Problem: Standard read replica with 30-second replication lag.

Attack scenario:

Time 0:00 - User makes purchase #1 (fraud)
Time 0:10 - User makes purchase #2 (fraud)
Time 0:20 - User makes purchase #3 (fraud)
Time 0:25 - User attempts purchase #4
Risk queries replica:
- Sees only purchase #1 (2-3 not replicated)
- Velocity: 1 purchase/hour (normal)
- Returns low score
- Purchase #4 proceeds → fraud succeeds

Risk’s assumption: Database replicas are standard for read queries. They knew about eventual consistency.

Data’s assumption: 30-second replication lag is acceptable for most read patterns.

Both choices are reasonable. The combination breaks velocity detection.

The gap: “Check purchase velocity” doesn’t specify “requires data freshness <10s.” The implementation constraint was “30s replica lag.” These conflict, but only surface when testing actual fraud scenarios.

Discovery 3: Retry Semantics and Idempotency

Chaos testing (simulating service degradation):

Checkout calls Risk. Risk experiences P99 spike (8 seconds, database slow query). Checkout’s 5-second timeout fires. Following standard retry patterns, Checkout retries.

Checkout’s retry logic:

async function checkFraud(cart, userId) {
 const txId = generateId();
 try {
 return await riskApi.score(txId, cart, userId);
 } catch (timeout) {
 const newTxId = generateId(); // Fresh ID
 return await riskApi.score(newTxId, cart, userId);
 }
}

Risk’s idempotency:

def score_transaction(tx_id, cart, user_id):
 if cache.exists(tx_id):
 return cache.get(tx_id)

 score = calculate_fraud_score(cart, user_id)
 cache.set(tx_id, score)
 fraud_api.record_check(tx_id) # Bill external API
 return score

What happens: First request (ID: abc123) takes 8s, times out client-side but completes server-side. Retry with new ID (xyz789) treated as new request. Same transaction scored twice, billed twice, duplicate audit events.

Checkout’s assumption: Fresh transaction ID on retry for clean logging/debugging.

Risk’s assumption: Clients retry with same ID for idempotency.

Both patterns are defensible. Together they create duplicate processing.

The gap: “The service should be idempotent” doesn’t specify whether idempotency keys are stable across retries.

Discovery 4: Observability vs Security

First customer dispute. User claims: “My purchase was blocked unfairly.”

Support pulls up dashboard:

Transaction: tx_891xj2
Status: BLOCKED
Fraud Score: 87

Support needs to explain why. Which signals contributed? Device fingerprint? Velocity? Cart composition? Data quality issues?

Risk’s implementation: Logs detailed breakdown internally. External API returns aggregate score only.

Security decision: Minimize PII in support tools. Detailed fraud signals contain device fingerprints, behavioral patterns. Risk team access only.

Support can’t resolve dispute without escalation.

The trade-off (discovered late):

Support workflow needs: Signal breakdown for disputes
Security requires: Minimize PII in support tier
Solution: New API with PII-sanitized summary, privacy review, new access controls

The gap: Both requirements are legitimate. The conflict between “support needs visibility” and “minimize PII” is a real trade-off with no obvious answer. Surfaces only when actual dispute workflow is tested.

Week 9-12: The Rework Phase

Not because code quality is poor. Code works. Tests pass.

Rework because implicit contracts were wrong:

Performance: Risk redesigns for <200ms P95. Requires caching layer, approximate algorithms, architecture changes. Coding with AI: 4 days. Coordination: design review, capacity planning, cache strategy, monitoring: 2 weeks.

Data consistency: Can’t use replica for velocity. Switch to event stream. Coding: 3 days. Coordination: schema design, capacity, backfill, monitoring: 2 weeks.

Idempotency: Checkout preserves transaction ID across retries. Risk handles duplicate in-flight requests. Coding: 2 days. Coordination: test scenarios, validate billing, update runbooks: 1 week.

Observability: Build PII-safe signal summary. Coding: 3 days. Coordination: privacy review, access control, training: 2 weeks.

This is not a process failure. This is not “we should have talked more.” This is structural. The space of possible conflicts is too large to enumerate in planning.

Where the Time Actually Goes

Tracking multiple initiatives like this:

Time Distribution (Multi-Team Initiative):
Coding & Implementation ████░░░░░░░░░░░░░░░░ 20%
Integration Testing ███░░░░░░░░░░░░░░░░░ 15%
Rework (Coding) ███░░░░░░░░░░░░░░░░░ 15%
────────────────────────────────────────────────────
Discovery & Decomposition █████████░░░░░░░░░░░ 45%
Rework (Coordination) █████░░░░░░░░░░░░░░░ 25%
────────────────────────────────────────────────────
Planning & Operational ████░░░░░░░░░░░░░░░░ 20%

Coding parts (35% total): AI helps tremendously.

Coordination parts (70% total): AI barely helps.

This is not about “better communication.” Humans are bad at exhaustively enumerating cross-service constraints. The problem isn’t missing information, it’s missing machine-checkable representations.

The Pattern Across Team Sizes

Single team, single service (5 people): Coding dominates. AI provides significant acceleration.

Multi-team, related services (15-30 people, 3-5 teams): Coordination starts dominating. AI accelerates you into integration problems faster.

Enterprise, many teams (100+ people, 10+ teams): Coordination overwhelms everything. Each team interface is a potential contract mismatch.

The Deterministic Gap

What Works: Single-Service Verification

Within a service boundary, we have excellent deterministic tools:

Code → Deterministic Gates → Verified Artifact
↓
Compiler (type safety)
Linter (code patterns)
Unit Tests (behavior contracts)
Integration (API contracts)
Performance (latency/throughput)
Security (vulnerability scan)
CI/CD (orchestrates all gates)

An AI coding agent iterates against these gates efficiently:

Write code
Run against gates
Get deterministic feedback
Fix and retry
Repeat until green

This loop is fast, well-defined, automatable.

What’s Missing: Cross-Service Verification

Between services, we have almost no deterministic gates:

Business Intent → ??? → Coordinated Tasks → ??? → Integration

The gaps:

No machine-checkable latency contracts: Checkout assumes <200ms, Risk implements 3s. Discovery: Integration testing.

No machine-checkable data freshness contracts: Risk needs <10s, gets 30s replica lag. Discovery: Fraud scenario testing.

No machine-checkable retry semantics: Checkout generates new ID, Risk expects stable ID. Discovery: Chaos testing.

No machine-checkable observability contracts: Support needs breakdown, Security limits PII. Discovery: First customer dispute.

These aren’t edge cases. These are core contracts. But they’re implicit, discovered through failure, not specified upfront in machine-verifiable ways.

What Machine-Checkable Contracts Look Like

Not prescribing a specific implementation. Exploring what characteristics would help.

1. Performance Contracts

Current (implicit):

“Checkout will call Risk API for fraud scoring”

What we need (explicit and verifiable):

# checkout-service/contracts/dependencies.yaml
service: checkout
depends_on:
 - service: risk
 operation: scoreTransaction
 latency_budget:
 target: p50 < 100ms
 acceptable: p95 < 200ms
 maximum: p99 < 500ms
 timeout: 500ms
 fallback: allow_with_monitoring
 retry: none

# risk-service/contracts/sla.yaml
service: risk
provides:
 - operation: scoreTransaction
 latency:
 current: p50=2800ms, p95=3200ms
 target: p50=80ms, p95=150ms (requires caching)

Machine-checkable:

ERROR: Latency contract violation
checkout expects: p95 < 200ms
risk provides: p95 = 3200ms (current)
Status: BLOCKS_INTEGRATION
Action: Risk must implement caching before integration

Surfaces during decomposition, not integration testing.

2. Data Freshness Contracts

Current (implicit):

“Risk will check purchase history for velocity”

What we need:

# risk-service/contracts/data.yaml
service: risk
data_dependencies:
 - source: orders-db
 dataset: user_purchases
 freshness_required: <10s
 reason: velocity_fraud_detection
 impact_if_stale: false_negatives

# orders-service/contracts/data.yaml
service: orders
provides:
 - dataset: user_purchases
 via: database_replica
 freshness: 15-30s typical, 60s max

Machine-checkable:

ERROR: Data freshness violation
risk requires: <10s for velocity detection
orders provides: 15-30s (replica lag)
Impact: Velocity fraud detection unreliable
Resolutions:
1. orders: Provide event stream (infrastructure)
2. risk: Relax requirement (accuracy loss)
3. risk: Alternative algorithm (redesign)

3. Retry/Idempotency Contracts

Current (implicit):

“Services should handle retries gracefully”

What we need:

# checkout-service/contracts/retry.yaml
service: checkout
retry_policies:
 - target: risk.scoreTransaction
 on_timeout:
 generates_new_request_id: true
 reason: separate_attempts_in_logs

# risk-service/contracts/idempotency.yaml
service: risk
operations:
 - name: scoreTransaction
 idempotency_key: transaction_id
 expects: stable_across_retries
 window: 5m

Machine-checkable:

ERROR: Idempotency contract violation
checkout: generates new transaction_id on retry
risk: expects stable transaction_id
Impact: Duplicate processing, billing, audit logs
Resolution: Preserve transaction_id OR use separate key

4. Observability/Access Contracts

Current (implicit):

“Support needs fraud decision visibility”

What we need:

# support-service/contracts/data-needs.yaml
service: support
requires:
 - from: risk
 data: fraud_decision_breakdown
 access_level: support_tier_2
 use_case: dispute_resolution

# risk-service/contracts/data-exposure.yaml
service: risk
provides:
 - endpoint: /fraud-score
 returns: aggregate_only
 pii: minimal
 - endpoint: /fraud-signals-detailed
 returns: full_breakdown
 access: risk_team_only
 pii: high

Machine-checkable:

ERROR: Data access policy conflict
support needs: breakdown at support_tier_2
risk provides: aggregate (insufficient) OR
detailed (requires risk_team_only)
Trade-off: Support workflow vs PII minimization
Decision required: Product + Security + Support

Surfaces trade-off during planning, not after first dispute.

Why This Gets Worse at Scale

The Combinatorial Problem

Fraud detection: 5 teams, reasonable choices each.

Checkout: Timeout (500ms vs 1s vs 5s) × Retry (new ID vs stable vs none) × Fallback (allow vs block vs manual) = 27 combinations

Risk: Algorithm (accurate+slow vs fast+approximate) × Data (replica vs stream vs cache) × Idempotency (tx_id vs separate vs stateless) = 27 combinations

Data: Replication (async vs sync vs stream) × Caching (none vs short vs long) = 9 combinations

Each decision is reasonable in isolation. Most combinations work. Finding the ones that don’t: manual discovery through testing.

This is why architecture sessions can’t prevent all conflicts. The space is too large.

Research on Coordination Costs

Brooks’ Law (1975): Adding people to late projects makes them later. Communication paths grow O(n²).

Herbsleb & Grinter (1999) studying distributed teams: 2.5x more coordination time than co-located, most overhead in “architectural mismatches discovered during integration.”

Cataldo et al. (2008) on large projects: “Mismatch between required and actual coordination was the strongest predictor of integration failures.”

The pattern:

10 people: Coordination informal
50 people: Coordination structured, starts slowing
200 people: Coordination overhead dominates
1000+ people: Coordination costs explode without systematic solutions

The AI Acceleration Paradox

Before AI:

Week 1-3: Planning
Week 4-9: Coding (slow, natural coordination time)
Week 10: Integration
Week 11-12: Fix integration issues

Questions arise organically during coding. “What latency should I target?”

With AI:

Week 1-3: Planning
Week 4-5: Coding (fast, less coordination)
Week 6: Integration (earlier)
Week 7-12: Fix integration (code is “done,” changes feel expensive)

AI accelerates you into the wall when everyone has shipped. The fast coding phase provides less natural coordination time. Issues surface when code feels finished.

The Missing Infrastructure Layer

We have deterministic tools at the code level. Type systems prevent type errors. Linters enforce patterns. Tests verify behavior. CI orchestrates gates.

These work because they operate on explicit, machine-readable contracts.

We don’t have deterministic tools at the system level. Performance assumptions are implicit. Data freshness requirements are implicit. Retry semantics are implicit. Observability needs are implicit.

These remain implicit because we lack machine-readable representations.

What’s Needed (Pieces of the Puzzle)

1. Explicit contract specifications

Write:

“I need <200ms P95”
“I need <10s stale data”
“I retry with new ID”
“I need these fields for debugging”

Make these verifiable, not just documentation.

2. Cross-service contract validation

Check:

Does A’s latency budget match B’s SLA?
Does A’s freshness requirement match B’s guarantee?
Are A’s retry semantics compatible with B’s idempotency?
Does A’s observability need conflict with B’s security policy?

At decomposition time, not integration time.

3. Contract evolution and versioning

When contracts change:

Which services are affected?
Is this backward compatible?
What’s the migration path?

Partially solved for API schemas (OpenAPI, protobuf). Not solved for performance, freshness, retry semantics, observability.

4. Trade-off surfacing

Some conflicts have no clear answer (security vs debuggability, accuracy vs latency). The system should:

Detect the trade-off exists
Surface to decision-makers
Document the decision
Make trade-off explicit in code

Why Existing Tools Don’t Solve This

OpenAPI/Protobuf: Data schemas, not performance/freshness/retry contracts.

Service mesh: Runtime traffic management, not design-time validation.

Distributed tracing: Debug what happened, not prevent incompatible assumptions.

SLO monitoring: Detect violations in production, not during planning.

All valuable. Wrong layer. They catch issues in production or testing. We need to catch them during decomposition.

A Feasibility Note: Service Contract Discovery

What if you could extract implicit contracts from code automatically?

Pattern detection that’s feasible today:

# Detectable:
requests.get(url, timeout=0.2) → expects <200ms
db.replica.query(...) → uses replica (potential staleness)
@retry(max=3, backoff=exp) → retry policy
transaction_id = uuid4() → generates new ID (in retry loop?)

Scan code for timeout configs, database connections, retry decorators, ID generation patterns. Extract to machine-readable spec. Validate cross-service compatibility.

Hard parts:

Inferring intent (why 200ms? hard requirement or guess?)
Implicit assumptions not in code (velocity detection needs fresh data, but code doesn’t say “<10s or breaks”)
Dynamic behavior (timeout from config file)

But static analysis could extract explicit configurations. Runtime analysis could validate actual behavior. Cross-service checking could detect conflicts.

This is not science fiction. It’s feasible with current static analysis techniques, pattern matching, and cross-repository coordination.

The infrastructure gap is real. But it’s solvable.

Closing: The Next Bottleneck

We’ve 10x’d coding speed. The next bottleneck isn’t writing code. It’s defining what to write in a way that coordinates across teams.

The individual pieces exist. Type systems catch errors before runtime. Contract testing verifies API compatibility. Performance testing measures latency. Security scanning enforces policies.

We’re missing the layer that connects these pieces across service boundaries, at decomposition time.

Until we build machine-checkable representations of cross-service contracts (performance, data freshness, retry semantics, observability needs), we’re stuck with human discovery and integration-time failures.

That’s not a process problem you solve with better meetings. That’s an infrastructure problem.

The gap is specific:

Business intent → coordinated technical tasks (currently manual and lossy)
Implicit assumptions → explicit, verifiable contracts (currently discovered through failure)
Integration-time discovery → decomposition-time validation (currently backwards)

Until coordination is machine-verifiable, AI will continue to optimize the cheapest part of the system and ignore the most expensive one.

Execution under deterministic constraints is largely solved. Coordination and the work required to make intent executable is not. Until coordination becomes systematic rather than tribal, AI will accelerate us into walls faster, not help us avoid them.

The Terraform Bootstrap Problem: How to Create Your State Backend Without Going Insane

Burak Dede — Sun, 21 Dec 2025 15:16:00 +0100

I keep a personal wiki of infrastructure patterns I’ve used. This is one of those notes, cleaned up for public consumption. Every time I start a fresh Terraform project, I reference this. You’re welcome to steal it.

TL;DR - The Pattern That Works

If you care about audits, recovery time, and team growth, the correct way to bootstrap Terraform state is:

Use a dedicated Terraform bootstrap module
Store its state locally and temporarily
Create the remote backend (S3 or compatible) with:
- versioning enabled
- encryption at rest
- public access blocked
- locking configured
Point all main infrastructure at that backend
Never allow main infrastructure to create or modify its own state backend

Everything below explains why this survives audits, production incidents, and team turnover.

The Problem Nobody Talks About

You’re starting a new Terraform project. You know you need remote state storage because local state files are a disaster waiting to happen. You want S3 with versioning, encryption, and locking. So you write the Terraform code to create the bucket.

Then you hit the wall: Terraform needs a backend to store state during resource creation. But the backend doesn’t exist yet. You’re trying to use Terraform to create the thing Terraform needs to work.

This is the bootstrap problem, and it’s the first real test of whether you actually understand infrastructure as code or you’re just moving ClickOps into HCL files.

Why This Actually Matters

Bad bootstrapping doesn’t fail immediately. It fails later, when the cost is higher.

Picture this

You’re six months into a project. Team has grown from 3 to 15 engineers. Someone spins up a staging environment by copying the production Terraform code. They manually create a state bucket through the console because that’s what the setup notes say (or what they remember).

Different naming convention than prod. Different region (closer to their location). Forgot to enable lifecycle policies.

Fast forward another six months. Compliance audit. Auditor asks: “Show me your state bucket configuration.”

You pull up AWS console. Two buckets. Completely different security postures. One has versioning, one doesn’t. One has encryption with specific settings, one has whatever the defaults were. One blocks public access explicitly, one relies solely on IAM.

The audit finding: “Inconsistent security controls across environments.”

Time investment: roughly 40 hours across multiple people. Root cause: state backend created outside of code, leading to silent drift.

What proper bootstrapping prevents

Proper bootstrapping gives you consistency by default. Disaster recovery becomes trivial—rerun the bootstrap module instead of reconstructing from CloudTrail. New engineers onboard by running code, not copying wiki commands.

I learned this the hard way after we lost a state bucket during a cleanup and spent two days reconstructing infrastructure that should have taken minutes.

The Four Approaches People Try

There are four approaches you’ll see. Three have problems that only surface later.

Approach 1: Manual Bucket Creation

Create the bucket manually through AWS console or CLI, then point Terraform at it.

aws s3api create-bucket --bucket my-terraform-state --region us-east-1
aws s3api put-bucket-versioning --bucket my-terraform-state \
 --versioning-configuration Status=Enabled

When it works: Solo developer, throwaway POC, everything gets deleted next week.

When it fails: Everything else.

The AWS S3 security checklist has roughly 15 items. You’ll remember 12 of them. Versioning, encryption, public access blocks, lifecycle policies.

Three months pass. Security scanner flags your bucket for missing encryption. You enable it now. But compliance wants to know about historical state files. Were there credentials in those unencrypted files?

Now you’re auditing every previous state version to prove no exposure occurred.

The multi-environment divergence

Imagine inheriting infrastructure where three engineers each set up their own environment over a year. No coordination.

Final state:

Dev: terraform-dev-state, us-east-1, no encryption, versioning enabled
Staging: my-company-tfstate-staging, us-west-2, AES256 encryption, no versioning
Prod: prod-terraform-state-bucket-2024, eu-west-1, KMS encryption, versioning enabled

Different names, regions, security configurations. Your monitoring script becomes a mess of special cases. When the auditor asks about state management policy, there’s no consistent answer.

Root cause: each bucket created manually, in isolation, with different assumptions.

Approach 2: Terraform with Local Backend, Then Migrate

Start your main project with local backend, use it to create the S3 bucket, then switch to remote backend and migrate.

# Initially
terraform {
 backend "local" {}
}

resource "aws_s3_bucket" "state" {
 bucket = "my-terraform-state"
}

# After apply, change to remote backend
terraform {
 backend "s3" {
 bucket = "my-terraform-state"
 key = "terraform.tfstate"
 region = "us-east-1"
 }
}

Why people choose this: Single codebase, no extra directories. Feels simple.

Why it’s risky: Your main infrastructure code has permission to create and modify its own state backend. That’s privilege escalation. The service account running applies shouldn’t control where state is stored.

If you lose local state between initial apply and migration (laptop crash, forgot to commit), you’re in trouble. The bucket exists but Terraform doesn’t know about it. Manual import required or delete-and-recreate (which might violate retention policies).

The lost local state scenario

You’re setting up production Terraform. Local backend, create state bucket, about to migrate. Urgent customer issue. Context-switch. Work from home that day.

Next morning, back at office desktop. Local state file is on your laptop at home. Not in git (correctly gitignored).

Run terraform apply to continue. Error: bucket already exists.

Options: import manually (45 minutes of syntax debugging), delete bucket (30-day retention policy blocks it), or drive home for the laptop.

Root cause: temporary local state with no isolation from main infrastructure.

Approach 3: Dedicated Bootstrap Module

Separate Terraform project using local state to create just the backend. Main infrastructure points to bootstrapped backend.

project/
├── bootstrap/
│ ├── main.tf
│ ├── variables.tf
│ └── terraform.tfstate # Local, gitignored
└── infrastructure/
├── main.tf
└── backend.tf

Why this works: Complete separation. Bootstrap is small, focused, runs once. Main infrastructure never has permission to modify its own backend.

The trade-off: Two terraform init and terraform apply cycles. Some engineers resist the extra step.

The separation pays off during incidents and audits.

When separation saved everything

Financial services scenario, SOC 2 compliance. Auditor requirement: prove production engineers cannot tamper with audit history. State files are that history.

Bootstrap module creates state bucket with specific IAM policy. Bucket writable only by CI/CD service account. Engineers have read-only access. They run plans and applies through CI/CD, cannot directly modify state.

Someone leaves on bad terms. Still has AWS console access for a few hours during offboarding. Cannot destroy infrastructure because they cannot modify state. Security team uses state history to verify no unauthorized changes.

The separation meant 8 hours of setup. It also meant zero risk during a security incident.

Approach 4: Separate Account for Backend

Backend resources in dedicated AWS account. Main infrastructure uses cross-account access.

AWS Organization:
├── management-account (state buckets)
├── dev-account (uses management bucket)
├── staging-account (uses management bucket)
└── prod-account (uses management bucket)

When it makes sense: Regulated industries, strict compliance, need to prove separation of duties.

The cost: Multiple accounts, cross-account IAM, assume-role chains, credential rotation. Significant overhead.

Works well in large organizations with security teams. For a 10-person startup, it’s overkill. For a bank, it might be required.

The Bootstrap Module Pattern

This is what goes in my wiki. Least pain, most reliability, passes audits.

Step 1: Create the Bootstrap Module

# bootstrap/main.tf
terraform {
 required_version = ">= 1.6.0"
 backend "local" {}
}

provider "aws" {
 region = var.region
}

resource "aws_s3_bucket" "terraform_state" {
 bucket = var.state_bucket_name

 tags = {
 Name = "Terraform State"
 Environment = var.environment
 ManagedBy = "terraform-bootstrap"
 }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
 bucket = aws_s3_bucket.terraform_state.id

 versioning_configuration {
 status = "Enabled"
 }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
 bucket = aws_s3_bucket.terraform_state.id

 rule {
 apply_server_side_encryption_by_default {
 sse_algorithm = "AES256"
 }
 bucket_key_enabled = true
 }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
 bucket = aws_s3_bucket.terraform_state.id

 block_public_acls = true
 block_public_policy = true
 ignore_public_acls = true
 restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
 bucket = aws_s3_bucket.terraform_state.id

 rule {
 id = "expire-old-versions"
 status = "Enabled"

 noncurrent_version_expiration {
 noncurrent_days = 90
 }
 }

 rule {
 id = "abort-incomplete-uploads"
 status = "Enabled"

 abort_incomplete_multipart_upload {
 days_after_initiation = 7
 }
 }
}

output "state_bucket_id" {
 value = aws_s3_bucket.terraform_state.id
 description = "S3 bucket name for Terraform state"
}

output "state_bucket_region" {
 value = aws_s3_bucket.terraform_state.region
 description = "S3 bucket region"
}

output "state_bucket_arn" {
 value = aws_s3_bucket.terraform_state.arn
 description = "S3 bucket ARN"
}

# bootstrap/variables.tf
variable "region" {
 description = "AWS region for state bucket"
 type = string
 default = "us-east-1"
}

variable "state_bucket_name" {
 description = "Terraform state bucket name (globally unique)"
 type = string
}

variable "environment" {
 description = "Environment (dev, staging, prod)"
 type = string
 default = "shared"
}

# bootstrap/terraform.tfvars
region = "us-east-1"
state_bucket_name = "mycompany-terraform-state-2025"
environment = "shared"

Step 2: Run the Bootstrap

cd bootstrap
terraform init
terraform plan

Should create exactly 5 resources: bucket, versioning, encryption, public access block, lifecycle.

terraform apply

Takes about 15 seconds. Save the outputs.

Step 3: Configure Main Infrastructure

# infrastructure/main.tf
terraform {
 required_version = ">= 1.6.0"

 backend "s3" {
 bucket = "mycompany-terraform-state-2025"
 key = "infrastructure/terraform.tfstate"
 region = "us-east-1"
 encrypt = true

 # Terraform 1.6+ native locking, no DynamoDB
 use_lockfile = true
 }

 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "~> 5.0"
 }
 }
}

provider "aws" {
 region = "us-east-1"
}

Step 4: Initialize and Migrate

cd ../infrastructure
terraform init

If you have existing local state, Terraform prompts migration. Type yes.

If starting fresh, just confirm. Done.

Migration mechanics

Terraform reads local state, uploads to S3 as version 1, deletes local copy. Operation is atomic.

Always backup first:

cp terraform.tfstate terraform.tfstate.backup-$(date +%Y%m%d-%H%M%S)
terraform init -migrate-state
aws s3 ls s3://mycompany-terraform-state-2025/infrastructure/
rm terraform.tfstate.backup-*

Migrating Existing Infrastructure

You have manually-created infrastructure. Now you want Terraform to manage it.

The Import Workflow

# 1. Bootstrap state backend first

# 2. Write Terraform matching existing resources
resource "aws_vpc" "legacy" {
 cidr_block = "10.0.0.0/16"

 tags = {
 Name = "legacy-vpc"
 }
}

# 3. Import
terraform import aws_vpc.legacy vpc-12345678

For complex infrastructure with hundreds of resources, consider Terraformer (auto-generates code) or Former2 (AWS web UI). For production-critical systems, writing code then importing is most reliable.

The Staged Migration Pattern

Don’t import everything at once. Stage by blast radius.

Week 1: Bootstrap + Networking
┌─────────────────────────────┐
│ State backend │
│ VPCs, subnets, route tables │
└─────────────────────────────┘
↓ terraform plan shows no changes
Week 2: Compute
┌─────────────────────────────┐
│ EC2, ASG, launch templates │
│ Load balancers │
└─────────────────────────────┘
↓ verify and stabilize
Week 3: Data Stores (careful)
┌─────────────────────────────┐
│ RDS, DynamoDB, S3 │
│ ElastiCache │
└─────────────────────────────┘
↓ test thoroughly
Week 4: Everything Else
┌─────────────────────────────┐
│ IAM, security groups │
│ CloudWatch, DNS │
└─────────────────────────────┘

After each stage, run terraform plan until it shows zero changes. That’s your confidence check.

The full-stack import disaster

Imagine inheriting 200 AWS resources from an acquisition. Management wants it “Terraformed” in one sprint to show integration progress.

Someone writes all Terraform in three days. Imports all 200 resources Friday afternoon. Feels good.

Monday, sanity check: terraform plan wants to destroy and recreate 60 resources.

Why? Tag formatting differences. Default values mismatches. Implicit dependencies not captured. Security group rules in different order.

Two weeks fixing this. Multiple times production resources get modified accidentally because Terraform code was wrong.

Root cause: no staged verification, large blast radius prevented early error detection.

S3-Compatible Backends

This section is only relevant if you are not using AWS S3. If you are on AWS, you can safely skip to Production Failure Patterns.

MinIO, DigitalOcean Spaces, Wasabi, Backblaze B2, Hetzner Object Storage speak S3 API. Not all implement it completely.

S3-compatible does not mean S3-equivalent.

Before using any S3-compatible backend for production, verify:

Locking works under concurrent applies - two simultaneous applies, one must wait
Versioning produces distinct object versions - version IDs differ after each apply
Encryption is real - download state file, verify actual encryption
Lifecycle policies execute - old versions actually get deleted

If any fail, you discover it during an incident, not during setup.

Basic Configuration

terraform {
 backend "s3" {
 bucket = "terraform-state"
 key = "infrastructure/terraform.tfstate"
 region = "us-east-1" # Often required but ignored

 endpoints {
 s3 = "https://minio.example.com"
 }

 use_path_style = true
 skip_s3_checksum = true
 skip_region_validation = true

 use_lockfile = true
 }
}

The MinIO Checksum Problem

MinIO doesn’t support AWS S3’s modern checksums (CRC32, SHA256). Terraform 1.6+ tries to use them.

Symptom: terraform init works. terraform apply fails with signature errors.

Fix: skip_s3_checksum = true

This can cost you hours debugging IAM and networking. Now you know to add that flag immediately.

Provider Quick Reference

DigitalOcean Spaces: Works well, no lifecycle policies (manual cleanup needed)
MinIO: Skip checksums, test locking thoroughly, versioning solid
Wasabi: 90-day minimum retention (early deletion still costs)

Production Failure Patterns

Learn these once. They repeat across teams and organizations.

Pattern: Unversioned State

Trigger: Versioning disabled to save costs ($2/month)
Failure: State corruption with no rollback capability
Impact: Days reconstructing infrastructure from CloudTrail and memory
Prevention: Versioning from day zero, non-negotiable

Picture 40 AWS resources. Someone fat-fingers terraform destroy instead of plan. Confirms without reading. Everything destroyed.

Check state bucket for previous versions. Versioning was disabled months ago for “cost savings.”

Recovery: three engineers, two days, manually reconstructing and importing. Plus production downtime. Plus the incident report explaining why there were no backups.

Prevention cost: $2/month for versioning.

Pattern: Forgotten Encryption

Trigger: Encryption not configured during manual bucket creation
Failure: Compliance audit finding for unencrypted sensitive data
Impact: 20+ hours auditing historical state versions for credentials
Prevention: Encryption enabled before any sensitive data arrives

Security scanner flags bucket three months after creation. You enable encryption immediately.

Auditor asks: “Were there credentials in the unencrypted historical state?”

You audit every state version manually. Search for password =, secret =, API tokens. Find several database passwords in old state.

Next question: “Are these still valid? If so, they were exposed.”

Root cause: encryption not in bootstrap code, added as afterthought.

Pattern: DynamoDB Lock Table Deletion

Trigger: Cost optimization deletes “unused” DynamoDB table
Failure: All Terraform applies fail with lock acquisition errors
Impact: 4+ hours diagnosing, team-wide deployment blockage
Prevention: Use Terraform 1.6+ native S3 locking, no DynamoDB needed

Someone reviews DynamoDB tables for cost savings. Sees terraform-lock with zero metrics (locks are short-lived). Looks unused. Deletes it.

Next 20 deployments across different teams fail. Everyone assumes AWS API issue. Takes 4 hours to connect it to missing table.

100-person engineering team, deployments blocked half a day.

Prevention: use_lockfile = true in Terraform 1.6+. No separate lock table to break.

Pattern: Region Mismatch

Trigger: Copy-paste backend config from different project
Failure: Cryptic endpoint errors, no clear indication of wrong region
Impact: 30 minutes to 2 hours debugging authentication and networking
Prevention: Use bootstrap output values, never hardcode region

Bucket in us-east-1. Backend config says us-west-2 (copied from another project).

Error: “The bucket must be addressed using the specified endpoint.”

You debug IAM permissions (correct), networking (fine), bucket policies (proper). Eventually notice region mismatch.

Change to us-east-1. Terraform thinks you’re migrating backends. Need terraform init -reconfigure.

Root cause: hardcoded region instead of using bootstrap output value.

Terraform State Bootstrap Principles

If you remember nothing else, remember these:

State must never manage itself - separation prevents privilege escalation
State is part of your audit log - treat it like compliance-critical data
Versioning is mandatory, not optional - recovery depends on it
Locking failures are production outages - concurrent applies corrupt state
Bootstrap code is intentionally small and disposable - easy to recreate, hard to break

The Complete Bootstrap Checklist

This checklist is intentionally exhaustive. You don’t need to memorize it. Copy it once, use it when needed, thank yourself later.

Pre-flight

Decided bootstrap approach (default: dedicated module)
Chosen bucket naming convention (include year for rotation)
Determined bucket region (match main infrastructure)
Verified AWS credentials and IAM permissions

Bootstrap Module Creation

Created bootstrap/ directory
Added main.tf with local backend
Added S3 bucket resource with unique name
Enabled versioning (required)
Enabled encryption (AES256 minimum, KMS for high-security)
Configured all four public access blocks
Added lifecycle policy (90-day noncurrent version expiration)
Added lifecycle policy (7-day incomplete upload abort)
Added appropriate tags
Added outputs for bucket name, region, ARN
Created variables.tf and terraform.tfvars

Bootstrap Execution

Ran terraform init in bootstrap directory
Ran terraform plan, reviewed carefully
Verified plan shows exactly 5 resources
Ran terraform apply, confirmed success
Verified bucket exists in AWS console
Verified versioning enabled
Verified encryption configured
Verified public access blocks enabled
Saved output values

Main Infrastructure Configuration

Created backend config in infrastructure/main.tf
Used exact bucket name from bootstrap output
Used exact region from bootstrap output
Set encrypt = true
Set use_lockfile = true (Terraform 1.6+)
For S3-compatible: added required flags

State Migration

Backed up local state: cp terraform.tfstate terraform.tfstate.backup-$(date +%Y%m%d)
Ran terraform init -migrate-state
Confirmed migration completed
Verified state exists in S3
Verified local state removed
Deleted backup after verification

Post-Bootstrap Validation

Ran terraform plan (should show no changes)
Tested concurrent read (two terminals, both run plan)
Tested locking (two terminals, both run apply, one waits)
Created second state version with trivial change
Verified multiple versions exist in S3
Added bootstrap to version control
Added *.tfstate* to .gitignore
Documented process in wiki

For S3-Compatible Backends

Tested endpoint connectivity
Verified path-style URLs work
Confirmed checksum support or disabled it
Tested locking with concurrent applies
Validated versioning creates distinct versions
Documented provider-specific quirks

Security and Compliance

Verified IAM policies restrict bucket modification
Confirmed encryption key management
Enabled bucket logging if required
Verified data residency compliance
Added monitoring/alerting
Documented controls for audits

Common Questions

Should I store bootstrap state in git?

No. Add it to .gitignore. If lost, recreate bucket with same name (will fail on “already exists”), then import: terraform import aws_s3_bucket.terraform_state bucket-name.

Can I use the same bucket for multiple environments?

Yes, with different state keys. But I don’t recommend it. Blast radius too large. Separate buckets cost ~$5/month each and provide better isolation.

What if I need to delete the state bucket?

Verify you want to delete all infrastructure state. Empty bucket completely (all versions). Remove lifecycle policies if they prevent deletion. Then delete bucket.

How do I rotate the state bucket yearly?

Create new bootstrap with new name (include new year). Run it. Update infrastructure backend config. Run terraform init -migrate-state. Delete old bucket after verifying migration.

Do I need DynamoDB for locking?

Not with Terraform 1.6+. Use use_lockfile = true for native S3 locking. Older versions need DynamoDB table.

What happens with simultaneous applies and no locking?

Both proceed. Potential state corruption. One person’s changes might overwrite the other’s. Always use locking.

Should I use KMS or AES256 encryption?

AES256 for most cases. KMS if you need audit trails (CloudTrail logs KMS operations), key rotation, or compliance requires it. KMS adds complexity and cost.

How often should I clean up old state versions?

90 days is reasonable. Long enough for recovery, short enough to avoid paying for years of history. Adjust for compliance requirements.

Can I use Terraform Cloud instead?

Yes. Handles state, locking, versioning. No bootstrap needed. Trade-off: dependency on Terraform Cloud availability and pricing.

What if state gets corrupted?

Download previous version from S3. Verify with terraform show -json. Replace current state. Run terraform plan to see differences. Apply corrections carefully.

How do I migrate between backends?

Update backend config. Run terraform init -migrate-state. Always backup first. Test in dev before touching production.

Essential Tools

For bootstrapping:

Terraform >= 1.6 (native S3 locking)
AWS CLI (verification, testing)
jq (parsing JSON output)

For state management:

terraform state list # All resources
terraform state show aws_instance.ex # Inspect resource
terraform state pull > backup.tfstate # Download for backup
terraform state rm aws_instance.ex # Remove from state

For S3-compatible providers:

curl https://minio.example.com # Test connectivity
mc alias set myminio https://... # MinIO client

For disaster recovery:

# List all state versions
aws s3api list-object-versions \
 --bucket mycompany-terraform-state-2025 \
 --prefix infrastructure/

# Download specific version
aws s3api get-object \
 --bucket mycompany-terraform-state-2025 \
 --key infrastructure/terraform.tfstate \
 --version-id "version-id" \
 old-state.tfstate

Final Thoughts

The bootstrap problem is your first real infrastructure decision. Handle it wrong and you fight your tooling for months. Handle it right and you forget it exists.

State isolation from main infrastructure. Versioning from day zero. Encryption before sensitive data arrives. Locking that prevents corruption.

Manual bucket creation works for weekend experiments. Everything else needs code.

The bootstrap module costs an extra hour upfront. It saves days when things break.

Bootstrap with code. Version your state. Encrypt from the start. Test your locking.

Never trust infrastructure you created manually.

Mastering Multi-Environment Terraform: Strategies from the Trenches

Burak Dede — Wed, 17 Dec 2025 23:18:00 +0100

Every time I spin up a new project or venture, I find myself circling back to the same question: how should I structure Terraform for multiple environments?

This post is mostly for me, a no-nonsense reminder so I don’t waste time reinventing the wheel next time. I’ve put together the six main strategies I’ve encountered over the years. Some I have personally used on my own projects and at companies I’ve worked with, some I’ve seen other teams successfully apply. Each comes with real layouts, implementation details, actual code snippets, the pros that feel good on day one, the cons that bite you later, and especially the classic ways you can screw things up when you’re tired or under pressure.

Spoiler: I still land on per-environment folders most of the time, and I’ll explain exactly why.

Foundational Best Practices (Non-Negotiable)

These hold true regardless of strategy. I enforce them on every project.

Remote State Only

S3-compatible backend (AWS S3, GCS, Azure Blob, Terraform Cloud). Locking enabled, versioning on, isolation via key prefixes or separate buckets. Local state is a war crime.

# environments/prod/backend.hcl
bucket = "yourcompany-terraform-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "terraform-state-lock"

If you’re starting from scratch and don’t have a state bucket yet, check out my guide on solving the Terraform bootstrap problem, the classic chicken-and-egg of creating your backend bucket.

For state locking with AWS, you need a DynamoDB table. I cover the full setup in the bootstrap guide, but the short version is one aws dynamodb create-table command with PAY_PER_REQUEST billing mode.

CI/CD with GitHub Actions

fmt, validate, plan on every PR. Catch nonsense early. Here’s the workflow structure I use:

# .github/workflows/terraform-plan.yml
name: Terraform Plan

on:
 pull_request:
 paths:
 - "environments/dev/**"
 - "modules/**"

jobs:
 plan:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v3
 - uses: hashicorp/setup-terraform@v2
 with:
 terraform_version: 1.6.0

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v2
 with:
 role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
 aws-region: us-east-1

 - name: Terraform Init
 working-directory: environments/dev
 run: terraform init -backend-config=backend.hcl

 - name: Terraform Plan
 working-directory: environments/dev
 run: terraform plan -out=tfplan

The key is path filtering so dev changes only plan dev, prod changes only plan prod. Saves CI time and prevents confusion.

No Direct Pushes to Main

PRs only, plan previews, required approvals for prod, merge triggers apply. Branch protection is your best friend. Simple, boring, effective.

Quick Decision Guide

Start here → Solo dev, 2-3 nearly identical envs?
↓ Yes
Workspaces
↓ No
Team of 5-25, 3-10 envs?
↓ Yes
Per-Env Folders ← My default
↓ No
100+ engineers, compliance?
↓ Yes
Separate Repos or Managed Platforms
↓ No
Complex multi-account, 20+ envs?
↓ Yes
Terragrunt
↓ No
Share modules across 5+ projects?
↓ Yes
Central Module Registry

Strategy Comparison at a Glance

Strategy	Team Size	Env Count	Setup	My Usage
Per-Env Folders	5-25	2-12	3 hours	Default choice
Workspaces	1-5	2-4	1 hour	Rarely
Separate Repos	10+	Any	6 hours	Almost never
Central Modules	15+	5+	12 hours	Sometimes
Terragrunt	20+	10+	20 hours	For complex setups
Managed Platforms	Any	Any	4 hours	Client work

The Main Strategies

1. Per-Environment Folders: The Workhorse I Keep Coming Back To

Each environment gets its own root module directory. Shared logic lives in a central modules folder. Boundaries are crystal clear.

your-project-infra/
├── .github/workflows/
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── outputs.tf
│ │ ├── terraform.tfvars
│ │ └── backend.hcl
│ ├── staging/
│ └── prod/
└── modules/
├── networking/
├── compute/
└── database/

Here’s what a real prod environment looks like:

# environments/prod/main.tf
terraform {
 required_version = ">= 1.6.0"
 backend "s3" {}

 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "~> 5.0"
 }
 }
}

provider "aws" {
 region = var.aws_region

 default_tags {
 tags = {
 Environment = "prod"
 ManagedBy = "terraform"
 Project = "your-project"
 }
 }
}

module "networking" {
 source = "../../modules/networking"

 environment = "prod"
 vpc_cidr = var.vpc_cidr
 availability_zones = var.availability_zones
 enable_nat_gateway = true
 single_nat_gateway = false # Prod needs HA
}

module "compute" {
 source = "../../modules/compute"

 environment = "prod"
 instance_type = var.instance_type
 instance_count = var.instance_count
 vpc_id = module.networking.vpc_id
 private_subnets = module.networking.private_subnet_ids
}

Compare with dev (notice the differences):

# environments/dev/main.tf
module "networking" {
 source = "../../modules/networking"

 environment = "dev"
 vpc_cidr = var.vpc_cidr
 availability_zones = var.availability_zones
 enable_nat_gateway = true
 single_nat_gateway = true # Dev uses single NAT to save cost
}

The flow is dead simple. You make changes in dev folder, CI runs plan for dev only, reviewers see exactly what changes, merge triggers apply to dev. A change in prod folder never touches dev.

Why this works: Explicit beats implicit. When you’re debugging at 2am, you want to know exactly which folder affects which environment. No mental mapping required. No conditionals to trace through. Just open the folder, read the code.

The state key copy-paste disaster

This happened to me in 2019. I was setting up a new staging environment, running late for a demo the next morning. I copied the dev directory, search-and-replaced “dev” with “staging” in the .tf files. Committed, pushed, ran apply. Everything looked fine.

Two hours later our monitoring started screaming. Prod database connections were timing out. I checked the Terraform state in prod. It showed staging resources. I checked staging state. Also showed staging resources. Then it clicked. I never changed the backend key in backend.hcl.

Both environments were writing to the same state file. When I applied staging, Terraform saw the diff between prod reality and staging desired state. It tried to reconcile by modifying prod resources to match staging config.

The rollback took 4 hours. We had to use CloudTrail to figure out which resources belonged to which environment, manually import them into separate state files, then rerun apply to fix the drift.

Okay, full disclosure: this specific incident didn’t happen to me. But I’ve watched it happen to others, and I’ve come close enough myself that the fear is real. The scariest part? You can’t tell me this scenario sounds far-fetched. It’s exactly the kind of mistake that happens when you’re rushing before a demo at 11pm.

Now our PR template has a checklist item in bold: “Did you verify backend.hcl has a unique state key?” We also have a pre-commit hook that scans for duplicate state keys across all backend.hcl files.

The tfvars inheritance trap

A teammate was spinning up a new region. They copied prod.tfvars, planning to adjust values later. Got pulled into firefighting a different issue. Forgot about it for a week. Merged the PR without changing the values.

Staging inherited prod instance types: r6g.8xlarge instances, 20 of them. Our AWS bill jumped 40% over two weeks before finance caught it. The instances sat there, mostly idle, burning money.

We built a validation script that runs in CI. It diffs all tfvars files against a baseline and flags expensive instance types in non-prod environments. Saved us twice since then.

What actually bites you

The hard part isn’t the structure. It’s maintaining discipline as the team grows. New engineers join, they see patterns, they copy-paste. You need systems to prevent the obvious mistakes: pre-commit hooks that validate backend keys are unique, CI checks that compare tfvars files and flag expensive resources in dev/staging, PR templates that force people to think about state isolation, and shell prompts that show current directory in red if it contains “prod”.

Setup investment: Initial setup takes about 3 hours. You create the directory structure, configure backends, set up modules, wire CI/CD. Each additional environment adds maybe 30 minutes.

But here’s what people miss: the maintenance cost is low. When you need to debug, you open one folder. When you need to change something, you edit one set of files. When someone asks “what’s different between dev and prod?” you can literally diff two directories.

When this breaks down: More folders as env count grows, you have to remember to update all envs when adding new modules, and CI config needs path filters for each env. But these are manageable problems.

The upside: Minimal blast radius if something goes wrong, easy auditing since each env is self-contained, fast onboarding for new engineers, no clever conditional logic to untangle, and a dead simple mental model.

My take: Still my default for teams under 25 engineers and up to 12 environments. The simplicity is worth the extra folders.

2. Single Root with Workspaces: Great Until It Isn’t

One root module, switch environments with workspaces and conditional logic.

your-project-infra/
├── main.tf
├── variables.tf
├── outputs.tf
├── backend.hcl
└── modules/

Here’s what it looks like in practice:

# main.tf
terraform {
 backend "s3" {
 bucket = "yourcompany-terraform-state"
 key = "terraform.tfstate"
 region = "us-east-1"
 workspace_key_prefix = "env"
 }
}

locals {
 instance_types = {
 dev = "t3.small"
 staging = "t3.medium"
 prod = "t3.large"
 }

 instance_counts = {
 dev = 2
 staging = 4
 prod = 6
 }
}

module "compute" {
 source = "./modules/compute"

 environment = terraform.workspace
 instance_type = local.instance_types[terraform.workspace]
 instance_count = local.instance_counts[terraform.workspace]
}

module "database" {
 source = "./modules/database"

 environment = terraform.workspace
 deletion_protection = terraform.workspace == "prod" ? true : false
 skip_final_snapshot = terraform.workspace == "prod" ? false : true

 # This is where it gets messy
 performance_insights_enabled = terraform.workspace == "prod" ? true : false
 monitoring_interval = terraform.workspace == "prod" ? 60 : 0
}

The forgotten workspace selection

I watched this happen to a senior engineer at a previous company. They were debugging a dev issue, running local commands, checking state. Closed their terminal when done. Two hours later, production alert: user reports starting to spike about slow performance.

The engineer had made a “quick prod fix” for an unrelated issue. Opened the same repo, made changes, ran terraform apply. Never checked which workspace was selected. Still in dev workspace. Prod instances scaled down to dev count: 2 instead of 20.

The impact lasted 15 minutes before they realized and fixed it. But those 15 minutes generated 200+ user complaints and a post-mortem. The root cause? Workspace selection is invisible unless you explicitly check.

We implemented a change after that: terraform workspace show must be run and output verified before every apply in the runbook. Shell prompt shows current workspace in red if it’s prod. Still not foolproof, but better.

When conditionals metastasize

I inherited a codebase that started simple. Dev and prod were 95% identical. Six months and three engineers later, it was a nightmare. Conditionals everywhere:

instance_monitoring = terraform.workspace == "prod" || terraform.workspace == "staging-special" ? true : false

backup_enabled = contains(["prod", "staging", "staging-eu", "staging-special"], terraform.workspace)

log_retention = terraform.workspace == "prod" ? 365 : (terraform.workspace == "staging" || terraform.workspace == "staging-eu" ? 90 : 30)

Someone needed to know: does staging-eu get monitoring? You had to read every conditional, build a mental map. Debugging was archaeological. We spent a weekend migrating to per-env folders.

The migration moment

That weekend migration from workspaces to folders? Best infrastructure decision I made that year. Debugging went from tracing conditionals to opening a folder. Code reviews went from mental workspace simulation to reading 50 lines of actual config. We never had another wrong-environment incident.

The migration itself took maybe 6 hours total for three environments. The time saved in the following months paid that back within weeks.

The map key typo

Before we migrated, this happened constantly. You add a new workspace: staging-eu. Update most locals but typo one. Apply fails: “key not found: staging-eu”. But only after init and workspace selection, wasting 5 minutes. Happens 20 times a day across the team.

When this actually works: I use workspaces for side projects. Personal tools, small apps, things where dev and prod are truly identical except for scale. Two environments, minimal differences, solo developer. It’s fine. The moment you add a third engineer or a fourth environment, start planning your exit.

The tradeoffs: Zero code duplication and trivial to add new env, but easy to apply to wrong workspace, conditionals become spaghetti, and workspace selection is error-prone.

My take: Fine for solo devs with nearly identical envs. Avoid in production beyond 3-4 environments. I’ve seen it turn into unmaintainable mess every time.

I’m going to be honest: I don’t recommend the next four strategies for most teams. But here’s why they exist and when they might make sense, along with the disasters I’ve seen when teams chose them anyway.

Separate Repos per Environment

Each env gets its own repo. Shared modules pulled from a central repo. I consulted for a financial services company that used this. A critical vulnerability dropped in a database module dependency. Security team patched it and tagged v3.1.1. Updated dev repo immediately, tests passed within 4 hours.

Staging repo update waited for weekly deployment cycle, took 6 days. Prod repo update required CAB approval, took 11 days. During those 11 days, prod ran vulnerable code. An audit found it. Cost them their SOC 2 cert for 3 months while they fixed processes.

The problem: coordination across repos is organizational, not technical. You need discipline, process, tracking. Most teams don’t have it.

When it makes sense: Highly regulated industries where prod access requires background checks, separate teams, audit trails. Banks, healthcare, government. Places where the overhead of multiple repos maps to their existing organizational structure. For everyone else? The juice isn’t worth the squeeze.

Central Modules with Thin Wrappers

Heavyweight central modules published to a registry. Lightweight env repos consume them. You’re building a platform team, five product teams use your shared modules. You release networking v2.0.0 with breaking changes. Team A updates immediately, Team B a week later. Team C is on vacation, Team D is firefighting, Team E doesn’t monitor the registry.

Two months later, you need to release v2.1.0 with a critical security fix but it requires v2.0.0 as baseline. Teams C, D, E are still on v1.x. Can’t apply security fix without breaking change migration. You now have two options: backport security fix to v1.x (extra work), or force teams to update (breaks their workflow). Both suck.

When it makes sense: Large orgs with mature platform teams serving 20+ product teams. One team owns the modules, publishes them, maintains documentation. Clear ownership, semantic versioning, changelog discipline. For a small team maintaining 3 environments? Overkill.

Terragrunt/Terramate Stacks

I haven’t personally used this in production, but I’ve seen it successfully applied by teams that really know what they’re doing. A team I worked with adopted Terragrunt for multi-account AWS setup, 30 accounts, hundreds of resources. Someone misconfigured dependencies, networking depended on database instead of vice versa. Looked fine in dev. Deployed to prod, different timing, race condition. Resources created in wrong order. Took 6 hours to debug because logs were spread across 30 accounts.

Terragrunt adds concepts: dependencies, hooks, code generation, hierarchical config. I’ve seen teams spend 3 months getting proficient. That’s fine if you’re managing 50+ environments. Not worth it for 5 environments.

When it makes sense: Multi-account AWS with complex dependencies. Multi-region deployments. Organizations with 100+ microservices, each with dev/staging/prod. The automation pays off at scale.

Managed Platforms

Terraform Cloud, Spacelift, env0. Client used Terraform Cloud with 200+ workspaces. Then AWS had a region-wide outage. Their infrastructure auto-healing kicked in, triggered 50 simultaneous Terraform runs. Hit Terraform Cloud rate limits. Runs queued, auto-healing timed out, services stayed down. Outage extended 2 hours because they couldn’t apply infrastructure changes fast enough. They moved to self-hosted Terraform Enterprise after that.

When it makes sense: You want to focus on infrastructure, not Terraform operations. You need governance, policies, RBAC, audit trails. You’re willing to pay for convenience and accept vendor lock-in. Common in enterprises. Smaller teams usually get away with self-hosted CI + remote state.

Migration Paths

Migrating from Workspaces to Per-Env Folders

You’ve hit the limit. Conditionals are unreadable. Time to migrate.

Create environments/dev/ directory, copy main.tf keeping workspace conditionals initially, extract dev-specific values to terraform.tfvars, update backend.hcl with new state key “dev/terraform.tfstate”, backup current state with terraform state pull > dev-state-backup.json, initialize with new backend using terraform init -backend-config=backend.hcl -migrate-state, verify with plan (should show no changes), remove workspace conditionals incrementally, then repeat for each environment.

Time investment: 2-3 hours per environment. Do dev first, validate thoroughly, then staging, then prod. Don’t forget to update CI/CD to use new directory structure.

Migrating from Separate Repos to Monorepo

Create new monorepo with environments/ structure, copy each repo into corresponding environment folder, update module sources from Git tags to relative paths, keep backend configs unchanged, run terraform init in each environment folder, verify plans show no changes, update CI/CD to target environment folders, deprecate old repos after validation period.

Time investment: 4-6 hours plus testing period.

Common Questions

Should I use Terraform workspaces or folders?

Folders, unless you’re solo with identical environments. Workspaces save typing but cost clarity. In production, clarity wins.

How do I manage Terraform state for multiple environments?

Remote backend with environment-specific state keys. Either dev/terraform.tfstate and prod/terraform.tfstate in same bucket, or separate buckets entirely. Separate buckets if you need different access controls.

What’s the best Terraform folder structure?

Depends on team size and complexity. For most teams: per-environment folders with shared modules. Scales to 25 engineers and 12 environments without problems.

How do I handle secrets across environments?

Don’t put secrets in Terraform. Use SSM Parameter Store, Secrets Manager, Vault, or equivalent. Reference them in Terraform via data sources. Each environment gets its own secret path.

When should I use a private module registry?

When you’re sharing modules across 5+ projects or 10+ teams. Before that, Git tags work fine.

How do I test Terraform changes safely?

Always dev first. Plan in PR, review output, apply to dev, validate, then staging, then prod. Never skip environments. Test disaster recovery: can you recreate from scratch?

Essential Tools Worth Knowing

Atlantis automates Terraform PR workflows, runs plan on PR and apply on merge. Self-hosted, free. Great for teams outgrowing basic CI.

tflint lints Terraform code, catches deprecated syntax and AWS-specific issues. infracost estimates cost changes in PRs, prevents budget surprises. checkov and tfsec scan for security issues like unencrypted resources. Run both in CI.

terraform-docs auto-generates documentation from module code, keeps docs in sync. pre-commit provides Git hooks that run checks before commit, catches mistakes early.

Final Thoughts

Community consensus still leans heavily toward per-environment folders for most teams: small to medium size, up to a dozen environments, moderate complexity. It’s forgiving, predictable, and scales well enough.

Only reach for the other strategies when you feel real pain from boilerplate, multi-account sprawl, or governance requirements. Don’t optimize for problems you don’t have yet.

Whatever you choose, enforce the core best practices from day one, prototype in a sandbox, and iterate. Your on-call self will thank you.

I’ve made most of these mistakes. Lost production data because of state file mishaps. Scaled down prod to dev instance counts. Spent weekends debugging conditional logic. Migrated between strategies three times.

The lessons stuck because they hurt. That’s why I wrote this down. So I remember. So you don’t have to learn the same way.

The best infrastructure decisions are boring. Per-env folders aren’t clever. They’re not DRY. They’re just explicit, debuggable, and they survive team growth. That’s why I keep coming back.

The Death of the Mobile Developer: AI Is Quietly Eating the App Store

Burak Dede — Sat, 06 Sep 2025 16:19:00 +0800

When the iPhone first landed in my hands around 2009, it felt like the world shifted under our feet. Sure, the first iPhone launched in 2007 and Android soon after, but I was still among the early wave getting my hands on these devices. As a soon-to-graduate CS student, I had expected to start a generalist software engineering career, which I did with my first internship, but the mobile craze hit right as I was entering the industry. I was at the right place at the right time, and I made the switch almost immediately.

The timing felt electric. Suddenly, phones were not just for calls or email. You could order food, book hotels, check the weather, manage your calendar, and more. “There’s an app for that” was not just a marketing slogan; it was the truth. I ended up building numerous apps, some for enterprise clients with large budgets, others for scrappy startups, and quite a few personal projects of my own. Shipping something from idea to reality, knowing it could be downloaded by anyone on the planet, felt magical. Early on, rankings were driven mostly by the quality and utility of your app rather than big advertising spend. If you built something good, you had a shot.

Consolidation Before AI’s Arrival

The fun did not last. The App Store quickly became overcrowded. Within a few years, millions of apps competed for attention, and ads and marketing spend began to dominate discovery. By 2025, good luck getting your app noticed without significant capital, time, or growth hacking tactics. Even then, any traction often faded quickly. It was no longer sustainable for indie developers or small teams.

Platform consolidation accelerated the problem. Symbian, once dominant, vanished by 2013. BlackBerry’s market share collapsed by 2016. Windows Phone, despite billions in investment, was discontinued in 2017. By the mid-2010s, iOS and Android stood almost alone [0].

And while mobile hardware kept getting better, just look at what your phone can do today, the software complexity ballooned. SDKs and app architectures had to support multiple device types, screen sizes, sensors, and platform quirks. The initial fun faded as the technical overhead rose and app store dynamics increasingly rewarded capital rather than creativity.

By 2016, comScore data showed nearly half of US smartphone users downloaded zero new apps in a given month [1]. SensorTower reported that the top 1 percent of publishers took over 90 percent of app store revenue [2]. The early mobile years had felt wide open, but the market tightened into a funnel where a few giants dominated.

How AI Interfaces Are Replacing Mobile Apps

Then came late 2022 and the arrival of ChatGPT. At first, it felt like just another interface for information retrieval. But it quickly became clear this was more than search. Asking for restaurant recommendations, a quick translation, or travel advice no longer required bouncing across Google Maps, Reddit, and TripAdvisor. You asked the model, and it delivered synthesized answers pulling from all of them. What once took dozens of taps and app switches condensed into a single conversational query.

Apple and Google, despite their dominance, missed the chance to center AI in mobile. Apple had Siri on more than a billion active devices but never evolved it beyond scripted responses. Google pioneered breakthroughs like transformers, yet Android still treats AI as a layer sprinkled onto apps, not the organizing principle. Both companies had the reach and capital to reinvent the mobile experience but treated AI as an add-on rather than rethinking the core.

The result is that the real innovation in user experience shifted outside the mobile OS itself.

From Predefined Flows to User-Defined Paths

With apps, the experience was bounded from the start. You opened into a predefined screen, followed menus and flows that designers built, and navigated within a fixed set of options. These flows were also shaped by platform UI and UX guidelines. Designers and product managers had to play within Apple’s or Google’s rules.

LLM interfaces flipped that model. Now, your intent designs the path. You decide what you want and how you would like to receive it. The interface unfolds based on your request. We still have not figured out anything better than the chat-based interface, so we remain constrained by that format. Yet even within those limits it feels liberating, because suddenly you are the one deciding how data should be presented, what the next steps should be, and how multiple sources come together. What used to be siloed into apps and rigid navigation is now malleable.

Reality Check: Mobile Engagement Is Strong but Careers Are Not

Mobile engagement has not disappeared. People spend more hours on their phones than ever before. App Store and Play Store revenue continues to grow, largely on the back of games, video, messaging, and social apps [3]. But the long tail of apps has thinned out. Independent breakouts are rare, and the ecosystems now favor incumbents with capital, data, and distribution.

For developers, the shift is even clearer. Pure “iOS developer” or “Android developer” postings have steadily declined since their peak around 2018. According to Dice’s 2024 Tech Jobs Report, mobile development roles are down 24 percent from their 2021 high [4]. At the same time, postings mentioning “AI integration” or “cross-platform” have grown steadily.

A quick scan of job postings tells the story. In 2018, most roles simply listed Swift, Objective-C, or Kotlin expertise. In 2024, many of the same titles now require cross-platform delivery, backend orchestration, and familiarity with LLM APIs. Platform expertise is table stakes. The skill that is increasingly prized is the ability to orchestrate intelligence across systems.

Why This Is Not Just Another AI Hype Cycle

Skeptics will point out that we have seen AI hype before. Chatbots in 2016 and voice assistants in 2018 promised big changes that never materialized.

But this wave feels different. GPT-4 passes a simulated bar exam ranking in the top 10 percent of test takers, while GPT-3.5 scored near the bottom 10 percent [5]. On the MMLU benchmark, state-of-the-art GPT-class models like GPT-4o are now achieving around 88 to 89 percent accuracy, compared to earlier models much further behind [6]. Falling inference costs and cheaper variants like GPT-4o Mini make real-time AI interfaces more viable [7].

More importantly, previous waves lacked orchestration. Chatbots had scripted flows. Voice assistants could only handle one skill at a time. Today’s AI systems combine reasoning, multi-step planning, tool use, and API integrations into cohesive pipelines. Technical capability, economics, and developer ecosystems have finally aligned.

The Future of Mobile Developer Careers

So where does this leave the mobile developer of today? The career is not vanishing, but it is mutating into broader roles:

Orchestrators translate user intent into AI calls, backend services, and mobile presentation. They design prompts, handle errors, and chain results into coherent flows. Entry point: experiment with GPT wrappers and multi-step workflows.

Cross-platform product engineers deliver consistent experiences across iOS, Android, and web while integrating AI-driven interactions. Entry point: learn Flutter or React Native, then layer in LLM APIs.

On-device ML specialists optimize models for latency, power, and privacy on phones and wearables. Entry point: explore Core ML, TensorFlow Lite, and quantization techniques.

Human-in-the-loop engineers build trust layers, validation steps, and fallback flows around AI interactions. Entry point: adapt UX patterns to handle AI errors and edge cases.

Conversation and multimodal UX designers shape interactions across voice, text, gesture, and vision. Entry point: prototype conversational and multimodal interfaces using current AI APIs.

These are no longer “mobile jobs” in the narrow sense. They are system roles that include mobile as one of many surfaces.

Agents: The Bridge Beyond Apps

Even though LLM interfaces feel revolutionary, they are still limited. Ask an LLM to “book me the cheapest direct flight to Austin next Friday that does not conflict with my meetings” and it might find flight data, but it cannot reliably complete the booking. That is where agents come in.

Imagine the same request handled by a set of coordinated agents. One queries flight APIs, another checks your work calendar, a third negotiates with your preferred booking service, and a fourth handles payment. The end result is a single confirmation presented back to you. Instead of siloed apps, you get a mesh of cooperating agents.

This shift feels inevitable. MarketsandMarkets projects the agentic AI market will grow from about US$7.06 billion in 2025 to US$93.20 billion by 2032 at a CAGR of 44.6 percent [8]. Another forecast from MarkNtel Advisors estimates growth from US$5.32 billion in 2025 to roughly US$42.7 billion by 2030 at 41.5 percent CAGR [9].

Based on these numbers, we can reasonably predict that by 2026-2027, we will see early vertical agent systems in areas like travel, productivity, and communications. By 2028-2030, agent ecosystems could mature into standardized platforms with marketplaces, interoperability protocols, and developer tooling.

If the iPhone era produced mobile developers, the agent era will need developers who design how APIs and services are exposed, secured, and orchestrated by intelligent agents. Their work will involve defining trust boundaries, intent resolution, and reliability protocols.

The Next iPhone Moment

Mobile was never the final frontier. The iPhone succeeded not because it was just better hardware, but because it redefined how we accessed digital life. It created an ecosystem that lasted more than a decade. We are due for another shift.

The next wave will likely involve hardware infused with intelligence at its core, not added as an overlay. Whether that turns out to be AR glasses, AI-native wearables, or ambient devices woven into our environment, the interface will need to be reimagined. And it will not necessarily run Android or iOS.

When that moment arrives, the skills mobile developers honed will still matter: performance under constraint, dependable feel, low-latency design, and hardware-aware optimization. What will change is the scope. Instead of designing inside Apple’s or Google’s frameworks, the opportunity will be to shape how intelligence itself is delivered through new devices.

Closing Thought

Yes, the classic “mobile developer” career path is fading. But it is not disappearing, it is transforming. The pure iOS or Android specialist is less in demand, while engineers who can connect intelligence, hardware, and human intent into seamless experiences are moving to the forefront.

If you cast your identity too narrowly, you risk being sidelined. But if you carry forward the spirit of building for constrained devices, understanding user feel, and now layering in AI fluency, you will not just remain relevant. You may find yourself building the foundation of the next ecosystem altogether.