Zero-Downtime Credential Rotation for Long-Running AI Agents

Long-running AI agents have a credential problem that doesn't exist in short-lived HTTP handlers. A web request starts, authenticates, and finishes in milliseconds. An agent might hold a database connection, an API session, or a vault-issued token for hours or days. When that credential expires or needs rotation, you can't just restart the process — the agent is in the middle of something.

This post covers how to build credential rotation that agents survive without dropping work.

Why Agents Make Rotation Hard

Most credential rotation guides assume you control the process lifecycle. Rotate the secret, redeploy, done. That works when your service is stateless and horizontally scaled. Agents break this assumption in several ways:

In-flight work. An agent coordinating a multi-step data migration or a long document analysis has state that lives in memory. Kill the process and you restart from scratch — or lose the work entirely if you didn't design for it.

Held connections. Database connections, websocket sessions, and streaming API calls are typically tied to the credential that opened them. Rotating the underlying secret doesn't automatically refresh these; the connection just starts failing at the next checkpoint.

Concurrent agents. If you have ten agent instances sharing a service account key, rotating that key means all ten need to pick up the new value without a gap where some have the old key and some have the new one — and the target service has already revoked the old one.

The goal is rotation that is atomic from the credential store's perspective and graceful from the agent's perspective.

Step 1: Model Credentials as Leased Resources

The first architectural change is to stop treating credentials as static config and start treating them as leased resources with explicit TTLs. Every credential your agent uses should have:

• A validUntil timestamp
• A refreshBefore threshold (typically 20% of TTL remaining)
• A credentialId that uniquely identifies this version

interface ManagedCredential {
  value: string;
  credentialId: string;
  validUntil: Date;
  refreshBefore: Date;
}

Your agent doesn't call process.env.API_KEY directly. It calls a credential manager that checks whether the current lease is still valid:

class CredentialManager {
  private cache = new Map<string, ManagedCredential>();

  async get(name: string): Promise<string> {
    const cred = this.cache.get(name);
    if (cred && cred.refreshBefore > new Date()) {
      return cred.value;
    }
    return this.refresh(name);
  }

  private async refresh(name: string): Promise<string> {
    const next = await vault.fetchCredential(name);
    this.cache.set(name, next);
    return next.value;
  }
}

This is the foundation everything else builds on. The agent never holds a bare string — it holds a reference it can re-resolve.

Step 2: Detect Staleness Before the Target Rejects It

Proactive refresh is better than reactive recovery. Your agent should check credential health on a background timer and refresh before expiry, not after the first 401 lands.

A background loop for Node:

async function watchCredentials(manager: CredentialManager, names: string[]) {
  setInterval(async () => {
    for (const name of names) {
      try {
        await manager.get(name); // refreshes if near expiry
      } catch (err) {
        // alert but don't crash — the current cached value may still work
        logger.warn({ name, err }, "credential prefetch failed");
      }
    }
  }, 60_000);
}

For agents in Python with async support, asyncio.create_task achieves the same result without blocking the main event loop.

The key design choice here: a prefetch failure should not crash the agent. Log it, alert on it, but let the agent continue on the still-valid cached credential until you're forced to act. Over-eager failure on a prefetch error turns a transient vault blip into an outage.

Step 3: Atomic Key Swaps at the Vault Layer

Proactive refresh handles expiry. Forced rotation — when you revoke a credential due to a suspected leak or a compliance requirement — is different. You need to swap the secret without any window where no valid credential exists.

The canonical pattern is the dual-active window:

1. Generate credential-v2 in your secret store while credential-v1 is still valid.
2. Tell the target service to accept both.
3. Propagate credential-v2 to all agents (they pick it up on next refresh).
4. Wait for confirmation that no agent is using credential-v1 (check access logs or use a short TTL).
5. Revoke credential-v1.

Most cloud secret managers support this natively. AWS Secrets Manager calls it rotation with a staging label: the new version is AWSPENDING, becomes AWSCURRENT once all consumers have acknowledged it, and the old version moves to AWSPREVIOUS before being deleted. Vault uses lease versioning.

The agents don't need to know rotation happened. They just get a new value the next time they call manager.get().

Step 4: Handle In-Flight Connections

Background credentials are the easy case. Held connections — database pools, gRPC streams, authenticated websockets — need explicit handling because they don't re-read credentials on every operation.

For database pools, the pattern is:

async function refreshPool(pool: Pool, manager: CredentialManager) {
  const newPassword = await manager.get("db-password");
  // drain existing connections gracefully
  await pool.end();
  // create a new pool with the rotated credential
  return new Pool({ connectionString: buildDsn(newPassword) });
}

For long-lived HTTP clients, most libraries let you inject auth at the request level rather than at construction time. Prefer that pattern:

// Worse: auth baked into the client instance
const client = new ApiClient({ apiKey: staticKey });

// Better: auth resolved per-request
const client = new ApiClient({
  authProvider: () => manager.get("api-key"),
});

Per-request auth resolution means the client transparently uses the current credential without needing to be rebuilt.

Step 5: Test Rotation in CI Before Production

Rotation failures are the worst kind of failure: they usually happen at 2am during a compliance audit, and they take down services that were working fine. You want rotation tested continuously, not discovered in an incident.

A minimal CI rotation test:

1. Spin up your agent in a test environment with a short-lived credential (TTL of 5 minutes, not the production 90 days).
2. Wait for the background watcher to trigger a refresh.
3. Assert that the agent continued processing without errors.
4. Trigger a forced rotation (simulate a leak by revoking the current credential early).
5. Assert that the agent picked up the new credential and recovered within your SLO window.

# Example: force rotation by deleting the secret version in a test vault
vault kv delete secret/test/api-key@v1

# Observe agent logs for successful recovery
sleep 30
assert_no_errors agent.log
assert_using_version agent.log v2

If your agent can't pass this test, it can't survive a real forced rotation. Build the test first; it will drive you to implement the patterns above.

Operational Checklist

Before shipping agents to production, verify:

• All credentials accessed through a CredentialManager, not process.env directly
• Background prefetch running with failure-tolerant error handling
• TTL and refreshBefore thresholds set (suggest: refresh at 20% remaining)
• Database pools and held connections have a refresh path
• HTTP clients use per-request auth injection, not constructor-time auth
• CI rotation test passing against a short-lived credential
• Alerting on prefetch failures and rotation events (for audit trail)

The Bigger Picture

Credential rotation feels like an ops detail until it isn't. The difference between a 30-second rotation window and a 30-minute outage is usually whether the agent was designed to participate in the rotation or just have it happen to it.

Treating credentials as leased resources — with explicit TTLs, proactive refresh, and per-request resolution — is the same principle that makes zero-knowledge encryption work at scale. You don't hold secrets longer than you need them, you refresh them before they expire, and you design for the revocation case from the start.

Agents that do this survive rotations invisibly. Agents that don't get paged about at 2am.