Encrypted Agent Logs: Shipping to Splunk and ELK Without Leaking Secrets

AI agents are verbose by nature. Every tool call, every decision trace, every API response gets logged somewhere — and for good reason: debugging a bad agent run without logs is like debugging a kernel panic without a core dump. The problem is that agent logs are full of sensitive data. Prompt contents, file paths, authentication tokens, PII scraped from third-party APIs — it all flows through your telemetry pipeline and lands in Splunk or Elasticsearch, where it sits in plaintext for years.

This post walks through a practical approach: log everything the agent does, but encrypt the sensitive payload fields before they leave the agent process. Your SIEM gets structured metadata for alerting and compliance, and the confidential fields stay unreadable without a key that never touches the log infrastructure.

The Threat Model

Traditional log pipelines trust the entire path: your Logstash cluster, the Elasticsearch data nodes, the Splunk indexers, the S3 cold-storage tier, every operator who can run a search query. You're one misconfigured IAM policy or disgruntled contractor away from a breach that leaks months of agent conversations.

The goal isn't to hide the existence of log events — you need those for anomaly detection and compliance audits. The goal is to separate structural metadata (timestamps, agent IDs, tool names, error codes, latency) from payload data (prompt text, file contents, API responses, user inputs). Metadata goes to Splunk or ELK in plaintext. Payloads get encrypted at the source and stored as opaque blobs that only authorized systems can decrypt.

Choosing a Payload Encryption Strategy

The right approach depends on who needs to decrypt later:

Symmetric envelope encryption (AES-256-GCM + key wrapping) works when a small number of trusted services need to read payloads — for example, a compliance tool that decrypts audit trails on demand. Each log entry gets a fresh data-encryption key (DEK) encrypted with a long-lived key-encryption key (KEK) stored in your KMS (AWS KMS, HashiCorp Vault, or similar). The DEK and the ciphertext travel together in the log event; decryption requires a live call to the KMS.

Asymmetric encryption (ECIES or RSA-OAEP) works when you want to write logs without any decryption capability in the log writer. The agent process holds only the public key. Only the party holding the private key — your compliance team, your legal hold system — can read the payload. This is the strongest separation: even if your entire logging infrastructure is compromised, the payloads are worthless.

For most agent deployments, envelope encryption with Vault or KMS is the right call. Asymmetric-only is better for high-assurance audit trails where you genuinely never want the log writer to be able to read its own output.

Schema: What Stays Plaintext, What Gets Encrypted

Define this explicitly and enforce it in code, not policy. A reasonable split:

{
  "ts": "2026-07-17T09:42:11.032Z",
  "agent_id": "billing-agent-prod-7f3a",
  "run_id": "run_01jx7m...",
  "tool": "read_file",
  "status": "success",
  "latency_ms": 142,
  "token_count": 831,
  "enc_payload": {
    "alg": "AES-256-GCM",
    "kid": "vault/agent-logs/2026-07",
    "iv": "base64...",
    "ct": "base64...",
    "tag": "base64..."
  }
}

The top-level fields are searchable in Splunk or Kibana: you can alert on status:error, plot latency_ms over time, count tokens per agent_id. The enc_payload blob contains the actual prompt, tool inputs, tool outputs, and any file contents — none of which a log query should need to reveal.

Implementation: Encrypting Before Shipping

Here's a minimal Python pattern using the cryptography library against HashiCorp Vault's Transit engine:

import os, json, base64
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import hvac

vault = hvac.Client(url=os.environ["VAULT_ADDR"], token=os.environ["VAULT_TOKEN"])

def encrypt_payload(payload: dict, key_name: str) -> dict:
    # Ask Vault to generate a DEK; Vault returns it encrypted with the KEK
    resp = vault.secrets.transit.generate_data_key(
        name=key_name, key_type="plaintext"
    )
    plaintext_dek = base64.b64decode(resp["data"]["plaintext"])
    encrypted_dek = resp["data"]["ciphertext"]   # vault:v1:...

    iv = os.urandom(12)
    aesgcm = AESGCM(plaintext_dek)
    ct_with_tag = aesgcm.encrypt(iv, json.dumps(payload).encode(), None)
    ct, tag = ct_with_tag[:-16], ct_with_tag[-16:]

    # plaintext_dek goes nowhere; only the Vault-encrypted version travels
    return {
        "alg": "AES-256-GCM",
        "kid": key_name,
        "dek": encrypted_dek,
        "iv": base64.b64encode(iv).decode(),
        "ct": base64.b64encode(ct).decode(),
        "tag": base64.b64encode(tag).decode(),
    }

The plaintext_dek never leaves process memory. Vault's Transit engine handles key rotation automatically — increment the key version monthly and old ciphertexts remain decryptable because the dek field carries the Vault key version in the vault:v1:... prefix.

To decrypt later, a privileged service calls Vault to unwrap the DEK, then decrypts the ciphertext locally. No other party in the logging pipeline can do this.

Forwarding to Splunk or Elasticsearch

Once you've structured your log events this way, the forwarding layer needs zero changes. Fluent Bit, Logstash, the Splunk Universal Forwarder — they just see JSON. The enc_payload field is an opaque string from their perspective.

A few practical tweaks help:

Set field type mappings explicitly in Elasticsearch. Without them, ES will try to index enc_payload as a nested object and may choke on the base64 strings. Map enc_payload as type: object, enabled: false to store it but skip indexing:

{
  "mappings": {
    "properties": {
      "enc_payload": { "type": "object", "enabled": false },
      "latency_ms": { "type": "long" },
      "tool": { "type": "keyword" }
    }
  }
}

In Splunk, use SHOULD_LINEMERGE = false and set the source type to _json. The enc_payload field will be extracted as a single string value. Create field aliases so dashboards work on the plaintext fields without touching the encrypted blob.

Size matters. Base64-encoded ciphertext for a typical agent turn (a few KB of prompt + output) adds roughly 30–40% overhead. Factor this into your Splunk license calculations and Elasticsearch storage estimates. For very high-volume pipelines, consider truncating payloads above a threshold (for example, cap tool output at 8 KB) and logging an enc_payload_truncated: true flag in the plaintext fields.

Compliance Patterns

If you're operating under GDPR or HIPAA, encrypted payloads solve a specific problem: the right to erasure. When a user requests deletion, you don't need to hunt through petabytes of logs for their data — you rotate (or destroy) the KEK for that user's key namespace, and all associated payloads become permanently unreadable. Your Splunk index shrinks to a compliance-friendly shell of metadata with no recoverable personal data.

For SOC 2 audits, the kid field (key ID) gives auditors a way to verify that specific key versions were active during specific time windows, without reading any payload contents. This is substantially cleaner than trying to redact plaintext logs retroactively.

What to Watch Out For

A few patterns that seem reasonable but cause problems in practice:

Don't encrypt the entire log event. You lose the ability to alert on errors or detect anomalies in real time. Splunk and ELK are useless if every field is opaque.

Don't store the DEK in the log. The example above stores the Vault-encrypted DEK (the one only Vault can unwrap), not the raw DEK. Storing a raw symmetric key alongside the ciphertext it encrypted is equivalent to no encryption at all.

Don't skip key rotation. Vault's Transit engine makes this easy; use it. A key that never rotates means a compromise today affects all historical logs.

Don't rely on network-layer encryption alone. TLS between your agent and Logstash is table stakes, not a substitute for field-level encryption. The data is plaintext the moment TLS terminates at the indexer.

Putting It Together

The pattern is simple in outline: encrypt sensitive fields at the agent process, ship structured events to your existing SIEM, and keep decryption keys in a separate trust boundary. None of your log infrastructure needs modification. Your alerting, dashboards, and compliance queries work on plaintext metadata. Payloads stay confidential until an authorized process explicitly decrypts them with a live KMS call.

For teams running agents at scale, this is quickly becoming a baseline expectation rather than a nice-to-have — particularly as regulators in the EU start asking pointed questions about where agent conversation data lives and who can read it.

The next time you configure a Fluentd pipeline or a Logstash ingest node, ask whether the downstream index should actually be able to read what the agent said.