We Replaced Every Long-Lived Secret in CI With OIDC — Six Months Later

Last spring, one of our customers had a credential incident. An AWS IAM user's access key embedded in CI got leaked somewhere. Tracking down where took eleven days — that key had not been rotated in three years, and the user/system/job ID columns were identical across every log entry it appeared in. Telling them apart was impossible.

After that incident, we made a plan: get every long-lived secret out of CI. It took six months. 47 access keys → 0. Audit trail per deploy: 47 lines → 1. This post is the six steps and the assumptions that broke along the way.

What OIDC workload identity actually is

The CI job carries a short-lived token that proves the job is what it says it is. The cloud verifies the token's signer (the IdP) and claims, then issues a short-lived credential.

That's the whole thing in one sentence. No secret pre-stored, job presents token at start, cloud issues credentials that are only valid during that job.

CI worker             IdP (e.g. GitHub OIDC)             Cloud IAM
─────────             ──────────────────                ──────────
Job starts                                                
  ↓                                                  
  Request (audience: aws)                                
        ─────────────────▶                          
                          Signed JWT issued             
        ◀─────────────────                           
  Present token to cloud                                
        ────────────────────────────────────────────▶
                                                  Verify signer
                                                  Match claims
                                                  (repo, branch,
                                                   workflow, etc.)
                                                  IF match:
                                                    issue temp creds
                                                    (typically 1h)
        ◀────────────────────────────────────────────
  Use creds for cloud calls                              
  Job ends, creds expire                               

No long-lived secret stored anywhere. That's the point.

Six-step migration

Step 1 — Inventory

Which secrets, in which jobs, calling which services. Without this, you can't start. We grepped the secret store and parsed build logs. Found 47 (well, 49 — two were dead keys nobody had used in six months).

Step 2 — Register the IdP

In each cloud (AWS / GCP / Azure), register our CI's OIDC issuer as a trusted IdP. We registered both GitHub Actions OIDC and our own CI's OIDC. The trap is that audience and thumbprint differ per cloud.

Step 3 — Write the trust policies

This is the load-bearing step. Specify which token claims can assume which IAM role.

{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:collabops/web:ref:refs/heads/main"
    }
  }
}

The sub claim is the load-bearing line. Too loose and another repo can assume the role. Too tight and legitimate jobs get rejected.

Step 4 — Migrate one job at a time

Pick one job. Move it to OIDC first. Usually a less critical build job. Stabilizing the first one typically takes 1–2 weeks.

Step 5 — Rotate then revoke the old keys

Once OIDC jobs are stable, rotate the original access keys, then revoke the rotated keys. This is the most-skipped step — OIDC works, so people leave the old keys around. Old keys must be revoked.

Step 6 — Migrate everything, delete the IAM users

Last step. With every job on OIDC, delete the underlying IAM users themselves. At this point your cloud console has no access keys for any human or automation.

Four assumptions that broke

"Tokens auto-renew, right?" No. Our CI requests a token once at job start. If the job runs longer than the token TTL, credentials expire mid-job. Fix: long jobs re-request creds per step.

"Claim matching is exact across platforms." The OIDC sub claim format differs per platform. GitHub Actions uses repo:<org>/<repo>:ref:refs/heads/<branch>. GitLab uses a different format. CollabOps uses yet another. Platform migration means rewriting every trust policy.

"No rotation needed." OIDC tokens themselves don't rotate, but the IdP's signing keys do. Cloud-registered thumbprints go stale and get auto-rejected. Quarterly check required.

"Short-lived tokens give automatic audit." Clouds log AssumeRoleWithWebIdentity in CloudTrail, but visibility into which sub claim was used varies by tool. We added our own audit layer that logs every OIDC token issuance + assume role call explicitly. That layer is where the real audit value is.

The audit trail, six months later

Before:
  one deploy = 47 lines of access-key usage logs
             + grep across jobs to find them
             + sort by timestamp
             + try to separate humans from automation (near impossible)

After:
  one deploy = 1 line (OIDC token sub: <exact workflow URL>)
             automation vs human separation: automatic
             which repo, which branch, which workflow: explicit
             validity bounded by token TTL (auto-expires)

That single table is the real result of six months of work. Not cost — auditability.

So when should you move CI to OIDC?

The time to move CI to OIDC workload identity is when your pipelines hold more than five long-lived secrets. Past five, the inventory itself starts to break down. Past fifty, an incident is a matter of time. And a small team is no reason to wait: smaller teams get more audit value per unit of migration cost, not less.