How I Lost All My Server Credentials in One Command

A production incident, a 126-restart crash loop, and what I learned about deploying with AI.

Marjuk · 2026-04-27 · 6 min read

bash — root@KnowGraph root@KnowGraph:~# rsync -az . root@server:/var/www/api/ sending incremental file list .env.production → /var/www/api/.env ⚠ 16 credentials overwritten with 1 variable. JWT_SECRET gone. DB gone. All gone. PM2: knowgraph-api crashed. Restarting... [1] PM2: knowgraph-api crashed. Restarting... [2] PM2: knowgraph-api crashed. Restarting... [3] total restarts 126 root@KnowGraph:~#

Today Claude Code wiped my entire production server in about three seconds.

I was building a new feature. I asked it to deploy. It ran an rsync command from the wrong directory and copied the frontend's .env.production file — which contains exactly one variable, VITE_API_URL — directly over the API's .env file, which contained 16 credentials. JWT secret. Database password. Google OAuth keys. Stripe keys. AWS keys. Resend key. All gone.

The API started crash-looping immediately. 126 restarts before I caught it.

The error was TypeError: OAuth2Strategy requires a clientID option. Passport throws that on startup when GOOGLE_CLIENT_ID is undefined. PM2 kept restarting the process. PM2 kept failing. Every user who hit the site saw "Could not connect to server."

Recovery

Recovery meant logging into every service dashboard I use — Google Cloud Console, AWS IAM, Stripe, Resend — and copying credentials one by one into a new .env file. Two hours of work to undo three seconds of damage.

The good news: the database was never touched. All user data was fully intact. The credentials were gone, not the data.

What I learned

The .env file is not backed up anywhere by default. It's in .gitignore for good reason — you don't commit secrets. But that also means the only copy is on the server. When it's gone, it's gone. Back it up in 1Password or equivalent. Update it every time you add a new credential.

AI tools can deploy code. This is powerful. It's also dangerous if your deploy commands aren't airtight. The fix was one flag: --exclude='.env' on every rsync. That flag now lives in the deploy command permanently. Claude Code cannot overwrite the server's credentials no matter what directory it runs from.

Document your incidents. I added a "Known failure modes" section to my CLAUDE.md — the file that instructs Claude Code how to work in this codebase. Future me, and future Claude, will know exactly what went wrong and why. The goal is to never have the same incident twice.

The site is back. Nothing was lost except two hours. But it was a good reminder that automation without guardrails is just a faster way to make mistakes.