@meilingzhou
Performance Engineer
Flamegraphs, Core Web Vitals, and careful caching.
We ran into this around background jobs while rolling back payments. I expected a tooling problem. It was an assumption in the schema. EXPLAIN and a short rollback window saved the afternoon. Writes slowed first, then a handful of reads timed out. Dropping the unused index and batching the backfill unblocked the queue. I now review write paths with the same care as the happy-path query.
We moved session checks to the edge to cut latency on every page load. It worked in staging, then failed on preview deploys when cookies crossed domains. Clock skew between edge and origin made short-lived tokens look expired. We fixed cookie domains per environment and added skew-tolerant expiry. Median auth path dropped about 120ms, with fewer cold-start surprises. Lesson: test cookies across every environment before calling a migration done.
I used to think more indexes always meant faster queries. Production taught me about write amplification and table bloat instead. We had three indexes that nothing queried, slowing every insert. EXPLAIN ANALYZE finally showed which plans actually used which indexes. After dropping the dead ones, writes got healthier without hurting reads. Now I review unused indexes in the same ritual as reviewing slow queries.
No spike in CPU. Error budgets looked fine at a glance. Users still reported blank pages in a thin slice of traffic. Logs only showed upstream resets with no clear application exception. The culprit was a stale keep-alive timeout between nginx and the app. Aligning idle timeouts stopped the intermittent 502s within an hour. We also added a dashboard for upstream reset reasons so the next page is faster.
We extracted one high-churn billing endpoint behind a strangler facade. Dual-writes ran for two weeks while we compared totals nightly. A feature flag controlled read traffic so we could roll back instantly. The hardest part was matching edge-case rounding in legacy invoices. Cutover finished with no customer-facing downtime and a smaller blast radius. We kept the facade until three more endpoints followed the same path.
We needed locks so queue workers did not process the same job twice. Redis SET NX was faster under load and easy to expire automatically. Postgres advisory locks were simpler operationally for our small team. Failover behavior mattered more than raw latency in our case. We chose Postgres first, then moved hot paths to Redis later. Pick the lock store you can operate confidently at 3am.