We ran into this around invoice generation when the cache went cold. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around the admin dashboard during a traffic spike. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around team onboarding after we split the monolith. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around invoice generation after we split the monolith. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around the admin dashboard while rolling back payments. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around team onboarding on a quiet Sunday incident. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around invoice generation on a quiet Sunday incident. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around the admin dashboard after the replica failover. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around invoice generation during a Friday deploy. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We ran into this around team onboarding during a Friday deploy. We did not rewrite the service. We moved one endpoint at a time. Dual writes ran until nightly totals matched for several days. A flag sent a small percentage of reads to the new path. The hard part was matching rounding the old code had hidden. Cutover finished with a small blast radius and an easy rollback.
We needed locks so queue workers did not process the same job twice. Redis SET NX was faster under load and easy to expire automatically. Postgres advisory locks were simpler operationally for our small team. Failover behavior mattered more than raw latency in our case. We chose Postgres first, then moved hot paths to Redis later. Pick the lock store you can operate confidently at 3am.
We needed locks so queue workers did not process the same job twice. Redis SET NX was faster under load and easy to expire automatically. Postgres advisory locks were simpler operationally for our small team. Failover behavior mattered more than raw latency in our case. We chose Postgres first, then moved hot paths to Redis later. Pick the lock store you can operate confidently at 3am.