7:23 AM. Client calls: "The admin dashboard won't load."

We'd deployed an update the night before—a new reporting feature, nothing major. Checked logs. No errors. Checked the server. CPU at 45%. Then I opened the database monitoring panel.

One query. Running for 8 minutes. Still going.

The "Simple" Feature

The new report pulled user activity across three tables, filtered by date range, grouped by department. Worked perfectly in test with 500 users. Production had 47,000.

The query was doing a full table scan on every request. No indexes on the join columns. No index on the date filter.

The Fix

We added composite indexes on the join keys and date column. Query time dropped from 8+ minutes to under 200ms. Total downtime: about 2 hours while we diagnosed, indexed, and verified.

What We Changed Going Forward

• Load-test with production-scale data volumes, not just realistic samples
• Review every new query with EXPLAIN before deploy
• Add database monitoring alerts for slow queries over 5 seconds

Indexes aren't optional optimization—they're infrastructure. One missing index can take down an entire dashboard.