Incident Review: Running Out of Database Connections

The failure hit at peak time. Pages spun, server logs filled with timeouts. We traced from gateway to database and found the cause in a loop.

The chain

A batch routine opened a database connection inside a loop without closing it, exhausting the pool within twenty minutes.

With the pool empty, every request queued, and the thirty-second timeout made the interface appear frozen.

Monitoring collected CPU and memory but not pool usage, so no alert fired.

Fix and improvements

The same connection is now reused, and code review includes a resource release check.

Monitoring tracks active connections and queue length, with an alert at eighty percent.

Timeouts dropped from thirty seconds to eight so users see an error instead of waiting.

An incident review should not hunt for a person; it should find the path nobody was watching. Cover it and the same failure stays away.