Security Logging and Monitoring Failures

9) Security Logging and Monitoring Failures

What it is

Your security controls might fail, but the bigger problem here is:

1. You do not log important security events, or
2. Logs exist but nobody monitors/alerts/responds.

Result:

1. Attack happens.
2. No one notices in time.
3. Attacker stays longer and causes more damage.

Simple way to understand it

Security is not only “prevent attack.” Security is also “detect attack quickly and respond quickly.”

If detection is weak, even small incidents become major breaches.

Common failures

1. Login failures not logged.
2. Privilege changes (user -> admin) not logged.
3. Logs have no user ID, source IP, request ID, timestamp.
4. Logs are only local and can be deleted by attacker.
5. No alerts for suspicious patterns.
6. Alert rules exist but are too noisy, so people ignore them.
7. No incident response playbook.
8. Log retention too short to investigate incidents.
9. Time is not synchronized across servers (timeline becomes confusing).

Step-by-step attack example

1. Attacker starts password spraying across many accounts.
2. App logs minimal info or nothing useful.
3. No threshold alert is configured.
4. Attacker succeeds on one account.
5. Attacker escalates privileges and exports data.
6. Weeks later company notices unusual behavior.
7. Forensics fail because logs are incomplete or gone.

Tricky concepts explained

1) Detection vs prevention

1. Prevention tries to block bad actions.
2. Detection finds attacks that bypass prevention.

You need both because prevention is never perfect.

2) Mean Time to Detect (MTTD) and Respond (MTTR)

1. MTTD = how fast you notice an attack.
2. MTTR = how fast you contain and recover.

Lower is better. Good logging/monitoring directly reduces both.

3) Audit trail

Audit trail = reliable sequence of who did what, when, from where.

Without audit trail:

1. You cannot prove impact.
2. You cannot confidently notify users/regulators.
3. You cannot learn and fix root cause.

4) Log integrity

If attacker can edit/delete logs, logs are not trustworthy.

Need:

1. Centralized log collection
2. Restricted access
3. Immutable/WORM storage for critical logs

What should always be logged (minimum baseline)

1. Authentication events (success/failure, MFA failure, lockouts)
2. Authorization failures (403, denied admin actions)
3. Account lifecycle events (create, disable, role change, password reset)
4. Sensitive data access/export events
5. Security configuration changes
6. API key/token creation/revocation
7. High-risk system errors and exceptions
8. Admin actions and privileged command execution

Each event should include:

1. Timestamp (UTC)
2. User/account/service identity
3. Source IP/device
4. Action + target resource
5. Outcome (success/fail)
6. Correlation/request ID

How to prevent/fix (practical order)

1. Define a security logging standard for all services.
2. Implement structured logs (JSON) for consistent parsing.
3. Send logs to centralized SIEM/log platform.
4. Protect logs from tampering/deletion.
5. Build high-value alerts first:
6. Many failed logins
7. Impossible travel
8. Privilege escalation
9. Large data export
10. New admin account creation
11. Tune alerts to reduce noise and false positives.
12. Create incident response runbooks and on-call process.
13. Test detection with tabletop and attack simulations.
14. Synchronize clocks with NTP across all systems.
15. Set retention period based on legal and forensic needs.

Detection checklist

1. If admin account is created now, who gets alerted and how fast?
2. Can we trace one user action across microservices?
3. Are logs centralized and immutable enough for forensics?
4. Are alert rules tested against real attack scenarios?
5. Do we have runbooks and trained responders?
6. Can we reconstruct a full timeline for the last 90 days?

Business impact

1. Late breach discovery (weeks/months).
2. Bigger data loss and higher recovery cost.
3. Compliance penalties for poor incident evidence.
4. Repeated attacks because root cause is unclear.