Module 10 · Linux and Networking Basics
"Final project: what happened on prod-web-01"
Plan your final project, a full investigation of Tallybook's web server from its logs and command output, with a timeline, findings, fixes and the scripts that would have caught each problem.
About 20 minutes
The problem
Tallybook's CTO has asked for a written investigation of prod-web-01, the server you've been examining. Investors and an enterprise customer will read the summary. It must say what happened, when, how you know, how serious it is, and what has been and will be done, with every claim backed by a command and its output.
The concept
An investigation report
| Section | Contents | From |
|---|---|---|
| Timeline | every event with its time, from the first attack to the outage's end | auth.log, access.log |
| Findings | the outage, the break-in, the miner, the disk, permissions, firewall, DNS | lessons 3 to 8 |
| Evidence | the command and output behind each finding | every lesson |
| Severity | how serious each finding is, and why | your judgement |
| Fixes | done now, and to do, with owners | lessons 4 to 8 |
| Detection | the check that would have caught each problem | lesson 9 |
Separate facts from conclusions
"The log shows a password login for backup from 194.26.29.120 at 02:14:51 on 30 August" is a fact. "The attacker installed the miner" is a conclusion, supported by the facts that the miner runs as backup and started after that login. Good reports make the difference clear.
Example
The start of the timeline, built with one command from both logs. Each awk prints a sortable timestamp and a short description:
Shell (bash)
%%bash
curl -sO https://academy.cloudtechanalytics.com/datasets/linux/auth.log
curl -sO https://academy.cloudtechanalytics.com/datasets/linux/access.log
{
grep "Failed password" auth.log | awk '!seen[$(NF-3)]++ {print "08-" $2, substr($3, 1, 5), "first ssh failure from", $(NF-3)}'
grep -E "Accepted password|NOT in sudoers" auth.log | awk '{print "08-" $2, substr($3, 1, 5), $6, $7, $8, $9, $10, $11}'
grep "kdevtmpfsi" auth.log | head -n 1 | awk '{print "08-" $2, substr($3, 1, 5), "first cron run of the miner"}'
awk '$9 >= 500 {print "08-31", substr($4, 14, 5), "first server error of the outage"; exit}' access.log
} | sort08-26 01:00 first ssh failure from 45.155.205.233
08-27 14:00 first ssh failure from 218.92.0.112
08-29 23:30 first ssh failure from 194.26.29.120
08-30 02:14 Accepted password for backup from 194.26.29.120
08-30 02:16 backup : user NOT in sudoers
08-30 02:30 first cron run of the miner
08-31 04:00 first ssh failure from 61.177.172.60
08-31 09:40 first server error of the outageEach line is a fact with a time. Add the outage's end, the snapshot times and your conclusions, and the timeline tells the whole story.
Walkthrough
- Complete the timeline with the outage's last error and the ps and df snapshots.
- Write each finding with its evidence (command and output) and severity.
- Write the list of fixes with owners, and the detection script for each problem.
- Open the project brief on the course page and plan the write-up.
Practice
Practice
How many minutes passed between the successful break-in (02:14 on 30 August) and the first cron run of the miner (02:30)?
Task
10 minWrite the executive summary of your investigation (100 to 200 words): what happened (the outage and the break-in), when, how serious it is, what has been fixed, what is still to do, and how such problems will be detected in future. Keep facts and conclusions distinct.
Your work is checked for
- Mentions the outage with times
- Mentions the break-in and the miner
- Gives dates
- Severity
- Fixed and still to do
- Detection in future
- Between 100 and 200 words
Check your understanding
Answer every question to check.