SSH and the auth log

01The problem

Lesson 5 found a cryptocurrency miner running as the user backup. Nobody at Tallybook logs in as backup: it's an old account for a backup script. So how did someone get in as that user?

Every login attempt to a Linux server is recorded in the authentication log. Tallybook's has a week of it.

02The concept

SSH

Secure Shell: how engineers log in to remote servers (ssh deploy@server). Two ways to prove who you are:

Brute-force attacks

Internet-wide bots try common usernames (root, admin, ubuntu) and passwords on every server with SSH open to the world, all day long. With password login allowed, any account with a weak password is eventually found.

Hardening SSH

• Allow keys only (PasswordAuthentication no).
• Don't allow root to log in (PermitRootLogin no).
• Only allow SSH from known networks (a firewall rule, lesson 7).
• Use a tool like fail2ban to block addresses after repeated failures.
• Remove accounts that aren't needed.

The auth log

Lines like Failed password for root from 45.155.205.233, Accepted publickey for deploy from 10.0.2.15, plus sudo (commands run as root) and CRON (scheduled jobs).

03Example

How many failed login attempts, and from where?

%%bash
curl -sO https://academy.cloudtechanalytics.com/datasets/linux/auth.log
grep -c "Failed password" auth.log
grep "Failed password" auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

2252
    900 45.155.205.233
    600 218.92.0.112
    400 61.177.172.60
    352 194.26.29.120

($(NF-3) is the fourth field from the end: the address, in lines ending from IP port N ssh2.) Four addresses, over two thousand attempts. Did any get in? List every successful login:

%%bash
grep "Accepted" auth.log | awk '{print $7, "for", $9, "from", $11}' | sort | uniq -c

1 password for backup from 194.26.29.120
      3 publickey for ada from 102.89.34.5
     20 publickey for deploy from 10.0.2.15

The deploy system and Ada log in with keys from known addresses. One login is different: a password login for backup, from 194.26.29.120, one of the attacking addresses. Look at that address's history:

%%bash
grep "194.26.29.120" auth.log | grep -oE "(Failed|Accepted) password for (invalid user )?[a-z]+" | sed 's/invalid user [a-z]*/an invalid user/' | sort | uniq -c | sort -rn
grep -E "backup" auth.log | grep -v "Failed password" | head -n 5

312 Failed password for backup
     40 Failed password for an invalid user
      1 Accepted password for backup
Aug 30 02:14:51 prod-web-01 sshd[23276]: Accepted password for backup from 194.26.29.120 port 50211 ssh2
Aug 30 02:14:51 prod-web-01 sshd[23276]: pam_unix(sshd:session): session opened for user backup(uid=1003) by (uid=0)
Aug 30 02:16:03 prod-web-01 sudo:   backup : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/backup ; USER=root ; COMMAND=/bin/bash
Aug 30 02:21:44 prod-web-01 sshd[23276]: pam_unix(sshd:session): session closed for user backup
Aug 30 02:30:01 prod-web-01 CRON[23277]: (backup) CMD (/tmp/.x/kdevtmpfsi >/dev/null 2>&1)

The attacker tried common usernames, then focused on backup and kept guessing its password until, on 30 August at 02:14, it worked. They tried to become root with sudo and were refused (backup isn't allowed). So they settled for what backup could do: start a miner, and add a cron job that restarts it every ten minutes, which is what you saw at the end of the log in lesson 1.

04Walkthrough

1. Run the cells. On which day did the most failed attempts happen?
2. Count the cron entries for the miner. Since when has it been running?
3. Which accounts did attackers try that actually exist on the server (Failed password for root versus for invalid user)?
4. Write the response plan (the task below).

05Practice

06Check your understanding