LOCKDOWN RANGE: LINUX - SOLUTION WALKTHROUGH ================================================================================ Companion to linux-range.html (Tool4That, Lockdown Range series). For instructors, study groups, and anyone checking their work AFTER trying a scenario on their own. All names, organizations, and addresses are fictional. Unofficial practice; not affiliated with or endorsed by Canonical or Linux Mint. CONTENTS 1. What the lab is (and is not) 2. The routine: power-on to Progress Check 3. Scenario 1 - Recruit: Makai Surf Supply (Ubuntu 24.04 desktop) 4. Scenario 2 - Operator: Lanikai Book Nook (Linux Mint 22.3) 5. Scenario 3 - Captain: Mauka Trail Outfitters (Ubuntu Server 24.04) 6. Linux hardening quick reference (commands and files) 7. Where a real machine differs from the simulator 8. Sources Format of the check lists below: [points] Check title as it appears in the Progress Check (hardening area) fix: the command or click-path (continuation lines end with \) why: one line on why it matters Hardening areas: Account management, Access control, Secure configuration, Patching, Malware defenses, Audit logging, and Data protection. Data protection covers the prohibited media files. The two questions per scenario are listed as Investigation. Defense Ladder s1 introduces the areas (https://freeapp4that.store/defense-ladder/#sophomore/s1). The Debrief links each check to its Defense Ladder lesson: - s3: accounts, sudo, SSH, setuid files, firewall, updates, software, media, time sync, Flatpak apps, and the web server banner. - s3 (One PAM line): password length, history, lockout, and empty-password sign-in. - s4: services and ports. - j3: persistence, sysctl, /etc/shadow permissions, repositories, AppArmor profiles, package file integrity, file capabilities, and database accounts. - s6: investigation questions. - Captain exceptions: the setuid question also links to s3, and the package question links to s4 and se1 (FreeBSD, Hunting on any Unix). -------------------------------------------------------------------------------- 1. WHAT THE LAB IS (AND IS NOT) -------------------------------------------------------------------------------- Linux Range is a single HTML page that behaves like a small Linux machine you have been asked to secure: a boot splash, a login screen, a minimal desktop (Ubuntu, Mint, or a text console on the server), and a bash-like terminal. The terminal runs over an in-memory filesystem with real paths and realistic contents: /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers, /etc/login.defs, /etc/pam.d/*, /etc/ssh/sshd_config, /etc/ufw, /etc/apt, cron, lightdm/gdm, and user homes. The terminal understands the commands Linux administrators use every day (section 6): sudo, apt, systemctl, ufw, useradd/usermod/userdel, passwd, chage, visudo, nano, vi, sed -i, find, grep, pipes, and redirection. Every change re-runs the scenario's checks and the Progress Check updates live. Availability checks fail (in red, with negative points) when something the Scenario Brief says the business needs gets broken. It deliberately does NOT run real binaries, simulate the whole GNOME or Cinnamon desktop, or emulate the network beyond listening ports. Nothing on the page touches your own computer. Progress is saved in the browser; "Reset lab" restores the scenario's original state and keeps the best score (like reverting a VM snapshot). Levels: 1 Recruit lists every check as a to-do with hints (two steps: the area and the principle, then the command); 2 Operator shows only the remaining count per hardening area; 3 Captain shows nothing but the checks you have passed. Versions: the machines follow Ubuntu 24.04.5 LTS "Noble Numbat" (kernel 6.8.0-146-generic) and Linux Mint 22.3 "Zena" (Cinnamon 6.6, Ubuntu noble package base). On Ubuntu 26.04 LTS the default sudo is sudo-rs, so its password prompt and messages look slightly different. -------------------------------------------------------------------------------- 2. THE ROUTINE: POWER-ON TO PROGRESS CHECK -------------------------------------------------------------------------------- [ ] 1. Pick the scenario (level pill in the header). The machine boots. [ ] 2. Sign in with the credentials from the Scenario Brief (user list on the desktop machines; typed at the "login:" prompt on the server). [ ] 3. Read "Scenario Brief.html" on the Desktop (server: the Scenario Brief button, or cat ~/Desktop/Scenario\ Brief.txt). Write down: authorized administrators, authorized users, critical services, policy numbers (password length, maximum age...). [ ] 4. Evidence first: answer the investigation questions before any change (Defense Ladder f1, https://freeapp4that.store/defense-ladder/#freshman/f1). Open "Investigation 1.txt" on the Desktop (Text Editor, or nano in the terminal), type the answer after "ANSWER:", and save (Ctrl+S in the editor; Ctrl+O Enter in nano). [ ] 5. Read before you change (cat, systemctl status, getent). Undo is cheap in the lab; on a real VM, take a snapshot first. [ ] 6. Start the long job early: once the package sources check out (ls /etc/apt/sources.list.d) and /etc/hosts looks right, run sudo apt update && sudo apt upgrade -y (on a real VM, in a second terminal while you keep working), and confirm it at the end with apt list --upgradable. [ ] 7. Work by hardening area, same order every time: Account management -> Access control (sudo, SSH, logins) -> Secure configuration (services, firewall, software) -> Patching -> Malware defenses (persistence, rogue services) -> Audit logging (auditd, the journal, the clock) -> Data protection. [ ] 8. Verify each area (getent group sudo, chage -l, ufw status verbose, ss -tulpn, dpkg -l | grep ...). Keep a short change log. [ ] 9. Reboot when a change needs it (sudo reboot), sign in again, confirm the checks still pass. [ ] 10. Finish / Debrief: read every miss and its fix. Reset lab, run it again faster. -------------------------------------------------------------------------------- 3. SCENARIO 1 - RECRUIT: MAKAI SURF SUPPLY (UBUNTU 24.04 WORKSTATION) -------------------------------------------------------------------------------- Login labuser / Easy!2026 Host ubuntu-lab (GDM user list) Scenario Brief summary Back-office desktop of a surf shop. IT contractor "kalani" manages it over SSH, which is a critical service: keep it running and reachable. Authorized administrators: labuser, kalani Authorized users: noe, pika, leilani, malia, tiko Policy: only the two admins have sudo; every account has a password of at least 10 characters that expires within 90 days and stays unchanged for at least 7 days after each change; no media files; no file-sharing clients or network scanners; firewall on; automatic security updates; nobody logged in automatically; the clock stays synchronized. Investigation Q1 Which account has an empty password? ANSWER: tiko (sudo cat /etc/shadow -> second field empty) Q2 Non-admin member of the sudo group? ANSWER: leilani (getent group sudo) Checks (100 points) [7] Unauthorized user rufus removed (Account management) fix: sudo userdel -r rufus why: an account nobody authorized is a door into the machine. [7] leilani removed from the sudo group (not an authorized administrator) (Access control) fix: sudo gpasswd -d leilani sudo (or: sudo deluser leilani sudo) why: sudo members are root in all but name. [7] Blank password on account tiko fixed (Account management) fix: sudo passwd tiko (type a strong password twice) why: an empty hash field lets anyone log in with Enter. [6] SSH root login disabled (PermitRootLogin no) (Access control) fix: sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' \ /etc/ssh/sshd_config && sudo systemctl restart ssh why: one guessed root password gives total control. [6] Password maximum age of 90 days or less (login.defs and every account) (Account management) fix: sudo sed -i 's/^PASS_MAX_DAYS.*/PASS_MAX_DAYS 90/' /etc/login.defs sudo chage -M 90 NAME for each human account: labuser, kalani, noe, pika, leilani, malia, and tiko (login.defs sets defaults for new accounts; chage updates each existing one, and sudo chage -l NAME shows its values; every human account still on the machine counts, so remove rufus first, or run chage on rufus too) why: passwords that never expire stay valid forever once leaked. [4] Password minimum age of 7 days or more (login.defs and every account) (Account management) fix: sudo sed -i 's/^PASS_MIN_DAYS.*/PASS_MIN_DAYS 7/' /etc/login.defs sudo chage -m 7 NAME for each of the same accounts (remove rufus first here too) why: stops cycling back to an old favorite password. [7] Minimum password length of 10 or more enforced via pam_pwquality (Account management) fix: sudo apt install libpam-pwquality sudo nano /etc/pam.d/common-password and make the line read: password requisite pam_pwquality.so retry=3 minlen=12 why: without the PAM module, passwd accepts "abc". [7] Firewall (ufw) enabled (Secure configuration) fix: sudo ufw allow ssh sudo ufw enable why: firewall off = every listener reachable. Allow SSH first, or the "Firewall blocks the required SSH service" availability check fails. [6] Automatic security updates enabled (Patching) fix: Software Updater > Settings... > Updates > "When there are security updates: Download and install automatically" (or: sudo dpkg-reconfigure -plow unattended-upgrades, answer yes) why: security updates close old bugs while nobody is watching. [6] System updates installed (apt upgrade) (Patching) fix: sudo apt-mark unhold sudo sudo apt update && sudo apt upgrade -y (apt upgrade lists sudo as "kept back": apt-mark showhold names the held package, and dpkg -l marks it hi. Software Updater leaves a held package out of its list, so unhold it first there too. The firefox snap is held as well: snap list shows "held" in Notes, and sudo snap refresh --unhold firefox lifts it.) why: the pending updates include fixes for sudo and OpenSSL. [6] Prohibited media files removed from user home directories (Data protection) fix: sudo find /home -type f \( -iname "*.mp3" -o -iname "*.mp4" \ -o -iname "*.m4a" -o -iname "*.mkv" -o -iname "*.flac" \) \ -delete why: policy bans them, and they hide pirated or malicious content. [7] Unauthorized software torrentbolt removed (file-sharing client) (Secure configuration) fix: sudo apt purge -y torrentbolt [5] Unauthorized software netprobe-ng removed (network scanner) (Secure configuration) fix: sudo apt purge -y netprobe-ng [6] Automatic (timed) login disabled in GDM (Access control) fix: sudo nano /etc/gdm3/custom.conf -> TimedLoginEnable = false (keep AutomaticLoginEnable off) why: timed login hands the desktop to whoever waits. [3] Clock synchronization on (systemd-timesyncd) (Audit logging) fix: sudo timedatectl set-ntp true (timedatectl status reads "System clock synchronized: no" and "NTP service: inactive" until then) why: logs line up across machines only when their clocks agree. [5] Investigation 1 answered tiko (Investigation) [5] Investigation 2 answered leilani (Investigation) Availability checks (points lost while broken) -8 Authorized user removed -6 Authorized administrator removed from sudo -6 Required service SSH stopped, disabled, or removed -4 Firewall blocks the required SSH service (port 22) -10 Your own account (labuser) deleted or locked -------------------------------------------------------------------------------- 4. SCENARIO 2 - OPERATOR: LANIKAI BOOK NOOK (LINUX MINT 22.3) -------------------------------------------------------------------------------- Login labuser / Medium!2026 Host mint-lab (starts auto-logged-in) Scenario Brief summary Bookshop reading-corner PC; patrons sign in with their own accounts. Technician "mkoa" maintains it. Complaints: the PC was already logged in when patrons sat down, and a strange "update helper" is running. No network services are required. Authorized administrators: labuser, mkoa Authorized users: hina, keanu, jace, nalu, aria Policy: only admins have sudo; never auto-login and no guest session; every account has a 10+ character password, no reuse of the last 5, lockout after repeated failures; no media; no miners or remote-shell servers; only approved apps (Flatpak apps included); every AppArmor profile in enforce mode; firewall on; automatic security updates. Investigation Q1 Account other than root with UID 0? ANSWER: sysadm (awk -F: '$3 == 0' /etc/passwd) Q2 Hidden folder of unauthorized tools? ANSWER: /home/jace/.hidden_tools (sudo ls -la /home/*) Checks (100 points) [5] Audit daemon running with a rule that watches /etc/sudoers (auditd) (Audit logging) fix: sudo apt install -y auditd printf "%s\n" "-w /etc/sudoers -p wa -k sudoers" | \ sudo tee /etc/audit/rules.d/50-sudoers.rules sudo augenrules --load sudo systemctl enable --now auditd why: logs are how you prove what changed; a watch on /etc/sudoers records every edit to who can become root. [5] Unauthorized user dax removed (Account management) fix: sudo userdel -r dax [6] keanu removed from the sudo group (not an authorized administrator) (Access control) fix: sudo gpasswd -d keanu sudo [6] Hidden root account sysadm removed (only root has UID 0) (Account management) fix: getent passwd sysadm (field 6: /home/sysadm, its own folder) sudo userdel -r -f sysadm (plain userdel refuses with "user sysadm is currently used by process 1": every root process runs as UID 0. When the home is /root or /, use sudo userdel -f sysadm, since -f with -r removes a shared home. Alternative: sudo vipw, change its UID from 0 to an unused number, then sudo userdel sysadm) why: any account with UID 0 IS root, whatever it is called. [6] Blank password on account aria fixed (Account management) fix: sudo passwd aria [5] Guest session disabled in LightDM (allow-guest=false) (Access control) fix: sudo nano /etc/lightdm/lightdm.conf -> allow-guest=false why: a guest session is an anonymous login. [5] Automatic login disabled in LightDM (Access control) fix: sudo sed -i '/^autologin-user/d' /etc/lightdm/lightdm.conf why: auto-login hands an admin session to the next person. [6] Minimum password length of 10 or more enforced via pam_pwquality (Account management) fix: sudo nano /etc/pam.d/common-password -> password requisite pam_pwquality.so retry=3 minlen=12 (libpam-pwquality is already installed on this machine) [4] Password history enforced (remember=5 or more) (Account management) fix: same file: append remember=5 to the pam_unix.so line (a pam_pwhistory.so remember=5 line also counts) why: stops toggling between two passwords forever. [5] Account lockout policy configured (pam_faillock, deny 10 or fewer) (Account management) fix: sudo nano /etc/pam.d/common-auth, before the pam_unix.so line: auth required pam_faillock.so preauth silent deny=5 unlock_time=900 and directly after it, these two lines: auth [default=die] pam_faillock.so authfail deny=5 unlock_time=900 auth sufficient pam_faillock.so authsucc (pam_unix's [success=1] skips exactly one line: without the authsucc line a CORRECT password lands on pam_deny and nobody can log in. pam_tally2 was removed in Linux-PAM 1.5.0 and current Ubuntu and Mint releases do not ship it, so a pam_tally2 line does not count. The managed way: on Ubuntu 24.04 and Mint 22 the faillock profiles are yours to write, and a name works once its profile exists: write the faillock and faillock_notify profiles in /usr/share/pam-configs/, then sudo pam-auth-update --enable faillock faillock_notify. It rebuilds the common-* files and refuses while they carry hand edits, unless you add --force.) why: lockout turns a million guesses into five. [3] Empty passwords rejected at sign-in (nullok removed from common-auth) (Account management) fix: sudo sed -i -E 's/[[:space:]]+nullok(_secure)?([[:space:]]|$)/\2/' \ /etc/pam.d/common-auth why: nullok turns a blank /etc/shadow field into a working login. [5] Firewall (ufw) enabled (Secure configuration) fix: sudo ufw enable (nothing needs allowing here) [4] Automatic security updates enabled (Patching) fix: Update Manager (shield icon, bottom right) > Edit > Preferences > Automation > Apply updates automatically (or: sudo mintupdate-automation upgrade enable; this enables the mintupdate-automation-upgrade.timer systemd timer and the flag file the job checks. unattended-upgrades counts only when 50unattended-upgrades also lists "Ubuntu:noble-security": on Mint its ${distro_id} entries expand to Linuxmint:zena-security) [4] Prohibited media files removed from user home directories (Data protection) fix: sudo find /home -type f \( -iname "*.mp3" -o -iname "*.mp4" \ -o -iname "*.m4a" \) -delete (there is an .m4a file!) [6] Unauthorized software minerd-x removed (cryptocurrency miner) (Malware defenses) fix: sudo apt purge -y minerd-x [4] Telnet server removed (nothing listening on port 23) (Secure configuration) fix: sudo apt purge -y telnetd inetutils-telnetd (found with: sudo ss -tulpn, then dpkg -l | grep -i telnet. On Ubuntu 24.04 / Mint 22 telnetd is only a transitional package; the server and its /etc/inetd.conf line belong to inetutils-telnetd. Purging telnetd alone leaves port 23 open and apt only says inetutils-telnetd "is no longer required". Alternative: sudo systemctl disable --now inetd) why: Telnet sends passwords in clear text. Check which package really provides a program before you remove it. [5] Rogue service libupdater disabled or removed (Malware defenses) fix: sudo systemctl disable --now libupdater sudo rm /etc/systemd/system/libupdater.service sudo rm -r /opt/.libupd && sudo systemctl daemon-reload (systemctl cat libupdater shows ExecStart=/opt/.libupd/libupd and Restart=always) why: Restart=always means kill alone brings it straight back. [3] cupsd AppArmor profile back in enforce mode (Secure configuration) fix: sudo apt install -y apparmor-utils sudo aa-enforce /etc/apparmor.d/usr.sbin.cupsd (sudo aa-status lists /usr/sbin/cupsd under complain mode) why: a profile in complain mode logs a violation and lets it happen. [3] Unauthorized Flatpak app removed (PixelQuest) (Secure configuration) fix: sudo flatpak uninstall -y io.example.PixelQuest (flatpak list --app names it; dpkg -l misses Flatpak apps) why: Flatpak apps sit outside apt, so they outlive a package audit. [5] Investigation 1 answered sysadm (Investigation) [5] Investigation 2 answered /home/jace/.hidden_tools (Investigation) Availability checks (points lost while broken) -8 Authorized user removed -6 Authorized administrator removed from sudo -10 Your own account (labuser) deleted or locked -------------------------------------------------------------------------------- 5. SCENARIO 3 - CAPTAIN: MAUKA TRAIL OUTFITTERS (UBUNTU SERVER 24.04) -------------------------------------------------------------------------------- Login labuser / Hard!2026 Host ubuntu-srv (text console) SSH on this server is socket-activated, as on every Ubuntu 24.04 install of openssh-server: ssh.socket (enabled) holds port 22 and starts ssh.service (shown as "disabled", "TriggeredBy: ssh.socket") on demand. - systemctl disable --now ssh does NOT close port 22 (systemctl warns "Stopping 'ssh.service', but its triggering units are still active: ssh.socket"); ss -tlnp shows systemd (pid 1) listening. - Authentication settings (PermitRootLogin...): sudo systemctl restart ssh. - Port / ListenAddress changes: sudo systemctl daemon-reload, then sudo systemctl restart ssh.socket (the sshd-socket-generator rewrites the socket's listeners from sshd_config). - systemctl status sshd says "Unit sshd.service could not be found": the sshd alias only exists while ssh.service is enabled. Use "ssh". - The SSH availability check fails if ssh.socket is stopped, disabled or masked (and ssh.service is not running/enabled instead), or if sshd_config would make sshd fail its syntax check. The package sources use the deb822 format: /etc/apt/sources.list only says "Ubuntu sources have moved to /etc/apt/sources.list.d/ubuntu.sources". One-line .list files in /etc/apt/sources.list.d/ are still read. Scenario Brief summary (cat ~/Desktop/Scenario\ Brief.txt) Web server (Apache + MySQL) for an online outdoor-gear store; admins work over SSH from the office network 10.20.4.0/24. The hosting intern left abruptly and the owner wants the server audited. Authorized administrators: labuser, moana Authorized users: bkup (backup operator, NOT an admin), kai, lehua Critical: SSH (keep running and allowed through the firewall), Apache on port 80, MySQL. FTP, Samba, and Telnet are not authorized. Root may not log in over SSH; sudo always asks for a password; empty passwords forbidden; IP forwarding off; no unauthorized software, repositories, setuid binaries, or scheduled jobs; install updates; firewall on; every system program matches its package, interpreters keep their normal privileges, the web server hides its version, and every database account has a name; every running process comes from a package or from a program the brief names. Investigation Q1 TCP port held by a program outside every installed package? ANSWER: 4444 (sudo ss -tulpn names each port's program, and ps aux shows the full path it started from, here /tmp/.sysupd. Ubuntu packages install programs in /usr/bin, /usr/sbin, and /usr/lib, which /lib points to. Files in /usr/local and /tmp were put there by hand, so a program that runs from /tmp sits outside every package. dpkg -S /tmp/.sysupd confirms it with "no path found", and dpkg -S /usr/sbin/vsftpd names vsftpd. dpkg -S /usr/lib/systemd/systemd-resolved names systemd-resolved. dpkg lists files by their /usr path, so dpkg -S /sbin/init says "no path found" even though /sbin points to /usr/sbin; dpkg -S /usr/sbin/init names systemd-sysv. Every other listener comes from a package: sshd (22, held by systemd), vsftpd (21), apache2 (80), smbd (139 and 445), systemd-resolved (53, loopback only), and mysqld (3306, loopback only). sudo ss -tnp also shows an ESTABLISHED connection to 4444 from 10.20.4.88. Also see /etc/rc.local and root's crontab, whose line is base64-encoded: echo THE-STRING | base64 -d shows the command.) Q2 Setuid binary that gives a root shell? ANSWER: /usr/local/bin/lsmon (find / -perm -4000 -type f 2>/dev/null) Q3 IP address the fake kernel worker keeps trying to reach? ANSWER: 203.0.113.50 (203.0.113.50:8443 is accepted too. The changed /usr/bin/ps leaves the process out, so ps aux | grep kworker shows only real kernel threads. ls /proc lists every PID; sudo apt install -y unhide, then sudo unhide proc reports it: Cmdline "[kworker/1:3]", Executable /var/tmp/.font-unix/fcsync. A kernel thread lacks a program file, so sudo ls -l /proc/PID/exe fails for the real ones; pstree -p and sudo systemctl status fontcache-sync show the impostor's PID too. sudo strace -p PID -e trace=network prints, every ten seconds: connect(4, {sa_family=AF_INET, sin_port=htons(8443), sin_addr=inet_addr("203.0.113.50")}, 16) = -1 EINPROGRESS Ctrl+C detaches. Without sudo: "Operation not permitted", because kernel.yama.ptrace_scope is 1.) Q4 File that process keeps reading? ANSWER: /etc/mysql/debian.cnf (sudo ls -l /proc/PID/fd shows 3 -> /etc/mysql/debian.cnf; sudo strace -p PID -e trace=file shows newfstatat() on it every pass, and sudo strace -p PID -s 200 shows the pread64() of the password. sudo fuser -v /etc/mysql/debian.cnf and sudo lsof -p PID name the process; sudo inotifywait -m /etc/mysql prints ACCESS debian.cnf.) Q5 Library that process preloads? ANSWER: /usr/lib/x86_64-linux-gnu/.fc/libfcsync.so.1 (/lib/x86_64-linux-gnu/.fc/libfcsync.so.1 is accepted too. sudo cat /proc/PID/maps and sudo pmap PID list it beside libc; sudo cat /proc/PID/environ | tr '\0' '\n' shows the LD_PRELOAD line. ldd /var/tmp/.font-unix/fcsync lists libc alone, so the extra library came from LD_PRELOAD. sudo ltrace -p PID shows inet_addr("203.0.113.50") after apt install ltrace.) Checks (100 points) [3] System journal kept across reboots (journald Storage=persistent) (Audit logging) fix: sudo sed -i 's/^Storage=.*/Storage=persistent/' \ /etc/systemd/journald.conf && \ sudo systemctl restart systemd-journald why: the intern set Storage=volatile; a journal in RAM vanishes at the first reboot, and the evidence goes with it (journalctl --list-boots lists only the current boot). [3] Unauthorized user vex removed (Account management) fix: sudo userdel -r vex [4] Unauthorized sudo rule for bkup removed (/etc/sudoers.d) (Access control) fix: sudo chattr -i /etc/sudoers.d/90-backup sudo rm /etc/sudoers.d/90-backup (a plain rm says "Operation not permitted": sudo lsattr /etc/sudoers.d/90-backup shows the i (immutable) flag) why: NOPASSWD: ALL in a drop-in file is silent root. [3] Sudo asks for a password again (Defaults !authenticate removed) (Access control) fix: sudo visudo, delete the line Defaults !authenticate (scripted: sudo sed -i '/^Defaults[[:space:]]*!authenticate/d' \ /etc/sudoers && sudo visudo -c) why: a quieter NOPASSWD: every sudoer runs root commands without a password. [3] Hidden root account ops removed (only root has UID 0) (Account management) fix: getent passwd ops, then sudo userdel -r -f ops (its home is /home/ops; plain userdel says "currently used by process 1", see scenario 2) [4] SSH root login disabled (PermitRootLogin no) (Access control) fix: sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' \ /etc/ssh/sshd_config && sudo systemctl restart ssh [4] SSH empty passwords disabled (PermitEmptyPasswords no) (Access control) fix: sudo sed -i \ 's/^#\?PermitEmptyPasswords.*/PermitEmptyPasswords no/' \ /etc/ssh/sshd_config && sudo systemctl restart ssh [3] SSH MaxAuthTries limited to 4 or fewer (Access control) fix: sudo sed -i 's/^#\?MaxAuthTries.*/MaxAuthTries 4/' \ /etc/ssh/sshd_config && sudo systemctl restart ssh [4] FTP server (vsftpd) removed or disabled (vsftpd) (Secure configuration) fix: sudo apt purge -y vsftpd (or: sudo systemctl disable --now vsftpd) [4] Samba file sharing removed or disabled (smbd/nmbd) (Secure configuration) fix: sudo apt purge -y samba (or: disable --now smbd nmbd) why: a guest-writable share of the website lets the LAN change it. [4] Firewall (ufw) enabled (Secure configuration) fix: sudo ufw allow ssh sudo ufw allow 80/tcp sudo ufw enable why: both allows first, or the SSH and web availability checks fail. [3] IPv4 forwarding disabled at boot (sysctl files) (Secure configuration) fix: sudo sed -i 's/^net.ipv4.ip_forward=1/net.ipv4.ip_forward=0/' \ /etc/sysctl.conf && sudo sysctl -p (or: put net.ipv4.ip_forward=0 in /etc/sysctl.d/99-hardening.conf, delete the sysctl.conf line, then sudo sysctl --system; Ubuntu 26.04 ships without /etc/sysctl.conf, so the drop-in is its route) why: a forwarding server is a pivot into the rest of the network. [4] Setuid bit removed from /usr/local/bin/lsmon (Access control) fix: sudo rm /usr/local/bin/lsmon (or: sudo chmod u-s it) why: a setuid copy of a shell is an instant root shell for anyone. [4] Persistence line removed from /etc/rc.local (Malware defenses) fix: sudo sed -i '/sysupd/d' /etc/rc.local sudo rm -f /tmp/.sysupd && sudo pkill sysupd why: rc.local runs as root at boot; the 4444 listener returns after every reboot until the line is gone. [4] Planted root cron job removed (Malware defenses) fix: sudo crontab -r (or: sudo crontab -e and delete the line) (the line pipes a base64 string into sh: echo L3RtcC8uc3lzdXBkIC1sIDQ0NDQgLWUgL2Jpbi9zaA== | base64 -d prints /tmp/.sysupd -l 4444 -e /bin/sh) why: cron restarts the backdoor every 10 minutes. [3] /etc/shadow permissions restored (640 root:shadow) (Access control) fix: sudo setfacl -b /etc/shadow sudo chmod 640 /etc/shadow && sudo chown root:shadow /etc/shadow (ls -l shows -rw-rw-r--+: the + is an ACL entry. sudo getfacl /etc/shadow names it: user:vex:rw-, or user:1006 once vex is deleted.) why: world-readable hashes can be cracked offline by any user. [3] Untrusted third-party repository removed (Patching) fix: sudo rm /etc/apt/sources.list.d/torrentbolt.list && sudo apt update (ubuntu.sources next to it is the official archive: keep it) why: a [trusted=yes] repository can push any package onto the box. [3] System updates installed (apt upgrade) (Patching) fix: sudo apt update && sudo apt upgrade -y [3] Unauthorized software netprobe-ng removed (network scanner) (Secure configuration) fix: sudo apt purge -y netprobe-ng [3] Apache hides its version (ServerTokens Prod, ServerSignature Off) (Secure configuration) fix: sudo sed -i -e 's/^ServerTokens .*/ServerTokens Prod/' \ -e 's/^ServerSignature .*/ServerSignature Off/' \ /etc/apache2/conf-available/security.conf sudo apache2ctl configtest && sudo systemctl reload apache2 (curl -sI localhost shows the Server header before and after) why: a version string tells an attacker which exploits to try. [4] Changed package file restored (debsums -s clean) (Malware defenses) fix: sudo apt install -y debsums && sudo debsums -s sudo apt install --reinstall -y procps (debsums -s reports /usr/bin/ps from procps; sudo dpkg --verify procps shows ??5?????? for it) why: a changed ps can hide processes from the administrator (here it leaves out the fake kworker); the package archive holds the real one. [4] cap_setuid removed from python3.12 (Access control) fix: sudo setcap -r /usr/bin/python3.12 (sudo getcap -r / 2>/dev/null lists file capabilities) why: an interpreter that may change its user ID is a root shell for everyone. [3] Anonymous MySQL account removed (Access control) fix: sudo mysql -e "DROP USER ''@'localhost';" (or: sudo mysql_secure_installation, y to Remove anonymous users. sudo mysql -e "SELECT user, host, plugin FROM mysql.user;" shows the empty user name.) why: an anonymous account lets anyone on the server connect without a name. [5] Disguised backdoor removed (the fake kworker, its program, its preload library, and its unit) (Malware defenses) fix: sudo systemctl disable --now fontcache-sync sudo rm /etc/systemd/system/fontcache-sync.service sudo systemctl daemon-reload sudo rm -r /var/tmp/.font-unix /usr/lib/x86_64-linux-gnu/.fc (systemctl status PID names the unit. Restart=always brings a plain sudo kill PID back under a new PID, and so does sudo fuser -k; stop the unit first.) why: it sends the database maintenance password toward 203.0.113.50 every ten seconds and returns at every boot until its unit, program, and library are all gone. [3] Investigation 1 answered 4444 (Investigation) [3] Investigation 2 answered /usr/local/bin/lsmon (Investigation) [3] Investigation 3 answered 203.0.113.50 (Investigation) [3] Investigation 4 answered /etc/mysql/debian.cnf (Investigation) [3] Investigation 5 answered /usr/lib/x86_64-linux-gnu/.fc/libfcsync.so.1 (Investigation) Availability checks (points lost while broken) -6 Authorized user removed -5 Authorized administrator removed from sudo -8 Required service Apache (apache2) stopped, disabled, or removed -8 Required service SSH stopped, disabled, or removed -5 Required service MySQL stopped, disabled, or removed -5 Firewall blocks the required SSH service (port 22) -4 Firewall blocks the required web (Apache) service (port 80) -10 Your own account (labuser) deleted or locked -------------------------------------------------------------------------------- 6. LINUX HARDENING QUICK REFERENCE -------------------------------------------------------------------------------- Account management cat /etc/passwd users; field 3 = UID (0 is root); 7 = shell sudo cat /etc/shadow field 2 = hash; "!" or "*" locked; EMPTY = no password at all sudo userdel -r NAME remove the user and the home directory sudo userdel -f NAME an extra UID 0 account; read its home with getent passwd NAME and add -r only when that home is its own sudo passwd NAME set a password (asked twice) sudo passwd -l NAME / -u lock / unlock sudo chage -l NAME aging; chage -M 90 -m 7 -W 14 NAME sets it Access control getent group sudo who can sudo (Debian, Ubuntu, Mint) sudo gpasswd -d NAME sudo remove sudo rights (also: deluser NAME sudo) sudo usermod -aG sudo NAME grant sudo rights sudo visudo edit /etc/sudoers with a syntax check sudo ls -la /etc/sudoers.d drop-in sudo rules hide here sudo lsattr FILE / chattr -i the i flag blocks every change, root's too sudo getfacl FILE / setfacl -b a + after ls -l modes means an ACL entry sudo getcap -r / 2>/dev/null file capabilities; sudo setcap -r FILE Password policy /etc/login.defs PASS_MAX_DAYS 90, PASS_MIN_DAYS 7, PASS_WARN_AGE 14 (defaults for new accounts; chage updates existing ones) /etc/pam.d/common-password pam_pwquality.so retry=3 minlen=12 pam_unix.so ... remember=5 /etc/pam.d/common-auth pam_faillock.so preauth / authfail / authsucc, deny=5 (see scenario 2) sudo apt install libpam-pwquality the module that enforces minlen sudo pam-auth-update --enable NAME profiles in /usr/share/pam-configs SSH (/etc/ssh/sshd_config, then sudo systemctl restart ssh) PermitRootLogin no PermitEmptyPasswords no MaxAuthTries 4 PasswordAuthentication per the brief X11Forwarding no sudo sshd -t syntax check before restarting ssh NAME@localhost test a sign-in from a second terminal before you log out; it asks about the key of the server first, and sshd uses the settings it read at its last start or reload systemctl status ssh "TriggeredBy: ssh.socket" = socket-activated (Ubuntu 24.04): ssh.socket owns port 22 sudo systemctl daemon-reload && sudo systemctl restart ssh.socket apply Port/ListenAddress changes (24.04) (sshd uses the FIRST value it reads for each keyword, and files in /etc/ssh/sshd_config.d/*.conf are read first) Services and ports sudo ss -tulpn what is listening (netstat -tulpn also) systemctl status NAME running? enabled? which unit file? sudo systemctl disable --now NAME stop now and after every reboot (check "TriggeredBy:" first: a socket can keep the port open and restart it) sudo systemctl mask NAME make it unstartable ls /etc/systemd/system homemade unit files live here systemctl cat NAME the unit file and its drop-ins systemctl list-unit-files --state=enabled what starts at boot systemctl --user list-unit-files your own units; others hide in ~/.config/systemd/user of each account sudo ss -tnp; sudo lsof -nP -i established connections and their programs curl -sI localhost a web server's banner (Server header) Firewall sudo ufw status verbose state and rules sudo ufw allow ssh allow required services FIRST sudo ufw default deny incoming then: sudo ufw enable sudo ufw show raw every netfilter table ufw loaded, IPv4 and IPv6, the shipped rules included Patching and software sudo apt update && sudo apt upgrade -y dpkg -l | grep -i NAME is it installed? sudo apt purge -y NAME remove with its config files ls /etc/apt/sources.list.d extra repositories (ubuntu.sources is the official archive in deb822 format) dpkg -l | grep -i NAME also shows transitional packages: the real program may come from another package apt-mark showhold held packages (apt upgrade keeps them back) sudo dpkg -i FILE.deb install a downloaded package (-r removes one, -P also its configuration files) /etc/apt/apt.conf.d/20auto-upgrades APT::Periodic::Unattended-Upgrade "1"; sudo mintupdate-automation upgrade enable Linux Mint automatic updates flatpak list --app; sudo flatpak uninstall ID apps installed outside apt (flatpak uninstall --unused then removes the runtime they leave behind) Malware defenses and files sudo find /home -type f \( -iname "*.mp3" -o -iname "*.mp4" ... \) find / -perm -4000 -type f 2>/dev/null setuid binaries ls -l /etc/shadow must be -rw-r----- root shadow (640) sudo crontab -l; cat /etc/crontab; ls /etc/cron.d scheduled jobs cat /etc/rc.local runs as root at boot ps aux | grep NAME; sudo kill PID running processes sudo debsums -s; sudo dpkg --verify PKG package files changed? sudo aa-status AppArmor profiles and their modes sudo clamscan -r -i DIR; sudo rkhunter --check --sk; sudo aide --check scanners and integrity checkers (apt install) Login managers /etc/gdm3/custom.conf AutomaticLoginEnable = false, TimedLoginEnable = false (Ubuntu) /etc/lightdm/lightdm.conf allow-guest=false, no autologin-user= (Linux Mint) gsettings set org.gnome.desktop.screensaver lock-enabled true screen lock for the current user (Mint: org.cinnamon.desktop.screensaver) Kernel and network /etc/sysctl.conf net.ipv4.ip_forward=0, then sudo sysctl -p sudo tcpdump -n -w case.pcap save a capture (the file belongs to the tcpdump user); tcpdump -n -r case.pcap reads it back without sudo Shell for u in keola pua; do sudo chage -l "$u"; done one command per name set -u a line that reads an unset variable stops set -euo pipefail the safety line for scripts; typed at a prompt, -e closes the shell at the first failed command Audit logging sudo tail -n 20 /var/log/auth.log logins, sudo use, su journalctl -u ssh -n 20 recent SSH service log journalctl --list-boots boots the journal kept last; sudo lastb; lastlog logins, failed logins, latest each sudo ausearch -k KEY -i; sudo aureport -x --summary audit events timedatectl status clock and NTP sync Process investigation ls /proc every process ID (a changed ps hides one) sudo ls -l /proc/PID/exe the program it runs; kernel threads lack one tr '\0' ' ' < /proc/PID/cmdline its command line (it can rewrite this) cat /proc/PID/status Name, PPid (2 for kernel threads), Uid sudo cat /proc/PID/environ | tr '\0' '\n' its environment (LD_PRELOAD) sudo ls -l /proc/PID/fd files and sockets it holds open sudo cat /proc/PID/maps; sudo pmap PID libraries mapped into it ldd PROGRAM the libraries the program asks for sudo strace -p PID -e trace=network its system calls until Ctrl+C (also trace=file, trace=process, -f, -o FILE, -s 200); strace CMD traces a command from its start sudo ltrace -p PID -e connect its library calls (apt install ltrace) sudo fuser -v FILE; sudo fuser -v -n tcp PORT who uses a file or port (-k kills each one: read the -v list first) sudo inotifywait -m -r -e modify,create,delete DIR file events until Ctrl+C (apt install inotify-tools) sudo unhide proc; sudo unhide-tcp processes and ports that ps and ss leave out (apt install unhide) Editors nano FILE Ctrl+O Enter = save, Ctrl+X = exit Search: F6 (Ctrl+W can close the browser tab here). sed -i 's/^#\?Key.*/Key value/' FILE change a config line in place -------------------------------------------------------------------------------- 7. WHERE A REAL MACHINE DIFFERS -------------------------------------------------------------------------------- - Real VMs are slow: a reboot takes a minute, apt upgrade several. Take a VM snapshot before destructive changes; Reset lab is the lab's equivalent. - The real desktop has many more windows and settings (Users and Groups, gufw, Software and Updates tabs). The commands are the same. - The lab reads file and service state. On a real machine many changes only take effect after a service restart (sshd) or a reboot; restart the service and verify (sudo sshd -T | grep -i permitrootlogin). - A broken PAM file on a real machine can lock everyone out, including you. Keep a root shell open in a second terminal while you edit /etc/pam.d/*, and test with a fresh login before closing it. - Tab completion, history, and man pages cover the common commands here; a real system has them for everything. - Out-of-scope commands get a "[Range] ..." note instead of pretending to work. If something you need is missing, try it on a practice VM. - md5sum, sha1sum, sha256sum, and sha512sum hash a text file's real content, so printf 'abc' | sha256sum matches any reference value. A simulated binary has no bytes, so it hashes a stand-in value: compare binaries on a real machine. debsums, dpkg --verify, AIDE, and rkhunter compare package programs with their shipped size in the same way. - The audit tools record events while auditd runs: each rule that auditctl or augenrules --load adds or removes (a CONFIG_CHANGE event that carries the rule's key), file watches (-w PATH -p wa -k KEY), execve rules, and the USER_CMD event sudo writes for each command. ausearch -k KEY and aureport read them. The SYSCALL records leave out the a0-a3 argument fields that a real log carries. - The network is the lab segment. The machine's own addresses reach its listening services (curl, wget, nc, ssh, ssh-copy-id), and ping gets answers from them and from the gateway. Names resolve from /etc/hosts and from the names systemd-resolved provides (the host name, _gateway, and localhost); every other name returns NXDOMAIN. An empty address on the segment answers ping with "Destination Host Unreachable", and a host past the gateway stays silent (a connection times out). ssh asks about the host key, then follows the sshd_config that sshd read at its last start or reload (PermitRootLogin for UID 0 accounts, PasswordAuthentication, PermitEmptyPasswords, MaxAuthTries, AllowUsers, and keys in authorized_keys, command="..." options included) and opens a session; known_hosts names are hashed with a stand-in digest. tcpdump shows a few modeled packets, sudo tcpdump -w saves them to a file that tcpdump -r reads without root, and traceroute stops at the gateway. ping -6, scp, and sftp run on a real machine. - A ufw command with a syntax error prints "ERROR: Invalid syntax"; a real ufw then also prints its command help. - The shell runs for loops (on one line, or across lines at the "> " prompt), NAME=value variables, and set -e, -u, -x, and -o pipefail, as bash does. if, while, until, case, and select blocks, command substitution with $( ), and scripts run on a real shell. - systemctl --user reaches the signed-in account's own user manager. For other accounts, read the files in their ~/.config/systemd/user: their managers run only while they are signed in. - ufw show raw prints the same tables a real machine does, the IPv6 rules from before6.rules included, with every packet counter at 0. iptables 1.8.10 with -n shows the protocol as a number (0 for all, 6 for tcp, 17 for udp, 58 for ipv6-icmp). - On the Captain server, nginx installed beside Apache fails to start: both want port 80, as they would on a real server. - Only the Captain server simulates ssh socket activation. On a real Ubuntu 24.04 desktop, installing openssh-server is socket-activated too; the Recruit desktop keeps the simpler ssh.service model. - On Mint, the ${distro_id} entries in /etc/apt/apt.conf.d/50unattended-upgrades expand to Linuxmint:zena-security, an origin that misses every Ubuntu security update. Only an entry that names "Ubuntu:noble-security" counts. - The lab's kernel is the noble 6.8 GA kernel on all three machines; a fresh Linux Mint 22.3 or Ubuntu 24.04.5 desktop install ships a newer HWE kernel (Mint 22.3: 6.14). - Ubuntu 26.04 LTS uses sudo-rs as the default sudo (slightly different prompts and messages), chrony for time sync, and apt 3. - strace and ltrace print the system and library calls each simulated command makes from the files it really touches, with a short loader sequence; ldd, /proc/PID/maps, and pmap list a short set of libraries for most programs. strace -p, ltrace -p, and inotifywait wait for Ctrl+C as the real tools do; inotifywait prints the events that running programs and the commands in another terminal window cause while it waits, and the planted program on the Captain server repeats its calls every ten seconds while something watches it. - Only change machines you own or are authorized to administer. Practice VMs and this simulator are the places to experiment. -------------------------------------------------------------------------------- 8. SOURCES -------------------------------------------------------------------------------- Ubuntu Server docs, automatic updates: https://ubuntu.com/server/docs/how-to/software/automatic-updates/ pam_faillock(8), Ubuntu 24.04: https://manpages.ubuntu.com/manpages/noble/man8/pam_faillock.8.html Linux-PAM 1.5.0 release notes (pam_tally2 removed): https://github.com/linux-pam/linux-pam/releases/tag/v1.5.0 userdel(8), login.defs(5), sshd_config(5), ufw(8), Ubuntu 24.04: https://manpages.ubuntu.com/manpages/noble/ sources.list(5) (deb822 format), Ubuntu 24.04: https://manpages.ubuntu.com/manpages/noble/man5/sources.list.5.html OpenSSH socket activation on Ubuntu 24.04 (README.Debian, ssh.socket): https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/README.Debian?h=ubuntu/noble-updates https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/systemd/ssh.socket?h=ubuntu/noble-updates telnetd on noble (transitional; inetutils-telnetd adds its inetd line): https://launchpad.net/ubuntu/noble/+package/telnetd https://git.launchpad.net/ubuntu/+source/inetutils/tree/debian/inetutils-telnetd.postinst?h=ubuntu/noble-updates Linux Mint 22.3 "Zena" release (Cinnamon 6.6, noble base): https://www.linuxmint.com/rel_zena_cinnamon_whatsnew.php Ubuntu 26.04 LTS changes for 24.04 users (sudo-rs default): https://documentation.ubuntu.com/release-notes/26.04/summary-for-lts-users/ Debian wiki, periodic and unattended upgrades: https://wiki.debian.org/PeriodicUpdates Linux Mint Update Manager source (automation): https://github.com/linuxmint/mintupdate