Jul 24, 2018 - Useful bash script snippets (1)


Useful bash script snippets (1)

Iterate though files in a directory and do something

for f in `ls someDirectory`
	echo $f

As one liner:

for f in `ls someDirectory`; do echo $f; done;

You want to do something in directories which match a certain pattern, e.g. start with a number:

IFS=$(echo -en "\n\b")

for d in `find [0-9]* -type d`
	echo $d

Imagine you have a file called ‘data.txt’ in various subdirectories and you want to process all these files:

for f in /home/floki/**/data.txt
	echo $f

Process CSV files

Iterate through a CSV file (does not take quotes into account!):


while IFS='\n' read -r row
	echo -n "Row $rowNo: "
	IFS=',' read -ra columns <<< "$row"
	for column in "${columns[@]}"
		# trim leading and trainling white spaces
		column=`echo $column | xargs`
   		echo -n "[$column] "
	echo ""
done < /home/floki/data.csv

Some other useful parameter substitutions

Use a default value if variable is not set:

HOME=${HOME-'/home/floki'} # HOME: /home/floki

String substitution, first occurrence:

dir=${HOME/floki/ragnar/} # dir: /home/ragnar

String substitution, global:

nonsense=${HOME//o/X/} # nonsense: /hXme/flXki

Substring removal


# Remove from beginning of the string

file=${fullPath##*/}  # file: test.txt`

ext=${file##*.}  # ext: txt

# Remove from the end of the string

name=${file%.*}  # name: test

path=${fullPath%/*}  # path: /home/floki

There are loads of amazing things you can do with Shell Parameter Expansion.

Jul 23, 2018 - Useful shell commands (1)


Useful shell commands (1)

A selection of useful shell commands

Find a program which blocks a certain port

For example listening on port 8080.

lsof -i :8080

Find a file which contains a certain string

For example find all java, xml and properties files which contain a the string myapp.db.user .

grep --include=\*.{java,xml,properties} -rnw "myapp.db.user" .

Check log files

Find specific entries using grep

You only want to see the errors. Grep for ‘error’ ignoring the case. And also add the 5 lines before and the 2 lines after the matching line, because they contain useful information.

grep -i -B 5 -A 2 "error" XYZ.log

Find specific entries using awk

You’re only interested in the 4th and 7th entry (word) of lines containing the string ‘Error’, e. g. because $4 is a filename and $7 is the error code.

cat XYZ.log | awk '/Error/ {print $4 $7}'

Image the log file contains entries like “… Uploaded Files: 5 ….” and you want to sum these figures up to get a total number of uploaded files:

cat XYZ.log | awk '/Uploaded/ {print $5}' | paste -sd+ | bc

SSH port forwarding


Imagine you have a postgres server running on your machine which only listens on localhost port 5432.

But from you desktop machine you want to use a GUI tool like pgAdmin.

As little extra challenge, the SSH server on the remote server doesn’t even listen on the default port 22, but on port 2222.

Solution: Connect to the database server via SSH and use local port forwarding, forwarding your local port 5555 to the server’s port 5432 . Then point pgAdmin to localhost:5555

ssh -L 5555:localhost:5432 user@ -p 2222

A slightly different case:

Image your database server runs on which is not directly accessible. But you can SSH into and this server can access

In that case simply forward your local port to

ssh -L 5555: user@ -p 2222


You have a postgres database running on your local machine (port 5432) and you want to test a program which expects the database running on port 1234.

Open a remote tunnel mapping the port 1234 on the remote machine to your local port 5432.

ssh -R 1234:localhost:5432 user@ 

Dynamic (SOCKS)

If you want to tunnel the data from/to your webbrowser through a certain server:

ssh -C -D 1080

This will create a SOCKS server listening on 1080 on your local machine. Simply add that for your webbrowser’s proxy configuration.

For the websites you view via this SOCKS proxy it will appear like you came from

Jul 11, 2018 - Protect your server


First things to do after ssh’ing into a fresh system

For simplicity I just assume a Debian system.

Update the system

apt-get update 
apt-get upgrade

I usually also make sure ‘vim’ is installed: apt-get install vim

Create user


I’ll refer to that user later as [user].

Generate/upload SSH public key

Generate an SSH key pair on your local machine, if you don’t already have one.


Then copy the public key to the server:

ssh-copy-id [user]@[host]

Restrict SSH access


LoginGraceTime 2m
PermitRootLogin no
StrictModes yes
MaxAuthTries 2

# only allow this user to connect
AllowUsers [user]

# only allow public key auth
PubkeyAuthentication yes
PasswordAuthentication no

Setup a firewall

Using ‘ufw’ (easier)

apt-get install ufw

ufw default deny incoming
ufw allow ssh

ufw enable

Using ‘iptables’ directly (bit more complex)

Create file ‘/etc/iptables_rules’ with following content

# Generated by iptables-save v1.6.1
:OUTPUT ACCEPT [91:8576]
:SSHATTACK - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m recent --set --name SSH --mask --rsource
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 3 --name SSH --mask --rsource -j SSHATTACK
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A SSHATTACK -j LOG --log-prefix "[iptables] SSH attack" --log-level 3

This will also log (/var/log/syslog) brute force attacks on the SSH port, and block an attacker for 600 seconds after 3 failed login attempts.

Apply the firewall rules: iptables-restore /etc/iptables_rules

Enable /etc/rc.local functionality again and load iptables rules on reboot

Create /etc/rc.local

#!/bin/sh -e

iptables-restore /etc/iptables_rules

exit 0

systemctl start rc-local