Aug 21, 2020 - More Unix tools

Comments

More Unix tools

Haven’t posted anything in ages, just too busy with various things… Anyway, here I’m just going to paste a few usage examples of some more very useful standard unix tools.

Awk

Often you have a lot of useful information encoded in file and directory names. In the past I used cut, tr, etc. to extract this information. This can sometimes get quite awkward. I knew there’s a tool called awk, but I never really bothered to use it. Until recently :-) It’s actually quite easy and very useful. Here’s an example:

Imagine you organized your holiday photos like this

/mnt/photos/2020/tenerife/DSC001.jpg
/mnt/photos/2020/tenerife/DSC002.jpg
/mnt/photos/2020/tenerife/DSC003.jpg
...

Now lets say you want to create a CSV file with an inventory of your photos:

cd /mnt/photos
echo "Year,Location,Filename" > ~/my_photos.csv
find * -type f | awk 'BEGIN { FS = "/" } ; {print $1","$2","$3}' >> ~/my_photos.csv

Note: This use case could be handled in various, probably simpler ways, however this hopefully demonstrates how awk works. As you can guess awk also quite handy to extract information from CSV files.

Find

I used it already in previous section, but find is also useful to process files serially. Example, calculate a checksum for all zip files:

find * -iname "*.zip" -exec sha1sum {} \; >> ~/checksums.sha1

Note: {} is substituted for every file find finds. The command which is executed has to be terminated with \;.

Parallel

Not really a standard tool, but very useful if you want to make most of your multicore CPU! See GNU Parallel

Example: We’ll do the same, calculate the checksums of all zip files:

# First get all absolute paths to the zip files and put them in a file:
find * -iname "*.zip" -exec readlink -f {} \; >> ~/zip_files.txt

# Process them in 5 parallel threads:
parallel -a ~/zip_files.txt --eta -j5 --joblog log.txt --delay 2 -k sha1sum {} >> ~/checksums.sha1

Notes: With parallel you don’t need the \;. Please ignore that this example doesn’t make much sense, because shasum1 is so fast you wouldn’t use parallel for that. But it shows a few options which can be very useful in other cases. --eta gives you some progress information. -j5 means use 5 threads. --joblog obviously save the logs. --delay delay the start of each job by 2 seconds. This option can be very important. For example if you kick off a process which right at the start hits a database (or other limited resource) very hard, you don’t want 5, 10, or whatever jobs to do that exactly at the same time. Finally -k which ensures that the output from each job is written in the same order as the input. Without this option the output order would be the order in which the jobs finish, which could be pretty random. Often you want to preserve the order so you can easily match up input and output files.

Apr 13, 2020 - Create an encrypted /home partition

Comments

Create an encrypted /home partition

In my previous Artix post I mentioned that I had trouble creating an encrypted /home partition during installation. I could do that, but then the system wouldn’t boot. So I went for a quite basic installation and set up things afterwards, among others the encrypted home partition.

Although at installation stage I already created an empty partition which should later be used for /home, in my case that was /dev/sda3.

So given you have an empty partition /dev/sda3 which you want to use for you /home: First set it up for encryption: (you might have to install cryptsetup first)

cryptsetup -y -v luksFormat /dev/sda3

Open the encrypted partition and create a file system:

cryptsetup luksOpen /dev/sda3 home

# Check if that worked, you should see a 'home' entry there:
ls -lh /dev/mapper/

# create file system:
mkfs.ext4 -m 1 /dev/mapper/home

# close the device again:
cryptsetup close home

Add entry to crypttab:

echo "home	/dev/sda3	none	luks" >> /etc/crypttab

Create a temporary mount directory and fstab entry to test if you can mount the partition at boot time.

mkdir /mnt/tmp

echo "/dev/mapper/home	/mnt/tmp	ext4	defaults,noatime	0	1" >> /etc/fstab

Reboot. You should be asked about your encryption credentials. When the system has booted up again, you should see it mounted as /mnt/tmp.

If that worked, then copy your old /home over to the encrypted partition:

cp -a /home/* /mnt/tmp/

# double check again if everything is ok, then
rm -rf /home/*

Finally adjust the fstab accordingly:

/dev/mapper/home	/home	ext4	defaults,noatime	0	1

Reboot.

Tip

If you haven’t dedicated a whole partition for this purpose and/or you just want to have an additional encrypted device, then you can simple use a file instead of a parition for that. For example you want something like /home/user/encrypted as place for sensitive data, not a whole parition.

Then you could do:

mkdir /home/user/encrypted

# create an empty 25GB file (take care of this file, it will
# hold all the encrypted data, do not delete accidentaly!)
dd if=/dev/zero of=/home/user/encrypted.luks bs=1G count=25

cryptsetup -y -v luksFormat /home/user/encrypted.luks

cryptsetup luksOpen /home/user/encrypted.luks encrypted

mkfs.ext4 -m 1 /dev/mapper/encrypted

cryptsetup close encrypted

echo "encrypted	/home/user/encrypted.luks	none	luks" >> /etc/crypttab

echo "/dev/mapper/encrypted	/home/user/encrypted	ext4	defaults,noatime	0	1" >> /etc/fstab

Apr 10, 2020 - Manage a blog with Gitlab

Comments

Manage a blog with Gitlab

Thought it’s actually time to write something about how this blog is built. All posts are written in Markdown language, which is then transformed by Jekyll into plain static HTML.

The Jekyll files and posts are all in a (private) Gitlab repository. Before I write a new entry I create a new branch, then commit the new post and changes to the branch. When I’m finished I create a ‘Merge request’ and merge it. The merge then triggers a pipeline job which builts the actual website using Jekyll. This is done by a .gitlab-ci.yml file in the root directory of the repository:

image: ruby:2.3

variables:
  JEKYLL_ENV: production

before_script:
- bundle install

buildSite:
  stage: build
  script:
  - ./createTagPages.sh
  - bundle exec jekyll build -d _site
  artifacts:
    paths:
    - _site
  only:
  - master
  after_script:
    - 'curl https://floki.blog/<SOME_IDENTIFIER>/deploy/$CI_JOB_ID'

It’s probably time to update this file at some point…

Anyway, what it does is: It uses the ruby docker image (Jekyll is written in Ruby). Installs Jekyll and the dependencies (which are specified in a ‘Gemfile’, see at the bottom of the page). Then calls Jekyll to build the website in the _site directory. Ignore the ./createTagPages.sh for now, this only - surprise - updates the tags which you see on the left hand side of the page. When Jekyll’s done, the _site directory is used as ‘artifact’ of the build, i. e. a zip file which contains the built website is created. When the build is finished I just ‘ping’ the webserver hosting it on a specific URL passing on the ID of the build job.

A script on the webserver runs periodically and checks for this call in the nginx logs and if it’s found, it will get the artifact from gitlab and deploy the new website:

#!/bin/bash

deployedId=`cat /var/www/version`

tmp=`cat /var/log/nginx/access.log | awk '/<SOME_IDENTIFIER>\/deploy/ {print $7}'`
jobId=${tmp##*/}
if [[ -n "$jobId" && "$jobId" != "$deployedId" ]]
then
	cd /tmp
	curl -L --header "PRIVATE-TOKEN: <GITLAB_TOKEN>" https://gitlab.com/api/v4/projects/<PROJECT_ID>/jobs/$jobId/artifacts >> artifacts.zip
	unzip artifacts.zip

	cd _site
	chown -R www-data:www-data *
	cp -r * /var/www/html/

	cd ..
	rm -rf _site artifacts.zip

	echo $jobId > /var/www/version

	dt=`date`
	echo "$dt - Deployed $jobId" >> /var/log/blog.log
fi

Finally the Gemfile for the Jekyll installation:

source "https://rubygems.org"

gem "jekyll", "~> 3.8.3"

group :jekyll_plugins do
  gem 'jekyll-paginate'
  gem 'jekyll-sitemap'
  gem 'kramdown'
  gem 'rouge'
end