Extracting and renaming files from zip archives

I had a pile of zip files all containing different files, but also containing a file called preview.png that I wanted to extract and rename. Since the names of the zip are descriptive I wrote a bash script to loop over the files, extract the preview file and rename it. The interesting bit is stripping the .zip extension.


for F in $FILES  
  echo "Processing $F file..."
  unzip "$F" preview.png
  FN=${F%.zip} # strip .zip
  mv preview.png "${FN##*/}.png"

For tar or gzipped tar the following will work in place of the 'unzip' line above:

tar -xf "$F" path/to/preview.png # extract from tar  
tar -zxf "$F" path/to/preview.png # extract from tar.gz  

GeoIP blocking and redirecting on Apache

Recently I was asked to configure a WordPress install to block any traffic from Russia, and redirect any traffic from the USA to another US-centric site.

First thing to do is install mod-geoip for Apache (in my case on Ubuntu):

sudo apt install libapache2-mod-geoip  

Next uncomment the GeoIP database in the GeoIP conf file (/etc/apache2/mods-available/geoip.conf):

<IfModule mod_geoip.c>  
  GeoIPEnable On
  GeoIPDBFile /usr/share/GeoIP/GeoIP.dat

Restart Apache:

sudo systemctl restart apache2  

With GeoIP enabled we can apply our block and redirect in .htaccess

GeoIPEnable On

# deny for Russia

# redirect for USA
RewriteEngine on  
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(US)$  
RewriteRule ^(.*)$ https://usa-site.com/$1 [L]  

A list of ISO Alpha-2 country codes can be found on WikiPedia, and many other places online.

Cache control using a map in Nginx

Google's delightful Lighthouse utility moaned at me about cache-control. For that particular site I use Nginx as a reverse proxy to NodeJS.

Nginx allows us to set a map for filetypes and apply that map to one or more server definitions. To define the map create the following near the top of your configuration file:

map $sent_http_content_type $expires {  
    default                    off;
    text/html                  epoch;
    text/css                   max;
    application/javascript     max;
    ~image/                    max;

In the above example, epoch sets the expiry time of all html to 1st January 1970 (which we all know was a Thursday). max on the other hand sets the expiry time to 31st December 2037 (a Friday). Other options off or a specific time.

The last entry, ~image will apply to rule to any content type beginning with image/, for example image/jpeg or image/png.

To apply the map add the following line inside the server block:

server {  

    expires $expires;

Plex transcoding using a RAM disk

My home server is a server in software only. Hardware wise it's an old Dell Inspiron desktop which was my main machine for a good five years. It's a core 2 duo with 6gb of RAM and on-board graphics.

I run the Plex media server on it, as well as some other services, all on Ubuntu Server 18.04. Without a fancy GPU, Plex does all the transcoding of media files using software, something which can tax an old system like mine given I encode to HEVC to save space.

Plex allows us to specify a directory where the transcoding data is written as it is used. It doesn't need to be very big. It occurred to me I could use a RAM disk for this and make it nice and snappy. Not to mention saving wear on the drive.

To create the drive first create a directory, then mount it as tmpfs

sudo mkdir /mnt/ramdisk  
sudo mount -t tmpfs -o size=512M tmpfs /mnt/ramdisk  

It's important the location is owned by root as we want to remount it at reboot. The default permissions for tmpfs make it writable by everyone, but just in case it's not chmod it to 1777.

To ensure the drive is remounted at boot add the following to the /etc/fstab file:

tmpfs /mnt/ramdisk tmpfs rw,size=512M 0 0  

We can use the df -h command to view the usage. The following shows my 1gb RAM disk with Plex using 67mb to transcode an SD video of a Porky Pig cartoon encoded in HEVC:

Filesystem      Size  Used Avail Use% Mounted on  
udev            2.9G     0  2.9G   0% /dev  
tmpfs           594M   33M  561M   6% /run  
/dev/sdc1        50G  9.3G   38G  20% /
tmpfs           2.9G   20K  2.9G   1% /dev/shm  
tmpfs           5.0M     0  5.0M   0% /run/lock  
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup  
/dev/sdb1       1.8T  1.1T  664G  62% /mnt/media
/dev/sda1       688G  282G  372G  44% /mnt/tor
cgmfs           100K     0  100K   0% /run/cgmanager/fs  
tmpfs           594M     0  594M   0% /run/user/1000  
tmpfs           1.0G   67M  958M   7% /mnt/ramdisk  

Repeating tasks with Watch

Today I had to call an API over one-thousand two-hundred times that would in turn insert rows in to a database.

I wrote the code to load the data, transform it as required, then post it to the API. It would do a handful of rows then the database server would fall over.

I could've thrown more resources at the database server but that seemed like unnecessary cost. I tweaked settings, added swap space, but nothing worked. I even tried adding a delay to the code, but I wrote it in a asynchronous way and it was becoming a headache.

Then it struck me. Alter the code so it only reads one row, transforms it, and posts it to the API. Then run the program every few seconds. Reducing the code was trivial, it just meant pulling out any loops. I had to add a way to mark each row as complete so it wouldn't keep processing the same data, but that was easy enough.

Initially I planned to use cron to repeat the program, but it's a one-off job. I only need it to process one set of rows then I won't need the cronjob anymore.

Enter watch. Watch is a unix/linux command to repeat a command, such as calling a program, at a given delay. It shows the first page of output on each run so I could see any errors or returned data. Since I wrote my program in NodeJS and wanted it to run every five seconds, I just called the following:

watch -n 5 "node process.js"