Extracting and renaming files from zip archives

I had a pile of zip files all containing different files, but also containing a file called preview.png that I wanted to extract and rename. Since the names of the zip are descriptive I wrote a bash script to loop over the files, extract the preview file and rename it. The interesting bit is stripping the .zip extension.


for F in $FILES  
  echo "Processing $F file..."
  unzip "$F" preview.png
  FN=${F%.zip} # strip .zip
  mv preview.png "${FN##*/}.png"

For tar or gzipped tar the following will work in place of the 'unzip' line above:

tar -xf "$F" path/to/preview.png # extract from tar  
tar -zxf "$F" path/to/preview.png # extract from tar.gz  

SSH config

I first encountered the SSH config when I had to set up SSH for two BitBucket accounts. One work, and one personal. Requiring separate login credentials, setting the remote on Git repositories to ssh://git@bitbucket.org/... wasn't going to work.

Each BitBucket account requires a different public SSH key so for two accounts I needed to generate two sets of keys. You'll likely already have a keyset called id_rsa so call the new one something else:

$ ssh-keygen 
Generating public/private rsa key pair.  
Enter file in which to save the key (/Users/you/.ssh/id_rsa): bb-personal  

Now in your ~/.ssh/ directory create a new file simply called config, for my example it looks like this:

Host bitbucket-work  
  User git
  Hostname bitbucket.org
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/bb_work

Host bitbucket-personal  
  User git
  Hostname bitbucket.org
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/bb_personal

Now instead of bitbucket.org in my git remote URIs I substitute the relevant Host from my config:


Of course, you need to ensure the public key is configured over on BitBucket, GitHub etc.

To use config with your own remote machines is just as easy. You can use any of the keys you already have on your system, or create a new one as shown above.

If it doesn't exist, create a file in the ~/.ssh/ directory on your remote machine called authorized_keys and copy the public key in to it, each one on a new line. Never the private key!

Now in your config file on your local machine create a new host for your remote machine:

Host me-production  
  User lewis
  Hostname <your host or ip>
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_rsa

Strictly speaking, I don't think the User line is necessary.

Now all I need to type in the terminal to ssh in to my production box is:

$ ssh me-production

Persistent NodeJS apps on restart

As far as I'm concerned the only two decent solutions to keeping NodeJS apps running properly are PM2 and Forever. The each have their strengths and weaknesses which is a matter for another post.

I recently found myself in a position where I had to use both. I had an application I'd written that I wanted to run in Node 4.x.x and a Ghost install that insists on Node 0.10.40. I use NVM to manage different versions on the same machine and found it best to use PM2 for one app and Forever for the other. Not very elegant, but it works and is holding up.

After running for a few days I patched the server OS and it dawned on me that if I restarted the server, I'd have to manually restart both Forever and PM2. So I present the method I used to have my apps launch properly using SystemV init system. This should be fairly easy to port to other init systems.

First thing to do is create new startup and shutdown scripts:

touch /var/www/startup.sh  
touch /var/www/shutdown.sh  

The first line in my startup.sh 'sources' the nvm script, otherwise bash complains it can't find it. So here is the startup script:

. ~/.nvm/nvm.sh
export NODE_ENV=whatever_this_should_be  
nvm use 4.2.1  
cd /var/www/webapp-one/  
pm2 start index.js  
nvm use 0.10.40  
cd /var/www/webapp-two/  
forever start index.js  

Hopefully the above is self-explanatory, it's just a series of commands to switch to the right Node install and get things up and running.

It's also important to have an elegant shutdown script, so here's mine:

. ~/.nvm/nvm.sh
nvm use 4.2.1  
cd /var/www/webapp-one/  
pm2 kill  
nvm use 0.10.40  
cd /var/www/webapp-two/  
forever stopall  

As before, we still need to 'source' nvm, then again it's just a series of commands to elegant shutdown.

Make both of these files executable:

chmod +x /var/www/startup.sh  
chmod +x /var/www/shutdown.sh  

It's a good idea here to test they work, hopefully you'll know by looking at the output:

cd /var/www  

Now we can get init.d to manage the startup and shutdown for us. So in the /etc/init.d directory create a new file:

sudo touch /etc/init.d/myapps  

Init scripts follow a certain format, the following is for SystemV:

#! /bin/sh
case "$1" in  
    su myuser -c $WWW_DIR/startup.sh
    su myuser -c $WWW_DIR/shutdown.sh
    sleep 3
    su myuser -c $WWW_DIR/shutdown.sh
    sleep 3
    su myuser -c $WWW_DIR/startup.sh
    echo "Usage: websites {start|stop|restart}" >&2
    exit 3

In the above file it should obvious what each section does. Pre-pending our bash scripts with su myuser -c tells the script to run the command as a certain user, you of course will pick whatever your username is. sleep simply pauses the script just to let things settle.

Once you've saved it, make it executable for all users, and test it out:

sudo chmod a+x /etc/init.d/myapps

cd /etc/init.d  
sudo ./myapps start # to test startup  
sudo ./myapps stop  # to test shutdown  
sudo ./myapps restart # to test restart  

If all is good you can update SystemV to run at system startup and shutdown:

sudo update-rc.d myapps defaults  

Now you can manage your apps like any other service.

sudo service myapps start  
sudo service myapps stop  
sudo service myapps restart  

Personally I prefer to have one service to handle everything. If I need to manage an individual app I'll do that manually, but you could create separate init.d scripts for all your separate apps.