Digital housekeeping

I'm finally getting my house in order. That is my digital house. I've never been very strict about files and folders though I try to keep on top of it. Recently I've felt a new vigour to get things in order.

Aside from a system SSD, where I do store some current work on my desktop, I try to keep my system drive free of anything but OS and software.

Along side the system drive is a 1tb WD Red that is my main drive. This is backed up nightly to a mirrored NAS giving me three local copies of these files. I'm also in the process of backing this up to a new Backblaze account, but the 300gb are taking a while to do.

Finally in my main system is a 640gb drive that I use as a scratch drive. There is stuff on there I wouldn't like to lose but it wouldn't be the end of the world if I did.

My small home server contains a small SSD for OS and software. Also in there are a 2.5in 750gb drive that I use for downloading and general work with a large amount of files. Also in here is my Media drive, a 2tb WD drive that contains films, tv shows, audiobooks etc.

The media drive is manually backed up to an external 2tb Buffalo drive.

In addition to the above I have a portable 2tb drive that lives in my bag. Another 2tb external that used to be my media drive. Two 1tb external Buffalo drives, and a 500gb external Buffalo drive.

So what am I storing where...

Documents and files (1tb WD Red)

All my JPG photos are further compressed using imagemin, in particular the jpeg-recompress binary.

Currently this drive contains about 300gb of data with about 650gb free, so should suffice for a few years to come.

Third drive (640gb)

At the time of writing the folder structure is:

As I said above, it wouldn't be the end of the world if this drive failed. But it's showing no signs and I plan to do something with a lot of the stuff on here soon.

Server - Media drive (2tb)

Back in October 2015 I decided to re-encode all video files to x265 and standardise audio formats and subtitle files. It's taken a long time to do this and I'm still doing it but the bulk of it is done. It's cut the space consumed by these files in half.

This drive is mirrored manually once every couple of weeks to an external 2tb drive.

2 x 1tb buffalo external

This is mirrored on two external drives. Just in case.

Reading List #1

Five articles I've read and enjoyed recently.

The Magic of Untidiness

I'm a very tidy, ordered person, trapped in the body of a clutterer.

Leviathan

David Sedaris on a family trip to the beach.

The Beginner's Guide to Getting Better Sleep

A life-long challenge for me.

I got destroyed at a web-surfing competition

Race across the internet from point A to B using only a one-button mouse: no keyboard, no search, no URL bar, no back button.

The New Essayists, or the Decline of a Form?

David Sedaris and a literary version of reality TV.

Neat Vanilla JS on HackerNews

I recently saw this video by Gordon Zhu highlighting the small but extraodinarily clever JavaScript on the Hacker News website.

I felt it would help me if I simplified it and broke down the steps for future reference. I'm using Paul Buchheit's code verbatim, I hope he doesn't mind, full credit to him.

What does this achieve?

This code is called when an anchor (a link) is clicked to vote up an item on Hacker News. It casts the vote and hides the vote up link so it can't be clicked again.

HTML

The markup for this is REALLY simple. It's a perfect example of progressive enhancement. The full URL is present in the HREF so if JavaScript is turned off the onclick is never called but the vote is still cast.

<a id="up_11659026" href="vote?for=11659026&dir=up" onclick="return vote(this);">Vote Up</a>

JavaScript

No JQuery here! This is concise and elegant vanilla JS. The only JS on the site is following two functions, with the latter calling the former.

function hide(id){
  var el = document.getElementById(id);
  if (el) { el.style.visibility = 'hidden'; }
}
function vote(node){
  var v = node.id.split(/_/);
  var item = v[1];
  hide('up_'   + item);
  hide('down_' + item);
  var ping = new Image();
  ping.src = node.href;
  return false;
}

How does it work

  1. When the link is clicked the vote() function is called with the anchor (<a>) element passed as this
  2. The id, up_11659026, is split at the underscore and the parts passed to an array, v
  3. A new var, item, is given the value at v[1], in this case 11659026
  4. Two calls are made to the hide() function, one with the value up_11659026, and one with the value down_11659026
  5. The hide() function looks for an element with the passed id, if it finds it, the visibility of the element is set to hidden
  6. A var is created, ping, and is an instance of JavaScript's Image object.
  7. By setting the src of ping to the href of the clicked element, the URL is called (in this case a vote is cast by the back-end)
  8. Nothing is returned or rendered as it's unnecessary.
  9. We return false to stop the URL being followed by the browser
  10. If JavaScript is turned off in the browser, the URL is followed when clicked and the same outcome occurs.

Node on Ubuntu on Vagrant on Hyper-V

The following is how I got NodeJS running on Ubuntu using Vagrant in Windows with Hyper-V.

Why Hyper-V?

Paying for VMWare goes against my open-source leanings, and I'm a lone dev so any expenses have to be seriously considered.

Virtualbox is a great product and I've used it a lot in the past but I never liked the networking stuff it added to my system. I've tried using Vagrant with Virtualbox on Windows before and always come against folder and file permissions and syncing problems. I have problems with Oracle too.

Hyper-V on the other hand is baked natively in to Windows 8.1 and higher (I'm using Windows 10). It's straightforward to enable, configure and plays nicely with the Windows filesystem and networking stack.

Why Ubuntu?

I've used it more than any other Linux distribution. I'm very familiar with configuring it and running it in production. I dare say the steps I outline below are fairly 'box' agnostic though.

Step 1. Install Vagrant

Head on over to https://www.vagrantup.com/downloads.html to download the installer, run it and wait for it to finish.

Step 2. Enable Hyper-V

Go to Uninstall or change a program, you can find this in the toolbar on This PC, via the control panel, or just search for it.

Next click on Turn Windows features on or off on the left side of the screen. And make sure Hyper-V is checked.

Enabled Hyper-V

You might have to reboot to fully enable Hyper-V but once it's enabled you can check it under the performance tab in the task scheduler

Check Hyper-V is enabled

Step 3. Create Hyper-V network switch

This step is really important. If you do not do this, Vagrant will not be able to connect to the box. So hit your Start button and search for Hyper-V Manager. Once in, find Virtual Switch Manager...

Hyper-V Virtual Switch Manager link

In the 'Virtual Switch Manager' select New virtual network switch, now you have three choices:

New virtual network switch

External creates a network switch that is connected to your system's physical network. This will allow your Vagrant box to connect to the outside world, and vice versa.

Internal creates a network switch that allows your host system and the virtual machines in Hyper-V to talk to each other. If you select this option, your Vagrant will not have internet access.

Private creates a network switch that can only be used by the virtual machines. This is useless for Vagrant.

I suggest using External as it means I can use apt-get etc. So select External and hit Create Virtual Switch. All you need do now is give your virtual switch a name. Hit 'OK' and close the Hyper-V Manager.

Step 4. The Vagrantfile

Now we have the host operating system set up and Vagrant installed it's time to actually create a Vagrant box.

In the directory your project with be in type the following command:

vagrant init

This will create a single file called Vagrantfile in your directory. This file is all you need and is where you'll put your instructions for setting up your Vagrant box.

Ignoring all the comments and remmed out statements, the basic Vagrantfile looks like this:

Vagrant.configure(2) do |config|
  config.vm.box = "base"
end

That's it. The base box is the default and is all well and good, but I want 64bit Ubuntu. So change "base" to "hashicorp/precise64". If you want a different base system, you can find more pre-built boxes at https://atlas.hashicorp.com/boxes/search

Next we have to tell Vagrant to use Hyper-V as I think it defaults to 'Virtualbox', so add the following line:

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.provider "hyperv"
end

Finally, we want to make sure the Vagrant box has access to the public network (the internet) so we can grab apt packages and the like. So add the following line:

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.provider "hyperv"
  config.vm.network "public_network"
end

This is the absolute basics we need. Save your Vagrantfile and we're ready to fire it up.

Step 5. vagrant up

One quirk of Vagrant on Hyper-V is that it must be run as Administrator. So whether you're using Command Prompt, Powershell, Cygwin, or Git Bash, you need to make sure you run it as Administrator.

So to get things going open your CLI window, navigate to your project folder and type:

vagrant up

If you see an error regarding the provider, you may need to force the use of Hyper-V:

vagrant up --provider=hyperv

If all goes well, you should see something along the lines of the following. It might take a while as it has to download the virtual hard drive for the Ubuntu version we selected. Also, because we're using Linux with Windows, Vagrant needs to set up a Samba share. So you'll need to enter your Windows credentials.

$ vagrant up
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Importing a Hyper-V instance
    default: Cloning virtual hard drive...
    default: Creating and registering the VM...
    default: Successfully imported a VM with name: precise64
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
    default: Timeout: 120 seconds
    default: IP: 192.168.1.174
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 192.168.1.174:22
    default: SSH username: vagrant
    default: SSH auth method: password
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
    default: You will be asked for the username and password to use for the SMB
    default: folders shortly. Please use the proper username/password of your
    default: Windows account.
    default:
    default: Username: xxxxx
    default: Password (will be hidden):
    default: Password (will be hidden): xxxxxxxx
==> default: Mounting SMB shared folders...
    default: C:/Users/lewis/Desktop/vagrant test => /vagrant

If you see something like above then everything ran fine. You can SSH in to your box by typing vagrant ssh

Some useful Vagrant commands are:

vagrant upCreate a box or start from halt
vagrant haltGraceful shutdown
vagrant destroyRemove the box
vagrant suspendPause box at exact state
vagrant resumeResume from suspend
vagrant reloadReboot, maybe after a config change
vagrant provisionRe-run the provisioning stuff
vagrant sshSSH in to your box

More can be found on the Vagrant website

Step 6. Provisioning and NodeJS

Every time you create a box from a Vagrantfile or vagrant up after a vagrant destroy, Vagrant will create your box from scratch. While we could install all the software we need each time, it makes sense to tell Vagrant to do it for us. This is known as provisioning.

To get started with provisioning NodeJS, create a new file called bootstrap.sh. This is a bash script where we'll put in the commands we need to run. I'm going to install NVM, the node version manager. This is because NodeJS release new versions very fast so I'm happy to actually select my NodeJS version manually.

#!/usr/bin/env bash

wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash

Now in your Vagrantfile add the following line. The important part here is at the end, priveleged: false. By default provisioning scripts run as sudo but we want NVM installed as the vagrant user.

config.vm.provision :shell, path: "bootstrap.sh", privileged: false

So now your whole Vagrantfile should look like this:

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.provider "hyperv"
  config.vm.network "public_network"
  config.vm.provision :shell, path: "bootstrap.sh", privileged: false
end

In your bootstrap.sh you would also put anything else you want run automatically. Such as setting environment variables, pulling from a remote repository, or even installing a database system like Redis.

All done

This is just basic steps to get NodeJS running in Ubuntu with Vagrant using Hyper-V on Windows. There is a lot more to Vagrant than I can cover here, so I suggest you start with their Getting Started guide.

A Node crisis of confidence

After over ten years of PHP, I've spent the last couple of years almost exclusively developing using Node.JS. I consider myself pretty proficient. Happy with prototypal inheritance, Promises, and much of the new ES6 stuff.

However, today I started to get the feeling that I'd been doing something wrong for a long time. It concerns the sharing of resources across modules.

The question being, should a shared resource be required once and passed to modules, or can we rely on Node's caching (and pseudo-singleton implementation) to prevent duplicating resources. Consider the following two approaches:

One

// app.js
require('db');
require('module_one')(db);
require('module_two')(db);

// module_one.js
module.exports = function(db){
  return {
    func_one : function(){
      ...
    }
  }
}

// module_two.js
module.exports = function(db){
  return {
    func_two : function(){
      ...
    }
  }
}

Two

// app.js
require('module_one');
require('module_two');

// module_one.js
require('db');
module.exports = {
  func_one : function(){
    ...
  }
}

// module_two.js
require('db');
module.exports = {
  func_two : function(){
    ...
  }
}

In the first approach, only one instance of db is ever created and used. This is explicitly clear. Though it does introduce dependency injection. The module itself cannot stand alone.

In the second approach, the db module is loaded twice, though because it's cached by Node, the second time gets the same instance as the first time. But the node developers acknowledge this is not guaranteed. If this fails, memory consumption could shoot up and database connections could grind to a halt.

I much prefer the second approach, the one I've used for two years without incident. But now I'm concerned it may be the wrong approach for larger projects.

If you've any thoughts, contact me on Twitter.

Update

After reading Liam Kaufman's article, How AngularJS Made Me a Better Node.js Developer and this thread on Stack Overflow I've come to the conclusion that the answer to the above is, it depends. Both approaches are equally valid and have different use-cases. I'd still be keen to see if there is any performance benefit to one approach over the other.