Repeating tasks with Watch

Today I had to call an API over one-thousand two-hundred times that would in turn insert rows in to a database.

I wrote the code to load the data, transform it as required, then post it to the API. It would do a handful of rows then the database server would fall over.

I could've thrown more resources at the database server but that seemed like unnecessary cost. I tweaked settings, added swap space, but nothing worked. I even tried adding a delay to the code, but I wrote it in a asynchronous way and it was becoming a headache.

Then it struck me. Alter the code so it only reads one row, transforms it, and posts it to the API. Then run the program every few seconds. Reducing the code was trivial, it just meant pulling out any loops. I had to add a way to mark each row as complete so it wouldn't keep processing the same data, but that was easy enough.

Initially I planned to use cron to repeat the program, but it's a one-off job. I only need it to process one set of rows then I won't need the cronjob anymore.

Enter watch. Watch is a unix/linux command to repeat a command, such as calling a program, at a given delay. It shows the first page of output on each run so I could see any errors or returned data. Since I wrote my program in NodeJS and wanted it to run every five seconds, I just called the following:

watch -n 5 "node process.js"  
Last updated: November 19, 2018

Raspberry Pi Waveshare TFT Trouble

In order to monitor a small Raspberry Pi 3 running as a server I bought a Waveshare 3.5" TFT touchscreen which connects through the GPIO pins.

I followed the instructions that came with it and all went well. It booted in to X and I could use the touchscreen without any further configuration. I don't need X as I'm likely just to run htop most of the time. This is where the problems started.

The Pi would show the boot process on the TFT then suddenly stop. After waiting a while I hit some keys and the screen moved, but didn't change. Seems the boot process was finished but I couldn't see it.

I tried everything I could think of. It wasn't until I hit ctrl-alt-f2 and saw a login prompt that things started to click. After much Googling I found myself looking at the /etc/rc.local file which contained the following conspicuous line:

fbcp &  

It turns out this command copies the framebuffer with a small delay (~25ms). I remmed out this line, rebooted and all was well.

I suspect if I wanted to load X I might need this line back, but for now I'm happy.

Javascript Media Queries

Javascript media queries used to be a pain in the butt. I won't go in to why because thanks to window.matchMedia it's now very easy.

window.matchmedia has a matches property which returns a boolean value, so to run it once just use it something like and if statement:

if(window.matchMedia("(min-width: 500px)").matches) {  
  /* the viewport is at least 500 pixels wide */
} else {
  /* the viewport is less than 500 pixels wide */

It's also possible to add a listener to call a function to run some code whenever the query return value changes:

if(matchMedia){ // Only run if matchMedia is supported  
  const media_query = window.matchMedia("(max-width : 500px)"); // Create media query
  media_query.addListener(widthChange); // Optional listener with function to run
  widthChange(media_query); // Initial run

function widthChange(media_query){  
  if(media_query.matches){ // If the media query resolves true
    // window width is under 500px wide
  } else {
    // window width is more than 500px wide

Paintings of the Weimar Republic

Painting of a Berlin street scene

Gustav Wunderwald's wonderful paintings of Weimar Berlin.

What to do about bookmarks

Browser bookmarks frustrate me. I need them for personal and professional use. Yet, like email, bookmark management just seems like a silo wherein I pile new bookmarks on top of old. I try to remember what concise list of tags I used in the past. But it's all a guessing game. Eventually the number becomes so great I export them all to an XML file and put that in a different kind of silo on a hard drive.

My bookmarks, and I suspect most folks', can be split in to three groups. The stuff we use daily, the stuff we're currently using but one day (work projects for example), and the stuff we want to come back to someday.

The first and second groups are, IMHO, what the bookmarks store in a browser is for. I want quick access to these. It's important to keep the two separate and clear out the transient stuff appropriately. When I finish a job I should collate all the bookmarks I've stored for that job in a text file and add it to the git repository and tarball for the job.

I've helped reduce the third group by embracing Pocket. But I still have bookmark folders full of long-forgotten links.

I tried using Pinboard and Delicious but that's even worse. They're like cold storage. I put stuff there and fool myself it'll be there when I want it. But it's easier just to Google something than to try and search these services.

So what to do? I don't know. I like the idea of keeping all those third-group links in appropriately named text files. But will I bother to keep them updated? Will I bother to look in them when I need a link?

What I'm more tempted to do is distill anything I learn from a link in to a document. A tutorial in my own words. Anything else can go in Pocket or simply be read and discarded.