Repeating tasks with Watch

Today I had to call an API over one-thousand two-hundred times that would in turn insert rows in to a database.

I wrote the code to load the data, transform it as required, then post it to the API. It would do a handful of rows then the database server would fall over.

I could've thrown more resources at the database server but that seemed like unnecessary cost. I tweaked settings, added swap space, but nothing worked. I even tried adding a delay to the code, but I wrote it in a asynchronous way and it was becoming a headache.

Then it struck me. Alter the code so it only reads one row, transforms it, and posts it to the API. Then run the program every few seconds. Reducing the code was trivial, it just meant pulling out any loops. I had to add a way to mark each row as complete so it wouldn't keep processing the same data, but that was easy enough.

Initially I planned to use cron to repeat the program, but it's a one-off job. I only need it to process one set of rows then I won't need the cronjob anymore.

Enter watch. Watch is a unix/linux command to repeat a command, such as calling a program, at a given delay. It shows the first page of output on each run so I could see any errors or returned data. Since I wrote my program in NodeJS and wanted it to run every five seconds, I just called the following:

watch -n 5 "node process.js"