3

I want to keep a nodejs app running, so I followed this: http://howtonode.org/deploying-node-upstart-monit and installed Monit and Upstart which have been working perfectly for a couple of months. I just added a feature to my app that allows disk writes and am getting permissions errors. I went looking for everything because it seemed the permissions errors shouldn't be the problem. It turns out that the user that I start with Upstart is not actually the user that is running the app!

When I run:

sudo -u deploy NODE_ENV=production node /srv/www/[name of my app]/dist/app.js

Either via Monit/Upstart, or just via SSH, it creates 2 processes. From ps ax: 

2869 ?        Ss     0:00 sudo -u deploy node /srv/www/[name of my app]/dist/app.js
2878 ?        Sl     0:01 node /srv/www/[name of my app]/dist/app.js

When I log in as the deploy user and just run:

NODE_ENV=production node /srv/www/[name of my app]/dist/app.js

There is only 1 process! 

3131 pts/1    Sl+    0:04 node /srv/www/[name of my app]/dist/app.js

This issue is that for some reason, the second process doesn't have the correct permissions to write and seems to be the one doing the work. Everything is perfect when I don't use Upstart including the writes, but I need Upstart/Monit to keep my nodejs app running.

What's the best way to configue Monit/Upstart in this case, then? I want to keep low permissions (hence the deploy user which does not have sudo) and I want the app to start when it crashes.

This is my Upstart script:

start on startup
stop on shutdown

script
    export HOME="/home/deploy"

    echo $$ > /var/run/[name of my app].pid
    exec sudo -u deploy NODE_ENV=production node /srv/www/[name of my app]/dist/app.js >> /srv/www/log/[name of my app].sys.log 2>&1
end script

Anything wrong there?

The tutorial says to use sudo -u user ... in order to run it as the user you want. But that's the thing that's causing the problem.

Thanks!

Paul

paintedbicycle
  • 199
  • 1
  • 3
  • 15
  • have you tried the obvious : manually change your user to "deploy" and just touch a file in that folder? Are you perhaps in a chroot environment and showing us the output not from root but from deploy perspective? – Dennis Nolte Jun 30 '14 at 14:25
  • @DennisNolte Good thought - I was logged in as my own user (not deploy nor root). But when I log in via deploy, the permissions on `locales` is the same (`drwxrwxr-x`) and the permissions on the enclosing folder is `drwxr-xr-x`. So don't think it helps much in this case - I can easily create and delete folders as deploy without `sudo`. – paintedbicycle Jun 30 '14 at 14:44
  • try this in test environment please: http://www.hacksparrow.com/two-common-nodejs-npm-permission-denied-errors-on-linux.html curl http://npmjs.org/install.sh | sudo sh – Dennis Nolte Jun 30 '14 at 15:17
  • @DennisNolte The `deploy` user is the one that I use to install npm packages all the time. It does it fine. In fact the app has been running and working for a month, but this was the first time I've needed the app to be able to make writes. `Deploy` doesn't have sudo access to I don't think I can run that. Anyway, as I said the `deploy` user can use `mkdir` via SSH, but for some reason the app cannot do it. – paintedbicycle Jun 30 '14 at 15:27
  • The two processes is a red herring. E.g. `root 51145 0.0 0.0 2443216 1864 s007 S+ 4:32PM 0:00.01 sudo -u mwagner NODE_ENV=production sleep 600` `mwagner 51137 0.0 0.0 2432764 500 s006 S+ 4:30PM 0:00.00 sleep 100` Run `ps auxw | grep node` and post that output. Edit: yay horrible formatting. – Mark Wagner Jun 30 '14 at 23:35
  • @MarkWagner The relevant output from `ps ax` is in the question. When I run via Upstart or via SSH as above, it get those two processes. When I log in as `deploy` I only get one process and the site works. – paintedbicycle Jun 30 '14 at 23:47
  • I said `ps auxw` not `ps ax`. – Mark Wagner Jun 30 '14 at 23:59
  • @MarkWagner, sorry, my mistake. I didn't even know that existed. Here it is: `root 2871 0.0 0.1 53528 1940 ? Ss 00:00 0:00 sudo -u deploy NODE_ENV=production node /srv/www/[name of my app]/dist/app.js deploy 2880 15.1 5.8 1000652 58908 ? Sl 00:00 0:01 node /srv/www/[name of my app]/dist/app.js paul 2992 0.0 0.0 9452 948 pts/1 S+ 00:00 0:00 grep --color=auto node` – paintedbicycle Jul 01 '14 at 00:01

1 Answers1

3

sudo persists as long as the process is running, so the solution is to use the setuid and setgid stanzas of upstart. The problem with that is that the line that sends the PID of the daemon needs root privileges. Here is a workaround:

[name].conf:

start on runlevel [2345]
stop on runlevel [016]

env HOME=/home/deploy
env NODE_ENV=production

setuid deploy
setgid deploy

exec node /srv/www/[name]/dist/app.js

[name]-pidfile.conf

start on started [name]
stop on stopped [name]

pre-start script
    initctl status name | cut -d" " -f4 > /var/run/[name].pid
end script

post-stop exec rm -f /var/run/[name].pid
CameronNemo
  • 399
  • 1
  • 6
  • Very interesting! I spent hours trying to get `setuid` and `setgid` working but didn't know about the second file. This gets rid of the root process and the app is only running once, under deploy. There still seems to be a difference between this and just logging in as `deploy` to run the app. Why? When Upstart starts the app, the app still doesn't have permission to write, but when I log in as `deploy` and start the app, it does. Running `ps auxw | grep node` I basically get the same output (deploy user running my app) – paintedbicycle Jul 01 '14 at 15:15
  • Well if the file already exists as the deploy user, then you can overwrite it as the same user. The problem is, /var/run is destroyed and recreated on boot, so the permissions fail. – CameronNemo Jul 01 '14 at 16:47
  • Dang, looks like another dead end. Thanks anyway, interesting answer! – paintedbicycle Jul 01 '14 at 17:14
  • On further inspection: when I reboot, the app starts, but doesn't have permissions to write, so my app fails (500 error). When Monit sees this 2 mins later, it restarts the app via Upstart and everything works. It's hacky and means my app is down for longer than I would want, but working. I'd love to get this cleaner. Any ideas? – paintedbicycle Jul 01 '14 at 18:59
  • Does your app write its pid file itself too? Maybe change the pidfile job to `start on starting [name]` and then use `pre-start exec touch /var/run/[name].pid` instead of the script. – CameronNemo Jul 01 '14 at 21:34
  • my app does not write that file, no. I tried this but it made no difference. Can't get my head around why, when starting an app as a user and even inspecting the process, seeing it is that user, that app then cannot do the things that the user when logged in can do. What's going on here? – paintedbicycle Jul 08 '14 at 04:14
  • sorry if this is stupid question, why u need to set `env HOME=/home/deploy` – slier Dec 09 '14 at 16:36
  • @slier I just added it because the OP had an `export HOME=/home/deploy` and I figured his app needed it. – CameronNemo Dec 09 '14 at 22:21