4

I have spent several days on an issue that's incredibly frustrating and I'm straight out of options.

I am currently using a CentOS 6.8 server at work for our website. CentOS 6 (Enterprise Linux 6 or EL6) uses the older init.d model for startup scripts. There is a program my boss requires me to use, the SonicWALL CDP Agent. He has a SonicWALL backup server, and their "agent" software runs in the background to back up specific files and folders to the server. It's actually built upon Acronis TrueImage, but that's another story.

The biggest issue is that their CDP agent uses Adobe AIR. They must have designed it this way to be cross-platform (as they have an installer for Windows/OSX/Linux), but of course Adobe completely stopped supporting AIR on 64-bit Linux some time ago. Their site gives some steps for getting it installed, but that dates back to Red Hat 5.5, not even 6.

I did manage to get AIR to install on EL7 by following the steps for EL6, and just noting where the package names had changed in the repos. Interesting to note that the normal GUI installer (AdobeAIRInstaller.bin) doesn't work and gives some error about "maybe your admin prohibits installs" even when running as root), but the .rpm files work.

I have the most current ones:
adobeair-core-2.6.0-19170.noarch.rpm
adobeair-2.6.0-19170.i686.rpm

Combined with the GTK2 i686 libraries, this actually works.

However, the CDP Agent is more complicated than this. There are two system services that need to run in order for the agent software to actually work (and if you think about it, this would be required for the automatic backups to work anyway.)

Their installer, however, is years old and places the startup scripts in init.d. This is "supposed" to work in EL7, but it doesn't. I've spent hours fiddling with this and it simply does not work.

So basically, there are two binaries that need to run. The commands to start them are: 

/sbin/cdp/cdpagentproxy  
/sbin/cdp/cdpdaemon start

If I open a terminal and run these manually, they work - and I can open the CDP Agent and it works. However, they aren't starting up at boot due to the init.d/systemd incompatibility.

So I made a couple simple "services" and placed them in the right spots. The cdpdaemon.service file, for example:

[Unit]
Description=CDP Daemon Service
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/sbin/cdp/cdpdaemon start

[Install]
WantedBy=multi-user.target

I placed this in /usr/lib/systemd/system/cdpdaemon.service and made a symlink to it in /etc/systemd/system/multi-user.target.wants/cdpdaemon.service.

But here's what happens when I try to check the status:

[root@localhost Desktop]# systemctl status cdpdaemon.service
● cdpdaemon.service - CDP Daemon Service
   Loaded: loaded (/usr/lib/systemd/system/cdpdaemon.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2016-08-21 22:08:13 EDT; 10s ago
      Process: 1596 ExecStart=/sbin/cdp/cdpdaemon start (code=exited, status=0/SUCCESS)
 Main PID: 1633 (code=exited, status=127)

Aug 21 22:08:13 localhost.localdomain systemd[1]: Starting CDP Daemon Service...
Aug 21 22:08:13 localhost.localdomain cdpdaemon[1596]: Starting  process.
Aug 21 22:08:13 localhost.localdomain systemd[1]: Started CDP Daemon Service.
Aug 21 22:08:13 localhost.localdomain systemd[1]: cdpdaemon.service: main process exited, code=exited, status=127/n/a
Aug 21 22:08:13 localhost.localdomain systemd[1]: Unit cdpdaemon.service entered failed state.
Aug 21 22:08:13 localhost.localdomain systemd[1]: cdpdaemon.service failed.

The other one does that same thing. It looks like it tries to run, but then stops and reports an error. If I open a terminal and simply type /sbin/cdp/cdpdaemon start, it works perfectly fine.

Am I doing something wrong here? I don't understand the different types of "target" options nor the "after" and type. (I basically took a working service, copied the text and changed the command)

I suppose another option would be to make a cronjob that runs at a certain time (let's say 11pm, if the backups run at midnight) that runs the two commands, but that seems like the wrong way to go about this.

The program DOES run, I just can't get it to run on its own without me starting the two helper processes.

Edit: I thought I'd provide the install .sh file in case it helps. You can see where it checks what distro you're running and creates the init.d scripts.

http://pastebin.com/FmrZcmcR

GregL
  • 9,030
  • 2
  • 24
  • 35
Danny Forche
  • 91
  • 1
  • 8
  • Try adding `Environment=SYSTEMD_LOG_LEVEL=debug` under `[Service]` – Ryan Babchishin Aug 22 '16 at 04:20
  • That didn't seem to make any difference. The "status" still shows the same thing, unless there's a difference place to check the logs at. – Danny Forche Aug 22 '16 at 04:52
  • I feel your pain... I'm still trying to catch up with systemd. What about running `journalctl -af` while typing to start the service? Does the service need to run as root or do you need to specify a different user by any chance? What if you had systemd run an `strace -f` on it? – Ryan Babchishin Aug 22 '16 at 05:19
  • Strangely, it appears you have a PATH problem. Check the old init script and see if it is doing something odd with the PATH. – Michael Hampton Aug 22 '16 at 05:43
  • OK, so when I try `journalctl -af` I get the same messages as my above post. Starting, main process exited with status 127, entered failed state, and service failed. strace says "command not found" – Danny Forche Aug 22 '16 at 12:55
  • @ Michael Hampton, I just checked PATH on both my CentOS 6 and CentOS 7 machines. They're the same. The only difference is it actually works on 6 and not on 7. I can upload the init scripts if you want, but they're super simple, basically just a quick check of where the binary is (whether in /usr/bin or /sbin) and then it runs them. – Danny Forche Aug 22 '16 at 13:16
  • When you type `/sbin/cdp/cdpagentproxy` or `/sbin/cdp/cdpdaemon start` in a terminal, does the terminal return, or does the command begin to output logs and status information? – GregL Aug 22 '16 at 13:31
  • I know this is a long shot, but maybe give https://github.com/jordansissel/pleaserun a try? Haven't tried it personally, but heard good things about it. – Mugurel Aug 22 '16 at 13:52
  • might it be that youre hitting a SELinux block ? as in, the file is actually there, but selinux denies access ? If thats the case you might look at the suggestion se troubleshooter app gives. – Tuncay Göncüoğlu Aug 22 '16 at 14:17
  • GregL, the terminal returns, but I can tell the process is running. If I type it again, it says "The cdpagentproxy is already running.." however this does not happen with the "service", only when I start it manually. As for SELinux, it's not that as I disabled SELinux entirely. – Danny Forche Aug 22 '16 at 14:36
  • Mugurel, I just tried that. It made a systemd script, but it has the same problem. No errors this time, but it still isn't running at startup. The program also can't detect what type of file it is: `No platform selected. Autodetecting... {:platform=>"systemd", :version=>"219", :level=>:warn}` – Danny Forche Aug 22 '16 at 15:03
  • Do either commands create a PID file? – GregL Aug 23 '16 at 12:03
  • Supposedly, but it isn't running when I check the status. – Danny Forche Aug 23 '16 at 16:31

3 Answers3

0

"Type=fokring" is for services that fork a child and put that into background and themselves exit normally. Looking at your output it feels like your programs do not do that. what happens if you change "Type=forking" to "Type=simple"?

Ektich
  • 81
  • 4
  • Tried that, this is what I get. `[root@localhost ~]# systemctl status cdpagentproxy ● cdpagentproxy.service - CDP Agent Proxy Service Loaded: loaded (/etc/systemd/system/cdpagentproxy.service; disabled; vendor preset: disabled) Active: inactive (dead) Aug 22 18:48:18 localhost.localdomain systemd[1]: Started CDP Agent Proxy Service. Aug 22 18:48:18 localhost.localdomain systemd[1]: Starting CDP Agent Proxy Service... ` – Danny Forche Aug 22 '16 at 22:48
  • And it should go without saying it doesn't actually do anything. The process isn't running. – Danny Forche Aug 22 '16 at 22:49
  • why does it say `/etc/systemd/system/cdpagentproxy.service; disabled` in this case? in your previous posts it was `enabled`. – Ektich Aug 23 '16 at 08:25
  • Oh, it was because I didn't enable it through systemtcl, I just put the .service file back in the folder. So when I enable it, I get the same thing. It says it exited. `Process: 1384 ExecStart=/sbin/cdp/cdpagentproxy (code=exited, status=0/SUCCESS) Main PID: 1384 (code=exited, status=0/SUCCESS)` It's almost as if it doesn't like being run from systemctl. – Danny Forche Aug 23 '16 at 12:55
  • one more question: any chance the cdpagenproxy is actually running? what does 'ps -aux' show? and let's have some consistency as well: in the original question you were showing outputs of `systemctl status cdpdaemon.service`, now you are showing `systemctl status cdpagentproxy.service`. – Ektich Aug 23 '16 at 19:41
  • OK, so if I do ps aux | grep cdp, the only thing I get is the grep result back. Doing ps aux and scrolling through the list, I didn't see anything either. Both services are the same, just two different binaries, but just for kicks I'll show you the cdpdaemon one. http://i.imgur.com/YNF0fV7.png – Danny Forche Aug 24 '16 at 16:23
  • Honestly, for now I'm thinking the best way to handle this is to just make a cronjob that runs every 5 minutes, checking if the process is running and starting it if not. But I don't know how to check that, I know you can do ps -aux | grep cdp, but that isn't something you could easily script. – Danny Forche Aug 24 '16 at 16:25
  • There is a difference between the situation where you have Type=forking and Type=simple. I didn't notice it at the beginning, but with "forking" you have "Main PID" exiting with status=127. So it is most likely that I was wrong about Type=simple. Would you be able to start either of the services via "strace -f" and put the output on http://pastie.org/ ? – Ektich Aug 24 '16 at 20:35
  • Hey, sorry for the late reply, I got caught up in other priorities. pastie.org seems to be down at the moment so I'll use a different site. http://hastebin.com/onapugacek.rb It's a ton of output. But towards the end I see it mentioning files that don't exist, I wonder if that's part of the problem? – Danny Forche Sep 09 '16 at 00:37
  • few things caught my attention: 1. the process finishes with exit code 127, so systemd is right, it is not finishing with code 0 so it fails from the systemd point of view. 2. it is trying to start /usr/bin/cdp/watchdog/watchdog.sh, but fails to find it: do you have that file in that location and is it executable? 3. line 6035 looks like it is trying to write something back to you: does anything appear on the terminal or in the log files when you are trying to run "/sbin/cdp/cdpdaemon start"? – Ektich Sep 15 '16 at 19:47
0

First, determine if these processes do fork or not. If you start a "forking" process in a terminal, it releases the terminal and "goes into background". Such processes need Type=forking. Otherwise, if a process does not release the terminal unless you terminate it with Ctrl+C or put & in the end of the command-line, then it is Type=simple. It's that simple.

Second, you have mentioned that there are two processes that have to be started in order. Did you write the unit files for both of them? If not, then do it right now.

Third, don't forget the dependencies. If the latter process (/sbin/cdp/cdpdaemon start) depends on the former (/sbin/cdp/cdpagentproxy), then you need to express requirement and ordering dependencies between them in the unit files.

If, for example, the unit files you've created are named cdpagentproxy.service and cdpdaemon.service respectively, then you have to put these lines in your cdpdaemon.service unit file:

[Unit]
Requires=cdpagentproxy.service
After=cdpagentproxy.service

(Put these lines inside the existing [Unit] section, of course. I've included the section header only for completeness.)

Then run systemctl daemon-reload, kill both processes and try to start the unit manually, or simply add it to auto-start and reboot.

intelfx
  • 134
  • 7
  • Someone already replied about the forking thing, see my reply to them about it not making any difference. As for the two processes - I made them separate services. I was only pasting the results of one as they both look identical. The two processes do not depend on each other, you can start either one before the other. However, as I've said above, none of that works. It says the service started and "exited normally", whereas if I just type the process name myself it starts and stays running as it should. – Danny Forche Aug 23 '16 at 21:23
0

Let's step back a sec.

The sonicwall process isn't compatible with RHEL/Centos7.

Stop there. Pass it up to the fine folks at Sonicwall for their fix; that's exactly what you should be paying them license fees to do, and they (should) have the expertise to support their software like that.

CentOS6 is under support for another half-decade, but they should get started on an eeeeeasy transition.

user2066657
  • 336
  • 2
  • 13