Lighttpd Not Printing Errors for 500 anywhere?

Question

i'm trying to use lighttpd as a simple "alive" server for some nodes i have running. However, the script I'm running throws 500 errors non-deterministically, and i can't find them anywhere.

Here's my debugging configuration:

server.document-root = "/var/www/health_checker"

server.port = 44445
server.modules += ("mod_auth", "mod_rewrite", "mod_accesslog", "mod_cgi", "mod_redirect", "mod_status" )

cgi.assign = ( ".sh" => "/bin/sh" )

url.rewrite-once = (
      "^/healthz"  => "/healthz.sh",
      "^/.*" => "/livez.sh" 
)

server.error-handler-404   = "/livez.sh"
server.errorlog            = "/home/aronchick/temperror.log"
server.breakagelog         = "/home/aronchick/tempcgi.log"
server.accesslog           = "/home/aronchick/tempaccess.log"

debug.log-request-header             = "enable"
debug.log-request-header-on-error    = "enable"
debug.log-response-header            = "enable"
debug.log-file-not-found             = "enable"
debug.log-request-handling           = "enable"
debug.log-condition-handling         = "enable"
debug.log-condition-cache-handling   = "enable"
debug.log-ssl-noise                  = "enable"
debug.log-timeouts                   = "enable"

And here's an example of the non-determinism:

aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
{ 'hostname': 'myservice-vm-0.us-east4-c.c.myservice-production.internal', 'date': '2022-06-19 19:38:37.445066814+00:00', 'ip': 'xx.xx.xx.xx'}
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
 </body>
</html>
aronchick@myservice-vm-0:~$ 
aronchick@myservice-vm-0:~$ curl 0.0.0.0:44445/livez.sh
{ 'hostname': 'myservice-vm-0.us-east4-c.c.myservice-production.internal', 'date': '2022-06-19 19:38:37.445066814+00:00', 'ip': 'xx.xx.xx.xx'}
aronchick@myservice-vm-0:~$ ps aux

I can't even figure out where the 500 errors are being written. Here's the /home/aronchick/temperror.log:

2022-06-19 19:29:37: (connections.c.771) fd:8 rqst: GET / HTTP/1.1
2022-06-19 19:29:37: (connections.c.771) fd:8 rqst: Host: 127.0.0.1:44445
2022-06-19 19:29:37: (connections.c.771) fd:8 rqst: User-Agent: curl/7.81.0
2022-06-19 19:29:37: (connections.c.771) fd:8 rqst: Accept: */*
2022-06-19 19:29:37: (connections.c.771) fd:8 rqst: 
2022-06-19 19:29:37: (response.c.407) -- parsed Request-URI
2022-06-19 19:29:37: (response.c.409) Request-URI     : /
2022-06-19 19:29:37: (response.c.411) URI-scheme      : http
2022-06-19 19:29:37: (response.c.413) URI-authority   : 127.0.0.1:44445
2022-06-19 19:29:37: (response.c.415) URI-path (clean): /
2022-06-19 19:29:37: (response.c.417) URI-query       : 
2022-06-19 19:29:37: (response.c.407) -- parsed Request-URI
2022-06-19 19:29:37: (response.c.409) Request-URI     : /livez.sh
2022-06-19 19:29:37: (response.c.411) URI-scheme      : http
2022-06-19 19:29:37: (response.c.413) URI-authority   : 127.0.0.1:44445
2022-06-19 19:29:37: (response.c.415) URI-path (clean): /livez.sh
2022-06-19 19:29:37: (response.c.417) URI-query       : 
2022-06-19 19:29:37: (response.c.495) -- logical -> physical
2022-06-19 19:29:37: (response.c.497) Doc-Root     : /var/www/health_checker
2022-06-19 19:29:37: (response.c.499) Basedir      : /var/www/health_checker
2022-06-19 19:29:37: (response.c.501) Rel-Path     : /livez.sh
2022-06-19 19:29:37: (response.c.503) Path         : /var/www/health_checker/livez.sh
2022-06-19 19:29:37: (response.c.522) -- handling subrequest
2022-06-19 19:29:37: (response.c.524) Path         : /var/www/health_checker/livez.sh
2022-06-19 19:29:37: (response.c.526) URI          : /livez.sh
2022-06-19 19:29:37: (response.c.528) Pathinfo     : 
2022-06-19 19:29:37: (response.c.164) fd:8 resp: HTTP/1.1 500 Internal Server Error
2022-06-19 19:29:37: (response.c.164) fd:8 resp: Content-Type: text/html
2022-06-19 19:29:37: (response.c.164) fd:8 resp: Content-Length: 365
2022-06-19 19:29:37: (response.c.164) fd:8 resp: Date: Sun, 19 Jun 2022 19:29:37 GMT
2022-06-19 19:29:37: (response.c.164) fd:8 resp: Server: lighttpd/1.4.63
2022-06-19 19:29:37: (response.c.164) fd:8 resp:

score 1 · Accepted Answer · answered Jun 25 '22 at 02:36

1

If your script fails to return a response, then lighttpd returns 500 Internal Server Error.

The CGI script is probably not following the CGI specification.

? Does your CGI script do something like:

printf "Status: 200\nContent-Type: application/json\n\n"

before printing { 'hostname': 'myservice-vm-0.us-east4-c.c.myservice-production.internal', 'date': '2022-06-19 19:38:37.445066814+00:00', 'ip': 'xx.xx.xx.xx'} ?

answered Jun 25 '22 at 02:36

gstrauss

221
1
5

Unfortunately I moved on to using nginx because I didn’t figure this out. I’m marking this as the answer as it sounds correct, but ymmv. – aronchick Jun 26 '22 at 10:36

Lighttpd Not Printing Errors for 500 anywhere?

1 Answers1