How can I log all process launches in Linux?

54

21

I would like to get a log of all processes that are launched with the time that they were launched and the arguments they were launched with. Is this possible in Linux?

Brandon DuRette

Posted 2010-12-16T23:42:55.193

Reputation: 643

Answers

42

Your starting point should be auditd.

Try something like this:

apt-get install auditd
auditctl -a task,always
ausearch -i -sc execve

Mikel

Posted 2010-12-16T23:42:55.193

Reputation: 7 890

1I'm getting error The audit system is disabled Where can I enable it? – Tombart – 2014-07-09T20:39:18.733

1well one problem could be solved by chmod 0750 /sbin/audispd but it's still not working (Debian Wheezy) – Tombart – 2014-07-09T20:43:27.853

is says Unable to set audit pid, exiting but I guess that the real problem will be that the system is running in LXC container – Tombart – 2014-07-09T20:49:10.233

How does audit integrate with systemd journald? Do their functions overlap? – CMCDragonkai – 2014-08-15T11:06:39.473

1I tried this on a live server and I effectively killed it, became almost unresponsive. I barely managed to remove this rule and make the server responsive again – Shocker – 2019-02-22T18:37:53.137

@Shocker I tested it with a sleep 10 command and it generated 276 lines of output. When I consider this and your comment I certainly won't try this in production, and I need something to find out what's killing an important process. – Rafał G. – 2020-02-11T07:28:58.170

10

I needed to do this, except (1) I didn't need the time and (2) I was only interested in processes that are started by a given process, and its children and subsequent descendants. Also, in the environment I was using, it wasn't possible to get auditd or accton, but there was valgrind.

Prefix the following to the process of interest on the command line:

valgrind --trace-children=yes

The information you need will be in the log output displayed on STDERR.

Evgeni Sergeev

Posted 2010-12-16T23:42:55.193

Reputation: 1 704

3By default, valgrind runs with the memcheck tool. To disable the tool and its related logging, and only print the creation of new commands (in addition to the usual output of your program), use the following command instead: valgrind --tool=none --trace-children=yes [command and args here] . Whenever a subprocess is spawned, Valgrind will then log the full command, including the arguments that were passed to it. – Rob W – 2018-03-02T09:39:47.023

6

You could use snoopy for this.

It is very simple to install, and since 2.x it can log arbitrary data (arguments, environmental variables, cwd, etc.).

Disclosure: Snoopy maintainer here.

Bostjan Skufca

Posted 2010-12-16T23:42:55.193

Reputation: 101

2

You could run startmon and follow its standard output, Ctrl-C when done. Here's how to compile and run startmon on recent Red Hat derived distros (RHEL, Fedora, CentOS):

sudo yum install git cmake gcc-c++
git clone https://github.com/pturmel/startmon
cd startmon
cmake .
make
sudo ./startmon -e

On Debian (and Ubuntu etc), the first line of the above changes to:

sudo apt-get install git cmake g++

Alternatively you can try the execsnoop script in perf-tools, see this answer. By default only the first 8 arguments are shown (9 including the program name); you can increase this via

sudo ./execsnoop -a 16

If you don't have root access to the system, the best you can do is keep polling /proc and hope it catches everything (which it won't), but for completeness here's a script to do that (I've put duplicate-removal in to simplify the output)—although this isn't as good as tracking them properly with one of the above methods, it does have the slight advantage of unambiguously displaying separators between the command-line arguments, in case you ever need to tell the difference between spaces inside an argument and space between arguments. This script is inefficient as it uses the CPU (well, one of its cores) 100% of the time.

function pstail () { python -c 'import os
last=set(os.listdir("/proc")) ; o=x=""
while True:
 pids=set(os.listdir("/proc"))
 new=pids.difference(last);last=pids
 for n in new:
  try: o,x=x,[j for j in open("/proc/"+n+"/cmdline")
    .read().split(chr(0)) if j]
  except IOError: pass
  if x and not o==x: print n,x' ; }

pstail

You can also patch execsnoop to tell you more explicitly which argument is which: grep -v sub.*arg < execsnoop > n && chmod +x n && mv n execsnoop

Silas S. Brown

Posted 2010-12-16T23:42:55.193

Reputation: 151

1

CONFIG_FTRACE and CONFIG_KPROBES through brendangregg/perf-tools

git clone https://github.com/brendangregg/perf-tools.git
cd perf-tools
git checkout 98d42a2a1493d2d1c651a5c396e015d4f082eb20
sudo ./execsnoop

On another shell:

while true; do sleep 1; date; done

First shell shows data of format:

Tracing exec()s. Ctrl-C to end.                                                        
Instrumenting sys_execve                                                               
   PID   PPID ARGS 
 20109   4336 date                                                                                       
 20110   4336 sleep 1                                                                                    
 20111   4336 date                                                                                                                                                                                                 
 20112   4336 sleep 1                                                                                    
 20113   4336 date                                                                                       
 20114   4336 sleep 1                                                                                    
 20115   4336 date                                                                                       
 20116   4336 sleep 1

CONFIG_PROC_EVENTS

Sample session:

$ su
# ./proc_events &
# /proc_events.out &
set mcast listen ok
# sleep 2 & sleep 1 &
fork: parent tid=48 pid=48 -> child tid=56 pid=56
fork: parent tid=48 pid=48 -> child tid=57 pid=57
exec: tid=57 pid=57
exec: tid=56 pid=56
exit: tid=57 pid=57 exit_code=0
exit: tid=56 pid=56 exit_code=0

CONFIG_PROC_EVENTS exposes the events to userland via a netlink socket.

proc_events.c adapted from: https://bewareofgeek.livejournal.com/2945.html

#define _XOPEN_SOURCE 700
#include <sys/socket.h>
#include <linux/netlink.h>
#include <linux/connector.h>
#include <linux/cn_proc.h>
#include <signal.h>
#include <errno.h>
#include <stdbool.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

static volatile bool need_exit = false;

static int nl_connect()
{
    int rc;
    int nl_sock;
    struct sockaddr_nl sa_nl;

    nl_sock = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_CONNECTOR);
    if (nl_sock == -1) {
        perror("socket");
        return -1;
    }
    sa_nl.nl_family = AF_NETLINK;
    sa_nl.nl_groups = CN_IDX_PROC;
    sa_nl.nl_pid = getpid();
    rc = bind(nl_sock, (struct sockaddr *)&sa_nl, sizeof(sa_nl));
    if (rc == -1) {
        perror("bind");
        close(nl_sock);
        return -1;
    }
    return nl_sock;
}

static int set_proc_ev_listen(int nl_sock, bool enable)
{
    int rc;
    struct __attribute__ ((aligned(NLMSG_ALIGNTO))) {
        struct nlmsghdr nl_hdr;
        struct __attribute__ ((__packed__)) {
            struct cn_msg cn_msg;
            enum proc_cn_mcast_op cn_mcast;
        };
    } nlcn_msg;

    memset(&nlcn_msg, 0, sizeof(nlcn_msg));
    nlcn_msg.nl_hdr.nlmsg_len = sizeof(nlcn_msg);
    nlcn_msg.nl_hdr.nlmsg_pid = getpid();
    nlcn_msg.nl_hdr.nlmsg_type = NLMSG_DONE;

    nlcn_msg.cn_msg.id.idx = CN_IDX_PROC;
    nlcn_msg.cn_msg.id.val = CN_VAL_PROC;
    nlcn_msg.cn_msg.len = sizeof(enum proc_cn_mcast_op);

    nlcn_msg.cn_mcast = enable ? PROC_CN_MCAST_LISTEN : PROC_CN_MCAST_IGNORE;

    rc = send(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0);
    if (rc == -1) {
        perror("netlink send");
        return -1;
    }

    return 0;
}

static int handle_proc_ev(int nl_sock)
{
    int rc;
    struct __attribute__ ((aligned(NLMSG_ALIGNTO))) {
        struct nlmsghdr nl_hdr;
        struct __attribute__ ((__packed__)) {
            struct cn_msg cn_msg;
            struct proc_event proc_ev;
        };
    } nlcn_msg;
    while (!need_exit) {
        rc = recv(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0);
        if (rc == 0) {
            /* shutdown? */
            return 0;
        } else if (rc == -1) {
            if (errno == EINTR) continue;
            perror("netlink recv");
            return -1;
        }
        switch (nlcn_msg.proc_ev.what) {
            case PROC_EVENT_NONE:
                printf("set mcast listen ok\n");
                break;
            case PROC_EVENT_FORK:
                printf("fork: parent tid=%d pid=%d -> child tid=%d pid=%d\n",
                        nlcn_msg.proc_ev.event_data.fork.parent_pid,
                        nlcn_msg.proc_ev.event_data.fork.parent_tgid,
                        nlcn_msg.proc_ev.event_data.fork.child_pid,
                        nlcn_msg.proc_ev.event_data.fork.child_tgid);
                break;
            case PROC_EVENT_EXEC:
                printf("exec: tid=%d pid=%d\n",
                        nlcn_msg.proc_ev.event_data.exec.process_pid,
                        nlcn_msg.proc_ev.event_data.exec.process_tgid);
                break;
            case PROC_EVENT_UID:
                printf("uid change: tid=%d pid=%d from %d to %d\n",
                        nlcn_msg.proc_ev.event_data.id.process_pid,
                        nlcn_msg.proc_ev.event_data.id.process_tgid,
                        nlcn_msg.proc_ev.event_data.id.r.ruid,
                        nlcn_msg.proc_ev.event_data.id.e.euid);
                break;
            case PROC_EVENT_GID:
                printf("gid change: tid=%d pid=%d from %d to %d\n",
                        nlcn_msg.proc_ev.event_data.id.process_pid,
                        nlcn_msg.proc_ev.event_data.id.process_tgid,
                        nlcn_msg.proc_ev.event_data.id.r.rgid,
                        nlcn_msg.proc_ev.event_data.id.e.egid);
                break;
            case PROC_EVENT_EXIT:
                printf("exit: tid=%d pid=%d exit_code=%d\n",
                        nlcn_msg.proc_ev.event_data.exit.process_pid,
                        nlcn_msg.proc_ev.event_data.exit.process_tgid,
                        nlcn_msg.proc_ev.event_data.exit.exit_code);
                break;
            default:
                printf("unhandled proc event\n");
                break;
        }
    }

    return 0;
}

static void on_sigint(__attribute__ ((unused)) int unused)
{
    need_exit = true;
}

int main()
{
    int nl_sock;
    int rc = EXIT_SUCCESS;

    signal(SIGINT, &on_sigint);
    siginterrupt(SIGINT, true);
    nl_sock = nl_connect();
    if (nl_sock == -1)
        exit(EXIT_FAILURE);
    rc = set_proc_ev_listen(nl_sock, true);
    if (rc == -1) {
        rc = EXIT_FAILURE;
        goto out;
    }
    rc = handle_proc_ev(nl_sock);
    if (rc == -1) {
        rc = EXIT_FAILURE;
        goto out;
    }
    set_proc_ev_listen(nl_sock, false);
out:
    close(nl_sock);
    exit(rc);
}

GitHub upsatream.

I don't think however that you can you get process data such as UID and process arguments because exec_proc_event contains so little data: https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80 We could try to immediately read it from /proc, but there is a risk that the process finished and another one took its PID, so it wouldn't be reliable.

Tested in Ubuntu 17.10.

Ciro Santilli 新疆改造中心法轮功六四事件

Posted 2010-12-16T23:42:55.193

Reputation: 5 621

0

Also you can use atop for viewing resource usage by processes. This is useful tools for logging and analyzing resource usage in each part of time

Quarind

Posted 2010-12-16T23:42:55.193

Reputation: 11

-3

You can try cat ~/.bash_history There is system log viewer,this can help you out.

Kracekumar

Posted 2010-12-16T23:42:55.193

Reputation: 125

1Let's also add that the command history is the usual way of accessing this information. – bryn – 2016-04-28T09:14:38.790

1~/.bash_history only contains commands I've executed in a terminal, apparently. I'm looking for a log of all the programs executed, for example when I click an icon to open up my e-mail client, gedit, or I open up my browser, and my browser executes another process by itself. new123456's answer did the trick. – runeks – 2011-06-01T19:22:35.623