8

I am seeing some strange behavior running the tail -f command inside a Docker container in CoreOS.

There are a number of variables I can think of that may be contributing to the problem but I am not sure what I need to do to troubleshoot first. On CoreOS I am running the newest version with overlayfs support, as well as a newer version of Docker (1.4.1).

What is interesting is that I am able to tail logs successfully on a different host OS (Ubuntu 14.04) which is running a different version of Docker (1.3).

I can produce strace logs if that helps to troubleshoot, they seem to be significantly different across hosts. For example, on the host that isn't working the strace stops after reading in the following in the strace output:

04:03:03 inotify_add_watch(4, "f017f0a1-a1e9-11e4-90bc-027e0f87cac6-paster.log", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1 <0.000028>
04:03:03 fstat(3, {st_mode=S_IFREG|0644, st_size=12229, ...}) = 0 <0.000022>
04:03:03 read(4, 0x7711f0, 64)          = ? ERESTARTSYS (To be restarted if SA_RESTART is set) <3.101545>

I am just not familiar enough with strace to be very good at interpreting the results.

jmreicha
  • 791
  • 1
  • 16
  • 29
  • Having exactly the same problem here. No luck figuring that one out yet? – Roberto Andrade Mar 10 '15 at 03:39
  • @RobertoAndrade still haven't figured this one out. My workaround for now is to use 'less +F file.log' but it isn't a great solution. Let me know if you find anything! – jmreicha Mar 11 '15 at 03:43
  • I switched to a Debian host with same Docker version and containers and all works well (same with a boot2docker host). All of this with the latest versions for all components. – Roberto Andrade Mar 11 '15 at 13:37
  • @RobertoAndrade Yep I had the same result. I would still like to figure out why CoreOS seemed to have issues though. Do you happen to know what version of CoreOS being used? – jmreicha Mar 11 '15 at 22:53

2 Answers2

4

Not a solution but a workaround:

I have the same problem on CoreOS 607.0.0 and reproduced this problem in containers based on Ubuntu or Fedora. However containers that use busybox do not have this issue. Two workarounds:

1) use a busybox-based container image such as alpine

2) install busybox in your existing container and run

busybox tail -F <logfile>
Christoph
  • 41
  • 2
3

This is a bug introduced in CoreOS 561 when the default file system was changed from btrfs to the overlayfs. I found a workaround to basically format the filesystem using btrfs when a host is first started, and that seems to be working for now.

Include the following in your CloudInit:

#cloud-config
coreos:
  units:
    - name: format-var-lib-docker.service
      command: start
      content: |
        [Unit]
        Before=docker.service var-lib-docker.mount
        ConditionPathExists=!/var/lib/docker.btrfs
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/truncate --size=25G /var/lib/docker.btrfs
        ExecStart=/usr/sbin/mkfs.btrfs /var/lib/docker.btrfs
    - name: var-lib-docker.mount
      enable: true
      content: |
        [Unit]
        Before=docker.service
        After=format-var-lib-docker.service
        Requires=format-var-lib-docker.service
        [Install]
        RequiredBy=docker.service
        [Mount]
        What=/var/lib/docker.btrfs
        Where=/var/lib/docker
        Type=btrfs
        Options=loop,discard
Brandon
  • 131
  • 3