Raspberry Pi pauses randomly while using OMX hardware acclerated encoding

0

I am streaming RTMP via ffmpeg (also happens with avconv) on a raspberry pi and it randomly stops working after a few hours. Using the nginx-rtmp-module as the server.

There are no ffmpeg or avconv errors, it simply stops encoding new frames from the source webcam. After killing / and starting back up ffmpeg it works fine again.

Example ffmpeg command:

ffmpeg -nostdin -hide_banner -input_format yuyv422 -f video4linux2 -s 800x600 -r 15 -i /dev/video0 -c:v h264_omx -r 15 -b:v 1500k -g 30 -an -f flv rtmp://localhost:1935/live/stream

This is on Raspbian GNU/Linux 10 (buster) and has been seen on both Pi 3B+ and Zero W using 3 different webcams. Happens with either single or multiple webcams being used at a time. Have also tried -threads 1 to no effect.

Nginx config just in case:

worker_processes  1;

events {
    worker_connections  1024;
}

rtmp_auto_push off;

rtmp {
        server {
                listen 1935;
                chunk_size 4096;

                application live {
                        live on;
                        record off;
                        allow publish 127.0.0.1;
                        deny publish all;
                        allow play 192.168.1.0/24;
                        deny play all;
                }
        }
}

Thanks for any suggestions!

Edit: Very minimal logs

/var/log/nginx/access.log

127.0.0.1 [26/Jul/2019:22:33:33 -0500] PUBLISH "live" "stream" "" - 301522662 529 "" "FMLE/3.0 (compatible; Lavf58.28" (33m 30s)
127.0.0.1 [26/Jul/2019:22:33:35 -0500] PUBLISH "live" "stream2" "" - 70667926 529 "" "FMLE/3.0 (compatible; Lavf58.28" (33m 32s)
192.168.1.2 [26/Jul/2019:22:34:03 -0500] PLAY "live" "stream" "" - 3361 301523158 "" "" (34m 9s)
192.168.1.2 [26/Jul/2019:22:51:08 -0500] PLAY "live" "stream2" "" - 355 11889 "" "" (42s)
..... <lot of the same>
192.168.1.2 [27/Jul/2019:06:34:47 -0500] PLAY "live" "stream2" "" - 355 10905 "" "" (29s)
192.168.1.2 [27/Jul/2019:06:35:18 -0500] PLAY "live" "stream2" "" - 355 13200 "" "" (30s)
192.168.1.2 [27/Jul/2019:06:35:48 -0500] PLAY "live" "stream2" "" - 355 14285 "" "" (29s)
192.168.1.2 [27/Jul/2019:06:36:18 -0500] PLAY "live" "stream2" "" - 355 18418 "" "" (30s)
192.168.1.2 [27/Jul/2019:06:37:07 -0500] PLAY "live" "stream2" "" - 355 37571 "" "" (49s)
192.168.1.2 [27/Jul/2019:07:28:06 -0500] PLAY "live" "stream" "" - 32011 3014672918 "" "" (8h 37m 44s)
192.168.1.2 [27/Jul/2019:07:28:06 -0500] PLAY "live" "stream2" "" - 2687 233538140 "" "" (50m 59s)
192.168.1.2 [27/Jul/2019:07:28:38 -0500] PLAY "live" "stream" "" - 354 472 "" "" (30s)

/var/log/nginx/error.log

2019/07/19 02:22:55 [notice] 4827#0: signal process started

ffmpeg

/usr/local/bin/ffmpeg -nostdin -hide_banner -input_format yuyv422 -f video4linux2 -s 800x600 -r 15 -i /dev/video0 -c:v h264_omx -r 15 -b:v 1500k -g 30 -an -f flv rtmp://localhost:1935/live/stream
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 68945.611935, bitrate: 115200 kb/s
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 800x600, 115200 kb/s, 15 fps, 15 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_omx))
[h264_omx @ 0x1ebd810] Using OMX.broadcom.video_encode
Output #0, flv, to 'rtmp://localhost:1935/live/stream':
  Metadata:
    encoder         : Lavf58.30.100
    Stream #0:0: Video: h264 (h264_omx) ([7][0][0][0] / 0x0007), yuv420p, 800x600, q=2-31, 1500 kb/s, 15 fps, 1k tbn, 15 tbc
    Metadata:
      encoder         : Lavc58.55.100 h264_omx
frame=259763 fps= 15 q=-0.0 size= 3175964kB time=04:48:37.40 bitrate=1502.4kbits/s speed=   1x

(the last line frame is just stuck, and no longer increasing at all, no other messages)

ffmpeg verbose logs from ffmpeg -v 48 -nostdin -input_format yuyv422 -f video4linux2 -video_size 800x600 -framerate 15 -i /dev/video0 -c:v h264_omx -b:v 1500k -g 30 -an -f flv rtmp://localhost:1935/live/stream

  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi --enable-nonfree --enable-libfreetype --enable-libx264 --enable-libmp3lame
  libavutil      56. 32.100 / 56. 32.100
  libavcodec     58. 55.100 / 58. 55.100
  libavformat    58. 30.100 / 58. 30.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 58.100 /  7. 58.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '48'.
Reading option '-nostdin' ... matched as option 'stdin' (enable or disable interaction on standard input) with argument 0.
Reading option '-input_format' ... matched as AVOption 'input_format' with argument 'yuyv422'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'video4linux2'.
Reading option '-video_size' ... matched as AVOption 'video_size' with argument '800x600'.
Reading option '-framerate' ... matched as AVOption 'framerate' with argument '15'.
Reading option '-i' ... matched as input url with argument '/dev/video0'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_omx'.
Reading option '-b:v' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '1500k'.
Reading option '-g' ... matched as AVOption 'g' with argument '30'.
Reading option '-an' ... matched as option 'an' (disable audio) with argument '1'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'flv'.
Reading option 'rtmp://localhost:1935/live/stream' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 48.
Applying option nostdin (enable or disable interaction on standard input) with argument 0.
Successfully parsed a group of options.
Parsing a group of options: input url /dev/video0.
Applying option f (force format) with argument video4linux2.
Successfully parsed a group of options.
Opening an input file: /dev/video0.
[video4linux2,v4l2 @ 0x2c304a0] fd:3 capabilities:84a00001
[video4linux2,v4l2 @ 0x2c304a0] Current input_channel: 0, input_name: Camera 1, input_std: 0
[video4linux2,v4l2 @ 0x2c304a0] Setting time per frame to 1/15
[video4linux2,v4l2 @ 0x2c304a0] All info found
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 259209.335638, bitrate: 115200 kb/s
    Stream #0:0, 1, 1/1000000: Video: rawvideo, 1 reference frame (YUY2 / 0x32595559), yuyv422, 800x600, 0/1, 115200 kb/s, 15 fps, 15 tbr, 1000k tbn, 1000k tbc
Successfully opened the file.
Parsing a group of options: output url rtmp://localhost:1935/live/stream.
Applying option c:v (codec name) with argument h264_omx.
Applying option b:v (video bitrate (please use -b:v)) with argument 1500k.
Applying option an (disable audio) with argument 1.
Applying option f (force format) with argument flv.
Successfully parsed a group of options.
Opening an output file: rtmp://localhost:1935/live/stream.
[rtmp @ 0x2c352f0] No default whitelist set
[tcp @ 0x2c357c0] No default whitelist set
[tcp @ 0x2c357c0] Original list of addresses:
[tcp @ 0x2c357c0] Address ::1 port 1935
[tcp @ 0x2c357c0] Address 127.0.0.1 port 1935
[tcp @ 0x2c357c0] Interleaved list of addresses:
[tcp @ 0x2c357c0] Address ::1 port 1935
[tcp @ 0x2c357c0] Address 127.0.0.1 port 1935
[tcp @ 0x2c357c0] Starting connection attempt to ::1 port 1935
[tcp @ 0x2c357c0] Connection attempt to ::1 port 1935 failed: Connection refused
[tcp @ 0x2c357c0] Starting connection attempt to 127.0.0.1 port 1935
[tcp @ 0x2c357c0] Successfully connected to 127.0.0.1 port 1935
[rtmp @ 0x2c352f0] Handshaking...
[rtmp @ 0x2c352f0] Type answer 3
[rtmp @ 0x2c352f0] Server version 13.14.10.13
[rtmp @ 0x2c352f0] Proto = rtmp, path = /live/stream, app = live, fname = stream
[rtmp @ 0x2c352f0] Window acknowledgement size = 5000000
[rtmp @ 0x2c352f0] Max sent, unacked = 5000000
[rtmp @ 0x2c352f0] New incoming chunk size = 4096
[rtmp @ 0x2c352f0] Releasing stream...
[rtmp @ 0x2c352f0] FCPublish stream...
[rtmp @ 0x2c352f0] Creating stream...
[rtmp @ 0x2c352f0] Sending publish command for 'stream'
Successfully opened the file.
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_omx))
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
detected 4 logical cores
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'video_size' to value '800x600'
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'pix_fmt' to value '1'
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'time_base' to value '1/1000000'
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x2c410e0] Setting 'frame_rate' to value '15/1'
[graph 0 input from stream 0:0 @ 0x2c410e0] w:800 h:600 pixfmt:yuyv422 tb:1/1000000 fr:15/1 sar:0/1 sws_param:flags=2
[format @ 0x2c41580] Setting 'pix_fmts' to value 'yuv420p'
[auto_scaler_0 @ 0x2c426e0] Setting 'flags' to value 'bicubic'
[auto_scaler_0 @ 0x2c426e0] w:iw h:ih flags:'bicubic' interl:0
[format @ 0x2c41580] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format'
[AVFilterGraph @ 0x2c38390] query_formats: 4 queried, 2 merged, 1 already done, 0 delayed
[auto_scaler_0 @ 0x2c426e0] w:800 h:600 fmt:yuyv422 sar:0/1 -> w:800 h:600 fmt:yuv420p sar:0/1 flags:0x4
[h264_omx @ 0x2c346f0] Using OMX.broadcom.video_encode
[h264_omx @ 0x2c346f0] OMX state changed to 2
[h264_omx @ 0x2c346f0] OMX state changed to 3
[h264_omx @ 0x2c346f0] OMX port 201 settings changed
Output #0, flv, to 'rtmp://localhost:1935/live/stream':
  Metadata:
    encoder         : Lavf58.30.100
    Stream #0:0, 0, 1/1000: Video: h264 (h264_omx), 1 reference frame ([7][0][0][0] / 0x0007), yuv420p, 800x600, 0/1, q=2-31, 1500 kb/s, 15 fps, 1k tbn, 15 tbc
    Metadata:
      encoder         : Lavc58.55.100 h264_omx
Clipping frame in rate conversion by 0.000008
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
    Last message repeated 1 times
Clipping frame in rate conversion by 0.459724
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.439308
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.470848
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.439461
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.459999
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.443169
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.471870
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.450981
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.429878
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600.66 bitrate=1028.7kbits/s speed=1.33x
Clipping frame in rate conversion by 0.461571
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.441002
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.470711
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.450264
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.431007
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.461479
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.435616
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.461662
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600.20 bitrate=1252.7kbits/s speed= 1.2x
Clipping frame in rate conversion by 0.438911
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.465172
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.435188
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.455467
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.433754
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.465797
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.442924
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600.66 bitrate=1363.5kbits/s speed= 1.1x
Clipping frame in rate conversion by 0.473045
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.452675
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.428154
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.461952
[rawvideo @ 0x2c32e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.441139

...
Clipping frame in rate conversion by 0.398155
[rawvideo @ 0x36c9e50] PACKET SIZE: 960000, STRIDE: 16004.53 bitrate=1456.1kbits/s speed=   1x
Clipping frame in rate conversion by 0.377571
[rawvideo @ 0x36c9e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.411201
[rawvideo @ 0x36c9e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.389839
[rawvideo @ 0x36c9e50] PACKET SIZE: 960000, STRIDE: 16004.73 bitrate=1456.1kbits/s speed=   1x
Clipping frame in rate conversion by 0.372490
[rawvideo @ 0x36c9e50] PACKET SIZE: 960000, STRIDE: 1600
Clipping frame in rate conversion by 0.407921

CasualDemon

Posted 2019-07-26T19:29:58.037

Reputation: 51

Share full log. – Gyan – 2019-07-27T10:58:32.540

added what logs there are @Gyan – CasualDemon – 2019-07-27T18:55:49.283

Share the log of ffmpeg -v 48 -nostdin -input_format yuyv422 -f video4linux2 -video_size 800x600 -framerate 15 -i /dev/video0 -c:v h264_omx -b:v 1500k -g 30 -an -f flv rtmp://localhost:1935/live/stream – Gyan – 2019-07-28T09:28:56.723

@Gyan that is the last section of the logs above, nothing else is displayed, no error messages or anything. Hence my confusion. – CasualDemon – 2019-07-28T15:56:31.467

1Right, run the command in my last comment and share its full log. – Gyan – 2019-07-28T16:14:25.197

Got ya now, I will let this run until it also stalls and report back! – CasualDemon – 2019-07-28T16:25:26.020

@Gyan stopped again and added log to post, sadly doesn't show any errors or additional stuff. – CasualDemon – 2019-07-29T02:15:27.673

I don't see the full log. – Gyan – 2019-07-29T04:40:22.197

I only copied what was visible when it stalled, copied the start of it as well this time. @Gyan – CasualDemon – 2019-07-29T19:00:37.420

No answers