3

i am running nginx v.1.9.3 here to serve my site and also video files for streaming. i am seeing that the mp4 files are always served with a code of 200 and that partial content requests are ignored. sometimes videos are able to be 'seeked' in browsers and sometimes not. i have used various file formats and the results vary from ok-ish to fatally bad.

i used curl -I to view the return data from nginx and saw that there was no mention of the server accepting byte-ranges and when i manually request parts of files using curl i see that in all cases the return code is 200.

i have tested with and without the mp4 module activated in the nginx config file build and neither made a significant advantage over the other.

an example file can be viewed here (this is mp4 h264 format as i heard this was better for modern browsers): https://www.ureka.org/file/play/18212/censored%20on%20google%202-h264.mp4

this is my site's config file:

server {
    server_name www.mysite.org;
    listen 443 ssl spdy default_server;
    ssl_certificate_key /mysitekey.key;
    ssl_certificate /mysitekey.crt;
    keepalive_timeout     300;
    spdy_keepalive_timeout 300;

    index index.php index.html index.htm;

    # https only, mode - Remember this setting for 365 days
    add_header Strict-Transport-Security max-age=31536000;

    access_log /mysite.log;
    error_log /mysite.log;
    root /mysite/path;

    client_header_buffer_size 1k;
    fastcgi_index index.php;
    client_max_body_size  2G;
    client_body_buffer_size 1K;
    proxy_read_timeout 600;

    error_page   500  /500.html;
    error_page   502  /502.html; #bad gateway
    error_page   503  /503.html;
    error_page   504  /504.html; # gateway timeout

    location ~ (^\.|/\.) {
        return 403;
    }

    set $cache_uri $request_uri;

    # POST requests and urls with a query string should always go to PHP
    if ($request_method = POST) {
        set $cache_uri 'null cache';
    }
    if ($query_string != "") {
        set $cache_uri 'null cache';
    }

    # Don't cache uris containing the following segments
    if ($request_uri ~* "(/admin/|/xml-rpc_handler.php|/(cron|sign-in|joyn|messages).php|/feed/|index.php|sitemap.xml|[a-z0-9_-]+-sitemap([0-9]+)?.xml)") {
        set $cache_uri 'null cache';
    }

    location /cache {
        rewrite ^/cache\/(.*)$ /engine/handlers/cache_handler.php?request=$1&$query_string;
    }

    location /export {
        rewrite ^/export\/([A-Za-z]+)\/([0-9]+)\/?$ /engine/handlers/export_handler.php?view=$1&guid=$2;
        rewrite ^/export\/([A-Za-z]+)\/([0-9]+)\/([A-Za-z]+)\/([A-Za-z0-9\_]+)\/$ /engine/handlers/export_handler.php?view=$1&guid=$2&type=$3&idname=$4;
    }

    location = /rewrite.php {
        rewrite ^(.*)$ /install.php;
    }

    location / {
        try_files $uri $uri/ /index.php?__elgg_uri=$uri&$query_string;
    }

    location ~ /\.well-known
    {
        access_log off;
        log_not_found off; 
    }

    location ~ /\. 
    {
        access_log off;
        log_not_found off; 
        deny all;
    }   

    # Prevent clients from accessing hidden files (starting with a dot)
    # This is particularly important if you store .htpasswd files in the site hierarchy
    location ~* (?:^|/)\. {
        deny all;
    }

    # Prevent clients from accessing to backup/config/source files
    location ~* (?:\.(?:bak|config|sql|fla|psd|ini|log|sh|inc|swp|dist)|~)$ {
        deny all;
    }

    location ~ \.php$ 
    {
    # Fastcgi cache
    set $skip_cache 1;
    if ($cache_uri != "null cache") {
        add_header X-Cache-Debug "$cache_uri $cookie_nocache $arg_nocache$arg_comment $http_pragma $http_authorization";
        set $skip_cache 0;
    }
    fastcgi_cache_bypass $skip_cache;
    fastcgi_cache microcache;
    fastcgi_cache_key $scheme$host$request_uri$request_method;
    fastcgi_cache_valid any 8m;
    fastcgi_cache_bypass $http_pragma;
    fastcgi_cache_use_stale updating error timeout invalid_header http_500;
    try_files $uri =404;
    include fastcgi_params;
    fastcgi_pass unix:/socket/php5-fpm.sock;
    fastcgi_index  index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_connect_timeout 60;
    fastcgi_send_timeout 180;
    fastcgi_read_timeout 600;
    fastcgi_buffers 256 4k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_intercept_errors on;
    fastcgi_param  QUERY_STRING     $query_string;
    fastcgi_param  REQUEST_METHOD   $request_method;
    fastcgi_param  CONTENT_TYPE     $content_type;
    fastcgi_param  CONTENT_LENGTH   $content_length;
}

## cache headers 

# cache.appcache, your document html and data
location ~* \.(?:manifest|appcache|html?|xml|json)$ {
  expires -1;
  add_header Cache-Control "private";
  add_header Pragma "private";
}

# Feed
location ~* \.(?:rss|atom)$ {
  expires 1h;
  add_header Cache-Control "public";
}

location ~ \.flv$ {
    flv;
}   
}

and here is the main nginx.conf file:

user nginx;
worker_processes  4;
worker_priority -5;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

# Maximum open file descriptors per process;
# should be > worker_connections.
worker_rlimit_nofile 40000;

events 
{
    worker_connections  8096;
    use epoll;
    multi_accept on;
}

http 
{   
    # Define the MIME types for files.
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    underscores_in_headers on;

    # do not allow content to be framed
    add_header X-Frame-Options DENY;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
    # caching using 50MB of RAM and 1000MB of disk space
    fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=microcache:50m max_size=1000m inactive=600m;

    # Force the latest IE version
    # Use ChromeFrame if it's installed for a better experience for the poor IE folk
    add_header "X-UA-Compatible" "IE=Edge"; 

 # cache file DESCRIPTORS

    open_file_cache max=2000 inactive=20s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 3;
    open_file_cache_errors on;
    keepalive_requests 100000;
    server_tokens off; # hides nginx version number

      # Speed up file transfers by using sendfile() to copy directly
      # between descriptors rather than using read()/write().
  sendfile        on;

  # Tell Nginx not to send out partial frames; this increases throughput
  # since TCP frames are filled up before being sent out. (adds TCP_CORK)
  tcp_nopush      on;

    ## Global SSL options
  ssl_session_cache builtin:1000 shared:SSL:10m; # a 1mb cache can hold about 4000 sessions, so we can hold 40000 sessions
  ssl_session_timeout  24h;

  # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
    ssl_dhparam /path/dhparam.pem;

    ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:HIGH:DH+AES:ECDH+3DES:DH+3DES:!eNULL:!NULL:!aNULL:!EDH:!MD5:!DSS; # previously produced A grade test result

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;

    #ssl_buffer_size 4k;
    ssl_buffer_size 1400; # 1400 bytes to fit in one MTU
    ssl_session_tickets off;

    # enable SPDY header compression
    spdy_headers_comp 6;

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them

    ssl_stapling on; # Requires nginx >= 1.3.7
    ssl_stapling_verify on; # Requires nginx => 1.3.7

    resolver 156.154.70.1 156.154.71.1 valid=300s;
    resolver_timeout 5s;

  # Compression

  # Enable Gzip compressed.
  gzip on;

  # Enable compression both for HTTP/1.0 and HTTP/1.1 (required for CloudFront).
  gzip_http_version  1.0;

  # Compression level (1-9).
  # 5 is a perfect compromise between size and cpu usage, offering about
  # 75% reduction for most ascii files (almost identical to level 9).
  gzip_comp_level    6;

  # Don't compress anything that's already small and unlikely to shrink much
  # if at all (the default is 20 bytes, which is bad as that usually leads to
  # larger files after gzipping).
  gzip_min_length    10000;

  # Compress data even for clients that are connecting to us via proxies,
  # identified by the "Via" header (required for CloudFront).
  gzip_proxied       any;

  # Tell proxies to cache both the gzipped and regular version of a resource
  # whenever the client's Accept-Encoding capabilities header varies;
  # Avoids the issue where a non-gzip capable client (which is extremely rare
  # today) would display gibberish if their proxy gave them the gzipped version.
  gzip_vary          on;

  # Compress all output labeled with one of the following MIME-types.
  gzip_types
    application/atom+xml
    application/javascript
    application/json
    application/rss+xml
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xhtml+xml
    application/xml
    font/opentype
    image/svg+xml
    image/x-icon
    text/css
    text/plain
    text/javascript
    application/font-woff
    text/x-component;
  # text/html is always compressed by HttpGzipModule

    gzip_static       on;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6]\.";

    ## Start: Timeouts ##
    client_body_timeout   30;
    client_header_timeout 30;
    keepalive_timeout     100;
    send_timeout          100;  
    ## End: Timeouts ##

    ## Reset lingering timed out connections. Deflect DDoS.
    reset_timedout_connection on;
    ### Directive describes the zone, in which the session states are stored i.e. store in slimits. ###
     ### 1m can handle 32000 sessions with 32 bytes/session, set to 5m x 32000 session ###
   limit_conn_zone $binary_remote_addr zone=perip:10m;
   limit_req_zone $binary_remote_addr zone=periprate:10m rate=1000r/s;
   limit_conn_zone $server_name zone=perserver:10m;
    #  limit_rate 1280k;
    ### Control maximum number of simultaneous connections for one session i.e. ###
    ### restricts the amount of connections from a single ip address ###
    limit_conn perip 200;
    ### limit connections for the total server
    limit_conn perserver 15000;


    ## Enable clickjacking protection in modern browsers. Available in
    ## IE8 also. See
    ## https://developer.mozilla.org/en/The_X-FRAME-OPTIONS_response_header
    add_header X-Frame-Options SAMEORIGIN;

    ## Include the cache map to decide when or not when to cache.
    include map_cache_piwik.conf;

    ## Include the php-fpm status allowed hosts configuration block.
    ## Uncomment to enable if you're running php-fpm.
    include php_fpm_status_allowed_hosts.conf;

    include /etc/nginx/sites-enabled/*;
}

if anyone knows why this is failing i'd love for you to share here.. thanks

tunist
  • 53
  • 1
  • 9
  • Do you have MOOV atom in the very beginning of content? – Anatoly Aug 02 '15 at 21:53
  • 1
    1. The `mp4` directive is needed for Flash streaming, it's not expected to be faster than range requests. 2. The `add_header` use is useless. 3. The `return 206` will prevent your configuration from working at all. 4. See http://stackoverflow.com/questions/14598565/serving-206-byte-range-through-nginx-django/24459321#24459321 for possible reasons why range requests may not work. – Maxim Dounin Aug 02 '15 at 23:14
  • Anatoly - i do have a function for relocating the moov atom and tested with it originally. now though it is not being used - i think that is because i found it didn't help, but i could go back to testing with that if i don't find another solution here. – tunist Aug 03 '15 at 11:07
  • Maxim - 1. the mp4 directive is needed for flash!? i thought the flv directive was for flash and the mp4 was for mp4? 4. my config file does not have any of the problem directives activated. ssi is not on and gzip_types only targets non video formats. – tunist Aug 03 '15 at 11:18
  • another strange aspect to this is that when i view the video in fedora 22 (firefox 39) - they play and seek fine! but other browsers do not. i have no idea how that is possible. – tunist Aug 03 '15 at 11:21
  • so now i have removed the accept-ranges header and all the other relevant directives except for gzip_off and gzip_static off - i am back to where i was originally - whereby the mp4 file that is not h264 format will seek/stream ok UNTIL i seek to a point that has not yet been buffered, wait for the video to start playing again.. and then finally seek to a point near the start of the video. at which point i always see 'Video can't be played because the file is corrupt'. i thought this might be a firefox bug, since i have seen similar logged. but other browsers do similar too here. – tunist Aug 03 '15 at 11:31
  • so i just used a moov atom library to show me that indeed the moov atom is not at the beginning of the files. i will play with that now and see what occurs. – tunist Aug 03 '15 at 11:49
  • so i have reuploaded the h264 file after having run the process on it to relocate the moov atom to the beginning. the outcome is that the video now performs similarly to the non h264 mp4 file, in that it can be played and seeked, until it is seeked several times and then an error is produced. when i view the file in chromium, the javascript video player shifts to it's flash mode and the result is different. in that case, there is no audio playback for that file at all. – tunist Aug 03 '15 at 13:32
  • 1
    It sounds like your problem is with how the MP4 is encoded. If it is not fully seekable, then there's nothing the nginx module can do for you. You might seek help (no pun intended) with correctly encoding your MP4 on our sister site, [video.se]. – Michael Hampton Aug 03 '15 at 17:32
  • thanks for the tip - i didn't know that the video site existed. that said, i think that the fact that i have never seen nginx output a 206 status code after it receives a partial content request suggests that there may be an issue beyond file formatting. – tunist Aug 03 '15 at 22:16
  • having done more testing here, it is clear that the issue is not MP4 format related. i have also tested other formats, that also behave as if the byte ranges are not being properly handled. this webm file, for example, does not handle seeking at all: https://www.ureka.org/file/play/7376/psychopath%20-%20ie%20video.webm – tunist Aug 05 '15 at 15:38
  • @tunist, please see the link in my comment above for some list of most common things which may result in range requests not being handled. If in doubt, please provide full configuration you are using. It's also good idea to try range requests by hand (e.g. with `curl -H 'Range: bytes=0-2' http://example.org/`). – Maxim Dounin Aug 06 '15 at 19:57
  • i did already try range requests using curl, yes - this is how i know they are not working. i did read the list you linked too - yes. none of those issues are relevant for me. i am using gzip but the types are explicitly listed and do not include video formats. i will update the question to include my config file shortly. thanks. – tunist Aug 06 '15 at 20:07
  • @tunist, configuration added only shows part of your real config. Notably, included files from `include /etc/nginx/sites-enabled/*;` are not shown, and they may contain anything. Note well there is at least one syntax error (missing `{` in `location ~ \.php$`, may be introduced while copying it here). Try `nginx -T` to get the full configuration. Also, the URL of a test file no longer works. – Maxim Dounin Aug 07 '15 at 00:04
  • @MaximDounin - yes, the missing bracket was removed when i copied the file to serverfault (i removed some un-needed lines and accidentally removed the bracket too). the sites-enabled folder only contains configs for admin services that all have unique server blocks and no nginx directives are inside those files that are outside of server blocks that only apply to specific domains. e.g. these nginx config files do not affect the site that is serving videos. – tunist Aug 07 '15 at 11:18
  • @MaximDounin - i have removed that test file, yes. this one will remain online: https://www.ureka.org/file/play/7369/nasa%20mars%20anomalies%202010.mp4 – tunist Aug 07 '15 at 11:18
  • oh, so it seems likely that i need to add extra logic into the PHP page that handles the video stream to support range requests - as detailed here: http://codesamplez.com/programming/php-html5-video-streaming-tutorial – tunist Aug 07 '15 at 11:37

1 Answers1

1

the cause of the problem was the lack of range handling in the PHP page i am using for streaming the files. i forgot that that is a requirement of the process! i have added the videostream class (http://codesamplez.com/programming/php-html5-video-streaming-tutorial) to the page and so far the streaming is working well in my tests :)

tunist
  • 53
  • 1
  • 9