I am using NGINX as a component in my system that accepts content streamed from clients (lets say, over RTMP) and passes it to some internal service for processing. The internal service fetches the content from the NGINX over HTTP.
But I don't want to wait until the client has completely streamed all their content before I start processing, so I'd like the internal service to start fetching the content as soon as the first byte becomes available - the problem is that if the client streams slower than how fast the internal service can pull from NGINX (which is always the case), NGINX reaches the current end of file and stops serving the content.
Is there a way to get NGINX to start serving a file whose final size is unknown (obviously with Transfer-Encoding: chunked) and not terminate the connection until all the (as yet to be provided) data has been sent?
I've tried serving the content of a FIFO, but NGINX simply ignores the FIFO returning 404 Not Found instead. I've also looked at the NGINX HLS module, but that seems to require the files to be in MPEG-4 format, which is not applicable to my use case.
