How to handle stream with ffmpeg and GPU

Question

So in my project, I need to capture a video stream, decode the video, run the individual frames through an object detection algorithm and then play the output as a video.

This whole operation is running on a Linux server with a couple of Tesla V100's, 40 core CPU (can't recall exact model right now) and a ridiculous amount of RAM. It should play the video which is 720x576i at 25fps (progressive), so de-interlacing also needs to be happening on the fly.

Needless to say, the object detection is by far the slowest operation happening here, but I'm currently looking on how best to approach the transcoding.

Now, since I have access to such a powerful GPU I thought it would be smart to utilize it, but I also read in a few places that transcoding on GPU sucks.

In any case, I have cuda 9.0 installed so I built ffmpeg with the following options enabled:

--enable-libavcodec  --enable-libx264 --enable-gpl --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp

(I'm 95% sure these are all, but if it really matters ill check once I have access to the server, which should be in a couple of days)

What is the best way to proceed from here? I'm not very familiar with ffmpeg and even less so when it comes to all the gpu related options.

I tried de-interlacing with -vf: yadif_cuda, but it failed, saying that this filter is not available. I thought it would be available if i enabled cuda in the build options.

I have no clue which options to use when decoding (e.g., -hwaccel ???, -c:v libx264 or h264_cuvid, etc..) and also I'm not sure how should I display the frames once the detection algorithm spits them out. Do I just 'play' them? Do I re-encode, and if so, how?

Any help is much appreciated, and if any more information is needed ill happily provide what I can.

I've made a diagram to clarify how the entire thing should play out:

Python is used for the algorithm, with probably skvideo for capture and playback. I could also just use ffmpeg directly (with a bash command, executed from the python script) though if necessary.

Everything that is between Capture and Play should happen live, i.e. while the video is transmitted to the end user.