1
I need to process >50,000 files using a third-party .exe command-line application. The application takes only one input file at a time, so I have to launch the application >50,000 times.
Each file (each job) usually takes about one second. However, sometimes the application hangs indefinitely.
I have written a Windows shell script that runs all the jobs serially, and checks every second to see whether the job is done. After 10 seconds, it kills the job and moves on to the next. However, it takes about 20 hours. I believe I can bring the total runtime down by a large amount if I run multiple jobs in parallel. The question is how?
In CMD I launch the task with Start, but there is no simple way to recover the process ID (PID) and therefore I cannot easily keep track of which instance has run for how long. I feel like I am trying to reinvent the umbrella. Any suggestions?
Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. – Xavierjazz – 2017-08-14T02:14:38.337
I have described my problem in detail in the post title and the first two paragraphs. The third paragraph talks about what I did. I changed the fourth paragraph but I don't know that the question is better now. – Mattia Landoni – 2017-08-14T02:20:14.483