I'm working on a project that with various data processing steps that naturally fit into a pipeline model. Steps involve things like:
- download data periodically
- generate a bunch of files
- upload those files to another service
- check periodically (once a day) until the service finishes processing
- get results and reprocess if needed
Gitlab CI pipelines are best way I can think of that can easily model such a pipeline with enough visibility to know what happened where etc. Another advantage is that Gitlab is open source allowing me to tinker.
Is this easily doable (I can setup a cron task + git submit to trigger the pipeline)? Is there a better way/ system to handle such a pipeline?
