Thanks for the response. I forgot to mention that the Make/CI jobs are run on a Slurm HPC cluster (actually multiple since the CI file uses runners running in multiple Slurm clusters for different resources). Make ensures we don’t redo stuff unless needed.

Point taken about restarts, but I usually want to see why a thing fails before retrying.

It seems like Pachyderm mainly offers better granularity, especially for the data management.