This pattern vs splitting into multiple jobs?

[mollerhoj]

Perhaps something worth mentioning in the README / tutorials?

I'm not sure I understand the advantage of this pattern over splitting these long jobs into multiple small jobs?

Since the enumerator here uses a cursor, it should be rather simple to have each finished job start the next one, recursively? (Having each job take the cursor as an argument)
Am I missing something?

I guess one advantage is that we don't have the overhead of starting multiple jobs (unless there's an interruption). But this should be a minimal performance advantage - and one can process multiple records in a single job (batching) if desired (and thus have fewer jobs)..

In short, are the some advantages to this pattern that I'm missing?

[fatkodima]

I described a few advantages here - https://github.com/fatkodima/sidekiq-iteration#faq

[Mangara]

If you're thinking of these small jobs running serially I'm not sure I understand the difference - this is essentially what job-iteration does, except it does so automatically, adapting to the size of the workload, and it uses the same job_id for the entire duration, so that it is easier to track the work through logs / tracing.

If you're thinking of running the small jobs in parallel, that is indeed very different and has different trade-offs. The work will get done much faster, but also put that much more strain on underlying resources (DB, API calls, etc). If you process each iteration in its own job, there is a large amount of overhead involved with enqueuing potentially millions of jobs versus one job iterating that many times. If you batch iterations, it is tricky to find the right amount of parallelism.

In the end, the serial model job-iteration uses is both simple, and good enough for many use cases.