Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?
Since Ruby's GIL means you can't do true parallel processing, it's common to kick off multiple worker processes and have them each take one job at a time.
If the job involves an expensive API call it will tie up a worker for the duration of the call.
So it's not so much ~thousands of jobs per second as it is a limitation of only 10 ruby processes per 1GB of memory. (A ruby process with the Rails env loaded is about 100MB.)
Elixir gives you the ability to make a few orders of magnitude more calls in a tiny amount of memory. (100,000 Elixir "workers" can be had in less than 1GB).
Sidekiq runs 25 worker threads by default. MRI will context switch upon I/O so typical server-side applications will get plenty of concurrency, even with the GIL, because they spend lots of time talking to the DB, memcached, redis, 3rd party APIs, etc.
tl;dr - Sidekiq can do a LOT of work, even on MRI with the GIL.
Comments
Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?
Since Ruby's GIL means you can't do true parallel processing, it's common to kick off multiple worker processes and have them each take one job at a time.
If the job involves an expensive API call it will tie up a worker for the duration of the call.
So it's not so much ~thousands of jobs per second as it is a limitation of only 10 ruby processes per 1GB of memory. (A ruby process with the Rails env loaded is about 100MB.)
Elixir gives you the ability to make a few orders of magnitude more calls in a tiny amount of memory. (100,000 Elixir "workers" can be had in less than 1GB).
Sidekiq runs 25 worker threads by default. MRI will context switch upon I/O so typical server-side applications will get plenty of concurrency, even with the GIL, because they spend lots of time talking to the DB, memcached, redis, 3rd party APIs, etc.
tl;dr - Sidekiq can do a LOT of work, even on MRI with the GIL.