Skip to content

Chapter V - Background Jobs and Process Management with Sidekiq

(avr. time for this chapter: 1 day)

This chapter introduces background job processing with Sidekiq and essential process management skills. You will learn how to handle long-running tasks asynchronously and manage system processes effectively.

Continue building upon the application from Chapter III.

Introduction to Background Jobs

In web applications, some operations are too slow to handle during a regular HTTP request. Examples include sending emails, processing files, generating reports, or calling external APIs. Background jobs allow these operations to run asynchronously, improving user experience and application responsiveness.

Why Use Background Jobs?

  • Better User Experience - Users don't wait for slow operations
  • Reliability - Jobs can be retried if they fail
  • Scalability - Workers can be scaled independently
  • Scheduling - Jobs can run at specific times

Sidekiq Setup

Sidekiq is a popular background job processor for Ruby. It uses Redis to manage job queues and is known for its efficiency and reliability.

Steps to implement:

  1. Add Sidekiq gem to your project
  2. Install and start Redis (required by Sidekiq)
  3. Configure Active Job to use Sidekiq as the queue adapter
  4. Verify the setup by starting Sidekiq and checking for errors

Reference: Sidekiq Documentation

Creating Background Jobs

Steps to implement:

  1. Generate a new job using Rails generator
  2. Implement job logic that sends purchase notification emails
  3. Call the job from your purchase controller using perform_later
  4. Experiment with scheduling jobs for future execution

Note: Always pass primitive types (IDs, strings) to jobs instead of objects. Objects may change between when the job is enqueued and when it runs.

Practical Exercise: Async Purchase Notifications

Refactor your ebook store to use background jobs for email notifications.

Steps to implement:

  1. Create a PurchaseNotificationJob that handles:
  2. Sending commission email to the seller
  3. Sending statistics email

  4. Any other email notifications from Chapter III

  5. Update your purchase flow:

  6. Keep the database transaction synchronous
  7. Move email sending to background jobs (outside the transaction)

  8. Ensure the user receives immediate feedback

  9. Create a StatisticsReportJob that:

  10. Calculates daily/weekly ebook statistics
  11. Can be scheduled to run periodically

Running Sidekiq

Steps to implement:

  1. Start Sidekiq in a terminal
  2. Configure and mount the Sidekiq Web UI to monitor jobs
  3. Make a purchase and observe the job being processed

Note: In production, Sidekiq should run as a managed process (systemd, Docker, etc.)

Process Management in Linux/macOS

Understanding how to manage processes is essential for any developer. You will frequently need to start, stop, and debug background processes.

Steps to implement:

  1. Learn to list all running processes using ps aux
  2. Filter processes by name using grep
  3. Use top or htop for interactive process monitoring
  4. Understand the key columns in process output:
  5. PID - Process ID, unique identifier
  6. %CPU - CPU usage percentage
  7. %MEM - Memory usage percentage
  8. STAT - Process state (S=sleeping, R=running, Z=zombie)

Killing Processes

Steps to implement:

  1. Practice finding process IDs (PIDs) for Sidekiq, Redis, and Ruby processes

  2. Learn different ways to terminate processes:

  3. Kill a single process by PID
  4. Force kill a process that won't terminate
  5. Kill all processes matching a name

  6. Kill a process using a specific port

  7. Understand signal types:

  8. SIGTERM (15) - Graceful shutdown, process can clean up
  9. SIGKILL (9) - Immediate termination, no cleanup
  10. SIGINT (2) - Interrupt (like pressing Ctrl+C)
  11. SIGHUP (1) - Hangup, often used to reload configuration

Reference: Linux Signals

Practical Exercise: Process Management

Steps to implement:

  1. Start multiple Sidekiq processes with different queue configurations

  2. Practice managing these processes:

  3. Find all Sidekiq processes and note their PIDs
  4. Kill one process gracefully and observe the shutdown behavior
  5. Force kill another and compare the behavior

  6. Clean up all remaining Sidekiq processes

  7. Verify processes are terminated using ps aux | grep sidekiq

Handling Stuck Ports

A common issue when Rails or other servers don't shut down properly.

Steps to implement:

  1. Learn to identify what process is using a specific port
  2. Practice killing processes that are blocking ports
  3. (Optional) Create a helper function in your shell configuration to simplify this task

Job Queues and Priorities

Steps to implement:

  1. Configure multiple queues in Sidekiq configuration file (critical, default, low)
  2. Assign different jobs to specific queues based on their importance
  3. Start Sidekiq with queue priority weights
  4. Test that critical jobs are processed before low priority jobs

Error Handling and Retries

Sidekiq automatically retries failed jobs. Understanding this behavior is crucial.

Steps to implement:

  1. Configure custom retry behavior for specific jobs
  2. Implement error handling for:
  3. Temporary failures that should be retried
  4. Permanent failures that should be discarded
  5. Test job failure scenarios and observe retry behavior

Final Exercise: Complete Integration

Steps to implement:

  1. Refactor all email sending in your ebook store to use background jobs

  2. Implement a scheduled job for daily statistics:

  3. Create DailyStatisticsJob
  4. Calculate total sales, popular ebooks, active sellers

  5. Discuss scheduling options with your tutor (e.g., sidekiq-scheduler gem)

  6. Practice the full development workflow:

  7. Start Redis, Sidekiq, and Rails server
  8. Make a purchase and verify emails are sent asynchronously
  9. View job status in Sidekiq Web UI
  10. Practice killing and restarting Sidekiq

  11. Observe how pending jobs are processed after restart

  12. Document the startup process:

  13. Create a README section or script that lists all required processes
  14. Include commands to start and stop each service

Tip: Consider using tools like foreman or overmind to manage multiple processes during development.

Checklist

Before moving to the next chapter, ensure you can:

  • [ ] Explain why background jobs are important
  • [ ] Set up and configure Sidekiq with Redis
  • [ ] Create and enqueue background jobs
  • [ ] Use ps, grep, kill, and pkill commands confidently
  • [ ] Find and kill processes using specific ports
  • [ ] Understand the difference between SIGTERM and SIGKILL
  • [ ] Configure job queues and priorities
  • [ ] Handle job failures and retries