Skip to content

Queues (Horizon) ​

Many actions don't finish in the web request. Order matching, order-book updates, emails and merchant webhooks are put on a queue in Redis and handled by background workers. Laravel Horizon runs and supervises those workers.

5 minutesServer administrationDevOpsVerified against the platform code

How it runs ​

  • Queue storage: Redis. Set QUEUE_CONNECTION=redis in .env.
  • Workers: Horizon, started as the PM2 process horizon (command php artisan horizon).
  • Supervision: the scheduler task pm:watcher checks every minute that horizon is running in PM2 and starts it again if it stopped. See Scheduler.

QUEUE_CONNECTION must be redis

.env.example ships with QUEUE_CONNECTION=sync, which runs every job inside the web request, so pages become slow and order processing blocks. On a real server always use redis.

Worker groups and queues ​

Horizon starts these supervisors (from config/horizon.php). In production (APP_ENV=production), three of them get larger limits:

SupervisorQueuesMax processes (production)Used for
supervisor-eventsevents3Live updates sent to one user: wallet balance, deposits, withdrawals, their orders, notifications. Own workers, so they never wait behind bulk jobs
supervisor-1default10Emails and notifications (including mobile push notifications), merchant jobs and other background work
supervisor-2orderbook10Live order-book updates pushed to browsers
supervisor-3market10Live prices, 24h statistics and the trades list pushed to browsers
supervisor-4merchant-webhooks10Delivering webhooks to merchants
supervisor-ordersorders15Creating and processing orders, stop-limit triggers (60-second timeout)
supervisor-lowlow5Market capitalisation recalculation after trades (runs at lower CPU priority)

Every job is tried 3 times before it counts as failed. Workers are balanced automatically: Horizon moves processes to the queues that have the most waiting jobs.

APP_ENV must be production or local

Horizon starts the supervisors listed for the current APP_ENV. The configuration has production and local sections. With any other value (for example staging), Horizon starts no workers and nothing in the queues is processed.

The Horizon dashboard ​

Horizon has a web dashboard at:

txt
https://yourdomain.com/horizoncctech

It shows the workers, queue lengths, throughput, recent jobs and failed jobs.

Who can open it: only the first account registered on the platform (the account with the lowest ID). Other admins, even superadmins, get an access error. This is set in app/Providers/HorizonServiceProvider.php.

Recent, completed and failed jobs are kept for 1 minute (trim in config/horizon.php), so the dashboard is for watching live activity, not for history. Errors from failed jobs are also written to storage/logs/laravel-YYYY-MM-DD.log.

Useful commands ​

Run these in the project folder:

bash
php artisan horizon:status      # running / paused / inactive
php artisan horizon:pause       # stop taking new jobs (running jobs finish)
php artisan horizon:continue    # resume after a pause
php artisan horizon:terminate   # stop gracefully; pm:watcher starts it again
php artisan queue:failed        # list failed jobs
php artisan queue:retry all     # retry every failed job

Restarting after an update ​

Workers load the code once when they start. After you deploy new code, restart them so they use it:

bash
php artisan horizon:terminate

Horizon finishes the jobs it's working on and exits. Within a minute, pm:watcher starts it again with the new code. To restart at once instead, run pm2w restart horizon (see Installation → Prepare PM2 for pm2w).

Scaling for busy markets ​

Signs that workers can't keep up: orders take seconds to appear, order books lag behind trades, or the Horizon dashboard shows long waits on a queue.

  1. Check the server first. If CPU is already near 100%, add CPU cores before adding workers. More processes on a full CPU makes things slower.

  2. Raise the limit for the busy queue in config/horizon.php, under environments → production. For example, for orders:

    php
    'supervisor-orders' => [
        'maxProcesses' => 25,
        'balanceMaxShift' => 2,
        'balanceCooldown' => 1,
    ],
  3. Apply it: php artisan config:clear (or config:cache if you cache config), then php artisan horizon:terminate.

Each worker uses up to 128 MB of memory, so 10 extra workers need about 1.3 GB of free RAM.

Redis is shared

Queues, cache and sessions all live in Redis. On busy platforms, give Redis its own CPU core and enough memory. Run redis-cli info memory to check its memory use.

Not processing? ​

SymptomCheck
Nothing happens after placing orderspm2w status: is horizon online? Is order-processor online?
horizon:status says inactiveThe scheduler isn't running, so pm:watcher never starts it. Check the cron entry.
Horizon runs but queues growAPP_ENV isn't production or local, or the server is out of CPU
Jobs fail repeatedlyRead storage/logs/laravel-YYYY-MM-DD.log for the error

What is kept in failed jobs

Live socket updates (order books, prices, wallet balances) are not kept when they fail, for example while Soketi is down: replaying a stale price later would be wrong. Every other failed job is kept for a week (queue:prune-failed, daily). Repeating merchant jobs (rates, invoice expiry, webhooks, clean-up) are unique: while the queue is down only one of each waits, instead of one per minute.

More fixes are in Troubleshooting.

Owner documentation for the CryptoTech exchange platform.