Appearance
Queues (Horizon)
Many actions don't finish in the web request. Order matching, order-book updates, emails and merchant webhooks are put on a queue in Redis and handled by background workers. Laravel Horizon runs and supervises those workers.
How it runs
- Queue storage: Redis. Set
QUEUE_CONNECTION=redisin.env. - Workers: Horizon, started as the PM2 process
horizon(commandphp artisan horizon). - Supervision: the scheduler task
pm:watcherchecks every minute thathorizonis running in PM2 and starts it again if it stopped. See Scheduler.
QUEUE_CONNECTION must be redis
.env.example ships with QUEUE_CONNECTION=sync, which runs every job inside the web request, so pages become slow and order processing blocks. On a real server always use redis.
Worker groups and queues
Horizon starts these supervisors (from config/horizon.php). In production (APP_ENV=production), three of them get larger limits:
| Supervisor | Queues | Max processes (production) | Used for |
|---|---|---|---|
supervisor-events | events | 3 | Live updates sent to one user: wallet balance, deposits, withdrawals, their orders, notifications. Own workers, so they never wait behind bulk jobs |
supervisor-1 | default | 10 | Emails and notifications (including mobile push notifications), merchant jobs and other background work |
supervisor-2 | orderbook | 10 | Live order-book updates pushed to browsers |
supervisor-3 | market | 10 | Live prices, 24h statistics and the trades list pushed to browsers |
supervisor-4 | merchant-webhooks | 10 | Delivering webhooks to merchants |
supervisor-orders | orders | 15 | Creating and processing orders, stop-limit triggers (60-second timeout) |
supervisor-low | low | 5 | Market capitalisation recalculation after trades (runs at lower CPU priority) |
Every job is tried 3 times before it counts as failed. Workers are balanced automatically: Horizon moves processes to the queues that have the most waiting jobs.
APP_ENV must be production or local
Horizon starts the supervisors listed for the current APP_ENV. The configuration has production and local sections. With any other value (for example staging), Horizon starts no workers and nothing in the queues is processed.
The Horizon dashboard
Horizon has a web dashboard at:
txt
https://yourdomain.com/horizoncctechIt shows the workers, queue lengths, throughput, recent jobs and failed jobs.
Who can open it: only the first account registered on the platform (the account with the lowest ID). Other admins, even superadmins, get an access error. This is set in app/Providers/HorizonServiceProvider.php.
Recent, completed and failed jobs are kept for 1 minute (trim in config/horizon.php), so the dashboard is for watching live activity, not for history. Errors from failed jobs are also written to storage/logs/laravel-YYYY-MM-DD.log.
Useful commands
Run these in the project folder:
bash
php artisan horizon:status # running / paused / inactive
php artisan horizon:pause # stop taking new jobs (running jobs finish)
php artisan horizon:continue # resume after a pause
php artisan horizon:terminate # stop gracefully; pm:watcher starts it again
php artisan queue:failed # list failed jobs
php artisan queue:retry all # retry every failed jobRestarting after an update
Workers load the code once when they start. After you deploy new code, restart them so they use it:
bash
php artisan horizon:terminateHorizon finishes the jobs it's working on and exits. Within a minute, pm:watcher starts it again with the new code. To restart at once instead, run pm2w restart horizon (see Installation → Prepare PM2 for pm2w).
Scaling for busy markets
Signs that workers can't keep up: orders take seconds to appear, order books lag behind trades, or the Horizon dashboard shows long waits on a queue.
Check the server first. If CPU is already near 100%, add CPU cores before adding workers. More processes on a full CPU makes things slower.
Raise the limit for the busy queue in
config/horizon.php, underenvironments → production. For example, for orders:php'supervisor-orders' => [ 'maxProcesses' => 25, 'balanceMaxShift' => 2, 'balanceCooldown' => 1, ],Apply it:
php artisan config:clear(orconfig:cacheif you cache config), thenphp artisan horizon:terminate.
Each worker uses up to 128 MB of memory, so 10 extra workers need about 1.3 GB of free RAM.
Redis is shared
Queues, cache and sessions all live in Redis. On busy platforms, give Redis its own CPU core and enough memory. Run redis-cli info memory to check its memory use.
Not processing?
| Symptom | Check |
|---|---|
| Nothing happens after placing orders | pm2w status: is horizon online? Is order-processor online? |
horizon:status says inactive | The scheduler isn't running, so pm:watcher never starts it. Check the cron entry. |
| Horizon runs but queues grow | APP_ENV isn't production or local, or the server is out of CPU |
| Jobs fail repeatedly | Read storage/logs/laravel-YYYY-MM-DD.log for the error |
What is kept in failed jobs
Live socket updates (order books, prices, wallet balances) are not kept when they fail, for example while Soketi is down: replaying a stale price later would be wrong. Every other failed job is kept for a week (queue:prune-failed, daily). Repeating merchant jobs (rates, invoice expiry, webhooks, clean-up) are unique: while the queue is down only one of each waits, instead of one per minute.
More fixes are in Troubleshooting.
