Scale & Operations guides
These guides explain how rConfig behaves as an estate grows from hundreds to many thousands of devices: how work flows from the scheduler through the queues to the workers, where the bottlenecks appear, and which settings trade one resource for another. The step-by-step tuning stays in the main docs, starting with the Horizon Queue Manager; these pages explain the model behind it.
How a scheduled backup becomes work
Almost everything rConfig does in the background follows the same path. Knowing it tells you where to look when things slow down.
| Stage | Component | What happens | Where it can bottleneck |
|---|---|---|---|
| 1. Schedule | Laravel scheduler (cron, web server user) | Every minute, due tasks are queued | Missing crontab entry: tasks never fire |
| 2. Queue | Redis | Each device in a task becomes its own job inside a batch | Redis memory, or eviction dropping jobs |
| 3. Execute | Horizon workers, kept alive by Supervisor | Workers take jobs by queue priority and connect to devices | Too few workers (long waits), too many (CPU, memory, DB connections) |
| 4. Store | Database and the storage directory |
Configurations, diffs, compliance results and logs are written | Buffer pool size, disk I/O, table growth |
Because every device is a separate job, one slow or failing device does not hold up the rest of the task. The queue keeps the web interface responsive however much background work is running. A test run with rConfig-sim, shown on the Horizon Queue Manager page, works through backups for 20,000 simulated devices.
Queues and workers
rConfig uses several named queues processed by priority, so a user’s manual download is not stuck behind a scheduled overnight run. Horizon groups them under three supervisors, and each group scales differently.
| Supervisor group | Queues | How it scales |
|---|---|---|
| HorizonTwo | TaskDownloadQueue, TaskSnippetQueue, ManualDownloadQueue |
Scales up to HORIZON_MAX_PROCESS under load. This is where backup throughput comes from |
| HorizonOne | HighPriorityQueue, PolicyCompliance, PurgeDataJobs, rConfigDefault, SnmpTrapQueue |
From V8.4.1, a fixed pool sized by HORIZON_MAX_PROCESS, split evenly and always running |
| HorizonThree | ReportBuildQueue (V8.4.0+) |
A small dedicated pool, so a backlog elsewhere never delays task reports |
The design decisions this creates:
- More workers is not free. Each worker can use a database connection, and every extra worker adds CPU and memory load. Raising
HORIZON_MAX_PROCESSalso raises the always-on HorizonOne pool, so check databasemax_connectionsand host memory before you raise it. - Timeouts must stay ordered.
REDIS_QUEUE_RETRY_AFTERmust stay greater than the job timeouts, or the same job can be processed twice. - Long-lived workers drift. Workers process thousands of jobs without restarting. The worker recycling settings restart a worker cleanly after a set time or job count, between jobs, rather than relying on a nightly restart of the whole Horizon process.
Scheduling at scale
The scheduler is simple. The judgement is in how you spread the work.
- Stagger tasks. Ten heavy tasks at midnight all land in the same queues at once. Spreading them flattens the peak the workers and database have to absorb.
- Fit the interval to the run time. A backup that takes two hours cannot run hourly. Start with daily intervals and shorten them only as capacity allows.
- Let targets follow the inventory. Tasks that target Smart Groups are evaluated at run time, so new devices join without editing the task. Disabled devices are always excluded.
See Scheduled Tasks.
Retention: keeping growth bounded
Every backup, log entry and compliance result adds rows and files. Without retention, the database keeps growing, which costs storage, lengthens backups and slows queries. rConfig handles this in two parts.
| Part | Covers | Behaviour that matters at scale |
|---|---|---|
| Data Purge | Timestamped tables such as activity_log, jobs, failed_jobs and the Vector agent tables |
Per-table retention, batch size and frequency. Large deletions are queued as background jobs on PurgeDataJobs |
| Config Retention | configs, config_changes and the configuration files on disk |
Disabled by default. Always keeps the latest version and a minimum number of versions per device and command. Never queued, always runs inline |
The trade-offs are batch size against database load during the purge, and retention length against storage and troubleshooting history. The main docs advise starting with conservative retention and dry runs. Read Data Purge and Config Purge & History Repair.
Tuning the host: one page, four cards
System Settings >> System Optimization compares the running values with recommendations for PHP, Horizon, Redis and the database, and flags what needs attention. It is the first place to look after a change in scale. Some cards are cached, so refresh after a change.
| Area | The decision | Read |
|---|---|---|
| PHP | memory_limit and max_execution_time. The CLI and Apache read separate files on Ubuntu and Debian, and Horizon workers use the CLI values |
PHP Settings |
| Database | The buffer pool (MariaDB/MySQL) or shared buffers (PostgreSQL) is the most important setting, followed by max_connections sized for workers plus web traffic |
Database Optimization |
| Redis | A maxmemory limit above the observed peak, with no eviction |
Redis Memory Optimization |
| Horizon | Whether Horizon runs, and whether workers are on the current code | Horizon Queue Manager |
PHP, the database, Redis and the workers all share the same RAM on a single server. The sizing tables in each page assume a dedicated host, so reduce the allocations if anything else runs there.
Monitoring: what rConfig watches, and what you must
rConfig runs its own health checks every five minutes (database, cache, Redis, Horizon, the scheduler, disk space, CPU load and others). From V8.4.1 a failing check raises a notification, which can go to Slack, Teams or a webhook. A health endpoint in the REST API lets an external monitor read the same results.
Those checks cover the application. The server still needs external monitoring for disk space and I/O, memory and swap, database connections and slow queries, and queue depth and wait time. A queue that grows and never drains is the clearest sign that workers, database or schedule need attention. See System Monitoring.
Beyond one server
When a single server is no longer enough, or one outage must not affect the whole estate, the options are covered elsewhere:
- Independent instances, each owning part of the inventory, in HA Options Overview and the High Availability guides.
- Distributed collection with Vector agents, in the Vector guides.
Self-managed Docker deployments reach their limits earlier than traditional installs. The reasons are in Docker Deployments.