LogstashAgent

A control-plane agent for LogstashUI that fully manages the Logstash instance it runs alongside.

Warning: Beta Release - This project is under active development. Features may change.

Overview

LogstashAgent is the host-side runtime for LogstashUI-managed instances.

It enrolls with LogstashUI, persists local agent state, checks in for policy and configuration changes, and applies those changes directly to the local Logstash installation.

Features

Enrollment + Reconciliation Loop - Enroll with LogstashUI and continuously reconcile desired state to the local Logstash instance.

• Enrollment mode: python src/logstashagent/main.py --enroll=<TOKEN> --logstash-ui-url=<URL>
• Controller mode: python src/logstashagent/main.py --run
• Agent state includes enrollment identity, policy assignment, and revision tracking.

Pipeline Management API - Create, update, delete, validate, and inspect Logstash pipelines.

• Endpoints include /_logstash/pipeline, /_logstash/pipeline/{pipeline_id}, /_logstash/pipeline/{pipeline_id}/logs, and /_logstash/pipelines/status.
• Config persistence is backed by pipelines.yml, conf.d, and metadata files.

Host Configuration Management - Apply managed configuration to local Logstash runtime files and secure settings.

• Controller updates logstash.yml, jvm.options, log4j2.properties, and keystore entries.
• Supports reconciliation and service restart flows for managed updates.

Local State + Credential Protection - Persist agent identity and encrypted sensitive fields under package-local data storage.

• State file: src/logstashagent/data/state.json
• Encryption key file: src/logstashagent/data/.secret_key
• Log file: src/logstashagent/data/logs/logstashagent.log

Requirements

Software

For Managed Agent mode

• Python 3.12+
• Logstash 8.x, 9.x

For Enrolled Controller mode (--run)

• Python 3.12+
• Access to managed Logstash settings/log paths
• Network reachability to your LogstashUI instance

For Local Development

• Python 3.12+
• uv (recommended) or pip

Quick Start - Agent Mode

Tip

Use --run only after successful enrollment, because controller mode requires persisted enrollment state.

Install

cd LogstashAgent
uv sync

Configure

Copy and adjust the example config:

cp src/logstashagent/config/logstashagent.example.yml src/logstashagent/config/logstashagent.yml

Run agent process

python src/logstashagent/main.py

By default this starts the agent service (including management API) on 0.0.0.0:9600 unless overridden in config.

---

Enroll And Run Controller

1. Enroll the agent

python src/logstashagent/main.py --enroll=<BASE64_TOKEN> --logstash-ui-url=http://localhost:8080

2. Start controller mode

python src/logstashagent/main.py --run

3. Verify state files

• src/logstashagent/data/state.json
• src/logstashagent/data/.secret_key

Updating

Pull latest source and resync dependencies:

git pull
uv sync

Then restart the running agent process.

Limitations

• Controller behavior depends on available host service managers (systemctl or service) for restart operations.
• Host filesystem permissions must allow managed writes to Logstash settings and metadata paths.

Roadmap

• Hardened host-mode lifecycle and service integration
• Expanded policy diff/apply visibility and diagnostics
• Additional keystore and secret-management workflows
• Broader automated test coverage around simulation and controller reconciliation paths

Reporting Issues

Found a bug or have a feature request? Open an issue.

Contributing

Contributions are welcome.

Please open an issue to discuss large changes before submitting a pull request.

License

Copyright 2024-2026 Elasticsearch and contributors.

Licensed under the Apache License, Version 2.0. See LICENSE for details.