Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,8 @@ load("@envoy_api//bazel:api_build_system.bzl", "api_proto_package")
licenses(["notice"]) # Apache 2

api_proto_package(
deps = ["@xds//udpa/annotations:pkg"],
deps = [
"//envoy/config/accesslog/v3:pkg",
"@xds//udpa/annotations:pkg",
],
)
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ syntax = "proto3";

package envoy.extensions.bootstrap.reverse_tunnel.downstream_socket_interface.v3;

import "envoy/config/accesslog/v3/accesslog.proto";

import "udpa/annotations/status.proto";

option java_package = "io.envoyproxy.envoy.extensions.bootstrap.reverse_tunnel.downstream_socket_interface.v3";
Expand Down Expand Up @@ -35,4 +37,10 @@ message DownstreamReverseConnectionSocketInterface {
// Optional HTTP handshake configuration. When unset, the initiator envoy uses the defaults
// provided by ``HttpHandshakeConfig``.
HttpHandshakeConfig http_handshake = 3;

// Access log configuration for reverse tunnel initiator lifecycle events.
// Logs are emitted on handshake success, handshake failure, and connection close.
// Reverse tunnel metadata (node_id, cluster_id, tenant_id, upstream cluster, etc.)
// is available via ``%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:*)%`` substitutions.
repeated config.accesslog.v3.AccessLog access_log = 4;
}
105 changes: 105 additions & 0 deletions docs/root/configuration/other_features/reverse_tunnel.rst
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,111 @@ The header priority order is:
be inferred from the request (e.g., the ``x-tenant-id`` header is missing or the formatter
evaluates to empty), host selection will fail and the request will not be routed.

.. _config_reverse_tunnel_access_logging:

Access logging
--------------

Both the initiator and responder bootstrap extensions support access logging for reverse tunnel
lifecycle events. Access logs are emitted at key connection lifecycle points, providing visibility
into tunnel establishment, handshake outcomes, and connection teardown.

Initiator access logging
~~~~~~~~~~~~~~~~~~~~~~~~

The initiator (downstream) Envoy can be configured to log reverse tunnel lifecycle events by adding
an ``access_log`` field to the downstream socket interface bootstrap extension:

.. code-block:: yaml

bootstrap_extensions:
- name: envoy.bootstrap.reverse_tunnel.downstream_socket_interface
typed_config:
"@type": type.googleapis.com/envoy.extensions.bootstrap.reverse_tunnel.downstream_socket_interface.v3.DownstreamReverseConnectionSocketInterface
stat_prefix: "downstream_reverse_connection"
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
log_format:
text_format_source:
inline_string: "[%START_TIME%] reverse_tunnel_initiator event=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:event)% node=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:node_id)% cluster=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:cluster_id)% tenant=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:tenant_id)% upstream_cluster=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:upstream_cluster)% host=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:host_address)% error=%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:error)%\n"

Any :ref:`access log <arch_overview_access_logs>` type supported by Envoy (file, stdout, gRPC, etc.)
can be used. The access log configuration follows the same format as access logs in other Envoy
components such as the :ref:`TCP proxy <config_network_filters_tcp_proxy>` and
:ref:`HTTP connection manager <config_http_conn_man>`.

Initiator lifecycle events
^^^^^^^^^^^^^^^^^^^^^^^^^^

The initiator emits access log entries at the following lifecycle points:

.. list-table::
:header-rows: 1
:widths: 25 75

* - Event
- Description
* - ``handshake_success``
- A reverse tunnel handshake completed successfully. The connection is now established
and available for data requests from the responder.
* - ``handshake_failure``
- A reverse tunnel handshake failed. The ``error`` field contains the failure reason
(e.g., HTTP status error, encode error, connection closed).
* - ``connection_closed``
- An established reverse tunnel connection was closed. This triggers re-establishment
on the next maintenance cycle.

Initiator dynamic metadata fields
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

All initiator access log fields are available under the ``envoy.reverse_tunnel.initiator`` dynamic
metadata namespace and can be referenced using the ``%DYNAMIC_METADATA(envoy.reverse_tunnel.initiator:FIELD)%``
:ref:`format string <config_access_log_format_strings>`.

.. list-table::
:header-rows: 1
:widths: 20 15 65

* - Field
- Type
- Description
* - ``event``
- string
- The lifecycle event that triggered this log entry. One of: ``handshake_success``,
``handshake_failure``, ``connection_closed``.
* - ``node_id``
- string
- The ``src_node_id`` of this initiator Envoy instance, as configured in the ``rc://``
listener address.
* - ``cluster_id``
- string
- The ``src_cluster_id`` of this initiator Envoy instance, as configured in the ``rc://``
listener address.
* - ``tenant_id``
- string
- The ``src_tenant_id`` of this initiator Envoy instance, as configured in the ``rc://``
listener address. Empty if tenant isolation is not used.
* - ``upstream_cluster``
- string
- The name of the upstream cluster that this reverse tunnel connects to.
* - ``host_address``
- string
- The resolved address of the specific upstream host that this connection targets.
* - ``connection_key``
- string
- A unique identifier for this specific reverse tunnel connection instance. Useful for
correlating handshake and close events for the same connection.
* - ``error``
- string
- The error message describing the failure reason. Only present on ``handshake_failure``
events. Examples: ``HTTP handshake failed with status 401``,
``HTTP handshake encode failed``, ``Connection closed``.

In addition to dynamic metadata fields, standard Envoy access log format strings such as
``%START_TIME%``, ``%DURATION%``, and ``%CONNECTION_TERMINATION_DETAILS%`` are also available.

.. _config_reverse_connection_security:

Security considerations
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,16 @@ envoy_cc_library(
hdrs = ["reverse_tunnel_initiator_extension.h"],
visibility = ["//visibility:public"],
deps = [
"//envoy/access_log:access_log_interface",
"//envoy/server:bootstrap_extension_config_interface",
"//envoy/stats:stats_interface",
"//envoy/thread_local:thread_local_interface",
"//source/common/access_log:access_log_lib",
"//source/common/common:logger_lib",
"//source/common/stats:symbol_table_lib",
"//source/common/stream_info:stream_info_lib",
"//source/extensions/bootstrap/reverse_tunnel/common:reverse_connection_utility_lib",
"//source/server:generic_factory_context_lib",
"@envoy_api//envoy/extensions/bootstrap/reverse_tunnel/downstream_socket_interface/v3:pkg_cc_proto",
],
)
Expand All @@ -66,6 +70,7 @@ envoy_cc_library(
deps = [
":reverse_connection_address_lib",
":reverse_tunnel_extension_lib",
"//envoy/access_log:access_log_interface",
"//envoy/api:io_error_interface",
"//envoy/grpc:async_client_interface",
"//envoy/network:address_interface",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,19 @@ ReverseConnectionIOHandle::~ReverseConnectionIOHandle() {
cleanup();
}

void ReverseConnectionIOHandle::emitAccessLog(const std::string& event,
const std::string& host_address,
const std::string& cluster_name,
const std::string& connection_key,
const std::string& error_message) {
if (!extension_) {
return;
}
extension_->emitAccessLog(getTimeSource(), event, config_.src_node_id, config_.src_cluster_id,
config_.src_tenant_id, cluster_name, host_address, connection_key,
error_message);
}

void ReverseConnectionIOHandle::cleanup() {
ENVOY_LOG_MISC(debug, "Starting cleanup of reverse connection resources.");

Expand Down Expand Up @@ -827,6 +840,8 @@ void ReverseConnectionIOHandle::onDownstreamConnectionClosed(const std::string&
// Remove connection state tracking.
removeConnectionState(host_address, cluster_name, connection_key);

emitAccessLog("connection_closed", host_address, cluster_name, connection_key, "");

// The next call to maintainClusterConnections() will detect the missing connection
// and re-initiate it automatically.
ENVOY_LOG(debug,
Expand Down Expand Up @@ -1140,6 +1155,8 @@ void ReverseConnectionIOHandle::onConnectionDone(const std::string& error,

trackConnectionFailure(host_address, cluster_name);

emitAccessLog("handshake_failure", host_address, cluster_name, connection_key, error);

} else {
// Handle connection success.
ENVOY_LOG(debug, "reverse_tunnel: Connection succeeded for host {}", host_address);
Expand All @@ -1148,6 +1165,8 @@ void ReverseConnectionIOHandle::onConnectionDone(const std::string& error,
updateConnectionState(host_address, cluster_name, connection_key,
ReverseConnectionState::Connected);

emitAccessLog("handshake_success", host_address, cluster_name, connection_key, "");

// Only proceed if connection is still valid.
if (!connection) {
ENVOY_LOG(error, "reverse_tunnel: Cannot complete successful handshake - connection is null");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#include <string>
#include <vector>

#include "envoy/access_log/access_log.h"
#include "envoy/network/io_handle.h"
#include "envoy/network/socket.h"
#include "envoy/stats/scope.h"
Expand Down Expand Up @@ -354,6 +355,21 @@ class ReverseConnectionIOHandle : public Network::IoSocketHandleImpl,
const std::string& host_address,
Upstream::HostConstSharedPtr host);

/**
* Emit an access log entry for a reverse tunnel lifecycle event.
* Creates an ephemeral StreamInfo populated with dynamic metadata containing
* reverse tunnel identifiers and event details.
* @param event the lifecycle event name (e.g., "handshake_success", "handshake_failure",
* "connection_closed")
* @param host_address the address of the remote host
* @param cluster_name the name of the upstream cluster
* @param connection_key the unique key identifying the connection
* @param error_message the error message (empty on success)
*/
void emitAccessLog(const std::string& event, const std::string& host_address,
const std::string& cluster_name, const std::string& connection_key,
const std::string& error_message);

/**
* Clean up all reverse connection resources.
* Called during shutdown to properly close connections and free resources.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@
#include "envoy/stats/stats_macros.h"
#include "envoy/thread_local/thread_local.h"

#include "source/common/access_log/access_log_impl.h"
#include "source/common/common/logger.h"
#include "source/common/stats/symbol_table.h"
#include "source/common/stats/utility.h"
#include "source/common/stream_info/stream_info_impl.h"
#include "source/server/generic_factory_context.h"

namespace Envoy {
namespace Extensions {
Expand All @@ -17,6 +20,68 @@ namespace ReverseConnection {
// Static warning flag for reverse tunnel detailed stats activation.
static bool reverse_tunnel_detailed_stats_warning_logged = false;

ReverseTunnelInitiatorExtension::ReverseTunnelInitiatorExtension(
Server::Configuration::ServerFactoryContext& context,
const envoy::extensions::bootstrap::reverse_tunnel::downstream_socket_interface::v3::
DownstreamReverseConnectionSocketInterface& config)
: context_(context), config_(config) {
stat_prefix_ = PROTOBUF_GET_STRING_OR_DEFAULT(config, stat_prefix, "reverse_tunnel_initiator");
// Configure detailed stats flag (defaults to false).
enable_detailed_stats_ = config.enable_detailed_stats();
if (config.has_http_handshake() && !config.http_handshake().request_path().empty()) {
handshake_request_path_ = config.http_handshake().request_path();
} else {
handshake_request_path_ =
std::string(ReverseConnectionUtility::DEFAULT_REVERSE_TUNNEL_REQUEST_PATH);
}
// Instantiate access loggers from config.
Server::GenericFactoryContextImpl generic_context(context, context.scope(),
context.messageValidationVisitor());
for (const auto& log_config : config.access_log()) {
access_logs_.emplace_back(AccessLog::AccessLogFactory::fromProto(log_config, generic_context));
}

ENVOY_LOG(debug,
"ReverseTunnelInitiatorExtension: creating downstream reverse connection "
"socket interface with stat_prefix: {}, access_logs: {}",
stat_prefix_, access_logs_.size());
}

void ReverseTunnelInitiatorExtension::emitAccessLog(
TimeSource& time_source, const std::string& event, const std::string& node_id,
const std::string& cluster_id, const std::string& tenant_id,
const std::string& upstream_cluster, const std::string& host_address,
const std::string& connection_key, const std::string& error_message) {
if (access_logs_.empty()) {
return;
}

// Create an ephemeral StreamInfo for this log entry.
StreamInfo::StreamInfoImpl stream_info(time_source, nullptr,
StreamInfo::FilterState::LifeSpan::Connection);

// Populate dynamic metadata with reverse tunnel identifiers and event info.
Protobuf::Struct metadata;
auto& fields = *metadata.mutable_fields();
fields["event"].set_string_value(event);
fields["node_id"].set_string_value(node_id);
fields["cluster_id"].set_string_value(cluster_id);
fields["tenant_id"].set_string_value(tenant_id);
fields["upstream_cluster"].set_string_value(upstream_cluster);
fields["host_address"].set_string_value(host_address);
fields["connection_key"].set_string_value(connection_key);
if (!error_message.empty()) {
fields["error"].set_string_value(error_message);
}
Comment on lines +73 to +75
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The error field is conditionally added to the metadata. To ensure consistent schema for log consumers, it is better to always include the field, setting it to an empty string if no error is present.

fields["error"].set_string_value(error_message.empty() ? "" : error_message);

stream_info.setDynamicMetadata("envoy.reverse_tunnel.initiator", metadata);

const Formatter::Context log_context{
nullptr, nullptr, nullptr, {}, AccessLog::AccessLogType::NotSet};
for (const auto& access_log : access_logs_) {
access_log->log(log_context, stream_info);
}
}

// ReverseTunnelInitiatorExtension implementation
void ReverseTunnelInitiatorExtension::onServerInitialized(Server::Instance&) {
ENVOY_LOG(debug, "ReverseTunnelInitiatorExtension::onServerInitialized");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
#include <memory>
#include <string>

#include "envoy/access_log/access_log.h"
#include "envoy/extensions/bootstrap/reverse_tunnel/downstream_socket_interface/v3/downstream_reverse_connection_socket_interface.pb.h"
#include "envoy/extensions/bootstrap/reverse_tunnel/downstream_socket_interface/v3/downstream_reverse_connection_socket_interface.pb.validate.h"
#include "envoy/server/bootstrap_extension_config.h"
Expand Down Expand Up @@ -33,22 +34,7 @@ class ReverseTunnelInitiatorExtension : public Server::BootstrapExtension,
ReverseTunnelInitiatorExtension(
Server::Configuration::ServerFactoryContext& context,
const envoy::extensions::bootstrap::reverse_tunnel::downstream_socket_interface::v3::
DownstreamReverseConnectionSocketInterface& config)
: context_(context), config_(config) {
stat_prefix_ = PROTOBUF_GET_STRING_OR_DEFAULT(config, stat_prefix, "reverse_tunnel_initiator");
// Configure detailed stats flag (defaults to false).
enable_detailed_stats_ = config.enable_detailed_stats();
if (config.has_http_handshake() && !config.http_handshake().request_path().empty()) {
handshake_request_path_ = config.http_handshake().request_path();
} else {
handshake_request_path_ =
std::string(ReverseConnectionUtility::DEFAULT_REVERSE_TUNNEL_REQUEST_PATH);
}
ENVOY_LOG(debug,
"ReverseTunnelInitiatorExtension: creating downstream reverse connection "
"socket interface with stat_prefix: {}",
stat_prefix_);
}
DownstreamReverseConnectionSocketInterface& config);

void onServerInitialized(Server::Instance&) override;
void onWorkerThreadInitialized() override;
Expand Down Expand Up @@ -105,6 +91,30 @@ class ReverseTunnelInitiatorExtension : public Server::BootstrapExtension,
*/
const std::string& handshakeRequestPath() const { return handshake_request_path_; }

/**
* @return reference to the configured access loggers for reverse tunnel lifecycle events.
*/
const AccessLog::InstanceSharedPtrVector& accessLogs() const { return access_logs_; }

/**
* Emit an access log entry for a reverse tunnel lifecycle event.
* Creates an ephemeral StreamInfo populated with dynamic metadata containing
* reverse tunnel identifiers and event details.
* @param time_source the time source for the stream info
* @param event the lifecycle event name
* @param node_id the node identifier of this Envoy instance
* @param cluster_id the cluster identifier of this Envoy instance
* @param tenant_id the tenant identifier of this Envoy instance
* @param upstream_cluster the name of the upstream cluster
* @param host_address the address of the remote host
* @param connection_key the unique key identifying the connection
* @param error_message the error message (empty on success)
*/
void emitAccessLog(TimeSource& time_source, const std::string& event, const std::string& node_id,
const std::string& cluster_id, const std::string& tenant_id,
const std::string& upstream_cluster, const std::string& host_address,
const std::string& connection_key, const std::string& error_message);

/**
* Increment handshake stats for reverse tunnel connections (per-worker only).
* Only tracks stats if enable_detailed_stats flag is true.
Expand Down Expand Up @@ -134,6 +144,7 @@ class ReverseTunnelInitiatorExtension : public Server::BootstrapExtension,
std::string stat_prefix_; // Reverse connection stats prefix
bool enable_detailed_stats_{false};
std::string handshake_request_path_;
AccessLog::InstanceSharedPtrVector access_logs_;

/**
* Update per-worker connection stats for debugging purposes.
Expand Down
Loading
Loading