Network Observability

Telegen provides deep network observability using eBPF.

Overview

Network observability includes:

DNS tracing - Query/response correlation
TCP metrics - RTT, retransmits, connection tracking
HTTP/gRPC tracing - Request/response details
Messaging protocol tracing - AMQP 0-9-1, AMQP 1.0, OpenWire, STOMP, Kafka, NATS, MQTT
Flow tracking - Connection topology
XDP packet analysis - High-performance packet inspection
Firewall/network infrastructure - Palo Alto PAN-OS, FortiGate FortiOS, Arista CloudVision, Cisco ACI
Per-connection statistics - Byte counters on TCP close

DNS Tracing

What’s Captured

Field	Description
Query	Domain name, type (A, AAAA, CNAME)
Response	Answer records, response code
Latency	Query-to-response time
Server	DNS server address

Sample Event

{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "attributes": {
    "dns.question.name": "api.example.com",
    "dns.question.type": "A",
    "dns.response_code": "NOERROR",
    "dns.answers": ["10.0.1.100", "10.0.1.101"],
    "dns.latency_ms": 2.5,
    "net.peer.ip": "10.0.0.2",
    "net.peer.port": 53,
    "process.pid": 12345,
    "k8s.pod.name": "my-app-xyz"
  }
}

Configuration

Network flow observability is a single switch. DNS capture is part of it and is not separately configurable.

ebpf:
  network:
    enabled: true

TCP Metrics

Metrics Collected

Metric	Description
`tcp_rtt_us`	Round-trip time in microseconds
`tcp_retransmits`	Packet retransmission count
`tcp_connections`	Connection count
`tcp_bytes_sent`	Bytes transmitted
`tcp_bytes_received`	Bytes received

Connection Tracking

# Metrics example
tcp_rtt_us{
  src_ip="10.0.1.50",
  dst_ip="10.0.2.100",
  dst_port="5432",
  k8s_src_pod="api-server",
  k8s_dst_service="postgres"
} 1250

tcp_retransmits_total{
  src_ip="10.0.1.50",
  dst_ip="10.0.2.100",
  dst_port="5432"
} 3

Configuration

TCP metrics are emitted whenever network observability is enabled. Per-metric toggles and connection sampling are not configurable.

ebpf:
  network:
    enabled: true

HTTP/gRPC Tracing

HTTP Details

Field	Description
`http.method`	GET, POST, PUT, DELETE, etc.
`http.url`	Full request URL
`http.route`	Matched route pattern
`http.status_code`	Response status
`http.request_content_length`	Request body size
`http.response_content_length`	Response body size

gRPC Details

Field	Description
`rpc.system`	grpc
`rpc.service`	Service name
`rpc.method`	Method name
`rpc.grpc.status_code`	gRPC status code

Configuration

Protocol coverage is selected through the traces exporter’s instrumentation list, and noisy endpoints are dropped with an attribute filter rather than a path list.

ebpf:
  network:
    enabled: true

  otel_traces_export:
    instrumentations:
      - http
      - grpc

  # Drop health and readiness probes
  filter:
    application:
      url.path:
        not_match: "/{health,healthz,ready,metrics,favicon.ico}*"

Service Topology

Telegen automatically builds a service dependency map:

        flowchart LR
    subgraph External
        LB["Load Balancer"]
    end
    
    subgraph Cluster["Kubernetes Cluster"]
        FE["Frontend"]
        API["API Gateway"]
        US["User Service"]
        OS["Order Service"]
        PG["PostgreSQL"]
        RD["Redis"]
        KF["Kafka"]
    end
    
    LB -->|HTTP| FE
    FE -->|HTTP| API
    API -->|gRPC| US
    API -->|gRPC| OS
    US -->|SQL| PG
    OS -->|SQL| PG
    API -->|TCP| RD
    OS -->|Produce| KF

Topology Data

topology:
  nodes:
    - id: "api-gateway"
      type: "service"
      attributes:
        k8s.deployment: "api-gateway"
        k8s.namespace: "default"
    
    - id: "user-service"
      type: "service"
      attributes:
        k8s.deployment: "user-service"
        k8s.namespace: "default"
  
  edges:
    - source: "api-gateway"
      target: "user-service"
      attributes:
        protocol: "grpc"
        requests_per_second: 150
        avg_latency_ms: 12
        error_rate: 0.01

XDP Packet Analysis

For high-performance packet inspection at the NIC level:

Configuration

Warning

XDP packet analysis has no configuration surface today. There is no xdp section; sample rate, interface selection, and packet filters are not configurable. Adding these keys stops the agent from starting.

Use Cases

DDoS detection - High packet rate anomalies
Protocol analysis - Non-HTTP traffic inspection
Network debugging - Low-level packet issues

Network Metrics

RED Metrics (Rate, Errors, Duration)

# Request rate by service
sum(rate(http_server_requests_total[5m])) by (service_name)

# Error rate
sum(rate(http_server_requests_total{status_code=~"5.."}[5m])) 
/ sum(rate(http_server_requests_total[5m]))

# Latency percentiles
histogram_quantile(0.99, 
  sum(rate(http_server_duration_bucket[5m])) by (le, service_name)
)

Connection Metrics

# Active connections by service pair
telegen_tcp_connections{state="established"}

# Connection errors
sum(rate(telegen_tcp_connection_errors_total[5m])) by (error_type)

# Retransmit rate
sum(rate(telegen_tcp_retransmits_total[5m])) 
/ sum(rate(telegen_tcp_segments_total[5m]))

DNS Metrics

# DNS query rate
sum(rate(telegen_dns_queries_total[5m])) by (domain)

# DNS latency
histogram_quantile(0.95, 
  sum(rate(telegen_dns_latency_bucket[5m])) by (le)
)

# DNS errors
sum(rate(telegen_dns_queries_total{response_code!="NOERROR"}[5m]))

Interface Filtering

Warning

Interface include/exclude lists have no configuration surface today. Network observability attaches to all interfaces.

Port Filtering

Ports are not filtered at capture time. Narrow the data instead by restricting which processes are instrumented, using discovery:

ebpf:
  discovery:
    instrument:
      - open_ports: "80,443,8080,3000,5432,6379"
    exclude_instrument:
      - open_ports: "22,2379,2380"

Network Security

Suspicious Connection Detection

Warning

Suspicious-connection detection has no configuration surface today. There is no network.security section; suspicious port lists, external-connection alerting, and IP blocklists are not configurable.

Example Alert

{
  "timestamp": "2024-01-15T10:30:00Z",
  "severity": "WARNING",
  "body": "Suspicious outbound connection to known bad IP",
  "attributes": {
    "network.event_type": "suspicious_connection",
    "net.peer.ip": "198.51.100.50",
    "net.peer.port": 4444,
    "process.pid": 12345,
    "process.executable.path": "/tmp/shell",
    "k8s.pod.name": "compromised-pod"
  }
}

Performance Considerations

Overhead

Feature	CPU Impact	Memory Impact
TCP metrics	~0.5%	10MB
DNS tracing	~0.2%	5MB
HTTP tracing	~1%	20MB
XDP (sampled)	~0.1%	5MB

Reducing Overhead

Shrink the BPF maps and drain events in larger, less frequent batches:

ebpf:
  tracer:
    maps_config:
      global_scale_factor: -1
    batch_length: 500
    batch_timeout: 5s

Best Practices

1. Filter Noisy Traffic

Drop health and readiness probes with an attribute filter:

ebpf:
  filter:
    application:
      url.path:
        not_match: "/{health,ready,metrics}*"

2. Narrow What Is Instrumented

For high-traffic environments, instrument fewer processes rather than sampling connections — there is no connection or packet sampling knob:

ebpf:
  discovery:
    exclude_instrument:
      - open_ports: "2379,10250"

3. Monitor Key Services

Focus on critical paths:

ebpf:
  discovery:
    instrument:
      - open_ports: "80,443,5432,6379"

Messaging Protocols

Telegen captures messaging traces at the eBPF level with no application code changes. Current wire-protocol coverage includes:

AMQP 0-9-1 (RabbitMQ semantics)
AMQP 1.0 (broker inferred from process/port hints)
OpenWire (ActiveMQ Classic)
STOMP
CQL (Cassandra) and NATS

AMQP-family tracing

Telegen emits spans from both kernel protocol hints and userspace fallback parsing for AMQP-family brokers. The span operation type is normalized (publish, receive, settle, create, process) while the raw protocol verb is preserved in messaging.operation.name.

What’s Captured

Field	Description
`messaging.system`	`rabbitmq`, `activemq`, `servicebus`, or `jms` (hint-resolved)
`messaging.operation.type`	`publish`, `receive`, `settle`, `create`, `process`
`messaging.operation.name`	Raw verb, for example `basic.publish`, `amqp1.transfer`, `openwire.message`, `stomp.send`
`messaging.destination.name`	Queue/topic/exchange-derived destination
`net.peer.ip` / `net.peer.port`	Broker address

Sample Span

{
  "name": "orders.created receive",
  "kind": "CONSUMER",
  "duration_ms": 0.8,
  "attributes": {
    "messaging.system": "activemq",
    "messaging.operation.type": "receive",
    "messaging.operation.name": "amqp1.transfer",
    "messaging.destination.name": "orders.created",
    "net.peer.ip": "10.0.2.50",
    "net.peer.port": 61616
  }
}

Configuration

ebpf:
  tracer:
    buffer_sizes:
      mq: 0   # shared large-buffer budget for AMQP/OpenWire/STOMP

Limits and caveats

Per-event in-kernel small buffers are capped (request 256 bytes, response 128 bytes).
Larger payload recovery depends on MQ large-buffer capture (ebpf.buffer_sizes.mq).
AMQP Go library coverage is available for github.com/rabbitmq/amqp091-go and github.com/streadway/amqp via uprobes; other language clients rely on TCP protocol parsing.

CQL (Cassandra) Tracing

Telegen parses the Cassandra Query Language binary protocol (CQL v3–v5) to capture query statements, keyspaces, batch operations, and prepared statement execution.

See Database Tracing for the full Cassandra tracing reference.

NATS Tracing

NATS is a lightweight, text-based publish/subscribe messaging system. Telegen captures PUB, MSG, and subscription operations from the NATS wire protocol.

What’s Captured

Field	Description
`messaging.system`	`nats`
`messaging.operation`	`publish` or `process`
`messaging.destination.name`	Subject name
`net.peer.ip` / `net.peer.port`	NATS server address

Sample Span

{
  "name": "sensor.readings publish",
  "kind": "PRODUCER",
  "duration_ms": 0.2,
  "attributes": {
    "messaging.system": "nats",
    "messaging.operation": "publish",
    "messaging.destination.name": "sensor.readings",
    "net.peer.ip": "10.0.3.10",
    "net.peer.port": 4222
  }
}

Configuration

NATS parsing is active whenever network observability is enabled. There is no per-protocol toggle.

ebpf:
  network:
    enabled: true

Connection Statistics

Telegen tracks byte-level connection statistics via TCP close events, providing a low-overhead measure of throughput per connection without full payload capture.

Metrics Emitted

Metric	Type	Labels	Description
`telegen.connection.bytes_sent`	Counter	src, dst, port	Bytes sent per connection lifetime
`telegen.connection.bytes_received`	Counter	src, dst, port	Bytes received per connection lifetime

These metrics are emitted when a TCP connection closes and complement the per-request span data produced by the protocol parsers.

Configuration

ebpf:
  network:
    enabled: true

Network Infrastructure Collection

Telegen collects metrics from network infrastructure devices via their REST APIs. This enables observability for firewalls, SDN controllers, and network appliances.

Supported Platforms

Vendor	Platform	Collects
Palo Alto	PAN-OS	System, interfaces (API key or username/password)
Fortinet	FortiGate FortiOS	System, interfaces (Bearer token)
Arista	CloudVision (CVP)	Inventory, interfaces, BGP, system
Cisco	ACI	Fabric health, node health, tenant health, interface stats

Configuration

netinfra:
  enabled: false
  collect_interval: 30s

  paloalto:
    - name: "pan-dc1"
      base_url: "https://10.10.10.20"
      api_key: "${PALOALTO_API_KEY}"
      verify_ssl: true
      timeout: 30s
      collect_interval: 30s
      collect: ["system", "interfaces"]
      labels:
        site: "dc1"
        team: "network"

  fortigate:
    - name: "fg-edge-1"
      base_url: "https://10.10.20.30"
      token: "${FORTIGATE_TOKEN}"
      verify_ssl: true
      timeout: 30s
      collect_interval: 30s
      collect: ["system", "interfaces"]
      labels:
        site: "edge"
        team: "security"

  cloudvision:
    - name: "cvp-prod"
      cvp_url: "https://cloudvision.example.com"
      token: "${ARISTA_CVP_TOKEN}"
      verify_ssl: true
      timeout: 30s
      collect_interval: 30s
      collect: ["inventory", "interfaces", "bgp", "system"]
      labels:
        site: "prod"

  aci:
    - name: "aci-fabric-a"
      apic_url: "https://apic.example.com"
      username: "${ACI_USERNAME}"
      password: "${ACI_PASSWORD}"
      verify_ssl: true
      timeout: 30s
      collect_interval: 30s
      collect: ["fabric_health", "node_health", "tenant_health", "interface_stats"]
      labels:
        site: "fabric-a"

Export Flow

Firewall metrics use the shared V3 pipeline metrics exporter (OTLP send_mode: failover / remoteWrite active). No private transport.

Runtime Notes

Use --mode collector or --mode unified for mixed black-box + host/eBPF collection
netinfra.enabled can remain false if you force mode via CLI and provide at least one target
If shared OTLP metrics exporter is unavailable at startup, Telegen logs degraded status and continues with other enabled sources

For full details, see Network Infra Firewalls.

Messaging Protocol Tracing

Telegen provides deep observability for message queue and event streaming platforms using eBPF protocol tracing. For full details, see Messaging Protocol Tracing.

Supported Messaging Protocols

Protocol	Version	Default System	Detection Method
Kafka	All	Kafka	Magic byte `0x00 0x00 0x00 0x00` + API key
AMQP 0-9-1	—	RabbitMQ	Preface `AMQP\x00\x00\x09\x01`
AMQP 1.0	—	ActiveMQ	Preface `AMQP\x00\x01\x00\x00`
OpenWire	—	ActiveMQ	“ActiveMQ” magic string in first 56 bytes
STOMP	1.0, 1.1, 1.2	ActiveMQ	Command matching (SEND, MESSAGE, etc.)
NATS	—	NATS	Text-based protocol detection
MQTT	3.1, 3.1.1, 5.0	MQTT	Fixed header detection

Messaging System Resolution

AMQP 1.0 is used by multiple brokers. Telegen resolves the correct messaging.system using hints:

Hint	Resolved System
`servicebus` / `azure-servicebus`	Azure Service Bus
`artemis` / `activemq` / `openwire` / `stomp`	ActiveMQ
`rabbit` / `beam.smp`	RabbitMQ
`qpid` / `solace`	JMS

Next Steps

Database Tracing - Deep database network tracing
Security Observability - Network security events
Agent Mode Configuration - Network configuration