ccproxy.observability¶
ccproxy.observability
¶
Observability module for the CCProxy API.
This module provides comprehensive observability capabilities including metrics collection, structured logging, request context tracking, and observability pipeline management.
The observability system follows a hybrid architecture that combines: - Real-time metrics collection and aggregation - Structured logging with correlation IDs - Request context propagation across service boundaries - Pluggable pipeline for metrics export and alerting
Components: - metrics: Core metrics collection, aggregation, and export functionality - logging: Structured logging configuration and context-aware loggers - context: Request context tracking and correlation across async operations - pipeline: Observability data pipeline for metrics export and alerting
RequestContext
dataclass
¶
RequestContext(
request_id,
start_time,
logger,
metadata=dict(),
storage=None,
)
PrometheusMetrics
¶
Prometheus metrics collector for operational monitoring.
Provides thread-safe, high-performance metrics collection using prometheus_client. Designed for minimal overhead in request processing hot paths.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
namespace
|
str
|
Metric name prefix |
'ccproxy'
|
registry
|
CollectorRegistry | None
|
Custom Prometheus registry (uses default if None) |
None
|
pushgateway_client
|
Any | None
|
Optional pushgateway client for dependency injection |
None
|
Source code in ccproxy/observability/metrics.py
record_request
¶
Record a request event.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
method
|
str
|
HTTP method (GET, POST, etc.) |
required |
endpoint
|
str
|
API endpoint path |
required |
model
|
str | None
|
Model name used |
None
|
status
|
str | int
|
Response status code or status string |
'unknown'
|
service_type
|
str | None
|
Service type (claude_sdk_service, proxy_service) |
None
|
Source code in ccproxy/observability/metrics.py
record_response_time
¶
Record response time.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
duration_seconds
|
float
|
Response time in seconds |
required |
model
|
str | None
|
Model name used |
None
|
endpoint
|
str
|
API endpoint |
'unknown'
|
service_type
|
str | None
|
Service type (claude_sdk_service, proxy_service) |
None
|
Source code in ccproxy/observability/metrics.py
record_tokens
¶
Record token usage.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
token_count
|
int
|
Number of tokens |
required |
token_type
|
str
|
Type of tokens (input, output, cache_read, cache_write) |
required |
model
|
str | None
|
Model name |
None
|
service_type
|
str | None
|
Service type (claude_sdk_service, proxy_service) |
None
|
Source code in ccproxy/observability/metrics.py
record_cost
¶
Record cost.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
cost_usd
|
float
|
Cost in USD |
required |
model
|
str | None
|
Model name |
None
|
cost_type
|
str
|
Type of cost (input, output, cache, total) |
'total'
|
service_type
|
str | None
|
Service type (claude_sdk_service, proxy_service) |
None
|
Source code in ccproxy/observability/metrics.py
record_error
¶
Record an error event.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
error_type
|
str
|
Type/name of error |
required |
endpoint
|
str
|
API endpoint where error occurred |
'unknown'
|
model
|
str | None
|
Model name if applicable |
None
|
service_type
|
str | None
|
Service type (claude_sdk_service, proxy_service) |
None
|
Source code in ccproxy/observability/metrics.py
set_active_requests
¶
Set the current number of active requests.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
count
|
int
|
Number of active requests |
required |
inc_active_requests
¶
dec_active_requests
¶
update_system_info
¶
is_enabled
¶
push_to_gateway
¶
Push current metrics to Pushgateway using official prometheus_client methods.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
method
|
str
|
Push method - "push" (replace), "pushadd" (add), or "delete" |
'push'
|
Returns:
Type | Description |
---|---|
bool
|
True if push succeeded, False otherwise |
Source code in ccproxy/observability/metrics.py
push_add_to_gateway
¶
Add current metrics to existing job/instance in Pushgateway (pushadd operation).
This is useful when you want to add metrics without replacing existing ones.
Returns:
Type | Description |
---|---|
bool
|
True if push succeeded, False otherwise |
Source code in ccproxy/observability/metrics.py
delete_from_gateway
¶
Delete all metrics for the configured job from Pushgateway.
This removes all metrics associated with the job, useful for cleanup.
Returns:
Type | Description |
---|---|
bool
|
True if delete succeeded, False otherwise |
Source code in ccproxy/observability/metrics.py
is_pushgateway_enabled
¶
Check if Pushgateway client is enabled and configured.
PushgatewayClient
¶
Prometheus Pushgateway client using official prometheus_client methods.
Supports standard pushgateway operations: - push_to_gateway(): Replace all metrics for job/instance - pushadd_to_gateway(): Add metrics to existing job/instance - delete_from_gateway(): Delete metrics for job/instance
Also supports VictoriaMetrics remote write protocol for compatibility.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
settings
|
ObservabilitySettings
|
Observability configuration settings |
required |
Source code in ccproxy/observability/pushgateway.py
push_metrics
¶
Push metrics to Pushgateway using official prometheus_client methods.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
registry
|
CollectorRegistry
|
Prometheus metrics registry to push |
required |
method
|
str
|
Push method - "push" (replace), "pushadd" (add), or "delete" |
'push'
|
Returns:
Type | Description |
---|---|
bool
|
True if push succeeded, False otherwise |
Source code in ccproxy/observability/pushgateway.py
push_add_metrics
¶
Add metrics to existing job/instance (pushadd operation).
Parameters:
Name | Type | Description | Default |
---|---|---|---|
registry
|
CollectorRegistry
|
Prometheus metrics registry to add |
required |
Returns:
Type | Description |
---|---|
bool
|
True if push succeeded, False otherwise |
Source code in ccproxy/observability/pushgateway.py
delete_metrics
¶
Delete all metrics for the configured job.
Returns:
Type | Description |
---|---|
bool
|
True if delete succeeded, False otherwise |
Source code in ccproxy/observability/pushgateway.py
get_context_tracker
¶
Get or create global context tracker.
request_context
async
¶
Context manager for tracking complete request lifecycle with timing.
Automatically logs request start/success/error events with accurate timing. Provides structured logging with request correlation.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
request_id
|
str | None
|
Unique request identifier (generated if not provided) |
None
|
storage
|
Any | None
|
Optional storage backend for access logs |
None
|
metrics
|
Any | None
|
Optional PrometheusMetrics instance for active request tracking |
None
|
**initial_context
|
Any
|
Initial context to include in all log events |
{}
|
Yields:
Name | Type | Description |
---|---|---|
RequestContext |
AsyncGenerator[RequestContext, None]
|
Context object with timing and logging capabilities |
Example
async with request_context(method="POST", path="/v1/messages") as ctx: ctx.add_metadata(model="claude-3-5-sonnet") # Process request ctx.log_event("request_processed", tokens=150) # Context automatically logs success with timing
Source code in ccproxy/observability/context.py
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
|
timed_operation
async
¶
Context manager for timing individual operations within a request.
Useful for measuring specific parts of request processing like API calls, database queries, or data processing steps.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
operation_name
|
str
|
Name of the operation being timed |
required |
request_id
|
str | None
|
Associated request ID for correlation |
None
|
**context
|
Any
|
Additional context for logging |
{}
|
Yields:
Type | Description |
---|---|
AsyncGenerator[dict[str, Any], None]
|
Dict with timing information and logger |
Example
async with timed_operation("claude_api_call", request_id=ctx.request_id) as op: response = await api_client.call() op["response_size"] = len(response) # Automatically logs operation timing
Source code in ccproxy/observability/context.py
182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 |
|
tracked_request_context
async
¶
Request context manager that also tracks active requests globally.
Combines request_context() with automatic tracking in the global context tracker for monitoring active request counts.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
request_id
|
str | None
|
Unique request identifier |
None
|
**initial_context
|
Any
|
Initial context to include in log events |
{}
|
Yields:
Name | Type | Description |
---|---|---|
RequestContext |
AsyncGenerator[RequestContext, None]
|
Context object with timing and logging |
Source code in ccproxy/observability/context.py
get_metrics
¶
Get or create global metrics instance with dependency injection.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
namespace
|
str
|
Metric namespace prefix |
'ccproxy'
|
registry
|
CollectorRegistry | None
|
Custom Prometheus registry |
None
|
pushgateway_client
|
Any | None
|
Optional pushgateway client for dependency injection |
None
|
settings
|
Any | None
|
Optional settings instance to avoid circular imports |
None
|
Returns:
Type | Description |
---|---|
PrometheusMetrics
|
PrometheusMetrics instance with full pushgateway support: |
PrometheusMetrics
|
|
PrometheusMetrics
|
|
PrometheusMetrics
|
|
Source code in ccproxy/observability/metrics.py
reset_metrics
¶
Reset global metrics instance (mainly for testing).
Source code in ccproxy/observability/metrics.py
get_pushgateway_client
¶
Get or create global pushgateway client instance.
Source code in ccproxy/observability/pushgateway.py
reset_pushgateway_client
¶
Reset global pushgateway client instance (mainly for testing).