Production Workload Level 1 Audit
Generated 2026-09-16T14:12:43+00:00 · lookback 2d · version 0.1.1
Interpretation: this report is a screening and evidence report. Thresholds identify workloads for Level 2 diagnostics; they do not establish recommended production values.
5
Total workloads
0
Priority review
1
Suspicious
4
Evidence gap only
0
No immediate concern
| Workload | Owner | Triage | Next level | HPA min/max | CPU p90 / p95 / p99 | Startup p95 | Top findings |
|---|---|---|---|---|---|---|---|
| debezium/courses-debezium-v2 Deployment | revex | SUSPICIOUS | Level 2 | — / — | — / — / — | — | [REVIEW] HPA: No HPA targets this workload. [REVIEW] Probes: One or more application containers have no readiness probe. [EVIDENCE_GAP] MCP: Required observability signals were not found for this workload. +24 more |
| default/affiliate-portal-api Deployment | — | EVIDENCE GAP | Fix telemetry / reassess | 2 / 20 | — / — / — | — | [EVIDENCE_GAP] MCP: Required observability signals were not found for this workload. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_max' failed. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_min' failed. +23 more |
| default/agency-dashboard-api Deployment | — | EVIDENCE GAP | Fix telemetry / reassess | 2 / 50 | — / — / — | — | [EVIDENCE_GAP] MCP: Required observability signals were not found for this workload. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_max' failed. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_min' failed. +23 more |
| default/agent-deployment-api Deployment | — | EVIDENCE GAP | Fix telemetry / reassess | 60 / 100 | — / — / — | — | [EVIDENCE_GAP] MCP: Required observability signals were not found for this workload. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_max' failed. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_min' failed. +23 more |
| default/agent-execution-api Deployment | — | EVIDENCE GAP | Fix telemetry / reassess | 5 / 20 | — / — / — | — | [EVIDENCE_GAP] MCP: Required observability signals were not found for this workload. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_max' failed. [EVIDENCE_GAP] MCP: Runtime query 'available_replicas_min' failed. +23 more |
debezium/courses-debezium-v2 SUSPICIOUS
Kind
Deployment
Deployment
Owner
revex
revex
Tier
—
—
Replicas now
configured=1, ready=1
configured=1, ready=1
HPA
None
None
HPA min/max
— / —
— / —
CPU request / limit
2.100 cores / 3.500 cores
2.100 cores / 3.500 cores
Memory request / limit
1.62 GiB / 2.75 GiB
1.62 GiB / 2.75 GiB
HPA metrics: —
HPA behavior: —
HPA ScalingLimited: False
Findings and Level 2 branches
| Severity | Area | Finding | Evidence | Next step |
|---|---|---|---|---|
| REVIEW | HPA | No HPA targets this workload. | Configured replicas: 1 | Confirm fixed capacity is intentional; otherwise review demand signal and autoscaling design. |
| REVIEW | Probes | One or more application containers have no readiness probe. | cloud-sql-proxy, debezium-v2 | Validate when a new pod becomes safe to route and add an appropriate readiness signal if needed. |
| EVIDENCE_GAP | MCP | Required observability signals were not found for this workload. | cpu_usage, memory_working_set, deployment_available_replicas, pod_restarts, pod_lifecycle, request_rate, request_latency | Validate metric names and labels, then add or repair coverage before final workload tuning. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_max' failed. | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_min' failed. | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_zero_pct' failed. | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_max_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p90_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p95_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p99_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_schedule_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'initialize_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'latency_p99_seconds' failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_max_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p90_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p95_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p99_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'oomkilled_observed' failed. | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_pct' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'restart_increase' failed. | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'schedule_to_initialize_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'latency_p99_seconds' query failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_pct' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_rps' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
Container resources and probes
| Container | Role | CPU req/limit | Memory req/limit | Readiness | Liveness | Startup |
|---|---|---|---|---|---|---|
| cloud-sql-proxy | Application | 100m / 500m | 128Mi / 256Mi | Missing | Missing | Missing |
| debezium-v2 | Application | 2 / 3 | 1536Mi / 2560Mi | Missing | Missing | Missing |
Runtime query evidence
| Signal | Value | Series | Error | Query |
|---|---|---|---|---|
| available_replicas_max | — | 0 | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_min | — | 0 | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | min_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_zero_pct | — | 0 | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | 100 *
avg_over_time(
(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)
== bool 0
)[${lookback}:${step}]
)
|
| cpu_max_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p90_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p95_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p99_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| create_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| create_to_schedule_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| initialize_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| latency_p99_seconds | — | 0 | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
| memory_max_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p90_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p95_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p99_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| oomkilled_observed | — | 0 | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | max(
max_over_time(
kube_pod_container_status_last_terminated_reason{
cluster="${cluster}",
pod=~"${pod_regex}",
container!="",
reason="OOMKilled"
}[${lookback}]
)
)
|
| peak_5xx_pct | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
rate(
http_requests_total{
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)[${lookback}:${step}]
)
|
| restart_increase | — | 0 | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | sum(
increase(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| schedule_to_initialize_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
MCP coverage checks
| Signal | Required | Status | Series | Error | Query |
|---|---|---|---|---|---|
| cpu_usage | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| memory_working_set | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| deployment_available_replicas | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" }[${lookback}] ) ) | count(
count_over_time(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}[${lookback}]
)
)
|
| pod_restarts | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| pod_lifecycle | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_rate | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( http_requests_total{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_requests_total{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_latency | Yes | Missing | 0 | Unresolved query placeholders ['container'] in: count( count_over_time( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[${lookback}]
)
)
|
Custom application signals
| Signal | Value | Triggered | Error | Query |
|---|---|---|---|---|
| peak_5xx_rps | rps | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_pct | percent | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| latency_p99_seconds | seconds | No | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
default/affiliate-portal-api EVIDENCE GAP
Kind
Deployment
Deployment
Owner
—
—
Tier
t1
t1
Replicas now
configured=2, ready=2
configured=2, ready=2
HPA
affiliate-portal-api
affiliate-portal-api
HPA min/max
2 / 20
2 / 20
CPU request / limit
1.000 cores / 1.100 cores
1.000 cores / 1.100 cores
Memory request / limit
1.00 GiB / 1.10 GiB
1.00 GiB / 1.10 GiB
HPA metrics: Resource/cpu: Utilization 65%; ContainerResource/affiliate-portal-api/cpu: Utilization 65%; Resource/memory: Utilization 65%; ContainerResource/affiliate-portal-api/memory: Utilization 65%
HPA behavior: Default controller behavior
HPA ScalingLimited: True TooFewReplicas the desired replica count is less than the minimum replica count
Findings and Level 2 branches
| Severity | Area | Finding | Evidence | Next step |
|---|---|---|---|---|
| EVIDENCE_GAP | MCP | Required observability signals were not found for this workload. | cpu_usage, memory_working_set, deployment_available_replicas, pod_restarts, pod_lifecycle, request_rate, request_latency | Validate metric names and labels, then add or repair coverage before final workload tuning. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_max' failed. | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_min' failed. | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_zero_pct' failed. | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_max_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p90_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p95_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p99_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_schedule_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'initialize_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'latency_p99_seconds' failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_max_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p90_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p95_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p99_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'oomkilled_observed' failed. | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_pct' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'restart_increase' failed. | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'schedule_to_initialize_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'latency_p99_seconds' query failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_pct' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_rps' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| INFO | HPA | HPA is currently limited by its configured minimum replica count. | TooFewReplicas: the desired replica count is less than the minimum replica count | No capacity action is required unless the configured minimum is unintended or creates an efficiency concern. |
Container resources and probes
| Container | Role | CPU req/limit | Memory req/limit | Readiness | Liveness | Startup |
|---|---|---|---|---|---|---|
| affiliate-portal-api | Application | 1 / 1100m | 1Gi / 1126Mi | HTTP /:6050; initial=60s, period=10s, timeout=2s, failureThreshold=2 | HTTP /:6050; initial=60s, period=20s, timeout=2s, failureThreshold=3 | HTTP /:6050; initial=5s, period=5s, timeout=2s, failureThreshold=30 |
Runtime query evidence
| Signal | Value | Series | Error | Query |
|---|---|---|---|---|
| available_replicas_max | — | 0 | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_min | — | 0 | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | min_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_zero_pct | — | 0 | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | 100 *
avg_over_time(
(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)
== bool 0
)[${lookback}:${step}]
)
|
| cpu_max_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p90_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p95_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p99_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| create_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| create_to_schedule_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| initialize_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| latency_p99_seconds | — | 0 | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
| memory_max_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p90_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p95_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p99_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| oomkilled_observed | — | 0 | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | max(
max_over_time(
kube_pod_container_status_last_terminated_reason{
cluster="${cluster}",
pod=~"${pod_regex}",
container!="",
reason="OOMKilled"
}[${lookback}]
)
)
|
| peak_5xx_pct | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
rate(
http_requests_total{
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)[${lookback}:${step}]
)
|
| restart_increase | — | 0 | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | sum(
increase(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| schedule_to_initialize_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
MCP coverage checks
| Signal | Required | Status | Series | Error | Query |
|---|---|---|---|---|---|
| cpu_usage | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| memory_working_set | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| deployment_available_replicas | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" }[${lookback}] ) ) | count(
count_over_time(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}[${lookback}]
)
)
|
| pod_restarts | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| pod_lifecycle | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_rate | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( http_requests_total{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_requests_total{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_latency | Yes | Missing | 0 | Unresolved query placeholders ['container'] in: count( count_over_time( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[${lookback}]
)
)
|
Custom application signals
| Signal | Value | Triggered | Error | Query |
|---|---|---|---|---|
| peak_5xx_rps | rps | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_pct | percent | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| latency_p99_seconds | seconds | No | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
default/agency-dashboard-api EVIDENCE GAP
Kind
Deployment
Deployment
Owner
—
—
Tier
t1
t1
Replicas now
configured=2, ready=2
configured=2, ready=2
HPA
agency-dashboard-api
agency-dashboard-api
HPA min/max
2 / 50
2 / 50
CPU request / limit
1.000 cores / 1.100 cores
1.000 cores / 1.100 cores
Memory request / limit
1.50 GiB / 1.65 GiB
1.50 GiB / 1.65 GiB
HPA metrics: Resource/cpu: Utilization 60%; ContainerResource/agency-dashboard-api/cpu: Utilization 60%; Resource/memory: Utilization 60%; ContainerResource/agency-dashboard-api/memory: Utilization 60%
HPA behavior: Default controller behavior
HPA ScalingLimited: True TooFewReplicas the desired replica count is less than the minimum replica count
Findings and Level 2 branches
| Severity | Area | Finding | Evidence | Next step |
|---|---|---|---|---|
| EVIDENCE_GAP | MCP | Required observability signals were not found for this workload. | cpu_usage, memory_working_set, deployment_available_replicas, pod_restarts, pod_lifecycle, request_rate, request_latency | Validate metric names and labels, then add or repair coverage before final workload tuning. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_max' failed. | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_min' failed. | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_zero_pct' failed. | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_max_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p90_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p95_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p99_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_schedule_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'initialize_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'latency_p99_seconds' failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_max_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p90_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p95_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p99_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'oomkilled_observed' failed. | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_pct' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'restart_increase' failed. | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'schedule_to_initialize_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'latency_p99_seconds' query failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_pct' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_rps' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| INFO | HPA | HPA is currently limited by its configured minimum replica count. | TooFewReplicas: the desired replica count is less than the minimum replica count | No capacity action is required unless the configured minimum is unintended or creates an efficiency concern. |
Container resources and probes
| Container | Role | CPU req/limit | Memory req/limit | Readiness | Liveness | Startup |
|---|---|---|---|---|---|---|
| agency-dashboard-api | Application | 1 / 1100m | 1536Mi / 1690Mi | HTTP /:6030; initial=60s, period=10s, timeout=2s, failureThreshold=2 | HTTP /:6030; initial=60s, period=20s, timeout=2s, failureThreshold=3 | HTTP /:6030; initial=5s, period=5s, timeout=2s, failureThreshold=30 |
Runtime query evidence
| Signal | Value | Series | Error | Query |
|---|---|---|---|---|
| available_replicas_max | — | 0 | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_min | — | 0 | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | min_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_zero_pct | — | 0 | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | 100 *
avg_over_time(
(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)
== bool 0
)[${lookback}:${step}]
)
|
| cpu_max_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p90_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p95_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p99_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| create_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| create_to_schedule_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| initialize_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| latency_p99_seconds | — | 0 | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
| memory_max_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p90_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p95_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p99_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| oomkilled_observed | — | 0 | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | max(
max_over_time(
kube_pod_container_status_last_terminated_reason{
cluster="${cluster}",
pod=~"${pod_regex}",
container!="",
reason="OOMKilled"
}[${lookback}]
)
)
|
| peak_5xx_pct | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
rate(
http_requests_total{
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)[${lookback}:${step}]
)
|
| restart_increase | — | 0 | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | sum(
increase(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| schedule_to_initialize_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
MCP coverage checks
| Signal | Required | Status | Series | Error | Query |
|---|---|---|---|---|---|
| cpu_usage | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| memory_working_set | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| deployment_available_replicas | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" }[${lookback}] ) ) | count(
count_over_time(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}[${lookback}]
)
)
|
| pod_restarts | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| pod_lifecycle | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_rate | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( http_requests_total{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_requests_total{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_latency | Yes | Missing | 0 | Unresolved query placeholders ['container'] in: count( count_over_time( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[${lookback}]
)
)
|
Custom application signals
| Signal | Value | Triggered | Error | Query |
|---|---|---|---|---|
| peak_5xx_rps | rps | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_pct | percent | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| latency_p99_seconds | seconds | No | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
default/agent-deployment-api EVIDENCE GAP
Kind
Deployment
Deployment
Owner
—
—
Tier
t1
t1
Replicas now
configured=60, ready=60
configured=60, ready=60
HPA
agent-deployment-api
agent-deployment-api
HPA min/max
60 / 100
60 / 100
CPU request / limit
1.000 cores / 1.100 cores
1.000 cores / 1.100 cores
Memory request / limit
1.00 GiB / 1.10 GiB
1.00 GiB / 1.10 GiB
HPA metrics: Resource/cpu: Utilization 65%; ContainerResource/agent-deployment-api/cpu: Utilization 65%
HPA behavior: Default controller behavior
HPA ScalingLimited: True TooFewReplicas the desired replica count is less than the minimum replica count
Findings and Level 2 branches
| Severity | Area | Finding | Evidence | Next step |
|---|---|---|---|---|
| EVIDENCE_GAP | MCP | Required observability signals were not found for this workload. | cpu_usage, memory_working_set, deployment_available_replicas, pod_restarts, pod_lifecycle, request_rate, request_latency | Validate metric names and labels, then add or repair coverage before final workload tuning. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_max' failed. | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_min' failed. | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_zero_pct' failed. | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_max_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p90_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p95_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p99_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_schedule_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'initialize_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'latency_p99_seconds' failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_max_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p90_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p95_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p99_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'oomkilled_observed' failed. | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_pct' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'restart_increase' failed. | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'schedule_to_initialize_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'latency_p99_seconds' query failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_pct' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_rps' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| INFO | HPA | HPA is currently limited by its configured minimum replica count. | TooFewReplicas: the desired replica count is less than the minimum replica count | No capacity action is required unless the configured minimum is unintended or creates an efficiency concern. |
Container resources and probes
| Container | Role | CPU req/limit | Memory req/limit | Readiness | Liveness | Startup |
|---|---|---|---|---|---|---|
| agent-deployment-api | Application | 1 / 1100m | 1Gi / 1126Mi | HTTP /:6050; initial=5s, period=5s, timeout=2s, failureThreshold=2 | HTTP /:6050; initial=5s, period=10s, timeout=2s, failureThreshold=3 | HTTP /:6050; initial=5s, period=5s, timeout=2s, failureThreshold=30 |
Runtime query evidence
| Signal | Value | Series | Error | Query |
|---|---|---|---|---|
| available_replicas_max | — | 0 | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_min | — | 0 | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | min_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_zero_pct | — | 0 | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | 100 *
avg_over_time(
(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)
== bool 0
)[${lookback}:${step}]
)
|
| cpu_max_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p90_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p95_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p99_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| create_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| create_to_schedule_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| initialize_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| latency_p99_seconds | — | 0 | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
| memory_max_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p90_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p95_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p99_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| oomkilled_observed | — | 0 | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | max(
max_over_time(
kube_pod_container_status_last_terminated_reason{
cluster="${cluster}",
pod=~"${pod_regex}",
container!="",
reason="OOMKilled"
}[${lookback}]
)
)
|
| peak_5xx_pct | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
rate(
http_requests_total{
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)[${lookback}:${step}]
)
|
| restart_increase | — | 0 | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | sum(
increase(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| schedule_to_initialize_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
MCP coverage checks
| Signal | Required | Status | Series | Error | Query |
|---|---|---|---|---|---|
| cpu_usage | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| memory_working_set | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| deployment_available_replicas | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" }[${lookback}] ) ) | count(
count_over_time(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}[${lookback}]
)
)
|
| pod_restarts | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| pod_lifecycle | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_rate | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( http_requests_total{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_requests_total{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_latency | Yes | Missing | 0 | Unresolved query placeholders ['container'] in: count( count_over_time( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[${lookback}]
)
)
|
Custom application signals
| Signal | Value | Triggered | Error | Query |
|---|---|---|---|---|
| peak_5xx_rps | rps | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_pct | percent | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| latency_p99_seconds | seconds | No | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
default/agent-execution-api EVIDENCE GAP
Kind
Deployment
Deployment
Owner
—
—
Tier
t1
t1
Replicas now
configured=5, ready=5
configured=5, ready=5
HPA
agent-execution-api
agent-execution-api
HPA min/max
5 / 20
5 / 20
CPU request / limit
2.000 cores / 2.200 cores
2.000 cores / 2.200 cores
Memory request / limit
4.00 GiB / 4.40 GiB
4.00 GiB / 4.40 GiB
HPA metrics: Resource/cpu: Utilization 75%; ContainerResource/agent-execution-api/cpu: Utilization 75%; Resource/memory: Utilization 75%; ContainerResource/agent-execution-api/memory: Utilization 75%
HPA behavior: Default controller behavior
HPA ScalingLimited: True TooFewReplicas the desired replica count is less than the minimum replica count
Findings and Level 2 branches
| Severity | Area | Finding | Evidence | Next step |
|---|---|---|---|---|
| EVIDENCE_GAP | MCP | Required observability signals were not found for this workload. | cpu_usage, memory_working_set, deployment_available_replicas, pod_restarts, pod_lifecycle, request_rate, request_latency | Validate metric names and labels, then add or repair coverage before final workload tuning. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_max' failed. | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_min' failed. | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'available_replicas_zero_pct' failed. | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_max_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p90_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p95_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'cpu_p99_cores' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'create_to_schedule_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'initialize_to_ready_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'latency_p99_seconds' failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_max_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p90_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p95_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'memory_p99_bytes' failed. | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'oomkilled_observed' failed. | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_pct' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_5xx_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'peak_rps' failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'restart_increase' failed. | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP | Runtime query 'schedule_to_initialize_p95_seconds' failed. | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | Correct the query, datasource access, or metric labels before drawing conclusions from this signal. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'latency_p99_seconds' query failed. | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_pct' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| EVIDENCE_GAP | MCP / Availability | Custom signal 'peak_5xx_rps' query failed. | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | Correct the query or datasource access. |
| INFO | HPA | HPA is currently limited by its configured minimum replica count. | TooFewReplicas: the desired replica count is less than the minimum replica count | No capacity action is required unless the configured minimum is unintended or creates an efficiency concern. |
Container resources and probes
| Container | Role | CPU req/limit | Memory req/limit | Readiness | Liveness | Startup |
|---|---|---|---|---|---|---|
| agent-execution-api | Application | 2 / 2200m | 4Gi / 4506Mi | HTTP /:6050; initial=5s, period=5s, timeout=2s, failureThreshold=2 | HTTP /:6050; initial=5s, period=10s, timeout=2s, failureThreshold=3 | HTTP /:6050; initial=5s, period=5s, timeout=2s, failureThreshold=30 |
Runtime query evidence
| Signal | Value | Series | Error | Query |
|---|---|---|---|---|
| available_replicas_max | — | 0 | Unresolved query placeholders ['cluster'] in: max_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_min | — | 0 | Unresolved query placeholders ['cluster'] in: min_over_time( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) )[${lookback}:${step}] ) | min_over_time(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)[${lookback}:${step}]
)
|
| available_replicas_zero_pct | — | 0 | Unresolved query placeholders ['cluster'] in: 100 * avg_over_time( ( ( sum( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" } ) ) == bool 0 )[${lookback}:${step}] ) | 100 *
avg_over_time(
(
(
sum(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}
)
)
== bool 0
)[${lookback}:${step}]
)
|
| cpu_max_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p90_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p95_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| cpu_p99_cores | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| create_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| create_to_schedule_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_created{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_created{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| initialize_to_ready_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
| latency_p99_seconds | — | 0 | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|
| memory_max_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( max_over_time( ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
max_over_time(
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p90_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.90, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.90,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p95_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.95, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.95,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| memory_p99_bytes | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max( quantile_over_time( 0.99, ( sum by (pod) ( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" } ) )[${lookback}:${step}] ) ) | max(
quantile_over_time(
0.99,
(
sum by (pod) (
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}
)
)[${lookback}:${step}]
)
)
|
| oomkilled_observed | — | 0 | Unresolved query placeholders ['cluster'] in: max( max_over_time( kube_pod_container_status_last_terminated_reason{ cluster="${cluster}", pod=~"${pod_regex}", container!="", reason="OOMKilled" }[${lookback}] ) ) | max(
max_over_time(
kube_pod_container_status_last_terminated_reason{
cluster="${cluster}",
pod=~"${pod_regex}",
container!="",
reason="OOMKilled"
}[${lookback}]
)
)
|
| peak_5xx_pct | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_rps | — | 0 | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( sum( rate( http_requests_total{ container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) )[${lookback}:${step}] ) | max_over_time(
(
sum(
rate(
http_requests_total{
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)[${lookback}:${step}]
)
|
| restart_increase | — | 0 | Unresolved query placeholders ['cluster'] in: sum( increase( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | sum(
increase(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| schedule_to_initialize_p95_seconds | — | 0 | Unresolved query placeholders ['cluster'] in: quantile( 0.95, max_over_time( ( kube_pod_status_initialized_time{ cluster="${cluster}", pod=~"${pod_regex}" } - on(cluster, namespace, pod) kube_pod_status_scheduled_time{ cluster="${cluster}", pod=~"${pod_regex}" } )[${lookback}:${step}] ) ) | quantile(
0.95,
max_over_time(
(
kube_pod_status_initialized_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
-
on(cluster, namespace, pod)
kube_pod_status_scheduled_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}
)[${lookback}:${step}]
)
)
|
MCP coverage checks
| Signal | Required | Status | Series | Error | Query |
|---|---|---|---|---|---|
| cpu_usage | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( sum:rate:container_cpu_usage_seconds_total_2m{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
sum:rate:container_cpu_usage_seconds_total_2m{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| memory_working_set | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( container_memory_working_set_bytes{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
container_memory_working_set_bytes{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| deployment_available_replicas | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_deployment_status_replicas_available{ cluster="${cluster}", namespace="${namespace}", deployment="${workload}" }[${lookback}] ) ) | count(
count_over_time(
kube_deployment_status_replicas_available{
cluster="${cluster}",
namespace="${namespace}",
deployment="${workload}"
}[${lookback}]
)
)
|
| pod_restarts | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_container_status_restarts_total{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_container_status_restarts_total{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| pod_lifecycle | Yes | Missing | 0 | Unresolved query placeholders ['cluster'] in: count( count_over_time( kube_pod_status_ready_time{ cluster="${cluster}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
kube_pod_status_ready_time{
cluster="${cluster}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_rate | Yes | Missing | 0 | Unresolved query placeholders ['cluster', 'container'] in: count( count_over_time( http_requests_total{ cluster="${cluster}", container="${container}", pod=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_requests_total{
cluster="${cluster}",
container="${container}",
pod=~"${pod_regex}"
}[${lookback}]
)
)
|
| request_latency | Yes | Missing | 0 | Unresolved query placeholders ['container'] in: count( count_over_time( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[${lookback}] ) ) | count(
count_over_time(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[${lookback}]
)
)
|
Custom application signals
| Signal | Value | Triggered | Error | Query |
|---|---|---|---|---|
| peak_5xx_rps | rps | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| peak_5xx_pct | percent | No | Unresolved query placeholders ['cluster', 'container'] in: max_over_time( ( ( 100 * sum( rate( http_requests_total{ status="5XX", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ) / clamp_min( sum( rate( http_requests_total{ status!="", container="${container}", cluster="${cluster}", pod=~"${pod_regex}" }[60s] ) ), 0.000001 ) ) or vector(0) )[${lookback}:${step}] ) | max_over_time(
(
(
100
*
sum(
rate(
http_requests_total{
status="5XX",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
)
/
clamp_min(
sum(
rate(
http_requests_total{
status!="",
container="${container}",
cluster="${cluster}",
pod=~"${pod_regex}"
}[60s]
)
),
0.000001
)
)
or
vector(0)
)[${lookback}:${step}]
)
|
| latency_p99_seconds | seconds | No | Unresolved query placeholders ['container'] in: max_over_time( ( histogram_quantile( 0.99, sum by (le) ( rate( http_request_duration_seconds_bucket{ container="${container}", pod_name=~"${pod_regex}" }[5m] ) ) ) )[${lookback}:${step}] ) | max_over_time(
(
histogram_quantile(
0.99,
sum by (le) (
rate(
http_request_duration_seconds_bucket{
container="${container}",
pod_name=~"${pod_regex}"
}[5m]
)
)
)
)[${lookback}:${step}]
)
|