[helm/charts] [stable/prometheus-operator] Missing TLS configuration for kubelet\'s resource endpoint (#23100)

From: notifications@github.com
Domain: IP info github.com
MX-server: IP info out-25.smtp.github.com
Size: 3611 Bytes
Create: 2020-07-07
Update: 2020-07-07
Score: 0
Safe: Yes

Outbound domains: github.com |

Describe the bug
Prometheus scraping requests to kubelet resource endpoint (metrics/resource/v1alpha1) for k8s 1.16 are rejected with:
Get "https://XXX.XX.XXX.XXX:10250/metrics/resource/v1alpha1": x509: cannot validate certificate for XXX.XX.XXX.XXX because it doesn't contain any IP SANs

Version of Helm and Kubernetes:
Helm: 2.16.9
Kubernetes: 1.16.11

Which chart: stable/prometheus-operator

What happened:
We have observed TargetDown alerts stemming from prometheus scraper not being able to scrape kubelet's resource metrics. All other kubelet metrics (/metrics, /metrics/cadvisor/, /metrics/probes) are scraped successfully over HTTPS.

After a quick look at kubelet's ServiceMonitor I've noticed that configuration of the resource endpoint is missing few properties that other endpoints have:

- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    path: /metrics/cadvisor
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    path: /metrics/probes
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - path: /metrics/resource/v1alpha1
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https

like:

bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

and

tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true

What you expected to happen:
The resource endpoint is scrapped successfully like other endpoints


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Want to protect your real email from messages like this? Use TempM email and be more secure on the internet.