How to add Cloudwatch metrics to Grafana

Hey guys, thanks for taking the time to read the following blog post. I hope it will benefit you in your Kubernetes learning path and help you create a positive impact in your organization.

Let's get into it…

As DevOps Engineers, making the workspace friendly for our developers, coworkers, and ourselves is essential. If we can reduce the number of tabs open to provide data of our application workload and infra, that’s a win-win situation for us. Integrating Cloudwatch metrics to Grafana allows a singular website to view our Kubernetes workloads along with the metrics the cloud provider generates regarding usage in our cloud.

Adding kube-prometheus-stack repo

With that being said, we are going to utilize the well-known kube-prometheus-stack helm chart and deploy it on an EKS cluster. You can add the kube-prometheus-stack helm chart with the following command

Implement Grafana IAM Role

In order to authorize Grafana pod to pull metrics from Cloudwatch we will need to set IRSA (IAM Role for Service Account) which is a system that automates the provisioning and rotation of IAM temporary credentials for kubernetes workloads instead of distributing static credintials

{
       "Version": "2012-10-17",
       "Statement": [
           {
           "Sid": "AllowReadingMetricsFromCloudWatch",
           "Effect": "Allow",
           "Action": [
               "cloudwatch:DescribeAlarmsForMetric",
               "cloudwatch:DescribeAlarmHistory",
               "cloudwatch:DescribeAlarms",
               "cloudwatch:ListMetrics",
               "cloudwatch:GetMetricData",
               "cloudwatch:GetInsightRuleReport"
           ],
           "Resource": "*"
           },
           {
           "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
           "Effect": "Allow",
           "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
           "Resource": "*"
           },
           {
           "Sid": "AllowReadingResourcesForTags",
           "Effect": "Allow",
           "Action": "tag:GetResources",
           "Resource": "*"
           }
       ]
}

Save the policy to grafana-policy.json on your current directory and execute the following command:

aws iam create-policy --policy-name GrafanaMetricsPolicy --policy-document

Secondly, We will create a trust relationship policy that will allow the Grafana pod to authenticate and access the AWS Cloudwatch from within the Kubernetes cluster. Please make sure that you feel in your AWS account ID, the EKS cluster unique OIDC, the service account name of Grafana, and the namespace it’s going to be located within the Kubernetes cluster.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/<OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<OIDC_ID>:sub": "system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT_NAME>",
         “<OIDC_ID>:aud”:”sts.amazonaws.com”
        }
      }
    }
  ]
}

Once you update the placeholder with your relevant information, you can save the file to trust.json and create Grafana role with the following command

aws iam create-role --role-name GrafanaMetricsRole --assume-role-policy-document

After the role had been created we can attach the grafana metric policy we had been creating in the first step to our Grafan role with the following command. Please make sure to feel in the right AWS account ID.

aws iam attach-role-policy --role-name GrafanaMetricsRole --policy-arn

So, the role is ready we can proceed with installing or update our helm chart with custom values file which I am going to call updated-values.yaml and update the following lines:

Update values file and install helm chart

grafana:
   serviceAccount:
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/GrafanaMetricsPolicy
   datasources:
     datasources.yaml:
       apiVersion: 1
       datasources:
       - name: CloudWatch
         type: cloudwatch
         jsonData:
           authType: default
           defaultRegion

The following lines are adding annotations for the service account with the role that we created in the previous steps and also cover a new data source so we would be able to access our CloudWatch metrics. Please make sure that you feel in the relevant region and AWS account ID. If done successfully, you will be able to view your new data source on the Grafana website under home > explore once you install your grafana helm chart.

helm install kube-prometheus-stack --create-namespace --namespace monitoring -f

To conclude,

we have covered which IAM policy, role, and trust relationship we need to set for our Grafana pod in order to access Cloudwatch metrics, and how to update the kube-prometheus-stack helm chart to use our role and add the Cloudwatch datasource. From here you can continue and implement dashboards our great community has been creating so your team can make sense of the Cloudwatch metrics.

Relevant links:

Grafana Cloudwatch integration - https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/

AWS IRSA - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Grafana IAM Policy - https://docs.aws.amazon.com/grafana/latest/userguide/adding--CloudWatch-manual.html