aws-cur-best-practice (#651)

Co-authored-by: bstuder99 <[email protected]>

aws-cloud-integrations.md

@@ -1,16 +1,16 @@
 # AWS Cloud Integration
 
-By default, Kubecost pulls On-Demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.
+By default, Kubecost pulls on-demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.
 
 You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.
 
-A GitHub repository with sample files which follow the below instructions can be found [here](https://github.com/kubecost/poc-common-configurations/tree/main/aws).
+This guide contains multiple possible methods for connecting Kubecost to AWS billing, based on user environment and preference. Because of this, there may not be a straightforward approach for new users. To address this, a streamlined guide contaning best practices can be found [here](./aws-cur-setup.md). This best practices guide has some assumptions to carefully consider.
 
-## Overview
+For the below guide, a GitHub repository with sample files can be found [here](https://github.com/kubecost/poc-common-configurations/tree/main/aws).
 
-Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.
+## Key AWS terminology
 
-### Terminology
+Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.
 
 **Cost and Usage Report**: AWS report which tracks cloud spending and writes to an Amazon Simple Storage Service (Amazon S3) bucket for ingestion and long term historical data. The CUR is originally formatted as a CSV, but when integrated with Athena, is converted to Parquet format.
 
@@ -384,7 +384,7 @@ This may be the preferred method if your Helm values are in version control and
 
 {% code overflow="wrap" %}
 ```bash
-$ kubectl create secret generic <SECRET_NAME> --from-file=service-key.json --namespace <kubecost> 
+$ kubectl create secret generic <SECRET_NAME> --from-file=service-key.json --namespace <kubecost>
 ```
 {% endcode %}
 
@@ -615,16 +615,16 @@ QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecu
 
 * **Resolution:** Previously, if you ran a query without specifying a value for query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default `aws-athena-query-results-MyAcctID-MyRegion` locations remain valid and you can continue to use them. The bucket should be in the format of: `aws-athena-query-results-MyAcctID-MyRegion` It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance. See also this AWS doc on [specifying a query result location](https://docs.aws.amazon.com/athena/latest/ug/querying.html#query-results-specify-location).
 
-#### Missing Athena Column
+#### Missing Athena column
 
 * **Symptom:** A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost `cost-model` container logs.
 
 {% code overflow="wrap" %}
 ```
 QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>
- 
-Checking the athena logs we see a syntax error:
-   
+
+Checking the Athena logs we see a syntax error:
+
 SYNTAX_ERROR: line 4:3: Column 'line_item_resource_id' cannot be resolved
 
 This query ran against the "<DB Name>" database, unless qualified by the query

aws-cur-setup.md

@@ -0,0 +1,255 @@
+# AWS Cost and Usage Report Setup
+
+There are many ways to configure Kubecost to pull the Cost and Usage Report (CUR) from AWS.
+
+This guide is intended as the best-practice method for users meeting the assumptions below. There is a separate guide which has more options: [AWS Cloud Integration](aws-cloud-integrations.md)
+
+The below guide assumes the following:
+1. Kubecost will run in a different account than the AWS Payer Account
+1. The IAM permissions will utilize AWS [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to avoid shared secrets
+1. The configuration of Kubecost will be done using [cloud-integration.json](https://docs.kubecost.com/install-and-configure/install/cloud-integration/multi-cloud) and not via Kubecost UI (following infrastructure as code practices)
+
+## Overview of Kubecost CUR integration
+
+This guide is a one-time setup per AWS Payer Account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.
+
+<details>
+
+<summary>Basic diagram when the below steps are complete:</summary>
+
+![cur-diagram](images/aws-cur/kubecost-cross-account-cur-diagram.png)
+
+</details>
+
+Kubecost supports multiple AWS Payer Accounts as well as multiple cloud providers from a single Kubecost Primary. For multiple Payer Accounts, create additional entries inside the array below.
+
+Detail for multiple cloud provider setups is [here](https://docs.kubecost.com/install-and-configure/install/cloud-integration/multi-cloud#aws).
+
+
+## Configuration
+
+Create a file called *cloud-integration.json* as below:
+```json
+{
+    "aws": [
+        {
+            "athenaBucketName": "s3://ATHENA_RESULTS_BUCKET_NAME",
+            "athenaRegion": "ATHENA_REGION",
+            "athenaDatabase": "ATHENA_DATABASE",
+            "athenaTable": "ATHENA_TABLE",
+            "athenaWorkgroup": "ATHENA_WORKGROUP",
+            "projectID": "ATHENA_PROJECT_ID",
+            "masterPayerARN": "PAYER_ACCOUNT_ROLE_ARN"
+        }
+    ]
+}
+```
+
+All of the values on the right will be replace following the steps in this guide.
+
+- `projectID` is the AWS payer account number where the CUR where the Kubecost Primary cluster is running.
+- `athenaWorkgroup` will be `primary` when following this guide.
+
+### Step 1: Create a CUR Export (and wait 24 hours)
+
+Follow the [AWS documentation](https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html) to create a CUR export using the settings below.
+
+* For time granularity, select _Daily_.
+* Select the checkbox to enable _Resource IDs_ in the report.
+* Select the checkbox to enable _Athena integration_ with the report.
+* Select the checkbox to enable the JSON IAM policy to be applied to your bucket.
+
+<details>
+
+<summary>Screenshots from select CUR creation in the above AWS documentation</summary>
+
+![CUR-export-config](images/aws-cur/1-cur-nav.png)
+![resource-ids](images/aws-cur/2-cur-create-step1.png)
+![bucket-permissions](images/aws-cur/3-cur-s3-bucket.png)
+![delivery-options](images/aws-cur/4-delivery-options.png)
+</details>
+
+{% hint style="info" %}
+If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.
+{% endhint %}
+
+Note the name of the bucket you create for CUR data `CUR_BUCKET_NAME`.
+
+AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
+
+### Step 2: Setting up Athena
+
+As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under `s3-path-prefix/cur-name` and typically has the filename *crawler-cfn.yml*. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this [here](https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html).
+
+
+{% hint style="info" %}
+Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.
+{% endhint %}
+
+Once Athena is set up with the CUR, you will need to create a *new* S3 bucket for Athena query results which will be the `ATHENA_RESULTS_BUCKET_NAME`. The bucket used for the CUR cannot be used for the Athena output.
+
+1. Navigate to the [S3 Management Console](https://console.aws.amazon.com/s3/home?region=us-east-2).
+2. Select _Create bucket._ The Create Bucket page opens.
+3. Use the same region used for the CUR bucket. This is the value for `athenaBucketName`
+4. Select _Create bucket_ at the bottom of the page.
+5. Navigate to the [Amazon Athena](https://console.aws.amazon.com/athena) dashboard.
+6. Select _Settings_, then select _Manage._ The Manage settings window opens.
+7. Set _Location of query result_ to the S3 bucket you just created, then select _Save._
+
+Navigate to Athena in the AWS Console, be sure it is in the region used in the steps above.
+
+* In your *cloud-integration.json*, update the following values with the values obtained in this step.
+<details>
+
+<summary>Screenshots from Athena</summary>
+
+![CUR-Config](images/aws-cur/6-cur-config.png)
+![athena-output-bucket](images/aws-cur/8-upload-cfn-template.png)
+</details>
+
+{% hint style="info" %}
+For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after 1 day of retention.
+{% endhint %}
+
+
+### Step 3: Download configuration templates
+
+The policy documents required can be cloned from: <https://github.com/kubecost/poc-common-configurations/tree/main/aws-attach-roles>
+
+### Step 4: Setting up Payer Account IAM permissions
+
+**From the AWS Payer Account**
+
+* In _kubecost-payer-account-cur-athena-glue-s3-access.json_, replace `ATHENA_RESULTS_BUCKET_NAME` with your Athena S3 bucket name (the default will look like `aws-athena-query-results-xxxx`).
+
+In the same location where your downloaded configuration files are, run the following command to create the appropriate policy (jq is not required):
+
+Update `SUB_ACCOUNT_222222222` in the file *iam-payer-account-trust-primary-account.json* with the account number of the account where the primary Kubecost cluster will run.
+
+{% code overflow="wrap" %}
+```sh
+aws iam create-role --role-name kubecost-cur-access \
+  --assume-role-policy-document file://iam-payer-account-trust-primary-account.json \
+  --output json | jq -r .Role.Arn
+```
+
+The output is the value for the `PAYER_ACCOUNT_ROLE_ARN`:
+
+```
+arn:aws:iam::PAYER_ACCOUNT_11111111111:role/kubecost-cur-access
+```
+
+Update the placeholders (everything with a `_` ex. `ATHENA_DATABASE`) in *iam-payer-account-cur-athena-glue-s3-access.json* and attach the required policies to the new role:
+
+```sh
+aws iam put-role-policy --role-name kubecost-cur-access \
+  --policy-name kubecost-payer-account-cur-athena-glue-s3-access \
+  --policy-document file://iam-payer-account-cur-athena-glue-s3-access.json
+```
+
+And allow Kubecost to read account tags:
+
+```sh
+aws iam put-role-policy --role-name kubecost-cur-access \
+  --policy-name kubecost-payer-account-list-tags-policy \
+  --policy-document file://iam-payer-account-list-tags-policy.json
+```
+{% endcode %}
+
+### 5. Setting up IAM permissions for Kubecost Primary Cluster
+
+**From the AWS Account where the Kubecost Primary will run**
+
+Update `PAYER_ACCOUNT_11111111111` with the AWS account number of the Payer Account and create policy allowing Kubecost to assumeRole in the Payer Account:
+
+```sh
+aws iam create-policy --policy-name kubecost-access-cur-in-payer-account \
+  --policy-document file://iam-access-cur-in-payer-account.json \
+  --output json |jq -r .Policy.Arn
+```
+
+Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):
+```
+arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account
+```
+
+Create namespace and set environment variables:
+
+```sh
+kubectl create ns kubecost
+export CLUSTER_NAME=YOUR_CLUSTER
+export AWS_REGION=YOUR_REGION
+```
+
+Enable the OIDC-Provider:
+
+```sh
+eksctl utils associate-iam-oidc-provider \
+    --cluster $CLUSTER_NAME --region $AWS_REGION \
+    --approve
+```
+
+Create the Kubernetes service account, attaching the assumeRole policy:
+
+**Note:** Replace `SUB_ACCOUNT_222222222` with AWS account number where the primary Kubecost cluster will run.
+
+{% code overflow="wrap" %}
+```sh
+eksctl create iamserviceaccount \
+    --name kubecost-serviceaccount \
+    --namespace kubecost \
+    --cluster $CLUSTER_NAME --region $AWS_REGION \
+    --attach-policy-arn arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account \
+    --override-existing-serviceaccounts \
+    --approve
+```
+{% endcode %}
+
+Create the secret (in this setup, there are no actual secrets in this file):
+
+{% code overflow="wrap" %}
+```
+kubectl create secret generic cloud-integration -n kubecost --from-file=cloud-integration.json
+```
+{% endcode %}
+
+Install Kubecost using the service account and cloud-integration secret:
+
+{% code overflow="wrap" %}
+```sh
+helm install kubecost \
+  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
+  --namespace kubecost \
+  --set serviceAccount.name=kubecost-serviceaccount \
+  --set serviceAccount.create=false \
+  --set kubecostProductConfigs.cloudIntegrationSecret=cloud-integration
+```
+{% endcode %}
+
+
+## Validation
+
+It can take over an hour to process the billing data for largre AWS accounts. In the short-term, follow the logs and look for a message similar to `(7.7 complete)`, which should grow gradually to `(100.0 complete)`. Some errors (ERR) are expected, as seen below.
+
+```
+kubectl logs -l app=cost-analyzer --tail -1 --follow |grep -i athena
+------------------
+Defaulted container "cost-model" out of: cost-model, cost-analyzer-frontend
+2023-05-24T19:41:31.63093249Z ERR Failed to lookup reserved instance data: no reservation data available in Athena
+2023-05-24T19:41:31.630973097Z ERR Failed to lookup savings plan data: Error fetching Savings Plan Data: QueryAthenaPaginated: athena configuration incomplete
+2023-05-24T19:41:34.577437245Z INF Adding AWS Provider to map with key: 440082503234/s3://aws-athena-query-results-
+2023-05-24T19:41:34.57901059Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Starting PipelineController
+2023-05-24T19:41:34.579037927Z INF CloudCost: IngestionManager: creating integration with key: 440082503234/s3://aws-athena-query-results-
+2023-05-24T19:41:34.581131953Z INF RunBuildProcess[CloudCost][440082503234/s3://aws-athena-query-results-]: build[NLzAH]: Starting build back to 2023-02-22 00:00:00 +0000 UTC in blocks of 7d
+2023-05-24T19:41:34.581715777Z INF CloudCost[440082503234/s3://aws-athena-query-results-]: ingestor: building window [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
+2023-05-24T19:41:34.581771159Z INF AthenaIntegration[440082503234/s3://aws-athena-query-results-]: StoreCloudCost: [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
+2023-05-24T19:41:34.608040758Z ERR ETL:  CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]:  failed to load range from back up 2023-05-18 00:00:00 +0000 UTC - 2023-05-25 00:00:00 +0000 UTC: file does not exist
+2023-05-24T19:41:34.60806207Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: QueryCloudUsage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000)
+2023-05-24T19:41:34.588661972Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Run[JkSYH]: QueryCloudUsage [2023-05-21T00:00:00+0000, 2023-05-25T00:00:00+0000)
+2023-05-24T19:41:50.528544821Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: coverage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000) (7.7 complete)
+```
+
+
+## Troubleshooting
+
+For help with troubleshooting, see the section in our original AWS integration guide [here](https://docs.kubecost.com/install-and-configure/install/cloud-integration/aws-cloud-integrations#troubleshooting).