Retrieve GCP Billing Data

You can configure Commander to retrieve Google Cloud Platform billing data. Retrieving billing data provides information for the Cloud Billing Report and Cost Analytics. As well as general usage costs, these reports include metered usage costs such as Storage and Network I/O. Any credits in the billing data are also accounted for, and the service costs are automatically adjusted for those credits.

Online, you can read about the basics of Google Cloud billing in general, including exporting billing data to BigQuery.

For resellers configuring billing for GCP subaccounts, see GCP Billing Subaccounts for Resellers.

About billing data retrieval

Commander can retrieve data from before the GCP account was added as a cloud account. New billing data is then retrieved nightly.

Unlike AWS and Azure, Google Cloud Platform doesn’t associate costs with VMs, so VM billing records won't be created and the Cloud Billing Report for GCP private clouds won't be populated. The Cloud Billing report for GCP will only show the projected cost model, not the historical or current cost models.

Commander also retrieves billing data for services that have never been managed by Commander. For example, if a GCP account was added as a cloud account 90 days ago, and you configure Commander to retrieve billing data today, Commander retrieves the last 60 days' worth of billing data for all services that have existed at any point over the last 60 days. As another example, if you add a GCP account as a cloud account today, and you configure Commander to retrieve billing data today, Commander retrieves the last 60 days' worth of billing data for all services that have existed at any point over the last 60 days, including services that were never managed by Commander.

GCP starts exporting billing data once export has been configured in the GCP Console, and doesn't backfill billing data. This means that Commander can only retrieve billing data from the time that billing data export is configured in GCP Console.

Billing data also includes costs for resources and services that aren't managed by Commander but will show up on your bill, such as Cloud Datastore, BigQuery, Stackdriver Logging and Cloud CDN. When a resource not managed by Commander is discovered in the billing data, ownership is assigned to all costs, based on the Default Ownership policy and the service's location in the Infrastructure or Applications tree, for the purposes of Cost Analytics in both Commander and the Service Portal.

Commander billing data are stored and queried in a universal date without time or time zone. For example, if you are retrieving or injecting billing data for November 30, 2021, Commander interprets this date without any time zone shift, regardless of where you are located. Cost information retrieved from this public cloud provider is split at midnight in the GMT time zone and then stored in Commander with a universal date without time or time zone.

GCP Billing Tags

Billing Tags allow you to group and filter your expenses in more detail. You can use these tags to filter expenses in Cost Analytics, the Service Portal Cost Dashboard, and the Cloud Billing Report.

Billing Tags are set up as labels and project.labels in your GCP account and automatically imported during billing data retrieval. If labels and project.labels contain the same tag keys with different values, the value from the labels data set will be used. GCP system.labels won't be retrieved. For more information GCP labels, see the GCP online documentation at Export Cloud Billing data to BigQuery.

Billing Tags can also be incorporated into formulas that are used to create Additional Line Items for cost analytics. For information on using Billing Tags in formulas for Additional Line Items, see Additional Line Items page.

• Before upgrade to 8.1.0, filtering Cost Analytics and the Service Portal Cost Dashboard by Custom Attributes affected the Recommendations. After upgrade, filtering by Billing Tags won't affect the Recommendations.
• Billing Tags and SKUs will only be retrieved from Version 8.1 and forward, unless billing records are reset. Please create a case through the Snow Support Portal if you would like to perform this activity.

How the GCP billing model maps to Commander costing

A GCP cloud account may include multiple projects, so a single cloud account may be associated with multiple billing accounts.

Likewise, if you've added multiple GCP cloud accounts, the same project may be included in multiple cloud accounts, meaning that multiple cloud accounts may be linked to the same billing account.

To help you understand this relationship, see the following diagram:

Note the following about this example:

• When you export a BigQuery dataset, you must select its project location. In this example, BigQuery Dataset 1 is located in Project 1, Big Query Dataset 2 is located in Project 3, and BigQuery Dataset 3 is located in Project 7.
• When you configure billing data retrieval for Cloud Account 1, Commander retrieves billing data from BigQuery Dataset 1 and 2. Because Project 1 is visible to both cloud accounts, billing data is automatically retrieved for this project in Cloud Account 2, even if you haven't configured billing data retrieval for Cloud Account 2.
• Because Cloud Accounts 1 and 2 share a billing account, once billing data is retrieved for Billing Account 1, charges for Project 1, 2 and 6 are added to both cloud accounts.
• If you want to retrieve billing data for Project 7 and 8, you must also configure billing data retrieval for Cloud Account 2, pointing to BigQuery Dataset 3.
• Billing isn't enabled for Project 5.

Overview of steps

To retrieve GCP billing data, there are several mandatory steps:

1. Export billing data to BigQuery
2. Enable the Cloud Billing API
3. Grant dataset permissions to the Commander service account
4. Configure the Default Ownership policy
5. Retrieve GCP billing data

Export billing data to BigQuery

To be able to retrieve GCP billing data in Commander, you must export billing data to Google BigQuery.

To learn how, see Export Billing Data to BigQuery in the Google Cloud documentation.

• Commander supports only v1 of the BigQuery data export tables; the original version isn't supported.
• Commander doesn't support GCP billing data exported to CSV or JSON. Only BigQuery datasets are supported.

Enable the Cloud Billing API

In the GCP Console, ensure that the Cloud Billing API is enabled for the project where the Commander service account is located.

To learn how to enable APIs for a project, see Enabling and Disabling Services in the Google Cloud documentation.

Grant dataset permissions to the Commander service account

Ensure that the service account has access to the exported BigQuery dataset. For more information, see Grant permissions to the Commander service account.

Configure the Default Ownership policy

GCP charges aggregated CPU and memory usage costs, grouped by project and service type. GCP billing data doesn't include costs for individual VMs. Before you retrieve billing data, you must configure the Default Ownership policy, to ensure that cost analytics are accurate, and costs are assigned to the proper owners and organizations.

For example, if you configure the Default Ownership policy to assign ownership of all services in GCP Project X to Commander Organization A, Commander can then assign the total cost of all services in Project X to Commander Organization A.

If you don't configure the Default Ownership policy before retrieving billing data, ownership won't be applied to the retrieved costs. This means that:

• Cost Analytics information won't be accurate.
• In the Service Portal, charges won't be visible.
• In Commander, no ownership information will be assigned to the charges.

For GCP, don’t create an ownership policy on a cloud account if it's sharing resources.

While the Default Ownership policy applies ownership only to new services, Commander also uses this policy to assign ownership to newly retrieved costs for the purposes of Cost Analytics.

To learn how to configure the Default Ownership policy, see Set Resource Ownership with Policies.

Retrieve GCP billing data

You need to configure billing data retrieval in Commander at the cloud account level.

If you have multiple GCP cloud accounts and they use the same single billing account, billing data for all of the cloud accounts will be retrieved at the same time. There's no need to configure billing data retrieval for the other cloud accounts.

Once billing data retrieval is configured for GCP, estimated costs are replaced with billing data costs in Cost Analytics. Cost analytics won't include costs for services associated with any billing accounts whose data hasn't been retrieved, so we recommend retrieving billing data for all associated billing accounts.

1. From the Inventory tree, select a cloud account.
2. Select Actions > Retrieve Billing Data.

   The Billing Data Settings dialog appears. The first column lists all billing accounts associated with this cloud account. The second column lists all projects associated with each billing account, whether or not the project is part of this cloud account.

   If you see the message "To retrieve billing data for this cloud account, you must enable the Cloud Billing API", see Enable the required APIs.

3. For Project Containing BigQuery Dataset, select the project where your BigQuery dataset is stored for this billing account.
4. Enter the Dataset Name.

   You can find the dataset name in the GCP Console. In the following example, the BigQuery dataset is located in Embotics Project X, and the dataset name is embotics_main_billing_export.

   

   Here's what this example looks like in Commander:

   

5. Click Test to verify that the dataset exists, and that this cloud account has access to it.
6. If you see the warning "Dataset found, but no billing data can be found in the dataset", this means that no billing data is available in the specified dataset. If you've recently exported billing data to this BigQuery dataset, you can ignore this warning, because it might take a few hours for data to be available from GCP. Commander will retrieve billing data once it's available.
7. If you see the error "The dataset 'x' cannot be found. Verify the dataset name and ensure that the service account used to add the selected project's cloud account has access to the dataset", verify that:
• the dataset is located in the selected project
• you have selected the project in the correct cloud account. If the project is visible to multiple cloud accounts, but the dataset isn't accessible to all cloud accounts, you must select the project under the cloud account that has access to the dataset. In the image below, "Embotics Project X" is visible to two cloud accounts, but the embotics_main_billing_export dataset is accessible only to the cloud account named GCP - Embotics Automation, so "Embotics Project X" must be selected under GCP - Embotics Automation.

  

8. Repeat steps 3-4 for each billing account that contains data you want to retrieve.
9. Click OK to close the dialog and retrieve billing data.

   If you see an error message, to close the dialog, you must either click Not Configured and OK, or click Cancel.

Disable billing data retrieval

1. From the Inventory tree, select a GCP cloud account for which you want to stop retrieving billing data.
2. Select Actions > Retrieve Billing Data.
3. In the GCP Billing Data Settings dialog, for each billing account that you want to disable, select Not Configured and click OK.

   Commander will no longer retrieve billing data for this billing account.

If other GCP cloud accounts share this billing account, billing data retrieval will also be disabled for those cloud accounts.