Metering
meshStack imports metering data from AWS Cost and Usage Reports
An AWS Cost and Usage Report should be created as specified in the documentation with the following properties
- Daily time granularity
- gzip compression
- txt/csv file format
- Should include resource ids
- Versioning should be setup to overwrite existing report
Note: If an S3 bucket is not already set up, it should be created as specified in AWS documentation
The following parameters are required to configure meshStack to process the AWS Cost and Usage Reports [More Details]
- AWS credentials that can access the S3 bucket where the reports are written
- The name of the S3 bucket where the report is written
- The region of the S3 bucket
- The name of the report
- The "Report path prefix" configured when creating the report
When processing the AWS Cost and Usage Report to generate the Usage Report in the meshPanel,
- You can configure which line item types
should be considered in the calculations. You can also configure whether you want to consider discounts or not.
We include the following columns in the calculations in order to come up with an amortized cost that should be charged to each account(the discount column is taken only if configured to consider discounts).
- If the line item type is RIFee, we take
reservation/UnusedAmortizedUpfrontFeeForBillingPeriod
,reservation/UnusedRecurringFee
anddiscounts/TotalDiscount
column - If the line item type is
SavingsPlanRecurringFee
then we take thediscounts/TotalDiscount
column - For other line item types, we take which ever is available from the columns
savingsPlan/SavingsPlanEffectiveCost
,reservation/EffectiveCost
andlineItem/UnblendedCost
in that order plus thediscounts/TotalDiscount
column
- If the line item type is RIFee, we take
- Only the line items with bill type
Anniversary
are taken into the calculation. In other words, line items with bill typePurchase
andRefund
are excluded.
IAM User Configuration
Policies
In order for meshStack to process AWS Cost and Usage Reports, following policies are required:
- Bucket Access Policy: This policy allows the Metering IAM user to fetch AWS cost and usage reports
- Organization Access Policy: This policy allows the Metering IAM user to list all accounts in the organization
- Assume Role Policy: This policy allows Metering IAM user to assume the IAM Role
MeteringRole
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-cur-bucket-name",
"arn:aws:s3:::my-cur-bucket-name/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "organizations:ListAccounts",
"Resource": "*"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::123456789123:role/MeteringRole"
}
]
}
Role
The 3 policies in the previous section should be attached to this role.
In order for the Metering IAM user to assume this role, following trust policy is required:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789123:user/Metering"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
Configuration Reference
This section describes the configuration of a AWS Platform Instance in the meshStack configuration model
at mesh.platforms
for AWS metering.
let AwsPlatformKrakenConfiguration =
{ platform :
{- The combination of meshPlatform and meshLocation identifiers, i.e. <platform-id>.<location-id> -}
Text
, region :
{- The region of the IAM user -}
Text
, meshfed-service-user :
{-
Metering IAM user credentials
access-key: "<AWS_ACCESS_KEY>"
secret-key: "<AWS_SECRET_KEY>"
-}
{ access-key : Secret, secret-key : Secret }
, cost-and-usage-report :
{-
Parameters required to process AWS cost and usage reports
name: The name of the report
bucket-name: The name of the S3 bucket where the report is written
bucket-region: The region of the S3 bucket
report-key-prefix: The report path prefix.
Generated reports will be located at <bucket-name>/<report-key-prefix>
-}
{ name : Text
, bucket-name : Text
, bucket-region : Text
, report-key-prefix : Text
}
, organization-root-account-role :
{- The role assigned to the metering IAM user -}
Text
, organization-root-account-external-id :
{- The external ID passed to "sts:AssumeRole" when used in privileged context -}
Optional Text
, cost-item-write-batch-size :
{- Cost items written per batch -}
Natural
, max-report-file-processing-retries :
{- The maximum number of attempts to process or download a single file in a report -}
Natural
, report-file-process-retry-delay-millis :
{- Time in milliseconds before an operation (report file processing) is retried -}
Natural
, max-bulk-insert-retries :
{- The maximum number of attempts to process a batch of cost items to the database -}
Natural
, bulk-insert-retry-delay-millis :
{- Time in milliseconds before an operation (cost item batch processing) is retried -}
Natural
, cost-item-read-page-size :
{-
Cost items to read per query when generating tenant usage reports.
This setting is useful to prevent read timeout errors in the case of accounts with
a large number of cost items.
-}
Natural
, apply-discounts :
{-
Whether the discounts contained in the AWS Cost and Usage Reports should be applied
when generating the tenant usage reports or not
-}
Bool
, reported-line-item-types :
{-
What line item types should be included in the report. See https://docs.aws.amazon.com/cur/latest/userguide/Lineitem-columns.html#l-L
for a list of options. Note that if you remove items from this list, some manual cleanup is still needed to remove
the already collected data from the database.
-}
List Text
}
let example
: AwsPlatformKrakenConfiguration
= { platform = "aws.aws-location"
, region = "eu-central-1"
, meshfed-service-user =
{ access-key = Secret.Native "AWS_ACCESS_KEY_KRAKEN"
, secret-key = Secret.Native "AWS_SECRET_KEY_KRAKEN"
}
, cost-and-usage-report =
{ name = "my-cur-name"
, bucket-name = "my-cur-bucket-name"
, bucket-region = "eu-central-1"
, report-key-prefix = "/my-cur-prefix"
}
, organization-root-account-role =
"arn:aws:iam::123456789123:role/MeteringRole"
, organization-root-account-external-id = Some
"abcd1234-12ab-12ab-12ab-abcdef123456"
, cost-item-write-batch-size = 10000
, max-report-file-processing-retries = 5
, report-file-process-retry-delay-millis = 120000
, max-bulk-insert-retries = 5
, bulk-insert-retry-delay-millis = 5000
, cost-item-read-page-size = 2000
, apply-discounts = False
, reported-line-item-types =
[ "DiscountedUsage", "Fee", "Usage", "SavingsPlanCoveredUsage" ]
}
Configuring Seller Information
If you would like to see the AWS costs attributed to a seller in the chargeback statments, a product needs to be created manually in the meshcloud Product Catalog. This product should have a resourceType
of sellerInfo
and should be scoped to a platformType
of Aws
. An example is shown below.
{
"resourceType": "sellerInfo",
"displayName": "AWS Seller Info",
"scope": {
"platformType": "Aws",
"location": null,
"platformInstance": null,
"localProjectId": null
},
"usageTypes": [],
"description": "",
"sellerId": "AWS",
"sellerProductGroup": null,
"@metadata": {
"@collection": "Products",
"Raven-Java-Type": "io.meshcloud.kraken.core.metering.Product"
}
}