Reading · Gab's Notes · RSS Amplifier

Context

Following my Terraform module adventures, I wanted to detect potential duplicates in our numerous ACLs, because they would only produce errors during application of the plan, after merging. And relying on humans to find a needle in the proverbial haystack wasn’t going to scale. I needed to automate this.

Duplicate you say?

Two ACLs are considered duplicates if they share the same source (token) and destination (endpoint).

Meme representing two spidermen pointing at each other, no caption.

Our module is called like this:

module "acl_foo" {
  source = "../../modules/acl"
  token    = "token"
  endpoint = "endpoint"
  # ...
}

We want to detect multiple instances of this module with the same source and destination, and raise an error.

We also want to detect duplicates for gateway_policys declared manually, without using the module.

resource "arsenal_gateway_policy" "foo" {
  token    = "token"
  endpoint = "endpoint"
  # ...
}

Linters to the rescue

Enter tflint. We already use it to make sure our code follows common best practices, and it advertises itself as “pluggable”. Unfortunately, it requires writing a separate Go program using a specific library, creating build binaries, releases, etc.

Go is in our tool set, but I had something much simpler in mind. Ideally, I could package everything in our Terraform repository to avoid having a separate thing to maintain.

tflint promotes another option with the opa plugin. It allows you to write custom policies in a language called Rego. It also provides custom functions to retrieve data from your Terraform definitions. We can also write tests for our rules.

The rule

Here is the rule I came up with for our duplicate detection. I’m no expert in Rego nor Terraform, so if you see something that I could improve, please let me know! The rest of this section is dedicated to explaining every step.

package tflint
import rego.v1
resource_acls := terraform.resources("arsenal_gateway_policy", {"token": "string", "endpoint": "string"}, {})
module_acls := terraform.module_calls({"token": "string", "endpoint": "string"}, {})
all_acls := array.concat(resource_acls, module_acls)
# filter out resources and modules
# which don't have a token or endpoint
# or which have undefined ones
# as we can't check for duplicates in this case.
acls := [
  acl |
    acl := all_acls[_]
    "token" in object.keys(acl.config)
    not acl.config.token.unknown
    "endpoint" in object.keys(acl.config)
    not acl.config.endpoint.unknown
]
acl_pairs := [
  acl_string |
    acl := acls[_]
    acl_string := concat(" -> ", [acl.config.token.value, acl.config.endpoint.value])
]
acl_duplicates := {
  i |
    pair := acl_pairs[i]
    count([x | some x in acl_pairs; x == pair]) > 1
}
deny_duplicate_acl contains issue if {
  _ := acl_duplicates[i]
  issue := tflint.issue(`Duplicate ACL found`, acls[i].decl_range)
}

The introduction documentation discusses the package and import lines, so let’s move on.

resource_acls := terraform.resources("gateway_policy", {"token": "string", "endpoint": "string"}, {})
module_acls := terraform.module_calls({"token": "string", "endpoint": "string"}, {})
all_acls := array.concat(resource_acls, module_acls)

terraform.resources and terraform.module_calls are documented here. They return a list of resources of the given type or module calls, and we request the token and endpoint values. We then concatenate everything to the all_acls list.

# filter out resources and modules
# which don't have a token or endpoint
# or which have undefined ones
# as we can't check for duplicates in this case.
acls := [
  acl |
    acl := all_acls[_]
    "token" in object.keys(acl.config)
    not acl.config.token.unknown
    "endpoint" in object.keys(acl.config)
    not acl.config.endpoint.unknown
]

We filter out the items we can’t work with: the modules which don’t have a token or endpoint (we have other module calls), and the ACLs for which one of the values is unknown. This can happen if one of them is defined as the output of another module, resource or data block for example, and in this case we have very little to work with, so we chose to exclude them.

acl_pairs := [
  acl_string |
    acl := acls[_]
    acl_string := concat(" -> ", [acl.config.token.value, acl.config.endpoint.value])
]

We use a list comprehension to transform this list of modules to a list of strings containing the token and endpoint of the modules. We end up with something like this:

[
    "token:foo -> endpoint:foo",
    "token:bar -> endpoint:foo",
    "token:foo -> endpoint:foo"
]

acl_duplicates := {
  i |
    pair := acl_pairs[i]
    count([x | some x in acl_pairs; x == pair]) > 1
}

Finally, we use a set comprehension to collect all indices of ACL pairs that appear at least two times in the module_acl_pairs list.

The result following our previous example would be: {0, 2}.

deny_duplicate_acl contains issue if {
  _ := acl_duplicates[i]
  issue := tflint.issue(`Duplicate ACL found`, acls[i].decl_range)
}

In the end, we declare our rule, which produces an issue for every item in module_acl_duplicates. It uses the indices stored in this list to retrieve the original module calls saved in module_acls in order to fetch their decl_range, which allows us to produce a nice error message pointing directly at the relevant source code.

One thing that is only written deep in the “introduction” docs of the plugin and which cost me quite some time was the fact that rules must conform to a specific naming scheme. Specifically, the rule name must start with a given string to be interpreted, in our case we chose deny_.

We enable the plugin like so:

plugin "opa" {
  enabled = true
  version = "0.9.0"
  source  = "github.com/terraform-linters/tflint-ruleset-opa"
}

And finally run tflint!

$ <span class="bash"><span class="bash"><span class="bash">tflint</span></span></span>
2 issue(s) found:
Error: Duplicate ACL found (opa_deny_duplicate_acl)
  on envs/prod/acl.tf line 97:
  97: module "acl_foo" {
Reference: ../../.tflint.d/policies/duplicate_acls.rego:31
Error: Duplicate ACL found (opa_deny_duplicate_acl)
  on envs/prod/acl.tf line 111:
 111: resource "gateway_policy" "bar" {
Reference: ../../.tflint.d/policies/duplicate_acls.rego:31

Tests

This rule can be tested using the facilities provided by the plugin. We test that we detect duplicates across resources, modules, a mix of both, and that we don’t report errors when everything is fine.

duplicate_resources(type, schema, options) := terraform.mock_resources(type, schema, options, {"main.tf": `
resource "arsenal_gateway_policy" "duplicate_1" {
   token    = "toto"
   endpoint = "tata"
}
resource "arsenal_gateway_policy" "duplicate_2" {
   token    = "toto"
   endpoint = "tata"
}
`})
test_deny_duplicate_acl_resource_failed if {
	issues := deny_duplicate_acl with terraform.resources as duplicate_resources
	count(issues) == 2
	issue := issues[_]
	issue.msg == `Duplicate ACL found`
}

duplicate_modules(schema, options) := terraform.mock_module_calls(schema, options, {"main.tf": `
module "acl_duplicate_1" {
  token    = "tete"
  endpoint = "tata"
}
module "acl_duplicate_2" {
  token    = "tete"
  endpoint = "tata"
}
`})
test_deny_duplicate_acl_module_failed if {
	issues := deny_duplicate_acl with terraform.module_calls as duplicate_modules
	count(issues) == 2
	issue := issues[_]
	issue.msg == `Duplicate ACL found`
}

unique_resources_overlap_modules(type, schema, options) := terraform.mock_resources(type, schema, options, {"main.tf": `
resource "arsenal_gateway_policy" "duplicate_1" {
   token    = "toto"
   endpoint = "tata"
}
resource "arsenal_gateway_policy" "duplicate_2" {
   token    = "tata"
   endpoint = "tata"
}
`})
unique_modules_overlap_resources(schema, options) := terraform.mock_module_calls(schema, options, {"main.tf": `
module "acl_unique_1" {
  token    = "tata"
  endpoint = "tata"
}
module "acl_unique_1" {
  token    = "titi"
  endpoint = "tata"
}
`})
test_deny_duplicate_acl_resource_module_failed if {
	issues := deny_duplicate_acl with terraform.resources as unique_resources_overlap_modules
	                             with terraform.module_calls as unique_modules_overlap_resources
	count(issues) == 2
	issue := issues[_]
	issue.msg == `Duplicate ACL found`
}

unique_resources(type, schema, options) := terraform.mock_resources(type, schema, options, {"main.tf": `
resource "arsenal_gateway_policy" "unique_1" {
   token    = "toto"
   endpoint = "tata"
}
resource "arsenal_gateway_policy" "unique_2" {
   token    = "titi"
   endpoint = "tata"
}
`})
unique_modules(schema, options) := terraform.mock_module_calls(schema, options, {"main.tf": `
module "acl_unique_1" {
  token    = "tete"
  endpoint = "tata"
}
module "acl_unique_1" {
  token    = "tutu"
  endpoint = "tata"
}
`})
test_deny_duplicate_acl_passed if {
	issues := deny_duplicate_acl with terraform.resources as unique_resources
	                             with terraform.module_calls as unique_modules
	count(issues) == 0
}

Tests can be run with a special environment variable before the tflint call.

TFLINT_OPA_TEST=1 tflint

`pre-commit` and multiple root modules

We have two more constraints:

we run tflint through pre-commit, using antonbabenko/pre-commit-terraform ;
our repository contains several Terraform root modules, for all our environments.

Here is how our directory structure looks like:

.
├── .git/
├── .gitignore
├── .pre-commit-config.yaml
├── .terraformignore
├── .tflint.d/
│   └── policies/
│       ├── duplicate_acls.rego
│       └── duplicate_acls_test.rego
├── .tflint.hcl
├── envs/
│   ├── staging/
│   │   ├── .terraform/
│   │   ├── .terraform.lock.hcl
│   │   ├── foo.tf
│   │   └── ...
│   ├── preprod/
│   │   ├── .terraform/
│   │   ├── .terraform.lock.hcl
│   │   ├── foo.tf
│   │   └── ...
│   └── prod/
│       ├── .terraform/
│       ├── .terraform.lock.hcl
│       ├── foo.tf
│       └── ...
├── modules/
│   └── custom_acl/
│       ├── .terraform/
│       ├── .terraform.lock.hcl
│       ├── main.tf
│       └── ...
└── README.md

This pre-commit config tells the hook to use the root .tflint.hcl configuration file for every subdirectory, and to delegate the directory change to tflint so the error messages include the whole path of the file and not the path relative to the directory.

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.99.5
    hooks:
      - id: terraform_tflint
        args:
          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
          - --args=--fix
          - --hook-config=--delegate-chdir

Then, we configure tflint using .tflint.hcl placed at the root of the repository:

plugin "opa" {
  enabled = true
  version = "0.9.0"
  source  = "github.com/terraform-linters/tflint-ruleset-opa"
  # this is relative to the directory in which tflint will be run: each subdir of envs/ and modules/
  policy_dir = "../../.tflint.d/policies"
}

As mentioned in the comment, we must declare the policy_dir relative to where tflint runs. Since it changes directory to run two levels deep, and we want our policies to be shared, we have to tell it to look for policies in the root of the repo.

Conclusion

This was quite a piece, but I’m glad we now have something to catch issues before they even happen. All of this can work locally, on the developer’s machine, before even spending time planning thus reducing the feedback loop to a minimum.

There is still a lot to learn about Rego, which I’m still deeply unfamiliar with but looks like a powerful tool. I don’t know whether I’ll spend much more time with it though, as I don’t have other immediate use cases.

Let me know if you see something I can improve in this setup! I’m still quite new to all this and I’m eager to learn.

Custom Terraform linter rules with Rego

Context

Duplicate you say?

Linters to the rescue

The rule

Tests

`pre-commit` and multiple root modules

Conclusion

Comments

Custom Terraform linter rules with Rego

Context

Duplicate you say?

Linters to the rescue

The rule

Tests

pre-commit and multiple root modules

Conclusion

Comments

`pre-commit` and multiple root modules