Validation Reference

• 1: Domains
• 1.1: API Domain
• 1.2: File Domain
• 1.3: Kubernetes Domain
• 2: Providers
• 2.1: Kyverno Provider
• 2.2: OPA Provider
• 3: Testing
• 4: Version Specification

lula-version: ">=v0.2.0"
metadata:
  name: Validate pods with label foo=bar
  uuid: 123e4567-e89b-12d3-a456-426655440000
domain:
  type: kubernetes
  kubernetes-spec:
    resources:
      - name: podsvt
        resource-rule:
          version: v1
          resource: pods
          namespaces: [validation-test]
provider:
  type: opa
  opa-spec:
    rego: |
      package validate

      import future.keywords.every

      validate {
        every pod in input.podsvt {
          podLabel := pod.metadata.labels.foo
          podLabel == "bar"
        }
      }

"yaml.schemas": {
    "${PATH_TO_LULA}/lula/src/pkg/common/schemas/validation.json": "*validation*.yaml"
},

1 - Domains

Domains, generically, are the areas or categories from which data is collected to be evaluated by a Lula Validation. Currently supported Domains are:

• Kubernetes
• API
• File

The domain block of a Lula Validation is given as follows, where the sample is indicating a Kubernetes domain is in use:

# ... Rest of Lula Validation
domain:
    type: kubernetes   # kubernetes or api accepted
    kubernetes-spec:
        # ... Rest of kubernetes-spec
# ... Rest of Lula Validation

Each domain has a particular specification, given by the respective <domain>-spec field of the domain property of the Lula Validation. The sub-pages describe each of these specifications in greater detail.

1.1 - API Domain

The API Domain allows for collection of data (via HTTP Get or Post Requests) generically from API endpoints.

[!Important] This domain supports both read and write operations (Get and Post operations), so use with care. If you configure validations with API calls that mutate resources, add the executable flag to the request so that Lula will ask for verification before making the API call.

Specification

The API domain Specification (api-spec) accepts a list of requests and an options block. options can be configured at the top-level and will apply to all requests except those which have embedded options. request-level options will override top-level options.

domain: 
  type: api
  api-spec:
    # options (optional): Options specified at this level will apply to all requests except those with an embedded options block.
    options:
      # timeout (optional, default 30s): configures the request timeout. The default timeout is 30 seconds (30s). The timeout string is a number followed by a unit suffix (ms, s, m, h, d), such as 30s or 1m.
      timeout: 30s
      # proxy (optional): Specifies a proxy server for all requests.
      proxy: "https://my.proxy"
      # headers (optional):  a map of key value pairs to send with all requests.
      headers: 
        key: "value"
        my-customer-header: "my-custom-value"
    # Requests is a list of URLs to query. The request name is the map key used when referencing the resources returned by the API.
    requests:
      # name (required): A descriptive name for the request.
      - name: "healthcheck" 
        # url (required): The URL for the request. The API domain supports any rfc3986-formatted URI. Lula also supports URL parameters as a separate argument.
        url: "https://example.com/health/ready"
        # method (optional, default get): The HTTP Method to use for the API call. "get" and "post" are supported. Default is "get".
        method: "get"
        # parameters (optional): parameters to append to the URL. Lula also supports full URIs in the URL.
        parameters: 
          key: "value"
        # Body (optional): a json-compatible string to pass into the request as the request body.
        body: |
stringjsondata
        # executable (optional, default false): Lula will request user verification before performing API actions if *any* API request is flagged "executable".
        executable: true
        # options (optional): Request-level options have the same specification as the api-spec-level options at the top. These options apply only to this request.
        options:
          # timeout (optional, default 30s): configures the request timeout. The default timeout is 30 seconds (30s). The timeout string is a number followed by a unit suffix (ms, s, m, h, d), such as 30s or 1m.
          timeout: 30s
          # proxy (optional): Specifies a proxy server for this request.
          proxy: "https://my.proxy"
          # headers (optional): a map of key value pairs to send with this request.
          headers: 
            key: "value"
            my-customer-header: "my-custom-value"
      - name: "readycheck"
      # etc ...

API Domain Resources

The API response body is serialized into a json object with the request name as the top-level key. The API status code is included in the output domain resources under status. raw contains the entire API repsonse in an unmarshalled (json.RawMessage) format.

Example output:

"healthcheck": {
  "status": 200,
  "response": {
    "healthy": true,
  },
  "raw": {"healthy": true}
}

The following example validation verifies that the request named “healthcheck” returns "healthy": true

provider: 
  type: opa
  opa-spec:
    rego: |
      package validate

      validate {
        input.healthcheck.response.healthy == true
      }

You can define a variable to simplify referencing the response:

provider: 
  type: opa
  opa-spec:
    rego: |
      package validate

      resp := input.healthcheck.response

      validate {
        resp == true
      }

1.2 - File Domain

The File domain allows for validation of arbitrary file contents from a list of supported file types. The file domain can evaluate local files and network files. Files are copied to a temporary directory for evaluation and deleted afterwards.

Specification

The File domain specification accepts a descriptive name for the file as well as its path. The filenames and descriptive names must be unique.

domain:
  type: file
  file-spec:
    filepaths:
    - name: config
      path: grafana.ini
      parser: ini         # optionally specify which parser to use for the file type

Supported File Types

The file domain uses OPA’s conftest to parse files into a json-compatible format for validations. Both OPA and Kyverno (using kyverno-json) can validate files parsed by the file domain.

The file domain includes the following file parsers:

• cue
• cyclonedx
• dockerfile
• dotenv
• edn
• hcl1
• hcl2
• hocon
• ignore
• ini
• json
• jsonc
• jsonnet
• properties
• spdx
• string
• textproto
• toml
• vcl
• xml
• yaml

The file domain can also parse arbitrary file types as strings. The entire file contents will be represented as a single string.

The file parser can usually be inferred from the file extension. However, if the file extension does not match the filetype you are parsing (for example, if you have a json file that does not have a .json extension), or if you wish to parse an arbitrary file type as a string, use the parser field in the FileSpec to specify which parser to use. The list above contains all the available parses.

Validations

When writing validations against files, the filepath name must be included as the top-level key in the validation. The placement varies between providers.

Given the following ini file:

[server]
# Protocol (http, https, socket)
protocol = http

The below Kyverno policy validates the protocol is https by including Grafana as the top-level key under check:

metadata:
  name: check-grafana-protocol
  uuid: ad38ef57-99f6-4ac6-862e-e0bc9f55eebe
domain:
  type: file
  file-spec:
    filepaths:
    - name: 'grafana'
      path: 'custom.ini'
provider:
  type: kyverno
  kyverno-spec:
    policy:
      apiVersion: json.kyverno.io/v1alpha1
      kind: ValidatingPolicy
      metadata:
        name: grafana-config
      spec:
        rules:
        - name: protocol-is-https
          assert:
            all:
            - check:
                grafana:
                  server:
                    protocol: https

While in an OPA policy, the filepath Name is the input key to access the config:

metadata:
  name: validate-grafana-config
  uuid: ad38ef57-99f6-4ac6-862e-e0bc9f55eebe
domain:
  type: file
  file-spec:
    filepaths:
    - name: 'grafana'
      path: 'custom.ini'
provider:
  type: opa
  opa-spec:
    rego: |
      package validate
      import rego.v1

      # Default values
      default validate := false
      default msg := "Not evaluated"
      
      validate if {
       check_grafana_config.result
      }
      msg = check_grafana_config.msg

      config := input["grafana"]
      protocol := config.server.protocol

      check_grafana_config = {"result": true, "msg": msg} if {
        protocol == "https"
        msg := "Server protocol is set to https"
      } else = {"result": false, "msg": msg} if {
        protocol == "http"
        msg := "Grafana protocol is insecure"
      }

    output:
      validation: validate.validate
      observations:
        - validate.msg

Parsing files as arbitrary strings

Files that are parsed as strings are represented as a key-value pair where the key is the user-supplied file name and the value is a string representation of the file contexts, including special characters, for e.g. newlines (\n).

As an example, let’s parse a similar file as before as an arbitrary string.

When reading the following multiline file contents as a string:

server = https
port = 3000

The resources for validation will be formatted as a single string with newline characters:

{"config": "server = https\nport = 3000"}

And the following validation will confirm if the server is configured for https:

  domain:
    type: file
    file-spec:
      filepaths:
      - name: 'config'
        path: 'server.txt'
        parser: string
  provider:
    type: opa
    opa-spec:
      rego: |
        package validate
        import rego.v1

        # Default values
        default validate := false
        default msg := "Not evaluated"
        
        validate if {
          check_server_protocol.result
        }
        msg = check_server_protocol.msg
        
        config := input["config"]
        
        check_server_protocol = {"result": true, "msg": msg} if {
          regex.match(
            `server = https\n`,
            config
          )
          msg := "Server protocol is set to https"
        } else = {"result": false, "msg": msg} if {
          regex.match(
            `server = http\n`,
            config
          )
          msg := "Server Protocol must be https - http is disallowed"
        }

      output:
        validation: validate.validate
        observations:
          - validate.msg

Note on Compose

While the file domain is capable of referencing relative file paths in the file-spec, Lula does not de-reference those paths during composition. If you are composing multiple files together, you must either use absolute filepaths (including network filepaths), or ensure that all referenced filepaths are relative to the output directory of the compose command.

Evidence Collection

The use of lula dev get-resources and lula validate --save-resources will produce evidence in the form of json files. These files provide point-in-time evidence for auditing and review purposes.

Evidence collection occurs for each file specified and - in association with any error will produce an empty representation of the target file(s) data to be collected.

1.3 - Kubernetes Domain

The Kubernetes domain provides Lula access to data collection of Kubernetes resources.

[!Important] This domain supports both read and write operations on the Kubernetes cluster in the current context, so use with care

Specification

The Kubernetes specification can be used to return Kubernetes resources as JSON data to the providers. The specification allows for both manifest and resource field data to be retrieved.

The following sample kubernetes-spec yields a list of all pods in the validation-test namespace.

domain:
  type: kubernetes
  kubernetes-spec:
    resources:                          # Optional - Group of resources to read from Kubernetes
    - name: podsvt                      # Required - Identifier to the list or set read by the policy
      resource-rule:                    # Required - Resource selection criteria, at least one resource rule is required
        name:                           # Optional - Used to retrieve a specific resource in a single namespace
        group:                          # Optional - empty or "" for core group
        version: v1                     # Required - Version of resource
        resource: pods                  # Required - Resource type (API-recognized type, not Kind)
        namespaces: [validation-test]   # Optional - Namespaces to validate the above resources in. Empty or "" for all namespace or non-namespaced resources
        field:                          # Optional - Field to grab in a resource if it is in an unusable type, e.g., string json data. Must specify named resource to use.
          jsonpath:                     # Required - Jsonpath specifier of where to find the field from the top level object
          type:                         # Optional - Accepts "json" or "yaml". Default is "json".
          base64:                       # Optional - Boolean whether field is base64 encoded

Lula supports eventual-consistency through use of an optional wait field in the kubernetes-spec. This parameter supports waiting for a specified resource to be Ready in the cluster. This may be particularly useful if evaluating the status of a selected resource or evaluating the children of a specified resource.

domain:
  type: kubernetes
  kubernetes-spec:
    wait:                               # Optional - Group of resources to read from Kubernetes
      group:                            # Optional - Empty or "" for core group
      version: v1                       # Required - Version of resource
      resource: pods                    # Required - Resource type (API-recognized type, not Kind)Required - Resource type (API-recognized type, not Kind)
      name: test-pod-wait               # Required - Name of the resource to wait for
      namespace: validation-test        # Optional - For namespaced resources
      timeout: 30s                      # Optional - Defaults to 30s
    resources:
    - name: podsvt
      resource-rule:
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]

[!Tip] Both resources and wait use the Group, Version, Resource constructs to identify the resource to be evaluated. To identify those using kubectl, executing kubectl explain <resource/kind/short name> will provide the Group and Version, the resource field is the API-recognized type and can be confirmed by consulting the list provided by kubectl api-resources.

Resource Creation

The Kubernetes domain also supports creating, reading, and destroying test resources in the cluster. This feature should be used with caution since it’s writing to the cluster and ideally should be implemented on separate namespaces to make any erroneous outcomes easier to mitigate.

domain:
  type: kubernetes
  kubernetes-spec:
    create-resources:                   # Optional - Group of resources to be created/read/destroyed in Kubernetes
      - name: testPod                   # Required - Identifier to be read by the policy
        namespace: validation-test      # Optional - Namespace to be created if applicable (no need to specify if ns exists OR resource is non-namespaced)
        manifest: |                     # Optional - Manifest string for resource(s) to create; Only optional if file is not specified
          <some manifest(s)>
        file: '<some url>'              # Optional - File name where resource(s) to create are stored; Only optional if manifest is not specified. Currently does not support relative paths.

In addition to simply creating and reading individual resources, you can create a resource, wait for it to be ready, then read the possible children resources that should be created. For example the following kubernetes-spec will create a deployment, wait for it to be ready, and then read the pods that should be children of that deployment:

domain: 
  type: kubernetes
  kubernetes-spec:
    create-resources:
    - name: testDeploy
      manifest: |
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: test-deployment
          namespace: validation-test
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: test-app
          template:
            metadata:
              labels:
                app: test-app
            spec:
              containers:
              - name: test-container
                image: nginx
    wait:
      group: apps
      version: v1
      resource: deployments
      name: test-deployment
      namespace: validation-test
    resources:
    - name: validationTestPods
      resource-rule:
        version: v1
        resource: pods
        namespaces: [validation-test]

[!NOTE] The create-resources is evaluated prior to the wait, and wait is evaluated prior to the resources.

Lists vs Named Resource

When Lula retrieves all targeted resources (bounded by namespace when applicable), the payload is a list of resources. When a resource Name is specified - the payload will be a single object.

Example

Let’s get all pods in the validation-test namespace and evaluate them with the OPA provider:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule:
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider: 
  type: opa
  opa-spec:
    rego: |
      package validate

      import future.keywords.every

      validate {
        every pod in input.podsvt {
          podLabel := pod.metadata.labels.foo
          podLabel == "bar"
        }
      }

[!IMPORTANT] Note how the rego evaluates a list of items that can be iterated over. The podsvt field is the name of the field in the kubernetes-spec.resources that contains the list of items.

Now let’s retrieve a single pod from the validation-test namespace:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podvt
      resource-rule:
        name: test-pod-label
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider: 
  type: opa
  opa-spec:  
    rego: |
      package validate

      validate {
        podLabel := input.podvt.metadata.labels.foo
        podLabel == "bar"
      }

[!IMPORTANT] Note how the rego now evaluates a single object called podvt. This is the name of the resource that is being validated.

We can also retrieve a single cluster-scoped resource as follows, where the rego evaluates a single object called namespaceVt.

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: namespaceVt
      resource-rule:
        name: validation-test
        version: v1
        resource: namespaces
provider: 
  type: opa
  opa-spec:  
    rego: |
      package validate

      validate {
        input.namespaceVt.metadata.name == "validation-test"
      }

Extracting Resource Field Data

Many of the tool-specific configuration data is stored as json or yaml text inside configmaps and secrets. Some valuable data may also be stored in json or yaml strings in other resource locations, such as annotations. The field parameter of the resource-rule allows this data to be extracted and used by the Rego.

Here’s an example of extracting config.yaml from a test configmap:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: configdata
      resource-rule:
        name: test-configmap
        group:
        version: v1
        resource: configmaps
        namespaces: [validation-test]
        field:
          jsonpath: .data.my-config.yaml
          type: yaml
provider: 
  type: opa
  opa-spec:
    rego: |
      package validate

      validate {
        configdata.configuration.foo == "bar"
      }

Where the raw ConfigMap data would look as follows:

configuration:
  foo: bar
  anotherValue:
    subValue: ba

This field also supports grabbing secret data using the optional base64 field as follows

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: secretdata
      resource-rule:
        name: test-secret
        group:
        version: v1
        resource: secrets
        namespaces: [validation-test]
        field: 
          jsonpath: .data.secret-value.json
          type: json
          base64: true

Evidence Collection

The use of lula dev get-resources and lula validate --save-resources will produce evidence in the form of json files. These files provide point-in-time evidence for auditing and review purposes.

The Kubernetes domain requires connectivity to a cluster in order to perform data collection. The inability to connect to a cluster during the evaluation of a validation with --save-resources will result in an empty payload in the associated observation evidence file.

Evidence collection occurs for each resource specified and - in association with any error will produce an empty representation of the target resource(s) to be collected.

2 - Providers

Providers are the policy engines which evaluate the input data from the specified domain. Currently supported Providers are:

• OPA (Open Policy Agent)
• Kyverno

The provider block of a Lula Validation is given as follows, where the sample is indicating the OPA provider is in use:

# ... Rest of Lula Validation
provider:
    type: opa   # opa or kyverno accepted
    opa-spec:
        # ... Rest of opa-spec
# ... Rest of Lula Validation

Each domain specification retreives a specific dataset, and each will return that data to the selected Provider in a domain-specific format. However, this data will always take the form of a JSON object when input to a Provider. For that reason, it is important that Domain and Providerspecifications are not built wholly independently in a given Validation.

2.1 - Kyverno Provider

The Kyverno provider provides Lula with the capability to evaluate the domain in against a Kyverno policy.

Payload Expectation

The validation performed should use the form of provider with the type of kyverno and using the kyverno-spec, along with a valid domain.

Example:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule:
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider: 
  type: kyverno
  kyverno-spec:
    policy:
      apiVersion: json.kyverno.io/v1alpha1          # Required
      kind: ValidatingPolicy                        # Required
      metadata:
        name: pod-policy                            # Required
      spec:
        rules:
          - name: no-latest                         # Required
            # Match payloads corresponding to pods
            match:                                  # Optional
              any:                                  # Assertion Tree
              - apiVersion: v1
                kind: Pod
            assert:                                 # Required
              all:                                  # Assertion Tree
              - message: Pod `{{ metadata.name }}` uses an image with tag `latest`
                check:
                  ~.podsvt:
                    spec:
                      # Iterate over pod containers
                      # Note the `~.` modifier, it means we want to iterate over array elements in descendants
                      ~.containers:
                        image:
                          # Check that an image tag is present
                          (contains(@, ':')): true
                          # Check that the image tag is not `:latest`
                          (ends_with(@, ':latest')): false

You can have mutiple policies defined. Optionally, output.validation can be specified in the kyverno-spec to control which (Policy, Rule) pair control validation allowance/denial, which is in the structure of a comma separated list of rules: policy-name1.rule-name-1,policy-name-1.rule-name-2. If you have a desired observation to include, output.observations can be added to payload to observe violations by a certain (Policy, Rule) pair such as:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule: 
        group: 
        version: v1 
        resource: pods
        namespaces: [validation-test] 
provider: 
  type: kyverno
  kyverno-spec:
    policy:
      apiVersion: json.kyverno.io/v1alpha1
      kind: ValidatingPolicy
      metadata:
        name: labels
      spec:
        rules:
        - name: foo-label-exists
          assert:
            all:
            - check:
                ~.podsvt:
                  metadata:
                    labels:
                      foo: bar
    output:
      validation: labels.foo-label-exists
      observations:
      - labels.foo-label-exists

The validatation and observations fields must specify a (Policy, Rule) pair. These observations will be printed out in the remarks section of relevant-evidence in the assessment results.

2.2 - OPA Provider

The OPA provider provides Lula with the capability to evaluate the domain against a rego policy.

Payload Expectation

The validation performed should use the form of provider with the type of opa and using the opa-spec, along with a valid domain.

Example:

domain: 
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule:
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider:
  type: opa
  opa-spec:
    rego: |                                   # Required - Rego policy used for data validation
      package validate                        # Required - Package name

      import future.keywords.every            # Optional - Any imported keywords

      validate {                              # Required - Rule Name for evaluation - "validate" is the only supported rule
        every pod in input.podsvt {
          podLabel == "bar"
        }
      }

Optionally, an output can be specified in the opa-spec. Currently, the default validation allowance/denial is given by validate.validate, which is really of the structure <package-name>.<json-path-to-boolean-variable>. If you have a desired alternative validation boolean variable, as well as additional observations to include, an output can be added such as:

domain: 
  type: kubernetes
  kubernetes-spec:
    resource-rules: 
    - group: 
      version: v1 
      resource: pods
      namespaces: [validation-test] 
provider:
  type: opa
  opa-spec:
    rego: |
      package mypackage 

      result {
        input.kind == "Pod"
        podLabel := input.metadata.labels.foo
        podLabel == "bar"
      }
      test := "my test string"
    output:
      validation: mypackage.result
      observations:
      - validate.test

The validatation field must specify a json path that resolves to a boolean value. The observations array currently only support variables that resolve as strings. These observations will be printed out in the remarks section of relevant-evidence in the assessment results.

Policy Creation

The required structure for writing a validation in rego for Lula to validate is as follows:

rego: |
  package validate 

  validate {

  }

This structure can be utilized to evaluate an expression directly:

rego: |
  package validate 

  validate {
    input.kind == "Pod"
    podLabel := input.metadata.labels.foo
    podLabel == "bar"
  }

The expression can also use multiple rule bodies as such:

rego: |
  package validate

  foolabel {
    input.kind == "Pod"
    podLabel := input.metadata.labels.foo
    podLabel == "bar"
  }

  validate {
    foolabel
  }

[!IMPORTANT] package validate and validate are required package and rule for Lula use currently when an output.validation value has not been set.

Reusing OPA modules

Custom OPA modules can be imported and referenced in the main rego module. The following example shows how to import a custom module and use it in the main rego module:

Let’s say we have a file that is called lula.rego with the following contents that verifies that pods have the label lula: "true":

package lula.labels

import rego.v1

has_lula_label(pod) if {
    pod.metadata.labels.lula == "true"
}

We can import this module and use it in the main rego module as follows:

domain:
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule:
        group:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider:
  type: opa
  opa-spec:
    modules:
      lula.labels: lula.rego
    rego: |                                   # Required - Rego policy used for data validation
      package validate                        # Required - Package name

      import future.keywords.every            # Optional - Any imported keywords
      import data.lula.labels as lula_labels  # Optional - Import the custom module

      validate {                              # Required - Rule Name for evaluation - "validate" is the only supported rule
        every pod in input.podsvt {
          lula_labels.has_lula_label(pod)     # Use the rules defined in the custom module
        }
      }

[!Note] The validate.rego module name is reserved for the main rego policy and cannot be used as a custom module name.

3 - Testing

Testing is a key part of Lula Validation development. Since the results of the Lula Validations are determined by the policy set by the provider, those policies must be tested to ensure they are working as expected. The Validation Testing framework allows those tests to be specified directly in the validation.yaml file, and executed as part of the validate workflows.

See the “Test a Validation” tutorial for an example walkthrough

Specification

In the Lula Validation, a tests property is used to specify each test that should be performed against the validation. Each test is a map of the following properties:

• name: The name of the test
• changes: An array of changes or transformations to be applied to the resources used in the test validation
• expected-result: The expected result of the test - satisfied or not-satisfied

A change is a map of the following properties:

• path: The path to the resource to be modified. The path syntax is described below.
• type: The type of operation to be performed on the resource
  • update: (default) updates the resource with the specified value
  • delete: deletes the field specified
  • add: adds the specified value
• value: The value to be used for the operation (string)
• value-map: The value to be used for the operation (map[string]interface{})

An example of a test added to a validation is:

domain:
  type: kubernetes
  kubernetes-spec:
    resources:
    - name: podsvt
      resource-rule:
        version: v1
        resource: pods
        namespaces: [validation-test]
provider:
  type: opa
  opa-spec:
    rego: |
      package validate

      import future.keywords.every

      validate {
        every pod in input.podsvt {
          podLabel := pod.metadata.labels.foo
          podLabel == "bar"
        }
      }
tests:
  - name: modify-pod-label-not-satisfied
    expected-result: not-satisfied
    changes:
      - path: podsvt[metadata.namespace=validation-test].metadata.labels.foo
        type: update
        value: baz
  - name: delete-pod-label-not-satisfied
    expected-result: not-satisfied
    changes:
      - path: podsvt[metadata.namespace=validation-test].metadata.labels.foo
        type: delete

There are two tests here:

• The first test will locate the first pod in the validation-test namespace and update the label foo to baz. Then a validate will be executed against the modified resources. The expected result of this is that the validation will fail, i.e., will be not-satisfied, which would result in a successful test.
• The second test will locate the first pod in the validation-test namespace and delete the label foo, then proceed to validate the modified resources and compare to the expected result.

Path Syntax

This feature uses the kyaml library to inject data into the resources, so the path syntax is adapted from this library. With that said, the implementation in Lula offers some additional functionality to resolve more complex paths.

The path should be a “.” delimited string that specifies the keys along the path to the resource seeking to be modified. Any key followed by [*=*] will be treated as a list, where the content inside the brackets specify the item to be selected from the list. For example, the following path:

pods[metadata.namespace=grafana].spec.containers[name=istio-proxy]

Will start at the pods key, then since the next item is formatted as [=] it assumes pods is a list, and will iterate over each item in the list to find where the key metadata.namespace is equal to grafana. It will then find the containers list item in spec, and iterate over each item in the list to find where the key name is equal to istio-proxy.

Multiple filters can be added for a list, for example the above example could be modified to filter both by namespace and pod name:

pods[metadata.namespace=grafana,metadata.name=operator].spec.containers[name=istio-proxy]

To support map keys containing “.”, [] syntax will also be used, however the data inside the brackets will need to be encapsulated in quotes, e.g.,

namespaces[metadata.namespace=grafana].metadata.labels["some.key/label"]

Additionally, individual list items can be found via their index, e.g.,

namespaces[0].metadata.labels

Which will point to the labels key of the first namespace. Additionally, a [-] can be used to specify the last item in the list.

[!IMPORTANT] The path will return only one item, the first item that matches the filters along the path. If no items match the filters, the path will return an empty map.

Path Rules

• Path resolution supports both path.[key=value] and path[key=value] syntax
• In addition to simple selectors for a list, e.g., path[key=value], complex filters can be used, e.g., path[key=value,key2=value2] or path[key.subkey=value]
• Use double quotes to access keys that contain periods, e.g., foo["some.key"=value] or foo["some.key/label"]
• To access the index of a list, use [0] (where 0 is any valid index) or [-] for the last item in the list
• In the scenario where you need to access a map key which is a stringified integer (e.g., the “0” in { "foo": { "0": "some-value" }}), either enclose the key in quotes, foo["0"], or access through the normal path syntax, foo.0.

Change Type Behavior

Add/Update

• These are nearly identical, except that add will append to a list, while update will replace the entire list
• All keys in the path must exist, except for the last key. If you are trying to update a key with anything other than a string, use value-map and specify the existing root key:

original data:

{
  "foo": {
    "bar": 1
  }
}

desired data:

{
  "foo": {
    "bar": 0
  }
}

change:

changes:
  - path: foo
    type: update
    value-map:
      bar: 0

Delete

• You can delete list entries by specifying the index, e.g., path[0], or by specifying a selector, e.g., path[key=value]
• When using delete, value nor value-map should be specified

A note about replacing a key with an empty map - due to the way the kyaml merge functionality works, simply trying to overwrite an existing key with an empty map will not yield a removal of all the existing data of the map, it will just try and merge the differences, which is possibly not the desired outcome. To replace a map with an empty map, you must combine the delete change type and an add change type, e.g.,

changes:
  - path: pods.[metadata.namespace=grafana].metadata.labels
    type: delete
  - path: pods.[metadata.namespace=grafana].metadata
    type: add
    value-map: 
      labels: {}

Which will delete the existing labels map and then add an empty map, such that the “labels” key will still exist but will be an empty map.

Executing Tests

Tests can be executed by specifying the --run-tests flag when running both lula validate and lula dev validate, however the output of either will be slightly different.

lula validate

When running lula validate ... --run-tests, a test results summary will be printed to the console, while the test results yaml file will be written to the same directory as the output data (either default to directory of the component-definition source file or to the directory specified by the --output-file flag).

E.g., Running validate on a component-definition with two validations, one with tests and one without:

lula validate -f ./component.yaml --run-tests

Will print the test results summary to the console as:

Test Results: 1 passing, 0 failing, 1 missing

And will print a test results yaml file to the same directory as the output data:

61ec8808-f0f4-4b35-9a5b-4d7516053534:
    name: test-validation
    test-results: []
82099492-0601-4287-a2d1-cc94c49dca9b:
    name: test-validation-with-tests
    test-results:
        - test-name: change-image-name
          pass: true
          result: not-satisfied
        - test-name: no-containers
          pass: true
          result: not-satisfied

Note that 61ec8808-f0f4-4b35-9a5b-4d7516053534 is the UUID of the validation without tests, and 82099492-0601-4287-a2d1-cc94c49dca9b is the UUID of the validation with tests.

lula dev validate

When executing lula dev validate ... --run-tests, the test results data will be written directly to console.

E.g., Running dev validate on a Lula validation with two tests:

lula dev validate -f ./validation.yaml --run-tests

Will print the test results to the console as:

  ✔  Pass: change-image-name
  •  Result: not-satisfied
  ✔  Pass: no-containers
  •  Result: not-satisfied

To aid in debugging, the --print-test-resources flag can be used to print the resources used for each test to the validation directory, the filenames will be <test-name>.json.. E.g.,

lula dev validate -f ./validation.yaml --run-tests --print-test-resources

4 - Version Specification

In cases where a specific version of Lula is desired, either for typing constraints or desired functionality, a lula-version property is recognized in the description (component-definition.back-matter.resources[_]):

- uuid: 88AB3470-B96B-4D7C-BC36-02BF9563C46C
  remarks: >-
    No outputs in payload
  description: |
    lula-version: ">=0.0.2"
    domain: 
      type: kubernetes
      kubernetes-spec:
        resources:
        - name: podsvt
          resource-rule:
            group:
            version: v1
            resource: pods
            namespaces: [validation-test]
    provider:
      type: opa
      opa-spec:
        rego: |                                   
          package validate

          import future.keywords.every

          validate { 
            every pod in input.podsvt {
              podLabel == "bar"
            }
          }

If included, the lula-version must be a string and should indicate the version constraints desired, if any. Our implementation uses Hashicorp’s go-version library, and constraints should follow their conventions.

If an invalid string is passed or the current Lula version does not meet version constraints, the implementation will automatically be marked “not-satisfied” and a remark will be created in the Assessment Report detailing the rationale.