API Conformance Scan

API Conformance Scan is a dynamic runtime analysis of your API to check that the implementation behind your API and the behavior of the backend service matches the contract set out in the OpenAPI (formerly known as Swagger) definition of the API.

You can run a scan on an API you have imported to 42Crunch Platform and deployed to find out if there are any mismatches between the API definition describing your API and what it actually does. If Conformance Scan testing finds any discrepancies, it reports the issues clearly so that you can fix them.

The scan generates real traffic to the selected API endpoint and could incur costs depending on your setup.

For best results, make sure that your OpenAPI definition is valid and well-formatted before you scan it. The API must be deployed so that the API endpoint is live, and the backend server your API uses must be accessible to Conformance Scan. Otherwise the API cannot be scanned.

If your account belongs to the free community organization, you cannot scan APIs in 42Crunch Platform, but you can still use the on-premises version of the scan.

Conformance Scan can have potential side effects: APIs can throw exceptions, fail, and data can be affected. As per our terms and conditions, you must only scan APIs that you own, and only against non-production systems and non-production data! Do not use Conformance Scan in production environment!

What you can scan

Both OpenAPI Specification v2 and v3 are supported. The file size of your API should not exceed 10 MB.

By default, Conformance Scan limits the maximum length for strings in the requests it sends during the scan to 4096. If the properties minLength or maxLength that you have defined for an API operation in your API definition conflict with this limit, it causes issues during the scan.

If minLength is longer than the string length limit allowed in Conformance Scan, scan cannot create the happy path request for that operation to establish a baseline. If maxLength is longer than the allowed string length limit, Conformance Scan can create the happy path request, but not the actual request during the scan.

In both cases, the operation is shown as a skipped operation in the scan report, but for different reasons. You must fix the operation in your API definition before it can be successfully scanned.

Conformance Scan does not support operations that require request bodies with the content type multipart/formData. Only request bodies with the content type application/json and application/x​-www-form-urlencoded are supported.

How Conformance Scan works

After you have created a scan configuration for your API, Conformance Scan takes over and starts the scan process:

The graphic shows the scand process as an arrow where the four steps follow each other from left to right.

  1. Preparation: Conformance Scan checks the scan configuration you created and prepares the pieces required for the scan.
  2. Happy path requests: Conformance Scan generates and sends a happy path request to all operations in your API to establish a successful benchmark. Any operations where the happy path request fails are skipped in the scan (see Happy path requests).
  3. Generating tests: The scan generates the test requests for the API operations in the scan based on the happy path requests. Each test request includes an intentionally tweaked element (header, body, parameters, HTTP method) so that the request no longer matches what the API expects. The implementation of the API should catch this and respond accordingly.
  4. Scan: Conformance Scan sends tests requests at a constant flow rate to the live API endpoint.
    • Conformance Scan waits for the API to respond within 30 seconds before it raises a timeout error for the request in the scan logs.
    • When the API responds, Conformance Scan analyzes the received response to determine if the API conforms to the contract it sets out in its OpenAPI definition (see Response validation).

Unlike the static testing in API Security Audit, Conformance Scan is dynamic testing and variable by nature. To better simulate real API traffic and more reliably test the API's behavior, the requests and parameter values that Conformance Scan generates are random, as is the order in which the requests are sent to the API. As a result, the API responses and the outcome of the scan can also vary. So do not be alarmed if you get slightly different results for your API in scans, that is completely normal.

Generating values for parameters

To successfully call the API operations in your API, Conformance Scan must follow the OpenAPI definition of the API and provide the required parameters in the calls. For this, when Conformance Scan loads the API definition in memory, it generates default values for each schema and parameter in the OpenAPI definition, and uses these values in the requests it sends. Because Conformance Scan scan does not generate any responses itself (it only validates the responses coming directly from the API), response schemas are excluded.

It may be very difficult to create valid values for some things, like some object schemas, or strings with a very specific pattern. To ensure best performance, if you have complicated schemas in your API, we recommend including some examples for these kind of values directly in the OpenAPI definition.

There are several properties you can use for this, in the order of preference:

  • x-42c-sample
  • default
  • enum
  • example

If the property x-42c-sample contains a value that is not valid against the schema, Conformance Scan tries to load the value from the property default, and so on, until it finds a sample value it can use. As a last resort, or if no samples are provided at all, Conformance Scan generates a value from scratch. If Conformance Scan cannot generate a default value that an API operation requires, that operation is skipped in the scan.

For more details on the vendor extension, see x-42c-sample. For more details on the other properties, see the OpenAPI Specification (OAS).

Happy path requests

Conformance Scan needs a benchmark to determine if the incorrect behavior of the API was caused by the test request or some other failure. To establish this benchmark, Conformance Scan first sends a happy path request to the operations in the API before it starts the actual scan.

A happy path request is a valid request generated directly from the OpenAPI definition of your API, designed and expected to always succeed. Conformance Scan generates and sends this request to each operation defined in your API, and validates the responses it received.

To be a success, the response must be either successful or expected, and the received HTTP status code must be 200399, or 404. Otherwise, the happy path request fails. If a happy path request fails, the operation in question is skipped in the scan, because any results for it would be inconclusive without a successful benchmark.

An example screenshot showing scanned and skipped operations on the filter sidebar.

Happy path request failing often indicates significant issues in your OpenAPI definition that you must fix before Conformance Scan can scan the failing operation. Running Security Audit and checking what kind of issues it raises helps you find the culprits.

The happy path request can also fail because the connection to the defined endpoint was broken, or the API took too long (over 30 seconds) to respond.

Response validation

How the API responds to the crafted test requests in the scan determines whether or not it conforms to the contract it sets out in its API definition. To catch the issues, Conformance Scan validates the response from the API and considers, for example, the following:

  • Is the provided HTTP status code defined in the response object in the API definition?
  • Are all headers in the response defined in the response object in the API definition and do they match the defined schema? Are all required headers present in the response?
  • Is the returned response body too big?
  • Should the response even have a body (method HEAD or returned status code HTTP 204)?
  • Does the Content-Type of the returned response match the types defined in the content map in the API definition, and does it match the defined schema?

Conformance Scan tests how the API handles requests to operations not defined in the OpenAPI definition at all, as well as misconfigured requests to existing operations.

For non-existing operations, Conformance Scan expects the API to respond with HTTP 405 Method not allowed, and any other response would be considered as nonconforming.

For misconfigured requests to existing operations, Conformance Scan expects the following:

  • The returned HTTP status code is greater than HTTP 400.
  • The returned HTTP status code (or a default response) is defined in the OpenAPI definition.
  • The returned response body matches what is defined in the OpenAPI definition for the returned HTTP status code.
  • The returned response headers match what is defined in the OpenAPI definition (if response headers are defined).

If the response body in the response from the API exceeds 8 KB, it is truncated.

API endpoints in Conformance Scan

By default, Conformance Scan lists endpoint URLs that are parsed directly from your OpenAPI definition. However, if you want to use a URL that is not listed, you can also enter it when you configure the scan settings.

A screenshot of selecting the API endpoint in the scan configuration wizard

The URL you enter for the API endpoint must fulfill the following criteria:

  • It is a public URL.
  • It specifies either http or https (for example, http://255.255.255.255 or https://api.example.com/v2).
  • It is not an internal address (for example, http://localhost or 255.255.255.255).
  • It does not include parameters (for example, http://www.example.com/products?id=1&page=2).
  • It does not include anchors (for example, http://www.example.com#up).

Authentication in Conformance Scan

If you have defined authentication for your API or its operations, Conformance Scan must provide the required details to authenticate when it sends requests to the API.

When you configure the scan, you must therefore fill in the details for the security schemes in your API that you want to use in the scan. The configuration wizard shows all security schemes defined in the OpenAPI definition of your API:

A screenshot of authentication configuration. Two security schemes and the fields for configuring their details are listed, one for API key and one for session ID.

Fill in the details for the security schemes you want to use in the scan. You can leave the security schemes that you do not want to use in the scan empty and Conformance Scan will ignore these schemes. Any API operations that use only these security schemes for authentication are skipped in the scan.

If you have not defined any security requirements in the API definition, no authentication is required.

With on-premises scan, instead of hard-coding the authentication details in the scan configuration, you can use environment variables. See Using environment variables with on-premises scan.

Conformance Scan currently supports the following authentication types:

  • Basic authentication
  • API keys in headers, query strings, or cookies
  • Bearer token
  • OAuth 2

To configure OAuth2 authentication, you must first manually obtain an access token that Conformance Scan can use to authenticate to your API. In the scan configuration wizard, authentication with OAuth token is configured like bearer token. For more details on OAuth2, see RFC 6749.

If needed, you can also configure mutual TLS for client authentication. The client certificate must be in p12 format. For more details, see Scan API conformance.

If your run Conformance Scan from the platform, the authentication details are only used in the current scan and are not stored anywhere. If your run Conformance Scan on premises, the authentication details are stored encrypted as part of the scan configuration in the platform. The authentication details are not retrievable and credentials are hidden.

Errors in Conformance Scan

Occasionally, Conformance Scan might fail to scan your API. The reason for this could be, for example:

  • Invalid OpenAPI definition: You API definition has critical errors that are preventing Conformance Scan from running. For example, the structure of your API might not conform to the OpenAPI Specification. Use API Security Audit to check your API definition and fix any found issues in Security Editor, then try Conformance Scan again.
  • Invalid scan configuration: The configuration you set up for the scan does not match your API definition and thus is not valid. For example, you might have chosen an authentication method that does not match the ones defined in your API definition. Try configuring and running Conformance Scan again, making sure the authentication details match your API definition.
  • Scan cannot reach API endpoint: Conformance Scan tried to run the scan but failed to reach the API endpoint you had selected for the scan. The API host could be down, or there could be an error in the URL, especially if entered a custom URL. Check the URL and the host of your API and try again.
  • Timeout: The scan took longer than the maximum scan duration (3600 seconds).

Scan report

Conformance Scan produces a scan report that provides valuable information on how well your API conforms to its API definition. The report summarizes what was scanned and how the scan went.

At the top, you can see how many tests were run, how many passed or failed, and if the scan skipped any operations that could not be tested. Below that is the list of issues that the scan found in the API implementation.

An example of a scan report

The filter sidebar on the left lets you choose by path and method which issues are shown. The sidebar also shows how many issues were found in each path and method, as well as which operations the scan had to skip.

An example screenshot showing scanned and skipped operations on the filter sidebar.

Clicking an issue provides further details on it, such as the attack that the scan performed, the URL the scan called, and the response time of the API, or the size and content type of the response.

To make it easier to reproduce the issues, the report also provides the cURL requests the scan used to detect each issue.

You can download a copy of the report, for example, to share the results outside the platform.

Like with the audit score, you can also see how the number of issues that Conformance Scan found in your API has changed over time in the Conformance Scan trends chart on the API summary page.

On-premises scan

Instead of running Conformance Scan from the platform UI, you can deploy and run it locally as a Docker image. You can, for example, integrate this as a task that your CI/CD pipeline runs on every push to your repository. This way, you can automate checking that the API implementation matches the API contract.

To run on-premises scan, you create a scan configuration and then pull and run the Conformance Scan Docker image from Docker Hub, using the configuration you created.

On-premises scan produces a scan report just like when you run the scan from 42Crunch Platform, and you can view it the on-premises scan tab of the API. This tab also shows you the scan logs, when the scan was last run, and the scan configuration of the API.

Only ERROR and CRITICAL level scan logs are uploaded to 42Crunch Platform. Other log levels are written as standard output (STDOUT) to console, and how you can retrieve them depends on the environment (like Docker or Kubernetes) where you are running Conformance Scan on-premises. For more details, see the documentation of your environment.

An example screenshot of the report from on-premises scan for the Pixi API.

At the moment, the on-premises scan status of an API is not shown on the API collection page, only the status of scans run in the platform are listed there. You have to remember which APIs you have run the on-premises scan on, or click through your APIs to view their on-premises scan tab. Only the latest report from the on-premises scan is currently retained for the API. We are working on improving this in later releases.

Regular users can only create scan configurations and run the on-premises scan on APIs in their own API collections. Organization administrators can create scan configurations and run the on-premises scan on all APIs in their organization.

Scan configuration and scan token

On-premises scan requires a scan configuration that provides the details and settings what kind of scan to run and on which API. This configuration is created and updated in 42Crunch Platform, and it is stored encrypted. Currently, each API can have only one scan configuration.

An example screenshot of a scan configuration. The scan token is hidden by default and shown as Xs.

Choosing the API and the API endpoint to be scanned and configuring the possible authentication are same as for a regular scan. In addition, for on-premises scan, you can configure additional settings to customize it. Configuring these settings is entirely optional and not needed in most cases, the default settings that Conformance Scan uses are usually enough. However, for more advanced use, the settings let you tweak some aspects of the scan, such as flow rate or memory limits. For more details, see Adjust the scan configuration for on-premises scan.

Creating a scan configuration also produces a scan token. The token indicates to the on-premises scan which API it should scan and with which settings. When you run the Docker container for on-premises scan, the scan token is passed in the environment variable SCAN_TOKEN. When the on-premises scan starts, it connects to 42Crunch Platform and fetches the scan configuration that matches the specified scan token. This ensures that the on-premise scan runs the correct configuration for your API.

When the on-premises scan starts, it establishes a a two-way, HTTP/2 gRPC connection to 42Crunch Platform at the address services.<your hostname> and the port 8001. Make sure that your network configuration (like your network firewall) authorizes these connections. The on-premises scan uses this connection to verify the scan token and to download the scan configuration you have created. During runtime, on-premises scan uses the connection to send the scan report and logs to the platform.

If you are a user in the free Community organization and access 42Crunch Platform at https://platform.42crunch.com, your hostname is 42crunch.com, and the endpoint you must enable is services.42crunch.com.

If you are an enterprise customer not accessing 42Crunch Platform at https://platform.42crunch.com, your hostname is the same one as in your platform URL.

For more details on how to run the on-premises scan and customize the scan configuration, see Run Conformance Scan on your premises.

Using environment variables with on-premises scan

Because each API can have only one scan configuration that on-premises scan uses, you must delete the existing configuration before you can reconfigure the scan. However, for authentication details, you can use environment variables and supply the values when you run the on-premises scan. This way you can easily test the different authentication methods for your API.

When configuring the authentication for on-premises scan configuration, you can enter an environment variable to any field instead of hard-coding a value. The environment variable can be called anything you want, as long as fulfills the following criteria:

  • Must be inside curly brackets ({})
  • Must start with $
  • Cannot contain other special characters than -, _, and .
  • Must be longer than one character (just {$} is not a valid environment variable)

An example screenshot showing configuring token authentication for scan. The header value has been replaced with an environment value.

Environment variables are currently not supported for mutual TLS password.

When you run the on-premises scan, you provide the values for the environment variables in your run command. The variables must have a prefix SECURITY_ added before them, for example:

docker run -e SCAN_TOKEN=<your scan token> -e SECURITY_ACCESS_TOKEN='<the access token value you want to use>' 42crunch/scand-agent:latest