Use Conformance Scan v2

API Conformance Scan is a dynamic runtime analysis of your API to check that the implementation behind your API and the behavior of the backend service matches the contract set out in the OpenAPI (formerly known as Swagger) definition of the API.

Both OpenAPI Specification v2 and v3 are supported.

The scan generates real traffic to the selected API endpoint and could incur costs depending on your setup.

For best results, make sure that your OpenAPI definition is valid and well-formatted before you scan it. The API must be deployed so that the API endpoint is live, and the backend server your API uses must be accessible to Conformance Scan. Otherwise the API cannot be scanned.

Conformance Scan can have potential side effects: APIs can throw exceptions, fail, and data can be affected. As per our terms and conditions, you must only scan APIs that you own, and only against non-production systems and non-production data! Do not use Conformance Scan in production environment!

We have introduced a new version of Conformance Scan, referred to as Scan v2. For backward compatibility and to avoid the adoption of the new version disrupting your day-to-day work, we have retained the previous version, Scan v1. Both versions of Conformance Scan share the same core features and operation, but the new Scan v2 offers additional features and more flexibility. Where applicable, the difference between the versions has been clearly indicated in this documentation.

You can run Scan v1 in 42Crunch Platform or on premises as a Docker image. Scan v2 is currently available as Docker image for on-premises scan, or in v1-compatibility mode for running in 42Crunch Platform.

Create a scan configuration

You must create a scan configuration so that Conformance Scan knows which API endpoint to scan and how to run the run the scan. For Scan v2, you can have as many scan configurations for a single API as needed.

1. Find the API that you want to scan. The quickest way to find API definitions in 42Crunch Platform is to click Find API in the main menu.
2. Click Conformance Scan > Add configuration, and select what kind of configuration you want to create:
   • Version 2: Creates a full, editable default scan v2 configuration for running scan in normal v2 mode either on premises or in IDE. The configuration leverages the full capabilities of Scan v2, but it does not currently support scan rules.
   • Version 2 in v1-compatible mode: Creates a scan configuration for running scan v2 in v1-compatible mode. This mode allows running the scan configuration both in 42Crunch Platform and on premises, and it supports customization rules defined in the platform. You can have only one configuration for v1-compatible mode at a time, but you can overwrite the existing configuration as needed. This configuration does not take advantage of all Scan v2 features, but does provide smooth transition from Scan v1 to Scan v2.
3. If you chose to create a normal Scan v2 configuration, enter a name for the scan configuration. You can also provide a short description of, for example, what this configuration is for, or when it should be used. If you chose to create a Scan v2 in v1-compatible mode configuration, the name is automatically set and you cannot change it. Like for Scan v1, you must select which API endpoint to use for the scan.

   If you use the same name that an existing Scan v2 configuration has, your new scan configuration will replace the existing one.

4. Do one of the following:
•  Version 2: Choose how to create the scan configuration, then click Confirm:
  • If this your first Scan v2 configuration for this API, select 42Crunch default configuration to create a basic scan configuration based directly on the OpenAPI definition of your API. You can edit this configuration later.
  • If you want to use an existing Scan v2 configuration for this API that you have edited and expanded, select Custom configuration in a JSON file and select the file to upload as the new scan configuration. This is especially useful when creating alternative, more complex configurations in an editor of your choice.
• Version 2 in v1-compatible mode: Enter all authentication details for your API, configure the settings you want to use, and create the configuration.
  • You must provide authentication details for all security schemes defined in your API, or the scan will not run properly.
  • The authentication details are stored encrypted as part of the scan configuration in the platform.

    Currently, if you export a Scan v2 configuration from the platform, the credentials are exported in plaintext. This will be fixed in a future release.

  • When creating a configuration, you can configure the same settings as when creating a Scan v1 configuration.
  • You cannot configure any Scan v2 settings in v1-compatible mode. These settings will use their default values not to interfere with the v1-compatibility.

You have now created a scan configuration for the API that you selected, as well as a scan token that tells the Scan v2 which API to scan and with what settings.

Run Scan v2 on premises

You can run both normal Scan v2 configurations and Scan v2 in v1-compatible mode on premises as a Docker image. Before you can run Conformance Scan on premises, you must have created a valid scan configuration for the API in 42Crunch Platform and have Docker installed. For more details on Docker, see the Docker documentation.

When the on-premises scan starts, it establishes a two-way, HTTP/2 gRPC connection to 42Crunch Platform at the address services.<your hostname> and the port 8001. Make sure that your network configuration (like your network firewall) authorizes these connections. The on-premises scan uses this connection to verify the scan token and to download the scan configuration you have created. During runtime, on-premises scan uses the connection to send the scan report and logs to the platform.

If you are an enterprise customer not accessing 42Crunch Platform at https://platform.42crunch.com, your hostname is the same one as in your platform URL.

1. Go to the API you want to scan, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want to use, and copy and paste the sample Docker command shown on the General information tab for editing.

   

   Scan configurations and tokens are specific to a scan version: you cannot run Scan v2 using a Scan v1 scan token, and vice versa. When running a scan, make sure you specify the right scan token for the scan version you are using, otherwise Conformance Scan cannot use the associated scan configuration and fails to run.

3. If your API requires authentication, add flags in the command for all required authentication methods you want to use. To find and copy the required environment variables (for example, SCAN42C_SECURITY_OAUTH2), go to the Web editor tab, scroll down to the environments object in your scan configuration, and add those variables to your Docker command, for example:

   docker run -e SCAN_TOKEN=<your scan token> -e PLATFORM_SERVICE=services.42crunch.com:8001 -e SCAN42C_SECURITY_OAUTH2='<the token value you want to use>' 42crunch/scand-agent:v2

   If you do not provide credentials to all required authentication methods listed in your scan configuration, Conformance Scan cannot run tests on all API operations because it cannot authenticate to your API.

4. When ready, run your finalized Docker command.

The on-premise scan starts and checks the scan configuration, then runs the scan according to the configuration. When the scan is finished, you can see the number of issues where your API implementation does not conform to the API contract. If Conformance Scan could not scan one or more API operations, for example, because a happy path request failed, the scan is marked as incomplete to reflect that more work is still needed.

If you are on the API summary page or in the list of scan reports on the UI when the scan finishes, you must refresh the page before the on-premises scan report is available on the UI.

You only need to pull the image when you first run an on-premises scan, or if your want to update the scan image version. For Scan v2, you must always specify the exact image version to use, not latest.

Run on-premises scan behind HTTP or HTTPS proxy

In addition to connecting to the API to be scanned, on-premises scan also connects to 42Crunch Platform when it runs to fetch the scan configuration and to upload logs. This is not an issue as long as the machine it runs on has direct internet access.

However, if the internet access is behind an HTTP or HTTPS proxy, some additional configuration is needed. Conformance Scan uses two environment variables that you must configure depending on which connections must go through the proxy.

1. Configure the proxy behavior for your Docker daemon. For more details, see Docker documentation.
2. Use the correct command to pull and run the Docker image for the on-premises scan depending on which connections must go through the proxy:
   • Scan must connect to 42Crunch Platform through proxy:

     docker run -e SCAN_TOKEN=<your token value>  -e PLATFORM_SERVICE=services.<your hostname>:8001 -e HTTP_PROXY=<replace with your proxy server address> -e HTTPS_PROXY=<replace with your proxy server address> 42crunch/scand-agent:v2

   • Scan must connect to the API through proxy:

     docker run -e SCAN_TOKEN=<your token value>  -e PLATFORM_SERVICE=services.<your hostname>:8001 -e HTTP_PROXY_API=<replace with your proxy server address> -e HTTPS_PROXY_API=<replace with your proxy server address> 42crunch/scand-agent:v2

Replace the placeholders for the variables with the actual values you want to use, and remember to add the environment variables for the authentication details. If you do not need both HTTP and HTTPS proxy, remove the unnecessary one. If you are a community user, remove the flag PLATFORM_SERVICE.

Run Scan v2 in 42Crunch Platform

You can use Scan v2 in v1-compatible mode to run Scan v2 in 42Crunch Platform or use scan rules defined in 42Crunch Platform.

1. Go to the API you want to scan, and click Conformance Scan.
2. On the Scan v2 configuration in v1-compatible mode, click > Run scan.
3. Review the details of the read-only configuration, and click Run scan.

Conformance Scan runs the scan and produces a scan report.

You can also run Scan v2 in v1-compatible mode in 42Crunch Platform the same way as Scan v1, just change the scan engine version that you want to use in the configuration dialog. If the API already has an existing configuration for v1-compatible mode, you can review the read-only configuration but not edit it. If you do not yet have a configuration, you can create one.

View scan reports

When the on-premises scan runs, it creates and uploads a scan report and scan logs to 42Crunch Platform. You can view the latest scan report for each of your scan configurations in the platform.

1. Go to the API you want, and click Conformance Scan.

   

2. Click on the time stamp of the report from Scan v2 that you want. You can see the a quick summary of the scan, as well as the list of tests the scan run and the found issues. If you want more room for viewing the test list, you can collapse the summary above it.

   

3. Use the filters to filter and order what is being shown in the result list, for example, to see where and why happy path requests failed.
4. To view the details of an issue and the test that was run, click the issue on the list.

   

5. To view the logs that the scan produced and check how the scan progressed, go to Logs.

Export scan reports

You can export a scan report from 42Crunch Platform to share with others. If you do not want the full report with all the details, you can also export just the found issues as a CSV list.

We recommend you treat the exported report as company-confidential information and store it safely because it can reveal information on security risks in your APIs to outsiders.

1. Click Conformance Scan, and go to the scan report you want to export.
2. Click Export, and choose what you want to export:
   • To export the full report, click Save report as JSON.
   • To export just the list of all found issues as CSV, click Save issue list as CSV.

Set a reference scan configuration

Reference scan configuration is the quality reference for the API: the statistics and results from the reference scan configuration are the ones shown, for example, on the API summary page. Once you know which of the scan configurations is most representative of the API as a whole, you can set that one as the reference configuration.

1. Go to the API you want, click Conformance Scan.
2. Find the scan configuration you want, and click > Set as reference.

The scan configuration you selected is set as the reference scan configuration for your API and its scan status on the API summary page and in the API collection is updated accordingly. If you run scan using another scan configuration, the results shown on the API summary do not change when the scan process finish, but you can view the results of other scan configurations through the list of scan reports.

Because the reference scan configuration is needed to provide scan statistics for your API, you cannot delete that scan configuration before you choose a new reference scan configuration.

Change the scan settings

You can edit the automatically generated scan configuration as required.

1. Go to the API you want, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want.
3. To check what settings the scan configuration is currently using, go to the Settings tab.

   

4. To quickly change a setting, go to the Web editor tab, and change the settings you want.

   

   You can, for example:

   • Change the time limits, like scan duration and timeout, and size limits for the scan report.

     If you run Conformance Scan in 42Crunch Platform, you cannot change maximum scan report size. However, when running the scan on premises, you can increase the maximum report size up to 128 MB.

     Large scan reports can take a long time to load in your browser. Depending on your browser memory limits and your device, if your scan report size exceeds 20 MB, you may not be able to view the scan report correctly on the platform UI. However, larger reports can be viewed in IDE. For more details on running Conformance Scan in IDE, see IDE integration.

   • Leave things that you do not need out of the scan report to keep the file size smaller.
   • Change scan behavior, such as following redirects, TLS verification, the flow rate for sent requests, or the testing mode (normal or lax).
   • Set the destination where Conformance Scan writes logs and on what level.

     If you set the log level to debug when running Scan v2, all the values of all variables in your scan configuration are printed in the clear in the scan logs. Depending on your API, this can include credentials.

   For the full list of settings and what they do, see API Conformance Scan settings.

5. When you are ready, click Save changes.

The changes to the configuration are saved and Conformance Scan validates the scan configuration against the OpenAPI definition of your API. If the configuration does not match the contract set out in the API definition, Conformance Scan flags the found errors in the scan configuration validation report (SCVR). You must resolve the errors before the configuration can be used in a scan.

When the configuration is valid, the updated settings are used the next time you run Conformance Scan using this configuration.

If you want to quickly test how changing a setting changes the way the scan operates, you can also specify the value as a flag in your Docker command. The flags must have the prefix SCAN42C_, followed by the name of the setting separated with underscores. For example, if you want to test how including only the found problems in the scan report (reportIssuesOnly) changes things, you would specify -e SCAN42C_REPORT_ISSUES_ONLY=true in your Docker command. The values from the Docker flags override the values in the scan configuration and are only used in that one scan, they do not change the underlying scan configuration.

Create a scan scenario

Scan scenarios allow you to configure a flow of requests for testing a particular behavior in your API. They are configured as part of the scan configuration for the API to be scanned. The basic scan configuration already includes some scenarios, but you can define as many additional scenarios as needed.

1. Go to the API you want, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want, and go to the Web editor tab to edit it. You can see the scenarios that already exist for each API operation, identified by the operationId (if available) or the path and method (verb) of the operation.
3. Find the API operation you want, and define a new scenario in the scenarios array. For more details on scan scenarios and their structure, see Scan token
4. When you are ready, click Save changes.

To view all scenarios in a scan configuration, as well as their details, go to the scan configuration you want and click Playbook.

Currently, playbooks are read-only: you can view them on the playbook tab, but if you want to change something, you must edit them in the web editor or in the editor of your choice. This will be improved on in a future release.

Export and import scan configurations

While doing simple changes to settings can be quick and easy in 42Crunch Platform, sometimes you might want to use the editor of your choice, especially when working on scan scenarios. In this case, you can export the configuration from 42Crunch Platform for editing and then import it back once you are ready.

We recommend always starting with an automatically generated scan configuration for the API in question, to be sure that the configuration that you are starting from is valid for that API. Do not try to export a scan configuration for one API and use it for a different one.

1. Go to the API you want, click Conformance Scan, and find the Scan v2 scan configuration you want.
2. To export the scan configuration as a JSON file, click > Download. You can now open the file in an external editor and edit as needed.

   Currently, if you export a Scan v2 configuration from the platform, the credentials are exported in plaintext. This will be fixed in a future release.

3. To import the scan configuration back into 42Crunch Platform, go to Conformance Scan, and do one of the following:
   • If you want to replace the contents of a scan configuration already in 42Crunch Platform, click > Upload on the scan configuration you want to replace, and select the JSON file to upload.
   • If you want to import the edited scan configuration as a new scan configuration, click Add configuration > Version 2, give the configuration a new name, select Custom configuration in a JSON file in the dropdown, and select the JSON file to upload.

     If you use the same name that an existing Scan v2 configuration has, your new scan configuration will replace the existing one.

The JSON file is uploaded to 42Crunch Platform and Conformance Scan validates the scan configuration against the OpenAPI definition of the API. If the configuration does not match the contract set out in the API definition, Conformance Scan flags the found errors in the SCVR. You must resolve the errors before the configuration can be used in a scan.

Delete scan configurations

You can delete scan configurations you no longer need from 42Crunch Platform.

Deleting a scan configuration permanently removes it from 42Crunch Platform. This action cannot be undone.

1. Go to the API you want, click Conformance Scan, and find the Scan v2 scan configuration you want.
2. Click > Delete.

The scan configuration is deleted and can no longer be used to run Conformance Scan. If you had created, for example, Docker scripts to run Conformance Scan, you must updated those to use a different scan token and scan configuration, otherwise they will fail. The latest scan report for the configuration is also deleted together with the configuration.