Use Conformance Scan v2

API Conformance Scan is a dynamic runtime analysis of your API to check that the implementation behind your API and the behavior of the backend service matches the contract set out in the OpenAPI (formerly known as Swagger) definition of the API.

Both OpenAPI Specification v2 and v3 are supported.

The scan generates real traffic to the selected API endpoint and could incur costs depending on your setup.

For best results, make sure that your OpenAPI definition is valid and well-formatted before you scan it. The API must be deployed so that the API endpoint is live, and the backend server your API uses must be accessible to Conformance Scan. Otherwise the API cannot be scanned.

Conformance Scan can have potential side effects: APIs can throw exceptions, fail, and data can be affected. As per our terms and conditions, you must only scan APIs that you own, and only against non-production systems and non-production data! Do not use Conformance Scan in production environment!

We have introduced a new version of Conformance Scan, referred to as Scan v2. For backward compatibility and to avoid the adoption of the new version disrupting your day-to-day work, we have retained the previous version, Scan v1. Both versions of Conformance Scan share the same core features and operation, but the new Scan v2 offers additional features and more flexibility. Where applicable, the difference between the versions has been clearly indicated in this documentation.

You can run Scan v1 in 42Crunch Platform or on premises as a Docker image. Scan v2 is currently available as Docker image for on-premises scan, or in v1-compatibility mode for running in 42Crunch Platform.

Create a scan configuration

You must create a scan configuration so that Conformance Scan knows which API endpoint to scan and how to run the run the scan. For Scan v2, you can have as many scan configurations for a single API as needed.

1. Find the API that you want to scan. The quickest way to find API definitions in 42Crunch Platform is to click Find API in the main menu.
2. Click Conformance Scan > Add configuration, and select what kind of configuration you want to create:
   • Version 2: Creates a full, editable default scan v2 configuration for running scan in normal v2 mode either on premises or in IDE. The configuration leverages the full capabilities of Scan v2, but it does not currently support scan rules.
   • Version 2 in v1-compatible mode: Creates a scan configuration for running scan v2 in v1-compatible mode. This mode allows running the scan configuration both in 42Crunch Platform and on premises, and it supports customization rules defined in the platform. You can have only one configuration for v1-compatible mode at a time, but you can overwrite the existing configuration as needed. This configuration does not take advantage of all Scan v2 features, but does provide smooth transition from Scan v1 to Scan v2.
3. If you chose to create a normal Scan v2 configuration, enter a name for the scan configuration. You can also provide a short description of, for example, what this configuration is for, or when it should be used. If you chose to create a Scan v2 in v1-compatible mode configuration, the name is automatically set and you cannot change it. Like for Scan v1, you must select which API endpoint to use for the scan.

   If you use the same name that an existing Scan v2 configuration has, your new scan configuration will replace the existing one.

4. Do one of the following:
•  Version 2: Choose how to create the scan configuration, then click Confirm:
  • If this your first Scan v2 configuration for this API, select 42Crunch default configuration to create a basic scan configuration based directly on the OpenAPI definition of your API. You can edit this configuration later.
  • If you want to use an existing Scan v2 configuration for this API that you have edited and expanded, select Custom configuration in a JSON file and select the file to upload as the new scan configuration. This is especially useful when creating alternative, more complex configurations in an editor of your choice.
• Version 2 in v1-compatible mode: Enter all authentication details for your API, configure the settings you want to use, and create the configuration.
  • You must provide authentication details for all security schemes defined in your API, or the scan will not run properly.
  • The authentication details are stored encrypted as part of the scan configuration in the platform.

    Currently, if you export a Scan v2 configuration from the platform, the credentials are exported in plaintext. This will be fixed in a future release.

  • When creating a configuration, you can configure the same settings as when creating a Scan v1 configuration.
  • You cannot configure any Scan v2 settings in v1-compatible mode. These settings will use their default values not to interfere with the v1-compatibility.

You have now created a scan configuration for the API that you selected, as well as a scan token that tells the Scan v2 which API to scan and with what settings.

Run Scan v2 on premises

You can run both normal Scan v2 configurations and Scan v2 in v1-compatible mode on premises as a Docker image. Before you can run Conformance Scan on premises, you must have created a valid scan configuration for the API in 42Crunch Platform and have Docker installed. For more details on Docker, see the Docker documentation.

When the on-premises scan starts, it establishes a two-way, HTTP/2 gRPC connection to 42Crunch Platform at the address services.<your hostname> and the port 8001. Make sure that your network configuration (like your network firewall) authorizes these connections. The on-premises scan uses this connection to verify the scan token and to download the scan configuration you have created. During runtime, on-premises scan uses the connection to send the scan report and logs to the platform.

If you are an enterprise customer not accessing 42Crunch Platform at https://platform.42crunch.com, your hostname is the same one as in your platform URL.

1. Go to the API you want to scan, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want to use, and copy and paste the sample Docker command shown on the General information tab for editing.

   

   Scan configurations and tokens are specific to a scan version: you cannot run Scan v2 using a Scan v1 scan token, and vice versa. When running a scan, make sure you specify the right scan token for the scan version you are using, otherwise Conformance Scan cannot use the associated scan configuration and fails to run.

3. If your API requires authentication, add flags in the command for all required authentication methods you want to use. To find and copy the required environment variables (for example, SCAN42C_SECURITY_OAUTH2), go to the Environment tab, and add those variables to your Docker command, for example:

   docker run -e SCAN_TOKEN=<your scan token> -e PLATFORM_SERVICE=services.42crunch.com:8001 -e SCAN42C_SECURITY_OAUTH2='<the token value you want to use>' 42crunch/scand-agent:v2

   If you do not provide credentials to all authentication methods listed as required in your scan configuration, Conformance Scan cannot run tests on all API operations because it cannot authenticate to your API. If you have defined an authentication method in you OpenAPI definition but your API does not actually use it, you can mark it as not required in your scan configuration to avoid having to provide credentials for it.

   

4. When ready, run your finalized Docker command.

The on-premise scan starts and checks the scan configuration, then runs the scan according to the configuration. When the scan is finished, you can see the number of issues where your API implementation does not conform to the API contract. If Conformance Scan could not scan one or more API operations, for example, because a happy path test failed, the scan is marked as incomplete to reflect that more work is still needed.

If you are on the API summary page or in the list of scan reports on the UI when the scan finishes, you must refresh the page before the on-premises scan report is available on the UI.

You only need to pull the image when you first run an on-premises scan, or if your want to update the scan image version. For Scan v2, you must always specify the exact image version to use, not latest.

Run on-premises scan behind HTTP or HTTPS proxy

In addition to connecting to the API to be scanned, on-premises scan also connects to 42Crunch Platform when it runs to fetch the scan configuration and to upload logs. This is not an issue as long as the machine it runs on has direct internet access.

However, if the internet access is behind an HTTP or HTTPS proxy, some additional configuration is needed. Conformance Scan uses two environment variables that you must configure depending on which connections must go through the proxy.

1. Configure the proxy behavior for your Docker daemon. For more details, see Docker documentation.
2. Use the correct command to pull and run the Docker image for the on-premises scan depending on which connections must go through the proxy:
   • Scan must connect to 42Crunch Platform through proxy:

     docker run -e SCAN_TOKEN=<your token value>  -e PLATFORM_SERVICE=services.<your hostname>:8001 -e HTTP_PROXY=<replace with your proxy server address> -e HTTPS_PROXY=<replace with your proxy server address> 42crunch/scand-agent:v2

   • Scan must connect to the API through proxy:

     docker run -e SCAN_TOKEN=<your token value>  -e PLATFORM_SERVICE=services.<your hostname>:8001 -e HTTP_PROXY_API=<replace with your proxy server address> -e HTTPS_PROXY_API=<replace with your proxy server address> 42crunch/scand-agent:v2

Replace the placeholders for the variables with the actual values you want to use, and remember to add the environment variables for the authentication details. If you do not need both HTTP and HTTPS proxy, remove the unnecessary one. If you are a community user, remove the flag PLATFORM_SERVICE.

Run Scan v2 in 42Crunch Platform

You can use Scan v2 in v1-compatible mode to run Scan v2 in 42Crunch Platform or use scan rules defined in 42Crunch Platform.

1. Go to the API you want to scan, and click Conformance Scan.
2. On the Scan v2 configuration in v1-compatible mode, click > Run scan.
3. Review the details of the read-only configuration, and click Run scan.

Conformance Scan runs the scan and produces a scan report.

You can also run Scan v2 in v1-compatible mode in 42Crunch Platform the same way as Scan v1, just change the scan engine version that you want to use in the configuration dialog. If the API already has an existing configuration for v1-compatible mode, you can review the read-only configuration but not edit it. If you do not yet have a configuration, you can create one.

View scan reports

When the on-premises scan runs, it creates and uploads a scan report and scan logs to 42Crunch Platform. You can view the latest scan report for each of your scan configurations in the platform.

1. Go to the API you want, and click Conformance Scan.

   

2. Click on the time stamp of the report from Scan v2 that you want. You can see the a quick summary of the scan, as well as the list of tests the scan run and the found issues. If you want more room for viewing the test list, you can collapse the summary above it.

   

3. Use the filters to filter and order what is being shown in the result list, for example, to see where and why happy path tests failed.
4. To view the details of an issue and the test that was run, click the issue on the list.

   

5. To view the logs that the scan produced and check how the scan progressed, go to Logs.

Export scan reports

You can export a scan report from 42Crunch Platform to share with others. If you do not want the full report with all the details, you can also export just the found issues as a CSV list.

We recommend you treat the exported report as company-confidential information and store it safely because it can reveal information on security risks in your APIs to outsiders.

1. Click Conformance Scan, and go to the scan report you want to export.
2. Click Export, and choose what you want to export:
   • To export the full report, click Save report as JSON.
   • To export just the list of all found issues as CSV, click Save issue list as CSV.

Set a reference scan configuration

Reference scan configuration is the quality reference for the API: the statistics and results from the reference scan configuration are the ones shown, for example, on the API summary page. Once you know which of the scan configurations is most representative of the API as a whole, you can set that one as the reference configuration.

1. Go to the API you want, click Conformance Scan.
2. Find the scan configuration you want, and click > Set as reference.

The scan configuration you selected is set as the reference scan configuration for your API and its scan status on the API summary page and in the API collection is updated accordingly. If you run scan using another scan configuration, the results shown on the API summary do not change when the scan process finish, but you can view the results of other scan configurations through the list of scan reports.

Because the reference scan configuration is needed to provide scan statistics for your API, you cannot delete that scan configuration before you choose a new reference scan configuration.

Change the scan settings

You can edit the automatically generated scan configuration as required.

1. Go to the API you want, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want.
3. Go to the Settings tab, and change the settings you want.

   

   You can, for example:

   • Change the time limits, like scan duration and timeout, and size limits for the scan report.

     If you run Conformance Scan in 42Crunch Platform, you cannot change maximum scan report size. However, when running the scan on premises, you can increase the maximum report size up to 128 MB.

     Large scan reports can take a long time to load in your browser. Depending on your browser memory limits and your device, if your scan report size exceeds 20 MB, you may not be able to view the scan report correctly on the platform UI. However, larger reports can be viewed in IDE. For more details on running Conformance Scan in IDE, see IDE integration.

   • Leave things that you do not need out of the scan report to keep the file size smaller.
   • Change scan behavior, such as following redirects, TLS verification, the flow rate for sent requests, or the testing mode (normal or lax).
   • Set the destination where Conformance Scan writes logs and on what level.

     If you set the log level to debug when running Scan v2, all the values of all variables in your scan configuration are printed in the clear in the scan logs. Depending on your API, this can include credentials.

   For the full list of settings and what they do, see API Conformance Scan settings.

4. When you are ready, click Save changes.

The changes to the configuration are saved and Conformance Scan validates the scan configuration against the OpenAPI definition of your API. If the configuration does not match the contract set out in the API definition, Conformance Scan flags the found errors in the scan configuration validation report (SCVR). You must resolve the errors before the configuration can be used in a scan.

When the configuration is valid, the updated settings are used the next time you run Conformance Scan using this configuration.

If you want to quickly test how changing a setting changes the way the scan operates, you can also specify the value as a flag in your Docker command. The flags must have the prefix SCAN42C_, followed by the name of the setting separated with underscores. For example, if you want to test how including only the found problems in the scan report (reportIssuesOnly) changes things, you would specify -e SCAN42C_REPORT_ISSUES_ONLY=true in your Docker command. The values from the Docker flags override the values in the scan configuration and are only used in that one scan, they do not change the underlying scan configuration.

Edit a scan scenario

Scan scenarios allow you to configure a flow of requests for testing a particular behavior in your API. They are configured as part of the scan configuration for the API to be scanned. The basic scan configuration already includes scenarios for all operations in your API and you can edit their details as needed.

1. Go to the API you want, and click Conformance Scan.
2. Click on the Scan v2 scan configuration you want, and go to the Scenarios tab. You can see the scenarios that already exist for each API operation, identified by the operationId (if available) or the path and method (verb) of the operation.
3. To view what happens in a scenario, click on the scenario you want. You can see the structure of the scenario and how many items each section has. You can expand the folds to view more details, such as which requests are included in the scenario.

   

4. To change the order of the request, click and choose the direction. You can also change which requests are the ones to be tested for conformance by switching Fuzzing on or off.
5. To change how the existing requests in a particular scenario work, such as creating variables or assigning different values to them, click on the request, edit the available details, and click Save changes. These changes apply only in the scope of this particular scenario.
6. To add more requests to the scenario, click Add request, and select the operation or external request you want. The request is added at the end of the existing scenario, but you can move its position as required and edit is details like with any request.

If your API changes (for example, you add a new operation), the scan configuration for the API is not automatically updated. To include the changes in the scan configuration, you must create a new configuration. If you have multiple scan configurations for different purposes where you need to include the changes, you can just copy the change you need from the new configuration and paste it into your existing ones.

Add external requests

Requests in a scenario usually target other operations defined in your API definition, but they can also be target external systems, for example, to fetch an authentication token or other data required in to execute the scenario. For this, you can add external requests to your scan configuration and use them in your scenarios.

1. On the scan configuration, go to the Operations tab. You can see the requests to all operations defined in your API.
2. To define a new external request, click Add, fill in the details for your request, and click Add external request. Your request is added to the list of available operations, and you can edit further details on it, like for any request.

   

3. To use your request in a scenario, add it as a request to the scenario you want.

Add multiple scenarios for an API operation

Scenarios in a scan configuration are organized by API operation and by default every API operation already has a happy path test scenario for them. If you need more happy path scenarios to test multiple HTTP codes for success or unhappy path scenarios to test error codes, you can manually add as many scenarios as you need. Currently, you must do so by editing the scan configuration in JSON format.

1. On the scan configuration, go to the Web editor tab. You can see the scenarios that already exist for each API operation, identified by the operationId (if available) or the path and method (verb) of the operation.
2. Find the API operation you want, and add a new scenario in the scenarios array. You can copy an existing scenario as the starting point.
3. Depending on what kind of scenario you are adding, make sure that the field key matches the type of the scenario you want:
   • For happy path scenario key: happy.path
   • For unhappy path scenario key: unhappy.path
   • For a custom scenario the key can be a custom string
4. When you are ready, click Save changes.

The scan configuration for the API operation in question now also includes the scenario you added and you can edit it like any of the default scenarios.

For more details on scan scenarios and their structure, see Scan scenarios

Add authorization tests to a scan scenario

Authorization tests let you check how your API implementation handles BOLA/IDOR (Broken Object Level Authorization, also known as Insecure Direct Object Reference) attack or BFLA (Broken Function Level Authorization).

1. On the scan configuration, go to the Authentication tab. You can see all security schemes defined in your API definition.

   If you need to test with a security scheme not defined in your API definition, you can click Add to add the new security scheme only in your scan configuration, without the need to touch your API definition.

2. Click the security scheme you want, and click + to add a new credential that you want to use to replace the correct one in the authorization tests.

   

3. To associate the credential you created for the test with operations where you want to test for BOLA or BFLA, click Add operation and select the operation or request you want.

   

4. Go to the Dynamic tests tab, click Add, give your test a name and select if it is a BOLA or BFLA test.
5. Select the credentials to be used in the authorization test and when ready, click Add authorization test:
   • Source: The correct credential that grants access to the resource
   • Target: The wrong credential that is not supposed to give access to the resource

   

6. To use the authorization test in a scenario, go to the scenario, click on Authorization tests > Add authorization test, and select the authorization test you want.

   

Next time you run scan using your scan configuration, Conformance Scan runs the authorization tests on the operations associated with the test credentials as the first thing in a scan, before it proceeds with the rest of the scan configuration.

Add preprocessing and postprocessing for the whole scan configuration

Sometimes same resources are needed across multiple operations in a scan, for example, the Petstore API would require an instance of a pet store for most tests. Global preprocessing and postprocessing scenarios let you define such overarching flows only once instead of doing this over and over again in each scenario.

1. On the scan configuration, go to the Pre-postprocessing tab. Click on the preprocessing or postprocessing section to open the section for viewing.
2. To add a new request to the flow, click Add operation and select the request you want. Just like with scenarios in the scan configuration, you can then edit the details of the request as needed and change the order of the requests in the flow.

   

3. To use your request in a scenario, add it as a request to the scenario you want.

Export and import scan configurations

While you can edit the scan configuration in 42Crunch Platform, sometimes you might want to edit the configuration as JSON in the editor of your choice. In this case, you can export the configuration from 42Crunch Platform for editing and then import it back once you are ready.

We recommend always starting with an automatically generated scan configuration for the API in question, to be sure that the configuration that you are starting from is valid for that API. Do not try to export a scan configuration for one API and use it for a different one.

1. Go to the API you want, click Conformance Scan, and find the Scan v2 scan configuration you want.
2. To export the scan configuration as a JSON file, click > Download. You can now open the file in an external editor and edit as needed.

   Currently, if you export a Scan v2 configuration from the platform, the credentials are exported in plaintext. This will be fixed in a future release.

3. To import the scan configuration back into 42Crunch Platform, go to Conformance Scan, and do one of the following:
   • If you want to replace the contents of a scan configuration already in 42Crunch Platform, click > Upload on the scan configuration you want to replace, and select the JSON file to upload.
   • If you want to import the edited scan configuration as a new scan configuration, click Add configuration > Version 2, give the configuration a new name, select Custom configuration in a JSON file in the dropdown, and select the JSON file to upload.

     If you use the same name that an existing Scan v2 configuration has, your new scan configuration will replace the existing one.

The JSON file is uploaded to 42Crunch Platform and Conformance Scan validates the scan configuration against the OpenAPI definition of the API. If the configuration does not match the contract set out in the API definition, Conformance Scan flags the found errors in the SCVR. You must resolve the errors before the configuration can be used in a scan.

Delete scan configurations

You can delete scan configurations you no longer need from 42Crunch Platform.

Deleting a scan configuration permanently removes it from 42Crunch Platform. This action cannot be undone.

1. Go to the API you want, click Conformance Scan, and find the Scan v2 scan configuration you want.
2. Click > Delete.

The scan configuration is deleted and can no longer be used to run Conformance Scan. If you had created, for example, Docker scripts to run Conformance Scan, you must updated those to use a different scan token and scan configuration, otherwise they will fail. The latest scan report for the configuration is also deleted together with the configuration.