Use Lucent Sky AVM with GitLab Pipelines

This article describes how to integrate Lucent Sky AVM with GitLab Pipelines. By integrating Lucent Sky AVM with an application's continuous integration pipelines, developers can ensure that only code changes meeting the security standards are committed or deployed.

The Lucent Sky AVM CLI commands used in this article favor simplicity over scalability. For example, asynchronous methods such as BeginAnalyze might be more suitable than their synchronous counterparts when working with a large application. To learn about more advanced functionalities of the CLI, view the following article in the Lucent Sky Knowledge Base:
Lucent Sky AVM CLI reference

In this article, you will learn how to:

• Prepare a GitLab runner with Lucent Sky AVM CLI.
• Prepare Lucent Sky AVM CLI for use in GitLab Pipelines.
• Start a scan in GitLab Pipelines.
• Download and evaluate a scan report in GitLab Pipelines.
• Download the remediated source code and create a pull request in GitLab Pipelines.

At the end, you will be use Lucent Sky AVM in GitLab Pipelines to start a scan, download and evaluate reports, and work with remediated source code in GitLab Pipelines.

Prepare a GitLab runner with Lucent Sky AVM CLI

This article assumes that a GitLab instance and a GitLab runner are available. To learn more about how to install GitLab, view the following article on the GitLab website:
Install GitLab

Alternatively, the following Docker Compose template can be used to quickly prepare a GitLab instance and a GitLab runner for testing purposes.

version: '3'

services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab
    restart: always
    # Replace <GitLabInstanceUrl> with the URL of the GitLab instance
    hostname: '<GitLabInstanceUrl>'
    environment:
      # Replace <GitLabInstanceUrl> with the URL of the GitLab instance
      GITLAB_OMNIBUS_CONFIG: |
        external_url '<GitLabInstanceUrl>'
    ports:
      - '80:80'
      - '443:443'
      - '22:22'
    volumes:
      - './config:/etc/gitlab'
      - './logs:/var/log/gitlab'
      - './data:/var/opt/gitlab'

  gitlab-runner:
    image: 'gitlab/gitlab-runner:latest'
    container_name: gitlab-runner
    restart: always
    depends_on:
      - gitlab
    volumes:
      - './runner/config:/etc/gitlab-runner'
      - '/var/run/docker.sock:/var/run/docker.sock'
    environment:
      # Replace <GitLabInstanceUrl> with the URL of the GitLab instance
      - 'CI_SERVER_URL=<GitLabInstanceUrl>'
      # Replace <RegistrationToken> with the registration token generated by GitLab
      - 'REGISTRATION_TOKEN=<RegistrationToken>'
      - 'RUNNER_EXECUTOR=shell'
      - 'DOCKER_IMAGE=alpine:latest'

If the GitLab runner is not a container, run the following commands in the GitLab runner to install and configure Lucent Sky AVM CLI and its dependencies:

sudo apt update
sudo apt install mono-complete uuid -y
wget -O ~/clear-cli.zip https://lsky.co/clearcli
sudo unzip ~/clear-cli.zip -d /opt/clear-cli
rm ~/clear-cli.zip
sudo chown gitlab-runner:gitlab-runner -R /opt/clear-cli
sudo chmod u+x -R /opt/clear-cli

If the GitLab runner is a container, run the following commands in the Docker host to install and configure Lucent Sky AVM CLI and its dependencies:

sudo docker exec -it gitlab-runner sudo apt update
sudo docker exec -it gitlab-runner sudo apt install mono-complete uuid -y
sudo docker exec -it gitlab-runner wget -O ~/clear-cli.zip https://lsky.co/clearcli
sudo docker exec -it gitlab-runner sudo unzip ~/clear-cli.zip -d /opt/clear-cli
sudo docker exec -it gitlab-runner rm ~/clear-cli.zip
sudo docker exec -it gitlab-runner sudo chown gitlab-runner:gitlab-runner -R /opt/clear-cli
sudo docker exec -it gitlab-runner sudo chmod u+x -R /opt/clear-cli

The URL https://lsky.co/clearcli points to the latest Lucent Sky AVM CLI. To pin the CLI to a specific version, append -version after the URL. For example, https://lsky.co/clearcli-2406.

Register the GitLab runner

Before the GitLab runner can be used by a pipeline. It needs to be registered with the GitLab instance. To learn more about how to register a runner, view the following article on the GitLab website:
Registering runners

Notate the tag value when registering the runner, as it is needed when creating the pipeline.

Start a scan in GitLab Pipelines

1. Navigate to the Settings > CI/CD section of the GitLab project. Select Add Variable to create a new variable named ApiKey with the value of an API key to the Lucent Sky AVM server, and visibility set to Masked.

   GitLab provides various ways to store secrets. To learn more about how to store and use secrets in a GitLab pipeline, view the following article on the GitLab website:
   Pipeline security

2. Open the .yml file of the pipeline, and add a scan stage in stages section:

    stages:
    - build
    - scan
    - test
    - deploy
   

3. In the variables section of the .yml file, insert the following code to add the necessary variables:

    variables:
      # Replace <InstanceFqdn> with the FQDN or IP address of the CLEAR Engine instance
      InstanceFqdn: "<InstanceFqdn>"
      # Replace <ApplicationId> with the value of the project's application ID on the Lucent Sky AVM instance.
      ApplicationId: "<ApplicationId>"
      # Replace <ToolsDirectory> with the directory where Lucent Sky AVM CLI will be installed to.
      ToolsDirectory: "<ToolsDirectory>"
   

4. In the .yml file, locate an appropriate location to add a scan job, such as after the application build is completed.

5. Insert the following code to the .yml file, which creates a scan under the application on Lucent Sky AVM server, and uploads the build artifact for analysis.

    scan-job:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        # Set the sources directory to the working directory when the runner starts
        SourcesDirectory=`pwd`
        # Generate scan ID
        ScanId=`uuid`
        # Configure CLI to connect to a remote Lucent Sky AVM instance
        mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Interface Config --Method Set --Value "endpoint = ${InstanceFqdn}:5759"
        Create scan
        mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Create --ApplicationId $ApplicationId --ScanId $ScanId
        # Start scan
        mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Analyze --ScanId $ScanId --SourceCodePath $SourcesDirectory
   

Download and evaluate a scan report in GitLab Pipelines

1. Open the .yml file of the GitLab Pipeline, and locate an appropriate location to evaluate the scan report, such as after the scan is completed.

2. Insert the following code to the .yml file, which generates and downloads the XML report of the scan.

    download-xml-report:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        # Download XML report
        mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Report --ScanId $ScanId --ReportFormat "xml" --ReportPath "$TMPDIR/Xml-Report.zip"
   

3. Insert the following code to the .yml file, which extracts the XML report.

    extract-xml-report:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script: 
      - |
        # Extract XML report
        mkdir "$TMPDIR/ScanResults"
        unzip "$TMPDIR/ScanResults/Xml-Report.zip" -d "$TMPDIR/ScanResults"
   

4. Insert the following code to the .yml file, which queries the XML report to evaluate if the scan has found any vulnerability with a priority score of 2 or higher.

    query-xml-report:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        # Query XML report
        resultCount=`mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Interface Query --Method Execute --QueryDataSource "$TMPDIR/ScanResults/Report.xml" --QueryStatement "SELECT COUNT(ID) FROM Results WHERE PRIORITY <= 2"`
   

5. Insert the following code to the .yml file, which publishes the build artifact as a pipeline artifact named war if no vulnerability with a priority score of 2 or higher was found.

    publish-build-artifact-pipeline-artifact:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" == "0" ]; then
          # Publish artifact
          # Insert artifact publication code
        fi
   

6. Insert the following code to the .yml file, which generates and downloads the HTML report when at least one vulnerability with a priority score of 2 or higher was found.

    download-HTML-report:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then
          # Download HTML report
          mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Report --ScanId $ScanId --ReportFormat "html" --ReportPath "$TMPDIR/Html-Report.zip"
        fi
   

7. Insert the following code to the .yml file, which extracts the HTML report when at least one vulnerability with a priority score of 2 or higher was found.

    extract-HTML-report:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then
          # Extract HTML report
          unzip "$TMPDIR/ScanResults/Html-Report.zip" -d "$TMPDIR/ScanResults"
        fi
   

Download the remediated source code and create a pull request in GitLab Pipelines

1. Open the .yml file of the GitLab Pipeline, and locate an appropriate location to work with the remediated source code, such as after the pipeline failed security policy evaluation.

2. Insert the following code to the .yml file, which configures Git on the GitLab Pipelines agent and creates a branch for the remediated source code when at least one vulnerability with a priority score of 2 or higher was found.

    configure-git:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then 
          cd $SourcesDirectory
          # Replace <UserEmail> with the user email to use for Git
          UserEmail="<UserEmail>"
          # Replace <UserName> with the user name to use for Git
          UserName="<UserName>"
          git config --global user.email "$UserEmail"
          git config --global user.name "$UserName"
          git checkout -b scan-$ScanId
        fi
   

3. Insert the following code to the .yml file, which generates and downloads the remediated source code when at least one vulnerability with a priority score of 2 or higher was found.

    download-remediated-source-code:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then 
          # Download the remediated source code
          mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Remediate --ScanId $ScanId --RemediatedSourceCodePath "$TMPDIR/ScanResults/RemediatedSource.zip" --RemediationOption 0
        fi
   

4. Insert the following code to the .yml file, which extracts the remediated source code over the original source code when at least one vulnerability with a priority score of 2 or higher was found.

    extract-remediated-source-code:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then 
          # Extract the remediated source code
          unzip -o $TMPDIR/ScanResults/RemediatedSource.zip -d $SourcesDirectory
        fi
   

5. Insert the following code to the .yml file, which commits the remediated source code to a branch when at least one vulnerability with a priority score of 2 or higher was found.

    commit-and-push-remediated-source-code:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then 
          # Replace <AplPath> with Application Protection Library files suitable for the framework of the project
          AplPath="<AplPath>"
          git add **/$AplPath
          git commit -m "Instant Fixes from scan $ScanId"
          git push -u origin scan-$ScanId
        fi
   

6. Insert the following code to the .yml file, which fails the pipeline when at least one vulnerability with a priority score of 2 or higher was found.

    fail-pipeline:
    stage: scan
    tags:
      # Replace <Tag> with the tag set when registering the runner
      - <Tag>
    script:
      - |
        if [ "$resultCount" != "0" ]; then
          echo "This build did not pass the scan criteria."
          exit 1
        fi
   

7. Navigate to the Code > Merge requests section of the GitLab project, and create a merge request from the branch containing the remediated source code.