This article describes how to integrate Lucent Sky AVM with GitLab Pipelines. By integrating Lucent Sky AVM with an application's continuous integration pipelines, developers can ensure that only code changes meeting the security standards are committed or deployed.
The Lucent Sky AVM CLI commands used in this article favor simplicity over scalability. For example, asynchronous methods such as BeginAnalyze
might be more suitable than their synchronous counterparts when working with a large application. To learn about more advanced functionalities of the CLI, view the following article in the Lucent Sky Knowledge Base:
Lucent Sky AVM CLI reference
In this article, you will learn how to:
- Prepare a GitLab runner with Lucent Sky AVM CLI.
- Prepare Lucent Sky AVM CLI for use in GitLab Pipelines.
- Start a scan in GitLab Pipelines.
- Download and evaluate a scan report in GitLab Pipelines.
- Download the remediated source code and create a pull request in GitLab Pipelines.
At the end, you will be use Lucent Sky AVM in GitLab Pipelines to start a scan, download and evaluate reports, and work with remediated source code in GitLab Pipelines.
Prepare a GitLab runner with Lucent Sky AVM CLI
This article assumes that a GitLab instance and a GitLab runner are available. To learn more about how to install GitLab, view the following article on the GitLab website:
Install GitLab
Alternatively, the following Docker Compose template can be used to quickly prepare a GitLab instance and a GitLab runner for testing purposes.
version: '3'
services:
gitlab:
image: 'gitlab/gitlab-ce:latest'
container_name: gitlab
restart: always
# Replace <GitLabInstanceUrl> with the URL of the GitLab instance
hostname: '<GitLabInstanceUrl>'
environment:
# Replace <GitLabInstanceUrl> with the URL of the GitLab instance
GITLAB_OMNIBUS_CONFIG: |
external_url '<GitLabInstanceUrl>'
ports:
- '80:80'
- '443:443'
- '22:22'
volumes:
- './config:/etc/gitlab'
- './logs:/var/log/gitlab'
- './data:/var/opt/gitlab'
gitlab-runner:
image: 'gitlab/gitlab-runner:latest'
container_name: gitlab-runner
restart: always
depends_on:
- gitlab
volumes:
- './runner/config:/etc/gitlab-runner'
- '/var/run/docker.sock:/var/run/docker.sock'
environment:
# Replace <GitLabInstanceUrl> with the URL of the GitLab instance
- 'CI_SERVER_URL=<GitLabInstanceUrl>'
# Replace <RegistrationToken> with the registration token generated by GitLab
- 'REGISTRATION_TOKEN=<RegistrationToken>'
- 'RUNNER_EXECUTOR=shell'
- 'DOCKER_IMAGE=alpine:latest'
If the GitLab runner is not a container, run the following commands in the GitLab runner to install and configure Lucent Sky AVM CLI and its dependencies:
sudo apt update
sudo apt install mono-complete uuid -y
wget -O ~/clear-cli.zip https://lsky.co/clearcli
sudo unzip ~/clear-cli.zip -d /opt/clear-cli
rm ~/clear-cli.zip
sudo chown gitlab-runner:gitlab-runner -R /opt/clear-cli
sudo chmod u+x -R /opt/clear-cli
If the GitLab runner is a container, run the following commands in the Docker host to install and configure Lucent Sky AVM CLI and its dependencies:
sudo docker exec -it gitlab-runner sudo apt update
sudo docker exec -it gitlab-runner sudo apt install mono-complete uuid -y
sudo docker exec -it gitlab-runner wget -O ~/clear-cli.zip https://lsky.co/clearcli
sudo docker exec -it gitlab-runner sudo unzip ~/clear-cli.zip -d /opt/clear-cli
sudo docker exec -it gitlab-runner rm ~/clear-cli.zip
sudo docker exec -it gitlab-runner sudo chown gitlab-runner:gitlab-runner -R /opt/clear-cli
sudo docker exec -it gitlab-runner sudo chmod u+x -R /opt/clear-cli
The URL https://lsky.co/clearcli points to the latest Lucent Sky AVM CLI. To pin the CLI to a specific version, append -version after the URL. For example, https://lsky.co/clearcli-2406.
Register the GitLab runner
Before the GitLab runner can be used by a pipeline. It needs to be registered with the GitLab instance. To learn more about how to register a runner, view the following article on the GitLab website:
Registering runners
Notate the tag value when registering the runner, as it is needed when creating the pipeline.
Start a scan in GitLab Pipelines
-
Navigate to the Settings > CI/CD section of the GitLab project. Select Add Variable to create a new variable named ApiKey with the value of an API key to the Lucent Sky AVM server, and visibility set to Masked.
GitLab provides various ways to store secrets. To learn more about how to store and use secrets in a GitLab pipeline, view the following article on the GitLab website:
Pipeline security -
Open the .yml file of the pipeline, and add a
scan
stage instages
section:stages: - build - scan - test - deploy
-
In the
variables
section of the .yml file, insert the following code to add the necessary variables:variables: # Replace <InstanceFqdn> with the FQDN or IP address of the CLEAR Engine instance InstanceFqdn: "<InstanceFqdn>" # Replace <ApplicationId> with the value of the project's application ID on the Lucent Sky AVM instance. ApplicationId: "<ApplicationId>" # Replace <ToolsDirectory> with the directory where Lucent Sky AVM CLI will be installed to. ToolsDirectory: "<ToolsDirectory>"
- In the .yml file, locate an appropriate location to add a
scan
job, such as after the application build is completed. -
Insert the following code to the .yml file, which creates a scan under the application on Lucent Sky AVM server, and uploads the build artifact for analysis.
scan-job: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | # Set the sources directory to the working directory when the runner starts SourcesDirectory=`pwd` # Generate scan ID ScanId=`uuid` # Configure CLI to connect to a remote Lucent Sky AVM instance mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Interface Config --Method Set --Value "endpoint = ${InstanceFqdn}:5759" Create scan mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Create --ApplicationId $ApplicationId --ScanId $ScanId # Start scan mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Analyze --ScanId $ScanId --SourceCodePath $SourcesDirectory
Download and evaluate a scan report in GitLab Pipelines
- Open the .yml file of the GitLab Pipeline, and locate an appropriate location to evaluate the scan report, such as after the scan is completed.
-
Insert the following code to the .yml file, which generates and downloads the XML report of the scan.
download-xml-report: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | # Download XML report mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Report --ScanId $ScanId --ReportFormat "xml" --ReportPath "$TMPDIR/Xml-Report.zip"
-
Insert the following code to the .yml file, which extracts the XML report.
extract-xml-report: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | # Extract XML report mkdir "$TMPDIR/ScanResults" unzip "$TMPDIR/ScanResults/Xml-Report.zip" -d "$TMPDIR/ScanResults"
-
Insert the following code to the .yml file, which queries the XML report to evaluate if the scan has found any vulnerability with a priority score of 2 or higher.
query-xml-report: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | # Query XML report resultCount=`mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Interface Query --Method Execute --QueryDataSource "$TMPDIR/ScanResults/Report.xml" --QueryStatement "SELECT COUNT(ID) FROM Results WHERE PRIORITY <= 2"`
-
Insert the following code to the .yml file, which publishes the build artifact as a pipeline artifact named war if no vulnerability with a priority score of 2 or higher was found.
publish-build-artifact-pipeline-artifact: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" == "0" ]; then # Publish artifact # Insert artifact publication code fi
-
Insert the following code to the .yml file, which generates and downloads the HTML report when at least one vulnerability with a priority score of 2 or higher was found.
download-HTML-report: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then # Download HTML report mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Report --ScanId $ScanId --ReportFormat "html" --ReportPath "$TMPDIR/Html-Report.zip" fi
-
Insert the following code to the .yml file, which extracts the HTML report when at least one vulnerability with a priority score of 2 or higher was found.
extract-HTML-report: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then # Extract HTML report unzip "$TMPDIR/ScanResults/Html-Report.zip" -d "$TMPDIR/ScanResults" fi
Download the remediated source code and create a pull request in GitLab Pipelines
-
Open the .yml file of the GitLab Pipeline, and locate an appropriate location to work with the remediated source code, such as after the pipeline failed security policy evaluation.
-
Insert the following code to the .yml file, which configures Git on the GitLab Pipelines agent and creates a branch for the remediated source code when at least one vulnerability with a priority score of 2 or higher was found.
configure-git: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then cd $SourcesDirectory # Replace <UserEmail> with the user email to use for Git UserEmail="<UserEmail>" # Replace <UserName> with the user name to use for Git UserName="<UserName>" git config --global user.email "$UserEmail" git config --global user.name "$UserName" git checkout -b scan-$ScanId fi
-
Insert the following code to the .yml file, which generates and downloads the remediated source code when at least one vulnerability with a priority score of 2 or higher was found.
download-remediated-source-code: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then # Download the remediated source code mono $ToolsDirectory/clear-cli/SkyAnalyzer.Interface.Console.exe --Key $ApiKey --Interface Scan --Method Remediate --ScanId $ScanId --RemediatedSourceCodePath "$TMPDIR/ScanResults/RemediatedSource.zip" --RemediationOption 0 fi
-
Insert the following code to the .yml file, which extracts the remediated source code over the original source code when at least one vulnerability with a priority score of 2 or higher was found.
extract-remediated-source-code: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then # Extract the remediated source code unzip -o $TMPDIR/ScanResults/RemediatedSource.zip -d $SourcesDirectory fi
-
Insert the following code to the .yml file, which commits the remediated source code to a branch when at least one vulnerability with a priority score of 2 or higher was found.
commit-and-push-remediated-source-code: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then # Replace <AplPath> with Application Protection Library files suitable for the framework of the project AplPath="<AplPath>" git add **/$AplPath git commit -m "Instant Fixes from scan $ScanId" git push -u origin scan-$ScanId fi
-
Insert the following code to the .yml file, which fails the pipeline when at least one vulnerability with a priority score of 2 or higher was found.
fail-pipeline: stage: scan tags: # Replace <Tag> with the tag set when registering the runner - <Tag> script: - | if [ "$resultCount" != "0" ]; then echo "This build did not pass the scan criteria." exit 1 fi
-
Navigate to the Code > Merge requests section of the GitLab project, and create a merge request from the branch containing the remediated source code.