5 posts tagged with "Security"

Often in software engineering, when we are developing new features, it is quite a common feature that we would encode certain sensitive information, such as passwords, secret keys, or tokens, for our code to do its intended functionality. Different professionals within the IT realm use it in different ways, such as the following:

• Developers use secrets from API tokens, Database credentials, or other sensitive information within the code.
• Dev-ops engineers might have to export certain values as environment variables and write the values in the YAML file for CI/CD pipeline to run efficiently.
• The cloud engineers might have to pass the credentials, secret tokens, and other secret information for them to access their respective cloud (In the case of AWS, even if we save these in a .credentials file, we still have to pass the filename in terraform block, which would indicate that the credentials are available locally within the computer.)
• The system administrators might have to send different logins and passwords to the people for the employees to access different services

But writing it in plain text or sharing it in plain text is quite a security problem, as anyone logging in to the code-base might access the secret or pull up a Man-in-the-Middle attack. To counter this, in the developing world, we have different options like Importing secrets from another file ( YAML, .py, etc.) or exporting them as an environment variable. But both of these still have a problem: a person having access to a singular config file or the machine can echo out the password ( read print). Given these problems, it would be very useful if we could deploy a single solution that would provide solutions to all the IT professionals mentioned above and more. This is the ideal place for introducing Vault.

HashiCorp Vault — an Introduction​

HashiCorp Vault is a secrets and encryption management system based on user identity. If we have to compare it with AWS, it is like an IAM user-based resource (read Vault here) management system which secures your sensitive information. This sensitive information can be API encryption keys, passwords, and certificates.

Its workflow can be visualized as follows:

Hosting Cost of Vault​

• Local hosting: This method is usually done if the secrets are to be accessed only by the local users or during the development phase. This method has to be shunned if these secret engines have to be shared with other people. As it is within the local development environment, there is no additional investment for deployment. This can be hosted directly in a local machine or by its official docker image
• Public Cloud Hosting ( EC2 in AWS/Virtual Machine in Azure): If the idea is to set up Vault to share with people across different regions, hosting it on Public cloud is a good idea. Although we can achieve the same with the on-prem servers, upfront costs and scalability is quite a hassle. In the case of AWS, we can easily secure the endpoint by hosting Vault in the EC2 instance and creating a Security group on which IPs can access the EC2. If you feel more adventurous, you can map this to a domain name and route from Route 53 so the vault is accessible on a domain as a service to the end users. In the case of EC2 hosting with an AWS-defined domain, the cost is $0.0116/hr.
• Vault cloud Hosting (HashiCorp Cloud Platform): If you don’t want to set up infrastructure in the Public Cloud Environments, there is an option of choosing the cloud hosted by vault. We can think of it as a SaaS-based cloud platform that enables us to use the Vault as a service on a subscription basis. Since hashicorp itself manages the cloud, we can expect a consistent user experience. For the cost, it has three production grade options: Starter at $ 0.50/hr, Standard at $1.58/hr, and Plus at $1.84/hr (as seen in July 2022).

Example of Self-Hosting in AWS Cloud​

Our goal in this Project is to create a Vault instance in EC2 and store static secrets in the Key—Value secrets engine. These secrets are later retrieved into the terraform script, which, when applied, would pull the secrets from the Vault Secrets Engine and use them to create infrastructure in AWS.

To create a ready-to-use Vault, we are going to follow these steps:

1. Create an EC2 Linux instance with ssh keys to access it.
2. SSH into the instance and install the Vault to get it up and running
3. Configure the Valve Secrets Manager

Step 1: Create an EC2 Linux instance with ssh keys to access it​

To create an EC2 instance and access it remotely via SSH, we need to create the Key pair. First, let's create an SSH key via the AWS console.

Once the Keys have been created and downloaded into the local workbench, we create an EC2 (t2.micro) Linux instance and associate it with the above keys. The size of the EC2 can be selected based on your requirements, but usually, a t2.micro is more than enough.

Step 2: SSH into the instance and install the secrets to get it up and running​

Once the status of the EC2 changes to running, open the directory in which you have saved the SSH(.pem) key. Open a terminal and type ssh -i <keyname.pem> ec2-user @<publicdns IP4> . Once we have established a successful SSH session into our Ec2 instance, we can install the Vault using the following commands:

wget -O- <https://apt.releases.hashicorp.com/gpg> | gpg — dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg  
  
echo "deb \[signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg\] <https://apt.releases.hashicorp.com> $(lsb\release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list  
  
sudo apt update && sudo apt install vault

The above command would install the vault in the EC2 environment. Sometimes the second command is known to throw some errors. In case of an error, replace $(lsb_release -cs) with “jammy”. [This entire process can be automated by copying the above commands to EC2 user data while creating an EC2 instance].

Step 3: Configure the Hashicorp valve​

Before initializing the vault, let's ensure it is properly installed by following the command:

vault

Let's make sure there is no environment variable called VAULT_TOKEN. To do this, use the following command:

$ unset VAULT_TOKEN

Once we have installed the Vault, we need to configure the Vault, which is done using HCL files. These HCL files contain data such as backed, listeners, cluster address, UI settings, etc. As we have discussed in the Vault’s Architecture, the back end on which the data is stored is very different from the vault engine, which is to be persisted even when the vault is locked (Stateful resource). In addition to that, we need to specify the following details:

• Listener Ports: the port/s on which the Vault listens for API requests.
• API address: Specifies the address to advertise to route client requests.
• Cluster address: Indicates the address and port to be used for communication between the Vault nodes in a cluster. To secure it much further, we can use TLS-based communication. This step is optional and can only be tried if you want to secure your environment further. The TLS Certificate can be generated using openssl in Linux.

# Installs openssl  
sudo apt install openssl  
  
#Generates TLS Certificate and Private Key  
openssl req -newkey rsa:4096 -x509 -sha512 -days 365 -nodes -out certificate.pem -keyout privatekey.pem 

Insert the TLS Certificate and Private Key file paths in their respective arguments in the listener “tcp” block.

• tls_cert_file: Specifies the path to the certificate for TLS in PEM encoded file format.
• tls_key_file: Specifies the path to the private key for the certificate in PEM-encoded file format.

#Configuration in config.hcl file  
  
storage "raft" {   
path = "./vault/data"   
node\id = "node1"   
}  
listener "tcp" {  
 address = "127.0.0.1:8200"   
tls\disable = "true"  
tls\cert\file = certificate.pem  
tls\key\file = privatekey.pem  
}  
disable\mlock = true  
api_addr = "http://127.0.0.1:8200"   
cluster_addr = "https://127.0.0.1:8201"  
 ui = true

Once these are created, we create the folder where our backend will rest: vault/data.

mkdir -p ./vault/data

Once done, we can start the vault server using the following command:

vault server -config=config.hcl

Once done, we can start our Vault instance with the backend mentioned in the config file and all its settings.

export VAULT_ADDR='http://127.0.0.1:8200'  
  
vault operator init

After it is initialized, it creates five Unseal keys called shamir keys (out of which three are used to unseal the Vault by default settings) and an Initial root token. This is the only time ever that all of this data is known by Vault, and these details are to be saved securely to unseal the vault. In reality, these shamir keys are to be distributed among key stakeholders in the project, and the Key threshold should be set in such a fashion that Vault can be unsealed when the majority are in consensus to do so.

Once we have created these Keys and the initial token, we need to unseal the vault:

vault operator unseal

Here we need to supply the threshold number of keys to unseal. Once we supply that, the sealed status changes to false.

Then we log in to the Vault using the Initial root token.

vault login

Once authenticated successfully, you can easily explore different encryption engines, such as Transit secrets Engine. This helps encrypt the data in transit, such as the Key-Value Store, which is used to securely store Key-value pairs such as passwords, credentials, etc.

As seen from the process, Vault is pretty robust in terms of encryption, and as long as the shamir keys and initial token are handled in a sensitive way, we can ensure the security and integrity

And you have a pretty secure Vault engine (protected by its own shamir keys) running on a free AWS EC2 instance (which is, in turn, guarded by the security groups)!

**Want to Connect?**  
  
If you want to connect with me, you can do so on [LinkedIn](https://www.linkedin.com/in/krishnadutt/).

Recently while trying to integrate a devsecops tool in my pipeline, i was trying to find the GitHub action which would simplify my workflow. Since i could not find it, i have to write the commands inline to run the command. Although it is not a hassle to write it within the script, it would be beneficial to have an action which we could directly call, pass parameters and run the action within the pipeline.

In this blog, i will walk you through different steps on how you can create a custom GitHub actions which would satisfy your requirement. The blog will be of 2 parts:

• Understanding what the GitHub action are
• Creating your custom GitHub actions

GitHub actions:​

Often when we write pipelines, we would have a set of actions which we would like to perform based on the type of application that we are developing. In order for us to run these actions across the repos in our organization, we would have to copy + paste this code across the repositories, which would make this process error prone and maintenance tussle. It would be better if we take DRY principle of software engineering and apply it to CI/CD world.

GitHub action is exactly this principle in practice. We create and host the required action in a certain GitHub public repository and this action is used across the pipeline to perform the action defined in the action. Now that we understand what GitHub action is, lets explore how we can build a custom GitHub action which can help automate set of actions. For this blog, i illustrate it with an example of SBOM enrichment tool Parlay, for which i have built a custom action.

Creating Custom Action — A case on Parlay​

We will be creating our custom action in the following steps:

• Defining inputs and outputs in action.yml
• Developing business logic in bash script
• Dockerize the bash application
• Test the action
• Publish it in GitHub action Marketplace

Defining inputs and outputs in action.yml

To start creating custom action create a custom git repository, clone that repo in your local system and open it up in your favourite code editor. We start by creation a file named actions.yml. This actions.yml defines the inputs that the action would take, the outputs that it would give and the environment it will run. For our use case we have 3 inputs and 1 output. The actions.yml should have following arguments:

• name: This would be the name of the action, which would be used to search in GitHub action marketplace. Since it would be published in marketplace, it’s name should be globally unique like s3 bucket.
• description: This describes what your action would do. This would be helpful to identify which action would be the right fit for our use case.
• inputs: Defines the list of options which would be used within the action. These can be compulsory or optional, which can be defined using “required” argument. In our current use case we are passing 3 arguments, input_file_name, enricher and output_file_name.
• outputs: This enlists the list of outputs that the action gives.
• runs: defines the environment in which action will execute , which in our case is docker

The action.yml will look something like this:

\# action.yaml  
name: "Parlay Github Action"  
description: "Runs Parlay on the given input file using the given enricher and outputs it in your given output file"  
branding:  
  icon: "shield"  
  color: "gray-dark"  
inputs:  
  input_file_name:  
    description: "Name of the input SBOM file to enrich"  
    required: true  
  enricher:  
    description: "The enricher used to enrich the parlay sbom. Currently parlay supports ecosystems, snyk, scorecard(openssf scorecard)"  
    required: true  
    default: ecosystems  
  output_file_name:  
    description: "Name of the output file to save the SBOM enriched using the parlay's enricher"  
    required: true  
outputs:  
  output_file_name:  
    description: "Prints the output file"  
runs:  
  using: "docker"  
  image: "Dockerfile"  
  args:  
    - ${{ inputs.input_file_name }}  
    - ${{ inputs.enricher }}  
    - ${{ inputs.output_file_name }}

Developing business logic in bash script​

Once we have defined the inputs, outputs and environment that we are going to use, we would like to define what we are going to do with those inputs ( basically our logic) in a file. We can either define this in JavaScript or bash. For my current use case, i am using bash.

In my current logic, i am going to check if all the inputs are first given, if not the action fails. Once i have these 3 arguments, i am going to construct the command to run the action and save the output in an output file. This file is printed in stdout and formatted using jq utility.

#!/bin/bash  
\# Check if all three arguments are provided  
if [ "$#" -ne 3 ]; then  
    echo "Usage: $0 <input> <input_file_name> <output_file_name>"  
    exit 1  
fi  
\# Extract arguments  
INPUT_INPUT_FILE_NAME=$1  
INPUT_ENRICHER=$2  
INPUT_OUTPUT_FILE_NAME=$3  
\# Construct command  
full_command="parlay $INPUT_ENRICHER enrich $INPUT_INPUT_FILE_NAME > $INPUT_OUTPUT_FILE_NAME"  
eval "$full_command"  
\# Check if the command was successful  
if [ $? -eq 0 ]; then  
    echo "Command executed successfully: $full_command"  
    cat $INPUT_OUTPUT_FILE_NAME | jq .  
else  
    echo "Error executing command: $full_command"  
fi

Dockerize the bash application​

Once we have the bash script ready, we will be dockerizing it using the following script. Whenever we invoke the action, this action which is defined in the bash script runs in an isolated docker container. In addition to the bash script in entrypoint.sh, we would also be adding the the required libraries such as wget, jq and installing parlay binary.

\# Base image  
FROM --platform=linux/amd64 alpine:latest  
\# installes required packages for our script  
RUN apk add --no-cache bash wget jq  
\# Install parlay  
RUN wget <https://github.com/snyk/parlay/releases/download/v0.1.4/parlay_Linux_x86_64.tar.gz>   
RUN tar -xvf parlay_Linux_x86_64.tar.gz   
RUN mv parlay /usr/bin/parlay  
RUN ls /usr/bin | grep parlay  
RUN parlay  
\# Copies your code file  repository to the filesystem  
COPY . .  
\# change permission to execute the script and  
RUN chmod +x /entrypoint.sh  
\# file to execute when the docker container starts up  
ENTRYPOINT ["/entrypoint.sh"]

Test the action​

No amount of software is good without running some tests on it. To test the action, lets first push the code to GitHub. Once pushed, lets define the pipeline in pipeline.yaml file in .github/workflows folder. For the sake of input file, i am using a sample sbom file in cyclonedx format and have pushed it to GitHub. In my pipeline.yaml file, i am cloning the GitHub repo and using my action called krishnaduttPanchagnula/parlayaction@main on cyclonedx.json.

on: [push]  
jobs:  
  custom_test:  
    runs-on: ubuntu-latest  
    name: We test it locally with act  
    steps:  
      - name: Checkout git branch  
        uses: actions/checkout@v1  
          
      - name: Run Parlay locally and get result  
        uses: krishnaduttPanchagnula/parlayaction@main  
        id: parlay  
        with:  
          input_file_name: ./cyclonedx.json  
          enricher: ecosystems  
          output_file_name: enriched_cyclonedx.json

Once the pipeline runs, this should give output in the std-out in pipeline console as follows.

Parlay Github action Output

Publish it in GitHub action Marketplace

Once we have tested the action and that is running fine, we are going to publish it GitHub actions market place. TO do so, our custom app should have globally unique name. To make it more unique we can add icon with our custom symbol and colour to uniquely identify the action in marketplace.

Once that is done you would see the button, “Draft a Release”. Ensure that your action.yml file has Name, description, Icon and color.

Once you have tick marks, you would be guided to release page where you can mention the title and version of the release. After that, click on “publish release” and you should be able to see your action in GitHub Actions marketplace.

Liked my content ? Feel free to reach out to my LinkedIn for interesting content and productive discussions.

Introduction​

In today’s era, the internet has become embedded into the very fabric of our lives. It has revolutionized the way we communicate, work, shop, and entertain ourselves. With the increasing amount of personal information that we share online, data security has become a major concern. Cyber-criminals are constantly on the lookout for sensitive data such as credit card information, social security numbers, and login credentials to use it for identity theft, fraud, or other malicious activities.

Moreover, governments and companies also collect large amounts of data on individuals, including browsing history, location, and personal preferences, to model the behavior of the users using deep-learning clustering models. This data can be used to coerce users psychologically to buy their products or form an opinion that they want us to form.

To overcome this issue, we can use a VPN which can be used to mask the user’s identity and route our traffic through a remote server. In addition, we can bypass internet censorship and access content that may be restricted in our region, which enables us to access our freedom to consume the data we want rather than what the governments/legal entities want us to consume. The VPNs we will discuss are of two types: Public VPNs such as Nord VPN, Proton VPN, etc., and private VPNs. Let’s try to understand the differences amongst them.

Private vs Public VPN​

Public VPNs are VPN services that are available to the general public for a fee or for free. These services typically have servers located all around the world, and users can connect to any of these servers to access the internet.

Private VPNs, on the other hand, are VPNs that are created and managed by individuals or organizations for their own use. Private VPNs are typically used by businesses to allow remote employees to securely access company resources, or by individuals to protect their online privacy and security.

Not all VPNs are created equal, and there are risks associated with using public VPN services over private VPN as follows:

Risks of Using Private VPNs​

1. Trustworthiness of the VPN Provider

When using a private VPN, you are essentially entrusting your online security to the VPN provider. If the provider is untrustworthy or has a history of privacy breaches, your data could be compromised. Unfortunately, many private VPN providers have been caught logging user data or even selling it to third-party companies for profit.

2. Potential for Malware or Adware

Some private VPNs have been found to include malware or adware in their software. This can be particularly dangerous, as malware can be used to steal sensitive information, while adware can slow down your computer and make browsing the web more difficult.

3. Unreliable Security of your Data

Private VPNs may not always provide reliable security. As the service is managed by the third-party service, it is difficult to understand how their system is working behind the closed doors. There may be logging data which can be easily used to identify the user, which would straight away remove the idea of anonymity of use.

Benefits of Creating Your Own Personal VPN​

1. Complete Control Over Your Online Security

By creating your own personal VPN, you have complete control over your online security. You can choose the encryption protocol, server locations, and other security features, ensuring that your data is protected to the fullest extent possible.

2. No Third-Party Involvement

When using a private VPN provider, you are relying on a third-party to protect your online security. By creating your own personal VPN, you eliminate this risk entirely, as there are no third parties involved in your online security.

3. Cost-Effective

While some private VPN providers charge high monthly fees for their services, creating your own personal VPN can be a cost-effective solution. By using open-source software and free server software, you can create a VPN that is just as secure as a private VPN provider, but without worrying about your browsing history privacy or the excessive costs.

Setting up OpenVPN in AWS​

OpenVPN is an open-source project that can be used to create your custom VPN using their community edition and setting things up on your VPN server. Once the VPN server is set up, we use the Open-VPN client to connect to our VPN server and tunnel our traffic through the instance. For setting up the Open-VPN server, we are going to need the following things:

1. An AWS account
2. A little bit of curiosity..!

We are going to set up the VPN server in an AWS EC2 instance, which would be used to connect with our Open-VPN client on all our devices.The Open-VPN company also provides a purpose-built OpenVPN Access Server as an EC2 AMI which comes out of the box with AWS-friendly integration , which we are going to use in this blog.

Setup Open-VPN server in AWS:​

1. Once you have setup the AWS, login to your AWS account and search for EC2.

2. Once you are in the AWS EC2 console, switch to the region you want you VPN to be in and then click “Launch instances” button on the right side of the screen.

3. In the Ec2 creation console, search for AMI named “openvpn”. You will see a lot of AMI images. Based on the number of VPN connections you require, select the AMI. For the Current demonstration, I am choosing AMI which serves two VPN connection.

4. Choosing the above VPN, sets the Security group by itself. Ensure that the Ec2 is publicly accessible ( Either with EIP or setting Ec2 in public-subset). Once done press “Launch Instance”.

5. When we connect to the Ec2 instance, we are greeted with the OpenVPN server agreement. Create the settings as shown below and at the end, create an password.

6. Once done, open https://:943/admin’ where you would see an login page. Enter your user name and login that you have set in the VPN server, which in my case, username is openvpn and enter your previously set password.

7. You would enter the openVPN settings page. In configuration>Vpn settings, scroll to the bottom and toggle “Have clients use specific DNS servers” to ON. In the primary DNS enter 1.1.1.1 and in secondary dns enter 8.8.8.8. After this, click save changes on the bottom of the screen.

8. If you scroll to the top you will see a banner with “Update Running Server”, click on it.

9. You are set on the Open-VPN server side !

Connecting to Open-VPN server from our device:​

1. Once the server is configured, we would require client to connect to out openVPN server. For that purpose we need to install “Open-VPN connect”

• For Windows : Download and install the open-VPN connect from here
• For Mobile : Search for “openvpn connect” in the play-store (for Android) and in app-store(for apple)
• For Linux:

First ensure that your apt supports the HTTPS transport:

apt install apt-transport-https

Install the Open-VPN repository key used by the OpenVPN 3 Linux packages

curl -fsSL <https://swupdate.openvpn.net/repos/openvpn-repo-pkg-key.pub> | gpg --dearmor > /etc/apt/trusted.gpg.d/openvpn-repo-pkg-keyring.gpg

Then you need to install the proper repository. Replace $DISTRO with the release name depending on your Debian/Ubuntu distribution. For distro list, refer here

curl -fsSL <https://swupdate.openvpn.net/community/openvpn3/repos/openvpn3-$DISTRO.list> >/etc/apt/sources.list.d/openvpn3.list 
apt update 
apt install openvpn3

1. Once installed open “Open-VPN connect“ , we should see the something like below.

2. In the URL form, enter the IP of your EC2 instance and click NEXT. Accept the certificate pop-ups you would get during this process.

3. In the user name form, enter the username that you have set in the server and same for the password. Then click IMPORT.

4. Once imported, click on the radio button and enter your credentials again.

5. Once connected you should see the screen link this. Voila ! enjoy using your private VPN in EC2.

Liked my content ? Feel free to reach out to my LinkedIn for interesting content and productive discussions.

Often in software engineering, when we are developing new features, it is quite a common feature that we would encode certain sensitive information, such as passwords, secret keys, or tokens, for our code to do its intended functionality. Different professionals within the IT realm use it in different ways, such as the following:

• Developers use secrets from API tokens, Database credentials, or other sensitive information within the code.
• Dev-ops engineers might have to export certain values as environment variables and write the values in the YAML file for CI/CD pipeline to run efficiently.
• The cloud engineers might have to pass the credentials, secret tokens, and other secret information for them to access their respective cloud (In the case of AWS, even if we save these in a .credentials file, we still have to pass the filename in terraform block, which would indicate that the credentials are available locally within the computer.)
• The system administrators might have to send different logins and passwords to the people for the employees to access different services

But writing it in plain text or sharing it in plain text is quite a security problem, as anyone logging in to the code-base might access the secret or pull up a Man-in-the-Middle attack. To counter this, in the developing world, we have different options like Importing secrets from another file ( YAML, .py, etc.) or exporting them as an environment variable. But both of these still have a problem: a person having access to a singular config file or the machine can echo out the password ( read print). Given these problems, it would be very useful if we could deploy a single solution that would provide solutions to all the IT professionals mentioned above and more. This is the ideal place for introducing Vault.

HashiCorp Vault — an Introduction​

HashiCorp Vault is a secrets and encryption management system based on user identity. If we have to compare it with AWS, it is like an IAM user-based resource (read Vault here) management system which secures your sensitive information. This sensitive information can be API encryption keys, passwords, and certificates.

Its workflow can be visualized as follows:

Hosting Cost of Vault​

• Local hosting: This method is usually done if the secrets are to be accessed only by the local users or during the development phase. This method has to be shunned if these secret engines have to be shared with other people. As it is within the local development environment, there is no additional investment for deployment. This can be hosted directly in a local machine or by its official docker image
• Public Cloud Hosting ( EC2 in AWS/Virtual Machine in Azure): If the idea is to set up Vault to share with people across different regions, hosting it on Public cloud is a good idea. Although we can achieve the same with the on-prem servers, upfront costs and scalability is quite a hassle. In the case of AWS, we can easily secure the endpoint by hosting Vault in the EC2 instance and creating a Security group on which IPs can access the EC2. If you feel more adventurous, you can map this to a domain name and route from Route 53 so the vault is accessible on a domain as a service to the end users. In the case of EC2 hosting with an AWS-defined domain, the cost is $0.0116/hr.
• Vault cloud Hosting (HashiCorp Cloud Platform): If you don’t want to set up infrastructure in the Public Cloud Environments, there is an option of choosing the cloud hosted by vault. We can think of it as a SaaS-based cloud platform that enables us to use the Vault as a service on a subscription basis. Since hashicorp itself manages the cloud, we can expect a consistent user experience. For the cost, it has three production grade options: Starter at $ 0.50/hr, Standard at $1.58/hr, and Plus at $1.84/hr (as seen in July 2022).

Example of Self-Hosting in AWS Cloud​

Our goal in this Project is to create a Vault instance in EC2 and store static secrets in the Key—Value secrets engine. These secrets are later retrieved into the terraform script, which, when applied, would pull the secrets from the Vault Secrets Engine and use them to create infrastructure in AWS.

To create a ready-to-use Vault, we are going to follow these steps:

1. Create an EC2 Linux instance with ssh keys to access it.
2. SSH into the instance and install the Vault to get it up and running
3. Configure the Valve Secrets Manager

Step 1: Create an EC2 Linux instance with ssh keys to access it​

To create an EC2 instance and access it remotely via SSH, we need to create the Key pair. First, let's create an SSH key via the AWS console.

Once the Keys have been created and downloaded into the local workbench, we create an EC2 (t2.micro) Linux instance and associate it with the above keys. The size of the EC2 can be selected based on your requirements, but usually, a t2.micro is more than enough.

Step 2: SSH into the instance and install the secrets to get it up and running​

Once the status of the EC2 changes to running, open the directory in which you have saved the SSH(.pem) key. Open a terminal and type ssh -i <keyname.pem> ec2-user @<publicdns IP4> . Once we have established a successful SSH session into our Ec2 instance, we can install the Vault using the following commands:

wget -O- <https://apt.releases.hashicorp.com/gpg> | gpg — dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg  
  
echo "deb \[signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg\] <https://apt.releases.hashicorp.com> $(lsb\release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list  
  
sudo apt update && sudo apt install vault

The above command would install the vault in the EC2 environment. Sometimes the second command is known to throw some errors. In case of an error, replace $(lsb_release -cs) with “jammy”. [This entire process can be automated by copying the above commands to EC2 user data while creating an EC2 instance].

Step 3: Configure the Hashicorp valve​

Before initializing the vault, let's ensure it is properly installed by following the command:

vault

Let's make sure there is no environment variable called VAULT_TOKEN. To do this, use the following command:

$ unset VAULT_TOKEN

Once we have installed the Vault, we need to configure the Vault, which is done using HCL files. These HCL files contain data such as backed, listeners, cluster address, UI settings, etc. As we have discussed in the Vault’s Architecture, the back end on which the data is stored is very different from the vault engine, which is to be persisted even when the vault is locked (Stateful resource). In addition to that, we need to specify the following details:

• Listener Ports: the port/s on which the Vault listens for API requests.
• API address: Specifies the address to advertise to route client requests.
• Cluster address: Indicates the address and port to be used for communication between the Vault nodes in a cluster. To secure it much further, we can use TLS-based communication. This step is optional and can only be tried if you want to secure your environment further. The TLS Certificate can be generated using openssl in Linux.

# Installs openssl  
sudo apt install openssl  
  
#Generates TLS Certificate and Private Key  
openssl req -newkey rsa:4096 -x509 -sha512 -days 365 -nodes -out certificate.pem -keyout privatekey.pem 

Insert the TLS Certificate and Private Key file paths in their respective arguments in the listener “tcp” block.

• tls_cert_file: Specifies the path to the certificate for TLS in PEM encoded file format.
• tls_key_file: Specifies the path to the private key for the certificate in PEM-encoded file format.

#Configuration in config.hcl file  
  
storage "raft" {   
path = "./vault/data"   
node\id = "node1"   
}  
listener "tcp" {  
 address = "127.0.0.1:8200"   
tls\disable = "true"  
tls\cert\file = certificate.pem  
tls\key\file = privatekey.pem  
}  
disable\mlock = true  
api_addr = "http://127.0.0.1:8200"   
cluster_addr = "https://127.0.0.1:8201"  
 ui = true

Once these are created, we create the folder where our backend will rest: vault/data.

mkdir -p ./vault/data

Once done, we can start the vault server using the following command:

vault server -config=config.hcl

Once done, we can start our Vault instance with the backend mentioned in the config file and all its settings.

export VAULT_ADDR='http://127.0.0.1:8200'  
  
vault operator init

After it is initialized, it creates five Unseal keys called shamir keys (out of which three are used to unseal the Vault by default settings) and an Initial root token. This is the only time ever that all of this data is known by Vault, and these details are to be saved securely to unseal the vault. In reality, these shamir keys are to be distributed among key stakeholders in the project, and the Key threshold should be set in such a fashion that Vault can be unsealed when the majority are in consensus to do so.

Once we have created these Keys and the initial token, we need to unseal the vault:

vault operator unseal

Here we need to supply the threshold number of keys to unseal. Once we supply that, the sealed status changes to false.

Then we log in to the Vault using the Initial root token.

vault login

Once authenticated successfully, you can easily explore different encryption engines, such as Transit secrets Engine. This helps encrypt the data in transit, such as the Key-Value Store, which is used to securely store Key-value pairs such as passwords, credentials, etc.

As seen from the process, Vault is pretty robust in terms of encryption, and as long as the shamir keys and initial token are handled in a sensitive way, we can ensure the security and integrity

And you have a pretty secure Vault engine (protected by its own shamir keys) running on a free AWS EC2 instance (which is, in turn, guarded by the security groups)!

**Want to Connect?**  
  
If you want to connect with me, you can do so on [LinkedIn](https://www.linkedin.com/in/krishnadutt/).

As we embrace digital progress and shift to using containers for our applications, we’re dealing with a variety of core images that form the basis of our apps. These images vary based on our app’s needs, given the different ways we build them. Unfortunately, in companies with a moderate level of DEVOPS experience (maturity levels 2 to 3), the focus often leans heavily towards getting new features out quickly, sometimes overlooking security.

For us developers, the challenge is keeping up with the ever-changing world of software. This includes the extra pieces they need (dependencies), the building blocks (components), and the tools (libraries) that make them work together across our whole system. All these parts combined are what we call the Software Bill of Materials (SBOM). To tackle this, we need to understand the SBOM of each image before we use it as a foundation for our apps. But doing this manually is complex and time-consuming. Plus, every time we want to update to a newer image version, we have to go through the whole process again, slowing down our ability to deliver our products.

The solution? Automation. By using automation, we can navigate the intricate process of understanding the SBOM. It helps us make organized lists of all the parts, find any known problems, and suggest ways to fix them — all done with tools like Trivy. Before we dive into the detailed steps, let’s make sure we’re all on the same page about what the SBOM really means.

What is SBoM?​

SBOM stands for Software Bill of Materials. Think of it like a detailed list of all the parts that make up our software. This list includes things like tools from other people, building blocks, and even the rules that guide how they’re used, all with exact version details.

SBOM is important because it gives us a big picture of all the different parts that come together to create a specific software. This helps our DevSecOps teams find out if there are any possible risks, understand how they could affect us, and take steps to fix them. All of this makes our software strong and secure.

Why create SBOM?​

1. Finding and Managing Outside Parts: SBOM helps us see all the software we use from others. It shows us different versions and even points out any possible security issues. With this info, we can make smart choices about what we use, especially when it comes to libraries and tools from other sources.
2. Making Our Supply Chain Secure: SBOM acts like a detailed map for our software. This map helps us make sure everything is safe and guards against any tricks or attacks on our software supply chain. We can even use SBOM to check if the people we get our software from follow good security rules.

SBOM format​

• Software Package Data Exchange (SPDX): This open standard serves as a software bill of materials (SBOM), identifying and cataloging components, licenses, copyrights, security references, and other metadata related to software. While its primary purpose is to enhance license compliance, it also contributes to software supply-chain transparency and security improvement.
• Software Identification Tags (SWID): These tags contain descriptive details about a specific software release, including its product and version. They also specify the organizations and individuals involved in producing and distributing the product. These tags establish a product’s lifecycle, from installation on an endpoint to deletion.
• CycloneDX (CDX): The CycloneDX project establishes standards in XML, JSON, and protocol buffers. It offers an array of official and community-supported tools that either generate or work with this standard. While similar to SPDX, CycloneDX is a more lightweight alternative.

In addition to creating SBOM, trivy has the capability to SCAN the SBOM generated either by trivy or other tools, to identify the severity of the problem. In addition to the vulnerability detection, it also suggests the possible fixes for the identified vulnerability.

SBOM with TRIVY​

Trivy is a comprehensive security scanner, maintained and built by aqua security team. It is reliable, fast, and has several in-built scanners to scan for has different security issues, and different targets where it can find those issues. It can be used to scan following use cases:

• OS packages and software dependencies in use (SBOM)
• Known vulnerabilities (CVEs)
• IaC misconfigurations
• Sensitive information and secrets

Today we will be focusing on the SBOM scanning capabilities of the Trivy. In this tutorial, we would doing the following :

• First, Create the SBOM using trivy
• Analyze the created SBOM to scan and find the vulnerabilites

For the sake of the demo we will be using one of the most used docker images, NGINX, particularly nginx:1.21.5. We would using its Docker hub image and run scanner to generate the SBOM. Once the SBOM is generated, we would use this SBOM to get the list of Vulnerabilities and possible fixes for the same.

Generating SBOM​

To generate the SBOM, make sure the trivy is first installed in you workbench. If not you can install using commands in this link. In my case , my pc is debian based, so i installed using following command

sudo apt-get install wget apt-transport-https gnupg lsb-release  
wget -qO - <https://aquasecurity.github.io/trivy-repo/deb/public.key> | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null  
echo "deb \[signed-by=/usr/share/keyrings/trivy.gpg\] <https://aquasecurity.github.io/trivy-repo/deb> $(lsb\_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list  
sudo apt-get update  
sudo apt-get install trivy

If you are using any flavour of debian, you might face issue with “lsb_release -sc” in the command. To overcome the issue you can use one of the following values: wheezy, jessie, stretch, buster, trusty, xenial, bionic.

Once installed you should see the following, when you run

trivy --version

Once the trivy is installed, we can scan the vulnerabilities in two ways, either for image, for the whole directory containing the files or an single file.

• For a folder containing group of language specific files

trivy fs --format cyclonedx --output result.json /path/to/your\_project

• For a specific file

trivy fs --format cyclonedx --output result.json ./trivy-ci-test/Pipfile.lock

• For a container image

trivy image --format spdx-json --output result.json nginx:1.21.5

Once done, the scanner will scan the files/image and determine which language is the application written. Once determined, it will download the database pertaining to that specific language and get the list of libraries that are present in that language and check against which are being used in the current context.

These libraries are listed down along with OS level libraries in an SBOM in the format that we have requested in.

Listing Vulnerabilities of SBOM​

Once the SBOM are generated, we can create the list of known vulnerabilities of the dependencies in the image/files such as libraries, OS packages etc. In addition to identifying these, trivy also suggests the version in which these vulnerabilities are fixed, making it not an effective tool to identify the vulnerabilities but also to resolve them.

In order for us to generate this list, we use the following command

trivy sbom results.json

This will generate the following list

As you can observe, we get the name of the library, the CVE vulnerability Number, Its Severity (HIGH,MEDIUM,LOW), The status of Vulnerability fix( fixed, not fixed or will_not fix) , if its fixed then the fixed version and along with the details on the vulnerability.

Based on this information ,we can upgrade the fixed vulnerable libraries, understand the vulnerability level in not fixed libraries and remove them if they are not required. In-addition to that, we would have the opportunity to look into the alternatives to the vulnerable libraries.

For more information and documentation, visit this site.

Liked my content ? Feel free to reach out to my LinkedIn for interesting content and productive discussions.