How to Use Python to Automate Data Enrichment

The amount of context that cybersecurity analysts and engineers require for assessing security alerts is overwhelming. But here’s the good news: Data enrichment can help. Practitioners often perform data enrichment to determine the severity and criticality of a security alert.

Data enrichment is the process of combining disparate data sources together to create a more meaningful story of existing data. For example, retrieving the hostname from a security alert and searching for the hostname in your CMDB, SIEM, and threat intelligence platform is a common data enrichment use case. 

The challenge that teams often face is proactively presenting enriched data for team members while they’re assessing security alerts. In some situations, technology solutions are missing integrations or are in isolated segments of the network, which makes data enrichment tedious and manual. 

It’s essential for teams to employ automation on tasks such as data enrichment to prevent burnout amongst team members and to translate team tradecraft into logic. Python is a great resource for security teams to leverage for automating data enrichment. 

This post is a step-by-step walkthrough of implementing a data enrichment use case using Python. Follow along by starting a free trial with Axonius.

Define Data Enrichment Use Case

Before writing code, it’s important to ensure that your team has defined the use case that your Python script will help solve. The example use case in this post is intended to assist team members with data enrichment by providing asset intelligence for security alerts.

Identify Data Sources

The primary goal when performing data enrichment is combining disparate data sources.  While selecting all the useful data sources is typically the end goal, it’s best to start with a few data sources and add additional sources as needed. While identifying data sources there are a few questions to consider answering:

1. Does my data source have an API or a mechanism to collect data?
2. Does my development environment have access to the data source?
3. Are there any requirements to keep in mind when handling the data collected?
4. Which attributes and fields are required to successfully implement my use case?

A goal for our use case is to provide asset intelligence from Axonius when a new JIRA issue is created with “Security Alert” as the issue type. For this use case, two data sources are required.

Define Security and Programming Logic

Translating your team’s tradecraft into documentation and automated steps is critical for maturing your security program. The more detailed your security logic, the easier it will be to translate it into Python code. This can be initially done from a high level as documented steps. 

Security Logic:

1. Retrieve new Security Alert type issues
2. Enrich hostname field with Asset Intelligence:
   > Hostname(s)
   > Date Last Seen(es)
   > Network Interface Addresses
3. Update case Additional Context field with Asset Intelligence

After your security logic is defined, it’s time to review API documentation to get an understanding of how to access data from the API. The field hostname is a custom field for the Security Alert issue type in JIRA. 

The API documentation mentions that an additional API call is required to convert the hostname custom field into JIRA’s custom field ID. Depending on your data sources, you may have the option to leverage a Python library to interact with a data source. For example, to leverage Axonius as a data source the Python API Client can be installed to facilitate API calls between the development environment and the Axonius platform.

Programming Logic:

1. Retrieve metadata for JIRA project
2. Retrieve new Security Alert type issues
3. Search for hostname in Axonius
4. Format response from Axonius 
5. Update JIRA issue field, Additional Context field with formatted response

Setting Up Development Environment

A development environment in Python is a combination of a text editor and a Python runtime implementation. To set up your Python environment, ensure that you have Python 3 installed and run the following shell commands:

When writing Python code it’s critical that you avoid saving secrets in your code. It’s best to store your secrets in environment variables if you’re not able to pull secrets directly from a secrets vault. Environment variables can also be used to store other static details like the URL of the JIRA instance.

Write Code

Writing Python code is a continuous cycle of design, implementation, and testing until the intended outcome is consistently achieved. When building Python applications, it’s important to consider the readability of your source code. After reviewing API documentation, outlining your programming logic, and setting up your development environment, you’re ready to begin taking the steps defined in your programming logic and converting them to Python functions.

2. Retrieve metadata for JIRA project

3. Retrieve new Security Alert type issues

4. Search for hostname in Axonius

5. Format response from Axonius

6. Update JIRA issue field, Additional Context field with formatted response

7. Combining Functions