Connectors → Splunk

About Splunk

Splunk is a software product that captures, indexes, and correlates real-time, machine-generated data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. Currently, the Splunk connector extracts data represented as Splunk reports.

Splunk Connector

The Incorta Splunk connector uses the Splunk Software Development Kit (SDK) for Java, which is built as a layer on top of the Splunk REST API. Version 1.0 of the connector supports Splunk reports. The Splunk connector creates a search job to retrieve the list of reports created in the system. When a report is selected during schema design, the Splunk Connector creates another search job to retrieve the fields of that report. Splunk retrieves the fields by discovering them from the last loaded job of the report, a mechanism that works for both scheduled and unscheduled reports.

The Splunk connector supports the following Incorta-specific functionality:

Feature	Supported
Chunking
Data Agent
Encryption at Ingest
Incremental Load	✔
Multi-Source	✔
OAuth
Performance Optimized	✔
Remote
Single-Source	✔
Spark Extraction
Webhook Callbacks	✔

Deployment Steps

The Splunk connector is an external connector. You deploy an external connector as a JAR file to each Incorta Node in an Incorta cluster as well as to Cluster Management Console (CMC) host. A System Administrator with root access to the operating systems for each host in the Incorta cluster, including the CMC, will need to deploy the external JAR file for the Splunk Connector. A CMC Administrator will need to restart the Analytics and Loader Services in the cluster. A Systems Administrator will need to restart the CMC.

Deployment to an Incorta Node

Here are the steps to deploy the incorta.connector.splunk.jar file to the extensions directory of an Incorta Node that is running the Analytics and/or Loader Services in an Incorta cluster.

Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.
As the root user for the hosts running Incorta Nodes, use Secure Copy for shell or similar to copy the incorta.connector.splunk.jar to the /tmp directory of the hosts.

PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
INCORTA_NODE_HOST_IPv4_LIST='1.1.1.1  2.2.2.2  3.3.3.3  4.4.4.4'
PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
HOST_ROOT_USER='ec2-user'
for i in ${INCORTA_NODE_HOST_IPv4_LIST}
do
 echo $i
 scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${i}:/tmp/${PATH_JAR_FILE}
 wait
done

Secure shell in to each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

sudo su incorta
sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar

For each Incorta Node, as the incorta user, create the splunk directory in the /extensions/connectors/ folder.

INCORTA_NODE_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/IncortaNode'
mkdir ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk

For each Incorta Node, as the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

mv /tmp/incorta.connector.splunk.jar ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk

Restart the Analytics and Loader Services

Here are the steps to restart the Analytics and Loader Services in an Incorta Cluster from the Cluster Management Console (CMC).

As the CMC Administrator, sign in to the CMC.
In the Navigation bar, select Clusters.
In the cluster list, select a Cluster name.
Select the Details tab, if not already selected.
In the footer, select Restart.

Deployment to the Cluster Management Console

Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.
Using Secure Copy for Shell, copy the incorta.connector.splunk.jar to the /tmp directory of the host running the CMC.

PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
CMC_HOST_IPv4='5.5.5.5'
PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
HOST_ROOT_USER='ec2-user'
scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${CMC_HOST_IPv4}:/tmp/${PATH_JAR_FILE}

Secure shell into each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

sudo su incorta
sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar

As the incorta user, create the splunk directory in the /extensions/connectors/ folder.

CMC_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/cmc'
mkdir ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk

As the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

mv /tmp/incorta.connector.splunk.jar ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk

As the incorta user, stop the CMC

cd ${CMC_INSTALLATION_PATH}
./stop-cmc.sh

As the incorta user, start the CMC

cd ${CMC_INSTALLATION_PATH}
./start-cmc.sh

Connect Splunk and Incorta

To connect Splunk and Incorta, here are the high level steps, tools, and procedures:

Create an external data source
Create a schema with the Schema Wizard
or, Create a schema with the Schema Designer
Load the schema
Explore the schema

Create an external data source

Here are the steps to create a external data source with the Splunk connector:

Sign in to the Incorta Direct Data Platform.
In the Navigation bar, select Data.
In the Action bar, select + New → Add Data Source.
In the Choose a Data Source dialog, in Application, select Splunk.
In the New Data Source dialog, specify the applicable connector properties.
To test, select Test Connection.
Select Ok to save your changes.

Splunk connector properties

Here are the properties for the Splunk connector:

Property	Control	Description
Data Source Name	text box	Enter the name of the data source
Authentication Method	drop down list	Options are: ● Using Splunk Username and Password ● Using AppleConnect
Username	text box	Splunk Username and Password authentication only
Password	text box	Splunk Username and Password authentication only
IdMS Account Name	text box	Splunk AppleConnect authentication only
IdMS Account Password	text box	Splunk AppleConnect authentication only
IdMS AppID Key	text box	Splunk AppleConnect authentication only
TOTP Secret Code	text box	Splunk AppleConnect authentication only
Hostname	text box	Splunk hostname
Port	text box	Splunk port

Create a schema with the Schema Wizard

Here are the steps to create a Splunk schema with the Schema Wizard:

Sign in to the Incorta Direct Data Platform.
In the Navigation bar, select Schema.
In the Action bar, select + New → Schema Wizard
In (1) Choose a Source, specify the following:
- For Enter a name, enter the schema name.
- For Select a Datasource, select the Splunk external data source.
- Optionally create a description.
In the Schema Wizard footer, select Next.
In (2) Manage Tables, in the Data Panel, first select the name of the Data Source, and then check the Select All checkbox.
In the Schema Wizard footer, select Next.
In (3) Finalize, in the Schema Wizard footer, select Create Schema.

Create a schema with the Schema Designer

Here are the steps to create a Splunk schema using the Schema Designer:

Sign in to the Incorta Direct Data Platform.
In the Navigation bar, select Schema.
In the Action bar, select + New → Create Schema.
In Name, specify the schema name, and select Save.
In Start adding tables to your schema, select Splunk.
In the Data Source dialog, specify the Splunk table data source properties.
Select Add.
In the Table Editor, in the Table Summary section, enter the table name.
To save your changes, select Done in the Action Bar.

Splunk table data source properties

For a schema table in Incorta, you can define the following Splunk specific data source properties as follows:

Property	Control	Description
Type	drop down list	Default is Splunk
Data Source	drop down list	Select the Splunk external data source
Report Entry Method	drop down list	Select an option for specifying the report to create the schema table from: ● Fully qualified name ● Select from list
Report's Fully Qualified Name	text box	This property appears when the value of Report Entry Method is Fully qualified name. Enter the full name of the report.
Report	drop down list	This property appears when the value of Report Entry Method is Select from list. Select an available report from the list.
Start Date	drop down list	Select the time window of the report
Full Load Start Date	text box	This property appears when the value of Start Date is Custom Date. Enter the custom date in yyyy-mm-dd format.
Page Size (in rows)	text box	Enter the number of records in a page for the REST API request
Callback	toggle	Enables the Callback URL field
Callback URL	text box	This property appears when the Callback toggle is enabled. Specify the URL.

Start date options

The start date options apply to unscheduled reports only:

Report’s Default Start Time: This option will use the default time window of the report.
All Time: This option will run the report to retrieve all available data without restricting the time window.
Custom Date: This option allows the user to enter a custom date to get the data from that date.

For scheduled reports, data is extracted from the last load job. In other words, incremental and full loading is supported for unscheduled reports, and full loading only is supported for scheduled reports.

View the schema diagram with the Schema Diagram Viewer

Here are the steps to view the schema diagram using the Schema Diagram Viewer:

Sign in to the Incorta Direct Data Platform.
In the Navigation bar, select Schema.
In the list of schemas, select the Splunk schema.
In the Schema Designer, in the Action bar, select Diagram.

Load the schema

Here are the steps to perform a Full Load of the Splunk schema using the Schema Designer:

Sign in to the Incorta Direct Data Platform.
In the Navigation bar, select Schema.
In the list of schemas, select the Splunk schema.
In the Schema Designer, in the Action bar, select Load → Full Load.
To review the load status, in Last Load Status, select the date.

Explore the schema

With the full load of the Splunk schema complete, you can use the Analyzer to explore the schema, create your first insight, and save the insight to a new dashboard.

To open the Analyzer from the schema, follow these steps: