Notes and tips for using the remote development environment (via SSH) with VS Code.
To develop and app for Splunk SOAR, you need an instance of Splunk SOAR (Community Edition or licensed) and your target asset, CSNA/StealthWatch. Splunk SOAR can be installed in AWS EC2 from the marketplace on in a lab environment.
Installing Splunk SOAR Community Edition from AWS Marketplace.
Select from AWS Marketplace version 5.1.0 and use a t2.2xlarge instnace with 8 cpu 32 Gig, 768G disk.
After starting the EC2 instance, login via SSH using the public key assigned to the instance.
ssh [email protected] -i customer_training.pemLog in to the Splunk SOAR (On-premises) web interface: https://docs.splunk.com/Documentation/SOARonprem/5.1.0/Install/Login
Initial install credentials are 'admin' and complete instance ID e.g. i-00a6c83b6fa826feb as the password. The password for the self-hosted version is password You should change the password following the initial login.
Follow the instruction to set up the development environment and create the user 'phantom', you can logoff and return by using:
ssh [email protected] -i customer_training.pem
su phantom
<enter the same password as the admin login>
cd /home/phantomAlternately, you can go directly to the development user by entering:
ssh [email protected] -i customer_training.pem and you will be put in the home directory of the development user: /home/phantom.
The userid phantom has permissions to view the apps and log directories. The installed app(s) directory is at: ls /opt/phantom/apps/*. The log files are in /var/log/phantom. If you choose to install an app via the GUI, and the install fails, the log file /var/log/phantom/app_install.log provides additional information.
Use VSCode Remote SSH development environment to connect to the AWS EC2 instance (or a lab instance)
Instructions for configuring VS Code for a remote compute environment (Cloud compute, VM, etc.) are available at: https://gitlab.com/joelwking/vs_code_remote_compute - along with a Meetup session recording and slides available at: https://www.wwt.com/community/programmability-and-automation-meetup-group
A sample SSH configuration is a follows:
Host Splunk_SOAR
User phantom
HostName 54.237.22.196
IdentityFile ~/.ssh/customer_training.pem
Note: IdentifyFile is optional, if not specified, you will be prompted for the password.
Use the VSCode Remote Explorer SSH Target and connect to the remote Splunk SOAR instance. Open a terminal window, the user phantom has access to git and python3.6.
Review and reference these URL for documenation on the API and app development.
First open the remote SSH to the Splunk SOAR instance in VSCode. Using the terminal prompt, create a directory in your home directory apps
mkdir apps
Set your username and email address in the git configuration. Substitute your name and email address for the user.name and user.email.
git config --global user.name "Joel Example"
git config --global user.email [email protected]
git config --global color.ui true
git config --global core.editor vim # Note esc + :q or esc + :q! to exit vim
Generate an SSH key and install it on your GitHub account.
ssh-keygen -t rsa -b 2048 -C "[email protected]"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/phantom/.ssh/id_rsa): /home/phantom/.ssh/id_github
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/phantom/.ssh/id_github.
Your public key has been saved in /home/phantom/.ssh/id_github.pub.
View the key and copy and paste it into your account, https://github.com/settings/keys
cat /home/phantom/.ssh/id_github.pub
Assuming the repository is available (this repo may be moved to the WWT organization in the future) at https://github.com/joelwking/csna.git
Clone the repository using the SSH key generated previously.
GIT_SSH_COMMAND='ssh -i /home/phantom/.ssh/id_github' git clone [email protected]:joelwking/csna.gitEnter the cloned directory.
cd csna/phCiscoSecureNetworkAnalyticsDevelop your app, and use git add / git commit to manage your changes. When you wish to push changes to the remote repository, git push using this command format.
GIT_SSH_COMMAND='ssh -i /home/phantom/.ssh/id_github' git push origin main
Note: If you wish to maintain a local copy your your laptop, don't forget to
git pullfrom the remote repository.
The program phenv is used to issue a number of subcommands to manage the installation. It is also used to execute Python apps using the correct python interpreter.
For help, phenv --help will show the commands.
[phantom@appdev ~]$ phenv
Usage: /opt/phantom/bin/phenv command [args]
phenv will set the PATH and PYTHONPATH as neccessary to invoke python/pip for Splunk Phantom
Example: /opt/phantom/bin/phenv python /opt/phantom/bin/ibackup.pyc --setup
You can also use the "phenv <command>" to administrate certain aspects of Splunk Phantom.
```shell
/opt/phantom/bin/phenv set_preference --help
Run "/opt/phantom/bin/phenv help " to get a list of available commandsYou can test your program without installing the application or using the web GUI by using the CLI. You need to create a test.json file to provide the configuration and parameters. Review the file test_jsons/readme.md for examples on how to create this input file.
cd csna/phCiscoSecureNetworkAnalytics
phenv python ./ciscosecurenetworkanalytics_connector.py -u admin -p adminpswd test_jsons/test.jsonYou will need to use phenv to determine the Python interpreter and PYTHONPATH to configure your .vscode/launch.json for debugging.
Rather than look at the source code of phenv, just run python from it, and check what path and executable is being used.
[phantom@appdev csna]$ phenv python
Python 3.6.13+ (heads/3.6-dirty:a64de63, Mar 26 2021, 15:25:25)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getenv('PYTHONPATH')
'/opt/phantom/www:/opt/phantom/lib3:/opt/phantom/pycommon3:/opt/phantom/www:/opt/phantom/lib:/opt/phantom/pycommon:/opt/phantom/www:/opt/phantom/lib:/opt/phantom/pycommon:/opt/phantom/www:/opt/phantom/lib:/opt/phantom/pycommon:/opt/phantom/www:/opt/phantom/lib:/opt/phantom/pycommon:'
>>> To get the absolute path of the Python interpreter:
>>> import sys
>>> sys.executable
'/opt/phantom/usr/bin/python'Note: VSCode when running in debugging mode, will looks in your home directory
/home/phantomfor.vscode/launch.json, so you need to copy fromcsna/.vscode/launch.jsonto your home directory before running the debugger!
Note: VSCode has some issues with the
main()function of the program, as it attempts to connect to thelocalhostI added modifications in commit to circumvent these issues, at least for now.
To install the app, you need to create a tar file. Refer to the app installation in this link https://docs.splunk.com/Documentation/SOAR/current/DevelopApps/Connector
cd ~/apps/csna
tar -zcvf phCiscoSecureNetworkAnalytics.tgz phCiscoSecureNetworkAnalytics
Use --exclude-from 'phCiscoSecureNetworkAnalytics/exclude_files.txt' to exclude files from the tarball.
tar --exclude-from phCiscoSecureNetworkAnalytics/exclude_files.txt -zcvf phCiscoSecureNetworkAnalytics.tgz phCiscoSecureNetworkAnalyticsThe tar file can be downloaded from the remote instance to your laptop (to your Downloads directory perhaps) and import it into your Splunk SOAR web UI.
Note: if the install fails in the GUI or the CLI, the log file
/var/log/phantom/app_install.logcan be used to determine the cause of the failure.
Rather than simply creating a tar file, you can compile your app and install it locally. Compiling also lints the code, and checks the format of the JSON file. Refer to https://docs.splunk.com/Documentation/SOAR/current/DevelopApps/Tutorial#Compiling_and_installing_the_app for more information.
If you are developing your app and wish to install (publish) on the development instance, issue:
cd csna/phCiscoSecureNetworkAnalytics
phenv python /opt/phantom/bin/compile_app.pyc -i -g /home/phantom/app/csna/phCiscoSecureNetworkAnalytics/exclude_files.txtNote: specify the absolute path for the
-g IGNORE_FILE.
The output of a successful install will appear as follows:
[phantom@appdev phCiscoSecureNetworkAnalytics]$ phenv python /opt/phantom/bin/compile_app.pyc -i -g /home/phantom/app/csna/phCiscoSecureNetworkAnalytics/exclude_files.txt
cd'ing into ./
Will be ignoring: .git*, .vscode*, phCiscoSecureNetworkAnalytics/__pycache__, phCiscoSecureNetworkAnalytics/test_jsons/*
Validating App Json
App json found at ./ciscosecurenetworkanalytics.json
Validating App Json
Validating actions
test connectivity
No further validation coded for "test connectivity" action
retrieve flows
Done
Compiling: ./__init__.py
Compiling: ./ciscosecurenetworkanalytics_consts.py
Compiling: ./ciscosecurenetworkanalytics_connector.py
Installing app...
Creating tarball...
..//home/phantom/app/csna/phCiscoSecureNetworkAnalytics.tgz
Calling installer...
Success
DoneThe compiler will create the tar file and leave it in your app home directory (e.g. csna) enabling you to include it in your remote GIT repository. Users can then download the tar file to their Splunk SOAR instance for use.
Note: Verify your files have been ignored! If they have not, use the
tarcommand to manually create the tar file before uploading to your Git remote!
To populate Splunk SOAR with test events to demonstrate your app, I wrote a Python class which uses the Splunk SOAR API to generate containers and artifacts.
You can follow these instructions to create an event from the user phantom and interactively from the Python interpreter, or you can use the code and instructions under the test directory to run a Docker container and define the events with a YAML file. refer to the test/readme.md.
The code is available at: https://github.com/joelwking/Phantom-Cyber/tree/master/REST_ingest
From the Splunk SOAR user interface (UI) naviate to Administration -> User Management -> Users and edit the user automation. You need to show the token or set a new token.
{
"ph-auth-token": "*********",
"server": "https://54.237.22.123"
}Change the Allowed IPs to any or the IP address from where you are initiating the program.
From the developent account CLI, create two environment variables with your token and Splunk SOAR instance IP address.
export PH_AUTH_TOKEN=2adnymvMJMredactedHOM+xBGUNs1wEk=
export PH_SERVER=54.237.22.123Create a directory and download the Python class. You can do this on your development account on your local laptop.
mkdir artifact
cd artifact
curl https://raw.githubusercontent.com/joelwking/Phantom-Cyber/master/REST_ingest/PhantomIngest.py -o PhantomIngest.pyDownload and run this program to create events (containers and artifacts) in Splunk SOAR so the app can be run against an artifact.
phenv pythonYou can cut-n-paste the following into your Python interpreter or write your own program to generate events with data specific to your use case. In this example, the IP address of 192.0.2.1 will be used as the malicious_ip for the app, so select an IP address that is represented in your Stealthwatch data.
import PhantomIngest as ingest
import os
p = ingest.PhantomIngest(os.getenv('PH_SERVER'), os.getenv('PH_AUTH_TOKEN'))
kontainer = {'name': 'Voltaire', 'description': 'French Enlightenment writer, historian, and philosopher.', 'label': 'events'}
container_id = p.add_container(**kontainer)
art_i_fact = {"name": "François-Marie Arouet", "source_data_identifier": "IR_3458575"}
cef = {'sourceAddress': '192.0.2.1', 'sourcePort': '6553', 'sourceUserId': '[email protected]'}
meta_data = {}
artifact_id = p.add_artifact(container_id, cef, meta_data, **art_i_fact)
print(p.status_code, container_id, artifact_id)Provided you have a status code of 200, navigate to the Splunk SOAR UI and a new event should appear on the home screen.
Joel W. King @joelwking