Merge for 6.3.5 Release (#169)

* Fix jinja template bug under SA-Eventgen app

* Feature timeMultiple (#141)

* Adjusting interval with timeMultiple

* Update issue templates

add the bug report and feature request issue templates

* changing the stanzas to produce data

* Windbag generator/count + end=0 edge cases (#145)

* Generator handling for count = -1, end = 0 working properly

* Update docs (#146)

* Updated docs, added release notes to changelog

* Bump version

* add python code style lint and format

* [Docs] update contribute docs (#148)

* [Docs] update contribute docs

* [Docs] update the contribute

* Fix make docs bug and summary anchor link error

* init unittest

* [Build] add ut for timeparser functions

* Add more UT for config module

* Add more unit tests for config module

* Pep8 (#151)

* Format to code standards, addressed linter errors/warnings

* Update docs

* Post-PEP8 Fixes (#157)

* skip sort, gitignore 3rd party libs

* fixed yapf ignore, ran format across repo

* Issue 160 (#163)

* Fixed timer and token

* Added a conditional for end=-1

* Update eventgentimer.py

* Fixed timer and token (#162)

* add extendIndexes feature (#154)

* add extendIndexes feature

* set extendIndexes as a list value

* correct log level

* upate doc, change num to weight

* Test fix (#168)

* Add sample functional test for replay mode

* Add token replacement functional tests

* skip failed case

* Added a timeout

* created a results dir

* Update version.json

.circleci/config.yml

@@ -17,6 +17,7 @@ jobs:
           name: Run Tests
           command: |
             make test
+          no_output_timeout: 30m
       - store_test_results:
           path: /home/circleci/project/tests/test-reports
       - store_artifacts:

.github/ISSUE_TEMPLATE/bug-report.md

@@ -3,7 +3,7 @@ name: Bug report
 about: Provide the details of the bug to us in order to further triage and fix
 title: "[BUG]"
 labels: bug
-assignees: li-wu, GordonWang
+assignees: lephino, arctan5x, jmeixensperger, li-wu, GordonWang
 
 ---
 

.github/ISSUE_TEMPLATE/feature_request.md

@@ -3,7 +3,7 @@ name: Feature request
 about: Suggest an idea or improvement for this project
 title: "[FEATURE/IMPROVEMENT]"
 labels: enhancement
-assignees: li-wu, GordonWang
+assignees: lephino, arctan5x, jmeixensperger, li-wu, GordonWang
 
 ---
 

.gitignore

@@ -17,7 +17,6 @@ stage/
 local/
 eventgen_wsgi.conf
 *.log
-dist
 *.egg-info
 **/*.tgz
 .cache
@@ -27,3 +26,4 @@ _book
 *.result
 venv/*
 *.log.*
+splunk_eventgen-*/

.yapfignore

@@ -0,0 +1,2 @@
+splunk_eventgen/lib/concurrent
+splunk_eventgen/lib/requests_futures

CONTRIBUTING.md

@@ -1,99 +1,16 @@
 # Contributing
 
-When contributing to this repository, please first discuss the change you wish to make via issue,
-email, or any other method with the owners of this repository before making a change.
+When contributing to this repository, please read over the [contributing document](http://splunk.github.io/eventgen/CONTRIBUTE.html). [Create a issue](http://splunk.github.io/eventgen/FILE_ISSUES.html) to discuss your demand and follow the [guidelines](http://splunk.github.io/eventgen/CONTRIBUTE_CODE.html) to make the code change
 
-Please note we have a code of conduct, please follow it in all your interactions with the project.
+Please note we have a [code of conduct](http://splunk.github.io/eventgen/CONTRIBUTE.html#code-of-conduct), please follow it in all your interactions with the project.
 
-## Pull Request Process
-
-1. Ensure any install or build dependencies are removed before the end of the layer when doing a
-   build.
-2. Update the splunk_eventgen/README/eventgen.conf.spec with details of changes to the interface, this includes new environment
-   variables, exposed ports, useful file locations and container parameters. Also, update the necessary documentation.
-3. Increase the version numbers in splunk_eventgen/version.json. The versioning scheme we use is [SemVer](http://semver.org/).
-4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you
-   do not have permission to do that, you may request the second reviewer to merge it for you.
-
-## Code of Conduct
-
-### Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, gender identity and expression, level of experience,
-nationality, personal appearance, race, religion, or sexual identity and
-orientation.
-
-### Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
-advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
-  address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-### Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-### Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-### Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at [email protected]. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-### Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at [http://contributor-covenant.org/version/1/4][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/
 
 ### Past / Active(marked as *) Contributors
 bbingham*
 arctan5x*
 jmeixensperger*
+li-wu*
+GordonWang*
 coccyx
 Jaykul
 allanwsplk

Makefile

@@ -13,7 +13,7 @@ XLARGE ?= 'tests/xlarge'
 NEWLY_ADDED_PY_FILES = $(shell git ls-files -o --exclude-standard | grep -E '\.py$$')
 CHANGED_ADDED_PY_FILES = $(shell git ls-files -mo --exclude-standard | grep -E '\.py$$')
 
-.PHONY: tests, lint, format
+.PHONY: tests, lint, format, docs
 
 all: egg
 
@@ -24,7 +24,7 @@ image: setup_eventgen egg
 	rm splunk_eventgen/default/eventgen_engine.conf || true
 	docker build -f dockerfiles/Dockerfile . -t eventgen
 
-test: egg image test_helper test_collection_cleanup
+test: egg image test_helper run_tests test_collection_cleanup
 
 test_helper:
 	docker run -d -t --net=host -v /var/run/docker.sock:/var/run/docker.sock --name ${EVENTGEN_TEST_IMAGE} eventgen:latest cat
@@ -41,15 +41,10 @@ test_helper:
 	@echo 'Installing test requirements'
 	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "pip install -r $(shell pwd)/tests/requirements.txt" || true
 
+run_tests:
 	@echo 'Running the super awesome tests'
 	docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "cd $(shell pwd); python tests/run_tests.py ${SMALL} ${MEDIUM} ${LARGE} ${XLARGE}" || true
 
-	echo 'Collecting results'
-	#TODO: Should be paramaterized or generalized so that we don't need to add this here
-	docker cp ${EVENTGEN_TEST_IMAGE}:$(shell pwd)/tests_results.xml tests_results.xml || echo "no tests_results.xml" || true
-
-	docker stop ${EVENTGEN_TEST_IMAGE} || true
-
 test_collection_cleanup:
 	@echo 'Collecting results'
 	#TODO: Should be paramaterized or generalized so that we don't need to add this here
@@ -97,9 +92,7 @@ run_controller: eg_network
 	docker run --name eg_controller --network eg_network -d -p 5672:5672 -p 15672:15672 -p 9500:9500 eventgen:latest controller
 
 docs:
-	npm install -g gitbook-serve
-	cd docs/
-	gitbookserve
+	cd docs/; bundle install; bundle exec jekyll serve
 
 build_spl: clean
 	python -m splunk_eventgen build --destination ./
@@ -124,3 +117,9 @@ else
 	@yapf -i $(NEWLY_ADDED_PY_FILES)
 endif
 
+lint-all:
+	@flake8 .
+
+format-all:
+	@isort -rc .
+	@yapf -r -i .

README.md

@@ -33,4 +33,4 @@ The Splunk Event Generator is licensed under the Apache License 2.0. Details can
 ### Support
 
 This software is released as-is. Splunk provides no warranty and no support on this software.
-If you have any issues with the software, please file an issue (https://github.com/splunk/eventgen/issues/new)
+If you have any issues with the software, please read over the [guidelines](http://splunk.github.io/eventgen/FILE_ISSUES.md) and file an issue.

docs/.gitignore

@@ -0,0 +1 @@
+_site

docs/ARCHITECTURE.md

@@ -19,7 +19,7 @@ These requirements have led to this new architecture in verison 4.  Eventgen is
     * Handles flattening of configs from global defaults and individual config files
 * Loading Plugins
     * Load plugins from the `lib/plugins` directory or App's `bin` directories, with three types of Plugins: Rating, Generator and Output
-    * For more detail on writing plugins, see the [Plugins documentation](Plugins.md)
+    * For more detail on writing plugins, see the [Plugins documentation](PLUGINS.md)
 * Creating timers for each sample
     * For samples with queueable generators (meaning we can have multiple threads generating sample simultaneously), the timer wakes up every interval and puts a task in the queue to be generated by a generator worker
     * For samples which are not queueable (meaning we can only run one simultaneous thread of the generator), the thread runs the generator plugin single threaded and handles sleeping the proper amount to manage intervals.
@@ -38,13 +38,13 @@ Depending on tunable parameters, these can be threads or processes and can eithe
 
 # Scaling
 
-This queue architecture allows significant advantages by distributing processing. At each timer execution a generation job gets put in the queue. This queue by default is a Python Queue (either in queue.Queue or multiprocessing.Queue). For details, see the [Performance guide](Performance.md)
+This queue architecture allows significant advantages by distributing processing. At each timer execution a generation job gets put in the queue. This queue by default is a Python Queue (either in queue.Queue or multiprocessing.Queue). For details, see the [Performance guide](PERFORMANCE.md)
 
 # Testing
 
 Given the complexity and the reimplementation of a number of features during refactors, we've also built out a number of test examples.  We currently don't have test scripts built with assertions etc, but there are a series of configs and samples under the `tests/` directory which should demonstrate and allow testing of basic functionality. Check out the ``tests/sample_eventgen_conf`` directory in the project for commonly-used plugins and configurations. You can execute a test config by doing:
 
-    python bin/eventgen.py generate tests/<dir>/<test>.conf
+    python -m splunk_eventgen generate tests/<dir>/<test>.conf
 
 # Server-Controller Architecture