Skip to content

Latest commit

 

History

History
executable file
·
195 lines (168 loc) · 7.13 KB

testing.md

File metadata and controls

executable file
·
195 lines (168 loc) · 7.13 KB

Recipe Tests

Recipe tests are integration tests, since StarThinker is mostly an integration platform. They are recipes with values hard coded and designed to be ran by anyone with access to the underlying resources. To help test recipes, StarThinker has a Test Task. It is like any other task in StarThinker except that it doesn't do any work. It simply compares and asserts endpoints.

StarThinker also comes with a test helper that loads and runs test recipes for complete integration testing. Tests are run in parallel and written to individual log files. The test helper uses StarThinker virtual environment and shell variables defined in starthinker_assets/development.sh or starthinker_assets/production.sh.

Architecture

When designing tests, put as small a burden on other developers as possible by hard coding as much as posible. For example the dataset in most tests is hard coded to just Test becuase it is not user specific. On the other hand an account id is almost always left as a field because each developer has unique access. Follow these practices:

  • Each test should inlude a script instead of re-coding it, this ensures a full stack test.
  • Inside the test file, hard code any fields that do not need to be customized by developers.
  • All remaining fields will be automatically inserted into the test configuration for the developer to provide.

StarThinker runs full integration tests, many tests require specific access to testing assets such as account identifiers and emails. To configure this, the helper will create or update test configuration files in the starthinker_assets/tests/ directory. After it is created edit the files in that directory and fill in required fields.

Not all fields need to be filled in, tests missing values will error, if you are not using those components, ignore the tests. Not all tests require fields, test not listed in the config file do not require user input.

Manual Testing

Steps

bash install/deploy.sh
  1. Option 1) Developer Menu
  2. Option 3) Test Tasks

Code

  • tests/ - The actual tests defined as workflow scripts.
  • starthinker/tool/test.py - Can be run from command line, finds, configures, and executes tests.
  • starthinker_assets/tests/[test].json - Generated by test tool when run, variables required for integration testing.
  • starthinker_assets/logs/[test].log - Generated by test tool when run, logs written during test execution.

Tips

  • For existing tests, keep a backup of your configuration file, its reusable.
  • Teams can share configuration files, keep any account login credentails seperate.
  • The best place to start a new recipe is the UI, prototype your workflow there.
  • Then download the JSON and create a test using the include block to connect to the script.
  • Test that the recipe runs with hard coded values as it did in the UI before adding fields.
  • When adding fields to the script, hard code the values in the test, do not use the config at this point.
  • Once the recipe is working, add optional test blocks to verify data.
  • Last, move the parameters from the test file to the config recipe by adding fields to the test file.
  • If the recipe is configured correctly, it will show up in the development UI.
  • Use the other tests as examples for how to structure yours.

Initializing the tests to create or update the configuration files:

source starthinker_assets/development.sh
python starthinker/tool/test.py --configure

Running all tests from the command line:

source starthinker_assets/development.sh
python starthinker/tool/test.py

Running ONLY some tests from the command line ( where tests are tests/scripts/[test].json):

source starthinker_assets/development.sh
python starthinker/tool/test.py --tests dt sheets

Viewing Test Logs

All tests run are logged to tests/logs/OK_[test].log and tests/logs/FAILED_[test].log. Running all tests clears the directory. Running a set of tests does NOT clear the directory but does overwrite the specific log files for each test run.

Adding Tests

A test is a script with additional test tasks, to create a test:

  1. Reference existing tests.
  2. Use instructions in step 1 and 2 for creating a script to create new tests.
  3. Modify the test tasks for any custom testing.

Test Task Samples

  1. generate an include block for any script
python starthinker/tool/test.py --include scripts/deal_finder.json
  1. inlcude another script in the test with parameters
{ "include": {
  "script": "scripts/deal_finder.json",
  "parameters":{
    "dataset": "Test",
    "recipe_name": "Deal_Finder_Test",
    "partners": {"field":{ "name":"partners", "kind":"integer_list", "order":1, "default":[], "description":"DBM partner id." }},
    "advertisers": {"field":{ "name":"advertisers", "kind":"integer_list", "order":2, "default":[], "description":"DBM advertiser id." }}
  }
}}
  1. check if sheet matches given values
{ "test": {
  "auth":"user",
  "sheets": {
    "url":"https://docs.google.com/spreadsheets/d/1h-Ic-DlCv-Ct8-k-VJnpo_BAkqsS70rNe0KKeXKJNx0/edit?usp=sharing",
    "tab":"Sheet_Clear",
    "range":"A1:C",
    "values":[
      ["Animal", "Age", "Weight ( lbs )"],
      ["dog", 7, 67],
      ["cat", 5, 1.5],
      ["bird", 12, 0.44],
      ["lizard", 22, 1],
      ["dinosaur", 1600, 273.97]
    ]
  }
}}
  1. check if bigquery table has given values or has data
{ "test": {
  "auth":"user",
  "bigquery":{
    "dataset":"Test",
    "table":"Sheet_To_BigQuery",
    "schema":[
      {"name": "Animal", "type": "STRING"},
      {"name": "Age", "type": "INTEGER"},
      {"name": "Weight_lbs", "type": "FLOAT"}
    ],
    "values":[
      ["dog", 7, 67],
      ["cat", 5, 1.5],
      ["bird", 12, 0.44],
      ["lizard", 22, 1],
      ["dinosaur", 1600, 273.97]
    ]
  }
}}
  1. assert recipe has executed all tasks before this point
{ "test": {
  "assert":"Completed all tasks."
}}
  1. check if path exists
{ "test": {
  "path":"somefile.txt"
}}
  1. check if storage file exists
{ "test":{
  "auth":"service",
  "storage":{
    "bucket":"bucket_name",
    "file":"file.png",
    "delete":true
   }
}}
  1. check if drive file exists
{ "test":{
  "auth":"service",
  "drive":{
    "file":"Some File Name Or URL",
    "delete":true
   }
}}

Notes

  • Ensure the test is re-entrant.
  • Ensure any test assets are shared with [email protected].
  • If test requires a sheet, use the template pattern to copy the master sheet to a user owned test sheet.
  • Only use fields for values that absolutely cannot be shared.
  • Keeping fields to a minimum is paramount.

© 2019 Google Inc. - Apache License, Version 2.0