Skip to content

Commit

Permalink
fix regressions tests
Browse files Browse the repository at this point in the history
  • Loading branch information
samansmink committed Sep 4, 2024
1 parent 51153a2 commit 2209eac
Show file tree
Hide file tree
Showing 4 changed files with 27 additions and 26 deletions.
44 changes: 22 additions & 22 deletions .github/regression/tpch_sf1_local.csv
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
benchmark/tpch/sf1/delta/q01.benchmark
benchmark/tpch/sf1/delta/q02.benchmark
benchmark/tpch/sf1/delta/q03.benchmark
benchmark/tpch/sf1/delta/q04.benchmark
benchmark/tpch/sf1/delta/q05.benchmark
benchmark/tpch/sf1/delta/q06.benchmark
benchmark/tpch/sf1/delta/q07.benchmark
benchmark/tpch/sf1/delta/q08.benchmark
benchmark/tpch/sf1/delta/q09.benchmark
benchmark/tpch/sf1/delta/q10.benchmark
benchmark/tpch/sf1/delta/q11.benchmark
benchmark/tpch/sf1/delta/q12.benchmark
benchmark/tpch/sf1/delta/q13.benchmark
benchmark/tpch/sf1/delta/q14.benchmark
benchmark/tpch/sf1/delta/q15.benchmark
benchmark/tpch/sf1/delta/q16.benchmark
benchmark/tpch/sf1/delta/q17.benchmark
benchmark/tpch/sf1/delta/q18.benchmark
benchmark/tpch/sf1/delta/q19.benchmark
benchmark/tpch/sf1/delta/q20.benchmark
benchmark/tpch/sf1/delta/q21.benchmark
benchmark/tpch/sf1/delta/q22.benchmark
benchmark/tpch/sf1/local/delta/q01.benchmark
benchmark/tpch/sf1/local/delta/q02.benchmark
benchmark/tpch/sf1/local/delta/q03.benchmark
benchmark/tpch/sf1/local/delta/q04.benchmark
benchmark/tpch/sf1/local/delta/q05.benchmark
benchmark/tpch/sf1/local/delta/q06.benchmark
benchmark/tpch/sf1/local/delta/q07.benchmark
benchmark/tpch/sf1/local/delta/q08.benchmark
benchmark/tpch/sf1/local/delta/q09.benchmark
benchmark/tpch/sf1/local/delta/q10.benchmark
benchmark/tpch/sf1/local/delta/q11.benchmark
benchmark/tpch/sf1/local/delta/q12.benchmark
benchmark/tpch/sf1/local/delta/q13.benchmark
benchmark/tpch/sf1/local/delta/q14.benchmark
benchmark/tpch/sf1/local/delta/q15.benchmark
benchmark/tpch/sf1/local/delta/q16.benchmark
benchmark/tpch/sf1/local/delta/q17.benchmark
benchmark/tpch/sf1/local/delta/q18.benchmark
benchmark/tpch/sf1/local/delta/q19.benchmark
benchmark/tpch/sf1/local/delta/q20.benchmark
benchmark/tpch/sf1/local/delta/q21.benchmark
benchmark/tpch/sf1/local/delta/q22.benchmark
4 changes: 2 additions & 2 deletions .github/workflows/LocalTesting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -281,13 +281,13 @@ jobs:
if: always()
shell: bash
run: |
python3 duckdb/scripts/regression_test_runner.py --old=duckdb_delta/build/release/benchmark/benchmark_runner --new=build/release/benchmark/benchmark_runner --benchmarks=.github/regression/tpch_sf1_local.csv --verbose --threads=2 --root-dir=.
python3 ./scripts/regression_test_runner.py --old=duckdb_delta/build/release/benchmark/benchmark_runner --new=build/release/benchmark/benchmark_runner --benchmarks=.github/regression/tpch_sf1_local.csv --verbose --threads=2 --root-dir=.
- name: Regression Test TPC-DS
if: always()
shell: bash
run: |
python duckdb/scripts/regression_test_runner.py --old=duckdb_delta/build/release/benchmark/benchmark_runner --new=build/release/benchmark/benchmark_runner --benchmarks=.github/regression/tpcds_sf1_local.csv --verbose --threads=2 --root-dir=.
python ./scripts/regression_test_runner.py --old=duckdb_delta/build/release/benchmark/benchmark_runner --new=build/release/benchmark/benchmark_runner --benchmarks=.github/regression/tpcds_sf1_local.csv --verbose --threads=2 --root-dir=.
- name: Test benchmark makefile
shell: bash
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,5 @@ include benchmark/benchmark.Makefile

# Generate some test data to test with
generate-data:
python3 -m pip install delta-spark duckdb pandas deltalake pyspark delta
python3 -m pip install delta-spark duckdb pandas deltalake pyspark
python3 scripts/generate_test_data.py
3 changes: 2 additions & 1 deletion scripts/generate_test_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,8 @@ def generate_test_data_pyspark(name, current_path, input_path, delete_predicate
builder = SparkSession.builder.appName("MyApp") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("spark.driver.memory", "8g")
.config("spark.driver.memory", "8g")\
.config('spark.driver.host','127.0.0.1')

spark = configure_spark_with_delta_pip(builder).getOrCreate()

Expand Down

0 comments on commit 2209eac

Please sign in to comment.