Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace lakefs_client with lakefs-sdk in Spark integration test harness #8403

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

replace lakefs_client with lakefs-sdk in Spark integration test harness #8403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4f752b9
replace lakefs_client with lakefs-sdk
mahdikhashan Nov 29, 2024
ff99bb6
use lakefs-sdk object and methods
mahdikhashan Nov 29, 2024
c2a9fe0
fix: use repositories_api instead of repositories
mahdikhashan Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion test/spark/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
lakefs_client==0.104.0
lakefs-sdk==1.43.0
python-on-whales==0.62.0
tenacity==8.2.2
12 changes: 6 additions & 6 deletions test/spark/run-test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import argparse
import sys

import lakefs_client
from lakefs_client import models
from lakefs_client.client import LakeFSClient
import lakefs_sdk
from lakefs_sdk import models
from lakefs_sdk.client import LakeFSClient
from python_on_whales import docker
from tenacity import retry, stop_after_attempt, wait_fixed

Expand All @@ -24,7 +24,7 @@ def get_spark_submit_cmd(submit_flags, spark_config, jar_name, jar_args):

@retry(wait=wait_fixed(1), stop=stop_after_attempt(7))
def wait_for_setup(lfs_client):
repositories = lfs_client.repositories.list_repositories()
repositories = lfs_client.repositories_api.list_repositories()
assert len(repositories.results) >= 0

def main():
Expand All @@ -48,11 +48,11 @@ def main():
submit_flags = ["--jars", "/target/client.jar"]

lfs_client = LakeFSClient(
lakefs_client.Configuration(username=lakefs_access_key,
lakefs_sdk.Configuration(username=lakefs_access_key,
password=lakefs_secret_key,
host='http://localhost:8000'))
wait_for_setup(lfs_client)
lfs_client.repositories.create_repository(
lfs_client.repositories_api.create_repository(
models.RepositoryCreation(name=args.repository,
storage_namespace=args.storage_namespace,
default_branch='main',))
Expand Down