Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read UC delta table #773

Merged
merged 63 commits into from
Jan 11, 2024
Merged

Read UC delta table #773

Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
ca70006
initial commit
xiaohanzhan-db Dec 1, 2023
34b1bb9
use databricks-sql to read delta table and convert to json
xiaohanzhan-db Dec 2, 2023
5944314
update
xiaohanzhan-db Dec 2, 2023
3cc004e
update
xiaohanzhan-db Dec 2, 2023
b22c4bb
update
xiaohanzhan-db Dec 2, 2023
b0dd43a
add mocked unittest
xiaohanzhan-db Dec 2, 2023
286650e
Fix lints
xiaohanzhan-db Dec 2, 2023
8a45576
update
xiaohanzhan-db Dec 2, 2023
b452ec3
update
xiaohanzhan-db Dec 2, 2023
c902b67
restructure code
xiaohanzhan-db Dec 4, 2023
f57c21c
Add timer for optimizing
xiaohanzhan-db Dec 4, 2023
7ff9e5a
Add db-connect
xiaohanzhan-db Dec 5, 2023
6d17cef
add wrapper
xiaohanzhan-db Dec 5, 2023
8116804
update
xiaohanzhan-db Dec 5, 2023
92cfeb2
add install dbconnect
xiaohanzhan-db Dec 5, 2023
fa9743b
update
xiaohanzhan-db Dec 5, 2023
00c3f73
update
xiaohanzhan-db Dec 5, 2023
7304c4c
patch dbconnect to allow multiple return formats
xiaohanzhan-db Dec 8, 2023
9a1916b
update
xiaohanzhan-db Dec 8, 2023
d3defc1
add arrow
xiaohanzhan-db Dec 9, 2023
da820eb
use compression
xiaohanzhan-db Dec 11, 2023
60a4be7
clean up
xiaohanzhan-db Dec 12, 2023
382a8fa
Add cluster rt check
xiaohanzhan-db Dec 12, 2023
f0286dd
Fix lints
xiaohanzhan-db Dec 13, 2023
09efbb0
remove patch.py for CI
xiaohanzhan-db Dec 13, 2023
64214de
update
xiaohanzhan-db Dec 13, 2023
13ce55b
update
xiaohanzhan-db Dec 13, 2023
5edc497
updat
xiaohanzhan-db Dec 13, 2023
ad06211
update
xiaohanzhan-db Dec 13, 2023
7a18e96
fix tests
xiaohanzhan-db Dec 13, 2023
89e658a
fix lint
xiaohanzhan-db Dec 13, 2023
acef77c
update
xiaohanzhan-db Dec 13, 2023
3e1b55f
update
xiaohanzhan-db Dec 13, 2023
6bd32a5
Add more tests
xiaohanzhan-db Dec 13, 2023
7205063
update
xiaohanzhan-db Dec 13, 2023
284b52f
update
xiaohanzhan-db Dec 13, 2023
6f8957a
update
xiaohanzhan-db Dec 13, 2023
7ce02e1
change to download_json
xiaohanzhan-db Dec 14, 2023
aba0f4d
update
xiaohanzhan-db Dec 14, 2023
040620f
fix lints
xiaohanzhan-db Dec 14, 2023
51522f8
Add decompressed option for arrow
xiaohanzhan-db Dec 15, 2023
7ae65e3
format json to jsonl
xiaohanzhan-db Dec 15, 2023
7f027fc
Add comments
xiaohanzhan-db Dec 16, 2023
34c5b72
Make cf_collect_type global option
xiaohanzhan-db Dec 16, 2023
0e9c029
fix comments
xiaohanzhan-db Dec 16, 2023
2007688
fix lints
xiaohanzhan-db Dec 22, 2023
10fd85a
fix comments
xiaohanzhan-db Dec 22, 2023
462f9b7
Fix lints
xiaohanzhan-db Jan 8, 2024
b7d9b04
change to use workspaceclient
xiaohanzhan-db Jan 8, 2024
626eeb1
Add CPT support
xiaohanzhan-db Jan 9, 2024
85cc21d
Rewire method assignment logic
xiaohanzhan-db Jan 9, 2024
d8c28ac
Fix bug in stripping https
xiaohanzhan-db Jan 10, 2024
3ff5d0e
Add tests for rewired method assignment logic
xiaohanzhan-db Jan 10, 2024
fdc7b14
Fix lints
xiaohanzhan-db Jan 10, 2024
a543b80
Fix lints
xiaohanzhan-db Jan 10, 2024
ba2dbc2
Removed logger set_level
xiaohanzhan-db Jan 10, 2024
88cad19
Remove pyspark. It conflicts with databricks-connect
xiaohanzhan-db Jan 10, 2024
ddc6d25
Merge branch 'main' into delta_input
xiaohanzhan-db Jan 11, 2024
21efd22
Update the comment
xiaohanzhan-db Jan 11, 2024
90d77fd
skip cluster version check when cluster_id is serverless
xiaohanzhan-db Jan 11, 2024
c8f8307
Add use_serverless flag
xiaohanzhan-db Jan 11, 2024
9a524be
update tests with use_serverless flag
xiaohanzhan-db Jan 11, 2024
00ca457
Fix lints
xiaohanzhan-db Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading