Skip to content

Commit

Permalink
added documentation for databricks connection "wait for cluster" feature
Browse files Browse the repository at this point in the history
  • Loading branch information
GoEddie committed Dec 4, 2024
1 parent b63f65a commit cd8c07f
Showing 1 changed file with 29 additions and 0 deletions.
29 changes: 29 additions & 0 deletions docs/connection-options.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Connection Options

When connecting to a Databricks cluster, it might be that the cluster is in the starting state so I have added some code to sit and poll the spark server until the cluster is available.

There are two options that help configure this and I added the options to the `SparkSessionBuilder`.

## DatabricksWaitForClusterMaxTime

This controls how long we will sit and wait for the cluster to start:

```csharp
var spark =
SparkSession
.Builder
.DatabricksWaitForClusterMaxTime(5)
.GetOrCreate();
```

## DatabricksWaitForClusterOnSessionCreate

The second option disables the behaviour completely:

```csharp
var spark =
SparkSession
.Builder
.DatabricksWaitForClusterOnSessionCreate(false)
.GetOrCreate();
```

0 comments on commit cd8c07f

Please sign in to comment.