Method saveAsBigQueryTable([class java.lang.String]) does not exist: Issue persists even after shading guava dependecies

[dduran28]

I am still experiencing the above issue when trying to run the PySpark example in Databricks.

`bq = bigquery.BigQuerySQLContext(session._wrapped._jsqlContext)
bq.setGcpJsonKeyFile(KEY_FILE)
bq.setBigQueryProjectId(BQ_PROJECT_ID)
bq.setGSProjectId(BQ_PROJECT_ID)
bq.setBigQueryGcsBucket(STAGING_BUCKET)
bq.setBigQueryDatasetLocation(DATASET_LOCATION)

bqDF = bigquery.BigQueryDataFrame(df._jdf)
bqDF.saveAsBigQueryTable(
"{0}:{1}.{2}".format(BQ_PROJECT_ID, DATASET_ID, TABLE_NAME)`

gives me the following error

Py4JError: An error occurred while calling o333.saveAsBigQueryTable. Trace:
py4j.Py4JException: Method saveAsBigQueryTable([class java.lang.String]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:341)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:349)
at py4j.Gateway.invoke(Gateway.java:286)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)>

This is my cluster configuration:

These are my installed JARs and libraries

[dduran28]

Bump

[samelamin]

Is this still an issue?

[dduran28]

Yes it is

[samelamin]

this is dependant on your cluster, can you let me know where you are running it?

spark version, manually create cluster vs EMR/Databricks/Dataproc