Spark SQL based Shark CLI #341
Conversation
Conflicts: project/SharkBuild.scala src/main/scala/shark/SharkServer2.scala
- Moved `CatalystContext` into the right package folder - Moved `getResultSetSchema` out of `CatalystContext` - Made `getResultSetSchema` to return a `Schema` rather than a `TableSchema` - `CatalystDriver.destroy` should call `super.destroy` - Coding style issues
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/Shark-Pull-Request-Builder/12211/ |
|
|
||
| import shark.LogHelper | ||
|
|
||
| class CatalystContext(sc: SparkContext) extends HiveContext(sc) with LogHelper { |
There was a problem hiding this comment.
Hmm... I'd rather just call it SharkContext. Catalyst is a query optimization framework, Spark SQL is more than that, but neither of them is a concept parallel to Hive. And this class is actually not very much related to JDBC.
| override def getSchema: Schema = tableSchema | ||
|
|
||
| override def getResults(res: JArrayList[String]): Boolean = { | ||
| if(hiveResponse == null) { |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/Shark-Pull-Request-Builder/12212/ |
|
@rxin @chenghao-intel, thanks for your comments :) I've removed the alternative Hive execution mode, and addressed issues you brought up. I cleaned up some code that doesn't follow reasonable coding conventions along the way until I realized that most of it was copied from the original Hive codebase. Maybe I should just left them unchanged... |
Merged PR #337 authored by @chenghao-intel, and then did some refactoring and update, mainly include:
CatalystContextas we don't care about response code for now, should update after SPARK-2106 is addressed.CatalystDriver.destroyshould callsuper.destroy.