spark-sql-datasource.adoc

DataSource

DataSource case class belongs to the Data Source API (along with DataFrameReader and DataFrameWriter).

Caution

FIXME Review DataSource case class.

write

Caution

FIXME

createSource

createSource(metadataPath: String): Source

Caution

FIXME

sourceSchema

Caution

FIXME

inferFileFormatSchema

inferFileFormatSchema(format: FileFormat): StructType

inferFileFormatSchema private method computes (aka infers) schema (as StructType). It returns userSpecifiedSchema if specified or uses FileFormat.inferSchema. It throws a AnalysisException when is unable to infer schema.

It uses path option for the list of directory paths.

Note	It is used by DataSource.sourceSchema and DataSource.createSource when FileFormat is processed.