Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot use #{address.city} in expression #38

Open
tsreaper opened this issue Mar 25, 2022 · 5 comments
Open

Cannot use #{address.city} in expression #38

tsreaper opened this issue Mar 25, 2022 · 5 comments

Comments

@tsreaper
Copy link
Contributor

Run the following SQL in SQL client:

create temporary table faker ( a string ) with ( 'connector' = 'faker', 'number-of-rows' = '10', 'fields.a.expression' = '#{address.city}' );
select * from faker;

Exception:

Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.faker'.

Table options are:

'connector'='faker'
'fields.a.expression'='#{address.city}'
'number-of-rows'='10'
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:159) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	... 11 more
Caused by: org.apache.flink.table.api.ValidationException: Invalid expression for column "a".
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.readAndValidateFieldExpression(FlinkFakerTableSourceFactory.java:187) ~[flink-faker-0.4.1.jar:?]
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:90) ~[flink-faker-0.4.1.jar:?]
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:21) ~[flink-faker-0.4.1.jar:?]
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:156) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	... 11 more
Caused by: java.lang.RuntimeException: Unable to resolve #{city_suffix} directive.
	at net.datafaker.service.FakeValuesService.resolveExpression(FakeValuesService.java:446) ~[flink-faker-0.4.1.jar:?]
	at net.datafaker.service.FakeValuesService.lambda$resolveExpression$1(FakeValuesService.java:450) ~[flink-faker-0.4.1.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at net.datafaker.service.FakeValuesService.resolveExpression(FakeValuesService.java:457) ~[flink-faker-0.4.1.jar:?]
	at net.datafaker.service.FakeValuesService.expression(FakeValuesService.java:407) ~[flink-faker-0.4.1.jar:?]
	at net.datafaker.Faker.expression(Faker.java:713) ~[flink-faker-0.4.1.jar:?]
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.readAndValidateFieldExpression(FlinkFakerTableSourceFactory.java:185) ~[flink-faker-0.4.1.jar:?]
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:90) ~[flink-faker-0.4.1.jar:?]
	at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:21) ~[flink-faker-0.4.1.jar:?]
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:156) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
	... 11 more
@knaufk
Copy link
Owner

knaufk commented Mar 28, 2022

@bodiam could you have a look at this when you have time? Seems to be rather releated to data-faker than flink-faker, no?

@bodiam
Copy link

bodiam commented Mar 28, 2022

@knaufk don't know, not sure what you're doing here. Can you reproduce this without flink? Then we can more easily fix it

@bodiam
Copy link

bodiam commented Mar 28, 2022

@knaufk which version of datafaker is this? Also, could you try 1.3.0-SNAPSHOT if you're running an earlier version?

@snuyanzin
Copy link
Contributor

snuyanzin commented Apr 2, 2022 •
edited
Loading

@tsreaper as a WA currently it should work for Address.city (with first capital letter)
like that

create temporary table faker ( a string ) with ( 'connector' = 'faker', 'number-of-rows' = '10', 'fields.a.expression' = '#{Address.city}' );
select * from faker;

@bodiam
Copy link

bodiam commented Jun 9, 2022

@tsreaper Is this still an issue or can this be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bodiam @snuyanzin @knaufk @tsreaper