Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: propagate SqlBuilder to Antlr Expression visitor [DHIS-16705] #19362

Merged
merged 5 commits into from
Dec 3, 2024

Conversation

luciano-fiandesio
Copy link
Contributor

@luciano-fiandesio luciano-fiandesio commented Dec 2, 2024

Description

This PR attempts at propagating the SqlBuilder into the Antlr parser, so that parser can generate SQL expression compatible with the underlying analytics database.

Changes

  • Configure the ExpressionService Spring Bean into a ServiceConfig, so that the SqlBuilder can be injected explicitly.
  • Modified DefaultProgramIndicatorService so that the expression map is now built externally (see ExpressionMapBuilder). This allow to use the new withSqlBuilder method on the expressions in the map and pass the SqlBuilder to each expression.
  • Add a "fluent" withSqlBuilder method to ProgramExpressionItem.
  • Use a custom builder for CommonExpressionVisitor to make possible to initialize the existing DefaultStatementBuilder with SqlBuilder
  • Add a new cast function to SqlBuilder to hanlde SQL casting

@luciano-fiandesio luciano-fiandesio requested review from larshelge, jimgrace and maikelarabori and removed request for larshelge December 2, 2024 13:54
@luciano-fiandesio luciano-fiandesio added the run-api-analytics-tests Enables analytics e2e tests label Dec 2, 2024

/**
* Used to collect the string replacements to build a description. This may contain names of
* metadata from the database.
*/
@Builder.Default private Map<String, String> itemDescriptions = new HashMap<>();
private Map<String, String> itemDescriptions;

Check notice

Code scanning / CodeQL

Exposing internal representation Note

getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
getItemDescriptions exposes the internal representation stored in field itemDescriptions. The value may be modified
after this call to getItemDescriptions
.
// Custom constructor
// -------------------------------------------------------------------------

@Builder(toBuilder = true)

Check notice

Code scanning / CodeQL

Use of default toString() Note

Default toString(): ExpressionInfo inherits toString() from Object, and so is not suitable for printing.
Default toString(): ExpressionState inherits toString() from Object, and so is not suitable for printing.
Default toString(): ProgramExpressionParams inherits toString() from Object, and so is not suitable for printing.
@@ -71,4 +87,25 @@ public DefaultOutboundMessageBatchService defaultOutboundMessageBatchService(
public ResourceBundleManager resourceBundleManager() {
return new DefaultResourceBundleManager();
}

@Bean("org.hisp.dhis.expression.ExpressionService")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if this definition is actually unnecessary. Since SqlBuilder is now a bean, you can now inject it into DefaultExpressionService as usual with (Lombok) constructor injection.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good catch, removed.

@@ -146,6 +147,10 @@ public class DefaultProgramIndicatorService implements ProgramIndicatorService {

private final Cache<String> analyticsSqlCache;

private final SqlBuilder sqlBuilder;

public static ImmutableMap<Integer, ExpressionItem> PROGRAM_INDICATOR_ITEMS;
Copy link
Member

@larshelge larshelge Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a little bit of an anti-pattern to initialize static variables in constructors in singletons. Since this is a Spring singleton bean, the constructor is guaranteed to be invoked once. It is more natural to make it consistent with other dependencies, and instead use private final:

private final Map<Integer, ExpressionItem> programIndicatorItems;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, my bad, this part is actually no longer like this. The problem is that this PROGRAM_INDICATOR_ITEMS variable is only used by Unit Tests and not by the runtime. But it can no longer be static, since we need to pass the SqlBuilder. Fixing the unit tests, this will change too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understand. Yes good to get it out of the implementation if it is only used in tests.

@@ -173,57 +180,63 @@ public DefaultProgramIndicatorService(
this.dimensionService = dimensionService;
this.i18nManager = i18nManager;
this.analyticsSqlCache = cacheProvider.createAnalyticsSqlCache();
this.sqlBuilder = sqlBuilder;

PROGRAM_INDICATOR_ITEMS = buildExpressionMap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this.programIndicatorItems = buildExpressionMap();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see comment above. The PROGRAM_INDICATOR_ITEMS variable no longer exists;

@Override
public String cast(String column, DataType dataType) {
return switch (dataType) {
case NUMERIC -> "toDecimal64(" + column + ", 8)"; // 8 decimal places precision
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use String.format over String concatenation.

@Override
public String cast(String column, DataType dataType) {
return switch (dataType) {
case NUMERIC -> "CAST(" + column + " AS DECIMAL)";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use String.format over String concatenation. Use lower-case SQL by default.

@@ -287,6 +288,16 @@ public interface SqlBuilder {
*/
String jsonExtractNested(String json, String... expression);

/**
* Generates a database-specific SQL casting expression for the given column or expression.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have to say "database-specific" in the Javadoc, as everything in SqlBuilder is database specific.

@larshelge
Copy link
Member

Looks good! Great to see these incremental improvements.

Copy link

sonarcloud bot commented Dec 2, 2024

@larshelge larshelge changed the title chore: propagate SqlBuilder to Antlr Expression visitor (DHIS-16705) chore: propagate SqlBuilder to Antlr Expression visitor [DHIS-16705] Dec 3, 2024
@larshelge larshelge merged commit 8050437 into master Dec 3, 2024
16 checks passed
@larshelge larshelge deleted the DHIS-16705_USE_SQLBUILDER_IN_EXPRESSION_PARSER branch December 3, 2024 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-api-analytics-tests Enables analytics e2e tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants