HPCC-30776 ECL mods to support pluggable file format syntax *** WIP ***

[dcamper]

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

[github-actions]

https://track.hpccsystems.com/browse/HPCC-30776
Jira updated

[ghalliday]

Looks like you're getting to grips with the structure. I would also add a new operator no_filetype, create that in setPluggableModeExpr
e.g.

targetAttr.setExpr(createValue(no_filetype, makeNullType(), createAttribute(fileFormat), *options));

Then search for no_csv in the sources in ecl/hql and add corresponding entries. You could also do the same for ecl/hqlcpp, but I would probably leave that to a separate PR, which meant the new syntax generated the generic disk read activity.

[ghalliday]

Is it worth special casing these? I was expecting these to all generate the new format, and by implication be a way of explicitly using the new generic disk read code.

[dcamper]

I am taking this change very slowly while I learn the vast array of helpers and structs. As of this writing, I was focused solely on changing the syntax and making sure it worked. Next step will be changing the underlying code and how it is called.

[dcamper]

Also: A plugin-style filetype will process options differently than flex. This is definitely not a problem with a new filetype such as Parquet, but may cause backwards-compatibility issues with an existing filetype (errors may be caught at different times and/or reported differently). Is that a concern, given that this is introducing a new syntax?

[ghalliday]

whitespace: tabs being used for indentation rather than spaces.

[dcamper]

Fixed.

[ghalliday]

Questionable whether it needs to check for a type at this point. Advantages and disadvantages...

[dcamper]

Closing while continuing work. Will open a new draft later.