Dozer Project Structure

[karolisg]

Current

At the moment configurations are stored in single file dozer-config.yaml. To be more flexible, we need to allow split files in different files. So far I have an idea of two different solutions. One solution uses predefined places for configuration contents, while the other allows users to decide where to include file content.

Proposed structure

|-- dozer-config.yaml
|-- connections.yaml
|-- endpoints.yaml
|-- sources.yaml
|-- queries
   |-- *.sql
|-- lambdas
   |-- *.{js}
   |-- *.{py}

Solution #1

In this structure, only dozer-config.yaml can contain all configurations or be split to separate files.

app_name: dozer-config-example
connections:
  {{ content of connections.yaml }}
  // other connections

sql: |
  // Some sql query
  {{ contents of *.sql }}
  
sources: 
  {{ content of sources.yaml }}
  // other sources
  
endpoints: 
  {{ content of endpoints.yaml }}
  // other endpoints

Solution #2

In this solution, only dozer-config.yaml can contain all configurations and user manually decides what file and where to include using !include.

app_name: dozer-config-example
connections:
  !include connections.yaml
  // other connections

sql: |
  // Some sql query
  !include *.sql
  
sources: 
  !include sources.yaml
  // other sources
  
endpoints: 
  !include endpoints.yaml
  // other endpoints

@chubei @v3g42 Any preferences?

[chubei]

Is !include legit yaml? If so I prefer solution #2.

[karolisg]

No, it is not supported by default, so for this, we would need to implement it.

[chubei]

I don't mean to ask if yaml supports include. I'm simply wondering if it's valid yaml synax.

[karolisg]

Good point. Probably with this approach we only can define like this

...
endpoints: !include endpoints.yaml
...

[v3g42]

@karolisg Is it worth going doing a plain merge of yaml files too?
https://docs.rs/yaml-merge-keys/latest/yaml_merge_keys/

And for SQL maybe we can support providing SQL inline or include file(s)

sql: !Inline
  select * from users

# or 

sql: !Path transforms.sql

• Also another thought, currently API server doesn't need connections and sources but just the endpoint names.
• Similary pipeline doesn't need index definitions and paths etc.

Is it worth splitting them ? So our files are modular?
@chubei @snork-alt @karolisg

[karolisg]

Is it worth going doing a plain merge of yaml files too?

Do you mean that we can have multiple dozer-config.yaml files?

[chubei]

I think splitting is reasonable. We may need to reorganize a little. Index definition is part of the schema now and is needed during migration.

[v3g42]

Yes. Don't know the best way to do this. I believe kubernetes supports something similar. By default we have one dozer-config.yaml. Either through explicit configuration using include or something like you suggested or by convention (if files present in the same folder we auto discover) having the file split into multiple would be good.

Instead of include at a lower level, I would rather combine multiple yaml files and parse.

[karolisg]

I think combining and merging multiple yaml files is better solution. It would be easier to explain users - you can define single configuration file dozer-config.yaml or split to several yaml files.

[universalmind303]

I like the idea of just merging the yaml files together

# config/connections/postgres.yaml
connections:
 # postgres config here

# config/connections/snowflake.yaml
connections:
  # snowflake conn

# config/sources.yaml
sources:  # ...
 
#  ... and so on. 

then being able to run via a glob pattern.

dozer -c config/**/*.yaml

seems like from an implementation standpoint, it'd also likely be a lot easier than implementing custom yaml directives.

[karolisg]

Me, @mediuminvader, @duonganhthu43 and @chubei had discussion about structure. What we decide is to have such a structure

|-- dozer-config.yaml
|-- 01.some-config.yaml
|-- 02.some-config.yaml
|-- some-endpoints.yaml
|-- all-sources.yaml
|-- queries
   |-- one_query.sql
   |-- another_query.sql
|-- lambdas
   |-- {lambda_function}
        |-- {function}.{js, py}
        |-- lambda-config.json (name and format are not decided yet)

In this structure, we would merge all configs defined in yaml files and also merge all sql queries.
Each lambda function has its own separate folder with implementation and configuration file.

There are couple things, which we need to think of:

• How we would resolve conflicting values in config files e.g. app_name is defined in two files, how we would decide which one is correct
• With such structure, we allow storing yaml only in root folder