Added data drift doc and modified anomaly detection to represent changes in fal_dbt_examples #92
Conversation
|
Should we review this? @omeroguz45 |
Yes, I think I forgot to mention it before, sorry. |
|
@omeroguz45 , please fix the conflicts so we can review |
|
Fixed conflicts. |
| For the entire Python script, you can use this [link](https://github.com/fal-ai/fal_dbt_examples/blob/main/fal_scripts/anomaly_detection.py). | ||
| >>>>>>> 910b4bb5a0c570b6137430e231bfb8c98b213b38:examples/anomaly-detection/README.md |
There was a problem hiding this comment.
Conflict here. Choose one of the two.
| df.pop('Day') | ||
| df.pop('Year') | ||
|
|
||
| df.to_csv(f'{PATH_PREFIX}/data/city_temperature.csv') |
There was a problem hiding this comment.
Would you consider this part of the process? Users could just dbt seed the complete csv and just use whatever they need in the SELECT ... FROM {{ ref(...) }}.
I do not think doing this adds that much.
| PATH_PREFIX = '' | ||
|
|
||
| import sys | ||
| sys.path.append(f'{PATH_PREFIX}/fal_scripts') |
There was a problem hiding this comment.
why do you add this? I think all imports are from libraries? Also, now it is possible to import natively. use the fal-scripts-path variable if necessary: https://fal-docs-ekvqb18d4-features-and-labels.vercel.app/Reference/fal-scripts-path
|
|
||
| ## Moving further | ||
|
|
||
| From the comparison, we can clearly see that anomaly detection and data drift systems cannot do each other's work. However, when combined, they create a powerful data analysis tool which can be implemented into a pipeline to provide machine learning models and data analysts with more insight towards the behaviour of the data. From there, notification, data cleaning and many more systems can be added downstream for various use cases. |
Added doc for data drift example and made small modifications to anomaly detection doc to represent the changes in fal_dbt_examples.