Reorganize Root-Level Scripts into Dedicated Scripts Directory

[wjbmattingly]

Current Status

The project root directory is cluttered with numerous script files (.py and .sh), making it difficult to navigate and maintain. Current script locations:

/
├── backup_tables.sh
├── checkDataUpdates.py
├── export_parallel.sh
├── full-build.sh
├── harvest_parallel.sh
├── import_parallel.sh
├── load_parallel.sh
├── run-harvest.py
├── run-load.py
├── run-merge.py
...

Problem

• Root directory is overcrowded with script files
• Difficult to quickly find relevant scripts
• Poor separation of concerns

Proposed Solution

Create a dedicated scripts/ directory with meaningful subdirectories based on script functionality:

scripts/
├── data/
│   ├── backup_tables.sh
│   ├── checkDataUpdates.py
│   └── import_parallel.sh
├── pipeline/
│   ├── run_harvest.py
│   ├── run_load.py
│   └── run_merge.py
└── utils/
    ├── export_parallel.sh
    └── full_build.sh

Implementation Steps

1. Create new directory structure
2. Move scripts to appropriate subdirectories
3. Update all references to script paths in:
   • Documentation
   • Other scripts that may call these scripts
   • Deployment configurations
4. Test that all scripts work from their new locations
5. Update README.md with new project structure

Migration Strategy

1. Create feature branch for reorganization
2. Move files in logical groups
3. Test thoroughly after each group move
4. Update relative path references
5. Run full test suite
6. Deploy to staging environment for verification

Breaking Changes

• All paths referencing these scripts will need to be updated
• Development workflows may need adjustment

Priority

Low

Tasks

• Analyze current structure
• Propose new structure
• Create a branch to modify structure
• Create new proposed structure - Warning, this will cause breaking changes
• Use this opportunity to remove unnecessary files
• Update codebase to align with refactoring

[wjbmattingly]

Proposed structure

scripts/
├── build/
│ └── full-build.sh
│
├── download/
│ └── harvest_parallel.sh
│
├── runs/
│ ├── run-all.sh
│ ├── run-export.py
│ ├── run-harvest.py
│ ├── run-load.py
│ ├── run-merge.py
│ └── run-reconcile.py
│
├── storage/
│ ├── backup_tables.sh
│ ├── export_parallel.sh
│ ├── google-sames-diffs.py
│ ├── import_parallel.sh
│ ├── load_parallel.sh
│ ├── load-csv-map2.py
│ ├── make_test_dataset.py
│ ├── make-wd-differents.py
│ ├── manage-data.py
│ ├── merge_parallel.sh
│ ├── merge-metatypes.py
│ ├── nt_parallel.sh
│ ├── populate-timestamps.py
│ ├── reconcile_parallel.sh
│ └── watch-mlcp.py
│
├── update/
│ ├── checkDataUpdates.py
│ ├── debug-reconcile.py
│ ├── updateExternal.py
│ └── updateExternal_parallel.sh
│
└── sources/