Environment Variables Not Preserved During gridtk resubmit

[183amir]

Issue:

When jobs are submitted using gridtk, only the command is saved, and the environment variables active at submission time (e.g., from an activated conda environment) are not captured or preserved. This becomes problematic during resubmission because gridtk uses sbatch to resubmit jobs, and sbatch only applies the environment variables present at resubmission time.

Since environment variables are critical for tasks such as selecting the correct Python binary in a virtual or conda environment, this oversight may cause unexpected behavior during resubmission if the original environment isn't re-established.

Example Problem:

If you activate a conda environment before submitting a job with gridtk submit --- python, the active conda environment’s version of Python will be used during the initial submission. However, during resubmission, if the conda environment is not activated, gridtk resubmit may invoke a different Python version from the system environment, leading to potential failures or incorrect execution.

Proposed Solution:

To avoid issues during resubmission, gridtk should either capture the environment variables at the time of the original submission or warn users to ensure they activate the same environment before resubmission. Alternatively, a mechanism could be added to gridtk resubmit to reapply the origin environment settings, ensuring consistent execution across submissions and resubmissions.