[Bug]: Git authentication errors aren't propagated to the CLI

[peterschmidt85]

Steps to reproduce

1. Create a GitHub token that will expire soon
2. dstack init --token <token>
3. Wait until the token is expeired
4. Run any run configuration

Actual behaviour

[10:46:17] DEBUG    dstack.api.server:124 POST /api/project/aana/runs/get                                                                                      
           DEBUG    dstack.api._public.runs:232 Refreshed run aana-episodic-retrieval-dev: RunStatus.PROVISIONING                                              
[10:46:22] DEBUG    dstack.api.server:124 POST /api/project/aana/runs/get                                                                                      
[10:46:23] DEBUG    dstack.api._public.runs:232 Refreshed run aana-episodic-retrieval-dev: RunStatus.PROVISIONING                                              
[10:46:28] DEBUG    dstack.api.server:124 POST /api/project/aana/runs/get                                                                                      
           DEBUG    dstack.api._public.runs:232 Refreshed run aana-episodic-retrieval-dev: RunStatus.FAILED                                                    
aana-episodic-retrieval-dev provisioning completed (failed)
           DEBUG    dstack.api.server:124 POST /api/project/aana/runs/get                                                                                      
           DEBUG    dstack.api._public.runs:232 Refreshed run aana-episodic-retrieval-dev: RunStatus.FAILED                                                    
Run failed with error code CONTAINER_EXITED_WITH_ERROR.
Check CLI, server, and run logs for more details.

Expected behaviour

The dstack apply command should propagate the error to logs / error column / CLI output

dstack version

Any

Server logs

No response

Additional information

It was quite difficult to troubleshoot the problem as RunPod deleted the pod quickly after the error and we couldn't see the container output. After several attempts, we managed to see the error:

2024-11-25T12:05:16.935341109Z time=2024-11-25T12:05:16.935097Z level=info msg=Run job log_level=trace
2024-11-25T12:05:16.935559345Z time=2024-11-25T12:05:16.935397Z level=trace msg=Fetching git repository
2024-11-25T12:05:16.935569104Z time=2024-11-25T12:05:16.935449Z level=trace msg=Credentials is not empty
2024-11-25T12:05:16.935570956Z time=2024-11-25T12:05:16.935467Z level=trace msg=Select HTTPS protocol
2024-11-25T12:05:16.935572776Z time=2024-11-25T12:05:16.935485Z level=trace msg=Checking out remote repo GIT URL=https://github.com/mobiusml/aana_episodic_retrieval
2024-11-25T12:05:16.935591289Z time=2024-11-25T12:05:16.935528Z level=info msg=git checkout branch=main hash=6b8d473f9c43f4c2ee08734a0dad562e27f09af8 path=/workflow auth=*http.BasicAuth url=https://github.com/mobiusml/aana_episodic_retrieval
2024-11-25T12:05:17.111475860Z time=2024-11-25T12:05:17.111049Z level=info msg=Job state changed new=failed
2024-11-25T12:05:17.111518173Z time=2024-11-25T12:05:17.111141Z level=error msg=Executor failed err=[executor.go:108 executor.(*RunExecutor).Run] [repo.go:24 executor.(*RunExecutor).setupRepo] [repo.go:65 executor.(*RunExecutor).prepareGit] authentication required
2024-11-25T12:05:17.111543642Z time=2024-11-25T12:05:17.11135Z level=info msg=Job finished, shutting down
2024-11-25T12:05:19.722483168Z time=2024-11-25T12:05:19.722065Z level=info msg=Logs streaming finished endpoint=/api/pull
2024-11-25T12:05:19.722510280Z time=2024-11-25T12:05:19.722182Z level=debug endpoint=/api/pull status=200 method=GET
2024-11-25T12:05:20.058534151Z 2024/11/25 12:05:20 http: response.WriteHeader on hijacked connection from github.com/dstackai/dstack/runner/internal/runner/api.NewServer.JSONResponseHandler.func8 (common.go:121)
2024-11-25T12:05:20.058565845Z 2024/11/25 12:05:20 http: response.Write on hijacked connection from fmt.Fprintln (print.go:305)
2024-11-25T12:05:20.058617112Z time=2024-11-25T12:05:20.0585Z level=debug endpoint=/logs_ws status=200 method=GET
2024-11-25T12:05:20.058639576Z time=2024-11-25T12:05:20.058606Z level=info msg=Logs streaming finished endpoint=/logs_ws