Testing Locally on a Remote Host

[youra6]

Hi,

I looked through the docs and I couldn't much on the topic. If I have a workspace.yaml configured along with my solids, pipelines on my local machine, is it possible to execute_pipeline() on a Dagster instance running on a remote server?

Not sure if this is even possible since I have not seen anyone else ask, but curious if it was.

[timothy-mugayi]

This is a question that makes sense sorta like running local code on SAAS platform better resources/scales on a remote cluster. My question is sorta inline with that remote execution but pre-deployed pipeline via web request #4397
I could imagine building a channel to ether pickle/raw upload your pipeline to a dag directory which dagster would pickup and you could then trigger the execution via the UI

[youra6]

Thanks for the response! Correct my ignorance since I'm not an engineer, but doesn't make sense for Dagster to include a feature like you described since they self proclaim to be better at making pipelines than it's competitors?

[timothy-mugayi]

@youra6 perhaps a combination of what I mentioned with #4397

Some other additional inspiration reloading of workspace via graphql #3803

[sryza]

Hi @youra6 - as @timothy-mugayi mentioned, the current recommended way to do this is to run a Dagit webserver on the remote machine running the DagsterInstance, and then to execute the job using the Python GraphQL API client - https://docs.dagster.io/concepts/dagit/graphql-client.

[youra6]

Thanks @sryza, its looks like you can create a client object where you can establish the host and port.

Prior to this, I've been running the execute_pipeline() in my main.py file.

However,client uses the submit_pipeline_execution method which takes in the PIPELINE_NAME. How does one set that up on the localhost?

[sryza]

To clarify, is your question about what value you should pass for the pipeline_name parameter?

[youra6]

Not exactly, I know the pipeline_name should be a string, but what I don't know is how do I set the pipeline_name on the client side (say I'm trying to run my script locally)?

In the execute_pipeline() function, you have to pass in PipelineDefinition object, but submit_pipeline_execution takes in the name of the pipeline it seems like.

Someone on the slack channel mentioned that I need to set up the repository on the server.

[sryza]

If you defined your pipeline using something like:

@pipeline
def my_pipeline():
    ...

Then you would want to set pipeline_name to "my_pipeline" when you invoke submit_pipeline_execution.