Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save_axs_table() deletes files from path #6

Open
ctslater opened this issue Jun 7, 2019 · 2 comments
Open

save_axs_table() deletes files from path #6

ctslater opened this issue Jun 7, 2019 · 2 comments
Labels
bug Something isn't working important Important to addresses

Comments

@ctslater
Copy link
Member

ctslater commented Jun 7, 2019

I was using save_axs_table() to save the new ZTF data and used the path= argument to put it on the new disk drives. Spark then deleted everything from the path I gave it, including a bunch of unrelated directories and all the ZTF data I was trying to import.

@zecevicp
Copy link
Collaborator

This is how Spark's saveAsTable() method works. I added a warning note in the save_axs_table() method's documentation about this.

@ctslater
Copy link
Member Author

Let's leave this open until we have more of a safeguard. I agree that it's how spark works, but I lost a few days of work to this and it could be much worse if someone specifies their home directory, for example, so I think it's a very risky thing to expose to the user.

@ctslater ctslater reopened this Jun 11, 2019
@mjuric mjuric added bug Something isn't working important Important to addresses labels Dec 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working important Important to addresses
Projects
None yet
Development

No branches or pull requests

3 participants