Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on really large files, silent fail reading connection #242

Open
jordansread opened this issue Feb 5, 2016 · 5 comments
Open

on really large files, silent fail reading connection #242

jordansread opened this issue Feb 5, 2016 · 5 comments
Labels

Comments

@jordansread
Copy link
Contributor

e.g., 2961595387 of 5289688432 bytes written, exception message: Connection reset

I think this is often silent with readLines(con), but shouldn't be. Not sure what to do (return NULL if partial?), but we should fix this and use a more robust way of getting/reading the file.

@jordansread jordansread added the bug label Feb 5, 2016
@jordansread
Copy link
Contributor Author

to be clear, that message is from the server logs, not geoknife. It ends up mostly swallowing a warning about incomplete end of line

@jordansread
Copy link
Contributor Author

@dblodgett-usgs 's idea is to check the Content-Length in the response headers w/ the file.

@jiwalker-usgs
Copy link
Contributor

The server is saying connection reset, what does it say on the client for the same connection? If the server thinks it is the client and the client thinks its the server, it is probably a middle man of some sort.

@jordansread
Copy link
Contributor Author

I just get warning incomplete line on ... from R, but I don't use a GET for the file, I read the connection directly which is probably a bad idea.
file is a connection in that case:
https://github.com/USGS-R/geoknife/blob/master/R/parseTimeseries.R#L20

@jordansread
Copy link
Contributor Author

switching to some other ways of getting the file to test if this is the weak point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants