Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster Date Conversion #56

Open
GoogleCodeExporter opened this issue Nov 3, 2015 · 0 comments
Open

Faster Date Conversion #56

GoogleCodeExporter opened this issue Nov 3, 2015 · 0 comments

Comments

@GoogleCodeExporter
Copy link

Hello,
When querying large data sets containing a Date column, the text to date 
conversion from as.Date() can sometimes be the largest speed bottleneck in the 
query. See: 
https://stat.ethz.ch/pipermail/r-devel/2013-August/067141.html

Consider using as.Date on unique values of the date only. For example, this 
patch to PostgreSQLSupport.R reduced a 130 second query that selection 4 
million rows from a table containing a Date column, to only 30 seconds:

diff PostgreSQLSupport.R PostgreSQLSupport.new
412a413,419
>     as.Date2 <- function(x) {            ## Significantly faster conversion 
with duplicated dates
>         if (anyDuplicated(x)) {
>             ux <- unique(x)
>             idx <- match(x, ux)
>             return(as.Date(ux)[idx])
>         } else as.Date(x)
>     }
433c440
<             rel[,i] <- as.Date(rel[,i])

---
>             rel[,i] <- as.Date2(rel[,i])

Alternatively, consider parsing the character string directly without making a 
system call, as in, for example the fasttime::fastPOSIXct function. See 
https://stat.ethz.ch/pipermail/r-devel/2013-August/067142.html

Another possibility, perhaps it would be more efficient to have Date data sent 
as an integer (days from epoch) rather than a character string.
--Robert

Original issue reported on code.google.com by [email protected] on 16 Aug 2013 at 8:39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant