Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to append_columns method for adding records #41

Open
ashander opened this issue Sep 20, 2017 · 0 comments
Open

Switch to append_columns method for adding records #41

ashander opened this issue Sep 20, 2017 · 0 comments

Comments

@ashander
Copy link
Owner

per discussion elsewhere, it's much faster but only if you avoid lists. as @molpopgen summarized:

passing lists to append_columns is a performance killer, too. One solution for ftprime will be to use numpy arrays (with the correct dtypes, which I can write down in the README for my project when I get a moment...) and act like you're in C:

x = np.zeroes([INITIAL_SIZE], dtype = the_right_type)
x_size = 0 #dummy variable

# fill X with new data, and x_size += 1 each time

When appropriate, send the slice x[:x_size] to the correct version of append_columns.

When filling, if x_size will become len(x) after adding data, create a new array that's the concatenation of x and a new empty array of size INITIAL_SIZE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant