-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed to work with newest version of Django. Fixes for viewing and downloading large reports. #54
base: master
Are you sure you want to change the base?
Conversation
Using StringIO to write one line at a time before flushing it is very slow. Reports download at about 5kbps. Also, previously the reports were created in memory and large reports would cause the server to run out of memory. This change causes the results to be written to the response while they are being read from the database. It never runs out of memory and runs at 600kbps. The same change is on the HTML view as well. The same problem can present itself if the user selects to view all records. The raw cursor is passed instead of a list of results.
Wow. Thank you so much! It's a lot of changes. I'll review and find a way to integrate all the changes. |
Fixed to work with newest version of Django. Using StringIO to write one line at a time before flushing it is very slow. Reports download at about 5kbps. Also, previously the reports were created in memory and large reports would cause the server to run out of memory. This change causes the results to be written to the response while they are being read from the database. It never runs out of memory and runs at 600kbps. The same change is on the HTML view as well. The same problem can present itself if the user selects to view all records. The raw cursor is passed instead of a list of results.
Looks like I left a pdb statement in there by accident. I have a lot of other changes - some of them are specific to my app so they're kind of hard to pull out. Heres my fill fields event code. It shows different criteria filters depending on the type of field.
|
I uploaded my whole build directory. Its tightly integrated so it won't work as is but theres some good snippits worth pulling out.
|
Wow, this is some pretty amazing work. Not sure when I'll be able to review
it all and integrate it, but thank you so much!
…On Thu, Jan 11, 2018 at 2:28 PM, Kyle Agronick ***@***.***> wrote:
I uploaded my whole build directory. Its tightly integrated so it won't
work as is but theres some good snippits worth pulling out.
***@***.***?diff=unified
<agronick@05913f7?diff=unified>
- In qbe.diagram.js there is code for making a good effort that the
tables in the diagram don't overlap.
- forms.py checks that everything is joined so you don't have queries
where everything is joined to every other thing.
- In operator and null operator
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#54 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAKl4WnKSbunAmCSmk5wVXxaCJDquuRdks5tJosAgaJpZM4OUgX8>
.
--
Javier de la Rosa
http://versae.es
|
I've been using qbe. I made a bunch of modifications to it - making it work outside the admin for less experienced users. Along the way I found a few bugs and things needed to be modified for the newer versions of Django. I pulled out some of the more important changes to integrate upstream. The version I'm committing works on Python 2 with Django 1.11.3. It doesn't add any more features. It allows reports of any size to be displayed or downloaded without overflowing the available memory. They also download several hundred times faster. I had to remove the ability to display links in the report in order to fix this.
I didn't commit the changes needed for Python 3. You should probably run 2to3 to get the codebase to work with Python3. I did that myself and it didn't take more than an hour. The changes I'm committing are only tested with Python 2.
Heres the git commit details:
Using StringIO to write one line at a time before flushing it is very slow. Reports download at about 5kbps. Also, previously the reports were created in memory and large reports would cause the server to run out of memory. This change causes the results to be written to the response while they are being read from the database. It never runs out of memory and runs at 600kbps.
The cursor is used on the HTML view as well. The same problem can present itself if the user selects to view all records. The raw cursor is read instead of a list of results.