Improve upload speed

[flavioamieiro]

As described in NAMD/mediacloud_backend#65 the upload speed is not enough to have it as part of a regular process. We need to benchmark the upload and see where we can improve speed.

[fccoelho]

I think that the reason why the add_document call takes a while to return
is because it has to wait for the pipeline to end, so that it can return
the document I'd to the caller. Perhaps if we modified the code to create
an empty document and return the Id immediately, The pipeline can then
update this document whenever it finishes. @turicas what do you think?
Em 27/06/2014 13:20, "Flávio Amieiro" [email protected] escreveu:

As described in NAMD/mediacloud_backend#65
NAMD/mediacloud_backend#65 the upload speed is
not enough to have it as part of a regular process. We need to benchmark
the upload and see where we can improve speed.

—
Reply to this email directly or view it on GitHub
#118.

[turicas]

@fccoelho, the add_document call (on client) does not wait for the pipeline to finish, it only waits for the Web interface to add the document pipeline on the backend (only one ZMQ call). We were thinking in not sending this message to the backend directly, but using a background daemon, so we can return directly to the API user, but I figured out in my tests that this part does not take some time.

[flavioamieiro]

As @turicas said, this was our first thought, but after some tests we realized creating the document was not our bottleneck. Also, I don't think any operation is going to be faster than just creating the document before returning as we do now. @turicas do you remember if we found any bottlenecks in our tests?

[turicas]

@flavioamieiro, the bottenecks I've found were related to running the services (broker and pypln-web, basically) in the same machine, so when the workers are consuming 100% CPU, the web process could not answer in a normal time (we need to modify the deployment process to permit it).

[fccoelho]

We must either renice the processes to give priority to the frontend or
simply do not use the same machine for both.
Em 30/06/2014 13:10, "Álvaro Justen" [email protected] escreveu:

@flavioamieiro https://github.com/flavioamieiro, the bottenecks I've
found were related to running the services (broker and pypln-web,
basically) in the same machine, so when the workers are consuming 100% CPU,
the web process could not answer in a normal time (we need to modify the
deployment process to permit it).

—
Reply to this email directly or view it on GitHub
#118 (comment).