API endpoint to fetch all the properties at once

[flavioamieiro]

@fccoelho opened this issue in NAMD/pypln.api#40, but I'm migrating it here because it demands a change in pypln.web itself (which would be naturally followed by pypln.api).

"""
currently when we want to acess properties of a PyPLN documents, we need to fetch each on a separate request. This impractical.

There should be an diferent url to fetch the properties data.

For example today when we point the browser to a document properties url, such as this http://fgv.pypln.org/documents/57894/properties/ we get back a json object with an array of property urls for that document:

{
    "properties": [
        "http://fgv.pypln.org/documents/57894/properties/average_sentence_length/",
        "http://fgv.pypln.org/documents/57894/properties/average_sentence_repertoire/",
        "http://fgv.pypln.org/documents/57894/properties/contents/",
        "http://fgv.pypln.org/documents/57894/properties/file_id/",
        "http://fgv.pypln.org/documents/57894/properties/file_metadata/",
        "http://fgv.pypln.org/documents/57894/properties/filename/",
        "http://fgv.pypln.org/documents/57894/properties/forced_decoding/",
        "http://fgv.pypln.org/documents/57894/properties/freqdist/",
        "http://fgv.pypln.org/documents/57894/properties/language/",
        "http://fgv.pypln.org/documents/57894/properties/lemmas/",
        "http://fgv.pypln.org/documents/57894/properties/length/",
        "http://fgv.pypln.org/documents/57894/properties/md5/",
        "http://fgv.pypln.org/documents/57894/properties/mimetype/",
        "http://fgv.pypln.org/documents/57894/properties/momentum_1/",
        "http://fgv.pypln.org/documents/57894/properties/momentum_2/",
        "http://fgv.pypln.org/documents/57894/properties/momentum_3/",
        "http://fgv.pypln.org/documents/57894/properties/momentum_4/",
        "http://fgv.pypln.org/documents/57894/properties/noun_phrases/",
        "http://fgv.pypln.org/documents/57894/properties/palavras_raw/",
        "http://fgv.pypln.org/documents/57894/properties/palavras_raw_ran/",
        "http://fgv.pypln.org/documents/57894/properties/pos/",
        "http://fgv.pypln.org/documents/57894/properties/repertoire/",
        "http://fgv.pypln.org/documents/57894/properties/semantic_tags/",
        "http://fgv.pypln.org/documents/57894/properties/sentences/",
        "http://fgv.pypln.org/documents/57894/properties/tagset/",
        "http://fgv.pypln.org/documents/57894/properties/text/",
        "http://fgv.pypln.org/documents/57894/properties/tokens/",
        "http://fgv.pypln.org/documents/57894/properties/upload_date/",
        "http://fgv.pypln.org/documents/57894/properties/wordcloud/"
    ]
}

I propose we add a new API endpoint which could have the form of either:

http://fgv.pypln.org/documents/57894/properties_data/

or

http://fgv.pypln.org/documents/57894/properties/gzip

which would return a gzipped JSON with all the data.

"""

[flavioamieiro]

I agree this is important, specially for our use of PyPLN in mediacloud, but not only for that. I think it will be very useful for everyone.

I would propose /documents/<id>/properties/all or /documents/<id>/properties/all_data/ as the endpoint, just so it would still be under /documents/<id>/properties/.

[fccoelho]

I Think these endpoints are pretty clear, but I think the one ending in
/all_data/ is more explicit. So if @flavioamieiro agrees, let's move on
with this.

On Mon, Sep 21, 2015 at 1:30 PM, Flávio Amieiro [email protected]
wrote:

I agree this is important, specially for our use of PyPLN in mediacloud,
but not only for that. I think it will be very useful for everyone.

I would propose /documents//properties/all or
/documents//properties/all_data/ as the endpoint, just so it would
still be under /documents//properties/.

—
Reply to this email directly or view it on GitHub
#134 (comment).

Flávio Codeço Coelho

+55(21) 3799-5551
Professor
Escola de Matemática Aplicada
Fundação Getulio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil

[flavioamieiro]

Just to document my recent findings: I've started implementing this in a branch but, as the commit message says, I've run into a problem. Values that work very well on their own, don't work at all when being rendered together.

Another weird thing about this bug specifically is that some strings with non-ascii chars trigger it and others don't. I managed to isolate the keys that (when being rendered together with the other properties): 'contents', 'lemmas', 'palavras_raw' and 'semantic_tags'. When I remove these four keys from the returned data, it works. When I add it back, it gives me a UnicodeDecodeError.

Also, automating tests for this has been hard, because of the way fixtures are right now. I decided to leave this untested for the time being since the next task is to remove MongoDict from the project.