Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: default JSON decoding should not use string #11

Closed
JulienPeloton opened this issue Dec 6, 2024 · 1 comment
Closed

bug: default JSON decoding should not use string #11

JulienPeloton opened this issue Dec 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@JulienPeloton
Copy link
Member

The function

@profile
def hbase_to_dict(hbase_output, escape_slash=False):
    """Optimize hbase output TreeMap for faster conversion to DataFrame"""
    gateway = JavaGateway(auto_convert=True)
    JSONObject = gateway.jvm.org.json.JSONObject

    # We do bulk export to JSON on Java side to avoid overheads of iterative access
    # and then parse it back to Dict in Python
    if escape_slash:
        hbase_output = str(hbase_output)
    optimized = json.loads(JSONObject(str(hbase_output)).toString())

    return optimized

should not have str(hbase_output) by default when calling JSONObject (only used when escaping slashes).

@JulienPeloton JulienPeloton added the bug Something isn't working label Dec 6, 2024
JulienPeloton added a commit that referenced this issue Dec 6, 2024
@JulienPeloton
Copy link
Member Author

See also #12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant