Merge pull request #1075 from girder/annotation-json-read-size

Increase maximum size of json file that will be read for annotations.

CHANGELOG.md

@@ -5,10 +5,11 @@
 ### Improvements
 - Allow ICC correction to specify intent ([#1066](../../pull/1066))
 - Make tile sources pickleable ([#1071](../../pull/1071))
-- Extract scale information from more bioformats files ([#1073](../../pull/1073))
+- Extract scale information from more bioformats files ([#1074](../../pull/1074))
 
 ### Bug Fixes
 - The cache could reuse a class inappropriately ([#1070](../../pull/1070))
+- Increase size of annotation json that will be parsed ([#1075](../../pull/1075))
 
 ## 1.20.1
 

girder_annotation/girder_large_image_annotation/handlers.py

@@ -124,7 +124,16 @@ def process_annotations(event):  # noqa: C901
         logger.error('Could not load models from the database')
         return
     try:
-        data = orjson.loads(File().open(file).read().decode())
+        if file['size'] > 1 * 1024 ** 3:
+            raise Exception('File is larger than will be read into memory.')
+        data = []
+        with File().open(file) as fptr:
+            while True:
+                chunk = fptr.read(1024 ** 2)
+                if not len(chunk):
+                    break
+                data.append(chunk)
+        data = orjson.loads(b''.join(data).decode())
     except Exception:
         logger.error('Could not parse annotation file')
         raise