Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MVP: Handle extra fields to export #278

Merged
merged 62 commits into from
Nov 11, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
4f81008
extend `try_get_number()` method to handle `inf` and `nan`
joshuaberetta Oct 21, 2021
d4e398a
add tests
joshuaberetta Oct 21, 2021
9cee720
change how fields are combined across versions for repeat groups
joshuaberetta Oct 27, 2021
920c8ff
add test for fix
joshuaberetta Oct 27, 2021
f1a1ad2
chance `field_section_name` to `section_field_name`
joshuaberetta Oct 27, 2021
0537c6a
update code comment
joshuaberetta Oct 27, 2021
8a6300c
make requested changes
joshuaberetta Oct 28, 2021
66adc80
make requested changes
joshuaberetta Oct 28, 2021
2352c61
add v2 fixture
joshuaberetta Oct 28, 2021
0dbb0d2
Merge pull request #276 from kobotoolbox/275-fix-repeat-group-versions
noliveleger Oct 29, 2021
ad78349
Merge pull request #274 from kobotoolbox/272-inf-and-nan
noliveleger Oct 29, 2021
6e238ef
clean slate with fixtures
joshuaberetta Nov 12, 2021
7273fdd
wip AnalysisForm class and basic passing test
joshuaberetta Nov 12, 2021
1e7233b
handle translations and choices, get all tests to pass again
joshuaberetta Nov 12, 2021
7d9f1b8
fix failing fixture test
joshuaberetta Nov 12, 2021
f4d396f
add support for `anaylsis_type` and `settings` field attributes
joshuaberetta Nov 13, 2021
4ff7a3f
minor edit
joshuaberetta Nov 13, 2021
9a66171
wip support for repeat groups
joshuaberetta Nov 19, 2021
978c9f3
fix typo
joshuaberetta Nov 19, 2021
98535f4
restructure fixtures with kpi model changes
joshuaberetta Nov 30, 2021
1883774
update tests with new syntax
joshuaberetta Nov 30, 2021
07a3cbf
return analysis form field labels to lists
joshuaberetta Nov 30, 2021
af46ed5
wip get tests passing again
joshuaberetta Nov 30, 2021
6ca0127
use defaultdict to refactor
joshuaberetta Nov 30, 2021
b22bf79
refactor extracting supplemental value for export
joshuaberetta Nov 30, 2021
3a7429f
handle labels better and refactor AnalysisForm class
joshuaberetta Nov 30, 2021
5624352
refactor and type annotate AnalysisForm class methods
joshuaberetta Dec 3, 2021
a8f60da
remove `[Any]` from annotations
joshuaberetta Dec 7, 2021
33bfed5
add more type annotations
joshuaberetta Dec 7, 2021
2cefe71
fix failing tests with missing import
joshuaberetta Dec 7, 2021
431ee61
clean up tests, remove `load_analysis_form_json()` from `build_fixtur…
joshuaberetta Dec 7, 2021
71d4c64
update autoreport test
joshuaberetta Dec 7, 2021
eb483c0
support updated transcript structure, ammend tests, remove support fo…
joshuaberetta Jan 24, 2022
1cbd7a4
handle translations
joshuaberetta Jan 24, 2022
bec0601
wip integrating changes from kpi
joshuaberetta Feb 8, 2022
1e24c63
handle filtering exports and different language transcriptions for th…
joshuaberetta Feb 9, 2022
15dbb28
use the source label for prefix of transcript and translation field l…
joshuaberetta Feb 15, 2022
aee1e0d
apply requested changes to transcript export format
joshuaberetta Feb 16, 2022
2f290b1
update handling of transcript export formatting
joshuaberetta Feb 16, 2022
79c65fb
minor cleanup
joshuaberetta Feb 17, 2022
3cb1673
Merge branch 'master' into 277-additional-fields-exports
joshuaberetta Mar 22, 2022
484a0eb
black formatting
joshuaberetta Mar 22, 2022
e6c4abe
fix breaking additional fields tests
joshuaberetta Mar 22, 2022
0bcbdb3
pin pyxform version to 1.7.0
joshuaberetta Mar 22, 2022
a41861a
clean up, use constants
joshuaberetta Mar 22, 2022
7988071
Merge branch 'master' into 277-additional-fields-exports
joshuaberetta Apr 13, 2022
4f772e7
remove `include_analysis_fields` flag
joshuaberetta Apr 13, 2022
a48975c
remove pyxform version pin from setup.py
joshuaberetta Apr 13, 2022
53afa08
make requested changes
joshuaberetta Apr 13, 2022
289b523
use `filter_fields` for controlling export of additional analysis fields
joshuaberetta Apr 13, 2022
4ea0c16
mvp filtering for multi-language transcripts
joshuaberetta May 12, 2022
0569abe
change translations field name from `translation_<code>` to `translat…
joshuaberetta May 17, 2022
4da4ec9
remove duplicate inserting of analysis fields
joshuaberetta Jun 1, 2022
17d53b9
split out languages into individual fields in analysis_form_json for …
dorey Oct 3, 2022
ba1fd14
unique name and path for transcript languages in json fixture
dorey Oct 3, 2022
b5d18d6
update handling of transcript fields with new format and update fixtures
joshuaberetta Oct 4, 2022
b71c7c6
key in supplementalData is qpath
dorey Oct 10, 2022
95507a1
Merge branch 'master' into 277-additional-fields-exports
joshuaberetta Oct 11, 2022
9fcec12
Merge branch 'master' into 277-additional-fields-exports
joshuaberetta Oct 11, 2022
c87f93d
Merge branch '277-additional-fields-exports' into 277-additional-fiel…
joshuaberetta Oct 11, 2022
15b2d44
Update tests to match qpath in supplementalData
jnm Nov 11, 2022
62df5d9
Merge pull request #308 from kobotoolbox/277-additional-fields-export…
dorey Nov 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/formpack/reporting/export.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,17 @@ def __init__(
tag_cols_for_header = []
self.tag_cols_for_header = tag_cols_for_header

self.t_lang_codes = []
_filter_fields = []
for item in self.filter_fields:
item = re.sub(r'^_supplementalDetails/', '', item)
if item.split('/')[-1].startswith('transcript'):
self.t_lang_codes.append(item.split('_')[-1])
_filter_fields.append(f"{item.split('/')[0]}/transcript")
else:
_filter_fields.append(item)
jnm marked this conversation as resolved.
Show resolved Hide resolved
self.filter_fields = _filter_fields

# If some fields need to be arbitrarily copied, add them
# to the first section
if copy_fields:
Expand Down Expand Up @@ -268,6 +279,7 @@ def get_fields_labels_tags_for_all_versions(
hierarchy_in_labels=hierarchy_in_labels,
multiple_select=self.multiple_select,
include_media_url=self.include_media_url,
t_lang_codes=self.t_lang_codes,
)
)

Expand Down Expand Up @@ -297,6 +309,7 @@ def get_fields_labels_tags_for_all_versions(
value_names = field.get_value_names(
multiple_select=self.multiple_select,
include_media_url=self.include_media_url,
t_lang_codes=self.t_lang_codes,
)
name_lists.append(value_names)

Expand Down Expand Up @@ -487,6 +500,7 @@ def _get_value_from_entry(
xls_types_as_text=self.xls_types_as_text,
attachment=attachment,
include_media_url=self.include_media_url,
t_lang_codes=self.t_lang_codes,
)

# save fields value if they match parent mapping fields.
Expand Down
46 changes: 33 additions & 13 deletions src/formpack/schema/fields.py
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,7 @@ def get_labels(
group_sep='/',
hierarchy_in_labels=False,
multiple_select='both',
t_lang_codes=[],
*args,
**kwargs,
):
Expand All @@ -493,10 +494,17 @@ def get_labels(
if self._is_translation:
return [f'{source_label} - translation ({self.language})']
elif self._is_transcript:
return [
f'{source_label} - transcript ({code})'
for code in self.languages
]
if t_lang_codes:
return [
f'{source_label} - transcript ({code})'
for code in self.languages
if code in t_lang_codes
]
else:
return [
f'{source_label} - transcript ({code})'
for code in self.languages
]
return [self._get_label(*args)]

def get_stats(self, metrics, lang=UNSPECIFIED_TRANSLATION, limit=100):
Expand All @@ -514,12 +522,21 @@ def get_stats(self, metrics, lang=UNSPECIFIED_TRANSLATION, limit=100):

return stats

def get_value_names(self, multiple_select='both', *args, **kwargs):
def get_value_names(
self, multiple_select='both', t_lang_codes=[], *args, **kwargs
):
if self._is_transcript:
return [
f'{self.source_field.name} - transcript ({code})'
for code in self.languages
]
if t_lang_codes:
return [
f'{self.source_field.name} - transcript ({code})'
for code in self.languages
if code in t_lang_codes
]
else:
return [
f'{self.source_field.name} - transcript ({code})'
for code in self.languages
]
return super().get_value_names()

@property
Expand All @@ -538,6 +555,7 @@ def format(
hierarchy_in_labels=False,
multiple_select='both',
xls_types_as_text=True,
t_lang_codes=[],
*args,
**kwargs,
):
Expand All @@ -550,11 +568,13 @@ def format(
except KeyError:
val = ''
elif self._is_transcript:
cells = dict.fromkeys(self.get_value_names(), '')
cells = dict.fromkeys(
self.get_value_names(t_lang_codes=t_lang_codes), ''
)
if isinstance(val, dict):
cells[
f'{self.source_field.name} - transcript ({val["languageCode"]})'
] = val['value']
name = f'{self.source_field.name} - transcript ({val["languageCode"]})'
if name in cells:
cells[name] = val['value']
return cells

return {self.name: val}
Expand Down
22 changes: 12 additions & 10 deletions tests/test_additional_field_exports.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,11 @@ def test_additional_field_exports_without_labels():
'versions': 'v1',
'filter_fields': [
'record_a_note',
'record_a_note/transcript',
'record_a_note/translation_en',
'record_a_note/translation_es',
'record_a_note/acme_timestamp',
'_supplementalDetails/record_a_note/transcript_en',
'_supplementalDetails/record_a_note/transcript_es',
'_supplementalDetails/record_a_note/translation_en',
'_supplementalDetails/record_a_note/translation_es',
'_supplementalDetails/record_a_note/acme_timestamp',
],
}
export = pack.export(**options)
Expand Down Expand Up @@ -55,10 +56,11 @@ def test_additional_field_exports_with_labels():
'versions': 'v1',
'filter_fields': [
'record_a_note',
'record_a_note/transcript',
'record_a_note/translation_en',
'record_a_note/translation_es',
'record_a_note/acme_timestamp',
'_supplementalDetails/record_a_note/transcript_en',
'_supplementalDetails/record_a_note/transcript_es',
'_supplementalDetails/record_a_note/translation_en',
'_supplementalDetails/record_a_note/translation_es',
'_supplementalDetails/record_a_note/acme_timestamp',
],
'lang': 'English (en)',
}
Expand Down Expand Up @@ -424,8 +426,8 @@ def test_additional_field_exports_all_versions_exclude_fields():
'versions': pack.versions,
'filter_fields': [
'record_a_note',
'clerk_details/name_of_shop',
'clerk_details/name_of_clerk',
'_supplementalDetails/clerk_details/name_of_shop',
'_supplementalDetails/clerk_details/name_of_clerk',
],
}
export = pack.export(**options)
Expand Down