Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to pyxform 1.5.0 #699

Closed
wants to merge 17 commits into from
Closed

Upgrade to pyxform 1.5.0 #699

wants to merge 17 commits into from

Conversation

duvld
Copy link
Member

@duvld duvld commented Apr 7, 2021

Upgrade pyxform dependency to latest version and fix broken tests.

Last of broken tests:

================================================== short test summary info ===================================================
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_corrupted_submission
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_simple_yes_submission
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_start_time_boolean_properly_set
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_start_time_submissions
FAILED onadata/apps/main/tests/test_csv_export.py::TestExport::test_csv_export_output
FAILED onadata/apps/main/tests/test_past_bugs.py::TestInputs::test_uniqueness_of_group_names_enforced
FAILED onadata/apps/main/tests/test_process.py::TestProcess::test_xls_export_content
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_build_sections_from_survey
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_child_record_parent_table_is_updated_when_sheet_is_renamed
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_delimiter_replacement_works_for_generated_gps_fields
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_delimiter_replacement_works_generated_multi_select_fields
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_gps_fields_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_mongo_encoded_fields_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_multi_selects_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xls_export_generates_valid_sheet_names
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xlsx_export_respects_custom_field_delimiter
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xlsx_export_works
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_column_header_delimiter_export_option
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_csv_http_response
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_csv_without_na_values
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_split_select_multiple_export_option
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_column_indices_in_groups_within_repeats
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_columns_for_gps_within_groups
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_dataframe_export_to
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_export_with_df_size_limit
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_format_mongo_data_for_csv
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_split_select_multiples
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_columns
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_columns_for_gps_within_groups
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_groups_within_repeats
============================ 30 failed, 341 passed, 15 skipped, 47 warnings in 620.81s (0:10:20) =============================

76cfa07 fixes:

FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_corrupted_submission
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_simple_yes_submission
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_start_time_boolean_properly_set
FAILED onadata/apps/logger/tests/test_simple_submission.py::TestSimpleSubmission::test_start_time_submissions

Tests failed because pyxform now doesn't like questions having the same name as surveys. The survey before the changed one already has a question as a no spaces truncation of the survey title, so style is copied over

b391112 fixes:

FAILED onadata/apps/main/tests/test_csv_export.py::TestExport::test_csv_export_output

Test failed because the duplicate naming error message is changed as error is thrown for form-question names, as well as group names

fb3a9fd and e394b1a fixes:

FAILED onadata/apps/main/tests/test_csv_export.py::TestExport::test_csv_export_output
FAILED onadata/apps/main/tests/test_process.py::TestProcess::test_xls_export_content
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_build_sections_from_survey
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_delimiter_replacement_works_generated_multi_select_fields
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_multi_selects_works
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_column_header_delimiter_export_option
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_csv_http_response
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_csv_without_na_values
FAILED onadata/apps/viewer/tests/test_exports.py::TestExports::test_split_select_multiple_export_option
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_column_indices_in_groups_within_repeats
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_columns_for_gps_within_groups
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_dataframe_export_to
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_csv_export_with_df_size_limit
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_format_mongo_data_for_csv
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_split_select_multiples
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_columns
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_columns_for_gps_within_groups
FAILED onadata/apps/viewer/tests/test_pandas_mongo_bridge.py::TestPandasMongoBridge::test_xls_groups_within_repeats

Tests failed because XLSForm/pyxform#410 changed bind types of select* questions from select and select1 to string. It's likely intention of question.bind.get("type") == MULTIPLE_SELECT_BIND_TYPE is to make sure the type is select and not select1. select all that apply should just replace the logic

be4eb8a fixes:

FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_child_record_parent_table_is_updated_when_sheet_is_renamed
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_delimiter_replacement_works_for_generated_gps_fields
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_delimiter_replacement_works_generated_multi_select_fields
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_gps_fields_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_mongo_encoded_fields_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_generation_of_multi_selects_works
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xls_export_generates_valid_sheet_names
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xlsx_export_respects_custom_field_delimiter
FAILED onadata/apps/viewer/tests/test_export_builder.py::TestExportBuilder::test_to_xlsx_export_works

Tests failed because pyxform.builder.create_survey_from_xls now needs a parameter to specify the names of the surveys

duvld added 8 commits March 31, 2021 11:42
pyxform.errors.PyXFormError: The name "start_time" is the same as the
form name. Use a different section name (or change the form name in the
"name" column of the settings sheet).
replacing `select` and `select1` for `string` as types for select* questions
means we can just check for `question.type` and pyxform's constant
instead
@duvld duvld requested review from jnm and noliveleger April 7, 2021 19:09
@duvld duvld marked this pull request as ready for review April 7, 2021 19:09
@duvld duvld requested a review from joshuaberetta April 7, 2021 20:04
@joshuaberetta joshuaberetta added the dependencies Pull requests that update a dependency file label Apr 8, 2021
@joshuaberetta joshuaberetta linked an issue Apr 8, 2021 that may be closed by this pull request
onadata/apps/viewer/tests/test_export_builder.py Outdated Show resolved Hide resolved
onadata/apps/viewer/tests/test_export_builder.py Outdated Show resolved Hide resolved
onadata/libs/constants.py Outdated Show resolved Hide resolved
@joshuaberetta
Copy link
Member

Note to use requirements.in file and pip-compile rather than editing the *.txt files.

@duvld duvld changed the title Upgrade to pyxform 1.4.0 Upgrade to pyxform 1.5.0 May 17, 2021
jnm added a commit that referenced this pull request May 25, 2021
* Recognize select questions from the question elements themselves
    instead of checking the bind type; see XLSForm/pyxform#168
* Recognize new error message when a group name matches the form name
    See XLSForm/pyxform#510
* Set `allow_choice_duplicates` to `yes` in cascade test XLSForms
    See XLSForm/pyxform#23; XLSForm/pyxform#373 (comment)
* Update tests to use `default_name` instead of relying on the XLSForm file name
    See XLSForm/pyxform#130
* Update expected XML in tests to match new pyxform behavior:
    * `<model>` becomes `<model odk:xforms-version="1.0.0">`
        * See XLSForm/pyxform#393
    * The bind type for `select` and `select1` becomes `string`
        * See XLSForm/pyxform#168
    * `calculate="concat('uuid:', uuid())"` becomes `jr:preload="uid"`
        * See XLSForm/pyxform#94

This squashed set of changes includes work from #699.
Thanks to @duvld for that contribution.
@jnm jnm mentioned this pull request May 25, 2021
jnm added a commit that referenced this pull request May 25, 2021
* Recognize select questions from the question elements themselves
    instead of checking the bind type; see XLSForm/pyxform#168
* Recognize new error message when a group name matches the form name
    See XLSForm/pyxform#510
* Set `allow_choice_duplicates` to `yes` in cascade test XLSForms
    See XLSForm/pyxform#23; XLSForm/pyxform#373 (comment)
* Update tests to use `default_name` instead of relying on the XLSForm file name
    See XLSForm/pyxform#130
* Update expected XML in tests to match new pyxform behavior:
    * `<model>` becomes `<model odk:xforms-version="1.0.0">`
        * See XLSForm/pyxform#393
    * The bind type for `select` and `select1` becomes `string`
        * See XLSForm/pyxform#168
    * `calculate="concat('uuid:', uuid())"` becomes `jr:preload="uid"`
        * See XLSForm/pyxform#94

This squashed set of changes includes work from #699.
Thanks to @duvld for that contribution.
@jnm
Copy link
Member

jnm commented May 25, 2021

@noliveleger, we missed an important caveat on is_equal_xml()

you have no other constructs than text and elements to worry about

As a result, pretty significant differences in XML, like the example below, were sailing through unit tests completely undetected. I've now removed pyxform_version_agnostic() and is_equal_xml() entirely (as part of #719), so it's no longer an issue.

@@ -1,9 +1,9 @@
-(Pdb) print(pyxform_version_agnostic(content))
+(Pdb) print(pyxform_version_agnostic(form_xml % data))
 <?xml version="1.0" encoding="utf-8"?>
 <h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:odk="http://www.opendatakit.org/xforms" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <h:head>
     <h:title>transportation_2011_07_25</h:title>
-    <model odk:xforms-version="1.0.0">
+    <model>
       <instance>
         <transportation_2011_07_25 id="transportation_2011_07_25">
           <formhub>
@@ -47,19 +47,19 @@
           </meta>
         </transportation_2011_07_25>
       </instance>
-      <bind nodeset="/transportation_2011_07_25/transport/available_transportation_types_to_referral_facility" type="string"/>
+      <bind nodeset="/transportation_2011_07_25/transport/available_transportation_types_to_referral_facility" type="select"/>
       <bind nodeset="/transportation_2011_07_25/transport/available_transportation_types_to_referral_facility_other" relevant="selected(../available_transportation_types_to_referral_facility, 'other')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/ambulance/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'ambulance')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/bicycle/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'bicycle')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/boat_canoe/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'boat_canoe')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/bus/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'bus')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/donkey_mule_cart/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'donkey_mule_cart')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/keke_pepe/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'keke_pepe')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/lorry/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'lorry')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/motorbike/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'motorbike')" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/taxi/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'taxi')" type="string"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/ambulance/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'ambulance')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/bicycle/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'bicycle')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/boat_canoe/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'boat_canoe')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/bus/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'bus')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/donkey_mule_cart/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'donkey_mule_cart')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/keke_pepe/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'keke_pepe')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/lorry/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'lorry')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/motorbike/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'motorbike')" type="select1"/>
+      <bind nodeset="/transportation_2011_07_25/transport/loop_over_transport_types_frequency/taxi/frequency_to_referral_facility" relevant="selected( /transportation_2011_07_25/transport/available_transportation_types_to_referral_facility , 'taxi')" type="select1"/>
       <bind nodeset="/transportation_2011_07_25/meta/instanceID" readonly="true()" type="string"/>
-      <bind nodeset="/transportation_2011_07_25/formhub/uuid" type="string" calculate="'ad758136cb284523815120ba0df3bb78'"/>
+      <bind calculate="'ad758136cb284523815120ba0df3bb78'" nodeset="/transportation_2011_07_25/formhub/uuid" type="string"/>
     </model>
   </h:head>
   <h:body>

@jnm
Copy link
Member

jnm commented May 25, 2021

Closed by #719

@jnm jnm closed this May 25, 2021
@jnm jnm deleted the 604-pyxform-upgrade branch May 25, 2021 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to pyxform 1.3.4
3 participants