Add CLI interface #18

ghing · 2017-03-06T22:50:37Z

While the original use case didn't want to consider how to download and unpack the XML results, I think it will make it a lot easier for volunteers to deal with counties or states that use Clarity systems for their results if they could just run a command to download the results as CSV and then use simpler scripts or manually update the data to do any post-processing.

@chagan and I started working on this at the #NICAR17 hackathon.

Tasks

Figure out which values from the Clarity detail XML to include in CSV and how to present them (e.g. # of precincts reporting)
Test! Test! Test! At the very least we should have an integration test for both a state and county XML file.
Write utility function for parsing jursidiction levels from Clarity URLS (@chagan)
Write utility function for retrieving lowest jurisdiction level from Clarity URL (@chagan)
Figure out why there are rows with no vote counts when calling clarify results (@ghing)
Update CLI to pull levels from url

The text was updated successfully, but these errors were encountered:

ghing · 2017-03-06T22:57:06Z

@dwillis, I made this ticket to have a place to track the work that Chris and I have done so far and to reference once I send a pull request.

Should the output columns match, as much as possible, the ones listed in Common Fields? Or the columns listed in the data entry instructions?

Another option that Chris and I discussed would to be to have the output match that from elex so users of that tool could easily pull results from Clarity into a system designed around using elex to pull in AP results. However, our thoughts at the time were that this might be a later feature to add, if someone requests it, and instead prioritize whatever's easiest for Open Elections.

dwillis · 2017-03-06T23:30:27Z

@ghing my thoughts here are that the output should probably be closer to the columns in the data entry instructions, with the caveat that if there are additional columns they should be provided as well. I like the elex integration idea, but agree that it's a future thing.

ghing · 2017-03-08T16:06:18Z

Note to self: Originally @chagan and I wanted to keep this as simple as possible and envisioned having one command to rule them all that would download all results for all jurisdictions. However, this is both slow and difficult to integration test (because I'm finding only one jurisdiction has issues while the others are fine). I also wonder if grabbing anything is the most common use case. On election night, folks probably just want racewide and for people doing further analysis, running a command for each county, or scripting that separately isn't too onerous. I'm going to try making separate jurisdictions and results subcommands. Having subcommands seems worth it anyway to make things more extensible.

ghing · 2017-04-10T16:16:18Z

Just an update on this. I wrote almost all of the code to do this at the beginning of the month, but in testing for the use case, I ran into some errors for some of the counties. I'm slammed with other projects, at least for this week, but I'm hoping to circle back around to this at the end of the month or beginning of May.

@chagan

Start adding a CLI interface to this package that will fetch a list of jurisdictions from a parent jurisdiction and download and parse results as CSV. This is a checkpoint commit so I can share the code with @chagan. It still needs a lot of work. Addresses openelections#18 openelections#18

ghing · 2017-04-20T03:02:24Z

A little background on the empty results:

clarify results http://results.enr.clarityelections.com/AR/Yell/63988/184121/Web01/en/summary.html

produces these results:

state,jurisdiction,office,candidate,party,votes
,,U.S. President & Vice President,,,
,Bluffton,U.S. President & Vice President,,,
,Briggsville,U.S. President & Vice President,,,
,Centerville,U.S. President & Vice President,,,
,Compton,U.S. President & Vice President,,,
,Crawford,U.S. President & Vice President,,,
,Danville,U.S. President & Vice President,,,
,Dardanelle Outside,U.S. President & Vice President,,,
,Dardanelle Ward 1,U.S. President & Vice President,,,
,Dardanelle Ward 2,U.S. President & Vice President,,,
,Dardanelle Ward 3 Other,U.S. President & Vice President,,,
,Dardanelle Ward 3 JP 10,U.S. President & Vice President,,,
,Dutch Creek,U.S. President & Vice President,,,
,Ferguson,U.S. President & Vice President,,,
,Galla Rock,U.S. President & Vice President,,,
,Gilkey,U.S. President & Vice President,,,
,Gravelly,U.S. President & Vice President,,,
,Herring,U.S. President & Vice President,,,
,Lamar,U.S. President & Vice President,,,
,Magazine 1,U.S. President & Vice President,,,
,Magazine 2,U.S. President & Vice President,,,
,Mason,U.S. President & Vice President,,,
,Mountain,U.S. President & Vice President,,,
,Prairie,U.S. President & Vice President,,,
,Richland,U.S. President & Vice President,,,
,Waveland,U.S. President & Vice President,,,
,Riley,U.S. President & Vice President,,,
,Rover,U.S. President & Vice President,,,
,Ward,U.S. President & Vice President,,,
,Ions Creek,U.S. President & Vice President,,,

I need to figure out why there are result objects that don't have any votes.

ghing · 2017-07-02T15:30:09Z

I added some logging to the Parser class and realized that my code in clarify.cli.results.result_as_dict was pulling the votes from Choice.total_votes which will be the total votes for the candidate (choice) for a particular type of vote and not the per-precinct votes.

Here's the code that doesn't work:

def result_as_dict(result, **addl_cols):
    """Return a result as a dictionary suitable for serialization"""
    result_dict = dict(**addl_cols)
    result_dict['office'] = result.contest.text
    # Cols:
    #  county, precinct, office, district, party, candidate, votes, winner (if
    #  it's in the data).

    if result.jurisdiction is not None:
        result_dict['jurisdiction'] = result.jurisdiction.name

    if result.choice is not None:
        result_dict['candidate'] = result.choice.text
        result_dict['party'] = result.choice.party
        result_dict['votes'] = result.choice.total_votes

    return result_dict

I'm going to get started on reworking this.

The data doesn't contain information about the jurisdiction for the results, that is whether the URL represents state results or county results. However, we've found that result URLs are either for a state or a county and that they have a distinct format. Based on this, we can detect level from the URL so the user doesn't have to specify it as an explicit command line argument. Add a utility function to detect jurisdiction levels from a results URL and use this function in the CLI commands. Addresses openelections#18 openelections#18

Add preliminary documentation for the CLI in the README. This is pretty sparse. It's mostly helpful for me to remember the CLI interface as I'm working on this project in spurts. Addresses openelections#18 openelections#18

The logic that prepared a `Result` object for CSV output was incorrectly setting the votes to the total votes for each cadidate instead of the votes for each candidate in each jurisdiction. Include the vote type in the CSV results. Reorder CSV columns to match examples in http://docs.openelections.net/data-entry/ Add logging in the `Parser` class and an option in the `results` command to make it easier to identify problems with parsing or application logic. Addresses openelections#18 openelections#18

ghing · 2017-07-02T16:22:51Z

While figuring out the problem I reported in this comment, I realized that there are a bunch of fields in the XML that we could surface in the CSV.

Jurisdiction:

total voters
ballots cast
voter turnout
percent reporting
precincts participating
precincts reported
precincts reporting percent

Contest:

# of candidates you can vote for
is the contest a ballot question
percent reporting
precincts participating
precincts reported
# of counties reporting

We should figure out which of these should be included in the output CSV and how they should be presented.

My intuition is that we should:

Include everything
Repeat values for each result that's for a given contest, jurisdiction
Leave values recorded at the county level (e.g. precincts reporting) blank for precinct results

@dwillis, do you have thoughts or questions?

dwillis · 2017-07-02T20:02:19Z

Let's include everything we can.

…

On Jul 2, 2017, at 12:22 PM, Geoffrey Hing ***@***.***> wrote: While figuring out the problem I reported in this comment, I realized that there are a bunch of fields in the XML that we could surface in the CSV. Jurisdiction: total voters ballots cast voter turnout percent reporting precincts participating precincts reported precincts reporting percent Contest: of candidates you can vote for is the contest a ballot question percent reporting precincts participating precincts reported of counties reporting We should figure out which of these should be included in the output CSV and how they should be presented. My intuition is that we should: Include everything Repeat values for each result that's for a given contest, jurisdiction Leave values recorded at the county level (e.g. precincts reporting) blank for precinct results @dwillis, do you have thoughts or questions? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

ghing · 2018-09-06T17:29:49Z

@GPHemsley I am so excited to see you giving some love to this project. I have some commits tucked away from when I was working on adding a CLI to this project. I wanted to check in and see if this was something you were also poking at. I'll try to merge in your improvements and just take stock of where I was at with adding a CLI.

GPHemsley · 2018-09-07T01:49:45Z

@ghing I was working my way towards this ticket, per @dwillis, but I don't have to if you have a WIP.

ghing · 2018-09-07T05:49:56Z

Let me merge your changes into my fork and see how far I got. I'm happy to hand this over to you, but just wanted to give you a heads up if it's a useful starting point.

…

On Thu, Sep 6, 2018, 8:49 PM Gordon P. Hemsley ***@***.***> wrote: @ghing <https://github.com/ghing> I was working my way towards this ticket, per @dwillis <https://github.com/dwillis>, but I don't have to if you have a WIP. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGvVLA_21ZkH_PEu8vP8Utra4CAQ03Lks5uYdC6gaJpZM4MUzfm> .

ghing · 2018-09-09T01:30:13Z

@GPHemsley I merged master into my feature branch over at https://github.com/ghing/clarify/tree/add-cli. Feel free to use that as a starting point for further work on this feature, and let me know if you have any questions. I'm happy to pass this off to you, but didn't want you to replicate effort.

nealmcb · 2021-11-03T03:09:09Z

Thanks for all your work folks. I'm wondering what the status of this CLI work is. It sounds like some things work at least?

For context, years ago I did my own interface for downloading Clarity ENR and saving snapshots in a database:
https://github.com/nealmcb/snapshot-election-results

I just started porting it to python3 in a new branch, then decided to look at how this project was progressing.

GPHemsley added the enhancement label Sep 8, 2018

GPHemsley added this to the 0.5.0 milestone Sep 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CLI interface #18

Add CLI interface #18

ghing commented Mar 6, 2017 •

edited

Loading

ghing commented Mar 6, 2017

dwillis commented Mar 6, 2017

ghing commented Mar 8, 2017

ghing commented Apr 10, 2017

ghing commented Apr 20, 2017

ghing commented Jul 2, 2017

ghing commented Jul 2, 2017 •

edited

Loading

dwillis commented Jul 2, 2017 via email

ghing commented Sep 6, 2018

GPHemsley commented Sep 7, 2018

ghing commented Sep 7, 2018 via email

ghing commented Sep 9, 2018

nealmcb commented Nov 3, 2021

Add CLI interface #18

Add CLI interface #18

Comments

ghing commented Mar 6, 2017 • edited Loading

ghing commented Mar 6, 2017

dwillis commented Mar 6, 2017

ghing commented Mar 8, 2017

ghing commented Apr 10, 2017

ghing commented Apr 20, 2017

ghing commented Jul 2, 2017

ghing commented Jul 2, 2017 • edited Loading

dwillis commented Jul 2, 2017 via email

ghing commented Sep 6, 2018

GPHemsley commented Sep 7, 2018

ghing commented Sep 7, 2018 via email

ghing commented Sep 9, 2018

nealmcb commented Nov 3, 2021

ghing commented Mar 6, 2017 •

edited

Loading

ghing commented Jul 2, 2017 •

edited

Loading