Find missing expungability data and join to existing charges data on github

[emilymo008]

Must be done before answering Sana's questions in #152

[laurafeeney]

My interpretation of the statutes (https://malegislature.gov/Laws/SessionLaws/Acts/2018/Chapter69, section 100J-K primarily) is that the default status is for offenses to be eligible for expungement unless explicitly listed in section 100J as ineligible. We may want to pare this sheet down to offenses we are not yet sure about from Sana or other trusted sources, fill in missing information with the information on eligibility, and highlight cases we are most unsure about. For example, I would feel confident that marine fishing violations don't meet any of the exclusion criteria for expungable offenses. We may want to make a (short) list of items to run by the partners.
Summary and documentation here: https://docs.google.com/document/d/18HP7UHs9bj7g6oDuiMx76ikEBYkC21BHiYArosmjXzA/edit?usp=sharing
this needs to be added to the Clean Slate Drive or perhaps a wiki in the github, but I'm not sure where

[emilymo008]

I marked all of the charges from chapter 265 as eligible for expungement since they did not match any of the ineligible parts of chapter 265 and none were felonies.

[jeremylang]

@laurafeeney will:

• 1. learn more about how it works
• 2. Take guesses at empty spots
• 3. Create easy to answer questions for Sana

[laurafeeney]

• I updated the google doc about determining offense-level eligibility. I added in summaries, annotations, and definitions for the sub-parts of 100J that define a series of other offenses which are not eligible for expungement.

• I reviewed the missing-expungability 7-21 file. I cross-referenced with "Master Crime List offense with Expunge categories.xlsx", the initial list from Sana (I believe). Some were already marked as Y/N, and just did not merge correctly. Others were very very close. In column J, I added the code number form that excel sheet for those relevant.

• For offenses that did not appear in the Excel Master Crime list, and did not seem to relate to any of the offenses in 100J, I marked them as eligible for expungement. If they were not eligible, I listed the portion of 100J that excludes the offense from being eligible for expungement, in column K.

• I marked y/n to mark under a column called "questionable" where I was more/less sure of my decision.

• There are many offenses that are in violation of the Code of Massachusetts Regulations, which seem a distinct kind of offense. I'm not sure whether these should be in our lists at all, they seem more like citations.

• I drafted questions or requests for confirmation for the decisions I made about eligibility for expungement to the bottom of the google doc.

• Lastly, i haven't used git / github in 6 years, and never with a shared setup. I haven't found how to push a csv file, since it's in the gitignore, and I can't directly upload from the website. I'll post on slack and see what help I can get!

[jeremylang]

Next steps:

• Update the crime list data with the information we learned from Sana tonight
• Waiting to hear back from Sana about the violations we should do with the violations (if it's safe to ignore them)

[jeremylang]

See thread on https://cfb-public.slack.com/archives/CPP9PAWH3/p1597192452010600 for ongoing conversation related to acquiring parts of the original Suffok data

[laurafeeney]

We updated the crime list in part manually, and in part through the R files in
notebooks. This gets merged back with the indvl level data in MA_Data-2_MergeCharges_alt.ipynb. A bit of a non-ideal setup, but works for now.

[laurafeeney]

issue #171 resolved whether we should include the 'violations', ie, CMR offenses. We should drop them from the dataset entirely.
we made the updates from the conversation with Sana on Aug 4. When we got data from Middlesex, we had additional need to cross-check the list of unmatched offenses with teh master crime list and with the original statute of non-eligible offenses, c276 s100J. These updates are in the data files: 8/02 updates, 7/21 updates, unmatched_middlesex_clean (still need to commit/update, will go in raw folder).

Data are joined in: Middlesex: Middlesex_Clean and Suffolk & Northwestern: MA_Data-2_MergeCharges_alt