[WIP] Output eval logging

[bmosaicml]

Evaluating with: output-logger-test-dsQsKa

Confirmed uri s3://mosaicml-internal-checkpoints-test/test_icl_output_logger_7b contains all the data. Samples:

context_tok	continuation_tok_target	continuation_tok_pred	correct	benchmark	question_tok	correct_choice	selected_choice
b'AMERICAN HISTORY: Name that completed an 1856 Republican slogan, Free soil, free speech, free men and...\\nAnswer: Fremont\\nAMERICAN HISTORY: After this waterway opened in 1825, freight rates between Buffalo & NYC were cut by more than 90%\\nAnswer: Erie Canal\\nAMERICAN HISTORY: In 1949 Henry H. Arnold became the first general of this branch of the armed forces\\nAnswer: Air Force\\nAMERICAN HISTORY: After his 1859 hanging, Civil War troops sang of this abolitionists body mouldering in the grave\\nAnswer: John Brown\\nAMERICAN HISTORY: On April 24, 1980 an attempt to rescue 53 hostages in this capital city was called off\\nAnswer: Tehran\\nAMERICAN HISTORY: Witchcraft trials held in this town in 1692 led to the hangings of 19 people\\nAnswer: Salem\\nAMERICAN HISTORY: On March 10, 1977, this Cesar Chavez labor group signed an agreement with the Teamsters union\\nAnswer: United Farm Workers\\nAMERICAN HISTORY: On April 2, 1917 President Wilson told Congress, The world must be made safe for this\\nAnswer: democracy\\nAMERICAN HISTORY: He shook up society in the 1960s & later published Revolution for the Hell of It & Steal This Book\\nAnswer: Abbie Hoffman\\nAMERICAN HISTORY: In searching for the 7 Cities of Cibola, he & his men traveled as far north as present-day Lyons, Kansas\\nAnswer: Francisco Vazquez de Coronado\\nAMERICAN HISTORY: On May 29, 1765 Patrick Henrys Stamp Act protest was interrupted with this one word\\nAnswer:'	b' Treason!'	b' Noason\\n'	False	jeopardy/10-shot/american_history_InContextLearningLMAccuracy			
b'AMERICAN HISTORY: Hostilities in the Spanish-American War ended when Manila surrendered to General Merritt & this admiral\\nAnswer: George Dewey\\nAMERICAN HISTORY: WWI sales of Liberty these, whose posters showed the statue, raised more than $15 billion\\nAnswer: bonds\\nAMERICAN HISTORY: In 1949 the U.S. joined this Western defense pact; our 1st ever peacetime alliance\\nAnswer: NATO\\nAMERICAN HISTORY: In 1843 Daniel Webster resigned as this presidents Secretary of State\\nAnswer: John Tyler\\nAMERICAN HISTORY: He was promoted to captain for burning the captured U.S. frigate Philadelphia in Tripoli Harbor\\nAnswer: Stephen Decatur\\nAMERICAN HISTORY: In 1798 Congress passed this collection of bills to control domestic dissent & conspiracy against the federal govt.\\nAnswer: the Alien & Sedition Acts\\nAMERICAN HISTORY: The British party of this name aimed to limit the kings powers; the U.S. party hoped to limit King Andrew Jackson\\nAnswer: the Whigs\\nAMERICAN HISTORY: He shook up society in the 1960s & later published Revolution for the Hell of It & Steal This Book\\nAnswer: Abbie Hoffman\\nAMERICAN HISTORY: On January 21, 1977 this president pardoned several thousand Vietnam War draft evaders\\nAnswer: Jimmy Carter\\nAMERICAN HISTORY: In May of 1780, Gen. Benjamin Lincoln was forced to surrender this South Carolina port to the British\\nAnswer: Charleston\\nAMERICAN HISTORY: This political party founded around 1789 stood for a strong central government\\nAnswer:'	b' the Federalists'	b' Federal Federalists'	False	jeopardy/10-shot/american_history_InContextLearningLMAccuracy

[irenedea]

@bmosaicml btw, you can use github draft PRs to signify that your PR is wip

[dakinggg]

Closing, as @maxisawesome is doing this.