Skip to content

Issues: UKGovernmentBEIS/inspect_ai

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

28 Open 204 Closed
28 Open 204 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

eval_set only retries "cancelled" or "errored" logs
#913 opened Nov 28, 2024 by max-kaufmann
7
support fractional unicode code points (e.g. ½, ¾) in numeric normalisation.
#897 opened Nov 25, 2024 by jjallaire
Add more documentation on CLI
#868 opened Nov 20, 2024 by mrahtz
1
[Feature Request] Add user_message solver
#864 opened Nov 19, 2024 by kaifronsdal
1
Inspect sometimes hangs on extremely large runs bug Something isn't working
#849 opened Nov 15, 2024 by MSchmatzAISI
@MSchmatzAISI
Multi-processing w/ s3 logs, unless you use spawn to launch your subprocesses
#840 opened Nov 13, 2024 by max-kaufmann
Log a "Sample error detected" if a sample has an error, so that we can see it in the console when we are running an eval
#824 opened Nov 8, 2024 by max-kaufmann
For certain Model API calls, if they error we don't have the API call logged to the transcript.
#823 opened Nov 8, 2024 by max-kaufmann
1
A scorer to compute costs based on model usages (for eval model, grader model, etc.)?
#795 opened Nov 3, 2024 by grigory93
[Question] Custom metric type
#775 opened Oct 30, 2024 by us2547
2
@dragonstyle
6
Adding a scorer without rerunning existing scorers.
#770 opened Oct 27, 2024 by sohaibimran7
2
Failed epoch doesn't stop rest of task's epochs
#746 opened Oct 23, 2024 by craigwalton-dsit
Web browser struggles with large pages web browser
#740 opened Oct 21, 2024 by art-dsit
1
plot_results(): Are there any frameworks that allow summarising and visualising inspect logs?
#704 opened Oct 16, 2024 by sohaibimran7
2
support for web browser navigations that open new windows web browser
#681 opened Oct 10, 2024 by jjallaire-aisi
@MariaIzobava
Error running inspect score with a scoring function that uses a task-level sandbox
#679 opened Oct 10, 2024 by bienehito
[Feature Request] Support live statistics in progress bar
#674 opened Oct 9, 2024 by tw-aisi
1
[Feature request] command line flag to capture the values of locals in the printed traceback
#673 opened Oct 9, 2024 by JasonGross
1
.json file is missing all messages when an error is thrown
#672 opened Oct 9, 2024 by JasonGross
1
Web Browser tool cannot see checkbox state web browser
#636 opened Oct 3, 2024 by sdtblckgov
@MariaIzobava
Feature Request: Support Anthropic on Vertex models.
#574 opened Sep 30, 2024 by simonbutt
1
[feature request] Support the -F flag for inspect eval
#551 opened Sep 27, 2024 by tw-aisi
1
[feature request] support passing -H / --host to docker sandbox
#540 opened Sep 27, 2024 by JasonGross
1
[feature request] Support video content as input and video-LLM models
#508 opened Sep 25, 2024 by alexandraabbas
1
[Bug report] hf_dataset crashes when split is not given
#365 opened Sep 10, 2024 by hanbyul-kim
ProTip! Updated in the last three days: updated:>2024-12-01.