Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Summarize actions to sentence + icon #741

Open
abrichr opened this issue Jun 12, 2024 · 8 comments
Open

Summarize actions to sentence + icon #741

abrichr opened this issue Jun 12, 2024 · 8 comments
Assignees
@Animesh404
Labels
$ bounty $ Please suggest a price range 🙏 💎 Bounty good first issue Good for newcomers help wanted Extra attention is needed

Comments

@abrichr
Copy link
Member

abrichr commented Jun 12, 2024

Feature request

When visualizing recordings, we would like to collapse each top level action into a one to two sentence description in a systematized way, ideally with an icon (eg prompt for a segment).

Motivation

Understanding recordings, humans and models.
@abrichr abrichr added good first issue Good for newcomers help wanted Extra attention is needed $ bounty $ Please suggest a price range 🙏 labels Jun 12, 2024
@abrichr
Copy link
Member Author

abrichr commented Jun 16, 2024

/bounty $100

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Jun 16, 2024 •
edited
Loading

💎 $100 bounty • OpenAdaptAI

Steps to solve:

  1. Start working: Comment /attempt #741 with your implementation plan
  2. Submit work: Create a pull request including /claim #741 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to OpenAdaptAI/OpenAdapt!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🔴 @IvanLauLinTiong Jun 16, 2024, 6:46:11 PM WIP
🟢 @onyedikachi-david Jul 5, 2024, 7:19:13 PM WIP
🟢 @vishwamartur Nov 17, 2024, 3:36:41 PM #922

@IvanLauLinTiong
Copy link

IvanLauLinTiong commented Jun 16, 2024 •
edited by algora-pbc bot
Loading

/attempt #741

Options

@abrichr abrichr changed the title Platform: Summarize actions to 1 sentence + icon Summarize actions to 1 sentence + icon Jun 16, 2024
@abrichr abrichr changed the title Summarize actions to 1 sentence + icon Summarize actions to sentence + icon Jun 17, 2024
@shashank40
Copy link
Contributor

@abrichr can you please explain the issue a bit more in detail?

@abrichr
Copy link
Member Author

abrichr commented Jun 18, 2024

@shashank40 from the README, here is a screenshot of the dashboard:

image

Here is a screenshot of Microsoft's recent updates to PowerAutomate:

image

Notice the one-sentence action summaries. We would like to implement the same type of thing, i.e. convert an action dict, window dict, and screenshot into a sentence and an icon. You can use the prompt adapter for this, e.g.

summary = prompt_adapter.prompt("Please provide a one sentence summary of the following contents: ...")

I expect the prompt should live in a .j2 file. You will need another prompt to generate an icon.

@onyedikachi-david
Copy link

onyedikachi-david commented Jul 5, 2024 •
edited by algora-pbc bot
Loading

/attempt #741

Algora profile Completed bounties Tech Active attempts Options
@onyedikachi-david 2 bounties from 1 project
JavaScript, Shell
Cancel attempt

@vishwamartur
Copy link

vishwamartur commented Nov 17, 2024 •
edited by algora-pbc bot
Loading

/attempt #741

Options

vishwamartur added a commit to vishwamartur/OpenAdapt that referenced this issue Nov 17, 2024
@vishwamartur
Add summary and icon to action events
c3c1a4a 
Related to OpenAdaptAI#741

Add functionality to summarize actions and generate icons for recordings.

* **openadapt/adapters/prompt.py**
  - Add `generate_summary_and_icon` function to generate a summary and icon for a given action using the `prompt` function.
* **openadapt/app/dashboard/api/recordings.py**
  - Import `generate_summary_and_icon` function.
  - Update `get_recording_detail` function to include summaries and icons for action events.
* **openadapt/app/dashboard/app/recordings/detail/page.tsx**
  - Update `Recording` component to display summaries and icons for action events.
@vishwamartur vishwamartur mentioned this issue Nov 17, 2024
Open
@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Nov 17, 2024

💡 @vishwamartur submitted a pull request that claims the bounty. You can visit your bounty board to reward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Animesh404 Animesh404

Labels
$ bounty $ Please suggest a price range 🙏 💎 Bounty good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@abrichr @IvanLauLinTiong @shashank40 @onyedikachi-david @Animesh404 @vishwamartur