Brave AI (Leo) test plan

Brave Leo AI Chat test plan (MVP)

Out-of-scope (for this round):

1. quality of suggested questions (the ones following Summarize this page)
2. quality of responses
3. quality of page-content summarizations
4. performance (specific metrics)

MVP Issue Lists:

https://github.com/orgs/brave/projects/67/views/2?filterQuery=MVP

QA will verify all QA/Yes issues within their respective milestone(s).

Shared Steps: <--- start here

1. install 1.60.x (or 1.61.x)
2. launch Brave
3. relaunch from the commandline using --env-leo=staging --env-ai-chat.bsg=dev --env-ai-chat-premium.bsg=dev (this will also pick up the Brave AI Chat Griffin seed)
4. click on Brave Leo in the sidebar -or-
5. type a search query/question for Leo into the URL bar and use the search terms - Ask Leo dropdown option

Sidebar integration

• expand/collapse via Brave Leo button
• remove/show Leo icon in the sidebar
• resize (text reflow)

Summarize this {page, video}

• Summarize this page
• Summarize this video
• Summarize this https://brave.com/static-assets/files/1.0-REVIEWERS-GUIDE-a5f1bca9-ea66-4445-806b-45e00038ff6c.pdf

Page/content types which should NOT have summarizations:

• internal chrome:// / brave:// URLs
• PDF rendered inline: https://brave.com/static-assets/files/1.0-REVIEWERS-GUIDE-a5f1bca9-ea66-4445-806b-45e00038ff6c.pdf
• https://account.bravesoftware.com and other basic-auth challenged URLs

Conversations

• Erase conversation history
  • UI changes
  • NO context changes

Contexts

• Summarize this page
• there should be a minimum of three (3) Suggested follow-ups questions
• negative cases/missing contexts

General chatbot/commands

• introduce yourself
• tell me about Zork I
• write me a Python program...
• translate this article
• solve a math problem
• summarize in brief
• summarize at length

Boundary/safety testing

• e.g. ask Leo to help gamble or evaluate "sensitive" content
• excessive safety: ask Leo about 2020 election results

Language models

• llama-2-13b-chat
• llama-2-70b-chat (Premium)
• claude-instant-v1 (Premium)

(large-language) Model-switching

• confirm model name & attribution
• confirm self-introduction/prompt
• confirm Summarize this page
• confirm 3 or more additional Suggested follow-ups questions

Error states - tested in https://github.com/brave/brave-browser/issues/31391 ✅

• network-disconnected (Wi-Fi/ethernet)
• server/endpoint unreachable (127.0.0.1 hostname)

Rate-limiting

• Retry
• Premium upsell
  • Go Premium button opens https://account.brave.com/?intent=checkout&product=leo in a new window

Accessibility

• tooltips (for controls)
• titletips (for cropped text)
• Keyboard navigation
  • tab to focus
  • spacebar to select
  • return to submit

Feedback - PENDING

• pending testing in https://github.com/brave/brave-browser/issues/32951 and https://github.com/brave/brave-browser/issues/32701 ⌛

  • Like answer
  • Dislike answer
    • Add feedback

account.bravesoftware.com

• Buy now
• Edit payment (handled by Stripe)
• Cancel plan
• Renew plan