alttexter-ghclient DEMO

[jonathanalgar]

Let's say we want to bootstrap a docs repo. We have five shiny new docs to start with (1x md, 1x mdx, 3x ipynb borrowed from langchain-ai/langchain for our demo purposes).

All the docs have images, but four are missing something important: alt text and title attributes for those images. (To learn why that's important, see here).

Let's commit and open the PR..

[github-actions]

Please check the LLM generated alt-text and title attributes in this file as they may contain inaccuracies. Explore how the LLM generated them.

[github-actions]

Please check the LLM generated alt-text and title attributes in this file as they may contain inaccuracies. Explore how the LLM generated them.

[github-actions]

Please check the LLM generated alt-text and title attributes in this file as they may contain inaccuracies. Explore how the LLM generated them.

[github-actions]

Please check the LLM generated alt-text and title attributes in this file as they may contain inaccuracies. Explore how the LLM generated them.

[jonathanalgar]

Using Chrome Screen Reader:

https://www.loom.com/share/1c8328607a904ea3a3bb4509d054a4a0?sid=bbf33acb-7df2-4ac1-a0f2-f33dac72fac3

[jonathanalgar]

The alttexter-ghclient doesn't like images without alt text and title attributes. It detects images defined in markdown formatted docs that were added or modified in a PR without alt text and/or title attributes and bundles them off, complete with the doc's text, in a request to the alttexter service to orchestrate generation by the language model.

So after the above commit, four docs were bundled off, one—the only doc where the images had complete alt text and title attributes—wasn't. The principle is that if someone has taken the time to write alt texts and titles for the images in a doc, they won't get overwritten.

It only bundles images of supported type (this currently corresponds to the supported types of gpt-4-vision-preview). So here it skipped the svg badges in one of the docs, for example.

The action handles both images defined locally and images defined by URL (for example).

All docs are handled fully asynchronously, so everything happens quite briskly (but not too briskly to avoid hitting token rate limits); the action took only 49 seconds to run. Granted this is primarily thanks to the model's speed!

The (optionally) returned public LangSmith trace URLs in the review comments are a powerful way to get folks with domain expertise—writing for accessibility in this case—but not necessarily Python expertise interested in improving the prompt. It means someone can click the link, open the model call in the LangSmith Playground and then tweak the prompt to see how that steers the output:

Such interaction encourages feedback in the comments both about the specific output with the current prompt and on continuous improvement of the prompt itself.

[jonathanalgar]

Examples of batch updates in real docs repos:

• OutSystems/docs-odc@acf3456
• Batch update of alt text and title attributes for images in md/mdx files across repo langchain-ai/langchain#15357