Update docs generation

[SchahinRohani]

Description

Update the docs generation by enhancing the md-to-mdx lib. Also add a polished version of the README.md

Preview Docs Link

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[aaronmondal]

I like the direction where this is going, but there are too many things happening at once here. Let's break this up into two parts:

• Your proposed refactor (which tbh seems somewhat unnecessary)
• The extension to the functionality, i.e. the tailwind logic.

I'd like to suggest to only focus on the functionality extension as the refactor doesn't seem necessary and adds review time/friction.

Reviewed 12 of 17 files at r1.
Reviewable status: 0 of 2 LGTMs obtained, and 12 of 17 files reviewed, and 3 discussions need to be resolved

---

docs/astro.config.ts line 13 at r1 (raw file):

import { rehypeMermaid } from "@beoe/rehype-mermaid"; // "rehype-mermaid";
import rehypeAutolinkHeadings from "rehype-autolink-headings";
// import { default as playformCompress } from "@playform/compress";

If this is not needed let'r remove this line entirely.

---

docs/package.json line 10 at r1 (raw file):

    "biome": "biome",
    "docs": "bun run docs.build && bun run docs.generate",
    "docs.build": "cd .. && unset TMPDIR TMP; bazelisk build nativelink-config:docs_json && cd docs && bun run src/utils/docs.build.ts",

Let's try to find more intuitive/descriptive names for this.

---

docs/src/utils/docs.build.ts line 1 at r1 (raw file):

import { generateDocs } from "./metaphase_aot";

Splitting this out seems unnecessary. I'm fine with renaming, but let's keep the logic in a single file.

[SchahinRohani]

Reverted the refactor completely as per your feedback. That said, I found it more productive to separate the raw file processing from the AST transformations, but I can definitely work within a more minimal file structure if that’s the preferred approach. "Unnecessary" is always in the eye of the beholder, right? 😄

Reviewable status: 0 of 2 LGTMs obtained, and 8 of 19 files reviewed, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Docs Deployment / macos-14, Docs Deployment / ubuntu-24.04, Installation / macos-13, Installation / macos-14, Local / ubuntu-22.04, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 3 discussions need to be resolved

---

docs/astro.config.ts line 13 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

If this is not needed let'r remove this line entirely.

Done.

---

docs/package.json line 10 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Let's try to find more intuitive/descriptive names for this.

How about bun run generate.docs and bun run build.docs?

---

docs/src/utils/docs.build.ts line 1 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Splitting this out seems unnecessary. I'm fine with renaming, but let's keep the logic in a single file.

Reverted back to "metaphase_aot.ts".

[aaronmondal]

Cool, this is a much nicer diff! Just some more adjustments to reduce diff size, then we should be fine. Also, links in the preview seem to have lost their underlines, which is an accessibility issue. Let's bring those underlines back.

Reviewed 11 of 13 files at r2, all commit messages.
Reviewable status: 0 of 2 LGTMs obtained, and all files reviewed, and 10 discussions need to be resolved

---

docs/package.json line 10 at r1 (raw file):

Previously, SchahinRohani (Schahin) wrote…

How about bun run generate.docs and bun run build.docs?

Hmm maybe something like docs.generate-config and docs.generate-mdx? In any case, I think it's fine if we change this later.

---

docs/src/utils/md_to_mdx.ts line 7 at r2 (raw file):

import remarkStringify from "remark-stringify";

export function generateFrontMatter(

Let's move this back to it's old place to reduce the diff size.

---

docs/src/utils/md_to_mdx.ts line 27 at r2 (raw file):

};

export async function transformMarkdownToMdx(

Move this back to it's old place to reduce diff size

---

docs/src/utils/md_to_mdx.ts line 44 at r2 (raw file):

    : [...content];

  // // Apply transformations for target IDs

nit: double comment

---

docs/src/utils/md_to_mdx.ts line 89 at r2 (raw file):

}

function removeValeAndGitCliffComments(markdown: string): string {

Move this back to it's old place to reduce diff size

---

docs/src/utils/md_to_mdx.ts line 95 at r2 (raw file):

}

export function parseMarkdown(markdown: string): Root {

Move this back to it's old place to reduce diff size

---

docs/src/utils/md_to_mdx.ts line 99 at r2 (raw file):

}

function processLines(lines: string[]) {

Move this back to it's old place to reduce diff size

---

docs/src/utils/md_to_mdx.ts line 183 at r2 (raw file):

}

function isCodeBlock(line: string): boolean {

Move this back to it's old place to reduce diff size

---

docs/src/utils/md_to_mdx.ts line 204 at r2 (raw file):

// AST
import type {

put imports to the top of the file

---

docs/src/utils/md_to_mdx.ts line 217 at r2 (raw file):

import { visit } from "unist-util-visit";

// import { findNodesById, logFirstNNodes } from "./debug";

nit: remove comment

---

docs/src/utils/md_to_mdx.ts line 467 at r2 (raw file):

      // Remove the entire <picture> tag and replace it with the two <img> tags
      .replace(
        /<picture>[\s\S]*?<source\s+media="[^"]*dark[^"]*"\s+srcset="[^"]*"[^>]*>\s*<source\s+media="[^"]*light[^"]*"\s+srcset="[^"]*"[^>]*>\s*<img\s+[^>]*src="[^"]*"[^>]*>\s*<\/picture>/g,

Since we're only looking for the picture tag, can't we simplify this regex to only look for the start and end picture tags?

[SchahinRohani]

Underlines are back, applied it with tailwind.

Reviewable status: 0 of 2 LGTMs obtained, and 17 of 19 files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Docs Deployment / macos-14, Docs Deployment / ubuntu-24.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, NativeLink.com Cloud / Remote Cache (Legacy Dockerfile Test), Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 8 discussions need to be resolved

---

docs/src/utils/md_to_mdx.ts line 7 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Let's move this back to it's old place to reduce the diff size.

Done.

---

docs/src/utils/md_to_mdx.ts line 27 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Move this back to it's old place to reduce diff size

Done.

---

docs/src/utils/md_to_mdx.ts line 89 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Move this back to it's old place to reduce diff size

Done.

---

docs/src/utils/md_to_mdx.ts line 95 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Move this back to it's old place to reduce diff size

Done.

---

docs/src/utils/md_to_mdx.ts line 99 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Move this back to it's old place to reduce diff size

Done.

---

docs/src/utils/md_to_mdx.ts line 183 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Move this back to it's old place to reduce diff size

Done.

---

docs/src/utils/md_to_mdx.ts line 204 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

put imports to the top of the file

Done.

---

docs/src/utils/md_to_mdx.ts line 467 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Since we're only looking for the picture tag, can't we simplify this regex to only look for the start and end picture tags?

For the Logo we are looking for an explicit html picture node in the ast , it could break different picture nodes if the regex match them.

[aaronmondal]

Has perfect lighthouse scores on desktop now. Nice work!

Reviewed 2 of 3 files at r3, all commit messages.
Reviewable status: 1 of 2 LGTMs obtained, and all files reviewed, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Docs Deployment / macos-14, Installation / macos-13, Local / ubuntu-22.04, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04 / stable, windows-2022 / stable, and 1 discussions need to be resolved

---

docs/src/styles/tailwind.css line 7 at r3 (raw file):

}

a.astro-3ii7xxms,

nit: This looks like it's some generated component tag. Will the hashes persist? If this is the only way to get the links, let's leave a comment explaining how these hashes came to be and what to look out for if/when they change.

[blakehatch]

Very ncie QOL changes and also got the same lighthouse scores as Aaron!

Reviewed 1 of 1 files at r5, all commit messages.
Reviewable status: 2 of 2 LGTMs obtained, and all files reviewed, and 1 discussions need to be resolved

[aaronmondal]

Reviewed all commit messages.
Reviewable status: 2 of 2 LGTMs obtained, and all files reviewed, and 1 discussions need to be resolved

[aaronmondal]

Reviewable status: complete! 2 of 2 LGTMs obtained, and all files reviewed