ai-scraping

Star

Here are 10 public repositories matching this topic...

mendableai / firecrawl

Star

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

markdown crawler data scraper ai html-to-markdown web-crawler scraping webscraping rag llm ai-scraping

Updated Nov 29, 2024
TypeScript

itsOwen / CyberScraper-2077

Sponsor

Star

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

scraper web-scraper openai webscraping gemini-api llm llm-scraper ai-scraping

Updated Nov 8, 2024
Python

spider-rs / spider

Star

A web crawler and scraper for Rust

rust crawler spider web-crawler web-scraper web-scraping indexer headless-chrome llm-crawler ai-scraping

Updated Dec 1, 2024
Rust

➖ Stripped down, stable version of firecrawl optimized for self-hosting and ease of contribution. Billing logic and AI features are completely removed. Crawl and convert any website into LLM-ready markdown.

search markdown crawler data scraper ai html-to-markdown web-crawler scraping embeddings webscraping rag llm ai-scraping

Updated Nov 22, 2024
TypeScript

mendableai / firecrawl-app-examples

Star

🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

markdown data ai examples html-to-markdown templates web-crawler scrapers rag llm ai-scraping

Updated Jun 22, 2024
TypeScript

ArchiveBox / abx-dl

Sponsor

Star

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...