Scrap agents

🔥 Search, scrape, and clean the web for AI agents.

Create agents that monitor and act on your behalf. Your agents are standing by!

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial…

Python scraper based on AI

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level…

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili…

An AI-powered research assistant that performs iterative, deep research on any topic by combining search…

Clone any website with one command using AI coding agents

The headless browser for AI agents and web scraping

Turn any webpage into structured data using LLMs

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM…

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as…

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in…

A community-driven way to read and chat with AI bots - powered by chatGPT.

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP…

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

Low latency web data collector

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with…

(Supports DeepSeek R1) An AI-powered research assistant that performs iterative, deep research on any topic…

Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI assistant access to profiles…

Linkedin Automation Tool: Describe your product. Define your target market. The AI finds the leads for you.

Ui.Vision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE …

Local-first AI job intelligence workbench for scraping roles, ranking fit, and generating tailored…

Get clean data from tricky documents, powered by vision-language models ⚡

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms…

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright…

🔥 AI-powered web monitoring platform. Create automated scouts that search the web and send email alerts when…

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps…

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust…

用AI创作高质量内容，用gpt-image-2创作的最佳生图工具，AI图片自动编排，小红书版Openclaw，自媒体创作者的AI工作台，小红书创作AI工具RedClaw，支持小红书图文下载、创作风格学习、小红书AI创作｜…

Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.

Free open-source Multilogin/Incogniton/Kameleo alternative for fingerprint spoofing…

🔥 This repository contains complete application examples, including websites and other projects, developed…

Turn any web app into an API. Chrome extension captures browser traffic, auto-generates schemas, lets AI…

The best CLI agent - The agent who could save AI — reads & edits files, runs Bash, greps your repo, browses…

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched…

The only browser automation that bypasses anti-bot systems. AI writes network hooks, clones UIs pixel-perfect…

ChatGPT Team(Business) 促销码自动扫描工具 — 批量发现/验证/价格收集，支持 17 国 34 个码，最高折扣 71% | ChatGPT Business promo code scanner…

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions…

Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation"…

AI agent skills for UnifAPI MCP and public-data workflows

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

OpenClaw-inspired autonomous AI agent built entirely in n8n. Adaptive RAG-powered memory, Skills via MCP…

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page…

CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents

Open-source AI agent for web automation and scraping.

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

Resume_Builder_AIHawk is a powerful Python tool that allows you to automatically customize your resume based…

Model Context Protocol (MCP) Server for Graphlit Platform

Browser automation, web crawling, and iOS + Android device control for AI agents. Zig-native, token-efficient…

Use LLMs to robustly extract web data

A completely private, locally-operated Ai Assistant/Chatbot/Sub-Agent Framework with realistic Long Term…

This AI Smart Speaker uses speech recognition, TTS (text-to-speech), and STT (speech-to-text) to enable voice…

Tools to build web AI agents that can authenticate, interact with and extract data from any website.

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an…

AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.

CloakBrowser Github: anti-detect browser download, source-level chromium patches, browser fingerprinting…

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web…

Extract knowledge from all information sources using gpt and other language models. Index and make Q&A…

Full-content web fetcher for AI agents — Chrome TLS fingerprinting, browser impersonation, and …

⚡ The Complete X/Twitter Automation Toolkit — Scrapers, MCP server for AI agents (Claude/GPT), CLI, browser…

The open-source execution engine for AI agents. 412 modules, MCP-native, triggers, queue, versioning…

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using…

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep…

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

This project provides a powerful web scraping tool that fetches search results and converts them into…

Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura…

Anti-detection browser server for AI agents — REST API wrapping Camoufox engine with OpenClaw plugin support

PocketGroq is a powerful Python library that simplifies integration with the Groq API, offering advanced…

Developed an AI application using LLM to analyze user resumes and provided the summarization, strengths…

Unofficial Claude API supporting direct HTTP chat creation/deletion/retrieval, messages with multiple file…

skilless.ai gives your AI Agents real data capabilities - web search, web scraping, video download/subtitle…

An implementation of Google Deep Search 🕵️ with support for 1000+ references, local inference, chatting with…

Model Context Protocol server that integrates AgentQL's data extraction capabilities.

AI-powered tool to audit and optimize website content by crawling URLs, analyzing H1s, and generating…

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

Scrape data from social media and chat with it using Langchain

Turn topics, links, and files into AI-generated research notebooks — summarize, explore, and ask anything.

Python package for webscraping in Natural language

JARVIS: a real-time agentic intelligence-gathering platform powered by autonomous web scraping & OSINT…

Open‑source alternative to Perplexity Comet, director.ai and firecrawl combined

OpenClaw skill for scraping any URL using the Decodo Web Scraping API.

GitHub Project: AI Job Application Automation 🚀 This project automates job searching, CV creation, and…

Token-efficient browser MCP server — structured web pages for AI agents, not raw accessibility dumps

AI tool for automating Upwork job applications using AI agents to find and qualify jobs, write personalized…

Intelligent Python system that extracts real estate property data as structured JSON using AI agents, Nebius…

A high level scripting API for bot builders, developers, and maintainers.

CLI tool for agents to quickly access browser telemetry (DOM, network, console) via Chrome DevTools Protocol.

A powerful MCP server extension providing web search and content extraction capabilities. Integrates…

Scrappy assistant that automates web3 bug hunting workflows. Tracks ongoing bug bounties and launches…

A useful drawer for MacOS. chatting, clipboard, webscraping, window managing, shotcuts. built with Rust and …

Automated Miulti AI Agent for company research with LangGraph— scrapes web data, extracts business insights…

Battle-tested skill library for AI agents. Save 98% of API costs with ready-to-use code for crypto, PDFs…

Fast, lightweight Firecrawl alternative in Rust. Web scraper, crawler & search API with MCP server for AI…

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

Streamlit demo of Scrapegraph-ai for GPT4-hackaton

AURA (Agent-Usable Resource Assertion) is an open protocol designed to make the web machine-readable. It…

Erotic conversations scraped from public resources on the internet

Twitter scraper API skill for tweet search, advanced Twitter search, profile tweets, follower export, media…

Official Oxylabs MCP integration

Civic Tech & Data AI For Good project. Tracks prosecutor election messaging, mass incarceration indicators…

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling…

A self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in…

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on…

linkedin-jobs-RAG

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation…

AI-powered browser automation for Go — describe tasks in plain English, let the agent handle the clicks.

CLI, MCP server, and npm library that turns any website into an API — no docs, no SDK, no browser.

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic…

AI web scraper built with Crawl4AI for extracting structured leads data from websites.

pebkac Chrome Nonautomation - A Local LLM-Driven Web Co-Browser using Smolagents, Zendriver, Trafilatura.

Fast, stateless CLI for web search and scrape. Built for AI agents.

Automated London flat/room hunt powered by Claude Code + Claude in Chrome + Gmail MCP. Scrapes 4 rental…

XML sitemap parser designed to extract and process millions of URLs while bypassing most modern anti-bot…

MCP server for scraping LinkedIn, Facebook, Instagram profiles and Google search.

Conversational agent that fuses chat data with live web results through Tavily search, extract, and crawl.

GPT4-powered Slack bot that can scrape URL contents

Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website…

Modern JAV metadata manager — multi-source scraping, Jellyfin integration, and AI-ready API. Built with…

Whiskey AI lets you create autonomous AI agents without code. Connect APIs, automate Solana actions, launch…

⚡ Build structured YouTube datasets at scale — effortlessly fetch transcripts and rich metadata for NLP, ML…

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text…

🦞 AI-powered job hunting agent — scrapes Boss直聘/LinkedIn, matches your profile, auto-applies. Built with…

Anti-detection browser MCP server for AI agents — navigate, interact, and automate the web without getting…

A unified web extraction and stateful automation engine for AI. Replaces heavy testing frameworks with…

Automated Deep Research with LLMs, web search, paper parsing, and didactic summarization.

The Offline Internet.

Reddit_Commentator_AIHawk is a Python project showcasing the power of artificial intelligence in social media…

CrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based…

A (really) easy way to web scrape

This repo provides guidance on setting up a bedrock agent to webscrape and internet search via action groups

A simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web…

Web scraping framework built for AI applications. Extract clean, structured content from any website with…

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and…

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and…

一句话监控网页内容变化，AI | 爬虫 | 网页监控 | 网页更新提醒 | 网页内容订阅

A tool that watches you browse, then writes HTTP-based automation scripts

A collection of cookbooks to help developers get started quickly with the Firecrawl API.

AI Lead Generation Agent that automatically discovers and qualifies potential leads from Quora. Using…

AI-powered agent that scrapes leads with Bright Data, qualifies them using OpenAI, and delivers…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

Local browser toolkit for AI agents: deep research and browser use automation with local Chrome (CDP) +…

🌐 Real-browser web fetching for AI agents.

Retrieval-augmented generation (RAG) for remote & local LLM use

Qurio brings multi-provider models, custom agents, reusable skills, MCP servers, HTTP tools, retrieval…

This demo repository illustrates how to use Python to scrape news articles from Google based on a given…

AI Agent which can do any kind of Browser Automation Task & Web Scraping, just using a single prompt!

OSINT Skill for AI agents (Claude Code, OpenClaw, Codex, OpenCode) — from a name to a scored dossier with…

Chew is a Go library for processing various content types into markdown/plaintext.

LLM orchestration toolkit for agent workflows: planner + workers + synthesis, optional router (LLM + learned …

A powerful integration that combines Browserbase's Stagehand with Mastra for advanced web automation…

Browser automation MCP server for Claude, powered by Puppeteer.

Compare the best SERP APIs by pricing, latency, coverage, and use cases for SEO tools, AI agents, rank…

Reddit AI Agent is an intelligent tool that helps you explore Reddit like never before! 🔎 It allows you to…

Extract Google Maps business leads and enrich contact details using AI & web scraping

⚠️ DEPRECATED — Use https://github.com/zoharbabin/web-researcher-mcp instead

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

A chatbot demo that scrapes a website and stores the result in a vector db, which can then be queried via…

We have developed a fully AI/ML-based itinerary recommendation system which when used by people coming to…

AI web agent to find answers to any question

Give your AI Agent the power to control Safari on macOS. No extensions, no separate browser.

Collection of Project Tutorials / blogs in python, web applications, machine learning, data science, deep…

ChatGPT selenium scraper written in Python

The official Python library for the Steel API

🕵️‍♂️ Privacy-focused AI job scraper, local storage, and interactive dashboard. Auto-scrapes AI/ML roles from…

Web scraping skill for Claude AI. Crawl websites, extract structured data with CSS/LLM strategies, handle…

Bricks is the open-source, fully local alternative to Clay.com. It combines AI agents, advanced web scraping…

AgentStack is a production-grade multi-agent framework built on Mastra, delivering 50+ enterprise tools, 25+…

Leverage modern open-source tools to create better web scraping workflows.

🔍 Model Context Protocol (MCP) tool for parsing websites using the Jina.ai Reader

Intelligent stealth browser MCP server for AI agents with 30 tools, 22 anti fingerprint scripts, and LLM…

Scraping Wikipedia by combining LangChain's agents and tools with OpenAI's LLMs and function calling

The AI-native browser automation library. Snapshot + ref targeting — born from OpenClaw, built for agents, by…

Open Comet is an autonomous AI agent integrated into your Chrome browser. It enables safe, transparent, and…

This project is a Python script that scrapes a Linkedin PDF, generates a customized portfolio site using…

A tool to scrape all files from a GitHub repository and turn it into a JSON or TXT file, Useful for AI and…

This is a darkweb forums tracker that monitors forum posts and sends alerts to Discord

A Twitter/X scraper built with Playwright for browser automation and OpenAI GPT-4 for AI-powered tweet…

How to guides on web-crawling or scraping

GPT-3.5-ON-STEROIDS combines GPT with Python tools, empowering dynamic web scraping, language processing, and…

Pyvigate: A Python framework that combines headless browsing with LLMs that assists you in your data…

Grabs all your Perplexity conversations data, spits it out into a nice file folder structure and allows you…

MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets…

Multi-provider web search CLI for AI agents — Brave, Serper, Exa, Jina, Firecrawl, Perplexity, xAI in one…

Just mention want you want and it will extract/scrape data from the Web. Useful to create AI web…

Zero-dependency browser automation CLI. 70+ commands, 10 test assertions, smart commands (click/fill by text…

Real-time Google Search API for AI Agents & RAG pipelines. Get structured SERP data instantly using remote…

Python, Javascript, and Rust libraries for the Spider Cloud API.

Build a Reddit Content Research Agent with LLMs, LangChain, SERP, Jupyter, Django, Bright Data, Celery…

AgentQL's integrations with workflow automation tools and AI agent frameworks let you extract structured data…

AI-powered deep research tool leveraging web scraping for cost-effective, comprehensive analysis. Open-source…

A :robot: which provides features from Wikipedia like summary, title searches, location API etc.

This is a template repository for building a web scraper with OpenAI support. The repository provides a basic…

[IEEE S&P'26] WebCloak: Characterizing and Mitigating the Threats of LLM-Driven Web Agents as Intelligent…