Scraper agents

🔥 Search, scrape, and clean the web for AI agents.

Create agents that monitor and act on your behalf. Your agents are standing by!

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial…

Python scraper based on AI

Turn any webpage into structured data using LLMs

A community-driven way to read and chat with AI bots - powered by chatGPT.

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP…

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

Get clean data from tricky documents, powered by vision-language models ⚡

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps…

用AI创作高质量内容，用gpt-image-2创作的最佳生图工具，AI图片自动编排，小红书版Openclaw，自媒体创作者的AI工作台，小红书创作AI工具RedClaw，支持小红书图文下载、创作风格学习、小红书AI创作｜…

Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.

🔥 This repository contains complete application examples, including websites and other projects, developed…

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched…

Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation"…

OpenClaw-inspired autonomous AI agent built entirely in n8n. Adaptive RAG-powered memory, Skills via MCP…

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

Resume_Builder_AIHawk is a powerful Python tool that allows you to automatically customize your resume based…

Use LLMs to robustly extract web data

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an…

AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web…

Extract knowledge from all information sources using gpt and other language models. Index and make Q&A…

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using…

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

This project provides a powerful web scraping tool that fetches search results and converts them into…

Developed an AI application using LLM to analyze user resumes and provided the summarization, strengths…

Unofficial Claude API supporting direct HTTP chat creation/deletion/retrieval, messages with multiple file…

Open‑source alternative to Perplexity Comet, director.ai and firecrawl combined

AI tool for automating Upwork job applications using AI agents to find and qualify jobs, write personalized…

A powerful MCP server extension providing web search and content extraction capabilities. Integrates…

Fast, lightweight Firecrawl alternative in Rust. Web scraper, crawler & search API with MCP server for AI…

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

Twitter scraper API skill for tweet search, advanced Twitter search, profile tweets, follower export, media…

A self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in…

linkedin-jobs-RAG

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation…

AI web scraper built with Crawl4AI for extracting structured leads data from websites.

XML sitemap parser designed to extract and process millions of URLs while bypassing most modern anti-bot…

MCP server for scraping LinkedIn, Facebook, Instagram profiles and Google search.

Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website…

Modern JAV metadata manager — multi-source scraping, Jellyfin integration, and AI-ready API. Built with…

⚡ Build structured YouTube datasets at scale — effortlessly fetch transcripts and rich metadata for NLP, ML…

Reddit_Commentator_AIHawk is a Python project showcasing the power of artificial intelligence in social media…

This repo provides guidance on setting up a bedrock agent to webscrape and internet search via action groups

A simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

Reddit AI Agent is an intelligent tool that helps you explore Reddit like never before! 🔎 It allows you to…

A chatbot demo that scrapes a website and stores the result in a vector db, which can then be queried via…

ChatGPT selenium scraper written in Python

Web scraping skill for Claude AI. Crawl websites, extract structured data with CSS/LLM strategies, handle…

Open Comet is an autonomous AI agent integrated into your Chrome browser. It enables safe, transparent, and…

A tool to scrape all files from a GitHub repository and turn it into a JSON or TXT file, Useful for AI and…

A Twitter/X scraper built with Playwright for browser automation and OpenAI GPT-4 for AI-powered tweet…

How to guides on web-crawling or scraping

Grabs all your Perplexity conversations data, spits it out into a nice file folder structure and allows you…

Just mention want you want and it will extract/scrape data from the Web. Useful to create AI web…

Python, Javascript, and Rust libraries for the Spider Cloud API.

A :robot: which provides features from Wikipedia like summary, title searches, location API etc.

This is a template repository for building a web scraper with OpenAI support. The repository provides a basic…

[IEEE S&P'26] WebCloak: Characterizing and Mitigating the Threats of LLM-Driven Web Agents as Intelligent…

Parse SaaS pricing page using Open AI - GPT-3.5

A Python project that extracts data from websites with the option to process the data through @openai's…

A bot that scrapes open-interest and liquidation heatmaps to alert traders when a "Short Squeeze" or "Long…

Claude Code Skill - 抓取微信公众号文章并转换为 Markdown，自动下载图片 | WeChat Article to Markdown Converter

⚡️ Real-time Knowledge Graph for AI Agents. Connect LLMs to verified weather, stock, and currency data via…

MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies…

The topic is about product matching via Machine Learning. This involves using various machine learning…

🧘 Reddit-powered AI optimization tips | Save your time and credits | can cover 380+ services | Real tips from…

Webpage to structured data in Rust & LLM

Instant, local access to complete Base44 documentation with AI assistant integration

AURORA (Artificial Unified Responsive Optimized Reasoning Agent) uses lobes and web research for RAG based…

Efficient RAG knowledge pack creator from online Julia documentation

RAG-based Web Scraping

ShopFilter 是由 OpenCode AI…

钉钉智能机器人，支持AI问答、知识库检索、JIRA管理和服务器维护、周报日报总结、快捷创建工单等等

Retrieval-augmented docs ingestion stack: Firecrawl + Crawl4AI + Qdrant vector search with FastAPI and MCP…

Scrapes headlines from CNN and FOX, then has ChatGPT do cross-analysis

Telegram bot which helps in promoting Instagram accounts

Telegram bot utilizing OpenAI's GPT to generate presentations and abstracts in PPTX and DOCX formats.

🦅 DestinyScout: 一款基于 Agent-Native LLM + Boss直聘的 L3 级自主个性化求职引擎。告别机械搬运，它能深度注入你的私人职场 DNA…

Infrastructure layer for AI agent swarms — 88 MCP tools · A2A · OmniMesh VPN · Scrapling scraper · COC sync ·…

This is the repository for a Streamlit application that helps with job applications. This app integrates…

A Python tool that automates LinkedIn job search, ranking, and export by combining Bright Data's LinkedIn Job…

A messenger bot that answers messages by scraping stackoverflow questions and answers

Python implementation of https://github.com/mishushakov/llm-scraper

The Most Powerful Open-source LLM Friendly Typescript Web Crawler & Scraper

AI News Scraper & Semantic Search: A Python application that scrapes news articles, uses GenAI to generate…

PromoBot - A web scraper that monitors promotion sites by searching keywords and reporting to a Telegram…

The Real Time Social Media Content Retrieval System fetches real-time LinkedIn posts based on user queries…

Tracking the systems that automate scientific research — from literature scrapers to full paper-writing…

Lightfeed SDK to search and filter web data

Pangolinfo 亚马逊爬虫与数据采集工具：基于 Pangolinfo Amazon Scrape API / 数据 API 实现 Amazon 实时数据采集（商品详情、关键词、评论、榜单、类目/利基），输出 AI…

AI-first web scraping engine with stealth bypass, MCP server, and multimodal output (Markdown, JSON, PDF) for…

A robust, local-first intelligence application for scraping and analyzing Dark Web data.

Paper Reading Agent Team…

Tired of LLMs citing fake papers? renderscholar is a Google Scholar scraper (inspired by Andrej Karpathy’s…

🆓 Access a collection of free, public JSON APIs with no limits or authentication required. Explore…

A simple npm package to perform requests as a user on the OpenAI ChatGPT page.

Find real pain points on Reddit and draft value-first replies. CLI + Claude Code skill. Serper + Reddit…

This AI bot goes online, gathers information about AI startups, and posts updates about them on X and Dev.to.

AI-native, agent-first web scraping for Python — cost-aware tiered fetching (HTTP → browser → stealth →…

Stone Scraper is an AI-powered tool for automated web data extraction. Built with Streamlit, Langchain, and…

This is the repository for a Streamlit application that helps with job applications. This app integrates…

This MCP server enables LLMs to retrieve and process web scraping requests using ScraperAPI.

Single-binary web scraper for AI agents. Headless Chrome + Readability → clean Markdown. Up to 99% token …

Integrating OpenAI Agents SDK with Bright Data Web Unlocker, enabling AI agents to access, extract, and…

Local MCP Server with Claude Desktop (Windows + WSL) with scrapping and crawling tools.

Otodom scraper and information retrieval

CrewAI Multiagent is an AI-powered automation suite for research, news, poetry, code execution, and PDF…

A lightweight MCP server for LinkedIn automation. Supports profile, job, company and post scraping. Enables…

Convert WeChat Official Account articles to clean Markdown with metadata extraction, image download, and code…

Enable AI agents to search, crawl, and extract web data with IP rotation, CAPTCHA handling, and rate limit…

Scrapes newly posted jobs from a variety of sites and an ai agent filters them based off of the users resume.

Autonomous AI agent skill for aggressive lead generation and growth hacking.

A Student hub, real-time plugin based web-application featuring a chat, marketplace, video conferencing, and…

Convert an X (Twitter) URL into clean markdown — tweets, photos, videos, long-form X Articles, and top…