ScrapeGraphAI: Build Lead Lists From Public Directories With Claude

promptRead the README at https://github.com/ScrapeGraphAI/Scrapegraph-ai and set up a small lead-list project in this folder.

1. Create a Python 3.12+ virtual environment and install scrapegraphai, langchain-anthropic, and the Playwright browser the README asks for. Tell me each command before you run it.
2. Write leads.py that uses SmartScraperGraph with Claude (model anthropic/claude-haiku-4-5, my key from the ANTHROPIC_API_KEY environment variable) and a Pydantic schema with these fields: name, category, website, phone, public_email, city.
3. Loop over these public directory pages:
   [PASTE PAGE URLS HERE, ONE PER LINE]
4. Only extract business contact details that are printed on the page. No guessing, no personal profiles, no pages behind a login.
5. Remove duplicates by name and website, add a source_url column, and save everything to leads.csv.
6. Before running on all pages, check the site's terms of service and robots.txt and tell me if anything forbids automated collection. Then test on ONE page and show me the first 5 rows.

bashpython3 -m venv .venv
source .venv/bin/activate
pip install scrapegraphai langchain-anthropic
playwright install
export ANTHROPIC_API_KEY=sk-ant-your-key-here

python# leads.py: turn public directory pages into a clean CSV of business contacts.
# Run inside a virtual environment where scrapegraphai is installed.
import csv
import os
from typing import List, Optional

from pydantic import BaseModel
from scrapegraphai.graphs import SmartScraperGraph

# 1) The public directory pages you want to read (one URL per results page).
PAGES = [
    "https://example-directory.com/plumbers/austin?page=1",
    "https://example-directory.com/plumbers/austin?page=2",
]

# 2) What one row in your sheet looks like.
class Business(BaseModel):
    name: str
    category: Optional[str] = None
    website: Optional[str] = None
    phone: Optional[str] = None
    public_email: Optional[str] = None
    city: Optional[str] = None

class Listing(BaseModel):
    businesses: List[Business]

PROMPT = (
    "List every business shown on this page. For each one return the business name, "
    "category, website, main business phone number, city, and a general business email "
    "only if it is printed on the page. Only use details that are publicly shown on this "
    "page. Do not guess, infer or invent anything. Leave a field empty if it is not shown. "
    "Skip private individuals and personal social media profiles."
)

# 3) The model. This uses Claude through your Anthropic API key.
#    For a fully local run, swap in the Ollama config from the ScrapeGraphAI README.
graph_config = {
    "llm": {
        "model": "anthropic/claude-haiku-4-5",
        "api_key": os.environ["ANTHROPIC_API_KEY"],
        "model_tokens": 200000,
    },
    "headless": True,
    "verbose": False,
}

rows, seen = [], set()
for url in PAGES:
    result = SmartScraperGraph(prompt=PROMPT, source=url, config=graph_config, schema=Listing).run()
    for b in (result or {}).get("businesses", []):
        key = ((b.get("name") or "").strip().lower(), (b.get("website") or "").strip().lower())
        if not key[0] or key in seen:
            continue
        seen.add(key)
        b["source_url"] = url
        rows.append(b)

fields = ["name", "category", "website", "phone", "public_email", "city", "source_url"]
with open("leads.csv", "w", newline="") as fh:
    writer = csv.DictWriter(fh, fieldnames=fields, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(rows)

print(f"Saved {len(rows)} businesses to leads.csv")

python# ollama pull llama3.2   (run this in your terminal first)
graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "model_tokens": 8192,
        "format": "json",
    },
    "headless": True,
    "verbose": False,
}

promptList every business on this page that is a dental clinic. For each one return the clinic name, street address, city, main phone number, website and booking page link if shown. Only use details printed on this page. Leave a field empty if it is not shown. Do not invent anything.

promptThis is a member directory of a local chamber of commerce. For each member business return the business name, industry, website and the general business phone. Ignore named individuals' personal phone numbers and personal emails. Only use details printed on this page.

promptI have a CSV called leads.csv. Clean it: fix capitalization in names, format every phone number the same way, remove rows with no website and no phone, flag any row where the email domain does not match the website domain, and save the result as leads_clean.csv. Show me what you changed.