Overview

ScrapeKit is a REST API for scraping public data from Instagram, Zillow, Redfin, Realtor.com, YouTube, and Meta Ad Library. Submit scrape jobs, poll for results, and optionally receive webhook notifications when jobs complete.

Base URL: https://scrapekit.dev

API Version: v1

Content Type: application/json

Quick example
# Submit a scrape job
curl -X POST https://scrapekit.dev/api/v1/scrape/profile \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk_live_your_key_here" \
  -d '{"username": "natgeo"}'

# Response
{
  "job_id": "abc123",
  "status": "queued",
  "type": "profile",
  "target": "natgeo",
  "estimated_seconds": 15
}

# Poll for results
curl https://scrapekit.dev/api/v1/jobs/abc123 \
  -H "X-API-Key: sk_live_your_key_here"

Authentication

All API requests require authentication via an API key sent in the X-API-Key header. You receive your key at registration and can regenerate it from the dashboard.

Header format
X-API-Key: sk_live_abc123def456abc123def456abc123de

Register

POST/api/v1/auth/register
Request
{
  "email": "you@example.com",
  "password": "min8chars"
}
Response (201)
{
  "user_id": "clx...",
  "email": "you@example.com",
  "api_key": "sk_live_abc123...",
  "tier": "free",
  "checkout_url": null,
  "message": "Store your API key securely. It cannot be retrieved later."
}

Login

POST/api/v1/auth/login
Request
{
  "email": "you@example.com",
  "password": "min8chars"
}

Sets an HTTP-only session cookie for dashboard access.

Get Session

GET/api/v1/auth/sessionSession Cookie
Response
{
  "userId": "clx...",
  "email": "you@example.com",
  "keyPrefix": "sk_live_abc1",
  "tier": "free"
}

Regenerate API Key

POST/api/v1/auth/keys/regenerateAPI Key or Session
Response
{
  "api_key": "sk_live_new_key_here...",
  "message": "New API key generated. Your old key no longer works."
}

Usage & Billing Info

GET/api/v1/auth/usageAPI Key or Session
Response
{
  "tier": "pro",
  "requests_this_month": 142,
  "monthly_limit": 1000,
  "remaining": 858,
  "month_resets_at": "2026-04-05T00:00:00.000Z",
  "features": ["hashtags", "comments", "bulk", "engagement_rate"]
}

Error Handling

All errors follow a consistent format with an error object containing a message and code.

Error response format
{
  "error": {
    "message": "Monthly credit limit exceeded. Please upgrade your plan.",
    "code": "RATE_LIMITED"
  }
}
CodeStatusDescription
UNAUTHORIZED401Invalid or missing API key
SUBSCRIPTION_REQUIRED403No active Stripe subscription
FEATURE_NOT_AVAILABLE403Feature not included in your tier
RATE_LIMITED429Monthly credit limit exceeded
VALIDATION_ERROR400Invalid request body
NOT_FOUND404Resource not found
EMAIL_EXISTS409Email already registered

Pricing Tiers

FeatureStarterGrowthBusinessScale
Credits/month501,0005,00020,000
Profile / Posts / Reels++++
Zillow / Redfin / Realtor++++
YouTube / Meta Ads++++
Hashtags / Comments-+++
Bulk scraping-1050200
Webhooks--++
Schedules---+

Instagram

All Instagram scrape endpoints return a job_id. Poll /api/v1/jobs/:id to get results. Each request costs 1 credit.

Scrape Profile

POST/api/v1/scrape/profileAPI KeyStarter+1 credit
Request
{
  "username": "natgeo",
  "include_recent_posts": false  // optional
}
Job result (data field)
{
  "username": "natgeo",
  "full_name": "National Geographic",
  "biography": "...",
  "followers": 283000000,
  "following": 156,
  "posts_count": 28500,
  "profile_pic_url": "https://...",
  "is_verified": true,
  "external_url": "https://natgeo.com"
}

Scrape Posts

POST/api/v1/scrape/postsAPI KeyStarter+1 credit
Request
{
  "username": "natgeo"
}

Scrape Reels

POST/api/v1/scrape/reelsAPI KeyStarter+1 credit
Request
{
  "username": "natgeo"
}

Scrape Hashtags

POST/api/v1/scrape/hashtagsAPI KeyGrowth+1 credit
Request
{
  "hashtag": "travel"
}

Scrape Comments

POST/api/v1/scrape/commentsAPI KeyGrowth+1 credit
Request
{
  "shortcode": "CxYzAbC1234"
}

Real Estate

Search property listings across Zillow, Redfin, and Realtor.com. All search endpoints accept the same filter parameters.

Zillow Agent Profile

POST/api/v1/scrape/zillow-profileAPI KeyStarter+1 credit
Request
{
  "slug": "Agent-Name"  // from zillow.com/profile/Agent-Name
}

Zillow Search

POST/api/v1/scrape/zillow-searchAPI KeyStarter+1 credit
Request
{
  "location": "San Francisco, CA",  // required
  "price_min": 200000,              // optional
  "price_max": 800000,              // optional
  "beds_min": 2,                    // optional (0-10)
  "baths_min": 1,                   // optional (0-10)
  "sqft_min": 1000,                 // optional
  "sqft_max": 5000,                 // optional
  "property_type": "house",         // optional: house|condo|townhouse|multi_family|land
  "status": "for_sale"              // optional: for_sale|sold|pending
}
Job result (data field)
{
  "location": "San Francisco, CA",
  "total_count": 1250,
  "properties": [
    {
      "address": "123 Market St, San Francisco, CA 94105",
      "price": 950000,
      "beds": 2,
      "baths": 2,
      "sqft": 1200,
      "property_type": "condo",
      "listing_url": "https://zillow.com/...",
      "thumbnail_url": "https://...",
      "latitude": 37.7749,
      "longitude": -122.4194
    }
  ],
  "metadata": {
    "pages_scraped": 8,
    "duration_seconds": 45,
    "price_buckets_used": false
  }
}

Auto-splits into price ranges when results exceed ~800 to ensure full coverage.

Redfin Search

POST/api/v1/scrape/redfin-searchAPI KeyStarter+1 credit
Request
{
  "location": "Denver, CO"  // city, zip, or address
}
Job result (data field)
{
  "location": "Denver, CO",
  "total_count": 350,
  "properties": [
    {
      "address": "2490 S Garfield St, Denver, CO 80210",
      "street": "2490 S Garfield St",
      "city": "Denver",
      "state": "CO",
      "zip": "80210",
      "price": 1950000,
      "beds": 5,
      "baths": 5,
      "sqft": 3607,
      "listing_url": "https://www.redfin.com/CO/Denver/...",
      "thumbnail_url": "https://ssl.cdn-redfin.com/...",
      "mls_id": "7892039",
      "latitude": 39.6712,
      "longitude": -104.9440,
      "source": "redfin"
    }
  ]
}

Realtor.com Search

POST/api/v1/scrape/realtor-searchAPI KeyStarter+1 credit
Request
{
  "location": "Austin, TX"  // city/state or zip code
}
Job result (data field)
{
  "location": "Austin, TX",
  "total_count": 2000,
  "properties": [
    {
      "address": "315 Leaning Rock Rdg, Austin, TX 78737",
      "city": "Austin",
      "state": "TX",
      "zip": "78737",
      "price": 936460,
      "beds": 4,
      "baths": 3,
      "sqft": 2850,
      "property_type": "single_family",
      "status": "for_sale",
      "listing_url": "https://www.realtor.com/...",
      "thumbnail_url": "https://...",
      "latitude": 30.2672,
      "longitude": -97.7431,
      "source": "realtor"
    }
  ]
}

YouTube

YouTube Channel

POST/api/v1/scrape/youtube-channelAPI KeyStarter+1 credit
Request
{
  "channel": "@MrBeast"  // handle, channel ID, or full URL
}

YouTube Video

POST/api/v1/scrape/youtube-videoAPI KeyStarter+1 credit
Request
{
  "video_url": "https://youtube.com/watch?v=dQw4w9WgXcQ"
}

Meta Ad Library

Meta Ad Library

POST/api/v1/scrape/meta-ad-libraryAPI KeyStarter+1 credit
Request
{
  "advertiser": "Nike"  // advertiser name, ad ID, or ad library URL
}

TikTok

TikTok Profile

POST/api/v1/scrape/tiktok-profileAPI KeyStarter+1 credit
Request
{
  "username": "@charlidamelio"  // @handle, username, or full URL
}
Job result (data field)
{
  "profile": {
    "username": "charlidamelio",
    "display_name": "Charli D'Amelio",
    "follower_count": 155000000,
    "following_count": 1200,
    "bio": "...",
    "profile_pic_url": "https://...",
    "video_count": 30,
    "total_views": 450000000,
    "avg_views": 15000000
  },
  "videos": [
    {
      "video_id": "7341...",
      "title": "Video title...",
      "view_count": 25000000,
      "like_count": 1200000,
      "comment_count": 8500,
      "share_count": 45000,
      "thumbnail_url": "https://...",
      "video_url": "https://...",
      "outlier_score": 1.67
    }
  ]
}

TikTok Video

POST/api/v1/scrape/tiktok-videoAPI KeyStarter+1 credit
Request
{
  "video_url": "https://www.tiktok.com/@user/video/7341..."
}

LinkedIn

LinkedIn Profile

POST/api/v1/scrape/linkedin-profileAPI KeyStarter+1 credit
Request
{
  "profile_url": "https://linkedin.com/in/satyanadella"
  // or just: "satyanadella"
}
Job result (data field)
{
  "name": "Satya Nadella",
  "headline": "Chairman and CEO at Microsoft",
  "location": "Greater Seattle Area",
  "profile_url": "https://linkedin.com/in/satyanadella",
  "profile_pic_url": "https://...",
  "summary": "...",
  "experience": [
    {
      "title": "Chairman and CEO",
      "company": "Microsoft",
      "duration": "2014 - Present"
    }
  ],
  "education": [...],
  "skills": [...]
}

LinkedIn Search

POST/api/v1/scrape/linkedin-searchAPI KeyStarter+1 credit
Request
{
  "query": "software engineer san francisco"
}

LinkedIn Jobs

POST/api/v1/scrape/linkedin-jobsAPI KeyStarter+1 credit
Request
{
  "query": "product manager",
  "location": "New York"  // optional
}

X / Twitter

X Profile

POST/api/v1/scrape/x-profileAPI KeyStarter+1 credit
Request
{
  "username": "elonmusk"
}
Job result (data field)
{
  "username": "elonmusk",
  "name": "Elon Musk",
  "bio": "...",
  "followers_count": 195000000,
  "following_count": 836,
  "tweet_count": 48291,
  "verified": true,
  "profile_image_url": "https://...",
  "banner_url": "https://...",
  "location": "Mars & Earth",
  "website": "https://..."
}

X Search

POST/api/v1/scrape/x-searchAPI KeyStarter+1 credit
Request
{
  "query": "AI startups funding"
}

X Jobs

POST/api/v1/scrape/x-jobsAPI KeyStarter+1 credit
Request
{
  "query": "hiring react developer",
  "location": "remote"  // optional
}

Bulk Scraping

Bulk Scrape

POST/api/v1/scrape/bulkAPI KeyGrowth+N credits

Submit up to 200 usernames in a single request. Costs 1 credit per username. Bulk cap depends on tier: Growth=10, Business=50, Scale=200.

Request
{
  "usernames": ["natgeo", "nasa", "spacex"],
  "type": "profile"  // profile | posts | reels
}
Response (202)
{
  "jobs": [
    { "job_id": "abc1", "target": "natgeo", "status": "queued" },
    { "job_id": "abc2", "target": "nasa", "status": "queued" },
    { "job_id": "abc3", "target": "spacex", "status": "queued" }
  ],
  "total_credits_used": 3
}

Jobs

All scrape requests are async. Submit a job, then poll for results. Jobs are retained for 30 days.

List Jobs

GET/api/v1/jobsAPI Key

Query params: limit (1-100, default 20), offset (default 0)

Response
{
  "jobs": [
    {
      "job_id": "abc123",
      "type": "profile",
      "target": "natgeo",
      "status": "completed",
      "created_at": "2026-03-05T10:00:00Z",
      "completed_at": "2026-03-05T10:00:15Z"
    }
  ],
  "total": 42,
  "limit": 20,
  "offset": 0
}

Get Job

GET/api/v1/jobs/:idAPI Key
Response (completed)
{
  "job_id": "abc123",
  "type": "profile",
  "target": "natgeo",
  "status": "completed",
  "data": { ... },
  "error": null,
  "created_at": "2026-03-05T10:00:00Z",
  "completed_at": "2026-03-05T10:00:15Z"
}

Status values: queued processing completed failed

Webhooks

Receive HTTP POST notifications when jobs complete or fail. Business tier and above. Webhook payloads are signed with HMAC-SHA256.

Create Webhook

POST/api/v1/webhooksAPI KeyBusiness+
Request
{
  "url": "https://your-app.com/webhooks/scrapeapi",
  "events": ["job.completed", "job.failed"]  // optional, defaults to both
}
Response (201)
{
  "id": "wh_abc123",
  "url": "https://your-app.com/webhooks/scrapeapi",
  "secret": "whsec_abc123...",
  "events": ["job.completed", "job.failed"],
  "active": true,
  "created_at": "2026-03-05T10:00:00Z"
}

Save the secret to verify webhook signatures.

List Webhooks

GET/api/v1/webhooksAPI KeyBusiness+

Update Webhook

PATCH/api/v1/webhooks/:idAPI KeyBusiness+
Request (all fields optional)
{
  "url": "https://new-url.com/hook",
  "events": ["job.completed"],
  "active": false
}

Delete Webhook

DELETE/api/v1/webhooks/:idAPI KeyBusiness+

Schedules

Automatically run scrape jobs on a recurring schedule. Scale tier only.

Create Schedule

POST/api/v1/schedulesAPI KeyScale
Request
{
  "type": "profile",           // profile | posts | reels | hashtags
  "target": "natgeo",          // username or hashtag
  "interval": "daily"          // daily | weekly | monthly
}
Response (201)
{
  "id": "sch_abc123",
  "type": "profile",
  "target": "natgeo",
  "interval": "daily",
  "active": true,
  "next_run_at": "2026-03-06T00:00:00Z",
  "created_at": "2026-03-05T10:00:00Z"
}

List Schedules

GET/api/v1/schedulesAPI KeyScale

Update Schedule

PATCH/api/v1/schedules/:idAPI KeyScale
Request (all fields optional)
{
  "interval": "weekly",
  "active": false
}

Delete Schedule

DELETE/api/v1/schedules/:idAPI KeyScale

Billing

Create Checkout Session

POST/api/v1/billing/checkoutAPI Key or Session
Request
{
  "tier": "pro"  // pro | business | enterprise
}
Response
{
  "url": "https://checkout.stripe.com/c/pay/..."
}

Redirect the user to the checkout URL to complete payment.

Billing Portal

POST/api/v1/billing/portalAPI Key or Session
Response
{
  "url": "https://billing.stripe.com/p/session/..."
}

Redirect users here to manage their subscription, update payment method, or cancel.

Questions? Reach out at support@scrapekit.dev