API Integration Guide

Complete documentation for integrating with the Magpie Scraping API.

🔐 Authentication

All endpoints require a Bearer token in the Authorization header:

Authorization: your-api-token

Contact us for trial access or manage your token in the dashboard.

⏳ Approval Flow

Jobs for Variant Sold, Merchant Items and Search Items endpoints go through manual approval:

  1. You submit a job → Status: AWAITING_APPROVAL
  2. Our team reviews and approves (within 24 hours)
  3. Job starts processing → Status: PROCESSING
  4. Results ready → Status: COMPLETED or PARTIAL_COMPLETE

Note: PDP tier 2, Tokopedia PDP, and Blibli PDP do NOT require approval - tasks are processed immediately.

1. Variant Sold

Batch scraping for Shopee product variant and sales data. Up to 50,000 items per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW

🔑 Endpoint

POST /shopee/variant_sold_v1/submit
GET  /shopee/variant_sold_v1/retrieve/{job_id}
GET  /shopee/variant_sold_v1/download/{job_id}

📥 Request Body

ParameterTypeRequiredDescription
countrystringYesCountry code: id, sg, th, ph, my, vn, tw
itemsarrayYesArray of objects with item_id and shop_id (max 50,000)

💻 Code Examples

# Submit Job
curl -X POST "https://nest-api.magpieiq.com/shopee/variant_sold_v1/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "country": "id",
    "items": [
      {"item_id": 13490493252, "shop_id": 651735813},
      {"item_id": 12345678901, "shop_id": 987654321}
    ]
  }'

# Poll Status
curl "https://nest-api.magpieiq.com/shopee/variant_sold_v1/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download Results
curl "https://nest-api.magpieiq.com/shopee/variant_sold_v1/download/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Content-Type": "application/json"
}

# 1. Submit Job
payload = {
    "country": "id",
    "items": [
        {"item_id": 13490493252, "shop_id": 651735813},
        {"item_id": 12345678901, "shop_id": 987654321}
    ]
}
response = requests.post(f"{API_HOST}/shopee/variant_sold_v1/submit", json=payload, headers=headers)
job_id = response.json()["job_id"]
print(f"Job submitted: {job_id}")

# 2. Poll until complete
while True:
    status_resp = requests.get(f"{API_HOST}/shopee/variant_sold_v1/retrieve/{job_id}", headers=headers)
    status = status_resp.json()["status"]
    print(f"Status: {status}")
    if status in ["COMPLETED", "PARTIAL_COMPLETE", "FAILED"]:
        break
    time.sleep(30)

# 3. Download results
download_resp = requests.get(f"{API_HOST}/shopee/variant_sold_v1/download/{job_id}", headers=headers)
print(download_resp.json())

Behavior

  • Cost: 10 credits per item
  • Approval: Required (24h timeout)
  • Batching: Items are chunked into batches of 50 for processing
  • Results: Merged into a single JSON file per job

Result Example

Variant Sold JSON Click to expand

2. PDP tier 2

Extended product details from Shopee product detail page. Single task per request. No Approval Required

Supported regions: ID · SG · TH · PH · MY · VN

🔑 Endpoint

POST /shopee/get_pc/submit
GET  /shopee/get_pc/retrieve/{task_id}

📥 Request Body

ParameterTypeRequiredDescription
urlstringYesShopee product URL (e.g., https://shopee.co.id/product/{shop_id}/{item_id})

💻 Code Examples

# Submit Task
curl -X POST "https://nest-api.magpieiq.com/shopee/get_pc/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shopee.co.id/product/34907167/40869177342"}'

# Retrieve Result (poll until status is "completed")
curl "https://nest-api.magpieiq.com/shopee/get_pc/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
resp = requests.post(f"{API_HOST}/shopee/get_pc/submit", 
    json={"url": "https://shopee.co.id/product/34907167/40869177342"}, headers=headers)
task_id = resp.json()["task_id"]

# Poll until ready
while True:
    result = requests.get(f"{API_HOST}/shopee/get_pc/retrieve/{task_id}", headers=headers).json()
    if result.get("status") == "completed":
        print(result["data"])
        break
    time.sleep(5)

⚡ Behavior

  • Cost: 8 credits per request
  • Approval: NOT required - immediate processing
  • Processing: Single task processing
  • Results: Full product detail page data including attributes, models, description

Result Example

PDP tier 2 JSON Click to expand

3. PDP tier 2 Bulk CSV

Submit many PDP tier 2 products as CSV and download one CSV or JSONL result file. No Approval Required

CSV columns: region,shop_id,item_id

🔑 Endpoint

POST /shopee/get_pc_bulk/submit
GET  /shopee/get_pc_bulk/retrieve/{task_id}

💻 Code Example

# Submit CSV (result_format defaults to csv; jsonl is faster for large batches)
curl -X POST "https://nest-api.magpieiq.com/shopee/get_pc_bulk/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -F "[email protected];type=text/csv" \
  -F "rescrape=false" \
  -F "result_format=jsonl"

# Poll until status is "completed"
curl "https://nest-api.magpieiq.com/shopee/get_pc_bulk/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download completed result
curl -L -o result.jsonl.gz "{download_url}"

Completed Result Example

{
  "status": "completed",
  "total": 216,
  "success": 216,
  "failed": 0,
  "pending": 0,
  "progress": 1,
  "download_url": "https://storage.googleapis.com/.../result.jsonl.gz?X-Goog-Signature=...",
  "result_gcs_path": "gs://magpie-universe-export/.../result.jsonl.gz",
  "export": {
    "signed_url": "https://storage.googleapis.com/.../result.jsonl.gz?X-Goog-Signature=...",
    "compressed": true
  }
}

Use download_url in the UI/client. result_gcs_path is for debugging.

4. Search / Category Items

Scrape all products from Shopee search or category pages. Up to 50 URLs per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW

🔑 Endpoint

POST /shopee/search-items/submit
GET  /shopee/search-items/retrieve/{job_id}

📥 Request Body

ParameterTypeRequiredDescription
urlsarrayYesArray of Shopee search/category URLs (max 50)

🔗 Supported URL Formats

  • Search: https://shopee.co.id/search?keyword=iphone
  • Category: https://shopee.co.id/Elektronik-cat.11044070

💻 Code Examples

# Submit Job
curl -X POST "https://nest-api.magpieiq.com/shopee/search-items/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://shopee.co.id/search?keyword=iphone",
      "https://shopee.co.id/Elektronik-cat.11044070"
    ]
  }'

# Poll Status & collect download URLs
curl "https://nest-api.magpieiq.com/shopee/search-items/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download a specific result (download_url requires the same Bearer token)
curl "https://nest-api.magpieiq.com/shopee/pgnt/jobs/{job_id}/items/{item_token}/data" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
payload = {"urls": ["https://shopee.co.id/search?keyword=iphone"]}
resp = requests.post(f"{API_HOST}/shopee/search-items/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]

# Poll until complete
while True:
    result = requests.get(f"{API_HOST}/shopee/search-items/retrieve/{job_id}", headers=headers).json()
    if result["status"] in ["COMPLETED", "PARTIAL_COMPLETE"]:
        for item in result["items"]:
            print(f"URL: {item['url']}, Status: {item['status']}, Download: {item.get('download_url')}")
        break
    time.sleep(30)

⚡ Behavior

  • Cost: 6 credits per URL
  • Approval: Required (24h timeout)
  • Region: Automatically extracted from URL (e.g., shopee.co.id → ID)
  • Results: One JSON file per URL containing all items from the search/category
  • Download: Each download_url points to /shopee/pgnt/jobs/{job_id}/items/{token}/data and requires the same Bearer token.
  • Data source: Returns items from search_items (all pages)

Result Example

Search Items JSON Click to expand

5. Merchant Items

Scrape all products from Shopee merchant/shop pages. Up to 50 URLs per request.

Supported regions: ID · SG · TH · PH · MY · VN · TW

🔑 Endpoint

POST /shopee/merchant-items/submit
GET  /shopee/merchant-items/retrieve/{job_id}

📥 Request Body

ParameterTypeRequiredDescription
urlsarrayYesArray of Shopee shop/merchant URLs (max 50)

🔗 Supported URL Formats

  • Shop ID: https://shopee.co.id/shop/24843249

💻 Code Examples

# Submit Job
curl -X POST "https://nest-api.magpieiq.com/shopee/merchant-items/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://shopee.co.id/shop/24843249",
      "https://shopee.co.id/shop/651735813"
    ]
  }'

# Poll Status & collect download URLs
curl "https://nest-api.magpieiq.com/shopee/merchant-items/retrieve/{job_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

# Download a specific result
curl "https://nest-api.magpieiq.com/shopee/pgnt/jobs/{job_id}/items/{item_token}/data" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
payload = {"urls": ["https://shopee.co.id/shop/24843249"]}
resp = requests.post(f"{API_HOST}/shopee/merchant-items/submit", json=payload, headers=headers)
job_id = resp.json()["job_id"]

# Poll & download results
while True:
    result = requests.get(f"{API_HOST}/shopee/merchant-items/retrieve/{job_id}", headers=headers).json()
    if result["status"] in ["completed", "partial_complete"]:
        for item in result["items"]:
            if item["status"] == "SUCCESS" and item["download_url"]:
                data = requests.get(f"{API_HOST}{item['download_url']}", headers=headers).json()
                print(item["url"], data.keys())
        break
    time.sleep(30)

⚡ Behavior

  • Cost: 6 credits per URL
  • Approval: Required (24h timeout)
  • Region: Automatically extracted from URL (e.g., shopee.co.id → ID)
  • Results: One JSON file per shop URL containing all products
  • Download: Use the download_url from retrieve to call /shopee/pgnt/jobs/{job_id}/items/{token}/data with your Bearer token.
  • Data sources: Returns items from both:
    • rcmd_items - Recommended/active items
    • search_items?filter_sold_out=1 - Sold out items (if any)

Result Example

Merchant Items JSON Click to expand

6. Tokopedia PDP Beta

Scrape product detail from Tokopedia product pages. Single task per request. No Approval Required

Region: ID (Tokopedia Indonesia)

🔑 Endpoint

POST /tokopedia/pdp/submit
GET  /tokopedia/pdp/retrieve/{task_id}

📥 Request Body

ParameterTypeRequiredDescription
product_keystringYesTokopedia product key (slug or slug-id from the product URL)
shop_domainstringYesTokopedia shop domain (subdomain part of the shop URL)

🔗 Finding product_key and shop_domain

From a Tokopedia product URL like:

https://www.tokopedia.com/novoidminds/no-void-minds-formlock-oversized-t-shirt-core-1729569390278181925
  • shop_domain = novoidminds (first path segment after domain)
  • product_key = no-void-minds-formlock-oversized-t-shirt-core-1729569390278181925 (second path segment)

💻 Code Examples

# Submit Task
curl -X POST "https://nest-api.magpieiq.com/tokopedia/pdp/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "product_key": "no-void-minds-formlock-oversized-t-shirt-core-1729569390278181925",
    "shop_domain": "novoidminds"
  }'

# Retrieve Result (poll until status is "completed")
curl "https://nest-api.magpieiq.com/tokopedia/pdp/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
resp = requests.post(f"{API_HOST}/tokopedia/pdp/submit",
    json={
        "product_key": "no-void-minds-formlock-oversized-t-shirt-core-1729569390278181925",
        "shop_domain": "novoidminds"
    }, headers=headers)
task_id = resp.json()["task_id"]

# Poll until ready
while True:
    result = requests.get(f"{API_HOST}/tokopedia/pdp/retrieve/{task_id}", headers=headers).json()
    if result.get("status") == "completed":
        print(result["result"])
        break
    time.sleep(5)

⚡ Behavior

  • Cost: 1 credit per request (beta pricing)
  • Approval: NOT required - immediate processing
  • Processing: Single task per request
  • Results: Full product detail page data including variants, pricing, shop info, and brand
  • Batch: For multiple products, submit each separately (batch endpoint coming soon)

Result Example

Tokopedia PDP JSON Click to expand

7. Blibli PDP Beta

Scrape product detail from Blibli product pages. Single task per request. No Approval Required

Region: ID (Blibli Indonesia)

🔑 Endpoint

POST /blibli/pdp/submit
GET  /blibli/pdp/retrieve/{task_id}

📥 Request Body

ParameterTypeRequiredDescription
formattedIdstringYesBlibli formatted product ID (e.g. ps--16M-70001-00016)
skustringYesBlibli SKU (e.g. 16M-70001-00016-00001)
pickupPointCodestringNoBlibli pickup point code (e.g. PP-3491245)
idstringNoProduct ID used as the GCS filename key; defaults to sku

🔗 Finding formattedId and sku

From a Blibli product URL like:

https://www.blibli.com/p/sample-product/ps--16M-70001-00016
  • formattedId = ps--16M-70001-00016 (last path segment)
  • sku = 16M-70001-00016-00001 (from product detail API or product variants)

💻 Code Examples

# Submit Task
curl -X POST "https://nest-api.magpieiq.com/blibli/pdp/submit" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "formattedId": "ps--16M-70001-00016",
    "sku": "16M-70001-00016-00001"
  }'

# Retrieve Result (poll until status is "completed")
curl "https://nest-api.magpieiq.com/blibli/pdp/retrieve/{task_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
View Python Example 
import requests
import time

API_HOST = "https://nest-api.magpieiq.com"
headers = {"Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json"}

# Submit
resp = requests.post(f"{API_HOST}/blibli/pdp/submit",
    json={
        "formattedId": "ps--16M-70001-00016",
        "sku": "16M-70001-00016-00001"
    }, headers=headers)
task_id = resp.json()["task_id"]

# Poll until ready
while True:
    result = requests.get(f"{API_HOST}/blibli/pdp/retrieve/{task_id}", headers=headers).json()
    if result.get("status") == "completed":
        print(result["result"])
        break
    time.sleep(5)

⚡ Behavior

  • Cost: 1 credit per request (beta pricing)
  • Approval: NOT required - immediate processing
  • Processing: Single task per request
  • Results: Full product detail page data including variants, pricing, stock, and merchant info
  • Batch: For multiple products, submit each separately

Result Example

Blibli PDP JSON Click to expand

📋 Job Status Reference

Job Statuses:

  • AWAITING_APPROVAL - Pending team approval
  • PROCESSING - Approved, scraping in progress
  • COMPLETED - All items successful
  • PARTIAL_COMPLETE - Some items failed
  • FAILED - All items failed
  • REJECTED - Rejected or timed out

HTTP Status Codes:

  • 200 - Success
  • 202 - Job accepted
  • 401 - Invalid/missing token
  • 402 - Insufficient credits
  • 404 - Job not found
  • 422 - Validation error

For interactive API documentation, see Swagger UI.