Nimbryxo Studio

Shopify Store Catalog & Pricing Extractor

Extract a Shopify store's product catalog with titles, prices, compare-at prices, variants, SKUs, images and stock availability from any custom or myshopify.com domain.

Input schema

Input fields, types, required flags, descriptions, defaults, and prefills come from the published definition. Prefill examples are distinct from defaults.

FieldTypeRequiredDescription
modestringrequiredWhat to scrape. `catalog` walks the public product feed, `collection` walks specific collections, `product` fetches exact product handles, `search` uses the storefront search suggestions, `collections` lists collection metadata, and `auto` detects the type of each URL in `startUrls`.
storeUrlsarrayoptionalStore domains or URLs, e.g. `deathwishcoffee.com` or `https://www.allbirds.com`. Accepts custom domains and `*.myshopify.com`. A product or collection URL is reduced to its store for catalog/collections/search modes.
startUrlsarrayoptionalStore, collection or product URLs. In `auto` mode each URL selects its own job: `/products/<handle>` fetches one product, `/collections/<handle>` fetches a collection, any other store URL fetches the catalog. In the other modes these URLs add stores and, for collection/product modes, their handles.
Show all input fields
FieldTypeRequiredDescription
modestringrequiredWhat to scrape. `catalog` walks the public product feed, `collection` walks specific collections, `product` fetches exact product handles, `search` uses the storefront search suggestions, `collections` lists collection metadata, and `auto` detects the type of each URL in `startUrls`.
storeUrlsarrayoptionalStore domains or URLs, e.g. `deathwishcoffee.com` or `https://www.allbirds.com`. Accepts custom domains and `*.myshopify.com`. A product or collection URL is reduced to its store for catalog/collections/search modes.
startUrlsarrayoptionalStore, collection or product URLs. In `auto` mode each URL selects its own job: `/products/<handle>` fetches one product, `/collections/<handle>` fetches a collection, any other store URL fetches the catalog. In the other modes these URLs add stores and, for collection/product modes, their handles.
collectionHandlesarrayoptionalCollection handles or collection URLs, e.g. `new-arrivals`, `sale`, or `https://store.com/collections/sale`. Handles are applied to every store in `storeUrls`; a full collection URL is applied to its own store.
productHandlesarrayoptionalProduct handles or product URLs, e.g. `dark-roast-coffee` or `https://store.com/products/dark-roast-coffee`. Handles are applied to every store in `storeUrls`; a full product URL is applied to its own store.
searchQuerystringoptionalIn-store search term. Shopify's public suggest endpoint returns at most 10 products per store and has no pagination.
currencystringoptionalISO 4217 code to pin prices to, e.g. `USD`, `EUR`, `GBP`. Use `store` to use the store's base currency from `/meta.json`. Pinning is important because Shopify localizes prices by IP.
countrystringoptionalTwo-letter ISO country code passed as `?country=` to the storefront, e.g. `US`, `GB`, `DE`.
vendorstringoptionalKeep products whose vendor contains this text (case-insensitive).
productTypestringoptionalKeep products whose product type contains this text (case-insensitive).
tagstringoptionalKeep products with at least one tag containing this text (case-insensitive).
minPricenumberoptionalDrop products whose lowest variant price is below this value.
maxPricenumberoptionalDrop products whose lowest variant price is above this value.
availablestringoptionalKeep all products, only products with at least one purchasable variant, or only sold-out products.
maxItemsintegeroptionalHard cap on dataset rows across all stores. When `flattenVariants` is on, each variant row counts as one.
maxPagesintegeroptionalPage cap per product/collection feed. Each page holds up to 250 items and Shopify refuses offsets above 25,000, so the effective maximum is 100.
includeDetailsbooleanoptionalFetch `/products/{handle}.json` for each emitted product to add barcode, weight/grams and inventory fields. Costs one extra request per product and is off by default. Product mode always uses the detail endpoint.
flattenVariantsbooleanoptionalEmit one `recordType: variant` row per variant instead of one product row with nested variants. Variant and product IDs are de-duplicated.
proxyConfigurationobjectoptionalLeave empty to fetch directly. After a 403 or 429 block the actor retries through Apify's datacenter proxy (AUTO) automatically. If you set this, your configuration is used for that retry instead of AUTO; direct requests are still tried first.

Download the published input schema

View complete JSON
{
  "description": "Scrape one or many Shopify storefronts: full catalog, collection, single product, in-store search, or the collection directory.",
  "properties": {
    "available": {
      "default": "any",
      "description": "Keep all products, only products with at least one purchasable variant, or only sold-out products.",
      "editor": "select",
      "enum": [
        "any",
        "available",
        "unavailable"
      ],
      "enumTitles": [
        "Any availability",
        "In stock only",
        "Sold out only"
      ],
      "prefill": "any",
      "title": "Availability",
      "type": "string"
    },
    "collectionHandles": {
      "default": [],
      "description": "Collection handles or collection URLs, e.g. `new-arrivals`, `sale`, or `https://store.com/collections/sale`. Handles are applied to every store in `storeUrls`; a full collection URL is applied to its own store.",
      "editor": "stringList",
      "prefill": [],
      "title": "Collection handles (mode=collection)",
      "type": "array"
    },
    "country": {
      "description": "Two-letter ISO country code passed as `?country=` to the storefront, e.g. `US`, `GB`, `DE`.",
      "editor": "textfield",
      "title": "Country (optional)",
      "type": "string"
    },
    "currency": {
      "default": "store",
      "description": "ISO 4217 code to pin prices to, e.g. `USD`, `EUR`, `GBP`. Use `store` to use the store's base currency from `/meta.json`. Pinning is important because Shopify localizes prices by IP.",
      "editor": "textfield",
      "prefill": "USD",
      "title": "Currency",
      "type": "string"
    },
    "flattenVariants": {
      "default": false,
      "description": "Emit one `recordType: variant` row per variant instead of one product row with nested variants. Variant and product IDs are de-duplicated.",
      "prefill": false,
      "title": "One row per variant",
      "type": "boolean"
    },
    "includeDetails": {
      "default": false,
      "description": "Fetch `/products/{handle}.json` for each emitted product to add barcode, weight/grams and inventory fields. Costs one extra request per product and is off by default. Product mode always uses the detail endpoint.",
      "prefill": false,
      "title": "Include barcode, weight and inventory",
      "type": "boolean"
    },
    "maxItems": {
      "default": 1000,
      "description": "Hard cap on dataset rows across all stores. When `flattenVariants` is on, each variant row counts as one.",
      "maximum": 50000,
      "minimum": 1,
      "prefill": 3,
      "title": "Max records",
      "type": "integer"
    },
    "maxPages": {
      "default": 100,
      "description": "Page cap per product/collection feed. Each page holds up to 250 items and Shopify refuses offsets above 25,000, so the effective maximum is 100.",
      "maximum": 100,
      "minimum": 1,
      "prefill": 100,
      "title": "Max pages per feed",
      "type": "integer"
    },
    "maxPrice": {
      "description": "Drop products whose lowest variant price is above this value.",
      "maximum": 1000000,
      "minimum": 0,
      "title": "Maximum price",
      "type": "number"
    },
    "minPrice": {
      "description": "Drop products whose lowest variant price is below this value.",
      "maximum": 1000000,
      "minimum": 0,
      "title": "Minimum price",
      "type": "number"
    },
    "mode": {
      "default": "catalog",
      "description": "What to scrape. `catalog` walks the public product feed, `collection` walks specific collections, `product` fetches exact product handles, `search` uses the storefront search suggestions, `collections` lists collection metadata, and `auto` detects the type of each URL in `startUrls`.",
      "editor": "select",
      "enum": [
        "catalog",
        "collection",
        "product",
        "search",
        "collections",
        "auto"
      ],
      "enumTitles": [
        "Full store catalog",
        "Specific collections",
        "Specific products",
        "In-store search",
        "Collection directory",
        "Auto-detect from URLs"
      ],
      "prefill": "catalog",
      "title": "Mode",
      "type": "string"
    },
    "productHandles": {
      "default": [],
      "description": "Product handles or product URLs, e.g. `dark-roast-coffee` or `https://store.com/products/dark-roast-coffee`. Handles are applied to every store in `storeUrls`; a full product URL is applied to its own store.",
      "editor": "stringList",
      "prefill": [],
      "title": "Product handles (mode=product)",
      "type": "array"
    },
    "productType": {
      "description": "Keep products whose product type contains this text (case-insensitive).",
      "editor": "textfield",
      "title": "Product type contains",
      "type": "string"
    },
    "proxyConfiguration": {
      "description": "Leave empty to fetch directly. After a 403 or 429 block the actor retries through Apify's datacenter proxy (AUTO) automatically. If you set this, your configuration is used for that retry instead of AUTO; direct requests are still tried first.",
      "editor": "proxy",
      "title": "Proxy configuration (optional)",
      "type": "object"
    },
    "searchQuery": {
      "description": "In-store search term. Shopify's public suggest endpoint returns at most 10 products per store and has no pagination.",
      "editor": "textfield",
      "title": "Search query (mode=search)",
      "type": "string"
    },
    "startUrls": {
      "default": [],
      "description": "Store, collection or product URLs. In `auto` mode each URL selects its own job: `/products/<handle>` fetches one product, `/collections/<handle>` fetches a collection, any other store URL fetches the catalog. In the other modes these URLs add stores and, for collection/product modes, their handles.",
      "editor": "stringList",
      "prefill": [],
      "title": "Start URLs (mixed, optional)",
      "type": "array"
    },
    "storeUrls": {
      "description": "Store domains or URLs, e.g. `deathwishcoffee.com` or `https://www.allbirds.com`. Accepts custom domains and `*.myshopify.com`. A product or collection URL is reduced to its store for catalog/collections/search modes.",
      "editor": "stringList",
      "prefill": [
        "https://deathwishcoffee.com"
      ],
      "title": "Shopify store URLs",
      "type": "array"
    },
    "tag": {
      "description": "Keep products with at least one tag containing this text (case-insensitive).",
      "editor": "textfield",
      "title": "Tag contains",
      "type": "string"
    },
    "vendor": {
      "description": "Keep products whose vendor contains this text (case-insensitive).",
      "editor": "textfield",
      "title": "Vendor contains",
      "type": "string"
    }
  },
  "required": [
    "mode"
  ],
  "schemaVersion": 1,
  "title": "Shopify Store Catalog & Pricing Extractor Input",
  "type": "object"
}

Output schema

Published result links and storage locations. This schema is not an exhaustive contract for every field in a dataset record.

FieldTypeRequiredDescription
itemsstringoptionalDataset containing product, variant and collection records (filtered by recordType).
runSummarystringoptionalKey-value store record named OUTPUT with per-store OK/CAPPED/PARTIAL and error statuses.

Download the published output schema

View complete JSON
{
  "actorOutputSchemaVersion": 1,
  "description": "Dataset rows for products, variants or collections, plus a per-store run summary record.",
  "properties": {
    "items": {
      "description": "Dataset containing product, variant and collection records (filtered by recordType).",
      "template": "{{links.apiDefaultDatasetUrl}}/items",
      "title": "Shopify records",
      "type": "string"
    },
    "runSummary": {
      "description": "Key-value store record named OUTPUT with per-store OK/CAPPED/PARTIAL and error statuses.",
      "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys",
      "title": "Per-store run summary",
      "type": "string"
    }
  },
  "title": "Shopify Store Catalog & Pricing Extractor Output",
  "type": "object"
}

Dataset schema

Published dataset views and any declared field definitions. A view can select only a subset of the returned record fields.

Download the published dataset schema

View complete JSON
{
  "actorSpecification": 1,
  "views": {
    "collections": {
      "display": {
        "component": "table",
        "properties": {
          "handle": {
            "format": "text",
            "label": "Handle"
          },
          "productsCount": {
            "format": "number",
            "label": "Reported products"
          },
          "storeUrl": {
            "format": "link",
            "label": "Store"
          },
          "title": {
            "format": "text",
            "label": "Collection"
          },
          "url": {
            "format": "link",
            "label": "Collection URL"
          }
        }
      },
      "title": "Collections",
      "transformation": {
        "fields": [
          "recordType",
          "title",
          "handle",
          "productsCount",
          "storeUrl",
          "url",
          "scrapedAt"
        ]
      }
    },
    "detailed": {
      "display": {
        "component": "table",
        "properties": {
          "available": {
            "format": "text",
            "label": "Available"
          },
          "currency": {
            "format": "text",
            "label": "Currency"
          },
          "price": {
            "format": "number",
            "label": "Price"
          },
          "productId": {
            "format": "text",
            "label": "Product ID"
          },
          "scrapedAt": {
            "format": "text",
            "label": "Scraped at"
          },
          "sourceUrl": {
            "format": "link",
            "label": "Source"
          },
          "storeUrl": {
            "format": "link",
            "label": "Store"
          },
          "title": {
            "format": "text",
            "label": "Title"
          },
          "url": {
            "format": "link",
            "label": "Product URL"
          }
        }
      },
      "title": "Detailed",
      "transformation": {
        "fields": [
          "recordType",
          "productId",
          "id",
          "handle",
          "title",
          "url",
          "vendor",
          "productType",
          "tags",
          "description",
          "createdAt",
          "updatedAt",
          "publishedAt",
          "publishedScope",
          "price",
          "priceMin",
          "priceMax",
          "compareAtPrice",
          "compareAtPriceMax",
          "discountPercent",
          "onSale",
          "currency",
          "available",
          "variantCount",
          "availableVariantCount",
          "variants",
          "images",
          "imageDetails",
          "featuredImage",
          "options",
          "collectionHandle",
          "store",
          "storeUrl",
          "source",
          "sourceUrl",
          "scrapedAt"
        ]
      }
    },
    "overview": {
      "display": {
        "component": "table",
        "properties": {
          "available": {
            "format": "text",
            "label": "Available"
          },
          "compareAtPrice": {
            "format": "number",
            "label": "Compare at"
          },
          "currency": {
            "format": "text",
            "label": "Currency"
          },
          "price": {
            "format": "number",
            "label": "Price"
          },
          "productType": {
            "format": "text",
            "label": "Type"
          },
          "storeUrl": {
            "format": "link",
            "label": "Store"
          },
          "title": {
            "format": "text",
            "label": "Title"
          },
          "url": {
            "format": "link",
            "label": "Product URL"
          },
          "vendor": {
            "format": "text",
            "label": "Vendor"
          }
        }
      },
      "title": "Products",
      "transformation": {
        "fields": [
          "recordType",
          "title",
          "price",
          "currency",
          "compareAtPrice",
          "available",
          "vendor",
          "productType",
          "storeUrl",
          "url"
        ]
      }
    },
    "variants": {
      "display": {
        "component": "table",
        "properties": {
          "available": {
            "format": "text",
            "label": "Available"
          },
          "barcode": {
            "format": "text",
            "label": "Barcode"
          },
          "compareAtPrice": {
            "format": "number",
            "label": "Compare at"
          },
          "inventoryQuantity": {
            "format": "number",
            "label": "Inventory"
          },
          "price": {
            "format": "number",
            "label": "Price"
          },
          "productTitle": {
            "format": "text",
            "label": "Product"
          },
          "productUrl": {
            "format": "link",
            "label": "Product URL"
          },
          "sku": {
            "format": "text",
            "label": "SKU"
          },
          "title": {
            "format": "text",
            "label": "Variant"
          }
        }
      },
      "title": "Variants",
      "transformation": {
        "fields": [
          "recordType",
          "productTitle",
          "title",
          "sku",
          "price",
          "compareAtPrice",
          "currency",
          "available",
          "barcode",
          "inventoryQuantity",
          "storeUrl",
          "productUrl"
        ]
      }
    }
  }
}

Key-value schema

Published key-value store definitions, when provided by this build.

This definition is not present in the default published build. See the published README for available documentation.

Published README

Read the full published documentation

Shopify Store Catalog & Pricing Extractor

Turn any public Shopify storefront into a structured product feed. Point the actor at one store or a list of stores, choose a mode, and get prices, compare-at prices, variants, stock status, images, options, collection metadata and optional barcode/weight/inventory detail.

No Shopify account, admin API key, cookie or login is required. The actor reads the same public storefront JSON that a shopper's browser receives. Proxy setup is also optional: requests go direct first, and the actor switches to Apify's datacenter proxy automatically only after a real 403 or 429 block.

Modes

Mode What it does Source endpoint
catalog Full published product feed for every store /products.json?limit=250&page=N
collection Products from one or more collection handles/URLs /collections/{handle}/products.json
product Exact products by handle or product URL /products/{handle}.json
search In-store product search, up to 10 results per store /search/suggest.json
collections Collection directory with titles, handles and reported counts /collections.json?limit=250&page=N
auto Mixed startUrls; each URL selects its own job Detected per URL

auto treats /products/<handle> as a product job, /collections/<handle> as a collection job, /collections as a collection-directory job, and any other store URL as a catalog job.

Input

Field Type Default Notes
mode select catalog One of the six modes above.
storeUrls string list - Store domains or full URLs; *.myshopify.com is supported. The prefill uses deathwishcoffee.com.
startUrls string list empty Mixed store/product/collection URLs. Used mainly by auto.
collectionHandles string list empty Handles or collection URLs for mode=collection.
productHandles string list empty Handles or product URLs for mode=product.
searchQuery string empty Required for mode=search.
currency string store ISO 4217 code such as USD or EUR; store uses /meta.json.
country string empty Optional ISO-2 country passed as ?country=.
vendor string empty Case-insensitive vendor substring filter.
productType string empty Case-insensitive product-type substring filter.
tag string empty Keep products with a tag containing this substring.
minPrice number empty Drop products below this lowest-variant price.
maxPrice number empty Drop products above this lowest-variant price.
available select any any, available or unavailable.
maxItems integer 1000 1-50,000 dataset rows; variant rows count individually when flattening.
maxPages integer 100 1-100 pages per feed. Shopify refuses offsets above 25,000.
includeDetails boolean false One extra /products/{handle}.json request per product for barcode, weight and inventory.
flattenVariants boolean false Emit one recordType: variant row per variant instead of nested product rows.
proxyConfiguration object empty Direct requests are always tried first. After a 403/429 the actor retries via AUTO, or via this configuration if provided.

The smallest useful run is also the daily prefill: deathwishcoffee.com, three records, no detail enrichment.

Examples

Full catalog from one store:

{
  "mode": "catalog",
  "storeUrls": ["https://deathwishcoffee.com"],
  "currency": "USD",
  "maxItems": 100
}

Collection and sale filters:

{
  "mode": "collection",
  "storeUrls": ["allbirds.com"],
  "collectionHandles": ["mens-sale"],
  "maxPrice": 100,
  "available": "available",
  "flattenVariants": true
}

Exact products by URL:

{
  "mode": "product",
  "startUrls": [
    "https://deathwishcoffee.com/products/valhalla-java",
    "https://deathwishcoffee.com/products/dark-roast"
  ],
  "includeDetails": true
}

Search a set of stores:

{
  "mode": "search",
  "storeUrls": ["allbirds.com", "rothys.com"],
  "searchQuery": "wool runner",
  "maxItems": 20
}

Mixed URLs in one run:

{
  "mode": "auto",
  "startUrls": [
    "https://allbirds.com",
    "https://allbirds.com/collections/mens",
    "https://allbirds.com/products/wool-runner"
  ]
}

Product output

Product records use recordType: "product" and contain:

  • productId and legacy id, handle, title, url
  • vendor, productType, tags
  • bodyHtml (raw) and description (plain text)
  • createdAt, updatedAt, publishedAt, publishedScope
  • price, priceMin, priceMax, compareAtPrice, compareAtPriceMax, discountPercent, onSale, currency
  • available, variantCount, availableVariantCount
  • variants: variant id, title, SKU, price, compare-at price, availability, option values and named options
  • images (URL list), imageDetails (URL/width/height/position), featuredImage
  • options: option names and their values
  • store: {url, name, currency}, plus legacy storeUrl
  • source, sourceUrl, scrapedAt, recordType

With includeDetails enabled, variants can also contain barcode, weight, weightUnit, grams, inventoryQuantity, inventoryPolicy, inventoryManagement, quantityRule, requiresShipping and taxable. Inventory quantities are only emitted when the store exposes inventory_management and the value is not Shopify's 999999 untracked sentinel.

When the detail endpoint omits the available flag but publishes inventory management, availability is derived from stock: positive quantity, or inventory_policy=continue, means available. Without published inventory the flag stays omitted rather than being guessed.

Variant output

With flattenVariants: true, each variant becomes its own row with recordType: "variant", plus the parent productId, productHandle, productTitle, productUrl, vendor and productType. Product and variant IDs are de-duplicated.

Collection output

mode=collections emits recordType: "collection" rows with collectionId, handle, url, title, description, productsCount (the store's reported count), timestamps, image, store, sourceUrl and scrapedAt.

Run summary

The dataset stays clean: products, variants and collections only. A separate key-value record named OUTPUT holds the run summary with each store's status, HTTP code, pages fetched, record counts, detail failures and capped flag.

Per-store statuses:

Status Meaning
OK The feed was read normally.
CAPPED The page cap or Shopify's 25,000-offset ceiling was reached.
PARTIAL Some pages or detail requests failed after data was collected.
PASSWORD_PROTECTED The store returned 401.
FROZEN The store returned 402.
BLOCKED The store returned 403, including after the automatic proxy retry.
NO_JSON The store returned 404 or a non-JSON response.
RATE_LIMITED The store returned 429 after retries.

If every store fails, the run fails with the per-store map instead of finishing silently with an empty dataset. If some stores succeed, the run succeeds and the summary reports the failures.

Pagination notes

  • 250 products per page, the Shopify maximum.
  • Short pages are not the end of the feed. The walker continues until an empty page, an error, maxPages, or the 25,000-offset ceiling.
  • Products are de-duplicated by store and product ID; variants by store and variant ID.
  • maxItems is a global cap across all stores and jobs.

Currency

Shopify localizes prices by IP. The actor pins ?currency= on every list, collection and product request. The default store setting reads the store's base currency from /meta.json; an explicit ISO code overrides it. Each product records the currency reported by its variants when available.

FAQ

Do I need a Shopify account or API key? No. The actor only reads public storefront JSON.

Do I need a proxy? No. Direct requests are tried first. Apify's datacenter proxy is used only after a 403 or 429.

Why does search return at most 10 products per store? That is Shopify's public suggest endpoint limit. There is no pagination on it. Use catalog or collection mode for larger feeds.

Why is the product count lower than the collection page says? Collection product endpoints expose only purchasable/published products, while products_count counts more. The reported count is kept as productsCount and is not treated as ground truth.

Why are some inventory fields missing? Inventory is a merchant setting. Stores that do not publish inventory_quantity cannot be forced to reveal it, and the untracked 999999 sentinel is never emitted as stock.

What happens on password-protected stores? Their public JSON returns 401. The actor reports PASSWORD_PROTECTED and does not retry with a proxy.

Why did a large store stop at 25,000 products? Shopify rejects page offsets above 25,000 on /products.json. Split large catalogs by collection to go deeper.

Are descriptions HTML or plain text? Both are available: bodyHtml keeps the original markup and description is a plain-text version.

Is this affiliated with Shopify? No. It is an independent third-party tool that reads publicly available storefront data.

Limits

  • Password-protected stores are not scrapable through the public JSON.
  • Stores that disable the JSON endpoints return NO_JSON.
  • Collection membership is not exposed publicly; the actor emits collection records instead of inventing per-product membership.
  • Search is capped at 10 products per store.
  • Inventory counts are available only where the store publishes them.
  • The product feed is limited to the newest 25,000 products per store/feed.

Python quickstart

Use a schema-derived input and the Python standard library to start a run. An Apify account and token are required for execution.

Open the step-by-step Python guide →

Source build rfdRrnEMOeqFiOPHE. Documentation is regenerated from the default published build; private drafts are not included.