Feed error
Item disapprovedPage not crawlable due to robots.txt
A "page not crawlable due to robots.txt" error means the store's robots.txt blocks Googlebot or Googlebot-Image from the product page or image, so Google cannot check it and disapproves the item. On Shopify it usually traces back to a customised robots.txt.liquid template or an app that added blocking rules.
How this appears in your account
The exact wording the channel shows, so you can match what you are looking at.
- Desktop page not crawlable due to robots.txt
- Mobile page not crawlable due to robots.txt
- Image not crawlable due to robots.txt
- Unable to do quality & policy checks on product pages
What it affects
- Channels
- Google Shopping
- Attributes
- link image_link
- Scope
- The whole feed
Why it happens on Shopify
The robots.txt template was customised
Shopify generates a sensible robots.txt by default and lets themes override it with a robots.txt.liquid template. Rules added there to hide pages from search can catch product URLs or block Google's crawlers entirely.
An app or developer added a blanket disallow
SEO tools and developers sometimes add a broad Disallow for a user agent, or for a path such as /products or a collection prefix, to manage crawl budget. Any rule matching the feed's links blocks the landing pages they point to.
Googlebot-Image is blocked but Googlebot is not
Images are fetched by a separate crawler. A rule targeting Googlebot-Image produces image errors on products whose pages crawl perfectly well.
Links point at blocked URL variants
Links that carry tracking parameters or go through a collection path can match a disallowed pattern even when the plain product URL is allowed.
How to fix it
-
Read the live robots.txt
Open /robots.txt on the domain used in the feed's links and look for any Disallow under User-agent: *, Googlebot or Googlebot-Image that matches a product or image URL.
-
Allow Googlebot and Googlebot-Image
Google's guidance is that both crawlers must be able to reach landing pages and images. Remove or narrow the blocking rule in robots.txt.liquid, or delete the template to fall back to Shopify's default.
-
Submit plain product URLs
Map link to the canonical product URL rather than a collection path or a URL with parameters that robots.txt might disallow.
-
Confirm with the URL inspection tool
Test an affected product URL in Search Console, using the smartphone fetch type for the mobile variant of the error, to confirm Google is no longer blocked.
-
Allow time for the recrawl
Once crawling is allowed and items are reprocessed, Google says the errors should clear within a couple of days. Changes can take 24 to 72 hours to show on the Needs attention page.
How feedlab handles this
feedlab builds link from each product's canonical Shopify URL rather than a collection or parameterised path, which keeps feed links clear of most path-based disallow rules. robots.txt itself lives in the Shopify theme, so the blocking rule still has to be removed there.
Frequently asked questions
Can I edit robots.txt on Shopify?
Yes, through a robots.txt.liquid template in the theme. Shopify's default file allows product pages, so if you have no custom template the cause is more likely elsewhere, such as links pointing at disallowed paths.
Which Google crawlers need access?
Google's guidance names Googlebot for landing pages and Googlebot-Image for images. Blocking either produces this error for the part it cannot fetch.
Will my products disappear immediately?
Not always. Google says that after a recent robots.txt change, products can keep showing with previously approved images for up to 30 days, which gives time to fix the file before they are disapproved.
Related errors
- Landing page not working An unavailable landing page error means Google could not load the page at the product's link when it crawled it, so the item is disapproved. On Shopify it usually means the product was unpublished, archived or renamed after the feed was built, or the storefront was password protected or briefly down.
- Image link cannot be crawled An image crawl failure means the channel could not retrieve the file at image_link, or retrieved it and judged it unusable. On Shopify the URL itself is rarely wrong, so the cause is usually a rate limit hit during a first full crawl, a temporary CDN response, or an image whose content breaks policy rather than one that is genuinely missing.
- Mismatched domains A "Mismatched domains" error means the product's link uses a different domain from the website registered in Merchant Center, so the item is disapproved. On Shopify it usually means the feed builds links on the myshopify.com address or an old domain while the account is registered to the store's custom domain.
Primary sources
Channel specifications change. These are the official documents this page is based on.