OpenParable
Skip to content
← Back to skills

Robots Meta Conflict

Business
robots-meta-conflict

Robots Meta Conflict is an Agent Skill for business published in the thedaviddias/Front-End-Checklist repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when applies to sites that use both robots.txt disallow rules and meta robots noindex tags. Use when investigating why deindexed pages still appear in search results.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/thedaviddias/Front-End-Checklist/blob/HEAD/skills/robots-meta-conflict/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/robots-meta-conflict, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Robots Meta Conflict

Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt — Google cannot read the noindex directive it cannot fetch.

Quick Reference

  • If a page is blocked in robots.txt, crawlers never fetch it and therefore never read its noindex meta tag
  • To properly deindex a page, use <meta name="robots" content="noindex"> without blocking it in robots.txt
  • If you block a URL in robots.txt, Google may still show it in results using anchor text from backlinks
  • Audit pages that need to be deindexed by confirming they are crawlable but carry noindex

Check

Cross-reference the site's robots.txt disallow paths against pages carrying <meta name="robots" content="noindex">. Flag any page where the URL matches a Disallow rule AND has a noindex tag — the noindex will never be seen by crawlers.

Fix

For pages that must be deindexed: remove the robots.txt Disallow rule for that URL and keep the noindex meta tag. For pages that must simply be uncrawled without appearing: keep the Disallow rule and remove the noindex tag (understanding Google may still show the URL).

Explain

Explain why a page blocked by robots.txt can still appear in Google search results, why the noindex meta tag inside a blocked page is never processed, and how to correctly deindex a page.

Code Review

Review metadata generation, rendered HTML, structured data, and response headers related to Robots Meta Conflict. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.

For full implementation details, code examples, and framework-specific guidance, see references/rule.md.

Rule page: https://frontendchecklist.io/en/rules/seo/robots-meta-conflict