User-agent: * Disallow: /*? Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /wishlist/ Disallow: /admin/ Disallow: /catalogsearch/ Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /review/product/ Disallow: /sendfriend/ Disallow: /enable-cookies/ Disallow: /LICENSE.txt Disallow: /LICENSE.html Disallow: /skin/ Disallow: /js/ Disallow: /directory/ Lets consider each groups of commands separately. Stop crawling user account and checkout pages by search engine robot: Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ Blocking native catalog and search pages: Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Sometimes Webmasters block pages with filters.. Disallow: /*?dir* Disallow: /*?dir=desc Disallow: /*?dir=asc Disallow: /*?limit=all Disallow: /*?mode* More reasonable to use canonical tag on these pages. Blocking CMS directories. Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /phpserver/