Support » Knowledge Base » Shop-Script » Tips & Instructions »
To prevent the server from being overloaded by search engine crawlers (e.g., Google, Yahoo!, etc.) indexing your website, it is often advisable to use a robots.txt file containing special directives limiting the range of URLs which crawlers are allowed to index. Besides, this approach allows to disallow indexing of pages containing low-relevant information by search engines.
This method can also be used with WebAsyst Shop-Script. Since the URLs of storefront pages are mostly different depending on whether SEO-friendly URLs are enabled in WebAsyst Installer, below are offered recommendations both for online stores with SEO-friendly URLs enabled and those where this option is turned off.
SEO-friendly URLs enabled | SEO-friendly URLs disabled | Comments |
| User-agent values | ||
User-agent: * | List of User-agent values of the search engine crawlers, further directives will be applied to. To apply the contents of the file to all crawlers, use the * character. | |
| For optimal indexing | ||
| Sitemap: http://mydomain.com/sitemap.php | URL of the Sitemap file generated in backend section "Tools -> Sitemaps". | |
| To reduce server load | ||
| Disallow: /pricelist/ | Disallow: /*ukey=pricelist | Price list page |
| Disallow: /search/ | Disallow: /*ukey=search* | Search results page |
| Disallow: /*sort=* | Disallow: /*sort=* | Products sorting results page |
| Disallow: /*/all/ | Disallow: /*show_all=yes | Page displaying all products in a category |
| To prevent indexing of irrelevant information | ||
| Disallow: /auth/ | Disallow: /*ukey=auth | Customer account login URL |
| Disallow: /cart/ | Disallow: /*ukey=cart | Shopping cart with products |
| Disallow: /checkout/ | Disallow: /*ukey=checkout | Order checkout page |
| Disallow: /feedback/ | Disallow: /*ukey=feedback | Feedback form page |
| Disallow: /installer/ Disallow: /published/wbsadmin/ | Disallow: /installer/ Disallow: /published/wbsadmin/* | WebAsyst Installer |
| Disallow: /login/ | Disallow: /login/ | WebAsyst user (store administrator)login page |
| Disallow: /logout/ | Disallow: /*ukey=logout | Customer logout link |
| Disallow: /myaccount/ | Disallow: /*ukey=office | Customer account page |
| Disallow: /order_status/ | Disallow: /*ukey=order_status | Order status view link |
| Disallow: /published/SC/html/scripts/ | Disallow: /published/SC/html/scripts/ | Various auxiliary scripts (e.g., RSSfeed) |
| Disallow: /register/ | Disallow: /*ukey=register | Customer registration page |
| Disallow: /remind_password/ | Disallow: /*ukey=remind_password | Customer password reminder link |
| Disallow: /*view=* | Disallow: /*view=* | Various auxiliary links (e.g., page printed version) |
| Disallow: /*/reviews/ | Disallow: /*ukey=discuss_product | Links to product discussion pages |
| Disallow: /facebook/ | Disallow: /*store_mode=facebook* | Simplified storefront version used for integration with Facebook |
| Disallow: /order_status/ | Disallow: /*order_status | Order status viewing pages |
| Disallow: /order_history/ | Disallow: /*ukey=order_history | Order history listing pages |
| Disallow: /print_form/ | Disallow: /*print_form* | Printer-friendly versions of various documents |
The load caused by search crawlers can also be reduced using Crawl-delay directive. It sets the minimum period in seconds after which a crawler is allowed to download the next website page after a previous page has been downloaded. An example of use:
Crawl-delay: 10
Crawl-delay directive should be specified just after lines containing Disallow or Allow directives.
If you have Shop-Script installed on your own or a web-hosting provider's server, you can upload a robots.txt file to the root directory of your online store using an FTP client or a file manager in your web-hosting control panel.
If you have created an online store using the WebAsyst online services, please email your robots.txt file to the WebAsyst support team at support@webasyst.net.
Login