jhv
07-30-2020
09:20 AM
Thread Options
- Subscribe to RSS Feed
- Mark Thread as New
- Mark Thread as Read
- Float this Thread for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Google Index Allowed??
Hi,
I submitted an indexing request with Google. Below is what shows in my robots.txt on my site. Will this allow google to index/crawl my site?
User-agent: NerdyBot Disallow: / User-agent: * Disallow: /ajax/ Disallow: /apps/
Reply
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report
1 REPLY 1
marketgoo
07-31-2020
05:35 PM
Thread Options
- Subscribe to RSS Feed
- Mark Thread as New
- Mark Thread as Read
- Float this Thread for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Hi @jhv , yes Google can crawl your site. Weebly automatically generates the robots.txt file for you, and what you're seeing is that it's blocking the "nerdybot" crawler from crawling your site, but all other search engines (like Google, Bing) can crawl and index your site without issue.
Reply
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report