How modify the file robot.txt???

[Tony7]

Hi, is there the posibility to modify the file robot.txt??

Now my file is like this:

User-agent: *
Disallow: /

and i want permit all...

Thanks

[Adam7]

---

@LisaM wrote:

Hi I am having an issue when I search my site in Google it says it doesn not contain more info because of the robot txt setting. I want to make sure that info about my site is showing in a Google search not simply the link.

---

It looks like Google is displayed outdated, cached info for your site.  They should update this automatically when they re-crawl your site again, although if you've verified your site with Google you can use Google's "Fetch as Google" option to try and speed that up.

http://hc.weebly.com/hc/en-us/articles/201723433-Verify-Site-with-Search-Engines

https://support.google.com/webmasters/answer/6065812?hl=en

[lvtheresa]

Sitemap: http://www.medisinfo2u.com/sitemap.xml

User-agent: *
Disallow: /

my website show this , but still there is no description come out when i search my site on google. 

please advice

thanks

Lv

[JasonS1]

Same problem. No matter how many times I update, it just won't change. I am also unable to verify on google webmaster.

[Adam7]

What site are you having trouble with, @JasonS, and what kind of changes are you making that aren't showing up with your robots.txt file?

[weeblyjbkfan]

SO YOU are telling me I need to Slide over the button to the (RIGHT)so it is highlighted dark blue in color to block my site from search engines? (has anyone heard of on or off Smiley Happy) That does not make sense at all. Please reply and let me know if I am understanding you. If I am please rename the name of the button. I am an ecommerce site and common sense would tell me I would never want to hide my site from search engines.

Help me fix these errors. Thanks in advance. Desperate weebly user

[weeblyjbkfan]

I am really confused... I have an ecommerce store.  With 1500 plus blocked resources on google search console.  blocks several that look like this:  /ajax/api/JsonRPC/Commerce/?Commerce[Checkout::getMiniCart,Product::isInStock,Product::isInStock,Product::isInStock,Product::isInStock,Product::isInStock,Product::isInStock,Product::isInStock,Product::isInStock,Product::i. 

SO YOU are telling me I need to Slide over the button to the (RIGHT)so it is highlighted dark blue in color to block my site from search engines?  (has anyone heard of on or off ) That does not make sense at all.  Please reply and let me know if I am understanding you.  If I am please rename the name of the button.  I am an ecommerce site and common sense would tell me I would  never want to hide my site from search engines.

Help me fix these errors.  Thanks in advance.  Desperate weebly user

[Adam7]

Yes, you want it to look like this if you want your site indexed by Google:

Also, anything in /ajax will be blocked regardless of that because it's backend scripting, and not the actual URLs of your products.

[weeblyjbkfan]

thank you.  OK so I just leave all of the errors then right?  and now google says my sitemap is blocking them from crawling it.  do I just need to wait 

[weeblyjbkfan]

It (sitemap) looks nothing like it did before....

[Adam7]

What's the address of your site, @weeblyjbkfan? It might help for me to see it myself.

[YouNet1]

Hi Adam,

I've a problem with my robot.txt file of the website https://www.erasmustrainingcourses.com

Among the many lines (that I want) there as well the following ones that I do not want (since those page do not even existany more) and that I cannot delete, can you help?

Disallow: /ict-and-new-technologies.html

Disallow: /http://www.erasmustrainingcourses.com/teachers-and-adult-education-staff-support.html

Disallow: /inclusion-and-diversity.html

Disallow: /innovative-teaching-methods1.html

Disallow: /internationalisation-and-languages.html

Disallow: /soft-skills.html

[Adam7]

Just to clarify, these were pages you had and had blocked from indexing, but then deleted the pages? It doesn't really hurt anything to have those in your robots.txt file, although you could make new pages with the same names, leave the option to block indexing for each unchecked, publish, then delete the pages. That should update the robots.txt file to remove those lines.

[AFM1]

if the site is not seen on the search engine does this not mean the searchers can not find you?
is not the whole purpouse of the page for people to find you then you finding them?

there must be a way to write a JS code to overcome this problem example:

find robot.txt for this weebly page 

up on finding the tobot text /* blank all data

divert to new location (x) 

provide this entry as the Robot text to your search engine.

I am not someone that writes code but I am sure some of you can write something along this lines 

the internet is no more than on off switches example type the word "orange" and everything that is not a colour or a fruit that represents an orange is blocked, right so now you are left with 2 items a color or a fruit right?

then you add a description that may read a drink for early in the morning, fruit at 1/2 time or a pigment to paint the wall etc the indexing based on the descrition can determine if one is looking for the fruit or the color paint. thus in this way I think the overright may do the same if the item is blocked what is the second option? 

the internet was more basic back in the 80s we had one word back then "PORN"     LMAO

[stumbledonto]

Wouldn't you want google to index your store pages for items that you are going to sell? Why would you disallow ajax and apps for that reason? Can it be changed?

[Marier]

Hi there!

I have a similar issue, I would like my website to be crawled and indexed in Google and for this I'd like to change my robots.txt because it is currently showing:

User-agent: *
Disallow: / 

However on my page settings > SEO, I have not hidden the page form search engines. 

I found this page https://www.weebly.com/app/help/us/en/topics/verify-your-site-with-search-engines very useful to link my site to GSC but nothing about robots.txt...

Thanks for your help,

Marie

[Adam7]

Thanks for your post, @Marier. Open your site in the website editor and go to Settings > SEO. It looks like you've probably checked the option to hide the site from search engines. Uncheck that, then publish your site again and it will update your .htaccess file.

[Marier]

Hi @Adam,

Thank you for your quick answer.

 I actually unticked that box from the beginning ... Is there something else I am missing? 

[Adam7]

It looks like you are looking at the option for individual pages, rather than the for the whole website. Go to the Settings tab using the link at the top of the editor, then go to the SEO section and look for the option there.

[Marier]

Great, thank you!

[Adam7]

Not a problem!

[PhVermeersch]

Hello,

I have a similar issue,  my robots.txt  is currently showing:

User-agent: Nerdybot
Disallow: / 

I've tried hiding in SEO and showing again, but nothing changes. The pages I explicitly want to hide are hidden on page level, but other pages which should get indexed are refused in Google search console.

https://www.digitale-infodagen.be/ 

Thank you in advance!

Philippe Vermeersch