Open up forums.topcoder.com to robots

[dmessing]

We should allow Google to index our forums (obviously still following our rules and regulations

[birdofpreyru]

@ajefts Where do we have the Forum related code? I guess, this issue is a simple modification of robot.txt in there? What are the rules and regulations to follow, is it necessary to still restrict robots access to some info?

[ajefts]

I believe this is the robots.txt that forums is using: https://apps.topcoder.com/robots.txt

But that looks ok...

[ajefts]

I just added apps.topcoder.com to our google webmaster console so we can see what's going on. It appears to be crawling. It might help us to describe the page structure to google in here to help it crawl and index....

[birdofpreyru]

Do you mean to setup an automatic generation of sitemap for the forums, which can be then fed to Google.Webmaster to improve the crawling?

[ajefts]

I connected the apps.topcoder.com domain to the webmaster console and I started to describe the page structure to it with page highlighting. I'll monitor it's indexing activity...

…

On Wed, Jan 18, 2017 at 3:26 PM, birdofpreyru ***@***.***> wrote: Do you mean to setup an automatic generation of sitemap for the forums, which can be then fed to Google.Webmaster to improve the crawling? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#933 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADorrlRhU7dVwYxzJXRRxrn3adF9GDcOks5rTnVcgaJpZM4LmeRS> .