CUTCODEDOWN
Minimalist Semantic Markup

Welcome Guest
Please Login or Register

If you have registered but not recieved your activation e-mail in a reasonable amount of time, or have issues with using the registration form, please use our Contact Form for assistance. Include both your username and the e-mail you tried to register with.

Pages: [1]

Author Topic: /for_others/ content is being indexed at Google  (Read 232 times)

fgm

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +5/-0
/for_others/ content is being indexed at Google
« on: 24 Feb 2023, 06:05:09 pm »
Hello, I've noticed the contents of /for_others directory are getting indexed at Google.

I'd add Disallow: /for_others/ to the current robots.txt file:
Code: [Select]
User-agent: *
Disallow: /for_others/
Logged

Jason Knight

  • Administrator
  • Hero Member
  • *****
  • Posts: 919
  • Karma: +171/-1
    • CutCodeDown -- Minimalist Semantic Markup
Re: /for_others/ content is being indexed at Google
« Reply #1 on: 26 Feb 2023, 09:06:27 am »
Logged
I'll fix every flaw, I'll break every law, I'll tear up the rulebook if that's what it takes. You will see, I will crush this cold machine.

fgm

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +5/-0
Re: /for_others/ content is being indexed at Google
« Reply #2 on: 26 Feb 2023, 09:35:27 am »
From RFC 9303 Section 2.3:
Quote
The rules MUST be accessible in a file named "/robots.txt" (all
   lowercase) in the top-level path of the service.
You can check the content is indexed using the search query site:cutcodedown.com/for_others
« Last Edit: 26 Feb 2023, 09:40:47 am by fgm »
Logged

Jason Knight

  • Administrator
  • Hero Member
  • *****
  • Posts: 919
  • Karma: +171/-1
    • CutCodeDown -- Minimalist Semantic Markup
Re: /for_others/ content is being indexed at Google
« Reply #3 on: 26 Feb 2023, 06:54:56 pm »
Ok, it used to work per directory... and since when has it even had a RFC?

Oh... since 2016. Must have missed that.  And it looks like that change wasn't made official until 2020... so yeah, totally missed a change in how it works. Disagrees with Koster's 1994 original which I've been relying on for decades...

Can't say I'm surprised. I know a lot of people started bitching about error log 404's from search engines accessing every blasted directory. Kind of like if you don't put a .ico in the root.
Logged
I'll fix every flaw, I'll break every law, I'll tear up the rulebook if that's what it takes. You will see, I will crush this cold machine.

mmerlinn

  • Jr. Member
  • **
  • Posts: 77
  • Karma: +9/-0
  • Nutcake
Re: /for_others/ content is being indexed at Google
« Reply #4 on: 26 Feb 2023, 08:15:41 pm »
Looks like Jason is too busy making money cleaning up after nitwits ignoring ADA standards to keep up with rules that keep changing, oftentimes for no apparent reason.l
Logged
The soul purr pus of a spell cheque cur is two valley date hour ignore ants.
Pages: [1]
 

SMF spam blocked by CleanTalk

Advertisement