Comment on Ask HN: Why isn't Google indexing information about the AT Protocol?Comments−fragmede2mohttps://firehose.directory/robots.txt isn't one, but comes up in Google despite that.https://atproto.at/robots.txt is good and findable with a direct searchhttps://atproto.wiki/robots.txt is empty but findable on Google.https://pulsar.feeds.blue/robots.txt is 404 and thus not indexed by Google.https://leaflet.pub/robots.txt is good. A logged in google search for this finds this for me.Tell the people at https://pulsar.feeds.blue to fix their robots.txt if they want to get indexed.
Comments
https://firehose.directory/robots.txt isn't one, but comes up in Google despite that.
https://atproto.at/robots.txt is good and findable with a direct search
https://atproto.wiki/robots.txt is empty but findable on Google.
https://pulsar.feeds.blue/robots.txt is 404 and thus not indexed by Google.
https://leaflet.pub/robots.txt is good. A logged in google search for this finds this for me.
Tell the people at https://pulsar.feeds.blue to fix their robots.txt if they want to get indexed.