I've been doing adjacent work for my search engine[1], and found substack to annoyingly be one of the sites that employ bot mitigation for its RSS endpoints. If you fetch at a very low rate it works fine, but for these types of bulk retrieval.
Substack is also a bit of a pain to integrate with because they have zero useful contact information and direct all inquiries to a chatbot that is beyond useless, makes it so you have to guess how they want you to interact with their servers since there is nobody to answer questions.
Comments
I've been doing adjacent work for my search engine[1], and found substack to annoyingly be one of the sites that employ bot mitigation for its RSS endpoints. If you fetch at a very low rate it works fine, but for these types of bulk retrieval.
Substack is also a bit of a pain to integrate with because they have zero useful contact information and direct all inquiries to a chatbot that is beyond useless, makes it so you have to guess how they want you to interact with their servers since there is nobody to answer questions.
[1] Preview of my take of the idea: https://mastodon.social/@marginalia/113670235590972416