Comment on Hister – A private, full content search index that you controlComments−vivzkestrel21d- lots of stupid questions to the author from a guy who has no idea about search engines- let us say I want to index every blog ever listed on HN- should be a small subset of the 400 billion pages out there on the internet no?- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?- how do you handle proxies? rotation? are there libraries you recommend for this?- what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind?- what are the filetypes you collect? only html or media as well?- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?- what is the frequency at which you refresh each page? once a day? once a week? something else?- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?−9935c101ab17a6617dA lot of these can be answered by checking the documentation: https://hister.org.Others could be answered by examining the source code, as it is open source.It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.
Comments
- lots of stupid questions to the author from a guy who has no idea about search engines
- let us say I want to index every blog ever listed on HN
- should be a small subset of the 400 billion pages out there on the internet no?
- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?
- how do you handle proxies? rotation? are there libraries you recommend for this?
- what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind?
- what are the filetypes you collect? only html or media as well?
- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?
- what is the frequency at which you refresh each page? once a day? once a week? something else?
- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?
- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?
A lot of these can be answered by checking the documentation: https://hister.org.
Others could be answered by examining the source code, as it is open source.
It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.