If you’re talking about SSL cert logs, that excludes a ton of web content.
I guess it depends on what you are looking for?
It's far less brute force than trying to randomly generate hostnames and then do lookups, and your solution also misses everything that is just random alphanumeric combinations, of which there are tons.
And you haven’t shown me that you can even get all those for free anyway
Yeah, but because it's an LLM it makes very good guesses. It's not so random, it's seen URLs in training data.
One of the points of the experiment was to see how much of a shortcut using an LLM could give me, as the developer.
I've done other LLM search experiments where I basically just prompt to "predict 25 search results for this query" and it's surprisingly good. You can also run a check to verify the URLs to splice hallucinations out of the result, but yeah, LLM is already doing a lot of the work for search.
Comments
Another option: https://www.certkit.io/tools/ct-logs/
Looks like they only give you 100 on a trial account
Sorry, let me be more obvious.
I am not pointing to a solution, but a pointer to the solution. Showing what is possible. The solution is pulling/parsing the CT logs yourself.
If you’re talking about SSL cert logs, that excludes a ton of web content.
And you haven’t shown me that you can even get all those for free anyway
I guess it depends on what you are looking for?
It's far less brute force than trying to randomly generate hostnames and then do lookups, and your solution also misses everything that is just random alphanumeric combinations, of which there are tons.
Free? Maybe.
Yeah, but because it's an LLM it makes very good guesses. It's not so random, it's seen URLs in training data.
One of the points of the experiment was to see how much of a shortcut using an LLM could give me, as the developer.
I've done other LLM search experiments where I basically just prompt to "predict 25 search results for this query" and it's surprisingly good. You can also run a check to verify the URLs to splice hallucinations out of the result, but yeah, LLM is already doing a lot of the work for search.