How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.
Totally agreed—this is a design choice that basically comes from our agent architecture, and the codegen-based architecture that we think will likely proliferate for web agent tasks in the future. We provide Tarsier's text/screenshot to an LLM and have it write code with generically written selectors rather than the naive selectors that Tarsier assigns to each element.
It's sort of like when you (as a human) write a web scraper and visually click on individual elements to look at the surrounding HTML structure / their selectors, but then end up writing code with more general selectors—not copypasting the selectors of the elements you clicked.
Great question, also situations where you have multiple CTAs with similar names/contexts on a page is still something I see LLM based automation struggle with.
Hm, not sure I follow why those situations would be especially difficult? Regarding website changes, the nice thing about using LLMs is that we can simply provide the previous scraper as context and have it regenerate the scraper to "self-heal" when significant website changes are detected.
Comments
How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.
Totally agreed—this is a design choice that basically comes from our agent architecture, and the codegen-based architecture that we think will likely proliferate for web agent tasks in the future. We provide Tarsier's text/screenshot to an LLM and have it write code with generically written selectors rather than the naive selectors that Tarsier assigns to each element.
It's sort of like when you (as a human) write a web scraper and visually click on individual elements to look at the surrounding HTML structure / their selectors, but then end up writing code with more general selectors—not copypasting the selectors of the elements you clicked.
Ooh that's a very neat approach, great idea! Chains of thought across abstraction layers. Definitely worth a blog post I reckon.
Good luck!
Thanks! We might put out a paper about it with some Carnegie Mellon collaborators this summer.
You might like to look at https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf
ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing.
Great question, also situations where you have multiple CTAs with similar names/contexts on a page is still something I see LLM based automation struggle with.
Hm, not sure I follow why those situations would be especially difficult? Regarding website changes, the nice thing about using LLMs is that we can simply provide the previous scraper as context and have it regenerate the scraper to "self-heal" when significant website changes are detected.