Selecting Icons

[reworkd]

Selecting Icons

[asim-shrestha]

Hey @llermaly, because Tarsier is typically for text parsing, we currently don't support icons. (Not sure how we'd best go about it in the future either)

For images it is quite straight forward. There is a page_to_image function in tarsier that will return the bytes of the image. Then you can pass that in as an image to a vision language model likeGPT-4-V. Let me know if that helps!

[asim-shrestha]

If you still want to go the text approach, you can manually find out which of those elements may be related to a search icon (through image name, or some other tag in the html itself) and provide that information in the prompt as well