Enable deep searching of specific URLs

[miurla]

Enable deep searching of specific URLs

[albertdbio]

Firecrawl is pretty slow when I tested it
Is this fine? I'm sure the user would understand as long as we provide a spinner

[albertdbio]

Firecrawl uses playwright + turndown 😆
We can also use this one https://jina.ai/reader

I'll give it a shot, already got the researcher able to refer to the previous sources it pulled, just that the types are a bit tricky.
Just need to integrate those API's

I'll work on this one first and I'll see if I can get back to video section next week, it's been busy at work

[miurla]

Thank you. I was stuck with the chat history feature. I'll be able to start working on a different task soon.

[albertdbio]

Hey @miurla, fixed the types related to enabling follow up conversations and also found a way to enable function calling with groq so now I'm moving forward with adding FireCrawl, planning to push a PR by Sunday!

[albertdbio]

@miurla not using the groq sdk, it turns out the vercel ai sdk needed to be upgraded to a minor version. That minor version set's content as optional in the OpenAI zod schema. Groq doesn't return content when doing function calls so it would throw an invalid response error. Going to test it out with the inquire agent to verify that it's working.
Sounds good, I'll use main!

[albertdbio]

Got caught up with work over the weekend. Will continue chipping away at it this week. First I'll rebase on to main.

[miurla]

Try this: https://github.com/mendableai/firecrawl

[albertdbio]

Ahh nevermind, the crawl api which scrapes multiple pages is slow, just scraping one page is decent

[miurla]

It's a good try!
I got the following advice on our Discord. I'm going to try this.

You don't need firecrawl for this, IMHO, can just use Playwright/fetch and then use turndown to convert html to markdown

[albertdbio]

jina seems to me to be a tad bit faster btw, if you wanna try it out

[miurla]

@albertdbio Wow, that's great. Is the function calling an implementation that uses groq's sdk to replace the researcher's model? Looking forward to the PR.

[miurla]

@albertdbio I've merged significant changes, so it's best to develop from main.

[miurla]

I'm working on this feature.