Idea: Web scraping example with a few different libraries

[danneu]

Idea: Web scraping example with a few different libraries

[avescodes]

Very interesting idea. The only thing I'm concerned with is how you would write a good "recipe" that compared and contrasted two alternative libraries (as opposed to solving a problem)?

Is there a clear winner (in your opinion?) Does one suit most use cases better than the other? If either of those is true, then perhaps the recipe could solve the "parsing html" problem, and have a sidebar on the alternative?

You seem like you're more informed on these two libraries than I, what are your thoughts?

[danneu]

Yeah, I didn't think of wording this as just "parsing html". 😅

I really just meant that the solution section could enumerate a few approaches instead of just one.

Two come to mind:

• CSS selectors, which most people are familiar with. ("clear winner")
• Function-based selectors applied to the zipper (incidental bonus approach)

It just so happens that I use Enlive and Laser for those respective approaches but I over-emphasized that. I certainly don't mean to assert mastery. There may be better ways to organize this recipe now that it makes more sense as "parsing html".

[avescodes]

@danneu I think regardless of how the recipes are shaped (we can always move things around) I'd really like to see what you come up with.

[semperos]

Especially for (1) web scraping that requires creating and keeping a session (authenticated or not), and (2) initial authoring of web scrapers, using clj-webdriver is quite helpful, since it drives a real browser. I'm biased, since I maintain it, but it has its uses in this area.