hey! I’m exploring how to make the following work:
from zyte_common_items import Product
from web_poet import ItemPage, handle_urls
@handle_urls("example.com")
class ExamplePage(ItemPage[Product]):
# ...In other words, how to make @handle_urls("example.com") work for an ItemPage[Product] subclass without a need to use instead_of in handle_urls, and without a need to use a base page object for instead_of.
I can see 2 main approaches here.
Approach 1: support is directly
In the example, handle_urls doesn't really define any override rule. Instead, we have a declaration that ExamplePage can return Product item for example.com page. This information should be enough to allow creation of a scrapy-poet provider for items:
def parse(self, response: DummyResponse, product: Product):
# ...We know the website, we know which item is needed, and can use Page Object registry to find a right page object, according to domain and priority.
To implement it, web-poet needs to be slightly refactored:
- We should rename “overrides”, “override rule” to something like “apply rules”. This includes changes to class names, module names, method names, and changes to documentation as well. In addition to the old
ApplyRule(for_patterns=..., use=ExampleProductPage, instead_of=GenericProductPage), it’d be possible to specify ApplyRule(for_patterns=..., use=ExampleProductPage, to_return=Product)
handle_urls decorator would pick up to_return=Product from the ItemPage[Product] automatically (probably unless instead_of argument is defined? but that's not clear, there are use cases for supporting both at the same time).
When implementing it, we should make sure that priorities work as intended. For example, it should be possible to define and configure a Page Object which provides Product instances using AutoExtract API, and have it behaving in a sensible way (enabled for all domains, custom Page Objects should take priority over this default one).
Approach 2: define standard generic Page Object classes
In this case, "override rules" stay override rules. There is a registry of {item_type: generic_page_object_class} defined somewhere, e.g. {Product: ProductPage}. Or, maybe, {Product: [ProductPage, AnotherGenericProductPage]}. There should be an API to extend and modify this registry. handle_urls looks up this registry, and picks instead_of argument based on it.
Pros/cons
Overall, I like an approach with items more, because it seems to provide a nicer API:
- less indirection;
- enables a use of items as dependencies in callbacks;
- only requires standardization of item classes, not base page objects.
A risk here is that we may decide that standardized page objects are actually very important. For example, unlike items, they allow to extract only some of the fields. They may also provide some methods other than to_item which can be used in overrides.
A challenge with standard base classes is how much logic you put there. On one hand, there is a lot of useful logic which can be put in a base class, and which can save developer time. For example, default implementation for some of the attributes, or helper methods. But the more powerful your base class is, the more you need to assume about the implementation. So, it might be wise to have these "standard" page objects mostly as "markers", to ensure that a wide range of page objects is compatible with each other. But, if we do so, we still need to put the "powerful" logic somewhere.
If you have separate "marker" base class used for overrides, and feature-full base class used in a project, usage becomes less straightforward - you'd probably need to do something like this - the original challenge is not solved:
# your project:
@handle_urls("example.com", instead_of=ProductPage)
class ExamplePageObject(PowerfulProductPage):
# ...
# generic spider:
def parse_product(self, response: DummyResponse, page: ProductPage):
# ...Alternative is
# your project:
@handle_urls("example.com")
class ExamplePageObject(PowerfulProductPage):
# ...
# generic spider:
def parse_product(self, response: DummyResponse, page: PowerfulProductPage):
# ...But it defeats the purpose of defining a standard ProductPage which is not tied to a project-specific spider - generic spider above doesn't support pages which are just ProductPage subclasses, it needs its own, PowerfulProductPage subclasses. It also requires solving a case when a page objects which uses handle_urls is not a strict subclass of PowerfulProductPage.
With "items" approach it's not an issue - as soon as page objects return the same item class, they're considered compatible, there is no need for them to share exactly same base class, besides having ItemPage[ItemClass] or Returns[ItemClass] somewhere in hierarchy.
Not extracting all fields with Approach 1
The main cons I see with the Approach 1 is a case where not all item fields are required in a generic spider.
@handle_urls("example.com")
class ExamplePage(ItemPage[Product]):
# ...
def parse(self, response: DummyResponse, product: Product):
# only some product attributes are used, not all of themThere is no ProductPage anymore, and if we define it, it's not stated that ExamplePage can be used instead of ProductPage. There are some possible solutions to it, e.g. using typing.Annotated:
def parse(self, response: DummyResponse, product: Annotated[Product, pick_fields(["price", "name"])):
# ...There could be also a "reverse" registry, {ProductPage: Product}, which scrapy-poet can use to pick a right Page Object from the registry if ProductPage is requested, though this approach has some challenges.
Using custom methods
Users might want to have a Page Object with some custom methods, and use it in a crawler:
class ProductListPage(ItemPage[ProductList]):
def links(self):
# example utility method
return [self.paginationNext.url] + [product.url for product in self.products]
def parse(self, response: DummyResponse, page: ProductListPage):
yield from scrapy.follow_all(page.links())
# ...This is compatible with both Approach 1 and Approach 2, if ProductListPage uses ProductList as a dependency:
@attrs.define
class ProductListPage(ItemPage[ProductList]):
item: ProductList
def links(self):
# example utility method
return [self.item.paginationNext.url] + [product.url for product in self.item.products]
def to_item(self):
return self.itemIn this case, one can define POs for ProductList using Approach 1, and they will be used automatically to create item dependency. It's also possible to override ProductListPage itself, if it's desired to have a custom links implementation. So, both pages below would be applied properly:
# for foo.com we state that "FooProductListPage returns productList"
@handle_urls("foo.com")
class FooProductListPage(ItemPage[ProductList]):
...
# for bar.com we override ProductListPage itself
@handle_urls("bar.com", overrides=ProductListPage)
class FooProductListPage(ProductListPage):
def links(self):
# ...
Conclusion
Approach 1 is tempting because it seems to provide a better API - one can receive items in callbacks, and just use them; there is no need e.g. to use item = await ensure_deferred(page.to_item()) in every callback, there is no need to use async def parse. It also gives fully typing support, which can be hit or miss with page objects.
For me it's also much easier to understand - there are no "overrides" anymore, no semi-magical replacement of one class with another. We're just telling "this Page Object returns Product item on example.com website", and then, when Product item is needed on example.com website, we know how to get it.
Approach 2 looks more powerful, especially if web-poet fields are used heavily, and a spider actually needs to access only some of the fields, but not all of them.
Currently I'm +0.25 to implement, follow and recommend Approach 1, mostly because of it's simplicity, even if it doesn't support some features which Approach 2 gives. But I do see advantages of Approach 2 as well, so it's not a strong opinon.
Thoughts?
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent