lmmx/bisque

Web scraping into structured Pydantic data models :stew:

bisque's Issues

Introduce Pydantic models to `elements`

Convert object representations of data models to Pydantic data models, using custom classes where necessary (e.g. for str subclasses representing strings that need to be coercible)

Parameterise `.find()` with data models rather than kwargs

The paradigm shift involved with introducing Pydantic could involve changing the form of element addressing.

• The built tree is used with a find() method that takes a tag name and other kwargs
• The results of a given find() call are chained to produce nested find()
• Functional Python is an antipattern (in GvR's informal style guide)
• Functional Python is also overly verbose and ad-hoc for highly nested tag access
• Data models can be highly nested and could be used to produce the same result which turns the kwargs into an object representing the nested path

Review I think this is actually already implemented with SoupStrainer?

Handle charset-conditional setup of `ContentMetaAttributeValue` following Pydantic conversion

Part of #3 in bisque.element.attributes

Make a shallow class interface

To resolve the circular import dependency, put the runtime types in the data model and access them by reference on self rather than directly from the module namespace (requiring lazy import hacks).

Demo in interface-separation-example

• Introduce type table
• Factor out all individual classes in clear hierarchical modules

Ensure no plain uses of `AttributeValueWithCharsetSubstitution` after Pydantic conversion

Part of #3 in bisque.element.attributes

The class AttributeValueWithCharsetSubstitution cannot be used on its own (it has no fields). If it's used on its own in the original code, it would create a str because it subclassed it. Now it can only do that if it has a known first field with which to create the __str__ representation.

If you do need to introduce fields, this would effectively prepend fields [thru dataclass inheritance] to the ContentMetaAttributeValue subclass, so that would need to instead inherit a base (i.e. we'd need to duplicate AttributeValueWithCharsetSubstitution_BASE as a base, and then add the field to the regularly named AttributeValueWithCharsetSubstitution so as to preserve its behaviour)

Reinitialise Bisque from Campbells

Before proceeding with Bisque features, fill the empty package shell with the Campbells package innards, and get the test suite to pass