mehmetozkaya/dotnetcrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

I am developing awsrun and aspnetrun repositories and create Udemy courses from real-world applications. Active Coupon APRI24 links below;

Software Architecture Udemy Courses

• Design Microservices Architecture with Patterns & Principles - Medium Articles - Github Repository
• Cloud-Native: Microservices, Kubernetes, Service Mesh, CI/CD - Medium Articles - Github Repository
• Microservices Interview: Essential Questions and Answers - Medium Articles

.Net Microservices Udemy Courses

• .NET 8 Microservices: DDD, CQRS, Vertical/Clean Architecture - Medium Articles - Github Repository
• Deploying .Net Microservices with K8s, AKS and Azure DevOps
• Secure .Net Microservices with IdentityServer4 OAuth2,OpenID
• Microservices Observability, Resilience, Monitoring on .Net
• Using gRPC in Microservices Communication with .Net 5

AWS Serverless Udemy Courses

• AWS Lambda & Serverless - Developer Guide with Hands-on Labs - Medium Articles - Github Repository
• AWS Serverless Microservices with Patterns & Best Practices - Medium Articles - Github Repository

dotnetcrawler's Issues

Cant open database

after launching eshop im trying to run dotnetcrawler sample but i receive error:

"Unhandled Exception: System.AggregateException: One or more errors occurred. (An error occurred while updating the entries. See the inner exception for details.) ---> Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Cannot insert the value NULL into column 'Name', table 'Microsoft.eShopOnWeb.CatalogDb.dbo.Catalog'; column does not allow nulls. INSERT fails.
The statement has been terminated."

How to fix that?

Can't do the database migration/update.

Hi,

I'm trying to run your code but met a problem on creating the database schema through using code-first approach as fellowing:

PM> add-migration MicrosofteShopOnWebCatalogDbContext -context MicrosofteShopOnWebCatalogDbContext
To undo this action, use Remove-Migration.
PM> update-database -context MicrosofteShopOnWebCatalogDbContext
System.ArgumentException: GenericArguments[0], 'DotnetCrawler.Data.Migrations.MicrosofteShopOnWebCatalogDbContext', on 'Microsoft.EntityFrameworkCore.Design.IDesignTimeDbContextFactory1[TContext]' violates the constraint of type 'TContext'. ---> System.TypeLoadException: GenericArguments[0], 'DotnetCrawler.Data.Migrations.MicrosofteShopOnWebCatalogDbContext', on 'Microsoft.EntityFrameworkCore.Design.IDesignTimeDbContextFactory1[TContext]' violates the constraint of type parameter 'TContext'.
at System.RuntimeTypeHandle.Instantiate(RuntimeTypeHandle handle, IntPtr* pInst, Int32 numGenericArgs, ObjectHandleOnStack type)
at System.RuntimeTypeHandle.Instantiate(Type[] inst)
at System.RuntimeType.MakeGenericType(Type[] instantiation)
--- End of inner exception stack trace ---
at System.RuntimeType.ValidateGenericArguments(MemberInfo definition, RuntimeType[] genericArguments, Exception e)
at System.RuntimeType.MakeGenericType(Type[] instantiation)
at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.FindContextFactory(Type contextType)
at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.FindContextTypes()
at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.FindContextType(String name)
at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(String contextType)
at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.UpdateDatabase(String targetMigration, String contextType)
at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
GenericArguments[0], 'DotnetCrawler.Data.Migrations.MicrosofteShopOnWebCatalogDbContext', on 'Microsoft.EntityFrameworkCore.Design.IDesignTimeDbContextFactory`1[TContext]' violates the constraint of type 'TContext'.

crawling from several website

Hi ,

I wana crawling from several wensite . The problem is that my entity is only capable receiving information from a site . Is there a solution to this dynamic?