stormid/robotify-netcore

Provides robots.txt middleware for .NET core

Robotify.AspNetCore

Robotify - robots.txt middleware for .NET core (.netstandard2.0)

Installation

Install package via NuGet:

Install-Package Robotify.AspNetCore

Configuration

Add Robotify to the middleware pipeline like any other:

Add to the ConfigureServices method to ensure the middleware and configuration are added to the services collection

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddOptions();

            services.AddRobotify();

            services.AddMvc();
        }

To configure what robots directives you want to appear in your robots file you can create and assign a IRobotifyRobotGroupProvider. Multiple providers can be created and registered in the standard DI container, each provider will then be iterated when generating the contents of the robots file.

A IRobotifyRobotGroupProvider contains a single method that returns a list of RobotGroup, below is an example of the RobotifyDisallowAllRobotGroupProvider:

    public class YourOwnProvider : IRobotifyRobotGroupProvider
    {
        public IEnumerable<RobotGroup> Get()
        {
            yield return new RobotGroup() 
			{			
				UserAgent = "something",
				Disallow = new[] { "/" }
			}
        }
    }

To register a provider, use the .AddRobotGroupProvider<TProvider>() extension method when adding robotify:

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddRobotify(c => c
                .AddDisallowAllRobotGroupProvider()
                .AddRobotGroupsFromAppSettings()
                .AddRobotGroupProvider<YourOwnProvider>()
            );
        }

Then use the middleware as part of the request pipeline, I would suggest adding it at least after the .UseStaticFiles() middleware, then if you have a static file robots.txt this will be used instead.

        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            app.UseStaticFiles();
            app.UseRobotify();
            app.UseMvc(routes =>
            {
                routes.MapRoute(
                    name: "default",
                    template: "{controller=Home}/{action=Index}/{id?}");
            });
        }

Robotify can be configured within the .UseRobotify() method or via json configuration:

{
  "Robotify": {
    "Enabled": true,
    "SitemapUrl": "https://www.example.com/sitemap.xml",
    "CrawlDelay": 10,
    "Groups": [
      {
        "UserAgent": "*",
        "Disallow": "/"
      },
      {
        "UserAgent": "Googlebot",
        "Disallow": ["/admin"]
      },
      {
        "UserAgent": "AnotherBot",
        "Allow": ["/search"]
      }
    ]
  }
}

Json configuration and code configuration will be merged at boot time. If no groups are specified a default deny all ill be added.

robotify-netcore's Issues

Update to .NET Core 2.1

Hi @Philo

Are you looking to update this to .NET Core 2.1?

Cheers
Greg

configuration not working anymore

Looks like the configurations for groups was removed. Why is that?

this configuration does not work anymore

"Robotify": {
    "Enabled": true,
    "SitemapUrl": "https://musclemonkey.io/sitemap.xml",
    "CrawDelay": 10,
    "Groups": [
      {
        "UserAgent": "*",
        "Allow": ["/"]
      }
    ]
  }

Sample always defaults to no Configuration Found

I did a fresh pull and I get this message in the Kestrel Window

[Robotify] - you have no rules configured, adding default deny all just to keep you safe!

I am guessing somewhere it isn't loading the appsettings.json values, but I thought you should know

Allow and Disallow options in appsettings.json need to be arrays

Hi @Philo

I have now implemented this in a project and have found that having a single path as a string in the Allow/Disallow options in appsettings.json does not parse correctly. Changing this to an array makes it function correctly.

i.e.
"Groups": [ { "UserAgent": "*", "Disallow": ["/"] }, { "UserAgent": "Googlebot", "Disallow": ["/admin"] }, { "UserAgent": "AnotherBot", "Allow": ["/search"] } ]

Cheers
Greg

Exposing library and framework details

Dear Robotify team,

I noticed you announce the library used, including the version, and also details about the server framework (asp.net core), on the robots.txt content.

# Generated with Robotify.AspNetCore (v0.2.3.0)

You also reveal that this file is being dynamically generated.

# created: 2018-10-10T13:26:12

This is not acceptable to me, for security reasons.
Could you please allow to disable this behaviour, for example, on the middleware configuration?

Thank you and best regards,
Fernando

Dotnet core 3 issue

I have project with dotnet core 3.0
When I start project with installed Robotify.AspNetCore - I've got the following message:

[2019-10-16T08:08:39.2047487+00:00 FTL] (//1) Host terminated unexpectedly
System.AggregateException: Some services are not able to be constructed (Error while validating the service descriptor 'ServiceType: Robotify.AspNetCore.RobotifyMiddleware Lifetime: Singleton ImplementationType: Robotify.AspNetCore.RobotifyMiddleware': Unable to resolve service for type 'Microsoft.AspNetCore.Http.RequestDelegate' while attempting to activate 'Robotify.AspNetCore.RobotifyMiddleware'.)
 ---> System.InvalidOperationException: Error while validating the service descriptor 'ServiceType: Robotify.AspNetCore.RobotifyMiddleware Lifetime: Singleton ImplementationType: Robotify.AspNetCore.RobotifyMiddleware': Unable to resolve service for type 'Microsoft.AspNetCore.Http.RequestDelegate' while attempting to activate 'Robotify.AspNetCore.RobotifyMiddleware'.

Setup sitemap without appsettings.json

Hi,

How is possible to setup the sitemap link without JSON ?

THanks

README doesn't show how to correctly use allow/disallow paths

PR#10 has been reverted in PR#11, the readme should include

"Groups": [
      {
        "UserAgent": "*",
        "Disallow": "/"
      },
      {
        "UserAgent": "Googlebot",
        "Disallow": ["/admin"]
      },
      {
        "UserAgent": "AnotherBot",
        "Allow": ["/search"]
      }
    ]