adamsalter/sitemap_generator

This plugin enables 'enterprise-class' Google Sitemaps to be easily generated for a Rails site as a rake task, using a simple 'Rails Routes'-like DSL. It allows you to take care of familiar Sitemap issues like: Gzip of Sitemap files, variable priority links, paging/sorting links (e.g. my_list?page=3), SSL host links (e.g. https:), Rails apps which are installed on a sub-path (e.g. example.com/blog_app/) and hidden AJAX routes. It includes a Sitemap Index file so it supports more than the standard 50,000 individual urls (up to a maximum of 2.5 billion), and pings all major search engines on completion (Google, Yahoo, MSN, Ask, SitemapWriter).

sitemap_generator's Issues

Sitemap_generator not compatible with Rails 3

Hi,

sitemap_generator is not compatible with Rails 3. Do you have plans to update it?

The error:
/home/nono/gem/gems/actionpack-3.0.0.beta/lib/action_controller/metal/url_for.rb:90: undefined method `class_attribute' for SitemapGenerator::Helper:Module (NoMethodError)

uninitialized constant ActionController::TestRequest

rake sitemap:refresh --trace

(in /var/www/marc-seeger.de)
** Invoke sitemap:refresh (first_time)
** Invoke sitemap:create (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke sitemap:clean (first_time)
** Execute sitemap:clean
** Execute sitemap:create
rake aborted!
uninitialized constant ActionController::TestRequest
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:440:in load_missing_constant' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:80:inconst_missing'
/var/www/marc-seeger.de/vendor/plugins/sitemap_generator-plugin/lib/sitemap_generator/helper.rb:5:in load_sitemap_rb' /var/www/marc-seeger.de/vendor/plugins/sitemap_generator-plugin/tasks/sitemap_generator_tasks.rake:29 /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:incall'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in execute' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:ineach'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in execute' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:ininvoke_with_call_chain'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/1.8/monitor.rb:242:in synchronize' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:ininvoke_with_call_chain'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:607:in invoke_prerequisites' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:ineach'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in invoke_prerequisites' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:596:ininvoke_with_call_chain'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/1.8/monitor.rb:242:in synchronize' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:ininvoke_with_call_chain'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in invoke' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:ininvoke_task'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in top_level' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:ineach'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in top_level' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:instandard_exception_handling'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in top_level' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:inrun'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in standard_exception_handling' /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:inrun'
/opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31
/opt/ruby-enterprise-1.8.6-20090421/bin/rake:19:in `load'
/opt/ruby-enterprise-1.8.6-20090421/bin/rake:19

And just FYI:
*** LOCAL GEMS ***

actionmailer (2.3.2)
actionpack (2.3.2)
activerecord (2.3.2)
activeresource (2.3.2)
activesupport (2.3.2)
addressable (2.1.0, 2.0.2)
builder (2.1.2)
chronic (0.2.3)
coderay (0.8.312)
cucumber (0.3.11)
data_objects (0.9.12, 0.9.11)
date-performance (0.4.7, 0.4.6)
diff-lcs (1.1.2)
dm-aggregates (0.9.11)
dm-core (0.9.11)
do_mysql (0.9.12, 0.9.11)
extlib (0.9.12, 0.9.11)
fastthread (1.0.7)
gchartrb (0.8)
hoe (2.3.2, 2.3.1, 2.1.0, 2.0.0)
lesstile (0.3)
mislav-will_paginate (2.3.11)
mysql (2.7)
nokogiri (1.3.2, 1.3.1, 1.2.3)
passenger (2.2.4, 2.2.3, 2.2.2, 2.2.1)
polyglot (0.2.6, 0.2.5)
postgres (0.7.9.2008.01.28)
rack (1.0.0, 0.9.1)
rack-cache (0.5)
rails (2.3.2)
rake (0.8.7, 0.8.4)
RedCloth (4.2.2, 4.2.1, 4.2.0, 4.1.9)
rspec (1.2.7, 1.2.6)
ruby-openid (2.1.7, 2.1.6)
rubyforge (1.0.3)
sequel (3.2.0, 3.1.0)
sinatra (0.9.2, 0.9.1.1)
sqlite3-ruby (1.2.4)
term-ansicolor (1.0.3)
treetop (1.2.6, 1.2.5)

REAME: Sitemap Generator has moved to kjvarga/sitemap_generator. Please create issues there.

Multi-lingual site sitemap generation

Hi,

I have multi-lingual site, where languages is separated by a first level (TLD) domain name, eg www.sitename.com, www.sitename.es.

Right now there are no way to generate sitemaps for different domain names (hosts). When it will be possible to do that? Any suggestions on that?

PS. Actually I need only possibility to generate sitemap files from rake tasks, everything else I can do through the nginx config (point different sitemaps on the different hostnames).

Thanks,
Dmitry

Rest In Peace

Dear Adam,
your project has helped me a lot and I find it very sad to hear that you have passed away.

Rest In Peace

Sitemap generator loads application controller when defining itself

This is giving me 2 issues:

1. My classes aren't reloading properly when sitemap gets loaded into my app. I think this has to do with my sitemap referencing some classes when generating urls, but it goes away whenever I comment out sitemap from my required gems.
2. My AppController references other classes. When I include sitemap it tries loading my app controller before all of those dependencies have been loaded, which ends up blowing up.

I think this can all be simplified by not touching application controller at all, and going directly at the underlying rails path utilities.

I think I have this working without ever touching the application controller, so I'll send a patch when I confirm.

tasks are polluting object with sitemap generator methods

The following lines live in sitemap:create:
task :create => [:environment] do
include SitemapGenerator::Helper
include ActionView::Helpers::NumberHelper

When those lines run, they're running their include on Object, adding a bunch of junk to it.

To prove this, add this to your Rakefile

at_exit { puts "I am #{self.class.name} and I have load_sitemap_rb defined on me? #{methods.include?('load_sitemap_rb')}" }

then run rake sitemap:create

I think the underlying issue is that there's quite a bit of code in the tasks file, that should maybe live somewhere else.

I'll take a crack at refactoring this and reference this issue.

gem

Can we please have this in a gem instead of a plugin?

Yahoo 403

Sitemap stats: 436,086 links, 142m20s
Successful ping of Bing
Successful ping of Ask
Successful ping of Sitemap Writer
Successful ping of Google
Ping failed for Yahoo: #<OpenURI::HTTPError: 403 Forbidden>

this is wrong: require 'sitemap_generator/tasks' rescue LoadError

require 'sitemap_generator/tasks' rescue LoadError does not work.

Example: http://codepad.org/bEQIuIS6

You can do this:
begin
require 'sitemap_generator/tasks'
rescue LoadError
end

Example: http://codepad.org/p27EDH1E

You could also do: begin; require 'sitemap_generator/tasks'; rescue LoadError; end

crc error of gz files, missing close for GZipWriter block

I'm receiving incorrect crc errors from all the sitemaps files.

This happens from time to time, so is not consistent.

After digging into this, found this:

http://ruby-doc.org/core/classes/Zlib/GzipWriter.html
(read the note)

The missing @close@ to GZipWritter could be the problem. Seems the following places:

http://github.com/adamsalter/sitemap_generator/blob/master/tasks/sitemap_generator_tasks.rake#L49-51
http://github.com/adamsalter/sitemap_generator/blob/master/tasks/sitemap_generator_tasks.rake#L62-64

Need addition of gz.close prior leaving the block

Adding that locally solved the erratic crc errors

Thank you.