Support for Heroku?

[kjvarga]

Support for Heroku?

[sidwood]

oop. found a fork that uses S3 http://github.com/shinjikuwayama/sitemap_generator

[kjvarga]

The problem is that the sitemap specification requires the sitemap to be served from the same host as the links in the sitemap. So putting it on a remote server is not an option. You need some kind of request proxy to proxy the sitemap request and stream it from the remote server I would think.

[sidwood]

Yes, I used Shinji's code to build a similar feature in v1.1.1 and ran into this issue. Uggg.

[kjvarga]

I spoke to Shinji about how he uses them and this was his reply:

It's often possible to have your Web server proxy the request to another location. Here's an nginx example:

location / {

proxy_pass http://localhost:8000;
proxy_set_header X-Real-IP $remote_addr;
}

If this isn't possible (say, at Heroku), then you might investigate using a middleware layer such as Refraction to do the proxying.

if req.path.include?('sitemap')

req.rewrite! "http://anotherhost/" + req.path
end

[kjvarga]

I think that's pretty easy to do. I want to break out the persistence mechanism so we can support anything else too: db, memcache, files, s3. So you just have to add an adaptor.

[kjvarga]

I've finished some much needed refactoring and re-architecting that will make plugging in different storage options a breeze.

I'm thinking a good option would be to add a database store. That way it works with ActiveRecord, so Amazon RDS, sqlite, mysql etc are no problem. And Mongoid support is easy then too. The thing I like about the db is that deployments are easier. No more Capistrano hooks to copy over your sitemaps.

[kerrizor]

I think the whole redirect thing might not be necessary

From the sitemap spec (http://sitemaps.org/protocol.php#location)

To submit Sitemaps for multiple hosts from a single host, you need to "prove" ownership of the host(s) for which URLs are being submitted in a Sitemap. Here's an example. Let's say that you want to submit Sitemaps for 3 hosts:

www.host1.com with Sitemap file sitemap-host1.xml

www.host2.com with Sitemap file sitemap-host2.xml

www.host3.com with Sitemap file sitemap-host3.xml

Moreover, you want to place all three Sitemaps on a single host: www.sitemaphost.com. So the Sitemap URLs will be:

http://www.sitemaphost.com/sitemap-host1.xml

http://www.sitemaphost.com/sitemap-host2.xml

http://www.sitemaphost.com/sitemap-host3.xml

By default, this will result in a "cross submission" error since you are trying to submit URLs for www.host1.com through a Sitemap that is hosted on www.sitemaphost.com (and same for the other two hosts). One way to avoid the error is to prove that you own (i.e. have the authority to modify files) www.host1.com. You can do this by modifying the robots.txt file on www.host1.com to point to the Sitemap on www.sitemaphost.com.

In this example, the robots.txt file at http://www.host1.com/robots.txt would contain the line "Sitemap: http://www.sitemaphost.com/sitemap-host1.xml". By modifying the robots.txt file on www.host1.com and having it point to the Sitemap on www.sitemaphost.com, you have implicitly proven that you own www.host1.com. In other words, whoever controls the robots.txt file on www.host1.com trusts the Sitemap at http://www.sitemaphost.com/sitemap-host1.xml to contain URLs for www.host1.com. The same process can be repeated for the other two hosts.

Now you can submit the Sitemaps on www.sitemaphost.com.

When a particular host's robots.txt, say http://www.host1.com/robots.txt, points to a Sitemap or a Sitemap index on another host; it is expected that for each of the target Sitemaps, such as http://www.sitemaphost.com/sitemap-host1.xml, all the URLs belong to the host pointing to it. This is because, as noted earlier, a Sitemap is expected to have URLs from a single host only.

[aukevs]

Great gem, worked perfectly untill i deployed and forgot about the read only filesystem of heroku.

Can anyone tell me what is the best solution right now? Found the fork that saves to S3 but can't make it work with my rails 3 app.

[kerrizor]

I built a cron'd rake task like so:

desc "build and deploy sitemap"
task :refresh_sitemap => :environment do
  s3_credentials = YAML.load_file("#{Rails.root.to_s}/config/amazon_s3.yml")[Rails.env].symbolize_keys!

  # builds the sitemap and deploys to S3 without notifying search engines
  Rake::Task["sitemap:refresh"].invoke

  AWS::S3::Base.establish_connection!(
    :access_key_id => s3_credentials[:access_key_id],
    :secret_access_key => s3_credentials[:secret_access_key]
  )

  Dir.glob("tmp/*xml.gz").each do |f|
    filename = File.join(Rails.root, f)
    AWS::S3::S3Object.store(File.basename(filename), open(filename), s3_credentials[:bucket], :access => :public_read)
     puts " [uploaded to S3:#{s3_credentials[:bucket]}]"
  end
end

Basically what's going on here is since Heroku allows us write-access to tmp/, we build the sitemap file there, then open an S3 connection and shove the files up. This could be more robust by checking the success of the sitemap:refresh task prior to opening the connection, and should probably spit out some errors if S3 is inaccessible, but its a decent starting point for a Heroku specific way to get your sitemaps built and moved up to S3.

** If you're having trouble, notice that I'm accessing S3 information from a yml file I built in config/.. since I do so much work with S3, I find having one central location for my access keys to be beneficial. It looks something like this:

test:
  bucket: [your test bucket]
  access_key_id: [your access key]
  secret_access_key: [your secret key]
  max_file_size: 734003200
  acl: 'public-read'
  url: [URL to this bucket]

development:
  bucket: [your development bucket]
  access_key_id: [your access key]
  secret_access_key: [your secret key]
  max_file_size: 734003200
  acl: 'public-read'
  url: [URL to this bucket]

staging:
  bucket: [your staging bucket]
  access_key_id: [your access key]
  secret_access_key: [your secret key]
  max_file_size: 734003200
  acl: public-read
  url: [URL to this bucket]

[dpickett]

I threw this together - happy to make a gem or incorporate it to the lib somehow - to me, I'm thinking it belongs in a separate gem - but I'm open to other thoughts.

https://gist.github.com/788471

Used it in conjunction with my paperclip setup.

[aukevs]

kerrizor: having some trouble with getting your rake task to connect to s3 but the solution is perfect, thanks for sharing!

[kjvarga]

Thanks for sharing some code guys. I'm on vacation right now but I'm happy to include solutions for s3/heroku when I return home in a few weeks. This is a much needed feature but I've not had the time so it's nice to see contributions. We will get there eventually :)

[mwawrusch]

Hi guys, with the help from Karl I was able to fix some url issues, I uploaded my take on sitemap_generator here https://github.com/mwawrusch/sitemap_generator under the heroku branch.

Here are the files that are not part of the gem modification.

Caveat: I am just learning ruby and rails, so the code is nowhere near production quality and should only be considered a workaround until karl comes back from vacation.

sitemap.rb

SitemapGenerator::Sitemap.default_host = "http://YOURHOST.com"
SitemapGenerator::Sitemap.exclude_root = true
SitemapGenerator::Sitemap.sitemaps_host = "http://BUCKETNAME.s3.amazonaws.com"
SitemapGenerator::Sitemap.public_path = "tmp/"
SitemapGenerator::Sitemap.sitemaps_path = "sitemaps/"

Sitemap_s3_publisher.rb

require 'aws/s3'

class SitemapS3Publisher

def self.publish
SitemapGenerator::Sitemap.create
upload
end

def self.publish_and_ping
SitemapGenerator::Sitemap.create
upload
SitemapGenerator::Sitemap.ping_search_engines
end

def self.store_file(file)
connect_to_storage_provider!
AWS::S3::S3Object.store("sitemaps/#{File.basename(file)}",
File.read(file),
bucket_name,
{
:access => default_access,
})

end

protected

def self.upload
Dir.glob("tmp/sitemaps/*xml.gz").each do |f|
filename = File.join(Rails.root, f)
self.store_file(f)
puts " [uploaded to S3:#{bucket_name}]"
end
end

def self.bucket_name
"BUCKETNAME"
end

def self.default_access
:public_read
end

def self.connect_to_storage_provider!
AWS::S3::Base.establish_connection!(storage_provider_credentials)
end

def self.storage_provider_credentials
@storage_provider_credentials = {
:access_key_id => "KEY",
:secret_access_key => "ACCESS"
}
@storage_provider_credentials
end

end

Rake tasks

desc "build and deploy sitemap to amazon s3"
task :publish_sitemap => :environment do
SitemapS3Publisher.publish
end

desc "build and deploy sitemap to amazon s3"
task :publish_sitemap_and_ping => :environment do
SitemapS3Publisher.publish_and_ping
end

[kerrizor]

So that's pretty much what my rake task does :) The problem is that if you put your S3 credentials into the gem, everyone who might want to use your fork would have to fork the gem as well and make changes to their version! Maintenance nightmare! In my Rake task on line 3 you can see where instead I'm opening up a yml file in config/ to read in the values I need to create the connection. Doing it this way I never need to touch the gem.

[mwawrusch]

Yes, I just included it here for reference. That file is not part of the gem.

The important part is that the reference urls to the sitemap files are wrong when created for s3, which was what I needed to change, and that involved some modifications in karls code (gladly he showed me the right direction) .

As i mentioned: "should only be considered a workaround until karl comes back from vacation." this is just for people who need to make a sitemap work for a couple of weeks.

[kerrizor]

Ok.. I guess I don't understand - could you explain how a fork of a gem with additional site- or user-specific information hardcoded into it is a better option than a rake task that lives within your own code base... or why whatever changes you need to make to the gem couldn't be wrapped into a pull request? Why wait on the gem author to "get back from vacation"? It seems like the point of github and OOS to craft our own viable solutions and offer them back to the community.

...to which point, yeah, I'll do that myself - wrap my rake task up into a pull request, right after I get back from walking my dog (which should be far sooner than Karl's vacation! :D)

[mwawrusch]

To put it into easy to understand sentences:

I needed to modify parts of the gem code from Karl. I started to use Ruby 2 weeks ago. The code I created works, but it might have side effects, etc. It is nothing that should be pulled into a production quality code base.

And yes, please create a working gem, publish a pull request, everyone will be happy. If you had done that in the first place I would not have had to do it myself.

[kerrizor]

I'm not going to write your code for you. I already provided a working Best Practices solution -- the one, in fact, that we use in our production code. Do what you want with it (or just ignore it, as you've already chosen to do so.)

[phaza]

Instead of reading credentials out of a config file, it would be great to read them from env variables. This would be more in line with Heroku mindset imho.
(Just a suggestion if this gets incorporated into the main gem. I can easily do this myself with the submitted rake task)

[mwawrusch]

Just to explain this here again, for the second time now. The changes I posted here (which are not part of the modifications I made in my fork that address the url issues) are here for completeness only and do not represent a best practice, on the contrary. For good design of the rake task look a couple of posts upward.

The maintainer of the gem has significant ruby experience and will know what to do.

[chak]

Hi Karl, just checking in on this thread. Do you plan to release support for db-backed sitemap storage? I would love to see that. Built-in heroku support would be great, but proxying from S3 seems like more hassle than its worth.

[kjvarga]

Hi chak,

I'm glad you mentioned that. I do like the idea of db-backed sitemaps. I'm 95% on the way to releasing the new Groups feature which gives you total control over how your sitemaps are generated, which is needed for S3 support. I think for now I'll add S3 support and the idea of Adapters to easily switch the persistence mechanism. After that we can look at adding an ActiveRecord Railtie and store in the database.

Anyone else prefer the db approach?

[kjvarga]

Oh yeah I should mention that it won't be necessary to do proxying if you're storing sitemaps on S3. It looks like the spec allows for storing sitemaps on remote servers. You just have to prove that you have write access to both...or something like that.

[mwawrusch]

Karl,

take a look at the fog gem regarding S3 and other cloud services.

I see the db approach as an additional option, but I really prefer the S3
based approach

Martin

On Tue, Apr 26, 2011 at 11:05 AM, kjvarga <
[email protected]>wrote:

Hi chak,

I'm glad you mentioned that. I do like the idea of db-backed sitemaps.
I'm 95% on the way to releasing the new Groups feature which gives you
total control over how your sitemaps are generated, which is needed for S3
support. I think for now I'll add S3 support and the idea of Adapters to
easily switch the persistence mechanism. After that we can look at adding
an ActiveRecord Railtie and store in the database.

Anyone else prefer the db approach?

Reply to this email directly or view it on GitHub:
#13 (comment)

[angelacode]

Hello --

I am a little confused...is the new solution now using DB and not S3 and is it released? Or does it use S3 and there is no need for proxying? Thanks! This is exactly what I am looking for :)

[kjvarga]

Hi Angela,

No it's not out yet! But it's coming. The solution will use S3 with no need for proxying. We've discussed adding in a DB solution later, if it's needed.

[angelacode]

I see that is good new, S3 sounds like it would be just right, when do you think it is available?

[kjvarga]

Hi all, I've just released v2.0! This was a TON of work. I had to rewrite a lot of the code and get it setup for new features and make it easier to extend going forward. The big features in 2.0 is a new API, Sitemap Groups and Sitemap Namers. I have not added S3 support yet!! BUT I can promise that it'll only be a week or two because it's not that hard. I've just been busy getting this out the door.

[mwawrusch]

Great, thank you. This is much appreciated.

[mokolabs]

Ping. Any update on S3 and/or Heroku support?

[kjvarga]

Yeah, I added Adapters last night and now the default adapter is a FileAdapter! Now I've just got to add the S3Adapter...I'll try to do that tonight. Will keep you posted.

[kjvarga]

Well, would you believe it?! The day has finally arrived!! Heroku support baby! I really hacked this together very quickly so it's not well tested. However, I have been successful using it on Heroku to upload sitemaps to S3. It uses CarrierWave, so anything that CW supports should work. Could you all try it out and let me know if you have issues. Make sure the issues are SitemapGenerator issues and not CW configuration issues...

I've released a pre-release Gem v2.0.1.pre1. So use that.

I've put up some configuration notes here https://github.com/kjvarga/sitemap_generator/wiki/Generate-Sitemaps-on-read-only-filesystems-like-Heroku

Let me know how it goes! (Oh, and if the mood strikes, consider donating to the project for good karma :)

[kjvarga]

Anyone had a chance to try this out yet?

[mwawrusch]

It's on my list for the weekend. Drowning in work here :P

On Thu, Jun 16, 2011 at 11:20 AM, kjvarga <
[email protected]>wrote:

Anyone had a chance to try this out yet?

Reply to this email directly or view it on GitHub:
#13 (comment)

[chak]

I have, and it's working well. Thanks!

On Thu, Jun 16, 2011 at 11:20 AM, kjvarga <
[email protected]>wrote:

Anyone had a chance to try this out yet?

Reply to this email directly or view it on GitHub:
#13 (comment)

[mwawrusch]

Got it up and running, works flawlessly. Tested it against 150k links, took a couple of minutes.
Great job, thank you.

[kjvarga]

Sweet! Good to hear. I'll add some docs and do a proper release soon.

[jgadbois]

I've also got this working, but Google doesn't recognize it (at least it's not showing up in webmaster tools). Any way to verify that the sitemap is being used?

[mokolabs]

To get this to work with Google Webmaster tools, I had to create a separate site profile for the subdomain where I'm storing my sitemaps and then submit the sitemap to Google Webmaster Tools from that secondary site profile.

So, if your main website is domain.com and you're storing your sitemaps at sitemaps.domain.com, then you'd need to create a new profile for sitemaps.domain.com on Google Webmaster Tools, and submit the sitemaps from within that new profile.

Hope that helps.

[jgadbois]

I'm actually storing them on s3 though (because I have a heroku app), so that option wouldn't work I don't think.

[mwawrusch]

you need to add a

Sitemap: http://.....xml.gz

into your robots.txt. Then it will work.

On Fri, Jun 24, 2011 at 9:18 AM, jgadbois <
[email protected]>wrote:

I'm actually storing them on s3 though (because I have a heroku app), so
that option wouldn't work I don't think.

Reply to this email directly or view it on GitHub:
#13 (comment)

Martin Wawrusch
p: [phone]

I blog at http://martinatsunset.com : O API Marketplace, Where Art Thou
Follow me on twitter at http://twitter.com/martin_sunset
Learn more about me http://about.me/martinw

[jgadbois]

Sorry if this is taking this issue OT. I do have the following in my robots.txt

Sitemap: http://weighttraining.com.s3.amazonaws.com/sitemaps/sitemap_index.xml.gz

[mwawrusch]

I think the culprit might be the

http://www.example.com/

in your actual site map, unless you own example.com :P

On Fri, Jun 24, 2011 at 9:26 AM, jgadbois <
[email protected]>wrote:

Sorry if this is taking this issue OT. I do have the following in my
robots.txt

Sitemap:
http://weighttraining.com.s3.amazonaws.com/sitemaps/sitemap_index.xml.gz

Reply to this email directly or view it on GitHub:
#13 (comment)

[jgadbois]

Ahh thank you. Is this a bug then? My generated sitemap_index is pointing to a generic sitemap1 instead of the sitemap1 in the same folder as sitemap_index

[mwawrusch]

Can you post your sitemap.rb file as a gist (gist.github.com)?

On Fri, Jun 24, 2011 at 9:34 AM, jgadbois <
[email protected]>wrote:

Ahh thank you. Is this a bug then? My generated sitemap_index is pointing
to a generic sitemap1 instead of the sitemap1 in the same folder as
sitemap_index

Reply to this email directly or view it on GitHub:
#13 (comment)

[jgadbois]

Ok, i see it, my sitemaps_host is wrong. Sorry about that. Thanks again for the help!

[mwawrusch]

np. Glad I could help.

On Fri, Jun 24, 2011 at 9:38 AM, jgadbois <
[email protected]>wrote:

Ok, i see it, my sitemaps_host is wrong. Sorry about that. Thanks again
for the help!

Reply to this email directly or view it on GitHub:
#13 (comment)

[jgadbois]

Ok, so my site map is pointing to the right child sitemap now, but Google still isn't recognizing it. Robots.txt points to the sitemap_index which poitns to sitemap1.xml.gz

[kjvarga]

Hey I took a look at your files and everything does look in order. Except for one thing, which is that sitemap1.xml contains a link back to the index file (which is on S3). This violates one of the rules for sitemaps which is that all the URLs in the sitemap must point to the same host.

Check out this info which is pretty useful (you may have read it): http://www.sitemaps.org/protocol.php#location

Try disabling adding the sitemap index link using SitemapGenerator::Sitemap. include_index = false and then see if Google accepts it.

When I run heroku rake sitemap:refresh:no_ping --app MYAPP I get the following error:
! Internal server error
That's the only error message I get. heroku logs doesn't give me anything about the error.

Any help appreciated. It works fine on my local machine. Here are my settings: https://gist.github.com/1089300

Fixed by itself. Now two hours later it seems to work. I didn't change anything. Maybe it was some temporary problem with Heroku.

[mokolabs]

Yeah... in my experience, internal server errors on Heroku seem to be intermittent problems with the Heroku platform itself, and nothing to do with your application code.

[kjvarga]

@jgadbois is Google happy with you sitemaps now? Can anyone else confirm that removing the sitemap index from the sitemap file (as mentioned in #13 (comment)) works and makes Google/other search engines happy?

FYI I've released v2.0.1.pre2 which should fix uploading to the root of the bucket.

I think we are ready for a release proper on this feature. I'd like to turn include_index off by default if using the WaveAdapter if that is causing issues.

[jgadbois]

I believe they are happy now. They aren't showing up in webmaster tools, but I see some crawl errors "In Sitemaps" in the Webmaster Tool dashboard, so I think they are indexing the sitemap.

I did remove the link back to the sitemap index, but not sure if that's when it started to work or now.

[kjvarga]

Ok we have a proper release v2.1.0 now, so you don't have to use --pre to install anymore. I still need to turn off include_index when using remote host, but I can just issue a patch for that.

[kjvarga]

As of v2.1.1 you no longer need to set include_index = false because it's done for you.

If someone could write a blog post about SitemapGenerator and linking to the repo that would be much appreciated. We don't rank very highly on Google.

Also very soon SitemapGenerator will be framework agnostic, so let me know if there are any frameworks you want me to integrate with :)