cannot find apikey for project

[scrapinghub]

cannot find apikey for project

[jdemaeyer]

Hey pawel, thanks alot for the feedback!

I haven't used ScrapyRT so far. What does deploying a project to it mean? From the docs it looks to me like you need to run it within an existing Scrapy project, and that there's no endpoint to overwrite the project. Did shub use to work with it?

In the past if I had default apikey it was used for all endpoints

Ah, hadn't thought of that. Any objections to reverting to this behaviour? (/cc @pablohoffman @dangra @eliasdorneles @josericardo)

[pawelmhm]

Hey @jdemaeyer

there are actually two issues here:

1. not using default apikey if you do shub deploy name_of_project
2. scrapyrt

1 is definitely more important IMO. If I have "default" apikey I would expect it to be "default" so that it is used when deploying to all projects I deploy to. Otherwise I will have to define and store same apikey for multiple projects which seems inconvenient for me (because I have 13 projects).

2 is less important. We use shub as parasites so we can accommodate all changes no problem. I see error is probably because there is no actual integer "project_id":https://github.com/scrapinghub/shub/blob/master/shub/config.py#L91 we can probably add some "project_id" if it's needed if project_id must be integer. Are there any important reasons why project is is integer?

What does deploying a project to it mean?

deploy is just POST with egg to "addversion" endpoint with Basic Auth header. It is not used in open source version it is only used in some internal SH projects (I know about 2 of them).

Did shub use to work with it?

yes, it worked fine

EDIT:
and I see it still works fine if I add some fake "project_id"

Perhaps you could consider validating config when migrating? e.g. I have following in scrapinghub.yml

projects
   realtime: realtime/realtime

this was generated automatically. After replacing above with:

realtime: realtime/00

it works fine.

[jdemaeyer]

1 is definitely more important IMO. If I have "default" apikey I would expect it to be "default" so that it is used when deploying to all projects I deploy too. Otherwise I will have to define and store same apikey for multiple projects which seems inconvenient for me.

Ohh sorry, I just saw that your scrapinghub.yml is a little misconfigured. You don't need to define an API key per project, but per endpoint. And you don't have to specify the endpoint if it is the default one. I.e. it should look more like this:

much neater, eh? :) the default API key will be used now for all projects that don't specify an extra endpoint

[jdemaeyer]

Are there any important reasons why project is is integer?

It's what Scrapy Cloud uses to designate projects. In shub its used to distinguish between target that is defined in scrapinghub.yml but was mistyped (shub deploy prof instead of prod) and a target that wasn't defined in scrapinghub.yml but is still valid (e.g. shub deploy 12345)

[pawelmhm]

Ohh sorry, I just saw that your scrapinghub.yml is a little misconfigured. You don't need to define an API key per project, but per endpoint.

I used default version generated by shub from scrapy.cfg. I didnt edit anything manually. So you say I should remove apikey part?

I just did that and I still get: Error: Could not find API key for endpoint prod.

[pawelmhm]

It's what Scrapy Cloud uses to designate projects. In shub its used to distinguish between a mistyped target (shub deploy prof instead of prod) and a valid target that is a SH project but doesn't have a shortcut defined scrapinghub.yml (e.g. shub deploy 12345)

sounds good I can just add some fake id see my edit here: #112 (comment)

[jdemaeyer]

Uh I see where this went wrong: the migration got confused because you use http (instead of https) for deploying, I'll fix that :)

So you say I should remove apikey part?

Nono, the API keys are fine. You need to remove the endpoints from your project definitions. I.e. ad-hoc: ad-hoc/12345 should be ad-hoc: 12345. I'll update the migration wizard in a minute, let's try if it works automatically then :)

sounds good we can just add some fake id see my edit here: #112 (comment)

could you give me one of the scrapy.cfg sections that you use for ScrapyRT?

[pawelmhm]

You need to remove the endpoints from your project definitions. I.e. ad-hoc: ad-hoc/12345 should be ad-hoc: 12345. I'll update the migration wizard in a minute, let's try if it works automatically then :)

do I need to remove this manually or can I just wait for update to shub that will migrate me nicely to proper scrapinghub.yml?

[dangra]

TBH, I am not sure to stay backward compatible between SH deploys and ScrapyRT, for now it is possible because both are using the egg-base deploys inherited from ScrapyD but in the long term I doubt we can keep the promise of shub->scrapyrt compatibility.