failing test 'it "handles searching for projects with special characters" do'

[markolson]

failing test 'it "handles searching for projects with special characters" do'

[benrugg]

Nice research and write-up. It looks like for the most part, characters outside the standard alpha-numerics don't yield any good search results anymore. (They used to before Kickstarter switched to Elasticsearch).

I put that test in there just to make sure we didn't screw things up with the way we escaped or encoded the characters, but honestly at this point it seems like more of a hassle than it's worth to support a wide set of punctuation or diacritics.

If you want to still keep working on it, by all means, go for it. But if your enthusiasm is waning, I think it's probably ok to get rid of that test.

I issued a pull request, with a couple of revised test and one left pending.

The problem has been noticed when a search including the ligature ae is done.

If lines 14-16 and 19-23 in the code below are commented out, then line 24 executes fine.
However, if either is left uncommented, then line 24 fails with

ArgumentError: invalid byte sequence in US-ASCII

(Absent the print statements, somewhere else in the routine will fail with the same error on a search including ae)

I've tried there various solutions of exporting environment variables and putting encoding lines at the tops of files that searches for the error message bring up, but none make the problem disappear.

cat -n ./lib/kickscraper/connection.rb
10    def call(env)
11  puts "ZZZ #{env[:url].request_uri()}"
12      # replace '+' symbols in the query params with their original spaces, because there
13      # seems to be a bug in the way some versions of Fararay escape parameters with spaces
14  #    env[:url].query_params.each { |key, value|
15  #      env[:url].query_params[key] = value.tr('+', ' ')
16  #    }
17
18      # add format=json to all public search requests, or add the oauth_token to all api requests once we have it
19  #    if env[:url].host.index('https://api.kickstarter.com').nil?
20  #      env[:url].query_params['format'] = 'json'
21  #    else
22  #      env[:url].query_params['oauth_token'] = Kickscraper.token unless Kickscraper.token.nil?
23  #    end
24  puts "ZZZ #{env[:url].request_uri()}"
25
26      # make the call
27      @app.call(env)
28    end
29  end

[benrugg]

Interesting... it seems like the issue might be related to the internal processing inside the Faraday code... that's just a speculation, but either way, I think leaving that test pending is fine. Thanks for working on this. I'll merge the pull request in a moment.