eugeneradionov/sky-fighters

sky-fighters's Issues

Store data to MariaDB (or MySQL)

Store data to Sqlite DB

There should be some gems: sqlite3 or sqlite3-ruby

Install SQLite browser http://sqlitebrowser.org/,
should be available in repo sudo apt-get install sqlitebrowser

Store data to MongoDB

Use ORM to manage DB queries

Article to read: http://www.sitepoint.com/orm-ruby-introduction/

• Sequel: http://sequel.jeremyevans.net/
• DataMapper: http://datamapper.org/
• ActiveRecord: http://guides.rubyonrails.org/active_record_basics.html (preferable)

Possibly implement all three variants to get familiar with them

Store data to PostgreSQL DB

• use separate class like: https://marcclifton.wordpress.com/2012/11/12/an-example-of-using-postgresql-with-ruby/ and keep Strategy pattern
• use prepared query:

conn.prepare('statement1', 'insert into table1 (id, name, profile) values ($1, $2, $3)')
conn.exec_prepared('statement1', [ 11, 'J.R. "Bob" Dobbs', 'Too much is always better than not enough.' ])

• use transaction:

@conn.transaction do |conn|
    ins = @conn.exec_prepared('insert', [i, "#{@words1[i % 100]} street"])
end

Multithreading for downloading HTMLs

possibly has no sense for Ruby

Use config file for storage configuration

Article to read: https://practicingruby.com/articles/configurable-applications-1

Skyfighters. Detect response encoding

The idea is to have default encoding set in global variable (possibly env. variable), and check response encoding. In case of response encoding doesn't much default one - do transformation

begin
  result = response.body.dup.force_encoding('UTF-8')
  unless result.valid_encoding?
    result = response.body.encode( 'UTF-8', 'Windows-1251' )
  end
  content = result  
rescue EncodingError
  result.encode!( 'UTF-8', invalid: :replace, undef: :replace )
end

via http://landrina.ru/development/ror-encoding-troubes-with-cyrillic/