Giter Site home page Giter Site logo

influxer's Introduction

Build

Influxer

Influxer provides an ActiveRecord-style way to work with InfluxDB with many useful features, such as:

Installation

Adding to a gem:

# my-cool-gem.gemspec
Gem::Specification.new do |spec|
  # ...
  spec.add_dependency "influxer", ">= 1.2.0"
  # ...
end

Or adding to your project:

# Gemfile
gem "influxer", "~> 1.2"

Usage

Metrics classes

To query InfluxDB or write to it, you should define a metrics class first. Each metrics class represents a measurement/series (or multiple related measurements):

class VisitsMetrics < Influxer::Metrics
  # Define tags...
  tags :account_id, :page_id
  # ...and attributes
  attributes :user_id, :browser
end

Querying

Now you can use your metrics classes in a similar way to Active Record models to build queries. For example:

VisitsMetrics.select(:account_id, :user_id).where(page_id: /^home\/.*/)

Influxer provides special query methods for dealing with time series:

  • Group by time: Metrics.time(:hour) => # select * ... group by time(1h).
  • Select only points for the last hour/minute/whatever: Metrics.past(:day) => # select * ... where time > now() - 1d.
  • Select only points since the specified time: Metrics.since(Time.utc(2014,12,31)) => # select * ... where time > 1419984000s.
  • and more.

See our Wiki for more.

Scopes support

You can define scopes to re-use query conditions:

class Metrics < Influxer::Metrics
  tags :account_id
  attributes :value

  default_scope -> { time(:hour).limit(1000) }

  scope :unlimited, -> { limit(nil) }
  scope :by_account, ->(id) { where(account_id: id) if id.present? }
end

Metrics.by_account(1)
# => select * from "metrics" group by time(1h) where account_id=1 limit 1000

Metrics.unlimited.by_account(1).time(:week)
# => select * from "metrics" group by time(1w) where account_id=1

Active Record integration

You can association metrics with Active Record models:

class UserVisits < Influxer::Metrics
end

class User < ActiveRecord::Base
  has_metrics :visits
end

user = User.find(1)
user.visits.write(page_id: "home")
#=> < creates point {user_id: 1, page_id: 'home'} in 'user_visits' series >

user.visits.where(page_id: "home")
#=> select * from user_visits where page_id='home'

Find more on Wiki.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/palkan/influxer.

License

The gem is available as open source under the terms of the MIT License.

influxer's People

Contributors

alekseenkoss77 avatar alexandershvaykin avatar ceritium avatar dimiii avatar gjhenrique avatar jklimke avatar mpursche avatar palkan avatar rodrigoapereira avatar vmeng avatar zumkorn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

influxer's Issues

Anyway update bug with influxer config

Hello palkan,

I'm trying to debug anyway_config 1.1.3 update:

wrong number of arguments (given 1, expected 0)
influxer (1.1.1) lib/influxer/config.rb:35:in `load'
anyway_config (1.1.3) lib/anyway/config.rb:69:in `initialize'

Do we need def load on influxer anymore?
Thank you for both gems =D

Broken write

Hi there!

Sadly, this #35 introduced a regression. Recently I tried for the first time to write data using model object. Due to parsed_timestamp in

    def write_point
      data = {
        values: values,
        tags: tags,
        timestamp: parsed_timestamp
      }
...
    end

it generates incorrect influxdb statements, like 'uptime_trace current_reading=1i s' or 'uptime_trace current_reading=1i 1520507736s'.

ArgumentError railse if the influxdb is not reachable.

Hi, thank you for your great gem! ๐Ÿ˜„

But I found an issue as is the title.

gem versions

  • influxer 0.5.1
  • influxdb 0.3.6

Model definition

class Target < ApplicationRecord
  hsa_metrics :metrics
end

class TargetMetrics < Influxer::Metrics
  set_series :cpu
  tags :target_id
  attributes :value, :kind
end

(I removed some codes..., sorry.)

How to reproduce?

  1. Stop your influxDB.
  2. send a request just like below.
irb(main):001:0> target = Target.last
  Target Load (0.3ms)  SELECT  `targets`.* FROM `targets` ORDER BY `targets`.`id` DESC LIMIT 1
=> #<Target id: "i-xxxxxx", name: "web server", group_id: 2, type_id: 41, created_at: "2016-08-11 01:37:11", updated_at: "2016-08-11 01:37:11">
irb(main):002:0> target.metrics
ArgumentError: wrong number of arguments (given 2, expected 1)
        from /Users/hirakiuc/Desktop/github_repos/watchtower/.bundle/ruby/2.3.0/gems/influxer-0.5.1/lib/influxer/rails/client.rb:15:in `log'
        from /Users/hirakiuc/Desktop/github_repos/watchtower/.bundle/ruby/2.3.0/gems/influxdb-0.3.6/lib/influxdb/client/http.rb:59:in `rescue in connect_with_retry'
        from /Users/hirakiuc/Desktop/github_repos/watchtower/.bundle/ruby/2.3.0/gems/influxdb-0.3.6/lib/influxdb/client/http.rb:68:in `connect_with_retry'

(...snipped...)

What is the problem?

I think that this problem will resolve if Influxer renames Influxer::Client#log method or Influxer::Client does not inherit InfluxDB::Client.

Here is my thought.


Influxer use influxdb-ruby internaly.

When http request timeout occured, this code is executed to log error message in influxdb-ruby.

But Influxer defines same name method which require only one argument in the same namespace.

So it seems like Influxer override log method which requires one argument, and Influxdb-ruby calls the log method with two arguments.

Finally, ArgumentError raised.


Can not write data

Bug report

System info: [Influxdb v1.2.0, influxdb-ruby v0.3.15, Influxer v1.0.0]

Steps to reproduce:

  1. create data.rb:
class Data < Influxer::Metrics
  set_series :data
  set_retention_policy :yearly

  attributes :value
end 
  1. 3.times { |i| Data.write(value: i) }

Expected behavior: Data stored in database

Actual behavior: Influx responses with 2xx, Influxer reports on successful storing, but there is no data in database

Additional info:
The old way of storing does not work:

$ curl -XPOST "http://localhost:8086/write?db=powerman" -d 'yearly.data value=1'
$ curl "http://localhost:8086/query?db=powerman&q=select+*+from+yearly.data"
{"results":[{"statement_id":0}]}

Influx expects retention policy is passed as a parameter (&rp=yearly):

$ curl -XPOST "http://localhost:8086/write?db=powerman&rp=yearly" -d 'data value=1'
$ curl "http://localhost:8086/query?db=powerman&q=select+*+from+yearly.data"
{"results":[{"statement_id":0,"series":[{"name":"data","columns":["time","value"],"values":[["2017-07-27T16:24:41.106977621Z",1]]}]}]}

See also:
https://github.com/influxdata/influxdb-ruby/blob/master/lib/influxdb/query/core.rb#L67
https://github.com/palkan/influxer/blob/master/lib/influxer/metrics/metrics.rb#L117

Rails generators

Something like:

rails generate metrics page_load req_time page_id load_time

force timestamp

It would be cool to be able to override the timestamp of the point.

Sometimes we need to process historical data and want to plot it with past dates, is this possible with influxer?

Ability to conncet to multiple databases/hosts

I have 3 processes that write into 3 databases on one host in an existing application (solaranzeige,de).

I want to summarize the data from these 3 processes (sum up power in watts, current in amperes and so on) and then write the data to another influx database with the same structure on another host, where Grafana is running.

As the standard ActiveRecord things like
self.abstract_class = true
establish_connection :central

do not work: How can I proceed to work with these 4 databases?

Influxer client does work with anyway config 2.1.0

What did you do?

A bundle update command updated anyway_Config to 2.1.0

What did you expect to happen?

Config should work as before or the gem should not be automatically updated

What actually happened?

The anyway config interface seems to have changes so that the initilization of the client config does not use the values defined in config/influx.yml in rails

Additional context

Environment

Ruby Version:
2.7.2

Influxer Version:
2.2.2

Influxer Client does not work with anyway_config 2.1.0

What did you do?

A bundle update command updated anyway_Config to 2.1.0

What did you expect to happen?

Config should work as before or the gem should not be automatically updated

What actually happened?

The anyway config interface seems to have changes so that the initilization of the client config does not use the values defined in config/influx.yml in rails

Additional context

Environment

Ruby Version:
2.7.2

Influxer Version:
2.2.2

Cache hash configuration cannot be applied

What did you do?

Tried to configure cache with expires_in value

What did you expect to happen?

Configuration with setting

cache: 
  expires_in: 60

should be loaded from yml file and assigned to the 'cache' key in config which defaults to false.

What actually happened?

Since the cache value is initializes as 'false' value, before loading and merging the yml file, anyway tries to assing hash values to 'false' object.

Possible solution could be to initialize the cache attribute of config to {enabled: false} or to fix anyway config to cope with those issues.

Additional context

backtrace:

     NoMethodError:
       undefined method `[]=' for false:FalseClass
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:56:in `block in merge_values'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:52:in `each'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:52:in `merge_values'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:54:in `block in merge_values'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:52:in `each'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:52:in `merge_values'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:174:in `trace!'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/anyway/loaders/yaml.rb:13:in `call'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/loaders/base.rb:10:in `call'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/config.rb:324:in `block in load_from_sources'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/loaders.rb:55:in `each'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/loaders.rb:55:in `each'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/config.rb:323:in `load_from_sources'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/config.rb:291:in `block in load'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/tracing.rb:132:in `capture'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/config.rb:288:in `load'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer/config.rb:37:in `load'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/gems/anyway_config-2.0.6/lib/.rbnext/2.8/anyway/config.rb:270:in `initialize'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer.rb:25:in `new'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer.rb:25:in `config'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer/client.rb:9:in `initialize'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer.rb:33:in `new'
     # /Users/jklimke/.rvm/gems/ruby-2.7.1/bundler/gems/influxer-18eb16522ad5/lib/influxer.rb:33:in `client'

Environment

Ruby Version:

2.7.1
Influxer Version:

Influxer::WhereClause#build_range error for negate=true case

the source code in https://github.com/palkan/influxer/blob/master/lib/influxer/metrics/relation/where_clause.rb#L60-L66 shows:

    def build_range(key, val, negate)
      if negate
        # FIX: the below maybe `or` instead of `and`  ???
        "#{key} < #{quoted(val.begin)} and #{key} > #{quoted(val.end)}"
      else
        "#{key} > #{quoted(val.begin)} and #{key} < #{quoted(val.end)}"
      end
    end

I didn't fork your repo and just reading your codes, I can fork it and send a new PR if needed.

ERR: invalid expression: 1 = 0

This may happen, when a custom scope is used, MWE:

scope :boom!, -> { where(time: []) }

It generates an incorrect query:

select * from year.trace where 1 = 0
ERR: invalid expression: 1 = 0

Probably, for influxdb time < 0 will be better than 1 = 0 as a clause returning none.

Depfu Error: No CI enabled on this project

Hello,

We've tried to activate or update your repository on Depfu and found something odd:

You don't seem to have some sort of CI or build service set up.

Without a service that will test the Depfu branches and pull requests, we can't guarantee you that
incoming updates of your will actually work with your dependencies. We think that this degrades the
service we're trying to provide down to a point where it is more or less meaningless.

What you can do now

  • Our friends at Travis-CI provide excellent service.
  • Circle CI are good, too, and have a free plan that will cover basic needs.
  • If you use something like Jenkins, make sure that you're using the github integration correctly so
    that it reports status data back to github.

It might be necessary to once deactivate and reactivate your project in Depfu for the CI service to
be properly detected.

If you don't want to set up CI for your project, please deactivate the project in Depfu, as we can't
do anything for you in this case.

How to deactivate the project

  • Login to Depfu
  • Go to the list of your repositories
  • Choose the correct organization
  • Disable the repo by clicking on the "Disable" button

If you think that this is a mistake

Please let us know by sending an email to [email protected].


This is an automated issue by Depfu. You're getting it because someone configured Depfu to automatically update dependencies on this project.

Rails 3 support

Hi.

First of all, thanks for the amazing library.

Would you like to support rails 3 or is it too old?

I'm working on that right now if you are interested.

Incorrect quoting of time for influx clients with a specific time_precision

Bug report

System info: [Influxdb v1.4.2, influxdb-ruby v0.3.17, Influxer v1.1.3]

Steps to reproduce:

  1. Use configuration with a specific time precision
Influxer.configure do |config|
   ....
  config.time_precision = 's'
end

  1. Execute any query with time filtering, for instance: where(time: time_vals), time_vals is an array of Time objects.

Expected behavior:
Influxer builds a correct query.

Actual behavior:
Influxer builds a query with an unquoted time parameter. Moreover, the parameter is formatted improperly (not rfc3339 as influx expects).

Additional info:
b7884c1#diff-9eee17a71fdc19385d4e9dec5fd4d31bL273

return val unless client.time_precision == 'ns'

query throw error after upgrade the gem to 0.3.0

I recently upgrade the gem from 0.2.4 to 0.3.0, as well as influxdb from 0.8.8 to 0.9.4.

I've got the following query:

VideoViewsMetrics.count('user_id').group(:traffic_source).fill(0).where(user_id: current_user.id).since(28.days.ago)

It starts to throw error afters upgrade to 0.3.0.

{"error":"error parsing query: found WHERE, expected SELECT, DELETE, SHOW, CREATE, DROP, GRANT, REVOKE, ALTER, SET at line 1, char 74"}

support for time_precision: 's'

The driver supports different time precisions, but influxer doesn't seem to. Here's what I need to do in my code to get the right times:

  def self.convert_time(time)
    time.to_i * 1_000_000_000 # shouldn't need this?
  end

  scope :between, lambda { |time1, time2|
    where(time: convert_time(time1)..convert_time(time2))
  }

  scope :at_time, lambda { |time|
    where("time <= #{convert_time(time)}")
  }

Can't delete data when retention policy is set for a metric

What did you do?

  1. Define a model with retention policy:
class ExampleMetrics < Influxer::Metrics
  set_retention_policy :yearly
end
  1. Delete data from model with retention policy:
ExampleMetrics.delete_all

What did you expect to happen?

Data is deleted from ExampleMetrics

What actually happened?

An influx error occurs and no data is deleted: ERR: error parsing query: retention policy not supported at line 1, char 1.

Additional context

Using the full name of a series(<retention_policy>.<series_name>) is currently not supported for DROP SERIES or DELETE FROM queries. Only the series name needs to be used. (See influxdata/influxdb#8088)

I will create an appropriate PR shortly.

Environment

Ruby Version: 2.6.5

Influxer Version: 1.2.1

Add calculation methods

Add support for calculations methods (such as count, mean, etc).

Metrics.count(:user_id)

Metrics.mean(:cpu_load)

Metrics.percentile(:req_time, 90)

Also think about multiple aggregations at once.

Bugs in wiki

build

point2 = MyMetrics.build user_id: 1, load_time: 0

MyMetrics do not respond to build but yes to new. build is just for Influxer::Relations

Found in https://github.com/palkan/influxer/wiki/Getting-Started#write-data-points

write!

Found in https://github.com/palkan/influxer/wiki/ActiveRecord-integration and https://github.com/palkan/influxer/wiki/Getting-Started#write-data-points

I tried to use write! for Influxer::Relations and for Influxer::Metrics objects but raise NoMethodError, instead I can use without problem write.

1.1.4 does not quote time ranges

Due to the changes done for version 1.1.4, time ranges are not escaped anymore in where clauses.

New quotation code:

def quoted(val, key = nil)
      if val.is_a?(String) || val.is_a?(Symbol) || @klass.tag?(key)
        "'#{val}'"
      elsif key.to_s == "time"
        quote_timestamp val, @instance.client
      else
        val.to_s
      end
    end

The code before checked if the value was a Time, DateTime, or Date instance and applied timestamp quoting appropriately:

  def quoted(val, key = nil)
       if val.is_a?(String) || val.is_a?(Symbol) || @klass.tag?(key)
         "'#{val}'"
-      elsif val.is_a?(Time) || val.is_a?(Date) || val.is_a?(DateTime)
+      elsif key.to_s == "time"
         quote_timestamp val, @instance.client
       else
         val.to_s
       end
     end

The solution would be either to pass the key with every call to quoted or to reintroduce time quoting by instance type

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.