Giter Site home page Giter Site logo

logs_api_integration's Introduction

Russian version

Integration with Logs API

This script can help you to integrate Yandex.Metrica Logs API with ClickHouse.

If you have any questions, feel free to write comments, create issues on GitHub or write me (e-mail: [email protected]).

Requirements

Script uses Python 2.7 and also requires requests library. You can install this library using package manager pip

pip install requests

Also, you need a running ClickHouse instance to load data into it. Instruction how to install ClickHouse can be found on official site.

Setting up

First of all, you need to fill in config

{
	"token" : "<your_token>", // token to access Yandex.Metrica API
	"counter_id": "<your_counter_id>",
	"visits_fields": [ // list of params for visits
	    "ym:s:counterID",
	    "ym:s:dateTime",
	    "ym:s:date",
	    "ym:s:firstPartyCookie"
	],
	"hits_fields": [ // list of params for hits
	    "ym:pv:counterID",
	    "ym:pv:dateTime",
	    "ym:pv:date",
	    "ym:pv:firstPartyCookie"
	],
	"log_level": "INFO", 
	"retries": 1, 
	"retries_delay": 60, // delay between retries
	"clickhouse": {
		"host": "http://localhost:8123", 
		"user": "", 
		"password": "",
		"visits_table": "visits_all", // table name for visits
		"hits_table": "hits_all", // table name for hits
		"database": "default" // database name
	}
}

On first execution script creates all tables in database according to config. So if you change parameters, you need to drop all tables and load data again or add new columns manually using ALTER TABLE.

Running a program

When running the program you need to specify a souce (hits or visits) using option -source.

Script has several modes:

  • history - loads all the data from day one to the day before yesterday
  • regular - loads data only for day before yesterday (recommended for regular downloads)
  • regular_early - loads yesterday data (yesterday data may be not complete: some visits can lack page views)

Example:

python metrica_logs_api.py -mode history -source visits

Also you can load data for particular time period:

python metrica_logs_api.py -source hits -start_date 2016-10-10 -end_date 2016-10-18

logs_api_integration's People

Contributors

lan143 avatar miptgirl avatar vmoshikov avatar yndx-metrika avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.