Comments (4)
Добрый день!
Я попробовала запустить скрипт с аналогичными настройками на чистой БД и проблема не воспроизводится.
Предположу, что после предыдущего запуска изменился список выгружаемых полей в конфиге и новые данные не могут записаться в старую структуру таблиц. В этом случае стоит удалить таблицу и перезапустить скрипт, чтобы он создал таблицу нужной структуры.
from logs_api_integration.
Взял другой счетчик, другой набор полей, другую операционную систему 😎, чистую БД, проблема так же выглядит:
('##### python', '2.7.11')
2017-08-06 16:04:59 MainProcess INFO CLI Options: Namespace(end_date=None, mode='history', source='hits', start_date=None)
2017-08-06 16:04:59 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:04:59 MainProcess INFO UserRequest(token=u'xxx', counter_id=u'3199561', start_date_str=u'2011-01-11', end_date_str='2017-08-04', source='hits', fields=(u'ym:pv:counterID', u'ym:pv:dateTime', u'ym:pv:date', u'ym:pv:clientID', u'ym:pv:title', u'ym:pv:URL', u'ym:pv:referer'))
2017-08-06 16:04:59 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:04:59 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:04:59 MainProcess INFO ### CREATING TASK
2017-08-06 16:04:59 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
{
"date1_str": "2011-01-11",
"date2_str": "2011-07-29",
"request_id": 194331,
"status": "created",
"user_request": [
"xxx",
"3199561",
"2011-01-11",
"2017-08-04",
"hits",
[
"ym:pv:counterID",
"ym:pv:dateTime",
"ym:pv:date",
"ym:pv:clientID",
"ym:pv:title",
"ym:pv:URL",
"ym:pv:referer"
]
]
}
2017-08-06 16:04:59 MainProcess INFO ### DELAY 20 secs
2017-08-06 16:05:19 MainProcess INFO ### CHECKING STATUS
2017-08-06 16:05:19 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:05:19 MainProcess INFO API Request status: created
2017-08-06 16:18:02 MainProcess INFO ### DELAY 20 secs
2017-08-06 16:18:22 MainProcess INFO ### CHECKING STATUS
2017-08-06 16:18:22 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:18:22 MainProcess INFO API Request status: created
2017-08-06 16:18:22 MainProcess INFO ### DELAY 20 secs
2017-08-06 16:18:42 MainProcess INFO ### CHECKING STATUS
2017-08-06 16:18:42 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:18:42 MainProcess INFO API Request status: processed
2017-08-06 16:18:42 MainProcess INFO ### SAVING DATA
2017-08-06 16:18:42 MainProcess INFO Part #0
2017-08-06 16:18:42 MainProcess INFO Starting new HTTPS connection (1): api-metrika.yandex.ru
2017-08-06 16:18:42 MainProcess INFO ### DATA SAMPLE
2017-08-06 16:18:42 MainProcess INFO ym:pv:clientID ym:pv:counterID ym:pv:date ym:pv:dateTime ym:pv:referer ym:pv:title ym:pv:URL
2017-08-06 16:18:42 MainProcess WARNING 1 rows were filtered out
2017-08-06 16:18:42 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:18:42 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:18:42 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:18:42 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:18:42 MainProcess INFO Starting new HTTP connection (1): localhost
2017-08-06 16:18:43 MainProcess CRITICAL Iteration #1 failed
Traceback (most recent call last):
File "metrica_logs_api.py", line 127, in <module>
integrate_with_logs_api(config, user_request)
File "metrica_logs_api.py", line 107, in integrate_with_logs_api
raise e
ValueError: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected \n at end of stream., e.what() = DB::Exception
from logs_api_integration.
Видимо, я что-то очень не так делаю. Как бы понять, что именно?
from logs_api_integration.
При том, скрипт создал нормальную структуру БД. Я скачал кусок с метрики запросом wget https://api-metrika.yandex.ru/management/v1/counter/3199561/logrequest/194328/part/1/download?auth_token=xxx -O c1
, и нормально загружается руками cat c1 | sed "1d" | clickhouse-client --query="insert into qwe1.hits_all FORMAT TabSeparated"
:) видимо вопрос к скрипту загрузки
from logs_api_integration.
Related Issues (20)
- Divison by zero when estimation returns max_possible_day_quantity = 0 HOT 1
- Ошибка TypeError при выгрузке данных HOT 3
- Missing fields in ch_types.json
- Oauth token must be passed in headers (not just in url) HOT 1
- Ошибка 'map' object is not subscriptable HOT 1
- Некорректная работа с запросами без поля Date
- Starting new HTTPS connection (1)
- Где найти значение "host"
- LastDirectClickOrder, OfflineCallFirstTimeCaller - Типы неверны.
- max_possible_day_quantity < 0 HOT 7
- 'map' object is not subscriptable
- raise ConnectionError(e, request=request)
- 'ascii' codec can't encode characters in position HOT 1
- Cannot parse input: expected [ before: "[\' for array with Cyrillic strings HOT 1
- Unknown field found in TSV header: 'ym:s:clientID' HOT 3
- Нужно изменить на UInt64, потому что в директе id группы пошли уже больше UInt32
- Ошибка Cannot parse input: expected '[' before: '"[\'
- Ошибка 117, DB::Exception: Unknown field found in TSV header: '1613536812404233133' at position 0 HOT 3
- Code: 130. DB::ParsingException: Cannot read array from text, expected comma or end of array HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logs_api_integration.