honeycombio / libhoney-js Goto Github PK
View Code? Open in Web Editor NEWJavascript library for sending data to Honeycomb
License: Apache License 2.0
Javascript library for sending data to Honeycomb
License: Apache License 2.0
Is your feature request related to a problem? Please describe.
Our ALB currently has a timeout of 60s, so idle connections from libhoney that time out around the 60s mark may be dropped.
Describe the solution you'd like
Retry once if there is an http timeout, similar to what is done in libhoney-go.
Describe alternatives you've considered
Considering increasing the ALB timeout to 65s to avoid common 60s intervals, but libhoney transmission should retry once regardless because this issue could technically occur at any timeout interval.
Currently it isn't possible to set the event timestamp using sendNow
. Ideally if the data passed into sendNow
had a timestamp
field, that would be used as the timestamp of the event.
I'm trying to include libhoney-js
in a webpack build for the browser, but it results in a whole load of failures to find NodeJS modules dns
, fs
, net
, and tls
as part of the superagent-proxy
dependent module included here:
libhoney-js/src/transmission.js
Line 10 in 2fe34fe
Commenting out the superagent-proxy
import in dist/libhoney.es.js
resolves the problem.
My webpack configuration is nothing (no conf file at all), and this is the line I use to include libhoney-js
:
import libhoney from 'libhoney'
I suppose that this dependent module isn't meant to work in the browser, although libhoney-js
says it itself does - is this meant to be supported, or should I give up now? :)
Within the express example, the dynamic fields are added on every call to the middleware:
libhoney-js/examples/express-dynamic-fields/express-honey.js
Lines 4 to 13 in f90a4a6
Is this required behavior or can you register these callbacks before that function declaration?
Hi.
I noticed that Go version of libhoney does gzip compression, and most likely it HUGELY optimizes the ingestion traffic.
Is there a simple way to enable gzip compression level 1 for Node version too?
The Go and Python libraries currently support the ability to emit events to stdout, is this a supported configuration planned for this library?
I found a way of enabling it by setting transmission
to writer
and a place holder key which is required but not used currently.
Sadly this configuration almost works, the JSON is emitted as a quoted string, I am not sure why.
"{\"time\":\"2019-11-08T23:35:26.136Z\",\"samplerate\":1,\"data\":{\"meta.local_hostname\":\"test\",\"service_name\":\"my-lambda-service\",\"meta.type\":\"fastify\",\"meta.package_version\":\"2.10.0\",\"name\":\"request\",\"trace.trace_id_source\":null,\"request.host\":\"localhost:3000\",\"request.original_url\":\"/\",\"request.remote_addr\":\"127.0.0.1\",\"request.method\":\"GET\",\"request.route\":null,\"request.query\":{},\"request.http_version\":\"HTTP/1.1\",\"trace.trace_id\":\"7a033493-5bec-490c-a811-995053787d1e\",\"trace.span_id\":\"11071ea0-f247-44d3-a659-3fd4de12574f\",\"duration_ms\":7.573277,\"response.status_code\":\"200\",\"meta.instrumentations\":[\"fastify\",\"http\",\"https\"],\"meta.instrumentation_count\":3,\"meta.beeline_version\":\"1.8.3\",\"meta.node_version\":\"v12.8.0\"}}"
I am keen to use this library with AWS lambda in concert with the cloudwatch logs and https://github.com/honeycombio/honeyaws as access to the endpoints in the US add latency to our lambda operations.
If you attempt to require
the package the nodejs process crashes because the libhoney.cjs.js
file does not exist in the package:
Here is a runkit link that reproduces the behaviour: https://runkit.com/brmatt/runkit-npm-libhoney
I have a feeling this might affect several users of the package because beeline-nodejs
has a dependency on "libhoney": "^2.2.1"
, which would appear to treat 2.2.2
as a safe upgrade.
We'd like to take note of which version of Node is being used.
This will help in providing support and debug information for operational environments as well as give our team signals as to who may be impacted by dropping support for end-of-life language versions.
To follow other implementations, it should go in this order: libhoney/v (transmission/v) beeline/v language/v
I recently started using Libhoney and it's been great. Thanks for your work on it!
I ran into one issue, described below:
When Libhoney is initialized with the option disabled
set to true, subsequent calls to the flush function on the instance of Libhoney result in a TypeError:
TypeError: Cannot read property 'flush' of null
This happens because the getAndInitTransmission
function returns null when disabled is set to true.
The flush
function calls getAndInitTransmission
, gets null back, and then tries to call the flush function of the result of the call to getAndInitTransmission
:
flush() {
const transmission = this._transmission;
this._transmission = getAndInitTransmission(
this._options.transmission,
this._options
);
return transmission.flush();
}
This results in the TypeError.
import Libhoney from 'libhoney';
const honey = new Libhoney({
writeKey: 'someKey',
dataset: 'someDataSet',
disabled: true,
});
honey.send({
message: "Test Honeycomb event",
});
honey.flush();
and see the errorI'm happy to submit a pull request but would want advice on if you would prefer to return from the Libhoney flush function if getAndInitTransmission returns null or handle in some other way to let the caller know that nothing is happening.
Here's my stack trace, if that helps:
at Libhoney.flush (node_modules/libhoney/dist/libhoney.cjs.js:1274:25) at flush (handler.js:111:26) at tryCatch (node_modules/regenerator-runtime/runtime.js:45:40) at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:271:22) at Generator.prototype.(anonymous function) [as next] (node_modules/regenerator-runtime/runtime.js:97:21) at asyncGeneratorStep (node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24) at _next (node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
Is your feature request related to a problem? Please describe.
Related to #88
TypeScript users are missing out on the benefits of a typed API for the library, because we don't have types.
Describe the solution you'd like
Add a type declaration file.
This is the first step in the TS port project, which will allow us to understand the library API and provide user value faster, while we continue on porting the rest of the library to TS.
Describe alternatives you've considered
Wait until we complete the TS port. In the long run, it's more maintainable to have the source code written in TS instead of the standalone types file. We intend to replace the standalone types file soon, so the tradeoff seems worthwhile in this case.
Additional context
We want to have a useful working example in the repo. To make sure it doesn't get out of date, and to gain more confidence when merging PRs, the example should be exercised in CI.
The disabled flag isn't actually in the released version on npm. This is currently my initial setup:
const hny = new Libhoney({
writeKey: process.env.HONEYCOMB_WRITE_KEY || 'something-non-empty',
dataset: process.env.HONEYCOMB_DATASET || 'linc-local-serve',
disabled: !process.env.HONEYCOMB_WRITE_KEY
});
Not pretty, but if I don't have the || 'something-non-empty'
I get a validation error for every request when no WRITE_KEY is passed in.
The type declaration file added in #277 should be checked in CI for validity, so we feel confident that when users use it, it doesn't cause errors.
At the very least we should run a TS linter. Are there other automated ways we can check the file?
Honeycomb uses write keys for auth tokens, and putting them on a webpage represents making public something you probably don't want to make public.
The actual work to fix this would live in the honeycomb service code, but putting an issue here to track progress on it.
Versions
Description
In vm2 for versions up to 3.9.19, Promise handler sanitization can be bypassed, allowing attackers to escape the sandbox and run arbitrary code. The vm2 project alerts to the existence of critical security issues and claims to have been discontinued, therefore recommending replacing vm2 with isolated-vm.
Hi.
I looked at the number of TCP (actually SSL) connections per second our infra opens, and it appeared that libhoney produces the absolute most of them. And SSL connections are CPU expensive (they even block event loop sensibly if done simultaneously many times) - we recently shaved off a lot by e.g. lowering the number of connections we reopen with pgbouncer.
Sounds like libhoney doesn't support persistent (keep-alive) connections?
Meanwhile, superagent supports passing a custom instance of Agent, which can be agentkeepalive. But libhoney doesn't expose an API to actually pass a custom Agent instance. (And even without agentkeepalive module, keep-alive mode can probably be turned on by sending a Keep-Alive http header - which libhoney doesn’t support injection of as well).
When writing to STDOUT, a writeKey is unnecessary and would save having to provide a writeKey to application lambda functions and publishing event data asynchronously from cloudwatch.
Encountered this error .writeKey must be a non-empty string
when setting transmission: "console"
and not supplying the writeKey.
This line breaks any downstream project which wishes to use Yarn, even transitively via honeycomb-beeline
:
Line 12 in a8cee1a
error [email protected]: The engine "yarn" is incompatible with this module. Expected version "YARN NO LONGER USED - use npm instead.". Got "1.22.5"
error Found incompatible module.
I'm guessing you didn't intentionally block all downstream projects from using Yarn :)
Format an event from the data shape that is sent to the API, then write that to stdout such that the events transmitted via the extension are realized properly in web app including traces.
https://docs.honeycomb.io/getting-data-in/integrations/aws/aws-lambda/
When using libhoney-js to send events/spans to Honeycomb, Chrome and Safari log a console error stating :client.js:759 Refused to set unsafe header "User-Agent"
. Per discussion in the Honeycomb Pollinator #general channel, it should be detected that we're running in a browser and avoid setting the User-Agent header. The UA is set in line 311 of transmission.js: https://github.com/honeycombio/libhoney-js/blob/master/src/transmission.js#L311
Trying to use libhoney
directly from the browser (CRA built with webpack)
import Libhoney from "libhoney";
let hny = new Libhoney({
writeKey: "API_KEY",
dataset: "DATASET_NAME",
// disabled: true, // uncomment for testing or development.
responseCallback: responses => {
responses.forEach(resp => {
console.log(resp);
});
}
});
let params = {
hostname: window.location.href,
applicationName: "test"
}
hny.add(params);
let builder = hny.newBuilder({ built: true });
let event = builder.newEvent();
event.addField("applicationName", "test");
event.addField("correlationId", "8f962726-57fc-441f-ab14-010a1921172e");
event.timestamp = new Date(Date.UTC(2016, 1, 29, 1, 1, 1));
event.metadata = { id: "recognize-me-later" };
event.send();
results in console output :
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.honeycomb.io/1/batch/datasetnamehere.
(Reason: missing token 'content-type' in CORS header 'Access-Control-Allow-Headers' from CORS preflight channel).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.honeycomb.io/1/batch/datasetnamehere.
(Reason: CORS request did not succeed).
{"duration":294,"metadata":{"id":"recognize-me-later"},"error":{"crossDomain":true,"method":"POST","url":"https://api.honeycomb.io/1/batch/DATASET_NAME"}}
preflight requests are made to:
Request URL:https://api.honeycomb.io/1/batch/DATASET_NAME
Request method:OPTIONS
Remote address:REDACTED:443
Status code:200
Version:HTTP/1.1
Referrer Policy:no-referrer-when-downgrade
Request headers:
Host: api.honeycomb.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: */*
Accept-Language: en-GB,pl;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,user-agent,x-hny-team
Referer: http://localhost:3000/test
Origin: http://localhost:3000
Connection: keep-alive
Response headers
Host: api.honeycomb.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: */*
Accept-Language: en-GB,pl;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,user-agent,x-hny-team
Referer: http://localhost:3000/test
Origin: http://localhost:3000
Connection: keep-alive
Did anyone came across same/similiar issue?
I need to disable libhoney in test and development environments. However if you pass an empty string as apiHost
(or writeKey
) you get errors printed in your output. This is not great for testing or in situations where you have an alternative logger.
Thanks!
Trying to run esdoc
(version 1.1.0
) spits back:
parse: /Users/cyen/src/honeycomb/libhoney-js/src/.external-ecmascript.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/builder_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/event_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/libhoney_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/transmission_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/builder.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/event.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/foreach.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/libhoney.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/transmission.js
/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:269
const body = node.body;
^
TypeError: Cannot read property 'body' of undefined
at Plugin._inferenceReturn (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:269:23)
at Plugin._inferenceMethod (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:31:42)
at Plugin._exec (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:17:10)
at Plugin.onHandleDocs (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:11:10)
at Plugin._execHandler (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/Plugin/Plugin.js:55:26)
at Plugin.onHandleDocs (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/Plugin/Plugin.js:135:10)
at Function.generate (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDoc.js:132:32)
at ESDocCLI.exec (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDocCLI.js:71:23)
at Object.<anonymous> (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDocCLI.js:182:7)
at Module._compile (module.js:649:30)
When shows only past 60 days dates in date range calendar. But when we select current month date its shows future month date also
Please let me know any face this issues before
The docs say to install using npm install libhoney --save-dev
. That should probably be npm install libhoney --save
Presumably the honeycomb library should be available in production :)
When I am trying to send my own timestamp,
const ev = hny.newEvent();
ev.addField('type', 'type');
ev.addField("value", 'event');
// ev.timestamp = new Date(Date.UTC(2018, 11, 5, 15, 29, 0));
ev.timestamp = '2018-12-06T14:12:06Z';
ev.metadata = { projectId: ''someId" };
ev.send();
It appears as added in rows examined: 30
But doesn't appear in the results
I have this record but with 7 hours of difference, how can I change it?
I noticed a fairly significant change in memory usage in my Node process after adjusting the sample rate for a high-frequency event, which led me to find that libhoney
leaks memory proportional to the number of events that have been sent, up to a ceiling that's proportional to the size of the response queue.
Appropriately enough, the data collected we're collecting via libhoney
helped diagnose the root cause -- we include the total allocated process memory in each event and after reducing the size of the event queue from 10k to 1k events and disabling response queueing entirely, we found that memory usage dropped by about 60%.
It looks like the bug is in _responseCallback()
:
Line 120 in 4209b25
Responses are pushed to the response queue on receipt, but even if the events are being consumed, they're never removed and so can't be garbage collected. With default settings this causes libhoney to eventually consume (10,000 * average_response_size)
bytes, so if response objects are reasonably large (more than a few Kb) this can approach a couple hundred Mb total.
Happy to try drafting a PR for this, but leaving record here since I'm unsure if I'll have time to anytime soon.
In Ruby this is: https://github.com/honeycombio/libhoney-rb/blob/bacadc7ae70ebe4cf8f68cca628f375a337f3a77/lib/libhoney/event.rb#L21
And I would expect the same here:
Line 58 in fdf579f
Not sure if this results in missing timestamps, or they get filled in at the event collector.
Is your feature request related to a problem? Please describe.
Describe the solution you'd like
Add TypeScript build process to the project.
any
)Similar to the inclusion of a user agent string, include the node version of the process sending the event.
This would be similar to the way beeline-nodejs sends the node version as part of its events. See here for its implementation detail: https://github.com/honeycombio/beeline-nodejs/blob/d1895731cd52e1fd848fe62f6bb5457c678aeec5/lib/api/libhoney.js#L221
part of the honeycomb sdk spec deals with how to handle responses from the api host: http://localhost:1313/docs/send-data/sdks/#responses
the JS sdk is different from go/python in that it can't assume threads, so maybe we can:
send
apis and deal with the response locally?neither sounds particularly nice.
First, I want to say I'm really liking Honeycomb so far! I used to use Scuba at FB and Honeycomb has been pretty good at filling that gap in my toolkit since leaving.
I see Typescript type definitions were added in #52 and then removed in #66. Curious, was there an issue with those types?
I'm working on instrumenting all our services with Honeycomb, but our new services are 100% Typescript with strict linting with type checking, making integrating Libhoney (and Beeline) a bit of a hassle. Happy to see Beeline now has a PR to add type definitions but I'd love to see the same in Libhoney too, as I want to log some events independently of what Beeline provides.
This is perhaps less of a problem now that batch support is in, but ideally the cost of using libhoney from within a webpage would be essentially zero.
If we can push all the batching/sending to a web worker, we should absolutely do that.
Would you accept a PR to add a .d.ts
file to support TypeScript consumers?
Given a libhoney configured with a v2 api key and without a dataset
When it sends HNY events
Then it should use unknown_dataset
as the dataset
When given a legacy key, the behavior remains the same as today.
Chatted with @pkanal and we think we should only keep and test for Node 14, 16, 18. The others are no longer security supported.
Testing end-of-life versions of node also blocks dependency updates (see: #293)
Places to change this:
I could be doing something wrong, but it looks like
Line 208 in f7ce6c0
this._fields
and this._dyn_fields
, when it seems like the relevant values are stored on this._builder
.Versions
Description
Dependent module superagent had a public vulnerability with formidable dependency in version 8, and hence have released v9.0.0+ with the fix.
More info is present in the link: ladjs/superagent#1800
Can you please look into it and upgrade dependency for superagent accordingly.
(Please include any relevant CVE advisory links)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.