honeycombio/libhoney-js

Javascript library for sending data to Honeycomb

libhoney-js's Issues

Retry once on send timeout

Is your feature request related to a problem? Please describe.

Our ALB currently has a timeout of 60s, so idle connections from libhoney that time out around the 60s mark may be dropped.

Describe the solution you'd like

Retry once if there is an http timeout, similar to what is done in libhoney-go.

Describe alternatives you've considered

Considering increasing the ALB timeout to 65s to avoid common 60s intervals, but libhoney transmission should retry once regardless because this issue could technically occur at any timeout interval.

[feature] ability to set timestamp in sendNow

Currently it isn't possible to set the event timestamp using sendNow. Ideally if the data passed into sendNow had a timestamp field, that would be used as the timestamp of the event.

Unable to include in build with Webpack for browsers

I'm trying to include libhoney-js in a webpack build for the browser, but it results in a whole load of failures to find NodeJS modules dns, fs, net, and tls as part of the superagent-proxy dependent module included here:

libhoney-js/src/transmission.js

Line 10 in 2fe34fe

import proxy from "superagent-proxy";

Commenting out the superagent-proxy import in dist/libhoney.es.js resolves the problem.

My webpack configuration is nothing (no conf file at all), and this is the line I use to include libhoney-js:

import libhoney from 'libhoney'

I suppose that this dependent module isn't meant to work in the browser, although libhoney-js says it itself does - is this meant to be supported, or should I give up now? :)

Dynamic Field registration time

Within the express example, the dynamic fields are added on every call to the middleware:

libhoney-js/examples/express-dynamic-fields/express-honey.js

Lines 4 to 13 in f90a4a6

	module.exports = function(options) {
	var honey = new libhoney(options);
	return function(req, res, next) {
	// Attach dynamic fields to the global event builder in libhoney.
	// Dynamic fields calculate their values at the time the event is created
	// (the event.sendNow call below)
	honey.addDynamicField('rss_after', () => process.memoryUsage().rss);
	honey.addDynamicField('heapTotal_after', () => process.memoryUsage().heapTotal);
	honey.addDynamicField('heapUsed_after', () => process.memoryUsage().heapUsed);

Is this required behavior or can you register these callbacks before that function declaration?

Does this library support gzip-level-1 compression?

Hi.

I noticed that Go version of libhoney does gzip compression, and most likely it HUGELY optimizes the ingestion traffic.

Is there a simple way to enable gzip compression level 1 for Node version too?

Support for writing to STDOUT for AWS users?

The Go and Python libraries currently support the ability to emit events to stdout, is this a supported configuration planned for this library?

I found a way of enabling it by setting transmission to writer and a place holder key which is required but not used currently.

Sadly this configuration almost works, the JSON is emitted as a quoted string, I am not sure why.

"{\"time\":\"2019-11-08T23:35:26.136Z\",\"samplerate\":1,\"data\":{\"meta.local_hostname\":\"test\",\"service_name\":\"my-lambda-service\",\"meta.type\":\"fastify\",\"meta.package_version\":\"2.10.0\",\"name\":\"request\",\"trace.trace_id_source\":null,\"request.host\":\"localhost:3000\",\"request.original_url\":\"/\",\"request.remote_addr\":\"127.0.0.1\",\"request.method\":\"GET\",\"request.route\":null,\"request.query\":{},\"request.http_version\":\"HTTP/1.1\",\"trace.trace_id\":\"7a033493-5bec-490c-a811-995053787d1e\",\"trace.span_id\":\"11071ea0-f247-44d3-a659-3fd4de12574f\",\"duration_ms\":7.573277,\"response.status_code\":\"200\",\"meta.instrumentations\":[\"fastify\",\"http\",\"https\"],\"meta.instrumentation_count\":3,\"meta.beeline_version\":\"1.8.3\",\"meta.node_version\":\"v12.8.0\"}}"

I am keen to use this library with AWS lambda in concert with the cloudwatch logs and https://github.com/honeycombio/honeyaws as access to the endpoints in the US add latency to our lambda operations.

v2.2.2 does not include the compiled code, and crashes the process when required

If you attempt to require the package the nodejs process crashes because the libhoney.cjs.js file does not exist in the package:

Here is a runkit link that reproduces the behaviour: https://runkit.com/brmatt/runkit-npm-libhoney

I have a feeling this might affect several users of the package because beeline-nodejs has a dependency on "libhoney": "^2.2.1", which would appear to treat 2.2.2 as a safe upgrade.

Add node version to the user-agent header

We'd like to take note of which version of Node is being used.

This will help in providing support and debug information for operational environments as well as give our team signals as to who may be impacted by dropping support for end-of-life language versions.

• Add node version to the user agent header similar to how we've done with Ruby and Python
• Follow the the user agent guidance at MDN re: format.

To follow other implementations, it should go in this order: libhoney/v (transmission/v) beeline/v language/v

Calling Libhoney.flush() results in TypeError when options.disabled === true

I recently started using Libhoney and it's been great. Thanks for your work on it!

I ran into one issue, described below:

Issue Description

When Libhoney is initialized with the option disabled set to true, subsequent calls to the flush function on the instance of Libhoney result in a TypeError:

TypeError: Cannot read property 'flush' of null

This happens because the getAndInitTransmission function returns null when disabled is set to true.

The flush function calls getAndInitTransmission, gets null back, and then tries to call the flush function of the result of the call to getAndInitTransmission:

 flush() {
    const transmission = this._transmission;

    this._transmission = getAndInitTransmission(
      this._options.transmission,
      this._options
    );

    return transmission.flush();
  }

This results in the TypeError.

Steps to reproduce

1. Create a new JavaScript file (I'm running unit tests with Jest, but this should work regardless of how you want to host your process)
2. Import Libhoney and create a new instance of the Libhoney class (make sure to set disabled to true):

import Libhoney from 'libhoney';
const honey = new Libhoney({
  writeKey: 'someKey',
  dataset: 'someDataSet',
  disabled: true,
});

3. Send an event:

honey.send({
  message: "Test Honeycomb event",
});

4. Call honey.flush(); and see the error

Additional Information

I'm happy to submit a pull request but would want advice on if you would prefer to return from the Libhoney flush function if getAndInitTransmission returns null or handle in some other way to let the caller know that nothing is happening.

Here's my stack trace, if that helps:

  at Libhoney.flush (node_modules/libhoney/dist/libhoney.cjs.js:1274:25)
  at flush (handler.js:111:26)
  at tryCatch (node_modules/regenerator-runtime/runtime.js:45:40)
  at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:271:22)
  at Generator.prototype.(anonymous function) [as next] (node_modules/regenerator-runtime/runtime.js:97:21)
  at asyncGeneratorStep (node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
  at _next (node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)

Add type declaration file

Is your feature request related to a problem? Please describe.
Related to #88
TypeScript users are missing out on the benefits of a typed API for the library, because we don't have types.

Describe the solution you'd like
Add a type declaration file.

This is the first step in the TS port project, which will allow us to understand the library API and provide user value faster, while we continue on porting the rest of the library to TS.

Describe alternatives you've considered
Wait until we complete the TS port. In the long run, it's more maintainable to have the source code written in TS instead of the standalone types file. We intend to replace the standalone types file soon, so the tradeoff seems worthwhile in this case.

Additional context

Example app is checked in CI

We want to have a useful working example in the repo. To make sure it doesn't get out of date, and to gain more confidence when merging PRs, the example should be exercised in CI.

Could you publish 1.0.0-beta8

The disabled flag isn't actually in the released version on npm. This is currently my initial setup:

const hny = new Libhoney({
    writeKey: process.env.HONEYCOMB_WRITE_KEY || 'something-non-empty',
    dataset: process.env.HONEYCOMB_DATASET || 'linc-local-serve',
    disabled: !process.env.HONEYCOMB_WRITE_KEY
});

Not pretty, but if I don't have the || 'something-non-empty' I get a validation error for every request when no WRITE_KEY is passed in.

Type declaration file is checked in CI

The type declaration file added in #277 should be checked in CI for validity, so we feel confident that when users use it, it doesn't cause errors.

At the very least we should run a TS linter. Are there other automated ways we can check the file?

libhoney is not safe for web pages

Honeycomb uses write keys for auth tokens, and putting them on a webpage represents making public something you probably don't want to make public.

The actual work to fix this would live in the honeycomb service code, but putting an issue here to track progress on it.

sec: critical vulnerability in transitive dep vm2 (via superagent-proxy)

Versions

• Libhoney: v4.0.1

Description

In vm2 for versions up to 3.9.19, Promise handler sanitization can be bypassed, allowing attackers to escape the sandbox and run arbitrary code. The vm2 project alerts to the existence of critical security issues and claims to have been discontinued, therefore recommending replacing vm2 with isolated-vm.

• CVE-2023-37466 (GHSA-g644-9gfx-q4q4)
• GHSA-g644-9gfx-q4q4

Keep-alive HTTP connections with api.honeycomb.io?

Hi.

I looked at the number of TCP (actually SSL) connections per second our infra opens, and it appeared that libhoney produces the absolute most of them. And SSL connections are CPU expensive (they even block event loop sensibly if done simultaneously many times) - we recently shaved off a lot by e.g. lowering the number of connections we reopen with pgbouncer.

Sounds like libhoney doesn't support persistent (keep-alive) connections?

Meanwhile, superagent supports passing a custom instance of Agent, which can be agentkeepalive. But libhoney doesn't expose an API to actually pass a custom Agent instance. (And even without agentkeepalive module, keep-alive mode can probably be turned on by sending a Keep-Alive http header - which libhoney doesn’t support injection of as well).

When transmission is set to console, allow the writekey to be undefined / empty string

When writing to STDOUT, a writeKey is unnecessary and would save having to provide a writeKey to application lambda functions and publishing event data asynchronously from cloudwatch.

Encountered this error .writeKey must be a non-empty string when setting transmission: "console" and not supplying the writeKey.

No-yarn engine requirement breaks use as dependency

This line breaks any downstream project which wishes to use Yarn, even transitively via honeycomb-beeline:

libhoney-js/package.json

Line 12 in a8cee1a

"yarn": "YARN NO LONGER USED - use npm instead."

error [email protected]: The engine "yarn" is incompatible with this module. Expected version "YARN NO LONGER USED - use npm instead.". Got "1.22.5"
error Found incompatible module.

I'm guessing you didn't intentionally block all downstream projects from using Yarn :)

Add support for use in conjunction with the AWS extension

Format an event from the data shape that is sent to the API, then write that to stdout such that the events transmitted via the extension are realized properly in web app including traces.

https://docs.honeycomb.io/getting-data-in/integrations/aws/aws-lambda/

Support marker

'User-Agent' header console error in Browser (detect if in browser and avoid setting UA)

When using libhoney-js to send events/spans to Honeycomb, Chrome and Safari log a console error stating :client.js:759 Refused to set unsafe header "User-Agent". Per discussion in the Honeycomb Pollinator #general channel, it should be detected that we're running in a browser and avoid setting the User-Agent header. The UA is set in line 311 of transmission.js: https://github.com/honeycombio/libhoney-js/blob/master/src/transmission.js#L311

Cross-Origin Request Blocked

Trying to use libhoney directly from the browser (CRA built with webpack)

import Libhoney from "libhoney";

let hny = new Libhoney({
  writeKey: "API_KEY",
  dataset: "DATASET_NAME",
  // disabled: true, // uncomment for testing or development.
  responseCallback: responses => {
    responses.forEach(resp => {
      console.log(resp);
    });
  }
});


let params = {
  hostname: window.location.href,
  applicationName: "test"
}

hny.add(params);

let builder = hny.newBuilder({ built: true });
let event = builder.newEvent();

event.addField("applicationName", "test");
event.addField("correlationId", "8f962726-57fc-441f-ab14-010a1921172e");

event.timestamp = new Date(Date.UTC(2016, 1, 29, 1, 1, 1));
event.metadata = { id: "recognize-me-later" };
event.send();

results in console output :

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.honeycomb.io/1/batch/datasetnamehere. 
(Reason: missing token 'content-type' in CORS header 'Access-Control-Allow-Headers' from CORS preflight channel).

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.honeycomb.io/1/batch/datasetnamehere. 
(Reason: CORS request did not succeed).

{"duration":294,"metadata":{"id":"recognize-me-later"},"error":{"crossDomain":true,"method":"POST","url":"https://api.honeycomb.io/1/batch/DATASET_NAME"}}

preflight requests are made to:

Request URL:https://api.honeycomb.io/1/batch/DATASET_NAME
Request method:OPTIONS
Remote address:REDACTED:443
Status code:200
Version:HTTP/1.1
Referrer Policy:no-referrer-when-downgrade

Request headers:

Host: api.honeycomb.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: */*
Accept-Language: en-GB,pl;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,user-agent,x-hny-team
Referer: http://localhost:3000/test
Origin: http://localhost:3000
Connection: keep-alive

Response headers

Host: api.honeycomb.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: */*
Accept-Language: en-GB,pl;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,user-agent,x-hny-team
Referer: http://localhost:3000/test
Origin: http://localhost:3000
Connection: keep-alive

Did anyone came across same/similiar issue?

[feature] option to disable libhoney

I need to disable libhoney in test and development environments. However if you pass an empty string as apiHost (or writeKey) you get errors printed in your output. This is not great for testing or in situations where you have an alternative logger.

Thanks!

Get ESDoc generation working again

Trying to run esdoc (version 1.1.0) spits back:

parse: /Users/cyen/src/honeycomb/libhoney-js/src/.external-ecmascript.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/builder_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/event_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/libhoney_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/__tests__/transmission_test.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/builder.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/event.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/foreach.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/libhoney.js
parse: /Users/cyen/src/honeycomb/libhoney-js/src/transmission.js
/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:269
    const body = node.body;
                      ^

TypeError: Cannot read property 'body' of undefined
    at Plugin._inferenceReturn (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:269:23)
    at Plugin._inferenceMethod (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:31:42)
    at Plugin._exec (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:17:10)
    at Plugin.onHandleDocs (/Users/cyen/src/honeycomb/libhoney-js/node_modules/esdoc-type-inference-plugin/src/Plugin.js:11:10)
    at Plugin._execHandler (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/Plugin/Plugin.js:55:26)
    at Plugin.onHandleDocs (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/Plugin/Plugin.js:135:10)
    at Function.generate (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDoc.js:132:32)
    at ESDocCLI.exec (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDocCLI.js:71:23)
    at Object.<anonymous> (/Users/cyen/.nvm/versions/node/v9.9.0/lib/node_modules/esdoc/out/src/ESDocCLI.js:182:7)
    at Module._compile (module.js:649:30)

Date range picker issues

When shows only past 60 days dates in date range calendar. But when we select current month date its shows future month date also

Please let me know any face this issues before

docs for libhoney options

right now the docs for libhoney's constructor don't mention any of the properties you can override:

this is pretty terrible, but I couldn't figure out how to get esdoc to let me specify name/type/attribute/description for each of the properties.

Docs bug: libhoney should probably not be a devDependency

The docs say to install using npm install libhoney --save-dev. That should probably be npm install libhoney --save Presumably the honeycomb library should be available in production :)

timezone libhoney

When I am trying to send my own timestamp,

  const ev = hny.newEvent();
  ev.addField('type', 'type');
  ev.addField("value", 'event');
  // ev.timestamp = new Date(Date.UTC(2018, 11, 5, 15, 29, 0));
  ev.timestamp = '2018-12-06T14:12:06Z';
  ev.metadata = { projectId:  ''someId" };
  ev.send();

It appears as added in rows examined: 30
But doesn't appear in the results

I have this record but with 7 hours of difference, how can I change it?

Queued responses aren't garbage collected, leaking memory

I noticed a fairly significant change in memory usage in my Node process after adjusting the sample rate for a high-frequency event, which led me to find that libhoney leaks memory proportional to the number of events that have been sent, up to a ceiling that's proportional to the size of the response queue.

Appropriately enough, the data collected we're collecting via libhoney helped diagnose the root cause -- we include the total allocated process memory in each event and after reducing the size of the event queue from 10k to 1k events and disabling response queueing entirely, we found that memory usage dropped by about 60%.

It looks like the bug is in _responseCallback():

libhoney-js/src/libhoney.js

Line 120 in 4209b25

_responseCallback(responses) {

Responses are pushed to the response queue on receipt, but even if the events are being consumed, they're never removed and so can't be garbage collected. With default settings this causes libhoney to eventually consume (10,000 * average_response_size) bytes, so if response objects are reasonably large (more than a few Kb) this can approach a couple hundred Mb total.

Happy to try drafting a PR for this, but leaving record here since I'm unsure if I'll have time to anytime soon.

Missing default to new Date()

In Ruby this is: https://github.com/honeycombio/libhoney-rb/blob/bacadc7ae70ebe4cf8f68cca628f375a337f3a77/lib/libhoney/event.rb#L21

And I would expect the same here:

libhoney-js/src/event.js

Line 58 in fdf579f

this.timestamp = null;

Not sure if this results in missing timestamps, or they get filled in at the event collector.

Add TypeScript to project

Is your feature request related to a problem? Please describe.

• Related to #88

Describe the solution you'd like
Add TypeScript build process to the project.

• all existing source and tests files are converted to TS with all types left open (any)
• tests continue to work
• code is compiled in CI
• library release process continues to generate a JS package
• type declaration file from #277 continues to be shipped with a released package

Include node version as field in all events

Similar to the inclusion of a user agent string, include the node version of the process sending the event.

This would be similar to the way beeline-nodejs sends the node version as part of its events. See here for its implementation detail: https://github.com/honeycombio/beeline-nodejs/blob/d1895731cd52e1fd848fe62f6bb5457c678aeec5/lib/api/libhoney.js#L221

missing response handling

part of the honeycomb sdk spec deals with how to handle responses from the api host: http://localhost:1313/docs/send-data/sdks/#responses

the JS sdk is different from go/python in that it can't assume threads, so maybe we can:

1. opportunistically steal some cycles from the main event loop to make sure the response queue doesn't grow without bound?
2. return a promise from the send apis and deal with the response locally?

neither sounds particularly nice.

Drop support for Node 8

Node 8 has been EOL since 2019 🤯 We feel it’s finally time to let it go. By dropping support for Node 8, we can upgrade the following dependencies:

• babel-jest: #139
• rollup: #138
• jest: #137

We are planning on dropping support in an upcoming major release.

Typescript types?

First, I want to say I'm really liking Honeycomb so far! I used to use Scuba at FB and Honeycomb has been pretty good at filling that gap in my toolkit since leaving.

I see Typescript type definitions were added in #52 and then removed in #66. Curious, was there an issue with those types?

I'm working on instrumenting all our services with Honeycomb, but our new services are 100% Typescript with strict linting with type checking, making integrating Libhoney (and Beeline) a bit of a hassle. Happy to see Beeline now has a PR to add type definitions but I'd love to see the same in Libhoney too, as I want to log some events independently of what Beeline provides.

web worker transmission implementation

This is perhaps less of a problem now that batch support is in, but ideally the cost of using libhoney from within a webpage would be essentially zero.

If we can push all the batching/sending to a web worker, we should absolutely do that.

Question: TypeScript Types?

Would you accept a PR to add a .d.ts file to support TypeScript consumers?

Update for E&S

Given a libhoney configured with a v2 api key and without a dataset
When it sends HNY events
Then it should use unknown_dataset as the dataset

When given a legacy key, the behavior remains the same as today.

Notes on trimming whitespace

Drop support for node versions that are no longer security supported.

Chatted with @pkanal and we think we should only keep and test for Node 14, 16, 18. The others are no longer security supported.

Testing end-of-life versions of node also blocks dependency updates (see: #293)

Places to change this:

• our CI tests "12.22", "14.17", "15.14", "16.2", "17.0", "18.0"
• our package.json we also bump it to node Version 14
• see what happens ✨

newBuilder on top-level Libhoney doesn't copy props

I could be doing something wrong, but it looks like

libhoney-js/src/libhoney.js

Line 208 in f7ce6c0

var b = new Builder(this, this._fields, this._dyn_fields);

won't work as intended, because it tries to grab this._fields and this._dyn_fields, when it seems like the relevant values are stored on this._builder.

dependencies upgrade of module superagent due to a public vulnerability

Versions

• Node: 20.1.0
• Libhoney: 4.2.0

Description

Dependent module superagent had a public vulnerability with formidable dependency in version 8, and hence have released v9.0.0+ with the fix.
More info is present in the link: ladjs/superagent#1800
Can you please look into it and upgrade dependency for superagent accordingly.
(Please include any relevant CVE advisory links)