t-pwk/flake-idgen

Flake ID generator yields k-ordered, conflict-free ids in a distributed environment in Node.js

flake-idgen's Issues

Why duplicate IDs are generated

function genFlakeId() {
    const flakeIdGen = new FlakeId();
    return intformat(flakeIdGen.next(), 'dec');
  }

const ids = [];
for (let i = 0; i < 50; i++) {
  ids.push(genFlakeId());
}
// ids
[
  "6901552045816283136",
  "6901552045820477440",
  "6901552045820477440",
  "6901552045833060352",
  "6901552045837254656",
  "6901552045841448960",
  "6901552045845643264",
  "6901552045849837568",
  "6901552045854031872",
  "6901552045858226176",
  "6901552045862420480",
  "6901552045875003392",
  "6901552045879197696",
  "6901552045879197696",
  "6901552045879197696",
  "6901552045883392000",
  "6901552045883392000",
  "6901552045883392000",
  "6901552045883392000",
  "6901552045883392000",
  "6901552045887586304",
  "6901552045887586304",
  "6901552045887586304",
  "6901552045887586304",
  "6901552045887586304",
  "6901552045891780608",
  "6901552045891780608",
  "6901552045891780608",
  "6901552045891780608",
  "6901552045891780608",
  "6901552045895974912",
  "6901552045895974912",
  "6901552045895974912",
  "6901552045895974912",
  "6901552045900169216",
  "6901552045900169216",
  "6901552045900169216",
  "6901552045900169216",
  "6901552045900169216",
  "6901552045904363520",
  "6901552045904363520",
  "6901552045904363520",
  "6901552045908557824",
  "6901552045908557824",
  "6901552045908557824",
  "6901552045912752128",
  "6901552045912752128",
  "6901552045912752128",
  "6901552045916946432",
  "6901552045916946432"
]

How to use with sequelize？When bulk insert, the generated id is always repeated

I use this, is not the method I use is wrong?

var intformat = require('biguint-format');
var FlakeId = require('flake-idgen');
var flakeIdGen = new FlakeId({epoch: 1300000000000});
var moment = require('moment')
module.exports = function(sequelize, DataTypes) {
    return sequelize.define('employment_detail', {
      id: {
        type: DataTypes.STRING(45),
        allowNull: false,
        primaryKey: true,
        // defaultValue: shortid.generate
        defaultValue: intformat(flakeIdGen.next(), 'dec')
      }
   }
}

Observing Duplicate Ids during load tests

I am using flake-idgen in a distributed environment and am seeing duplicate Id errors on a the primary Id field under the circumstances below during K6 load testing (50 Vus):

2+ replicates of Id service
Datacenter = 0
Worker = 1,2,3,n

The issue does not exist when I run a single instance of the Id generation service (datacenter = 0, worker = 0). Based on observation, the Duplicate collisions are being produces by different instances of the service initialized with a different worker Id.

I'll attempt incrementing both datacenter and worker and leave a comment on the results.

42 bit timestamp?

@T-PWK From what I understand, the ms timestamp is 41 bits but in the readme 42 bits are reserved for the timestamp, what is the extra bit?

ReferenceError: Buffer is not defined

I tried to use this package in my Sveltekit project and got Buffer is not defined.

Here is my code:

  import FlakeId from "flake-idgen";
  import intformat from "biguint-format";

  const flakeIdGen = new FlakeId();
  let id = intformat(flakeIdGen.next(), "dec");

Here is my error log:

ReferenceError: Buffer is not defined
    at module.exports.next (flake-id-gen.js:58:13)
    at instance (MainComponent.svelte:11:49)
    at init (index.mjs:2165:11)
    at new MainComponent (MainComponent.svelte:24:23)
    at createProxiedComponent (svelte-hooks.js?v=8eec1919:341:9)
    at new ProxyComponent (proxy.js?v=8eec1919:242:7)
    at new Proxy<MainComponent> (proxy.js?v=8eec1919:349:11)
    at create_fragment (+page.svelte:12:97)
    at init (index.mjs:2180:37)
    at new Page (+page.svelte:141:20)

Not sure if this package will work in a Jamstack project.

Version 1.4.0 crash in Webpack 4 step "UglifyJS"

Hi,

Your package works fine until 1.3.0 but version 1.4.0 introduce some changes that makes build impossible with Webpack on UglifyJS step.

Error is:

ERROR in assets/bundle.js from UglifyJs
Unexpected character '`' [assets/bundle.js:203079,32]

Extract from devDependencies in our file package.json:

    "babel-core": "6.26.3",
    "babel-loader": "7.1.5",
    "babel-polyfill": "6.26.0",
    "babel-preset-env": "1.7.0",
    "babel-preset-react": "6.24.1",
    "babel-preset-stage-0": "6.24.1",
    "babel-register": "6.26.0",
    "copy-webpack-plugin": "5.1.1",
    "css-loader": "3.5.2",
    "generate-json-webpack-plugin": "0.3.1",
    "html-webpack-plugin": "4.2.0",
    "imagemin-webpack-plugin": "2.4.2",
    "style-loader": "1.1.4",
    "uglifyjs-webpack-plugin": "2.2.0",
    "url-loader": "4.1.0",
    "webpack": "4.42.1",
    "webpack-bundle-analyzer": "3.7.0",
    "webpack-cli": "3.3.11",
    "webpack-dev-server": "3.10.3",
    "webpack-merge": "4.2.2"

How to use with mongodb?

I generate id with flake-idgen and biguint-format but when try to save this id to mongodb get next error:

Error: Argument passed in must be a single String of 12 bytes or a string of 24 hex characters at new ObjectID

Is it real fix?

Min and Max values ?

Hello,
I would like to know if the generated id converted in decimal stay inside the range of the ?
Number.MAX_SAFE_INTEGER ?

Thank you and have a nice day.

Duplicate ids generated in torture test?

First, thanks for creating this. It's really cool. I'm considering using it as a cryptographic nonce generator (not the intended use case I know, see SO question here), so I want to make sure it doesn't generate duplicate IDs.

I put together a simple test for it and ran it on my computer. I killed the process half way through to simulate an unexpected server shutdown and then resumed it again. After generating about 200,000,000 total ids in 4 processes running in parallel, I checked for duplicates and found a little under 2,000 (each with exactly 2 copies).

Is there something wrong with my testing methodology? Is this behavior expected?

Easiest way to get datacenter/worker id from generated id

Maybe it's a good idea to add a get function for retrieving the datacenter or worker id from a generated id?