ulid/javascript

Universally Unique Lexicographically Sortable Identifier

javascript's Issues

Closed

Monotonically strictly increasing clock source

I'm just wondering, did you find a monotonic strict increasing clock source for the timestamp generation? Otherwise there's a small change at high resolution scenarios where the time will be the same, we could get an equal time and then the ids are not strictly ordered. Compare with: https://github.com/boundary/flake And with reference to: http://learnyousomeerlang.com/time

On a single machine it should be possible with just a counter though, but then you need to keep state of the old counter somewhere. If the OS could provide this counter, it would be much easier.

Maybe UCID a better name?

This guy has a good point:

Sorry, erm.... What makes you say UUIDs are not lexicographically sortable?
They're not chronologically sortable, sure - but I'm confused by your choice of names.

Unique Chronologically sortable Identifier
or simply
Unique Chronological Identifier

Random bias?

From the code:

export function randomChar(prng) {
    let rand = Math.floor(prng() * ENCODING_LEN);
    if (rand === ENCODING_LEN) {
        rand = ENCODING_LEN - 1;
    }
    return ENCODING.charAt(rand);
}

I believe this biases the final character in the ENCODING array, which has two different random values it can be created from.

I believe what you want is to do is select a new random number in this case, since while it is difficult to get this value (a 1 in 256 chance) that is still an increase in bias.

While the increase is small, it feels like using a strong prng only to add unnecessary bias should be addressed.

Not available in web workers

This repo looks for crypto on window before falling back to Node detection and other means of finding a crypto API. This misses the fact that web workers use self as their global reference, so supporting crypto in web workers would be an easy addition.

issue 1

Descending ULID

I would like to create a ULID using a negative timestamp so when sorted in lexicographical order we get a descending sort. However it seems, at least in the nodejs implementation, that an error is thrown if I specify a negative timestamp.

It is JS not in Python. ;-)

Great work. I ported it to Python. I hope the acknowledgement is sufficient.
https://github.com/mdipierro/ulid

Please Upgrade Your Package

Please Upgrade Your Package

First and Last IDs for Timestamp

Is it possible to obtain the first and last lexicography sorted IDs for a timestamp?

If I understand correctly, it seems possible to get the first ID with monotonicFactory().

However, I'm not sure how to go about obtaining the last ID.

Any help would be much appreciated. 👍

Error in worker

in order to detect the browser, you need to take into account that the code can be executed in the worker and there is no window, but there is self

root = typeof window !== "undefined" ? window : null;

should be rewritten as

root = self instanceof Window || self instanceof ServiceWorkerGlobalScope || self instanceof Worker ? self : null;

Publish version 1.0.x

We depend on the changes from #48 - please publish a new release via npm.

Fundamental problem

Ulid is defined as 128 bit binary and base32 is a 5 bit space.

So if you divide 128/5 = 25.6 so if 80 bits / 16 chars is random them time is 10 chars / 50 bits
80 bits + 50 bits = 130 bits !!

So when decoding a 26 chars to 16 octets you will have data-loss
When encoding back from octets to chars you will have missing data

Specs is not clear what to do. Time atm is zero padded so from octets to chars you can assume the 2 extra bits are zero to the left of the 128 and on decoding you have to trim the 2 bits from the left before anything.

License violation

FYI: @Lewiscowles1986 just violated your MIT License by changing it to AGPL:
Lewiscowles1986/php-ulid#17

I find it very disrespectful to the spirit of Open Source and to your work :(

I hope, he'll come to his senses and will change it back to MIT, or you might contact the Free Software Foundation (https://softwarefreedom.org/) and ask them to explain to him the legal violation.

Command line interface

It would be cool if you could add a cli so one can call $ ulid on the command line.

Tests confusing

What would be the use of these tests?

it('should change length properly', function() {
  assert.strictEqual('0001ARYZ6S41', ulid.encodeTime(time, 12))
})

it('should truncate time if not enough length', function() {
  assert.strictEqual('ARYZ6S41', ulid.encodeTime(time, 8))
})

The specs clearly read:

Specification

...

...

 Timestamp          Randomness
  10 chars           16 chars
   48bits             80bits

...

Components

Timestamp

• 48 bit integer

...

Randomness

• 80 bits

...

So there would (should?) never be a need for any other length than 10? I see others implement the same test:

• fvilers/ulid.net
• imdario/go-ulid
• Lewiscowles1986/ulid
• Lewiscowles1986/jULID

Also the len argument to encodeRandom() and encodeTime() pop up in all other implementations; again: since these are fixed per the spec they don't make sense since they will/should never be called with any other argument.

Don't get me wrong, I understand completely that the (original?) intent probably is/was "you may want to 'vary' with, for example 12 chars timestamp and 14 chars randomness" but then it wouldn't be a ulid anymore, would it? Then it (and a ulid) would simply be a base32 encoded representation of 128 bits of data.

I propose the referred tests are dropped, the arguments ditto and the lengths either hardcoded in their respective methods, or, for readability be put in some sort of const (like const TIMEPARTLENGTH = 10, RANDOMPARTLENGTH = 16).

If a ulid is to become as widespread as UUID/GUID's then, IMHO, it would be better to start off with people not 'tweaking' their ulid's to their need causing a 'proliferation' of all sorts of 'sorta compatible' ulids.

But that's just my $0.02

issue2

Improper use of crypto.getRandomValues

Officially crypto.getRandomValues doesn't return a value. Some implementations do return a value, but older ones (like on older android browsers do not) and it causes this library to fail.

The following line expects getRandomValues to return the array back, but it does not.
https://github.com/alizain/ulid/blob/6915ff233a4f2274f6726b8fc2ad71eca87de292/lib/index.js#L105

This should be replaced by:

return () => {
  const buffer = new Uint8Array(1);
  browserCrypto.getRandomValues(buffer);
  return buffer[0] / 0xff;
};

error when bundle for producation

ulid version

v2.2.1

webpack version

v3.8.1

library or framework

React v15.6.1

error message

ERROR in static/js/vendor.d444dbbb14fee2232a46.js from UglifyJs
Unexpected token: name (done) [./node_modules/ulid/lib/index.js:20,0][static/js/vendor.d444dbbb14fee2232a46.js:92761,8]

Rust language version

Made a rust-lang.org version

https://github.com/mmacedoeu/rulid.rs

Is there a way to get the timestamp from a given ULID?

Use case:

If I store the ULID in the DB and I can retrieve timestamp information from the ULID, I won't need to store a separate timestamp column anymore.

Support for RFC 4648 base32?

It would be convenient for me personally if this library supported a variant of ulid that was encoded in RFC 4648 base32 instead of Crockford's base32. (Obviously, this would violate the ulid spec.)

Decoding ULID string into bytes

Hi,

I've been trying to decode ULID into bytes for more efficient storage. I didn't notice there is a way to retrieve the raw bytes from the library itself so I tried using Crockford's base32 encoding to do it manually, as the docs say it uses this specific encoding. However when using base32-encode and base32-decode sometimes there seems to be precision errors, last encoded character won't match the original ULID. Docs do mention that there is 2 bit difference between Crockford's encoding and ULID's length (130 bit vs 128), which I suspect may lead to encoding error. My question is how to do it properly?

Example:

import { ulid } from 'ulid';
import b32decode from 'base32-decode';
import b32encode from 'base32-encode';

const uid = ulid();
console.log('UID', uid);

const decoded = b32decode(uid, 'Crockford');
console.log('DECODED', decoded);

const encoded = b32encode(decoded, 'Crockford');
console.log('ENCODED', encoded);

Output:
UID 01FZD39998855SS2YG4XP4T14P
DECODED ArrayBuffer {
[Uint8Contents]: <00 5f f6 8d 29 4a 10 52 e7 22 f4 09 db 13 41 25>,
byteLength: 16
}
ENCODED 01FZD39998855SS2YG4XP4T14M

Generate time from shortened ulid?

I use shortened ulids where I truncate the randomness part of the ulid (last 16 characters) and use the timestamp portion (first ten characters) as keys in a database. The reason being that the timestamp alone offers sufficient resolution, with the randomness increasing the storage size with no benefit. Now, if I want to recover the time from the timestamp portion, decodeTime() requires a full length (26 char) ulid, though, strictly speaking, only the first 10 chars encode time information. Can decodeTime() be modified to allow both a 26 char and a 10 char ulid, or is there something else I am missing? I know I can generate my own random portion, concatenate it with the truncated ulid, and then call decodeTime(), but that seems circuitous.

Monotonically Increasing Guarantee

This is related to #21 and ulid/spec#11

If multiple UUIDs are generated in the same millisecond, the random bits are not guaranteed to be monotonically increasing. So given const [a, b, c] = [ulid(), ulid(), ulid()]
c and b could sort lower than a

This implementation already has an algo for incrementing the base32 of the ulid. It should just change to increment the most significant bits instead of the least significant bits of the random portion.

Delphi/Object Pascal implementation

because dinosaurs still live 😬

https://github.com/martinusso/ulid

dist _index.js#L59

javascript/dist/index.js

Line 59 in a583120

if (Number.isInteger(now) === false) {

to

if (Number.isInteger(Number(now)) === false) { 

better for some situation like ulid("150000")?

Binary implementation in readme

Hey everyone,

I've added a column in the readme for binary implementations in your libraries. If you have implemented it, please let me know here or submit a PR so I can add the ✓!

@SuperPaintman @savonarola @merongivian @imdario @Lewiscowles1986 @ararslan @RobThree @fvilers @mdipierro @rafaelsales

isUlid

A solid isUlid implementation?

Crypto API isn't detected on alternative platforms

I run across this using Cloudflare Workers. It should be possible to use this library by instead checking for a crypto global instead of trying to detect `window. Documentation from Cloudflare here: https://developers.cloudflare.com/workers/reference/apis/web-crypto/.

Add config to output lowercase

I'm not a fan of all upper case. Would be cool to be able to change the default to all lower case.

Not sure I like the new spec's "increment within same ms" change + more thoughts

You changed the spec to include:

To ensure sortability for IDs generated within the same millisecond, implementations should remember the most recently generated randomness string and increment the most significant character that is not the last character in the encoding.

I don't think you understand the impact of this change; it brings a whole lot of hurt into your world. For instance, you can't rely on wall-clock-time (which you fell for). Unless you can get a reliable, monotonic, strictly increasing timesource (which is (near?) impossible AFAIK in JS) you'll run into trouble later. Why? Because: wall-clock-time goes back (DST) / stands still (leap seconds are sometimes implemented as such) / does other weird stuff (and there's even more). Trust me, I know 😉 Sidenote: some systems may not even have 'ms-resolution' timers, something to consider too.

There's 80 bits of random data; I think it's safe to assume (even with the birthday paradox in mind) that the chance on a collision (within any, random, millisecond) is similar to the chance of a GUID collision. However, that would be if the ULID's would be spread evenly across time (which would, in effect, make a ULID similar or even equal to a GUID). So, given that ULID's are typically 'clustered' in a certain timespan the chance on a collision goes up. Worst-case would be a DST change + some other 'random event' that would cause the same 'millisecond'-point-in-time to exist (say) 3 or 4 times. The chance on a collision would increase a lot but (I haven't done the math yet) would still suffice for most purposes unless you're Google maybe and generating many millions or even billions of ULID's p/ms.

So I guess my advice is to leave that part out of the spec (and, let's be honest, this method is no beauty either (it could be improved but wouldn't be worth the trouble IMHO)) and keep implementations much simpler that way. You could compare a ULID it with a v4 UUID with the only difference that the first 6 bytes (48 bits) are reserved for a timestamp for better ordering. However, it does mean that, within the millisecond, the ordering is 'random' so that the spec may need to be more clear about it / reflect that fact.

Besides time, by the way, this tiny change in the spec also requires you to keep state between generating ULID's (so you'll need to keep the generator-object around for example), introduces chances for race conditions (multithreading; N threads generating ULID's will, without proper locking etc. inevitably generate exactly the same ULID's) and all other sorts of things that either need to be taken for granted (which will bite you in the *ss sooner than later) or need to be worked out / specified.

For inspiration about some more of the problems you might run into you might want to look at Twitter's Snowflake (retired but still available) or have a look at my "Snowflake-alike" IdGen. Both are 64 bit (actually 63*) equivalents of ulid.

* Come to think of it: You might also want to steer clear of the sign-bit; when systems would order on the (internal) 128 bit representation / byte-array (for 'speed') the order may 'break' as soon as the sign bit is set to 1. Twitter thought of it way before it happened but I think it's a good idea. Sacrificing 1 bit out of 128 should't be that much of a problem (though it does halve your "ulid-space" from 2^128 to 2^127).

I was/am doing a ulid .Net port (waaay more extensive than fvilvers', with all due respect!) and also implemented the binary form, created methods to convert to/from UUID's (since ULID's and UUID's are both 128 bit you can easily convert one to the other and vice versa). I also implemented a Parse(...) method and some others and will publish on GitHub very soon have put it on GitHub. There are some things I thought of / came across which might need 'agreement' and be put in the spec:

• ulid's are, IMHO, case-insensitive (e.g. NOT case-sensitive) We seem to agree on that; missed it in your README 😛
• We may want to allow / think about / specify a(n optional?) separator for readability. No { and } mess like (G/U)UID's but something like ttttt-ttttt-rrrrrr-rrrrr-rrrrr (lengths 5-5-6-5-5) might make ulid's more readable. Other formats/separators are ofcourse a possibility (as well as none of this ofcourse; I'm just bringing my ideas to the table)
• You/we may want to decide on pronunciation and document it: "you-lid"? "you-el-i-dee"? "uhl-id"? Something else? You/we want to avoid the GIF's "Jif" v.s. "Gif" wars
• Besides the pronunciation, same goes for a "standardized" writing. I catch myself writing ulid, ULID, UlId, ... so that may have to be standardized too
• I had more... but can't think of it now. Will add here / open new issue when I think of it.

So far, for now, another $0.02 of mine 😝

---

Edit: FYI; I just released V1.0 of NUlid. 🎉 Maybe you can add it to your list of community contributed ports?

PRNG inconsistency

Disclaimer: I have not conducted any research into how the following affects information security or may lead to errors.

These pieces of code generate random numbers in the range from 0 to 1 inclusive, because you divide the value received from the browser or node by 255, not 256 (because 1 byte takes 256 different values, not 255):
return buffer[0] / 0xff
and
nodeCrypto.randomBytes(1).readUInt8() / 0xff

While Math.random() used in case of absence of a good prng always returns a value less than 1.
Moreover, your functions with good PRNG return only 256 different values, thus being much inferior even to Math.random().

is_ulid(x)

Hello, is there a validation / checking method provided with the library? Thank you.

Where is ulid.js

In the documentation says:

Browser

<script src="/path/to/ulid.js"></script> <script> ULID.ulid() </script>

where is this ulid.js file?
How can I generate this file using the project files?

String form has 2-bits of redundancy

We discovered by accident in my team that the string version (26 characters of Crockford's base32) encodes 130 bits (26 * 5) rather than 128 bits. These extra two bits are correctly stripped by the implementations we've played with (Python, Julia, Golang).

See for example: oklog/ulid#9

This means that there are four string encodings mapping to the same binary ULID, for every ULID.

This is not something you encounter generating them normally, since you will always generate them with a timestamp of right now. But in our random fuzz testing of an internal API, we encountered them.

As an implementation side note, I would recommend rejecting string ULIDs with a prefix above 7 (a ULID beginning with 8 has the same binary as one beginning with 0) with an error rather than parsing them.

Use Typescript module support

Since you are switching to Typescript, you can remove the manual code for the "UMD" wrapper and instead use ES2015 exports and let Typescript build your module wrappers for you with compile options in the tsconfig.json.

Also, you can remove index.d.ts if you already have an index.ts as Typescript will refer to that for its own definitions instead. (In fact the error I'm seeing with recent npm version is that the current index.ts is not a module and cannot be imported (because of the manual module wrapper confusing Typescript's type senses.)

I'd be happy to help with a PR if you would like, but this is the sort of thing might be better to do as learning experience, so I'll leave that up to you.

What is the most accessible way to get ulid in Docker image?

Installing the entire Node.js as a dependency is ... inefficient to say the least.

None of the Python libraries seem to provide CLI interface.

Is there a recommended program?

I need ulid ID as part of a build process.

Maintainers

It's quite clear that this repository is lacking maintainers to help review and merge PRs.

I can't offer a day-to-day availability to do so, but I would definitely have time each week to dedicate to helping keep this library alive and help merge the occasional bugfix etc.. I use it very regularly in many projects, both personal and work. I'd love to become a maintainer of ulid/javascript, and I'm sure there are others that would readily do the same.

@alizain would you consider adding someone as a maintainer? I'd be happy to discuss the responsibility further if you'd accept help for this project.

go test -bench=. is failing

🚀: (master)>go test -bench=.
--- FAIL: TestTimestamp (0.00s)
ulid_test.go:287: got timestamp 4773815605011, want 281474976710655
--- FAIL: TestParseRobustness (0.00s)
ulid_test.go:245: runtime error: index out of range
ulid_test.go:260: #1: failed on input [26]uint8{0x1, 0xc0, 0x73, 0x62, 0x4a, 0xaf, 0x39, 0x78, 0x51, 0x4e, 0xf8, 0x44, 0x3b, 0xb2, 0xa8, 0x59, 0xc7, 0x5f, 0xc3, 0xcc, 0x6a, 0xf2, 0x6d, 0x5a, 0xaa, 0x20}
FAIL
exit status 1
FAIL _/Users/hsawhney/code/ulid 4.985s
🚀: (master)>

Typescript definition file

Would you be interested in adding a Typescript definition file to your npm package? I'd be happy to contribute one via PR and help maintain it. There are other places to host such files, but it's always nicest when they exist right along the npm install.

(I ask first as an issue instead of sending a PR first as a courtesy, because I know not everyone wants to maintain Typescript definitions and "no" is a perfectly fine answer.)