microwavekonijn/ps2census

The PS2 Census Client is a tool to simplify the interaction with the Planetside 2 Census API.

PS2 Census Library

About

The PS2 Census library is here to simplify the interaction with the Planetside 2 Census API for Javascript and Typescript projects.

Requirements

Event Stream

• Node.js v14+ or any modern browser with WebSocket support;
• DBG Census API Service ID.

Rest API

• ES6;
• DBG Census API Service ID(optional, but recommended).

Installation

npm install ps2census

For better performance it is recommended to install bufferutil and utf-8-validate, though this is optional. The --save-optional flag can be used to install them as optionalDependencies.

Getting started

Event Stream

const { CensusClient } = require('ps2census');

const subscriptions = {
  worlds: ['10'],
  eventNames: ['MetagameEvent'],
};

const client = new CensusClient('ServiceID', 'ps2', {
  streamManager: {
    subscription: subscriptions,
  },
});

client.on('ps2Event', event => {
  // Handle the event, for more information see http://census.daybreakgames.com/#websocket-details
});
// or
client.on('facilityControl', event => {
}); // Note that the event always starts with a lower case letter

client.on('subscribed', subscription => {
}); // Notification of a subscription made by the event stream
client.on('duplicate', event => {
}); // When a duplicate event has been received
client.on('ready', () => {
}); // Client is ready
client.on('reconnecting', () => {
}); // Client is reconnecting
client.on('disconnected', () => {
}); // Client got disconnected
client.on('error', error => {
}); // Error
client.on('warn', error => {
}); // Error, when receiving a corrupt message

client.watch();

// To terminate the client
client.destroy();

Rest API

const { Rest } = require('ps2census');

const client = new Rest.Client('ps2ps4eu', { serviceId: 'example' });

// Get character with items
client
  .getQueryBuilder('character')
  .resolve('item')
  .exactMatchFirst(true)
  .get({ 'name.first_lower': '^microwave' })
  .then(characters => {
    // Process the data
  });

Nanite Systems ESS

With the Census ESS being having some issues the NS ESS has made its entrance. Basic usage with Typescript:

new CensusClient('serviceid', 'ps2', {
  streamManager: {
    endpoint: 'wss://push.nanite-systems.net/streaming',
  },
});
// or
new CensusClient('serviceid', 'all' as any, {
  streamManager: {
    endpoint: 'wss://push.nanite-systems.net/streaming',
  },
});

Documentation

• Stream Client;
• Testing Stream.

Contribution

Please make sure not to create duplicate issues. When making a PR make sure to check if eslint doesn't report any problems.

Disclaimer

The PS2 Census Library comes with no warranties. This software is as is and usage is at the developer/users own discretion.

ps2census's Issues

Comparing multiple websocket connections through different proxies

Some people have suggested running multiple websockets and comparing the events between them. The most reliable way (if not the only one) seems to be running them through different proxies. In order to not completely reinvent the wheel I thought perhaps I could add this as a feature to ps2 census instead. I won't promise to write such a feature, but I'll consider it if you wouldn't be opposed to merging a PR like that. So is there any chance that would be worth a consideration?

Add zone_id to MetagameEvent class

The MetagameEvent class is missing the zone_id.

See issue 47 comment for context.

Add autodocumentation using comments

Auto document ps2census using TypeDoc or TSDoc. Before this can happen some functions and classes need to be documented using comments.

MetagameEvent misses instance_id

MetagameEvent misses instance_id in its declaration.

Integrate Census Rest API into the Client

Add helper function to the objects of the event stream so they can easily enrich the data received.

Add option to ignore expired certs

Yes, this is what it has come to sad noises

Move Websocket connector to own class

Client will be a more general class that manages the different aspects of the API. The Client will still emit all the events.

Split Census Rest API types

Basically all types should be curated so that there are less duplicates as for instance the character type is used at multiple places. Split them so that for each endpoint an base type can be composed and a map can be created(indexed type) for the resolvable types.

Readd type guards for commands

A few collections only allow a few commands(all support tree and join):

• charactersEvent: limit;
• charactersEventGrouped: limit;
• charactersFriend: only base commands;
• charactersItem: only base commands;
• charactersLeaderboard: start, and `limit;
• charactersOnlineStatus: only base commands;
• charactersWorld: only base commands;
• event: limit;
• leaderboard: start, and limit;
• map: only base commands;
• worldEvent: limit.

Event stream client - no output.

Hello,

I'm trying to use ps2census to output deaths from the event stream. However, I'm not getting any output.

My code:

import { Client, Events } from "ps2census";


const subscriptions = [{
  worlds: ['10'],
  eventNames: ['Death']
}];

const client = new Client('serviceID', {
  streamManagerConfig: {
      subscriptions
  },
});

client.on('subscribed', (subscription) => {`Sub event: ${subscription}`}); // Notification of a subscription made by the event stream
client.on('duplicate', (event) => {console.log(`Dupe event: ${event}`)}); // When a duplicate event has been received
client.on('reconnecting', () => {console.log('reconnecting')}); // Client is reconnecting
client.on('disconnected', () => {console.log('disconnected')}); // Client got disconnected
client.on('error', (error) => {console.log(error)}); // Error
client.on('warn', (error) => {console.log(error)}); // Error, when receiving a corrupt message

client.on(Events.PS2_DEATH, async (e) => {
  console.log(e);
});

client.on('ready', () => {console.log('ready')}); // Client is ready
client.watch();

After the 'ready' output, nothing else happens.

Node version: v16.4.0
ps2census version: 2.0.3

I'm new to nodejs so please let me know if I have done something incorrectly :-)

Any help is much appreciated.

Bug - TypeError: Client is not a constructor

Hi,

I just ran into an issue with the new version where the following error occurs: TypeError: Client is not a constructor.

Looking at census.client.ts the export class is in fact CensusClient.

I fixed this issue by doing the following:

import census from 'ps2census';
const { CensusClient, Events } = census;

const subscriptions = [{
    // Connery, Miller, Cobalt, Emerald, SolTech
    worlds: ['1', '10', '13', '17', '40'],
    characters: ['all'],
    eventNames: ['MetagameEvent'],
    logicalAndCharactersWithWorlds: true,
}];

// Initalise ps2census event stream client
const client = new CensusClient(serviceID, 'ps2', {
    streamManager: {subscriptions}
});

Fix export entry point

Exporting under export default doesn't behave as expected. Also interfaces/types need to be exported.

Ratelimit and combine manager request to Census

Right now every request is triggered immediately and directly. It might be a good idea to add a ratelimit queue with retry functionality to the managers that are also able to combine request(e.g. https://census.daybreakgames.com/get/ps2:v2/character?character_id=5428569415095835281,5428010618035323201)

Wrap results returned from managers

ASP is not included on PS4(prestige_level on characters), so a wrapper with computed values for results returned from any of the rest managers can resolve this problem.

ps2alerts/aggregator#299

Subscription manager

The client on start up uses an array of subscription objects to subscribe to events on a connection to the Event Stream. It would be better to have a manager for this that is able to more dynamically add and remove subscriptions.

Some investigations is required with regards to how they work. E.g. when you subscribe to the 'PlayerLogin' event for all characters and then unsubscribe from that event for character A, what consequences will that have. Similarly what if you take that event and do the same thing and unsubscribe world A?

Additionally it might be useful to add a health check as I encountered an issue where I subscribed to the MetagameEvent for all worlds and after a while I only received this event for Soltech.

Fix export of certain usefull types related to Rest

Certain types related to collections are not exported, but might be useful. Need to be exported.

Wrapper for Rest in Client

Queries run from the CensusClient already know their environment. Needs to be determined how to solve this.

Split zone ids

Zone ids can be separated into a instance id(first 16bits) and a definition id(last 16bits). Add functions to the client events to compute these values.

CharacterManager caches invalid responses

Currently the CharacterManager is caching invalid character responses.

Scenario:

Brand new player is requested from Census, world ID is missing (BR 0). CharacterManager considers this a valid response (it currently isn't able to detect if a response is invalid in that manner) and continues to serve an invalid response.

Underscored Census endpoints such as map_region present an index error

Example query:

       await get(
            rest.mapRegion,
            {
                world_id: String(instance.world),
                zone_ids: String(instance.zone),
            },
       });

TS error:

TS2345: Argument of type 'baseRequest<"map_region">' is not assignable to parameter of type 'baseRequest<"singleCharacterById" | "character" | "characterName" | "charactersAchievement" | "charactersCurrency" | "charactersDirective" | "charactersDirectiveObjective" | "charactersDirectiveTier" | ... 102 more ... | "worldEvent">'.   Type '"map_region"' is not assignable to type '"singleCharacterById" | "character" | "characterName" | "charactersAchievement" | "charactersCurrency" | "charactersDirective" | "charactersDirectiveObjective" | "charactersDirectiveTier" | ... 102 more ... | "worldEvent"'.

Faking events documentation changes

The faking events documentation is outdated.

The example uses the old Client constructor and the fake event is missing the zone_id.

Fix test.ts due to websocket cert expiry

The test.ts file needs fixing for the websocket certificate expiring.

Add documentation

• Add a docs folder for any documentation;
• Generate wiki based of md files in docs;
• Documentation with describes the current collections/types of the Rest API;
• Documentation on usage of the Client and EventStream;
• Documentation on usage of the Rest API methods.

Routing of events

Events like 'PlayerLogin', 'BattleRankUp', etc are all emitted through the 'event' event(I know, it needs a better name). Depending on the configuration of the client these events are emitted under their own event name.

logicalAndCharactersWithWorlds not working with login and logouts events

Probably want to resolve some of the oddities that come with subscribing to the event stream by allowing the client to split subscriptions into multiple that help with resolving some of the issues.

Resubscribe on an interval

Hi,

I'm trying to use client.resubscribe() to rerun all subscriptions. However, I keep receiving the error:

TypeError: Cannot read properties of undefined (reading 'resubscribe')
    at Timeout._onTimeout (file:///usr/src/app/publisher.mjs:85:18)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

Here is my code:

...
const client = new CensusClient(serviceID, 'ps2', {
    streamManager: {
        subscription: subscriptions
    },
});
...

// Configure interval to rerun all subscriptions to the websocket API
setInterval(async function (client) {
    console.log('Rerun all subscriptions.');
    await client.resubscribe(true);
}, 120000);

Can you shed some light as to why this might be happening?

Thanks :)

Make query conditions optional

Query conditions are always required, which are optional. Current solutions is an empty object, but should allow undefined.

Reimplement Tree and Join formats

Tree and Join commands alter the response format. With the REST refactor it is possible to adjust the format when these commands are applied. The Tree command has the priority and should be the simplest to implement. For Joins there are workarounds.

Create Event Classes rather than GenericEvent type

It would be extremely useful if you're able to send back formatted objects of each event type, e.g.:

class DeathEvent implmentents GenericEventInterface {
    public constructor(event: GenericEvent) {}

    public isSuicide() => {
        // Code to figure out if an death is actually a suicide
    }
}

This way we can under the hood use methods on the class itself to figure out particular data / statistical points without having the need to duplicate code.

Update documentation

Documentation is out of date. Will be updated after rest refactor.

Improvements utility types Census Rest API

The rest functions rely heavily on the types. Some improvements might be needed.

• Automatic type intersection when using resolve to generate new type;
• Allow timing to be executed anywhere in the chain;
• Better type guard of operations(now it is based on the return type);
• Improvement to types returned from tree, distinct, and (maybe) join command;
• Fix so that timings are not ignored when included.

Add ratelimiter to retry mechanism rest

Ratelimit retry attempts to decrease error rates of the api.

Refactor rest collections

The initial goal was to have self contained collection types which would contain information about the name, format, resolvables, etc. This was scrapped as there were some issues with the format. However after some experimentation there is a way which makes the collections better maintainable.

interface CensusRequest<C extends Collection<any, any>> {
    collection: C extends { name: infer N } ? N : never;
    params: Record<string, string | number | boolean>;
}

interface CensusTreeRequest<C extends Collection<any, any>> extends CensusRequest<C> {
    tree: string[];
}

interface Collection<C, F, R = never> {
    collection: C;
    format: F;
    resolve: R;
}

interface CharacterCollection extends Collection<'character',
    {
        character_id: string,
        name: {
            first: string,
            first_lower: string,
        }
    }> {
}

function factory<C extends Collection<any, any>>(x: C['collection']): CensusRequest<C> {
    return {
        collection: x,
        params: {},
    };
}

function tree<C extends Collection<any, any>>(x: CensusRequest<C>): CensusTreeRequest<C> {
    return {
        ...x,
        tree: [],
    };
}

function retry<R extends CensusRequest<any>>(x: R): R {
    return {
        ...x,
    };
}

declare function execute<R extends CensusRequest<any>>(x: R):
    R extends CensusTreeRequest<infer C>
        ? Promise<C['format']>
        : R extends CensusRequest<infer C>
        ? Promise<C['format']>
        : never;

const y = factory<CharacterCollection>('character');

const x = tree(y);

const a = retry(x);

const b = execute(a).then((c) => {
    console.log(c.character_id, c.name.first);
});

It does not, and should not, change anything from a functional standpoint.

GenericEvent cannot be ContinentUnlock

Conversion of type 'GenericEvent' to type 'ContinentUnlock' may be a mistake because neither type sufficiently overlaps with the other.

[Thread] Census Rest API

This thread is for reporting issues with the rest module of this library. When reporting type related bugs please be clear and try to give an example. When reporting issues with the code, please supply an example of code that should work but doesn't.

Rebalance default DuplicateFilter

The hashes generated internally require some attention to reduce the amount of false positives.

PS2 Census REST API integration

Integration of the Census REST API for PS2, that can be used to retrieve additional data based on a event from the Event Stream as well as being able to cache results(externally or internally). Maybe with an auto hydrate feature for the Event Stream.

• Add endpoints;
• Integrate endpoints into the Client.

Improve debug events with labels

Beside a string a debug event should contain information of the source of the debug event. E.g.:

client.on('debug', (info, label) => {});

Event Stream example still incorrect

The Event Stream example in the README.md is still incorrect. I was having similar issues to #42.

I will submit a PR to fix this 👍🏻

PS2Event should always contain a world_id

The PS2Event should always contain a world_id.
Every event contains a world_id, so it should be moved to the abstract class

Fix / Re-implement Retry Mechanism

As per ps2alerts/aggregator#174 we've had to implement a retry system. This shoud live within the Census package ideally with configurable number of retries (suggest default be 3)

Add team_id to event objects

Support for team_id is going to be added. Adjustment on the event stream object to reflect this as typescript will deny its existance while it is clearly there(silly typescript).

https://discord.com/channels/251073753759481856/451032574538547201/870391423214682143

Preparation release v1.0.0

• Cleaner export system for exposing items by removing default exports;
• Implementing #31;
• Implement class-transformer on incoming events;
• Small bug fixes:
  • Default cache implementation;
  • Improved ClientOptions.

Add forget method on CharacterManager

CharacterManager needs to be able to forget cached entries. Addition of forget and forgetAll methods should resolve this, also giving voice to the methods of the Cache.

Enrich event data

• zone_id for MetagameEvent can be computed;
• is_suicide for Death can be computed;
• is_teamkill for Death can be computed;
• Timestamps to Date;
• Parse booleans;
• kill_type for Death to determine if suicide or teamkill.

Any implementation for zone_id on MetagameEvent would be to unreliable. Therefor it can't be a statically computed value, can be implemented later through a manager. is_suicide and is_teamkill merged to kill_type in Death.

Testing

Some tests that can be used to verify between iterations if everything is working as expected.

Sub class of PS2Events for character events

Introduce a new sub class of PS2Events that will be a super class for the event classes that have character_id.

DNS problems fail to resolve push.planetside2.com

NOTE: Might not be a DNS issue but a network routing issue between DO and DBG.

The Census stream seem to have some DNS troubles. It can happen that your service will fail to connect to the stream. To resolve this add a entry to your hosts file that link push.planetside2.com to 64.37.171.38.~~

If you are using Docker you can use--add-host to achieve this. https://docs.docker.com/engine/reference/commandline/run/

Add isomorphic-ws

The dependency isomorphic-ws will allow the client to be integrated into a browser client as opposed to only a NodeJS environment. Only minor changes required.

Implement response object

Right now information from the debug command is ignored when used. A rest query should return a response object of the type array that can hold any additional information from the request.