ekino/rendr

A SSR toolkit for Headless CMS and more.

SSR with dynamic page structure

There are many challenges when doing Server Side Rendering with React (or any other similar stacks), it is not only about setting up a state management, or select the best CSS framework. The first challenge is on the logical architecture level, where do we put business logic, how can we do data aggregation, how can we do http caching with user's data, etc ... The second challenge, is how can we add flexibility to the end user to administrate the page layouts or templates (and not only contents).

This project try to resolve these two challenges.

Getting started

Once you are familliar with the architecture described in this document, use following guide to create your first Rendr API. Then we will create a simple Rendr Engine following this second guide

Architecture

When do you need this kind of solution?

• You have internal data sources (microservices or legacy system)
• You have a CMS but don't want to add custom codes or don't want to expose it to the world
• Your CMS cannot do the aggregation from different data sources
• You need to add some cache mechanisms to protect your internals systems

Understanding the different layers

• Data: Data sources, can be anything as long as the aggregation layer can access to those data. This data layer can be protected or cached by some middlewares. This is up to the implementation to do that work.
• Aggregation: The aggregation knows how to fetch the data and how to compose a page definition with all the required contents. The page is a set of information: page title, blocks with container name, cache information, etc ... The page structure can either be hard coded in the project or coming from an external CMS with a proper page builder or similar headless solutions.
• Rendering engine: This layer knows how to communicate with the aggregation layer, and render the final output to the client. The output can be anything, most of the case this will be html. But the rendering engine can be a PDF engine, a mobile application, mainly anything that can display information and interact with the user.

Note: The rendering engine should not contain any business rules, but only visual rules. All business rules have to be in the aggregation layer.

On a nutshell, the architecture is something like this:

    +-------------------------------+
    |        Rendering Engine       |
    +-------------------------------+
    +-------------------------------+
    |          Aggregation          |
    +-------------------------------+
    +--------+ +--------+ +---------+
    | Data 1 | | Data 2 | |   CMS   |
    +--------+ +--------+ +---------+

Code organisation

The project uses lerna to handle multiple packages in one git repository. Packages are located in the packages folder. Each packages try to resolve one thing:

Base modules

• core: contain main definitions (Page, Context), so its the main dependency of other packages. The package contains code to normalize page definition.
• aggregator: a registry for service aggregators. A block from a page might need extra informations to be used. A closure can be attached to that registry, so it will be call once the page reference knows about the service. This can be used either on the Rendering engine or in the aggregation layer, of course the latter is better.
• api: expose using json format a Page definition coming from a loader.
• loader: load a page from a data source and return a page object with all information loaded from the datasource. It is possible to encapsulate this loader by others loaders if you need to add more information into the page definition.
• template-react: take a page definition and render the page to the client. This package works with React, contains registry to store block types with their rendering components.

Integration modules

CMS integration

• loader-contentful: integrate a page loader from contentful, the module provides a set of migrations to install required models.

• cms-drupal: WIP

Rendering engines

• rendering-nextjs: integrate NextJS as a rendering engine.

• rendering-gatsby: integrate Gatsby as a rendering engine.

Main data types

• Page: a CMS (Headless) must produce this object in order to work with any renderer. If the CMS cannot do that, a loader must be done, see loader-contentful for more information. The page can be exposed through a json API and so NO confidential information must be attached to it. It you need to store value, use the settings fields.
• RendrCtx: this is a object containing information about the current request, the object will be defined in the different lifecycle of the different component. If you need to store private information, use the settings fields.

Examples

Add information => need to explain the examples folder

Contributing

Please see our CONTRIBUTING.md.

Publishing new version

Please see our PUBLISHING.md.

rendr's Issues

`createApiLoader` can not handle chunked data response

Requests for large page data via createApiLoader never resolve. Only the first data chunk is ever processed by the Writable stream.

This seems to affect data responses with a content-length somewhere in the region of 65000 bytes or more.

More information

I've isolated this issue to the pipePageToClient function in the file: packages/loader/src/services/api.ts.

A new Writable stream is created to build the data from chunks via the .write method:

let data = "";
  const dest = new Writable({
    write(chunk) {
      data += chunk;
    }
  });

However, .write is never called more than once. For streams coming in multiple chunks, the whole of the data is not collated and the request does not complete.

The issue can be resolved by moving the functionality of .write into a callback triggered by the "data" event of the Readable stream, e.g.:

let data = "";
source.on("data", (chunk) => {
  data += chunk;
});

Steps to reproduce

The issue can be reproduced in the test suite packages/loader/src/services/api.test.ts, in the "test rendr/document mode, ie a Page" test, by passing a large Page object to the strPage variable:

const strPage = JSON.stringify(createPage(MOCK_DATA));

I've attached some suitable JSON to use as MOCK_DATA: ekino-rendr-loader-mock-page-object.json.zip.

The test will timeout without passing. Increase the Jest timeout setting to 30,000 (jest.setTimeout(30000)) and the test will still fail to complete.

add blockRenderer into the Component call in createBlockRenderer

Update createBlockRenderer signature to add blockRenderer for recursive rendering.

export function createBlockRenderer(
  blockRegistry: BlockRegistry
): BlockRenderer {
  return function blockRenderer(block, key) {
    const { component: Component, settings: props } = blockRegistry(
      block.type,
      block.settings
    );

    return <Component key={key} {...props} blockRenderer={blockRenderer} />;
  };
}

Attach Rendr fields to the user form on plugin install

Description

Rendr plugin adds 2 new fields to the User (field_rendr_allowed_channels and field_rendr_preview_token).

By default, these fields are not displayed on the user edit form.

Objective

• After installing the plugin, field_rendr_allowed_channels should appear on the user edition form
• After installing the plugin, field_rendr_preview_token should appear on the user edition form as readonly
  NOTE: Make sure it does not affect the account creation form for end users

Fix security alert and add github action to start audit

Something like this can be used to alter packages resolution

"resolutions": {
    "**/**/handlebars": "^4.5.3",
    "**/**/mem": "^4.0.0"
  }

Channel behaviour documentation

Add some documentation on Channel domain/locale.
How the channel is resolved.
How to pass the channel in query parameter or via domain + prefix

Auto generate and refresh Rendr user token

Description

The preview system of Rendr relies on a unique token attached to admin users with permission to view or edit Rendr pages.
There is currently no way to generate the token for all users

Objective

• Create a drush command that generates a unique token for each users with the right permissions
• This command can be attached to the Drupal cron so the tokens are refreshed regularly

Loader should act as a middleware

We should review the Loader interface so it can be use as a local middleware for page management.

So we can have an usage like this:

const loader = loaderChain([
  cacheLoader,
  apiLoader,
  redirectLoader,
  statusCodeLoader
]);

Channel is not loaded when both domain and locale are not empty

rendr/packages/cms-drupal/src/Resolver/ChannelResolver.php

Line 48 in 0916472

$channels = $this->entityTypeManager->getStorage('ekino_rendr_channel')->loadByProperties(

Steps to reproduce:

1. Create a channel with a domain www.my-domaun.com and a locale en
2. Contact the page api url with the right parameter /?rendr_channel=www.my-domaun.com/en

Expected The channel should be loaded

Actual The channel is not loaded

drupal - add filters to pages views.

Fix status code type in error page example

In the nextjs-with-remoteapi,
the status code for the error page should be a number not a string

Add support for Drupal

Add a drupal module to expose a rendr like compatible page.

Requirements

• The Drupal integration should support virtual site management (or channel); (ie not using the multi-site integration feature from Drupal).
• Each page is a set of properties and blocks, main properties are: statusCode, template, cache information, etc ... see an example here: https://nextjs-with-remoteapi.rande.now.sh/api/

Main tasks

• create a drupal module: ekino_rendr for drupal
• add dependencies to required external modules
• add any required content types
• add a default set of fixtures so external users can start with a real use case
• add a post content type (or module) to show how work the split between the structure and the content
• try to create a set of helper to improve the DX (ex: create a helper to define a block with mandatory fields: container, settings, order, type)
• expose the Page API and document how to get the filter per website/channel. So we can create a javascript example with a dedicated loader.

Notes

• For now, we use a mono-repo, so the PHP code need to go in the packages folders. This will likely not work with composer, so I plan to use https://github.com/splitsh/lite to make sure we can have a dedicated read-only repository.

PageCreator should return Promise<Page> or Page or void

For now the PageCreator should allows to return multiple types:

• Page: if there is no async code in the function
• void: if the pageCreator take over the request (ie: a case where the function return binary data)

Improve the error boundary loader

The errorBoundary should check if a previous loader already send http headers or content to the response stream. If so, we cannot send a page object.

Also the error is always the last one, we need to use a proper helper function to dump the full error stack, like this:

function dumpErrorMessage(err: Error | RendrError, stack: string[] = []) {
  if (err instanceof RendrError && err.previousError) {
    dumpErrorMessage(err.previousError, stack);
  }

  stack.push(`${err.message}: ${err.stack}`);

  return stack;
}

drupal - page inheritance for blocks

Fix build package.

The current command tsc -m es6 --outDir dist/lib-esm is used to generate module or jsnext:main for the packages.json references.

This generate issue with some Gatsby build as eslint expects to have import reference on top of the file.

Actions:

• check if there is no a better way to generate package (rollup.js) seems to appear very frequently these days.
• try to proper test with package downloaded from registry, and not only from local filesystem.

var __extends = (this && this.__extends) || (function () {
    var extendStatics = function (d, b) {
        extendStatics = Object.setPrototypeOf ||
            ({ __proto__: [] } instanceof Array && function (d, b) { d.__proto__ = b; }) ||
            function (d, b) { for (var p in b) if (b.hasOwnProperty(p)) d[p] = b[p]; };
        return extendStatics(d, b);
    };
    return function (d, b) {
        extendStatics(d, b);
        function __() { this.constructor = d; }
        d.prototype = b === null ? Object.create(b) : (__.prototype = b.prototype, new __());
    };
})();
import React from "react";
export function createGatsbyPage(templateRegistry, containerRenderer) {
    return (function (_super) {
        __extends(GatsbyPage, _super);
        function GatsbyPage() {
            return _super !== null && _super.apply(this, arguments) || this;
        }
        GatsbyPage.prototype.render = function () {
            if (!(this.props.pageContext instanceof Object)) {
                return React.createElement("div", null, "\"pageContext\" attribute is missing or not an Object from the Gatsby.ReplaceComponentRendererArgs Props");
            }
            if (!('page' in this.props.pageContext)) {
                return React.createElement("div", null, "No `page` props defined in the `pageContext`.");
            }
            var page = this.props.pageContext.page;
            var Template = templateRegistry(page.template);
            if (!Template) {
                return React.createElement("div", null, "Template is not yet defined");
            }
            return (React.createElement(Template, { page: page, blocks: page.blocks, containerRenderer: containerRenderer }));
        };
        return GatsbyPage;
    }(React.Component));
}

Add support for Gatsby

Contentful list articles should used a fixed date

The current request is https://cdn.contentful.com/spaces/xxxxxxx/environments/master/entries?limit=32&skip=0&content_type=rendr_article&fields.published_at%5Blte%5D=2020-08-28T22%3A03%3A29.885Z&fields.website.sys.id=7eWm8MzzDMaNSaoHV8Cpr3&order=-fields.published_at

The date is 2020-08-28T22%3A03%3A29.885Z but it should be something more cacheable => rounded to the hour or the day so contentful can cache the response.