neos/contentrepository-development-collection

Work in Progress on the Event Sourced Content Repository as installable set of packages

The Neos package

Note

This repository is a read-only subsplit of a package that is part of the Neos project (learn more on www.neos.io).

Neos is an open source Content Application Platform based on Flow. A set of core Content Management features is resting within a larger context that allows you to build a perfectly customized experience for your users.

If you want to use Neos, please have a look at the Neos documentation

Contribute

If you want to contribute to Neos, please have a look at https://github.com/neos/neos-development-collection - it is the repository used for development and all pull requests should go into it.

Build frontend assets

If you need to rebuild the frontend assets you need to run the following commands within the Neos.Neos package directory:

For develpment:

yarn
yarn build

For production:

yarn
yarn build:production

contentrepository-development-collection's Issues

Move new NodeInterface and TraversableNodeInterface to core

neos/neos-development-collection#2202

Re-Implement Workspaces module

Project Management (tracking ticket)

fix Behavioral Tests

fix "not in menu" (as part of Node properties)

-> node property should be called "_notInMenu" (or so) with underscore)

Code Cleanup: Null Pointer exceptions

We assume objects to exists in some places where they can be null currently¹.
To avoid nasty PHP errors we should cater for those cases and check for null or adjust the API to avoid null cases to begin with.

---

¹ The EA inspections plugin can help to find possible null pointers in PhpStorm

Discussion of Show/Hiding Nodes

Implementation ticket: #13

I am yet unsure which approach to use for showing/hiding nodes.

The problem

• When you hide a node, all sub nodes should automatically be unreachable as well; so hiding/showing is recursive.
• Regarding working at hiding/showing nodes with dimension fallbacks - I think the rule must always be: When an anchestor node (in a particular CS/DSP) is hidden, the node itself is not reachable.
  • thus, we can either store the hidden/non-hidden information on the Node itself (i.e. the row in the DB); or we store it differently).
  • Problem when storing it on the Node itself: you effectively hide only the node in the Visible Dimension Space Points; i.e. if you hide "german" and another node exists for "swiss german", this is not affected by the hiding operation. Maybe this is what the user expects?? Unsure still about this!

Solution 1: introducing Restriction Edges

(as suggested by Bernhard)

The idea is to capture the relationship between the user's intention (i.e. hiding a certain node); and the effects of that operation (i.e. all nodes which are now hidden because of this intention).

This would mean we have a separate DB table restrictionEdge looking as follows:

• ContentStreamIdentifier
• DimensionSpacePoint
• OriginNodeAggregateIdentifier: the node which was hidden by the user
• AffectedNodeAggregateIdentifier: the node which should actually be hidden by the user's action

i.e. for a tree of three nodes:

• a
  • b
  • c

If I hide "a", I would get the edges "a->a" (to hide a itself), and the edges "a->b", "a->c"; all in their respective CS Identifiers / DSPs.

On forking a ContentStream, all these edges would need to be copied (similar to HierarchyEdges)

Enforcing these constraints

• on the query level, we would add a join with restrictionEdge on CS, DSP and AffectedNodeAggregateIdentifier
• if we find a match, we know this node is hidden (because some node above it was hidden)

Benefits of this solution

• pretty straightforward implementation
• can be easily extended to roles / starttime / endtime by adding properties on the hierarchyEdge

Drawbacks of this solution

• when e.g. having 100.000 nodes in 3 dimensions, when hiding the root node, we actually add 300.000 rows to the DB table.

Open Questions

• still unsure how much the additional join will performance-wise impact the already-quite-complex DB queries
• How to model this in Events? I think we have two "camps" here:
  • I (Sebastian) would only capture the infos in the event that a certain node was hidden, and create all the connections in the DB itself (similar as I implemented it for deletion). However, this somehow limits the usefulness of the event stream (e.g. you cannot answer the question "is node X currently visible" by looking at the event stream only)
  • Bernhard/Bastian would either have a huuuge event (e.g. in the example above with 300.000 affectedNodeAggregateIdentifiers); or 300.000 events; which I (sebastian) see as a big problem performance wise.
  • This discussion is IMHO a specialization of #21

Alternative: Check access at Query time

Alternatively to materializing the restrictions at the write side, we could also check the access of nodes at query time.

While this may sound prohibitively expensive at first (if things are checked for every node), we
can probably spend things up a lot because the Nodes needed to render a single page are very
often highly-connected trees -- e.g. when rendering a menu, you need a certain subtree of Document Nodes; and when rendering content, you need a certain
subtree of content.

The idea now is that we only need to do this recursive checking when fetching the root of the
tree; and not when fetching leaves (as we then have all information readily consumable).
Hopefully, we can create a Cache in ContentSubgraph which allows to speed this up tremendously.

The idea is still pretty rough; but I'd like to discuss it to see if it is relevant.

Benefits of this solution

• hiding/showing nodes does not depend on the number of subnodes below

Drawbacks of this solution

• by looking at the event stream, we cannot know whether a node is currently visible or not.

Alternative: Application Layer

We could also decide that the recursive checking is NOT done at all; but you are responsible
yourself to handle this in the application when fetching roots. This is actually pretty much
the (undefined) behavior of the old CR when checking this in detail.

Discussion: "Primary" events on the event stream vs "Derived" events?

(WIP)

Maybe an idea: Primary events capture what the user "actually" did; and Derived events are generated e.g. to be able to watch a certain Node's visibility changes.

However, unsure to me what the correctness constraints on this are; and how to be able to guarantee eventual consistency in this case.

UPDATE: See #100 for a more detailed concept

Improve Behat Test Setup for E2E Behat tests

see comments in #43

Improve Create Node

• basic "Create into" behavior for document nodes
  • "Create Into" for content nodes
• Changes\Create() should generate a unique node name using $this->nodeService->generateUniqueNodeName($parent->findParentNode()) (see TODO)
• implement "Create After" for document nodes
  • "Create after" for content nodes
• implement "Create Before" for document nodes
  • "Create before" for content nodes
• DocumentTitleNodeCreationHandler: re-enable nodeUriPathSegmentGenerator (see TODO inside)

add Travis CI support

REFACTOR: be able to use InternalRequestEngine again; instead of the forked CustomizedInternalRequestEngine

The only change in CustomizedInternalRequestEngine is that we comment out $this->securityContext->clearContext(); for the session to prevail between different requests. Thsi needs to be cleaned up maybe a bit further.

Fix CreateNode

allow modifying properties in backend (should emit new events)

Discussion: Possible Higher Level Write API

Hey everybody,

I have been thinking about creating a higher-level write-side API so people do not have to fiddle with the commands and correct identifiers etc for many cases.

My idea would be as follows:

• you can create an object NodeMutator (or something similar to that) which you can create like NodeMutator::fromNode(NodeInterface $node)
• this NodeMutator can be used like $mutator->setProperty(...) which will then emit the SetProperty Command.
  • This should return a MutationResponse (or s.th. like this); where you e.g. could call block() to ensure the projection has actually been updated. (Currently this would be a no-op until we introduce the async graph projection)
• to create a new node, one could do s.th. like $mutator->newChildNode() -> returning a NodeBuilder
  • then you can call methods on the NodeBuilder, to specify how your new node should look like.
  • ... and then, when you call .build(); which will emit the CreateNodeAggregateWithNode command. This will also return a new "NodeMutator" for the newly created node; so you can directly e.g. set properties for this.

What do you think about it? :)

All the best,
Sebastian

Fix E2E Behat tests (Small Detail regarding DimensionSpacePoint / DimensionSpacePointSet constructor still wrong)

When moving a HIDDEN node itself (and none of its children), it will LOOSE its hidden state

See TODO comment in GraphProjector::whenNodesWereMoved() -> can be properly fixed with a testcase; the query to write is just difficult. So I am deferring this a bit (as #27 is already huge).

Additional case: a CHILD of a to be moved node is hidden; so it should keep its hidden state as well.

Make backend work with current (TypeScript) UI, i.e. get rid of patches to TS UI

neos/neos-ui#2178

Discussion: Fully Consistent Graph Aggregate

(braindump)

• concept idea: create a single graph aggregate with snapshot per Content Stream, which can be used for immediate consistency of our invariants (e.g. node paths must be unique; the Nodes in a NodeAggregate must be all of the same Node Type; Nodes in a NodeAggregate must have disjoint DimensionSpacePoints)
  • makes the model a lot easier to reason about
• Implementation idea: basically copy our graph projection and make it contain only the structure (e.g. not the properties) -- use this as a snapshot of the aggregate state.
  • i.e. the aggregate would NOT reconstitute itself from an event stream, but would instead query the DB.
  • then the aggregate basically says: "when the event stream (for Content Stream X) is still at version A, the following events be applied: [...]"
• don't implement now, can be done later

Invariant Check: Content Graph integrity violation detection

e.g. "does a node really have only a single parent?"

Implement Show/Hide Nodes

see #27

implement move node

should be done by #20

Handle/Log errors at projection time

(Re)playing projections should never fail so possible invalid operations (due to failed soft constraint checks or bugs) should be ignored.

However we should log those cases in order to make them discoverable

Ensure projection:replayall works

remove NodeAggregate from ES stream names

fix dimension menu in demo site

Implement blocking Command handling in NodeCommandHandler

Rename "Show / Hide" to "Enable / Disable"

`get-policy-info` ajax request fails

from @kitsunet

• it fails earlier about not finding a site (in the ES CR node routing)
• seems to me an actual bug, the question is what would be the desired behavior? It grabs the Neos.Neos:Sites root node which (after transforming the demo site to events) has no dimension values. So in that dimensionspacepoint there is no child nodes (because they have dimensions). in current CR the code flow is somewhat different but we would have a default context with default dimensions and the sites node would have been found in that as well as the children. I think the ContentSubGraph that is used there is just wrong because it doesn't default to anything
• eg. what happens in \Neos\EventSourcedNeosAdjustments\EventSourcedRouting\EventSourcedFrontendNodeRoutePartHandler::fetchSubgraphForParameters
• I guess though the real problem is still in the fetchSubgraphForParameters
• I will try to make sense of that and compare to how it works with frontend

make Backend Rendering basically work

!!! protect access to non-public content streams

• Subgraph creation (constructor or so)
  • UNCOOOL ALTERNATIVE?? reading is not possible if unauthenticated for a user content stream (or another user's content stream) (Front Controller logic)
• when publishing commands, ensure they always target my content stream. (Command Handler)
  • NodeCommandHandler abstain logic; add custom privileges.

Interface / API Cleanup: Remove getSubgraph() from TraversableNode(Interface) (only used at 1 position)

Discussion with Bernhard:

• remove getSubgraph() from TraversableNode (only used once anyways)
• remove getVisibleDimensionSpacePoint() from Node
• add getVisibleDimensionSpacePoint() in TraversableNode
• implementation of TraversableNode::getVisibleDimensionSpacePoint() delegates to embedded subgraph

Node Schema Migration (planned changes)

• Schema Migrations
  • "Auto created child nodes have changes"
  • "default values have changed" in NodeTypes.yaml (today: node:repair)
• Node Type Renaming

(Bastian has started on that)

EventSourcedContentRepository Workspace Read Model -> get rid of ORM and use DBAL directly

this way, the Event Sourced CR should have NO dependencies on the ORM anymore!

do performance profiling with many nodes // try out editing on neos.io

REFACTOR: remove TraverseHierarchy API from ContentSubgraph

is difficult to use and error-prone

fix absolute FlowQuery stuff (e.g. footer in demo site)

Divide the GraphProjection into further "Aspects" (no AOP!!) // Strategies

Idea: Code complexity management; NOT further scalablity

Naming: "Constraint" // "Concern"?

!!! fix navigation to different pages in Backend

Fix Shortcut Support

Creating content nodes not working in the UI due to UI being reliant on parentContextPath

Currently, the Neos UI in https://github.com/neos/neos-ui/blob/master/packages/neos-ui-redux-store/src/CR/Nodes/selectors.js#L18 determines the parent context path by manipulating the context path string directly, which of course does not work anymore for node addresses which have no hierarchy.

We need to refactor the UI to not rely on this behavior anymore.

Make Content Stream versions known to Content Graph

In ContentStreamCommandHandler::handleForkContentStream() we currently have to iterate over all events of a CS in order to determine its version.
Instead we should make the versions of the individual CS known to the ContentGraph.
That would also be a requirement for #23

Discuss: remove NodeIdentifier

Why is NodeIdentifier actually needed? example "SetProperty" - why is this using NodeIdentifier and not NodeAggregateIdentifier, DimensionSpacePoint, CSI?

Need to think through how the shine through with multiple DSPs should work / works........

Example where this breaks: PublishingService::getUnpublishedNodes -> not possible to create TraversableNode, because we are missing DSP.

Remove NodeIdentifier from all commands and Events and replace by NodeAggregateIdentifier

Discussion: #16

Discuss handling of NodeTypeConstraints

currently not consistent I'd say. either we use fully a whitelist, or we use the old handling.

Discussion: Remove find* Methods on ContentGraph

I do not think we actually need these anymore; and they introduce ambiguity AFAIK.

Import / Export

Export

• Allow the export of all events for a given content stream 1:1 (line-delimited JSON)
• Include used assets
• Optionally: create zip file on the fly
• Later: "compact event stream" (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
  • (longer-term): “compact event stream” up to time X

Import

• Import into Event Store without constraint checks

Create Content Stream Pool (for improving performance)

Problem

Forking of content streams takes longer the more nodes exist (currently roughly 3 seconds for 50k nodes).
When publishing nodes partially this could lead to decreasing UX with larger growing projects.

At the "CR Sprint @ Dresden, April 2019" we collected some ideas on how to prevent this.
The approach we consider a good balance between performance & complexity rise is to pro-actively copy base content streams asynchronously into a "Content Stream Pool" so that we don't have to copy the Content Stream for every fork but can use a prepared one until the pool is used up.

Content Stream Pool

Table in the form (simplified):

ReferenceContentStreamId	ContentStreamId
live-xyz	cs1
live-xyz	cs2
shared-xyz	cs3

Maintained by the GraphProjector (see below)

NodeCommandHandler:

• Command ForkContentStream =>
  • get 1st Content Stream Identifier (CSI) from "Content Stream Pool" with ReferenceContentStreamId => Content Stream to fork =>
    • 0 results ? => create new ContentStreamId
    • else => use CSI from Pool
  • Publish Event ContentStreamWasForked with expect version = -1 (No Stream)

New Process Manager for "Content Stream Pools":

• Event ContentStreamWasForked (cs = Content Stream, v = event version) =>
  • How many new pools are required (n) (initially e.g. 5)
  • if n == 0 => die
  • else =>
    • copy edges (n times, each with a new CSI) of cs =>
      • publish event ContentStreamWasCopied(newCSI, referenceCSI) with expect version v + 1
      • (later: split into batches and publish events ContentStreamWasPartiallyCopied with expect version v + m + 1 (m = number of batches)
      • ConcurrencyException (= events have been published to cs in the meantime) =>
        • publish event ContentStreamCopyProcessWasIntercepted => "Restart" PM

Content Graph Projector

• Event ContentStreamWasForked =>
  • Remove corresponding CSI from "Content Stream Pool" table (if exists)
• Event ContentStreamWasCopied =>
  • Insert newCSI(s) and referenceCSI(s) to "Content Stream Pools" table
• Event NodeWas... (all content stream related events) =>
  • project to CSI and to all CSIs from the "Content Stream Pool"

EventSourcing: Centralized/Extensible Event serialization with symfony serializer

neos/Neos.EventSourcing#169

make frontend rendering basically work