onzag/itemize

Easy, Fast, Scalable, Reliable; ReactJS based FullStack engine.

itemize's Issues

Realtime search behaviour is not very predictable and not realtime

Currently search works this way.

• Every search record contains a created_at property
• The event listeners for requesting updates uses this property in order to fetch new records and request feedback.
• Item definition providers are in charge of keeping the item up to date.

There are flaws with this method.

1. If an item gets deleted it doesn't dissapear out of the search, it just turns blank.
2. If an item gets updated it just updates in place, not affecting the ordering or even its matching for the search.

The search should work instead

• Every search record has a last_modified property
• The event listeners request for updates in the specific domain "by-parent" or "by-owner" since the last_modified event, if we see records affected then we can inform about records edited and records added (last_modified given to check is smaller than created_at)
• In order to check for records deleted we should check the deleted records table, with the given type; given that itemize delete are hard deletes in order to keep GDPR compliance, this table was created to keep track of deleted records without any information.

This should fix the flaws, some things to consider:

1. For non-cached search t(both traditional, and standard) hat doesn't run in the cache worker as locally this would mean that it has to re-ask the server for brand new information, since it has no other records.
2. Cached search should be exceptionally performant and predictable.

Payment Service/Type with item integration

This will be part of the way to monetize Itemize, providing a service to add payment support for safe monetary transactions; it should be easy as just a simple definition and an API key; the service should be linked to a payment provider.

Tags type

Proposing the introduction of the tags type, which represents itself as an array of text type, and allows for tagging of the content with special predefined tags.

The valid tags are given via the values, and expect to get translations each.

Custom tags might be possible to be added, and will affect how search is done, whether it is a standard tag or a custom tag; the problem with custom tags is translation, custom tags can't be translated or have i18n content.

Random ids/string ids?

Right now ids follow a standard auto increment mechanism in the database, this is not optimal as it allows to easily guess ids as well as to easily find out the structure of the database and override what could be protected namespaces; such as in the case of fragments.

Two solutions come to find.

1. https://wiki.postgresql.org/wiki/Pseudo_encrypt
2. Change the type of the id to be a text type, and use random alphanumeric identifiers for the id (no solution comes to mind on how to generate this index).

The second solution proves to be the most flexible as the id type being a string will prove to be incredibly effective as well to allow for custom identifiers to be set, however it would require a major refactor where every single id in every single file would need to be changed to be string/text.

The first would be easier to implement since it would require a simple change to the way the index works as well as how values are created; but it will not allow for custom ids.

I feel like gravitating towards the first because it's easier and for the end user there is no much difference, however this creates a coding situation where numeric values hold some meaning to them; but this is a hurdle to the programmer, we can probably make the first 1000 values to be protected and unable to be held by an index.

Copy/Paste/Drag/Drop in fast prototyping rich text editor

Right now copy and paste doesn't work quite right as soon as images/files are added into the mix, dropping images/files doesn't work as well.

Testing module client side mode needs more work

Currently there's no much testing that happens with puppeteer, when the testing mode is enabled, there are a couple of things that need to be established.

1. Mocking of the entry fields, by adding some form of "mock", "mockWith" property to the entry type.
2. Clicking buttons and actions, exploring the website; by adding some form of special react component, in order to track where this special component leaves a trace we leave a dangling hidden DOM piece as previous sibling, and then events can be broadcasted to the subtree of the next sibling.
3. These events should allow submitting and any other action as if the user was interacting with the site.

Mutation and merging happens even if the file limit is reached

In certain unlikely scenarios mutation can merge via the gql query merger, and create a request that will be rejected by the server because the sheer amount of files that have been passed.

In order to fix this bug, we should ensure that the file limitations are not exceeded by the merger.

This should be as easy as counting the files, their size, and ensuring the limit is not reached to toggle the boolean merge flag.

Workflow Improvements/Refactor

1. Generate interfaces for all the types and all the structures of every schema generated in build data so that it can be used in the source.
2. Raw DB must be easier to use and be integrated with these.
3. SQL table row value must be more meaningful.

CMS improvements

The Standard CMS that comes in itemize is just not very good.

1. A mechanism to prevent types overriding, right now fragments are identifiable by a given numeric id; but they can be overriden by articles, the sharing of these protected ids is not good nor safe. This might be fixed by the changing of the way the database index works.
2. The CMS should display all the available fragments and articles in an easy to use list.
3. By ID creation should be enabled, where an id can be manually specified.
4. A mechanism to allow to define fragments options as well as predefined template fragments so these are easily accessible and easily designed for.

Socket is unidentified

In some situations the socket is unidentified when registering, there seems to be a timing issue that does not allow for the token to be registered via the handshake on the socket io remote protocol.

It can be easily replicated by forcefully making the identify request wait for 3 seconds before replying, the client still sends the registering requests despite not having received confirmation from the server that is allowed to do so, causing an error.

Add a by-property search domain

Once #10 is fixed a by-property search domain might be added to support integer and string identifier types.

Currently the domains that exist are:

• by-parent (requires a user id)
• by-owner (requires a parent_id, parent_type and parent_version)

The by-property domain would allow to choose a property and a property value to act as domain and event listeners will be fired and registered based on that property, the cache worker in the client side should also be able to handle these results in this domain.

Handling of the cache (what happens on cluster disconnect?)

If a cluster manager loses connection of its redis clients to the global cache it should wipe the cache and remove all the listeners as it doesn't know whether during the outage it lost events and as such the cache suddenly becomes invalid.

So all the cache should be marked as invalid, no feedback to check, simply blow it; and add a log message as error type because this shouldn't have happened to start with.

Knex should have no issue with this as during an outage will cause the endpoints to crash giving INTERNAL_SERVER_ERROR and once it recovers any knex related functionality should be mantained, no such case is with the local cluster cache, so the cache should be blown.

Remembering that sometimes the cache is the same, we should ensure not to blow global cache variables; now this should mean that it shouldn't even happen because then there's a single cluster and the global is the same, but just to keep consistency.

Flagging, blocking, modding and banning is not implemented

Flagging and modding are already properties in the system, even modRoleAccess is there, but there's nothing that can be done with these properties.

All the necessary properties are there, it's just not implemented, there are even commented out lines where flagging should take part in.

The search for the modding behaviour needs to be added nevertheless, both in search and traditional search for these items that need moderation, flagged, and blocked as well; there need to be args for them, that moderators can use.

Archival Support

We should provide archival support, the way it should be handled.

1. Every module will have an archiveAge property that represents how old the item is when it needs to be archived in milliseconds, similar to requestLimiters since attribute.
2. A new base property archived_at is added.
3. Archives (should, might) be readonly, if this is the case we might need an ARCHIVED error type.
4. When retrieving an item to put it in the cache, the primary table is used first, if not found there it fallbacks to the archived table.
5. When performing search, using the since limiter should be able to make it possible to know whether we should use the standard primary table or the archived table, as such we should endorse using since then; if no since is specified, use the archival table; endorsing the use of since then provides a optimization.

Archived elements are simply downprioritized.

Archival should mostly be necessary when we are talking about large data, we shouldn't endorse the use of archival for small units of data.

Check https://www.postgresql.org/docs/9.3/ddl-partitioning.html to see how it might be implemented, this might affect the way that the schema is created as well and build, so the build-database might need to change to enable the archiving support.

Third party services

We need a way to add third party services via the customization attributes of the config.

Optimally they would also be services like the mail service, storage service, etc... and they should be added to the service configuration file and consume information from the config custom attributes of both sensitive and standard config.

Keyword search is non-functional

Currently there are language keywords that are added in each item definition for search mode.

These keywords are not really used anywhere.

Module based search should allow filtering by item definition types given the usage of these keywords and consume these keywords.

Unsubscribe from emails directly from the email

Users should be allowed to unsubscribe from emails from the email itself, and a rest endpoint should be added that allows users, given a token, to unsubscribe and change the property that represents the subscription to false.

As well sending emails via the mail service should be linked to a property and should not send the email if the user that it represents proves to be unsubscribed, an alternative unvalidated endpoint should be offered as well.

The ubsubscribe link should be somehow injected into the template, when we talk about templates.

But since headers are also a thing, this requires integration with whatever the email service provider is; for the default provider, mailgun, the documentation must be check to ensure things are in sync, since the process should work in a provider agnostic way and not make a hard link.

Emails are going to spam due to the lack of unsubscription, this should solve this.

OnzaSystems website

The website that contains information about Itemize in a more user friendly way is down currently.

The repository is at https://github.com/onzag/onzasystems but uses a very old version of itemize that has been heavily reworked so most likely it will need to be redone.

The website is important and should contain key features about what itemize is and what it tries to achieve in a less "geeky" way than github.

Improve the standard CMS

1. We should be able to list all the available fragments.
2. The versioning should be a dropdown of what we should version as.
3. Depends on the support of the whole text specs #6 issue and the brand new editor.
4. We should have a fragment fast prototyping loader component, that doesn't do much overall than be an disguised item provider; however this fragment loader should detect if we have edit access on fragments and provide an edit button in such a case, so that designers can work on the site and get a redirection to the CMS.
5. Programmers can specify in the fragment what context each fragment is in, and if the fragment is supposed to be a template; as some form of CMS property; since fragments are supposed to be identifiable; this again refers to #6

This will be a powerful change that separates a lot of the work of programmers and translators with the work of designers, leading to efficiency as designers will not anymore need programmers to do even complex changes in the structure of the application, and can just modify the website in real time in production builds.

Documentation should be done

The documentation is very incomplete, and almost nothing is documented this prevents any newcomer or literally anyone from using this platform.

Documentation should be written for at least the stable modules and go on from there.

Server side search caching

The server side cache is very powerful and provides ways to cache what it refers as IDEFQUERY or item definition single get queries.

Item definitions could be marked for caching, via a cacheSince attribute which preferably would be equal to the request limiter since attribute; another option is cacheAll to literally cache everything for that item.

Add event listeners for records added, records removed, and records edited (similar to #10 so #10 should be done) for a none domain, so that is just inform for these changes on a table level to keep them updated in each cluster level cache by the cluster manager.

Every record should then register and store each of its respective IDEFQUERY as well, similar to how the cache worker does it in the client side.

The records stored in the cache should be different than the standard records and during the cache and indexing event we should retrieve both last_modified and created_at in order to be able to keep these records clean as they age, because we don't want a forever growing cache, however we would only clean it as we need to change it.

This will be the job of the cluster manager to mantain these records, and even download and register them; each extended node will have nothing to do with this.

However the appData.cache will then need a search method so that it can then use those cached records to perform its search and will use the local search functions in order to do the matching.

After filtering is done using the local search functions then we would have to use local ordering as well, these are the same functons used in the client side for the cache worker.

Pitfalls.

• No support for full text search, this means that if detected will have to fallback to postgresql; probably by noticing that the local search function is null/undefined for one of the search fields; this same flaw exists on the client side; however it might also be possible to just pass the tsquery to postgresql to check whether it matches true or false since we are on the server side.

Unsure whether this will be implemented at all because postgresql is fast as it is, and searches are not very common, so it doesn't have a lot of value unless the system is so big that searches are constantly performed and the database needs a relief.

Support for analytics

A way must be devised so that analytics can be supported (for whichever analytics platform is used, eg. google analytics).

Analytics are a powerful tool for management of a website.

While currently they are not required, they certainly should be added later, as having support for these adds quite some value to the platform. Probably via an event driven system that can be linked via a plugin, and informing these actions from the item provider, others will come from popstate.

Includes are untested/non-implemented

The functionality for includes, which allows to extend an item definition with another is not yet implemented or is otherwise untested.

These are.

• Include schema functionality (untested)
• Include search (untested)
• Include client side operations (not implemented)

The includes should be added and make work at some point.

Includes have proven less useful than expected so this issue has little priority and in fact it might be better to remove them altogether if it proves messy to implement them.

Files type, handler, fast prototyping entry and view

The files type has no handler and no fast prototyping entry nor view.

While this is not currently very relevant because the text type can inject the file property as media, and that's the best way to utilize such property, there should be a handler and fast prototyping entry and view in order to maintain consistency.

It should be similar to the file type but allowing to display lots of files.

Text editor does not support the entirety of text-specs

QuillJS does not support the entirety of the text specification and does not provide a nice way to fully edit templatized fragments and/or build fragments.

SlateJS https://github.com/ianstormtaylor/slate should support all these configurations out of the box and should replace quill.

SSR search results and resources

Automatic search and search results are not added into consideration for the SSR functionality, this means they do not SSR.

With the beforeSSRRender method this should be achievable as we can now capture props and state on the fly, however caution must be advised with the search state that is loaded from the navigation key, specially after this state must be loaded from the constructor and might not match the server state vs client state; this used to cause issues in the past and was moved to componentDidMount due to the search state loaded from navigation disrupting SSR.

Resources loaded via the HTMLResource loader should also be added into this list of SSR resource, this one should be rather easy.

This should drastically improve page responsiveness and SEO where these items appear, granted, they are not common use for this.