ostis-ai/sc-machine

Software implementation of semantic network storage and processing

sc-machine's Issues

Add opportunity to expand sc-segments

Is your feature request related to a problem? Please describe.
Sc-segments amount is always fixes and can't be changed.

Describe the solution you'd like
In future there will be need to expand sc-segments dynamically.

Describe alternatives you've considered
The solution is a method in sc-memory that will allow to change sc-segements.

Add strings hashing in sc-fs-strorage

Is your feature request related to a problem? Please describe.
Problem caused by PR #41. Sc-dictionary-based sc-fs-storage doesn't support secure level for strings. Strings must be hashing at this level.

Describe the solution you'd like
A solution may be as a solution for sc-rocksdb-based sc-fs-storage that uses algorithm SHA256.

Add support on Windows

Is your feature request related to a problem? Please describe.
Current state of sc-machine doesn't support on Windows OS.

Describe the solution you'd like
Sc-machine must be supported on all OS.

Add integration tests utils

Describe the solution you'd like

• Add utils for agents workflow to tests them;
• Add utils to check memory state, compare it versions.

Describe alternatives you've considered

Firstly, it would be cool to check the performance of agents, for example, through memory states before and after. It is necessary to create specialized utilities that would allow checking the initial state of memory by templates, and then the final state of memory, which has changed as a result of the execution of some functionality. Secondly, there is also a need to create test chains for agents. Since agents are usually executed sequentially, it is necessary to implement the ability to configure these chains: agents in it, inputs, outputs of these agents, as well as branches

Add new levels of log messages

Is your feature request related to a problem? Please describe.
Current log messages (info, debug, error) are not convient for usage.

Describe the solution you'd like
I propose to expand log messages types set.

Describe alternatives you've considered
There will be types of log messages such as:

1. sc-memory log message;
2. customs and others.

Implement sc-dictionary nodes masking to avoid large memory usages

Is your feature request related to a problem? Please describe.
After #41 each node of sc-dictionary has 255 childs, some ones of that never appear. There is needed a mechanism to masking nodes so that use small memmory segments.

Describe the solution you'd like
You can explicitly specify the ranges of those characters that can be stored in storage, and filter out others. You can also use bits in the mask to indicate the language of the text.

Cover code by integration tests

Is your feature request related to a problem? Please describe.
All our platform projects has only module tests. It is awful. To provide system without errors and problems we need cover modules communication with tests.
Code to cover:

• sc-kpm agents;
• sc-kpm agent API.

Compilers optimization

Is your feature request related to a problem? Please describe.
In the current state of sc-machine Clang and GNU compilers are optimized as o3. See 65th and 73th lines in main CMakeLists.txt set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3"), set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3"). Optimization level can be correctly configured.

Usage of invalid addresses after rerun failed sc-server

Describe the bug
If sc-server wasn't normally closed, then after rerun of sc-server there will be invalid addresses in memory.

To Reproduce
Steps to reproduce the behavior:

1. Run sc-server;
2. Add elements with sc-server API;
3. Press on Ctrl+Z;
4. Rerun sc-server;
5. Find last created with sc-server API.

Expected behavior
There wiil be opportunity to save kb version history and checkout the last correct of them.

Desktop (please complete the following information):

• OS: [e.g. Ubuntu 20.04]
• Version [e.g. 0.7.0]

Up codecov level to 90%+

Describe the solution you'd like

• Remove glib -based tests;
• Add tests for sc-kpm;
• Update tests for sc-memory;
• Update tests for sc-builder.

Add codestyle documentation and update clang

Is your feature request related to a problem? Please describe.
We don't have actual documentation for code style and clang doesn't cover all code style rules

Describe the solution you'd like
Add code style documentation and update clang

Add opportunity to automatically update project development part of documentation

Is your feature request related to a problem? Please describe.
The documentation must record the history of development and operation, the current state and planning of the machine. To this end, it is necessary to add the ability to automatically change this part of the documentation.

Describe the solution you'd like
I think there are needed to record:

• PRs merges;
• Changelog changes;
• Questions solvings and applyings;
• Plans;
• Calls and speaks.

Memory issues

Describe the bug
The sanitizers detected memory leaks and the use of unallocated memory.

Links
Check this link

Invalid addrs crushes sc-iterator methods usages

Describe the bug
If sc-iterator was created with invalid not-exist addrs, then its methods is dangerous for usage, because they call errors.

To Reproduce
Steps to reproduce the behavior:

1. Create non-valid sc-address, for example ScAddr const & addr = ScAddr(125);;
2. Create 3-constr iterator `ScIterator3Ptr const iter3 = m_ctx.Iterator3(addr, ..., ...);
3. Call Next and Get methods from it.

Expected behavior
Iterators must check themself. It can be solved by IsValid() calls at the begin of iterators methods.

Desktop (please complete the following information):

• OS: [e.g. Ubuntu 20.04]
• Version [e.g. 0.6.1]

Implement strings effectively removing logic in sc-memory from sc-fs-storage

Is your feature request related to a problem? Please describe.
After #41 strings from file storage don't remove because there are ideas how to do it quickly.

Describe the solution you'd like
Strings must removing by sc-dictionary method sc_dictionary_remove and use cache to mind which strings wasn't removed.

Provide loop-ranged API method for sc-iterators and sc-templates

Is your feature request related to a problem? Please describe.
Current sc-iterators and sc-templates use only index-ranged methods. It's not convinient.

Describe the solution you'd like
Sc-iterators and sc-templates must have loop-ranged API method, such as for (auto const & item : iterator) {}.

[scs] SCs builder wrong behaviour if sentence starts from file

Describe the bug
Builder incorrectly processes scs-sentence if it starts from file (both kind of file representation, see examples)

To Reproduce
Try these sentences^

[IMS.ostis]
<- file;
=> nrel_sokraschenie:
[Метасистема IMS.ostis];;

"file://files/file0.html"
<- file;
=> nrel_sokraschenie:
"file://files/file1.html";;

Builder not fails, but output construction in memory is quite strange in both cases.

Expected behavior
It is expected, that builder creates two files in sc-memory with correspondent content and connects them with relation.

Screenshots
First case (sc-element with system identifier [IMS.ostis] was created)

Second case (sc-element with system identifier "file://files/file0.html" was created)

Add healthcheck endpoint to sc-server

Is your feature request related to a problem? Please describe.
After we deploy platform on server we regularly need to check its state. Now it's implemented in inconvenient way via external scripts .

Describe the solution you'd like
Add /health endpoint to sc-server.

Rename basic type classes for SCs sources in sc-builder

Is your feature request related to a problem? Please describe.
There are mistakeable sc-elements syntactic types.

Describe the solution you'd like
It is needed to rename basic type classes for SCs sources in sc-builder in accordance with document https://docs.google.com/document/d/1nW18hnETmqBRCvyUq6JywVLkVxx-MU3W4uZf143K3JI/edit

Config file overhaul

Is your feature request related to a problem? Please describe.
Currently configuring components of OSTIS is clunky and most of this configuration happens inside sc-machine. Some parameters are a legacy of previous versions. Config file makes other components FS-dependent on sc-machine. Partially this is solved by #43, but we should look into improving it further in such areas as:

1. Clarity and readability
2. Extensibility (for future modules or implementations)
3. Unification (unified config parser for both C++ code and Python modules/scripts)
4. Standardisation

Describe the solution you'd like
Most of the discussion about the config file happened upon reviewing #43. Some notable points:

1. currently naming of the variables is not unified and is not intuitive
2. It's not clear which parameters are required and which are optional. In case they're optional, it's not clear what's the default
3. There is no clear separation between configuration of each component. The file should either be unified and used across all instances\parts of the OSTIS-based system (then it would make sense to split the configuration into common parts which are used across most components, such as network/storage/caching/authentication, and then also split it into configuration of optional components, think [sc-inference] section or something like this).

Describe alternatives you've considered

1. We could split the configuration on a per-component basis for more independence between them, but it may become harder to set up a complex system like that. I would not like to do it the way ostis-web-platform did before #43 appeared.
2. We could also leave the old config format if you consider the problems described above non-critical, but we would still have to unify the parser, create some kind of standard and stick to it in the future.

Integrate scp-interpeter with sc-machine

Is your feature request related to a problem? Please describe.
To process kb programs and agents scp-interpeter and sc-machine integration is needed.

Describe the solution you'd like
To integrate scp-interpeter with sc-machine there are needed:

1. to correct process control (process deadlocks and races);
2. to choose relation type between them.

Additional context
In my opinion, scp-interpeter can be develop as a independent project.

Bug with usage of utilities with role relations

Describe the bug
IteratorUtils::getIterator5 and IteratorUtils::getAnyByOutRelation don't work correctly with role relations.

I tried to get a role-related object from this structure:

With role relations, except rrel_1, utilities return an empty address. But it works correctly if I use iterators in this case.

To Reproduce
Steps to reproduce the behavior:

1. Try to get a message from rrel_last relation in the structure above.
2. See error of empty address

Expected behavior
Correct work of utilities with role relations.

Desktop:

• OS: Ubuntu 20.04

Remove python from sc-machine

Is your feature request related to a problem? Please describe.
Sc-machine has dependence from python to run sc-builder and sc-server.

Describe the solution you'd like
It makes sense to leave one form of code sources in sc-machime on C languages.

Iterator utils include

Describe the bug
Try to build ostis-example-app on branch 0.6.0 from scratch and get next error

Scanning dependencies of target exampleModule
[ 98%] Building CXX object problem-solver/cxx/exampleModule/CMakeFiles/exampleModule.dir/agents/SubdividingSearchAgent.cpp.o
In file included from /home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-memory/sc-memory/kpm/../sc_object.hpp:9,
                 from /home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-memory/sc-memory/kpm/sc_agent.hpp:9,
                 from /home/viktor/work/ostis-example-app/problem-solver/cxx/exampleModule/agents/SubdividingSearchAgent.hpp:9,
                 from /home/viktor/work/ostis-example-app/problem-solver/cxx/exampleModule/agents/SubdividingSearchAgent.cpp:7:
/home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-kpm/sc-agents-common/generated/coreKeynodes.generated.hpp:60:18: error: ‘coreKeynodes_hpp_SubdividingSearchAgent_impl’ does not name a type
   60 | #define ScFileID coreKeynodes_hpp
      |                  ^~~~~~~~~~~~~~~~
/home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-memory/sc-memory/kpm/../sc_defines.hpp:11:45: note: in definition of macro ‘SC_COMBINE_INTERNAL’
   11 | #define SC_COMBINE_INTERNAL(v1, v2, v3, v4) v1##v2##v3##v4
      |                                             ^~
/home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-memory/sc-memory/kpm/sc_agent.hpp:147:3: note: in expansion of macro ‘SC_COMBINE’
  147 |   SC_COMBINE(ScFileID, _, __AgentName__, _impl) \
      |   ^~~~~~~~~~
/home/viktor/work/ostis-example-app/ostis-web-platform/sc-machine/sc-memory/sc-memory/kpm/sc_agent.hpp:147:14: note: in expansion of macro ‘ScFileID’
  147 |   SC_COMBINE(ScFileID, _, __AgentName__, _impl) \
      |              ^~~~~~~~
/home/viktor/work/ostis-example-app/problem-solver/cxx/exampleModule/agents/SubdividingSearchAgent.cpp:19:1: note: in expansion of macro ‘SC_AGENT_IMPLEMENTATION’
   19 | SC_AGENT_IMPLEMENTATION(SubdividingSearchAgent)
      | ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [problem-solver/cxx/exampleModule/CMakeFiles/exampleModule.dir/build.make:63: problem-solver/cxx/exampleModule/CMakeFiles/exampleModule.dir/agents/SubdividingSearchAgent.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2010: problem-solver/cxx/exampleModule/CMakeFiles/exampleModule.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

when remove this include and paste code of functions instead of functions, build finished successfully

To Reproduce
Follow the README.md installation instructions

Expected behavior
Successful build

Desktop

• OS: Ubuntu 20.04

Generation of the node with Russian identifier

Describe the bug
After resolving a node with a Russian identifier search return empty addr of node

To Reproduce
Steps to reproduce the behavior:

1. Generate node with the identifier on Russian
2. Find node by the same identifier

Expected behavior
Node should be returned, or shouldn't be generated

Screenshots

Desktop (please complete the following information):

• Version [e.g. 0.7.0]

Sanitizers work only on Clang

Is your feature request related to a problem? Please describe.
Sanitizers work only on Clang, as code coverage works only on GNU. It is a big problem in project developement. Such problems must be solved in the nearest future.

Add opportunity to specify a set of sc-elements as a sc-agent events

Is your feature request related to a problem? Please describe.
Now sc-agents can be initiated by only one sc-element. There is need to initiate sc-agents by a condition set.

Fix deadlocks in sc-memory

Is your feature request related to a problem? Please describe.
In sc-iterators when there are certain users deadlock can be runned

Code coverage works only on GNU

Is your feature request related to a problem? Please describe.
Code coverage works only GNU. May be code coverage can be launched on another compilers. Its problem can be discussed and researched.

Binary stripping in compilers

Is your feature request related to a problem? Please describe.
With using GNU compiler binary files are stripping. See 75th line in main CMakeLists.txt set(CMAKE_EXE_LINKER_FLAGS_RELEASE "${CMAKE_EXE_LINKER_FLAGS_RELEASE} -s") ## Strip binary. Such task doesn't solved with another compilers.

Describe the solution you'd like
Compilers must be correctly configured. There is needed to change configs in main CMakeLists.txt.

Handle crush errors

Is your feature request related to a problem? Please describe.
Sc-machine doesn't handle crush errors. It is needed to fix.

Describe the solution you'd like
A solution is sigactions adding, for example, default segmentation fault sigaction.

Search and generation by empty template

Describe the bug
If a sc-template has no triples, then search or generation by it causes an error (errors).

To Reproduce
Steps to reproduce the behavior:

1. Create empty template - ScTemplate scTemplate;;
2. Search sructures by this template - m_ctx.HelperSearchTemplate(template, result);;
3. An access error will be occured in search or generation method.

Expected behavior
Search or generation by empty template must be correctly process and return empty result.

Desktop (please complete the following information):

• OS: [e.g. Ubuntu 20.04]
• Version [e.g. 0.6.1]

Add ctrl+C for sc-builder

Is your feature request related to a problem? Please describe.
Now sc-builder has no opportunity to break itself in runtime with Ctrl+C.

Describe the solution you'd like
Sc-builder must have opportunity to break itself in runtime with Ctrl+C.

Do not install deps ubuntu in check-style ci

Is your feature request related to a problem? Please describe.
Check-style ci workflow mustn't install deps ubuntu. There is needed more effective solution for this ci.

Add tool to visualize scn-latex on Github

Is your feature request related to a problem? Please describe.
After repaclement of scn-latex there are no docs automatic visualizing on Github.

Increase code coverage by tests

Is your feature request related to a problem? Please describe.
There is a low code coverage.

Describe the solution you'd like
Not all code is covered by tests.

Export problems

Describe the bug
There are several problems regarding the export:

1. If you export existing knowledge to .gwf format, on next rebuild it doesn't get merged properly (you get 2 copies of sc-links)
2. .gwf export doesn't export file format (HTML and images are borked after rebuild)

_Originally posted by @FallenChromium

To Reproduce
Steps to reproduce the behavior:

1. Go to scg_code entity
2. Click on export button in SCg code mode
3. Put exported file to kb and rebuild kb.
4. Run system and see error

Expected behavior
Correct links and no duplicates.

Desktop (please complete the following information):

• OS: Ubuntu 20.04

Implement users authorization

Is your feature request related to a problem? Please describe.
Knowledge kb is shared by users in online mode. There are no mechanismes to define users.

Describe the solution you'd like
There are needed to:

1. describe implementation;
2. describe sc-elements structures and sc-elements users access levels;
3. and implement authorization.

Communication between `kb_scripts` and `build_kb.py` is shallow

Is your feature request related to a problem? Please describe.
After the merge of #43, all scripts related to building KBs will be moved to sc-machine repository. There's a problem with these scripts: kb_scripts use an error log file (...existence of it, rather) as a measure to message build_kb.py that something went wrong.

Describe the solution you'd like
The UNIX way for scripts to communicate with each other is by using exit codes - non-zero exit code would indicate that there was a problem.

Describe alternatives you've considered
Considering that there are non-fatal errors in kb_scripts, I would advise to establish an exit code system, so that we can be vocal about the errors, but continue the build process anyway. Alternatively, we could expose a standardised prepare() function (or something along those lines) in kb_scripts and import the logic as modules opposed to running in a subprocess.

Divide sc-machine into components

Is your feature request related to a problem? Please describe.
The sc-machine is overloaded by modules, such as sc-tools with sc-builder and sc-server, and sc-kpm.

Describe the solution you'd like
Sc-machine must contain only sc-core.

[scs] Allow specification of the structure into this structure itself

Is your feature request related to a problem? Please describe.
in OSTIS Standard there are a lot of constructions like this^

struct1
[*
=>nrel_idtf:
[Identifier1];;
*];;

We use them when we need include the structure specification into this structure itself. There are no other ideas how to do that shortly.

Describe the solution you'd like
Support this type of construction in scs-builder and in SCs grammar. Shortly the idea is - when scs-sentence is included in scs-contour and the first element of this sentence is not presented, that means that the first element of this sentence is the structure, defined by this scs-contour.

Describe alternatives you've considered
I don't see other "normal" ways, only idea is to write everything in triples and give temporal names to all sc-elements

Implement search by substring in sc-dictionary and sc-rocksdb fs-storage

Is your feature request related to a problem? Please describe.
To search by identifier dynamically user needs opportunity to search, while not known the whole identifier.

Leaks and uninitialized memory usage

Bug
During moving sanitizers I've found that there are a lot of memory leaks and uninitialized memory usage. Cause of nonzero return code it breaks build process.

To Reproduce
Steps to reproduce the behavior:

cd sc-machine/
cmake -B build -DSC_USE_SANITIZER=address
make -C build

cd sc-machine/
cmake -B build -DSC_USE_SANITIZER=memory
make -C build

Expected behavior
Build without errors.

Desktop

• OS: Ubuntu 20.04
• Version 0.6.0

Additional context
You may find tested code here
It's possible to ignore errors using -i and -k options of make, but it'd be perfect to
eliminate this problems.

Improve event queue management

Is your feature request related to a problem? Please describe.
Current event queue implementation doesn't solve tasks, such as event identification for use ids in clients, events creators detection
and non-linear container structure for quick completing. It make problem to solve tasks in user distributed system environment.

Describe the solution you'd like
I'd like to implement:

• events identification for search by id;
• events creators detection;
• events multiprocessing improvement.

Describe alternatives you've considered
I think it is needed to remake current events queue, add new models and change events structure for more appropriate one.

Build fails due to invalid casts after GLib update

Describe the bug
Building sc-machine on OpenBSD\macOS\Linux gives a following compilation error under clang++ (doesn't affect g++, because it only gives a warning rather than an error)

...error: cannot initialize a parameter of type 'gint *' (aka 'int *') with an rvalue of type 'void *'
    g_atomic_int_compare_and_exchange...

The error happens in the sc_lock.cpp file. This is caused by a recent GLib update. The fix is already on the way: https://gitlab.gnome.org/GNOME/glib/-/merge_requests/2578/diffs#f7f66e2c465806264320da3c0bf65728e0c5aff5

To Reproduce
Steps to reproduce the behavior:

1. Install glib 2.72.0.
2. Make a clean build of sc-machine

Expected behavior
A successful build

Screenshots

Desktop (please complete the following information):

• OS: macOS Monterey 12.3.1
• Version 0.6.0

Solution
Downgrade to glib 2.70.4 or upgrade to a release with the patch merged (there are none at the moment of opening this issue)

Additional context
It is not a problem related to sc-machine, I'm opening this issue so that other people facing this problem could easily find the solution.

Rewrite docs about sc-fs-storage into scn-latex docs

Describe the solution you'd like
Moves:

• Write formal text about sc-fs-storage
• Remove mkdocs

Unify sc-memory interface for link contents manipulating

Is your feature request related to a problem? Please describe.
Sc-memory API methods, such as SetLinkContent and GetLinkContent, gets and returns ScStreamPtr instances respectively. But methods HelperResolveSystemIdtf and HelperFindBySystemIdtf get std::string instances.

Describe the solution you'd like
I suppose to add analogs for SetLinkContent and GetLinkContent procedures, that will get and return std::string instances.

Describe alternatives you've considered
Old methods can be deprecated or used with new ones.

The optimization flag is -O3 for coverage & sanitizers

Describe the bug
Issue with cmake code, where the optimization flag is -O3 for coverage & sanitizers.

To Reproduce
Disable sanitizers and enable coverage, then build. Or disable coverage and enable sanitizers, then build.

Expected behavior
-O0 for coverage and -O1 for sanitizers.

Screenshots
Lines 91-97 and 104-109.

Desktop (please complete the following information):

• OS: [e.g. Ubuntu 20.04]
• Version [e.g. 0.7.0]

Fix cmake for apps

If build sc-machine that locate in <app_name>/ostis-web-platform/sc-machine by commands

# in <app_name>
cd ostis-web-platform/sc-machine
mkdir build 
cd build
cmake ../../..
make

CMakeLists.txt can't find cmake/tests.cmake file, so we get next error:

Unite mkdocs and latex docs into latex docs

Is your feature request related to a problem? Please describe.
To achieve the objectives of the platform project:

1. it is necessary to get rid of the informal representation of knowledge about the system in the form of documentation in the format mkdosc;
2. integrate with the formal description in the form of format latex text;
3. correct errors in the documentation, update and supplement it.

Describe the solution you'd like
Solution is:

1. unite documentation;
2. fix it;
3. and supplement it.