accelerated-text / accelerated-text Goto Github PK

Accelerated Text is a no-code natural language generation platform. It will help you construct document plans which define how your data is converted to textual descriptions varying in wording and structure.

Home Page: https://www.acceleratedtext.com

License: Other

Makefile 0.14% Clojure 11.43% Dockerfile 0.05% JavaScript 86.79% Shell 0.13% Python 0.39% Grammatical Framework 0.09% Ruby 0.02% Sass 0.97% Gherkin 0.01%

clojure graphql javascript natural-language-generation nlg nocode text-generation

accelerated-text's Introduction

A picture is worth a thousand words. Or is it? Tables, charts, pictures are all useful in understanding our data but often we need a description – a story to tell us what are we looking at. Accelerated Text is a natural language generation tool which allows you to define data descriptions and then generates multiple versions of those descriptions varying in wording and structure.

Documentation • About • Philosophy • Features • Get Started • Demo • Development • Contact

About

Accelerated Text can work with all sorts of data:

descriptions of business metrics
customer interaction data
product attributes
financial metrics

With Accelerated Text you can use such data to generate text for your business reports, your e-commerce platform or your customer support system.

Accelerated Text provides a web based Document Plan builder, where:

the logical structure of the document is defined
communication goals are expressed
data usage within text is defined

Document Plans and the connected data are used by Accelerated Text's Natural Language Generation engine to produce multiple variations of text exactly expressing what was intended to be communicated to the readers.

Philosophy

Natural language generation is a broad domain with applications in chat-bots, story generation, and data descriptions to name a few. Accelerated Text focuses on applying NLG technology to solve your data to text needs.

Data descriptions require precision. For example, generated text describing weather conditions should not contain things beyond those provided in the initial data – temperature: -1C, humidity: 40%, wind: 10km/h. Despite this, the expression of an individual fact – temperature – could vary. It could result in "it is cold", or "it is just below freezing", or "-1C", but this fact will be stated because it is present in the data. A data to text system is also not the one to elaborate on a story adding something about the serenity of some freezing lake – again, it was not in the supplied data.

Accelerated Text follows the principle of this strict adherence to the data-bound text generation. Via its user interface it provides instruments to define how the data should be translated into a descriptive text. This description – a document plan – is executed by natural language generation engine to produce texts that vary in structure and wording but are always and only about the data provided.

Key Features

Document plan editor to define what needs to be said about the data.
Data samples can be uploaded as CSV files to be used when building Document Plans.
Text structure variations to provide richer reading experience going beyond rigid template generated text.
Language and vocabulary control to match each of your reader groups.
Build-in rule engine to allow the control of what is said based on the different values of data points.
Live preview to see variations of generated text.

Get Started

The easiest way to get started is to use Accelerated Text Project Template. It will provide you with the necessary project configuration structure.

If you want to start tinkering and run it based on the latest code in the repository, first make sure that you have make and docker-compose installed, then clone the project and run

make run-app

After running this command the document plan editor will be availabe at http://localhost:8080, while AMR and DLG editors will be reachable via http://localhost:8080/amr/ and http://localhost:8080/dlg/ respectively.

For more detailed description of text generation workflow visit the Documentation.

Demo

For a demonstration of how Accelerated Text can be used to provide descriptions for various items in an e-commerce platform (https://www.reactioncommerce.com/) please check the following repository: https://github.com/tokenmill/reaction-acc-text-demo.

Development

To get started with a development environment for Accelerated Text please follow the instructions in our developer's guides for the front-end, api and the text generation engine.

Contact Us

If you have any questions, do not hesitate asking us at [email protected]

If you'll submit an Issue this will help everyone and you will be able to track the progress of us fixing it. In order to facilitate it please provide description of needed information for bug requests (like project version number, Docker version, etc.)

License

Distributed under the The Apache License, Version 2.0.

accelerated-text's People

Contributors

Stargazers

Watchers

Forkers

emilis joyfulsun whiteeyehansel zorrock expovation francisjervis philippoes thelatch marcograham chenxian01 atlaspilotpuppy praveenvattem amit1nayak randyh0120 fakegit shaileshj2803 yclolivier harshalshende oytuntez motaword simbaninja917 gitter-badger dddominikk asmaswapna site-command dwtcourses bruto001 ahmadfikrimasyhur edvillan15 mehranattari merlin-lacuna marceloluk fullbright 5l1v3r1 tawawhite gg-big-org pauloduarte43 chillskellingtom vitaly-z disnofprosde techxbase sohaib0399 dat0106 ekakit reanimatedmanx owami munirh80

accelerated-text's Issues

Merge three parts of frontend into single project

Document Plan Editor, AMR editor and RGL editor now are three separate projects, but basically these are copies of same project. Need to invest time and do proper single project

Use unicode in GF service

@(client/request {:url     "http://localhost:8001/parse"
                  :method  :post
                  :headers {"Content-type" "application/json"}
                  :body    (json/write-value-as-string
                             {:module   "Lang"
                              :instance ""
                              :content  {}
                              :text     "£"}
                             (json/object-mapper {:escape-non-ascii true}))})

produces 'ascii' codec can't encode character u'\xa3' in position 0: ordinal not in range(128)

Read only json files in the document plan dir

Filter files in document plan dir by the JSON extension to prevent attempts to parse all sorts of random files that might get into that folder.

Move `traffic` to `.traffic`

Traffic stats should go into the hidden folder .traffic instead of a prominent traffic

GF API should return parse structure

Using GF shell we can get parse structure like this:

i alltenses/LangEng.gfo
p "this wine is good"
PhrUtt NoPConj (UttS (UseCl (TTAnt TPres ASimul) PPos (PredVP (DetCN (DetQuant this_Quant NumSg) (UseN wine_N)) (UseComp (CompAP (PositA good_A)))))) NoVoc

API call is needed to get this. API parameters:

text
dictionary [ [ "word" : "pos" ], ... ]

Dictionary should be used dynamically during parse time (if possible).

Exported document plans should have human readable file names

Current file names are like this api/resources/document-plans/vsqYQzbYQfOGaQtU.json.
This makes it hard to manage them manually. We should have DP name in the file name - My Plan - vsqYQzbYQfOGaQtU.json or something like that.

Spec for Document-Plan

At the moment there's only partial spec, we need to create proper one

Async parameter was removed from text generation endpoint

There was this nice feature added in PR #152, which seems to be removed now.

It could ease the use of this endpoint in various cases.

Bring back UI tests

Now they're disabled and most of them fails due to changes.
Lets make them work

Just an idea: try to cover some scenarios as a behaviour driven tests?

Document reader-model setup

Document how to launch on Windows

https://docs.docker.com/docker-for-windows/install-windows-home/

Xlsx data file support

Currently only .csv is supported

Formatting dates

Hi there,

Is it possible to format date values? If yes, what kind of format would AccText require?

Non-existing or empty values in data rows

Currently, if you don't provide the value for a data cell, it will yield a missing value exception:
https://github.com/tokenmill/accelerated-text/blob/a322f9b6436923a3f624a0bdd53d6466541e047e/core/src/acc_text/nlg/graph/data.clj#L8

(defn get-data [data key]
  (if (contains? data key)
    (get data key)
    (throw (Exception. (format "Missing value for data cell: `%s`" key)))))

I think this should be a warning and default value of a missing key should be null/empty string. Otherwise, this kind of document plans are not possible and API client needs to provide all values which limits the document plans to a good degree:

See client.corporate.name check:

List of plans won't refresh after deletion in AMR and DLG editors

Create new plan in either of editors, press delete button, confirm. It won't refresh until after page reload. Do the same process in main editor for desired outcome.

Segment headings

Segments act as structural text blocks like chapters in the article. We need to be able to set a Header of the Segment.

For example this book description:

Overview
The connotations of words drawn from usage in English literature are studied to recover lost meanings and analyze function in this classic study of verbal communication by an authoritative analyst of the English language.

Editorial Reviews

"Rarely is so much learning displayed with so much grace and charm. My only regret is that the book was not twice as long." The New York Times Book Review
"...a brilliant book addressed to students and to lay people alike, unbaffling, deeply informative, and timelessly persuasive." Robert Burchfield, Editor of the Oxford English Dictionary

Hardcoded port in front end service

If I change the port in docker-compose from 8080 to, for example, 8088, then http://localhost:8088/amr works, but http://localhost:8088/amr/ redirects to http://localhost:8080/amr

Use profiles in docker-compose

https://docs.docker.com/compose/profiles/

Missing hints in UI

Reader models and AMRs, when empty, should have [i] as well:

Unify code formatting

https://github.com/weavejester/cljfmt

Question: how to create/edit AMRs

Hi, I cannot find the answer to this in the docs - for instance I want to create "is-owned-by." Not sure if this just undocumented or a new feature request?

EDIT: looking into the code this is of course defined in the the concept-net YML files, so I guess this is a feature request for editing those in the front end!

Azure pipeline status code 0 when tests fail

Upgrade dictionary item implementation in core

Using graph generated from Paradigm functions, should be able to create solution to work for any type/ language.

Smarter subset of preview data rows

Now it takes 20 first. In case of current Eval example, first 20 rows varies very little, while actually interesting rows are left behind. Would be awesome to get as much variety as possible.

Maybe just a random shuffle each time, or select by nil/non-nil columns

Add default document plans to ReactionCommerce Demo

ReactionCommerce repo should have doc. plans, AMRs and dictionaries for:

Dresses
Pants

Quick menu dictionary items have null part of speech tag

Blockly XML with dictionary item from sidebar:

<xml
	xmlns=\"http://www.w3.org/1999/xhtml\">
	<block type=\"Document-plan\" id=\"rviqrSdvHPXTyKGwFhQt\" deletable=\"false\" x=\"260\" y=\"120\">
		<statement name=\"segments\">
			<block type=\"Segment\" id=\"PnmqkUxnotmTNrcQJUaI\">
				<mutation value_count=\"2\" value_sequence=\"value_\"></mutation>
				<value name=\"value_0\">
					<block type=\"AMR\" id=\"MVbThWgFjmbVQTCkemLk\">
						<mutation concept_id=\"PlIxwTBumcCfGFGz\" concept_kind=\"Text\" concept_label=\"on the hill\" roles=\"[{&quot;id&quot;:&quot;stozLtQLvfWMFVxs&quot;,&quot;fieldLabel&quot;:&quot;thing&quot;,&quot;__typename&quot;:&quot;ThematicRole&quot;,&quot;fieldType&quot;:[&quot;List&quot;,&quot;Str&quot;,&quot;N&quot;]}]\"></mutation>
						<value name=\"stozLtQLvfWMFVxs\">
							<block type=\"Dictionary-item\" id=\"635YfTe_wQd)?cpC3)h%\">
								<mutation id=\"house_N_Eng\" name=\"house\" pos=\"N\"></mutation>
							</block>
						</value>
					</block>
				</value>
			</block>
		</statement>
	</block>
</xml>

Blockly XML with dictionary item from quick menu:

<xml
	xmlns=\"http://www.w3.org/1999/xhtml\">
	<block type=\"Document-plan\" id=\"rviqrSdvHPXTyKGwFhQt\" deletable=\"false\" x=\"260\" y=\"120\">
		<statement name=\"segments\">
			<block type=\"Segment\" id=\"PnmqkUxnotmTNrcQJUaI\">
				<mutation value_count=\"2\" value_sequence=\"value_\"></mutation>
				<value name=\"value_0\">
					<block type=\"AMR\" id=\"MVbThWgFjmbVQTCkemLk\">
						<mutation concept_id=\"PlIxwTBumcCfGFGz\" concept_kind=\"Text\" concept_label=\"on the hill\" roles=\"[{&quot;id&quot;:&quot;stozLtQLvfWMFVxs&quot;,&quot;fieldLabel&quot;:&quot;thing&quot;,&quot;__typename&quot;:&quot;ThematicRole&quot;,&quot;fieldType&quot;:[&quot;List&quot;,&quot;Str&quot;,&quot;N&quot;]}]\"></mutation>
						<value name=\"stozLtQLvfWMFVxs\">
							<block type=\"Dictionary-item\" id=\"x:,ly@ARnV(^b}^eT;,O\">
								<mutation id=\"house_N_Eng\" name=\"house\" pos=\"null\"></mutation>
							</block>
						</value>
					</block>
				</value>
			</block>
		</statement>
	</block>
</xml>

Update API documentation

can not access to swagger API

Hi, I am new in Accelerated-Text !

I try download and run software, but I have problems.

First I clone repo, and run "make run-app". Then conteiners are created and run correctly. I can open web UI correctly, and create documents, etc. in http://192.168.1.131:8080/

Iam using

Hostname	desktop
OS Information	linux x86_64 Ubuntu 18.04.4 LTS
Kernel Version	4.15.0-112-generic
Total CPU	4
Total memory	11.7 GB

Version	19.03.7 (API: 1.40)
Root directory	/var/lib/docker
Storage Driver	overlay2
Logging Driver	json-file
Volume Plugins	local
Network Plugins	bridge, host, ipvlan, macvlan, null, overlay

But when I try see API in http://localhost:3001/index.html then I get error:

acc-text-api_1 | 2020-08-15 10:22:08,001 ERROR a.error - Uri: /swagger.json Stacktrace: acc-text-api_1 | java.lang.Exception: Unable to resolve spec: :api.nlg.service.request/documentPlanName acc-text-api_1 | at clojure.spec.alpha$reg_resolve_BANG_.invokeStatic (alpha.clj:69) acc-text-api_1 | clojure.spec.alpha$reg_resolve_BANG_.invoke (alpha.clj:64) acc-text-api_1 | clojure.spec.alpha$fn__1803.invokeStatic (alpha.clj:132) acc-text-api_1 | clojure.spec.alpha/fn (alpha.clj:130) acc-text-api_1 | clojure.spec.alpha$fn__1782$G__1777__1791.invoke (alpha.clj:121) acc-text-api_1 | clojure.spec.alpha$specize.invokeStatic (alpha.clj:152) acc-text-api_1 | clojure.spec.alpha$specize.invoke (alpha.clj:151) acc-text-api_1 | clojure.spec.alpha$form.invokeStatic (alpha.clj:177) acc-text-api_1 | clojure.spec.alpha$form.invoke (alpha.clj:173) acc-text-api_1 | spec_tools.visitor$spec_dispatch.invokeStatic (visitor.cljc:13) acc-text-api_1 | spec_tools.visitor$spec_dispatch.invoke (visitor.cljc:9) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:238) acc-text-api_1 | spec_tools.visitor$visit.invokeStatic (visitor.cljc:52) acc-text-api_1 | spec_tools.visitor$visit.invoke (visitor.cljc:26) acc-text-api_1 | spec_tools.visitor$eval22199$fn__22200$fn__22201.invoke (visitor.cljc:60) acc-text-api_1 | clojure.core$mapv$fn__8445.invoke (core.clj:6912) acc-text-api_1 | clojure.core.protocols$fn__8159.invokeStatic (protocols.clj:168) acc-text-api_1 | clojure.core.protocols/fn (protocols.clj:124) acc-text-api_1 | clojure.core.protocols$fn__8114$G__8109__8123.invoke (protocols.clj:19) acc-text-api_1 | clojure.core.protocols$seq_reduce.invokeStatic (protocols.clj:31) acc-text-api_1 | clojure.core.protocols$fn__8146.invokeStatic (protocols.clj:75) acc-text-api_1 | clojure.core.protocols/fn (protocols.clj:75) acc-text-api_1 | clojure.core.protocols$fn__8088$G__8083__8101.invoke (protocols.clj:13) acc-text-api_1 | clojure.core$reduce.invokeStatic (core.clj:6828) acc-text-api_1 | clojure.core$mapv.invokeStatic (core.clj:6903) acc-text-api_1 | clojure.core$mapv.invoke (core.clj:6903) acc-text-api_1 | spec_tools.visitor$eval22199$fn__22200.invoke (visitor.cljc:60) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:239) acc-text-api_1 | spec_tools.visitor$visit.invokeStatic (visitor.cljc:52) acc-text-api_1 | spec_tools.visitor$visit.invoke (visitor.cljc:26) acc-text-api_1 | spec_tools.visitor$eval22348$fn__22349.invoke (visitor.cljc:143) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:239) acc-text-api_1 | spec_tools.visitor$visit.invokeStatic (visitor.cljc:52) acc-text-api_1 | spec_tools.visitor$visit.invoke (visitor.cljc:26) acc-text-api_1 | spec_tools.swagger.core$transform.invokeStatic (core.cljc:96) acc-text-api_1 | spec_tools.swagger.core$transform.invoke (core.cljc:87) acc-text-api_1 | spec_tools.swagger.core$eval22738$fn__22739.invoke (core.cljc:105) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:234) acc-text-api_1 | spec_tools.swagger.core$eval22788$fn__22789$fn__22791.invoke (core.cljc:142) acc-text-api_1 | clojure.core$map$fn__5866.invoke (core.clj:2755) acc-text-api_1 | clojure.lang.LazySeq.sval (LazySeq.java:42) acc-text-api_1 | clojure.lang.LazySeq.seq (LazySeq.java:51) acc-text-api_1 | clojure.lang.RT.seq (RT.java:535) acc-text-api_1 | clojure.core$seq__5402.invokeStatic (core.clj:137) acc-text-api_1 | clojure.core$apply.invokeStatic (core.clj:660) acc-text-api_1 | clojure.core$mapcat.invokeStatic (core.clj:2783) acc-text-api_1 | clojure.core$mapcat.doInvoke (core.clj:2783) acc-text-api_1 | clojure.lang.RestFn.invoke (RestFn.java:423) acc-text-api_1 | spec_tools.swagger.core$eval22788$fn__22789.invoke (core.cljc:142) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:244) acc-text-api_1 | spec_tools.swagger.core$expand_qualified_keywords$fn__22805$fn__22806.invoke (core.cljc:165) acc-text-api_1 | clojure.core$fn__8429$fn__8431.invoke (core.clj:6840) acc-text-api_1 | clojure.core.protocols$iter_reduce.invokeStatic (protocols.clj:49) acc-text-api_1 | clojure.core.protocols$fn__8140.invokeStatic (protocols.clj:75) acc-text-api_1 | clojure.core.protocols/fn (protocols.clj:75) acc-text-api_1 | clojure.core.protocols$fn__8088$G__8083__8101.invoke (protocols.clj:13) acc-text-api_1 | clojure.core$reduce.invokeStatic (core.clj:6828) acc-text-api_1 | clojure.core$fn__8429.invokeStatic (core.clj:6830) acc-text-api_1 | clojure.core/fn (core.clj:6830) acc-text-api_1 | clojure.core.protocols$fn__8167$G__8162__8176.invoke (protocols.clj:175) acc-text-api_1 | clojure.core$reduce_kv.invokeStatic (core.clj:6856) acc-text-api_1 | clojure.core$reduce_kv.invoke (core.clj:6847) acc-text-api_1 | spec_tools.swagger.core$expand_qualified_keywords$fn__22805.invoke (core.cljc:162) acc-text-api_1 | clojure.walk$walk.invokeStatic (walk.clj:50) acc-text-api_1 | clojure.walk$postwalk.invokeStatic (walk.clj:53) acc-text-api_1 | clojure.walk$postwalk.invoke (walk.clj:53) acc-text-api_1 | spec_tools.swagger.core$expand_qualified_keywords.invokeStatic (core.cljc:159) acc-text-api_1 | spec_tools.swagger.core$expand_qualified_keywords.invoke (core.cljc:157) acc-text-api_1 | spec_tools.swagger.core$swagger_spec.invokeStatic (core.cljc:184) acc-text-api_1 | spec_tools.swagger.core$swagger_spec.invoke (core.cljc:176) acc-text-api_1 | spec_tools.swagger.core$swagger_spec.invokeStatic (core.cljc:182) acc-text-api_1 | spec_tools.swagger.core$swagger_spec.invoke (core.cljc:176) acc-text-api_1 | reitit.coercion.spec$create$reify__22889._get_apidocs (spec.cljc:87) acc-text-api_1 | reitit.coercion$get_apidocs.invokeStatic (coercion.cljc:147) acc-text-api_1 | reitit.coercion$get_apidocs.invoke (coercion.cljc:139) acc-text-api_1 | reitit.swagger$create_swagger_handler$create_swagger__25892$transform_endpoint__25903.invoke (swagger.cljc:93) acc-text-api_1 | clojure.core$keep$fn__8559.invoke (core.clj:7337) acc-text-api_1 | clojure.lang.LazySeq.sval (LazySeq.java:42) acc-text-api_1 | clojure.lang.LazySeq.seq (LazySeq.java:51) acc-text-api_1 | clojure.lang.RT.seq (RT.java:535) acc-text-api_1 | clojure.core$seq__5402.invokeStatic (core.clj:137) acc-text-api_1 | clojure.core/seq (core.clj:137) acc-text-api_1 | reitit.swagger$create_swagger_handler$create_swagger__25892$transform_path__25912.invoke (swagger.cljc:97) acc-text-api_1 | clojure.core$map$fn__5866.invoke (core.clj:2753) acc-text-api_1 | clojure.lang.LazySeq.sval (LazySeq.java:42) acc-text-api_1 | clojure.lang.LazySeq.seq (LazySeq.java:51) acc-text-api_1 | clojure.lang.RT.seq (RT.java:535) acc-text-api_1 | clojure.core$seq__5402.invokeStatic (core.clj:137) acc-text-api_1 | clojure.core$apply.invokeStatic (core.clj:660) acc-text-api_1 | clojure.core$apply.invoke (core.clj:660) acc-text-api_1 | reitit.swagger$create_swagger_handler$create_swagger__25892$map_in_order__25919.invoke (swagger.cljc:99) acc-text-api_1 | reitit.swagger$create_swagger_handler$create_swagger__25892.invoke (swagger.cljc:100) acc-text-api_1 | reitit.ring.middleware.exception$wrap$fn__20547$fn__20548.invoke (exception.clj:49) acc-text-api_1 | muuntaja.middleware$wrap_format_response$fn__27254.invoke (middleware.clj:132) acc-text-api_1 | api.server$wrap_response$fn__29685.invoke (server.clj:43) acc-text-api_1 | ring.middleware.params$wrap_params$fn__27201.invoke (params.clj:67) acc-text-api_1 | muuntaja.middleware$wrap_format_negotiate$fn__27247.invoke (middleware.clj:96) acc-text-api_1 | reitit.ring$ring_handler$fn__20513.invoke (ring.cljc:292) acc-text-api_1 | clojure.lang.AFn.applyToHelper (AFn.java:154) acc-text-api_1 | clojure.lang.AFn.applyTo (AFn.java:144) acc-text-api_1 | clojure.lang.AFunction$1.doInvoke (AFunction.java:31) acc-text-api_1 | clojure.lang.RestFn.invoke (RestFn.java:408) acc-text-api_1 | clojure.lang.Var.invoke (Var.java:384) acc-text-api_1 | org.httpkit.server.HttpHandler.run (RingHandler.java:115) acc-text-api_1 | java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) acc-text-api_1 | java.util.concurrent.FutureTask.run (FutureTask.java:266) acc-text-api_1 | java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) acc-text-api_1 | java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) acc-text-api_1 | java.lang.Thread.run (Thread.java:748)

Thanks !

Numbers in excel files get converted to floats

Domain name is capitalized

Hi there,

Just noticed an issue with full stops:

How should I handle the capitalization issue here?

How would you handle inline HTML formatting?

Hi everyone,

We have a case where we generate a large chunk of text, a few paragraphs, and we need some formatting in HTML format. We currently simply entered the HTML tags/CSS rules inside the text in AccText... :( it really doesn't feel good to do that...

A typical formatted text looks like this on AccText:

And here is a sample output:

You can see the inline formatting in blue/green phrases.

How would you handle this? The only thing that comes to my mind is to use transformers/enrichers (which @zmedelis reminded me in issue #324).

"mkDigits" operation produces numbers that have spaces between them

expected value: "99"
generated text: "9 9"

Support AMRs with different arrity

Let's say I want to generate two sentences with the same meaning (same AMR):

Apple is red
Apple has a red color

This can be implemented in a single HasProperty AMR and it currently works generating good sentences:

If I do not specify the property kind argument, I'd still expect the first "Apple is red" version to be generated. Currently, this does not work.

Data file upload error

I merged our fork with upstream (thanks for the frontend env var updates), but now I keep receiving the error below when I try to upload a sample data file.

Any ideas?


2021-03-26 13:08:28,972 INFO  d.e.data-files - Searching for data file: `books.csv`
--
2021-03-26 13:08:28,975 ERROR a.error - Uri: /accelerated-text-data-files/ Stacktrace:
datomic.impl.Exceptions$IllegalArgumentExceptionInfo: :db.error/not-an-entity Unable to resolve entity: :data.spec.data-file/id
{:entity :data.spec.data-file/id, :db/error :db.error/not-an-entity}
at datomic.error$arg.invokeStatic (error.clj:57)
datomic.error$arg.invoke (error.clj:52)
datomic.db$require_id.invokeStatic (db.clj:589)
datomic.db$require_id.invokePrim (db.clj:-1)
datomic.db$require_attrid.invokeStatic (db.clj:689)
datomic.db$require_attrid.invoke (db.clj:686)
datomic.db$resolve_lookup_ref.invokeStatic (db.clj:534)
datomic.db$resolve_lookup_ref.invoke (db.clj:526)
datomic.db$extended_resolve_id.invokeStatic (db.clj:568)
datomic.db$extended_resolve_id.invoke (db.clj:564)
datomic.db$resolve_id.invokeStatic (db.clj:579)
datomic.db$resolve_id.invoke (db.clj:572)
datomic.pull$pull_STAR_.invokeStatic (pull.clj:351)
datomic.pull$pull_STAR_.invoke (pull.clj:310)
datomic.pull$pull_1.invokeStatic (pull.clj:404)
datomic.pull$pull_1.invoke (pull.clj:401)
clojure.lang.Var.invoke (Var.java:393)
datomic.impl.Circular.pull (Circular.java:22)
datomic.db.Db.pull (db.clj:1819)
datomic.api$pull.invokeStatic (api.clj:162)
datomic.api$pull.invoke (api.clj:160)
data.datomic.entities.data_files$pull_entity.invokeStatic (data_files.clj:14)
data.datomic.entities.data_files$pull_entity.invoke (data_files.clj:13)
data.datomic.impl$eval9970$fn__9971.invoke (impl.clj:44)
clojure.lang.MultiFn.invoke (MultiFn.java:234)
data.datomic.impl$db_access$reify__10050.read_item (impl.clj:94)
data.db$read_BANG_.invokeStatic (db.clj:6)
data.db$read_BANG_.invoke (db.clj:6)
data.entities.data_files$read_data_file.invokeStatic (data_files.clj:38)
data.entities.data_files$read_data_file.invoke (data_files.clj:36)
data.entities.data_files$store_BANG_.invokeStatic (data_files.clj:47)
data.entities.data_files$store_BANG_.invoke (data_files.clj:44)
api.server$fn__31552.invokeStatic (server.clj:95)
api.server/fn (server.clj:93)
reitit.ring.middleware.exception$wrap$fn__22396$fn__22397.invoke (exception.clj:49)
muuntaja.middleware$wrap_format_response$fn__29103.invoke (middleware.clj:132)
api.server$wrap_response$fn__31535.invoke (server.clj:53)
ring.middleware.params$wrap_params$fn__29050.invoke (params.clj:67)
muuntaja.middleware$wrap_format_negotiate$fn__29096.invoke (middleware.clj:96)
reitit.ring$ring_handler$fn__22362.invoke (ring.cljc:292)
clojure.lang.AFn.applyToHelper (AFn.java:154)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.lang.AFunction$1.doInvoke (AFunction.java:31)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.lang.Var.invoke (Var.java:384)
org.httpkit.server.HttpHandler.run (RingHandler.java:115)
java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
java.lang.Thread.run (Thread.java:748)

AccText should not have any demo data

Currently in api folder we have demo AMRs and dictionaries. Those should not be present here. Let's have a separate repository (or multiple) where we store demo doc. plans, AMRs and dictionaries.

Namespacing document plans

We have to have the ability to group document plans (as well as AMRs and DGLs) into namespaces. In order to do that, we can adopt the following naming convention:

namespace: dp-name

Like:

concept-net: Part Of
cooking: Cooking Instruction

Names without namespace will be accepted.

We can not enforce this naming pattern at the UI level. Thus we'd have to follow the agreed convention when naming DPs.

When importing plans from the specified locations, the namespace comes from the document plan's immediate parent. If the direct parent is the root of the configured DP folder, then there is no namespace.

Let us say that Acc Text started with this configuration: DOCUMENT_PLANS: "/acctext/document-plans"

Folder contents (file names equal DP name for simplicity here):

/acctext/document-plans/a.json
/acctext/document-plans/x/a.json
/acctext/document-plans/x/b.json
/acctext/document-plans/y/a.json

We will have DP names like this:

a
x: a
x: b
y: a

Two points to discuss:

Note that inside DP JSON file name element will not have namespace embedded. That is not "x: a", but "a". Though if we do it like this, we run the risk of name collisions.
Issue #238 about DP file naming is related here. If we adopt namespacing, maybe we can have human-readable names?

Create utils docker image

It will be easier to export data on windows - there won't be a need to install clojure for this

No docker image history is kept

When we push docker images on commit to master, only tag used is "latest". We should add another tag with commit hash.

Crash when adding new item to dictionary

Steps to reproduce:
Enter string in entry field and click "add" in Dictionary pane.
Logs:

2020-03-28 07:09:16,015 DEBUG a.n.g.service - ** Incomplete concrete ** acc-text-api_1 | incomplete concrete DefaultBody of Default = open Syntax, DefaultLex, ParadigmsEng in { acc-text-api_1 | lincat acc-text-api_1 | Segment02, DocumentPlan01 = Str ; acc-text-api_1 | lin acc-text-api_1 | Function01 Segment02 = Segment02 ; acc-text-api_1 | Function02 = "" ; acc-text-api_1 | } acc-text-api_1 | acc-text-api_1 | 2020-03-28 07:09:16,016 DEBUG a.n.g.service - ** Concrete ** acc-text-api_1 | concrete DefaultInstance of Default = DefaultBody with acc-text-api_1 | (Syntax=SyntaxEng), acc-text-api_1 | (DefaultLex = DefaultLexEng); acc-text-api_1 | 2020-03-28 07:09:16,017 DEBUG a.n.g.service - ** Lex interface ** acc-text-api_1 | interface DefaultLex = { acc-text-api_1 | } acc-text-api_1 | acc-text-api_1 | 2020-03-28 07:09:16,017 DEBUG a.n.g.service - ** Lex data ** acc-text-api_1 | resource DefaultLexEng = open SyntaxEng, ParadigmsEng, MorphoEng in { acc-text-api_1 | } acc-text-api_1 | acc-text-api_1 | 2020-03-28 07:09:20,431 ERROR a.error - Uri: /_graphql Stacktrace: acc-text-api_1 | java.lang.NullPointerException: null acc-text-api_1 | at clojure.core$name.invokeStatic (core.clj:1595) acc-text-api_1 | clojure.core$name.invoke (core.clj:1589) acc-text-api_1 | api.graphql.translate.dictionary$schema__GT_default_multilang_dict_item.invokeStatic (dictionary.clj:89) acc-text-api_1 | api.graphql.translate.dictionary$schema__GT_default_multilang_dict_item.invoke (dictionary.clj:84) acc-text-api_1 | api.graphql.domain.dictionary$create_dictionary_item.invokeStatic (dictionary.clj:28) acc-text-api_1 | api.graphql.domain.dictionary$create_dictionary_item.invoke (dictionary.clj:27) acc-text-api_1 | clojure.lang.Var.invoke (Var.java:393) acc-text-api_1 | com.walmartlabs.lacinia.schema$wrap_resolver_to_ensure_resolver_result$fn__12518.invoke (schema.clj:612) acc-text-api_1 | com.walmartlabs.lacinia.executor$invoke_resolver_for_field.invokeStatic (executor.clj:137) acc-text-api_1 | com.walmartlabs.lacinia.executor$invoke_resolver_for_field.invoke (executor.clj:115) acc-text-api_1 | com.walmartlabs.lacinia.executor$resolve_and_select.invokeStatic (executor.clj:451) acc-text-api_1 | com.walmartlabs.lacinia.executor$resolve_and_select.invoke (executor.clj:344) acc-text-api_1 | com.walmartlabs.lacinia.executor$eval14396$fn__14397.invoke (executor.clj:230) acc-text-api_1 | clojure.lang.MultiFn.invoke (MultiFn.java:234) acc-text-api_1 | com.walmartlabs.lacinia.executor$combine_selection_results_sync$fn__14424.invoke (executor.clj:322) acc-text-api_1 | com.walmartlabs.lacinia.resolve.ResolverResultImpl.on_deliver_BANG_ (resolve.clj:109) acc-text-api_1 | com.walmartlabs.lacinia.executor$combine_selection_results_sync.invokeStatic (executor.clj:318) acc-text-api_1 | com.walmartlabs.lacinia.executor$combine_selection_results_sync.invoke (executor.clj:313) acc-text-api_1 | com.walmartlabs.lacinia.executor$execute_nested_selections_sync$fn__14431.invoke (executor.clj:340) acc-text-api_1 | clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58) acc-text-api_1 | clojure.core.protocols$fn__8154.invokeStatic (protocols.clj:136) acc-text-api_1 | clojure.core.protocols/fn (protocols.clj:124) acc-text-api_1 | clojure.core.protocols$fn__8114$G__8109__8123.invoke (protocols.clj:19) acc-text-api_1 | clojure.core.protocols$seq_reduce.invokeStatic (protocols.clj:31) acc-text-api_1 | clojure.core.protocols$fn__8146.invokeStatic (protocols.clj:75) acc-text-api_1 | clojure.core.protocols/fn (protocols.clj:75) acc-text-api_1 | clojure.core.protocols$fn__8088$G__8083__8101.invoke (protocols.clj:13) acc-text-api_1 | clojure.core$reduce.invokeStatic (core.clj:6828) acc-text-api_1 | clojure.core$reduce.invoke (core.clj:6810) acc-text-api_1 | com.walmartlabs.lacinia.executor$execute_nested_selections_sync.invokeStatic (executor.clj:340) acc-text-api_1 | com.walmartlabs.lacinia.executor$execute_nested_selections_sync.invoke (executor.clj:331) acc-text-api_1 | com.walmartlabs.lacinia.executor$execute_query.invokeStatic (executor.clj:489) acc-text-api_1 | com.walmartlabs.lacinia.executor$execute_query.invoke (executor.clj:458) acc-text-api_1 | com.walmartlabs.lacinia$execute_parsed_query_async.invokeStatic (lacinia.clj:54) acc-text-api_1 | com.walmartlabs.lacinia$execute_parsed_query_async.invoke (lacinia.clj:29) acc-text-api_1 | com.walmartlabs.lacinia$execute_parsed_query.invokeStatic (lacinia.clj:63) acc-text-api_1 | com.walmartlabs.lacinia$execute_parsed_query.invoke (lacinia.clj:56) acc-text-api_1 | com.walmartlabs.lacinia$execute.invokeStatic (lacinia.clj:121) acc-text-api_1 | com.walmartlabs.lacinia$execute.invoke (lacinia.clj:70) acc-text-api_1 | api.graphql.core$handle.invokeStatic (core.clj:68) acc-text-api_1 | api.graphql.core$handle.invoke (core.clj:66) acc-text-api_1 | api.server$fn__28937.invokeStatic (server.clj:53) acc-text-api_1 | api.server/fn (server.clj:50) acc-text-api_1 | reitit.ring.middleware.exception$wrap$fn__19795$fn__19796.invoke (exception.clj:49) acc-text-api_1 | muuntaja.middleware$wrap_format_response$fn__26502.invoke (middleware.clj:132) acc-text-api_1 | api.server$wrap_response$fn__28933.invoke (server.clj:45) acc-text-api_1 | ring.middleware.params$wrap_params$fn__26449.invoke (params.clj:67) acc-text-api_1 | muuntaja.middleware$wrap_format_negotiate$fn__26495.invoke (middleware.clj:96) acc-text-api_1 | reitit.ring$ring_handler$fn__19761.invoke (ring.cljc:292) acc-text-api_1 | clojure.lang.AFn.applyToHelper (AFn.java:154) acc-text-api_1 | clojure.lang.AFn.applyTo (AFn.java:144) acc-text-api_1 | clojure.lang.AFunction$1.doInvoke (AFunction.java:31) acc-text-api_1 | clojure.lang.RestFn.invoke (RestFn.java:408) acc-text-api_1 | clojure.lang.Var.invoke (Var.java:384) acc-text-api_1 | org.httpkit.server.HttpHandler.run (RingHandler.java:115) acc-text-api_1 | java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) acc-text-api_1 | java.util.concurrent.FutureTask.run (FutureTask.java:266) acc-text-api_1 | java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) acc-text-api_1 | java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) acc-text-api_1 | java.lang.Thread.run (Thread.java:748) acc-text-api_1 |

Prevent duplicate file creation

When files are put in data-files folder and datomic is used as storage, after every restart, duplicate files get created.

Review and improve current eval

Change from BLEU to ROUGE
Evaluation calculation is inaccurate
Possibly change it to integration test

Increase LOG verbosity

If user launches accelerate-text manually and forgets one of the services, eg. gf

we get error: 2021-02-10 09:22:39,077 ERROR a.n.core - Connection refused

which is not clear at all. It should be clear what is missing

Sequences do not work correctly when one or more AMRs attached to them returns an empty text

Commas along with 'and' are missing

"TypeError: failed to fetch resource" when uploading data file

Unable to change environment variables after front-end docker image is built

Need to rebuild the image to make it work it seems. This prevents changing ports in other services. Maybe front-end should communicate inside docker network by default?

Add part of speech tag to dictionary items

Some supplied dictionary items are marked as verbs (VV) etc. It does not seem to be possible to add these tags to newly created items.

Docker install fails to run on Ubuntu 20.04

Just tried to install on Ubuntu 20.04 using
git clone https://github.com/tokenmill/accelerated-text.git
and then
make run-app

Am greeted with this page:

Console output:

$ make run-app
docker-compose -p dev -f docker-compose.yml -f docker-compose.front-end.yml down && \
docker-compose -p dev -f docker-compose.yml -f docker-compose.front-end.yml up --remove-orphans
Removing dev_acc-text-api_1       ... done
Removing dev_datomic-transactor_1 ... done
Removing dev_gf_1                 ... done
Removing dev_front-end_1          ... done
Removing network dev_default
Removing network dev_datomic
Creating network "dev_default" with the default driver
Creating network "dev_datomic" with the default driver
Creating dev_gf_1                 ... done
Creating dev_front-end_1          ... done
Creating dev_datomic-transactor_1 ... done
Creating dev_acc-text-api_1       ... done
Attaching to dev_front-end_1, dev_datomic-transactor_1, dev_gf_1, dev_acc-text-api_1
datomic-transactor_1  | Recursively changing ownership of /srv/datomic/data to datomic:datomic.
datomic-transactor_1  | Launching with Java options -server -Xms3968m -Xmx3968m -XX:+UseG1GC -XX:MaxGCPauseMillis=50
front-end_1           | (nginx -g "daemon on;" && npm start)
front-end_1           |
front-end_1           | > [email protected] start /usr/src/app
front-end_1           | > node -r esm node_modules/.bin/webpack-dev-server --host 0.0.0.0 --config packages/webpack/run.config.js
front-end_1           |
gf_1                  | [2021-04-14 08:45:26 +0000] [1] [INFO] Starting gunicorn 19.10.0
gf_1                  | [2021-04-14 08:45:26 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
gf_1                  | [2021-04-14 08:45:26 +0000] [1] [INFO] Using worker: sync
gf_1                  | [2021-04-14 08:45:26 +0000] [9] [INFO] Booting worker with pid: 9
front-end_1           | ℹ ｢wds｣: Project is running at http://0.0.0.0:8080/
front-end_1           | ℹ ｢wds｣: webpack output is served from /
front-end_1           | ℹ ｢wds｣: Content not from webpack is served from /usr/src/app/dist, /usr/src/app/assets
front-end_1           | ℹ ｢wds｣: 404s will fallback to /index.html
datomic-transactor_1  | Starting datomic:free://datomic-transactor:4334/<DB-NAME>, storing data in: data ...
datomic-transactor_1  | System started datomic:free://datomic-transactor:4334/<DB-NAME>, storing data in: data
front-end_1           | ℹ ｢wdm｣: Hash: 4f70cede9e544df5520e
front-end_1           | Version: webpack 4.28.3
front-end_1           | Time: 17651ms
front-end_1           | Built at: 04/14/2021 8:45:51 AM
front-end_1           |              Asset       Size  Chunks             Chunk Names
front-end_1           | assets/favicon.png   5.24 KiB          [emitted]
front-end_1           |         index.html  236 bytes          [emitted]
front-end_1           |     main.bundle.js   8.46 MiB    main  [emitted]  main
front-end_1           | Entrypoint main = main.bundle.js
front-end_1           | [0] multi (webpack)-dev-server/client?http://0.0.0.0:8080 (webpack)/hot/dev-server.js ./packages/app/start-in-browser.js 52 bytes {main} [built]
front-end_1           | [./node_modules/preact/debug/dist/debug.module.js] 19.4 KiB {main} [built]
front-end_1           | [./node_modules/preact/dist/preact.module.js] 8.94 KiB {main} [built]
front-end_1           | [./node_modules/strip-ansi/index.js] 161 bytes {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/index.js?http://0.0.0.0:8080] (webpack)-dev-server/client?http://0.0.0.0:8080 4.29 KiB {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/overlay.js] (webpack)-dev-server/client/overlay.js 3.51 KiB {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/socket.js] (webpack)-dev-server/client/socket.js 1.53 KiB {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/utils/createSocketUrl.js] (webpack)-dev-server/client/utils/createSocketUrl.js 2.91 KiB {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/utils/log.js] (webpack)-dev-server/client/utils/log.js 964 bytes {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/utils/reloadApp.js] (webpack)-dev-server/client/utils/reloadApp.js 1.59 KiB {main} [built]
front-end_1           | [./node_modules/webpack-dev-server/client/utils/sendMessage.js] (webpack)-dev-server/client/utils/sendMessage.js 402 bytes {main} [built]
front-end_1           | [./node_modules/webpack/hot sync ^\.\/log$] (webpack)/hot sync nonrecursive ^\.\/log$ 170 bytes {main} [built]
front-end_1           | [./node_modules/webpack/hot/dev-server.js] (webpack)/hot/dev-server.js 1.61 KiB {main} [built]
front-end_1           | [./node_modules/webpack/hot/emitter.js] (webpack)/hot/emitter.js 75 bytes {main} [built]
front-end_1           | [./packages/app/start-in-browser.js] 203 bytes {main} [built]
front-end_1           |     + 1052 hidden modules
front-end_1           | Child html-webpack-plugin for "index.html":
front-end_1           |      1 asset
front-end_1           |     Entrypoint undefined = index.html
front-end_1           |     [./node_modules/html-webpack-plugin/lib/loader.js!./node_modules/html-webpack-plugin/default_index.ejs] 376 bytes {0} [built]
front-end_1           |     [./node_modules/lodash/lodash.js] 530 KiB {0} [built]
front-end_1           |     [./node_modules/webpack/buildin/global.js] (webpack)/buildin/global.js 472 bytes {0} [built]
front-end_1           |     [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {0} [built]
front-end_1           | ℹ ｢wdm｣: Compiled successfully.
acc-text-api_1        | WARNING: requiring-resolve already refers to: #'clojure.core/requiring-resolve in namespace: datomic.common, being replaced by: #'datomic.common/requiring-resolve
acc-text-api_1        | 2021-04-14 08:46:11,319 INFO  d.d.utils - Applying Datomic migration: 2019-10-28-initial-schema.edn
acc-text-api_1        | 2021-04-14 08:46:11,782 INFO  a.server - Running server on: localhost:3001. Press Ctrl+C to stop

Here's the output from the browser console:

GF parse result conversion to AccText data

From

PhrUtt NoPConj (UttS (UseCl (TTAnt TPres ASimul) PPos (PredVP (DetCN (DetQuant DefArt NumSg) (UseN wine_N)) (AdvVP (UseComp (CompCN (UseN shop_N))) (PrepNP in_Prep (DetCN (DetQuant DefArt NumSg) (UseN city_N))))))) NoVoc

Needs to be converted to be converted to DLG functions. AccText represents this via Semantic Graph, therefore the above will be converted to AccText semantic graph.

Add structural words to document plan editor

Can not toggle between "Most relevant" and "Top 20" data rows

Try to toggle between those two data row selections. After clicking on "Top 20" selection jumps back to "Most relevant".
Error indication icon at the top-right changes to red.