Comments (5)
Yes, this is a somewhat annoying thing about hacking on duckling.
from duckling.
I've been thinking about how we could approach this issue. My idea is to add a rule to each language that splits single words so we can reuse the rules that already exist.
Assume that we have Language X
and it has rules for hello
, world
and intersection
, so it correctly parses hello
(with hello rule
), world
(with world
rule), and hello world
, world hello
, hello hello
, world world
(with intersection
rule).
What I would like to do is add a single-word decomposition
rule that will handle helloworld
, hellohello
, worldhello
and worldworld
. The idea is that this rule will only split the word into subwords which then will be picked up by the intersection
rule.
For example, rule single-word decomposition
converts helloworld
into hello world
. Then, rule intersection
picks hello world
and works as expected.
The single-word decompositon
rule would need to know in advance all possible subwords so then it can separate them. A rough implementation would be something like:
ruleSingleWordDecomposition :: Rule
ruleSingleWordDecomposition = Rule
{ name = "single-word decomposition"
, pattern =
[ regex "(\\S+)" -- we match 'single' words
]
, prod = \tokens -> case tokens of
(Token RegexMatch (GroupMatch (w:_)):_) -> do
let rgx = "(hello|world)" :: String -- we need to compute in advance a big regex of possible "subwords"
let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)
Just $ Token RegexMatch $ GroupMatch ws -- How do we return the "subwords" so other rules can pick them up?
_ -> Nothing
}
Note: we need to split the "single word" in Haskell (let ws = Text.pack <$> getAllTextMatches (show w =~ rgx)
) since we can't do it with pure regexes AFAIK
In this case, our single-word decomposition
rule knows that "single words" can be composed only from the "subwords" hello
and world
.
The main issue that I found with this is how to use the output of this rule as input for the remaining rules (they don't pick it up). So, even if we're able to build the required regex and split the "single word" into "subwords" I can't seem to redirect them to other rules.
I'm trying to avoid adding special cases or hacking into the internals of Duckling (Engine.hs
and the like), so any hint would be appreciated.
from duckling.
One quick note, the mentioned examples in the first comment aren't permalinks so I can't be sure to what they refer to. I understand that this issue is related to:
from duckling.
@chessai Do you think that this could work?
from duckling.
Regarding @emlautarom1's interesting suggestion, I see two problems that should be considered before going this route.
First, decomposing German number words like "dreiundzwanzig" 'three.and.twenty' (23) can lead to unnwanted ambiguities because "drei und zwanzig" 'three and twenty' is also a grammatical phrase in German. Consider, for instance, the sentence "sie wählten zwischen dreiundzwanzig Kandidaten" 'they made a selection between twenty three candidates'. The problem is that the sentence "sie wählten zwischen drei und zwanzig Kandidaten" 'they made a selection between three candidates and twenty candidates'. Hence, the suggested decomposition makes the two sentences indistinguishable.
Similarly, if single-word decomposition
is applied the compound "zweihundertjährige" 'two.hundred.year.old.PL' and the phrase "zwei hundertjährige" 'two hundred.year.old.PL' are converted to the same string. Thus, the difference between sentences like "sie fällten zweihundertjährige Eichen" 'they chopped oak trees that were two hundred years old' and "sie fällten zwei hundertjährige Eichen" 'they chopped two oak trees that were one hundred years old' is erased.
So the upshot is that we lose information by applying single-word decomposition
.
from duckling.
Related Issues (20)
- Issue in capturing two date fields in a sentence HOT 1
- Support docker container for Mac M1 HOT 3
- smallest possible range/body
- stack not found when running on M1 HOT 6
- Empty reply from server
- Empty reply from server
- [german][time] dayofMonth monthAsNumber at timeOfDay not detecting
- Alpine image is not working
- [CS] Czech language support HOT 1
- [HU] Hungary language Support
- Does duckling has support for sentence containing entities with negation
- Have plan to support Cantonese? HOT 1
- Support Persian(fa) Languages HOT 4
- [Persian][Numeral] Extract incorrect number from text
- ghc: panic! when stack build.
- Error building on MacOS with M1 Pro HOT 4
- Can't build Docker image HOT 2
- [it_IT] add time corpus/rules
- Default log files are created even if different paths are provided via options
- [ERROR] Network.Socket.sendBuf: resource vanished HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from duckling.