google / budoux Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://google.github.io/budoux/
License: Apache License 2.0
Home Page: https://google.github.io/budoux/
License: Apache License 2.0
A JavaScript unit test case is failing at https://github.com/google/budoux/actions/runs/7726982481/job/21064428934
HTMLProcessor.getBlocks should skip <rt> and <rp> elements for <ruby> FAILED
Since the same test case passes on Chrome Headless M120, but not on M121, I believe the change is brought by the Chrome update.
Hello, I'm porting budoux to zig (and C).
I wonder what exactly is the unicode format of the keys in the model. Are they normalized?
When constructing a key should the key be constructed from range of codepoints or graphemes?
I'm asking because currently all the tests pass expect for the thai model and this might be the issue I'm hitting.
Consider to add the --lang
option to the JavaScript CLI tool similar to #55
input: ここから先はチケットを購入されたお客様のみ入ることができます。
actual: ここから/先は/チケットを/購入された/お客様の/み入る/ことができます。
expected: ここから/先は/チケットを/購入された/お客様のみ/入ることが/できます。
input: 基本ギアパワーのみ有効だ
actual: 基本ギアパワーの/み有効だ
expected: 基本ギアパワーのみ/有効だ
HTMLProcessor.getText()
calls:
return Jsoup.parseBodyFragment(html).text();
It looks like this collapses whitespaces. Example:
String html = " H e ";
String result = HTMLProcessor.getText(html);
The result
becomes H e
, collapsing consecutive spaces into one space, and leading and trailing spaces to none.
/cc @tushuhei
Input: あなたの意図したとおりに情報を伝えることができます。
Actual:
あなたの/意図したと/おりに/情報を/伝える/ことができます。
Expected:
あなたの/意図したとおりに/情報を/伝える/ことができます。
あなたの/意図した/とおりに/情報を/伝える/ことが/できます。
Found in Blink web_tests.
Dart is not supported yet.
Input: <img>abcdef
Expected: <img>abcdef
Actual: <img></img>abcdef
Unlike Python HTMLParser
, Java version uses Jsoup.parseBodyFragment
, which supports HTML parsing algorithm, so we don't have to worry about issues like #355.
But the serialization code should be aware of self-closing tags.
Input: いよいよはじまる
Expected: いよいよ/はじまる
Actual: いよいよは/じまる
With budoux 0.6.0
budoux --html "foo</p>"
Traceback (most recent call last):
File "/home/johnc/.local/bin/budoux", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/johnc/.local/pipx/venvs/budoux/lib/python3.11/site-packages/budoux/main.py", line 187, in main
print(_main(test))
^^^^^^^^^^^
File "/home/johnc/.local/pipx/venvs/budoux/lib/python3.11/site-packages/budoux/main.py", line 171, in _main
res = parser.translate_html_string(inputs_html)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/johnc/.local/pipx/venvs/budoux/lib/python3.11/site-packages/budoux/parser.py", line 102, in translate_html_string
return resolve(chunks, html)
^^^^^^^^^^^^^^^^^^^^^
File "/home/johnc/.local/pipx/venvs/budoux/lib/python3.11/site-packages/budoux/html_processor.py", line 124, in resolve
resolver.feed(html)
File "/home/johnc/.pyenv/versions/3.11.3/lib/python3.11/html/parser.py", line 110, in feed
self.goahead(0)
File "/home/johnc/.pyenv/versions/3.11.3/lib/python3.11/html/parser.py", line 172, in goahead
k = self.parse_endtag(i)
^^^^^^^^^^^^^^^^^^^^
File "/home/johnc/.pyenv/versions/3.11.3/lib/python3.11/html/parser.py", line 413, in parse_endtag
self.handle_endtag(elem)
File "/home/johnc/.local/pipx/venvs/budoux/lib/python3.11/site-packages/budoux/html_processor.py", line 84, in handle_endtag
self.to_skip = self.element_stack.get_nowait()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/johnc/.pyenv/versions/3.11.3/lib/python3.11/queue.py", line 199, in get_nowait
return self.get(block=False)
^^^^^^^^^^^^^^^^^^^^^
File "/home/johnc/.pyenv/versions/3.11.3/lib/python3.11/queue.py", line 168, in get
raise Empty
_queue.Empty
Hi there,
First thanks for the lib, it's impressive the results from such a small footprint😄
The results were not exactly what I wanted for japanese tokenization, so I decided to train my own model and it was quite simple and straightforward. Sadly after importing the generated model in javascript it doesn't work.
import { Parser, loadDefaultJapaneseParser } from 'budoux'
import model from './mymodel.json'
// obviously the following works
const parser = loadDefaultJapaneseParser()
console.log(parser.parse('今日は天気です。'))
// but this doesn't
const parser = new Parser(model)
console.log(parser.parse('今日は天気です。'))
Uncaught TypeError: this.model.values is not a function or its return value is not iterable
at Parser.parse (parser.js:120:47)
As shown in https://github.com/google/budoux/runs/6555048469, our "Style Check" GitHub Action won't be interrupted even if mypy
finds an error. Any mypy
error should pause the process to flag contributors about its type error.
Run sasanquaneuf/mypy-github-action@a0c442aa252655d7736ce6696e06227ccdd62870
/opt/hostedtoolcache/Python/3.9.12/x64/bin/mypy .
build/lib/budoux/__init__.py: error: Duplicate module named "budoux" (also at
"./budoux/__init__.py")
build/lib/budoux/__init__.py: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules for more info
build/lib/budoux/__init__.py: note: Common resolutions include: a) using `--exclude` to avoid checking one of them, b) adding `__init__.py` somewhere, c) using `--explicit-package-bases` or adjusting MYPYPATH
Found 1 error in 1 file (errors prevented further checking)
Have you considered directly building a DomFragment
instead of returning strings from your parser ?
This would allow you to avoid unnecessary serialization/parsing/sanitization steps.
Something like this:
// in dom.ts
function parseFromString(html: string): Node {
const template = document.createElement('template');
template.innerHTML = html;
return template.content;
}
// parser.ts
function translateHTMLString(html: string): Node {
if (html === '') return new DocumentFragment();
const fragment = parseFromString(html);
if (Parser.hasChildTextNode(fragment)) {
const wrapper = document.createElement('span');
wrapper.append(...fragment.childNodes);
fragment.appendChild(wrapper);
}
this.applyElement(fragment.firstChild as HTMLElement);
return fragment;
}
// in budoux-base.ts
sync() {
const translated = this.parser.translateHTMLString(this.innerHTML);
this.shadow.textContent = '';
this.shadow.appendChild(translated);
}
You can even avoid having to parse anything at all if you clone the existing nodes instead of grabbing this.innerHTML
.
// in budoux-base.ts
sync() {
let translated: HTMLElement;
if (Parser.hasChildTextNode(this)) {
translated = document.createElement('span');
translated.append(...this.childNodes.map(node => node.cloneNode(true)));
} else {
translated = this.firstElementChild!.cloneNode(true) as HTMLElement;
}
this.parser.applyElement(translated);
this.shadow.textContent = '';
this.shadow.appendChild(translated);
}
This is also likely to be more performant than parsing and serializing the tree multiple times.
Originally posted by @engelsdamien in google/safevalues#256 (comment)
To align with #80, we need to amend the JavaScript port to remove the P features.
I've noticed a few examples where "禁則処理" is not working properly in BudouX, although they are rare. I suspect this might be due to it not being included in the training data and so on.
I used the latest main branch https://github.com/google/budoux/tree/cb21dadb92fce3ee21157457539e061d9f04d99a
Input:
Adobe Illustratorとは?デザイン・レイアウトの決定版
Actual:
Adobe Illustratorとは/?デザイン・レイアウトの/決定版
Expected:
Adobe Illustratorとは?/デザイン・レイアウトの/決定版
Input:
『バクマン。』に影響を受け、漫画家を目指す
Actual:
『バクマン。/』に/影響を/受け、/漫画家を/目指す
Expected:
『バクマン。』に/影響を/受け、/漫画家を/目指す
Input:
[動画で見る]魚眼レンズで撮影したような動画を作成する方法
Actual:
[動画で/見る/]魚眼レンズで/撮影したような/動画を/作成する/方法
Expected:
[動画で/見る]/魚眼レンズで/撮影したような/動画を/作成する/方法
Input:
電子サインの法的な効力は?
Actual:
電子サインの/法的な/効力は/?
Expected:
電子サインの/法的な/効力は?
Context: #33
Support --thres
in the JavaScript CLI as the Python version does.
Input: ありがとうございます。
Expected: ありがとう/ございます。
Actual: ありが/とうございます。
Found in griffin-stewie/BudouX.swift#22
When a ZWSP or WBR element appears at the end of a line in source HTML, the space that should be introduced by the line break may be removed. The behavior may vary by browser. Possible solution from the BudouX side is not to insert a separator right before \n
.
Demo: https://codepen.io/tushuhei/pen/GRbraYN
HTML:
<p>
これは
<b>テスト</b>
です。
</p>
<p class="zwsp" style="word-break: keep-all; overflow-wrap: anywhere;">
これは​
<b>テスト</b>
です。​
</p>
<p class="wbr" style="word-break: keep-all; overflow-wrap: anywhere;">
これは<wbr>
<b>テスト</b>
です。<wbr>
</p>
single quotes in data attributes get converted into double quotes and that breaks when there is a double quote in the data attribute.
<div data-number-list='["1","2","3"]'>
becomes <div data-number-list="["1","2","3"]">
Demo
https://codepen.io/tushuhei/pen/VwqMywj
Setup
<p>xyz<wbr>abcabc</p>
const parser = new HTMLProcessingParser({
UW4: {a: 1001}, // means "should separate right before 'a'".
});
const paragraph = document.querySelector('p');
parser.applyElement(paragraph);
console.log(paragraph.innerHTML);
Expected
<p>xyz<wbr>abc<wbr>abc</p>
Actual
<p>xyz<wbr><wbr>abc<wbr>abc</p>
We may want to remove duplicated separators in case we need to apply BudouX to the same element multiple times (e.g. Web Components that reuse their Light DOM).
@kojiishi Could you take a look what changes should be applied to html_processor.ts
?
Input:
あのイーハトーヴォのすきとおった風、夏でも底に冷たさをもつ青いそら、うつくしい森で飾られたモリーオ市、郊外のぎらぎらひかる草の波。
Actual:
あの/イーハトーヴォの/すきと/おった/風、/夏でも/底に/冷たさを/もつ青い/そら、/うつくしい/森で/飾られた/モリーオ市、/郊外の/ぎらぎら/ひかる/草の/波。
Expected:
あの/イーハトーヴォの/すきとおった/風、/夏でも/底に/冷たさを/もつ/青い/そら、/うつくしい/森で/飾られた/モリーオ市、/郊外の/ぎらぎら/ひかる/草の/波。
Add an option to control the train/test data split ratio. The value needs to be between 0 and 1.
When BudouX is used in a browser context, it throws assertion error in console.
https://jsfiddle.net/wfg4m8a6/
BudouX may consume HTML elements whose styles are not computed yet, which causes assertion failure apparently at
budoux/javascript/src/html_processor.ts
Line 112 in 43b3569
@kojiishi Could you look into this issue?
If display: flex;
is applied to the parent element, the overflow-wrap: break-word;
becomes ineffective, resulting in text overflow.
overflow-wrap: anywhere;
resolves the issue. It's supported on modern web browsers.
https://developer.mozilla.org/en-US/docs/Web/CSS/overflow-wrap
The primary difference between overflow-wrap: anywhere;
and overflow-wrap: break-word
is reflected in the way soft wrap opportunities introduced by word-break are handled when calculating min-content intrinsic sizes.
https://developer.mozilla.org/en-US/docs/Web/CSS/overflow-wrap
anywhere
To prevent overflow, an otherwise unbreakable string of characters — like a long word or URL — may be broken at any point if there are no otherwise-acceptable break points in the line. No hyphenation character is inserted at the break point. Soft wrap opportunities introduced by the word break are considered when calculating min-content intrinsic sizes.
break-word
The same as the anywhere value, with normally unbreakable words allowed to be broken at arbitrary points if there are no otherwise acceptable break points in the line, but soft wrap opportunities introduced by the word break are NOT considered when calculating min-content intrinsic sizes.
Go to the example https://codepen.io/tamanyan/pen/KKyyxMj
<div class="flex">
<div>
<h3>5. word-break: keep-all; + overflow-wrap: break-word; + <wbr> + flex</h3>
<p class="keep-all-break-word box">
グレートブリテン<wbr>および<wbr>北アイルランド連合王国という<wbr>言葉は<wbr>本当に<wbr>長い言葉<wbr>ですね
</p>
</div>
</div>
.flex {
display: flex;
}
.keep-all-break-word {
word-break: keep-all;
overflow-wrap: break-word;
}
.box {
padding: 10px;
border: 1px solid;
font-size: 40px;
}
Use overflow-wrap: anywhere; instead of overflow-wrap: break-word;
I have a blog site especially focused on interviews and budoux
is a great tool for its readablitiy, but I have a question whether or not it has any negative impacts on SEO when appying it on each article bodies like this:
because the text is divided into many parts by span
tags. Maybe this is a bit out of scope for budoux development, but users might face this issue when using it so I posted it here. I would like to hear your opinions.
Thanks.
npm test
results in:
> [email protected] test
> ts-node node_modules/jasmine/bin/jasmine tests/*.ts
src/parser.ts:20:36 - error TS2307: Cannot find module './data/models/zh-hans' or its corresponding type declarations.
20 import {model as zhHansModel} from './data/models/zh-hans';
~~~~~~~~~~~~~~~~~~~~~~~
Demo: https://codepen.io/tushuhei/pen/QWzpMQK
The current implementation does not propagate global styles in the custom element because of the shadow DOM.
We may want to consider outputting the translated HTML to Light DOM instead to let users control the styles of child elements from outside of the custom element.
The module is not useable in web worker without significant patching. Imo, this can be solved either:
input: お気軽にお問い合わせください。
actual: お気軽にお問い/合わせください。
expected: お気軽に/お問い合わせ/ください。
can i set parameter to let budoux output fixed line
By setting <meta name="format-detection" content="telephone=no">
, numbers such as phone numbers become text on iOS.
However, the "telephone=no"
setting does not work for text in <budoux>
, and numbers such as phone numbers become <a href="tel:">
.
Is this an iOS issue or a budoux issue?
Sample:
https://codepen.io/rhsk/full/RwEVLBa
<h2>normal</h2>
<p>090-1234-5678</p>
<p>テキストテキストテキストテキストテキスト 090-1234-5678 テキストテキストテキストテキスト</p>
<h2>budoux</h2>
<p><budoux-ja>090-1234-5678</budoux-ja></p>
<p><budoux-ja>テキストテキストテキストテキストテキスト 090-1234-5678 テキストテキストテキストテキスト</budoux-ja></p>
Input: 要点をまとめる必要がある。
Expected: 要点を/まとめる/必要が/ある。
Actual: 要点を/まと/める/必要が/ある。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.