garfieldnate/kengdic

Joe Speigle's Korean/English dictionary database

Kengdic

Kengdic is a large, open Korean/English dictionary database created by Joe Speigle. It was originally hosted by Joe at ezcorean.com.

The bulk of the usable data is in kengdic_2011.tsv. The raw directory contains data which still needs to be examined. The scripts directory contains some automated QA checks for the data; these are run whenever a change is made in this repository.

Contributors

The dictionary data is still quite dirty, and contributions are very welcome. Some ways you can help:

• Fix some of the issues which have been automatically flagged. The list of these issues is available on the repository website: http://garfieldnate.github.io/kengdic/.

• Check existing entries for correctness (흰색 means white, not gray, "the m prophets", lowercase "british", etc.)

• Add new entries

• Assess the current coverage. Are we missing any particularly basic words, or words related to any specific subject?

• Fix grammatical, spelling or formatting issues

• Help come up with editorial and style guidelines.

• Come up with new automatic checks we can run on the dictionary to find possible issues (see scripts/lint.py).

By contributing data, you release it under the same license terms as Kengdic itself (see below).

Example Datapackage Usage

We've provided a datapackage.json for convenience. To retrieve and load the data in python:

• pip install datapackage

    $ python
    >>> from datapackage import Package
    >>> package = Package('https://raw.githubusercontent.com/garfieldnate/kengdic/master/datapackage.json')
    >>> resource = package.get_resource('kengdic')
    >>> data = resource.read(keyed=True)
    >>> data[20977]
    {'id': 20978, 'surface': '급조', 'hanja': '急造', 'gloss': None, 'level': None, 'created': datetime.datetime(2009, 1, 1, 20, 23, 14), 'source': 'mr.hanja-213889@ezcorean:213889'}

License

The Kengdic data is released under dual licenses: users may choose to use MPL 2.0 or the LGPL, version 2.0 or later.

kengdic's Issues

Generate definitions

Try generating definitions, and possibly check for and flag low similarity with other known-good definitions.

Unify data

There are 3 separate dictionary files in the repo. Figure out which one is the master file and delete the others, or combine the data if appropriate.

clean up fields

Some of the fields do not seem to contain useful data. Clean them up or remove them.

Come up with editorial guidelines

The data needs a lot of cleaning. Since there are 130K entries, it probably won't be possible to do it alone, and in order for the result to be consistent we're going to need to come up with editorial guidelines of some kind.

hanja field should be an array type in datapackage.json

May in fact be better to generalize the concept to simplify alternative spellings.

Working notes:

hanja field should be an array:
- 淵魚,鰱魚
- 俛,袝
- 남동생 男同生 can also be written 男동생
- 청사초롱 靑紗초籠 can also have 燭 I feel like the 초 is 灯; but it's not kyoyuk
- 어음교환조합은행 can appear with spaces: 어음교환 조합은행 or 어음 교환 조합 은행, etc.
- 最정상급 for 최정상급 might get used instead of 最頂上級
- should 어음 交換 be also 語音 交換? Online they seem to use hangeul everywhere, but the etymology seems sensical...

random issues

missing:
- 결합사
- 訟事辮

nth

,

=

@

Japanese vinegared rice vegetables = sushi?

?

\t)

the home of ..

should 어음 交換 be also 語音 交換? Online they seem to use hangeul everywhere, but the etymology seems sensical...

~

Unalignable hanja and surface fields

These are fairly straightforward to fix. See the list of issues at http://garfieldnate.github.io/kengdic/hanja%20length.txt.

determine future structure

Currently we have 7 fields; some information is missing (part of speech, morphology) and some information is contained in the gloss field (regional variants, multiple senses). Come up with a structure that we can use moving forward that will make the dictionary a bit more comprehensive and usable. Start by reviewing Jim Breen's work.