Posts Tagged ‘PLS’

« Older Entries

Advantages of Ralf’s French dictionary

Wednesday, April 7th, 2010

At the moment, I am thinking about the advantages that Ralf's French dictionary has to offer. Let me show you the advantages:

french-advantages

1. The dictionary is stored as XML file. This means that you can edit the dictionary with gedit. Or you can transform the dictionary with XSLT.

2. The encoding is UTF-8. With this encoding there shouldn’t be encoding problems:

2.a. UTF-8 means no encoding problems within the <grapheme> element: Hebrew or Tamil are no problem. And this means that French accents are displayed correctly inside the <grapheme> element. No crap characters should occur.

2.b. UTF-8 means that the IPA phonemes are displayed correctly inside the <phoneme> elements. The dictionary can easily be edited by human editors. It is difficult to edit a phonetic dictionary that contains X-SAMPA or Kirshenbaum characters. IPA phonemes can easily be read by humans. And it is no problem to process IPA characters with saxonb-xslt (type saxonb-xslt into the Ubuntu terminal). Every detail of the French language can be catched.

3. The license of Ralf's French dictionary is GPLv3. Maybe the simon developers are interested to offer an automatic dictionary import? The license would permit this. You can see that the dictionary design is very developer-friendly.

4. The <lexicon> element contains the attribute alphabet="ipa". Not all of my dictionaries contain IPA characters. Some dictionaries contain eSpeak charakters; these dictionaries contain the attribute alphabet="espeak". The PLS standard allows us to use different alphabets for the <phoneme> elements. Personally, I prefer the IPA alphabet. But of course, other alphabets could be used. A future version of simon could differentiate between the different alphabets during dictionary import. I am thinking about the following solution:
- alphabet="ipa" is used by Ralf's German dictionary, Ralf's French dictionary, Ralf's Spanish dictionary;
- alphabet="espeak" can be used by other dictionaries (or alternatively, I transform the eSpeak phonemes into IPA phonemes). I am not sure whether it is good to use eSpeak phonemes inside some of my PLS dictionaries. Maybe it would be better to convert them into IPA phonemes?

Currently, the simon dictionary import process doesn’t differentiate between the tags alphabet="ipa" and alphabet="espeak". As far as I know, the eSpeak phonemes are probably almost the same for all languages. So maybe it would be a good idea if simon would be able to import PLS dictionaries with eSpeak phonemes. I am saying maybe because I am not sure whether this would be a good decision. In the long run, the IPA is the better decision (because it is easily editable by humans).

You can see that I spent a lot of time thinking about the different phoneme characters. It is great to see that simon handles almost all German phonemes. At the moment, there are import problems with the French IPA phonemes ɔ̃ — ɛ̃ — ɑ̃ — ɥ. The tilde is imported by simon as the SAMPA sequence n a s. This is wrong, and should be corrected.

5. Each of my dictionaries contains a language tag. Ralf's German dictionary contains xml:lang="de-DE" because it is Standard German. Ralf's French dictionary indicates Standard French by using the language tag xml:lang="fr". It would be possible to develop a phonetic dictionary for Canadian French. In this case, the tag would be xml:lang="fr-CA".

It is possible that a future version of simon automatically “understands” the language of the specific PLS dictionary. E.g. Ralf's Austrian German dictionary contains the language tag xml:lang="de-AT". Maybe it would be possible to ask the simon user via a wizard:

Which language do you want to use for dictation? Please select the appropriate language.

◊ Standard German (BOMP)
◊ Standard German (PLS)
◊ Standard German (PLS) + Austrian German (PLS)
◊ Only Austrian German (PLS)
◊ Standard French (PLS)
◊ Standard German (PLS) + Medical German (PLS)
◊ [Afrikaans, Catalan, Croatian, ...]

Thanks to the language tag xml:lang="fr" (each language contains a specific language tag) it should be not too difficult to develop an automatic dictionary import function for the 27 PLS dictionaries.

simon offers automatic import of the German BOMP dictionary (obviously, this dictionary does have a very good quality). But what about other languages? It would be good for the marketing if simon offered automatic dictionary import for all 27 PLS dictionaries. Almost all of my dictionaries are in an early stage of development. But this is no problem at the moment. Each dictionary can be improved easily. It is just necessary to run an XSLT style-sheet that transforms the eSpeak phonemes into IPA phonemes. No big deal. I did this for German, for French, and for a couple of other languages.

How many phonemes are needed? I would say that we don’t need all IPA phonemes. We can stick to the existing ones plus 4 French phonemes plus 4 Spanish phonemes. That should do the job for all 27 languages. My goal is to help making simon usable for 27 languages. Usable means that all major characteristics (=phonemes) of each specific language are covered by the specific phonetic dictionary.

My personal focus are the following dictionaries: German, French, Spanish, English (maybe I will transform the English Voxforge dictionary into PLS format). The other languages (Afrikaans, Catalan, Croatian, …) will have to use the phonemes that are used by these four languages. I don’t plan to add more phonemes to my PLS dictionaries. I want to keep it as simple as possible, and as complex as necessary.

6. I am experimenting with the role attribute in Ralf's French dictionary. I tried the following role attributes: lettre and abrévation. simon displays the terminal abrévation correctly:

abreviation

This means that French terminals (= value of the role attribute) can use an apostrophe.

Conclusion: Ralf's French dictionary is user-friendly and developer-friendly. I propose that the simon developers do the following two things:
- add the 4 French phonemes ɔ̃ — ɛ̃ — ɑ̃ — ɥ to the simon import process;
- offer automatic PLS dictionary import for 27 languages (simon should download the specific dictionary directly from the internet, and import it automatically after the download has finished).

Tags: , ,
Posted in Ubuntu, dictionary | Comments Off

Ralf’s French dictionary 0.1.2 released

Tuesday, April 6th, 2010

A few minutes ago, I uploaded Ralf's French dictionary version 0.1.2 (license: GPLv3). Download the dictionary, and import it into simon as PLS dictionary. I applied the following changes to the dictionary:

1. I added an empty role attribute to each <lexeme> element. At the moment, Ralf's French dictionary doesn’t contain any terminal information (noun, verb, adjective). It is possible to add terminal information to this dictionary with a simple text editor.

2. I changed another thing: The previous version of Ralf's French dictionary contained about 60.000 duplicate <lexeme> elements. I removed these elements with the following XSLT expression:

<xsl:for-each-group select="lexeme" group-by="grapheme">

You can find this line in the style-sheet improve-french-dictionary.xsl (license: GPLv3). I used this style-sheet to generate version 0.1.2 using the Ubuntu terminal:

am3msi@am3msi-desktop:~/Documents/201004/french-dictionary$ saxonb-xslt -ext:on -s:french-dictionary.xml -xsl:improve-french-dictionary.xsl -o:french-dictionary-0.1.2.xml

3. The language tag is now correct: xml:lang="fr"

Unfortunately, at the moment the French phonemes ɔ̃ — ɛ̃ — ɑ̃ — ɥ aren’t transcribed into the correct SAMPA phonemes during the simon import process. It shouldn’t be a big deal to fix this. As soon as this issue has been fixed, Ralf's French dictionary should be usable for training of French words. Remember: the dictionary contains more than 300.000 French <lexeme> elements. It would be nice if a native speaker from France would take a closer look at Ralf's French dictionary, and make suggestions for improvements.

Tags: ,
Posted in Ubuntu, dictionary | Comments Off

What are my targets?

Friday, January 22nd, 2010

I have two targets:

1. Produce more PLS dictionaries that can be imported into simon. I am planning to explain the development steps in this blog. It might be a little off-topic, but I think it is important to inform the people. This means that I will provide the reader with details of dictionary development / improvement.

I want that people understand how to handle the development / improvement of PLS dictionaries.

2. I want to learn about the simon source code. How does simon work internally? I don’t need to understand every detail, but I would like to be able to understand what is going on “behind the scenes” (scenes = simon GUI; behind = simon source code). Where can I start?

The article Becoming a KDE developer contains some useful links (e.g. I could type qtdemo into the Ubuntu terminal). Qt looks like a very interesting development software. How can I get involved?

Let me make an example: I read the C++ tutorial. The first chapters were easy. Then suddenly, it became extremely difficult. What are pointers? What is an array? At least I know how to compile a simple C++ program under Ubuntu. That is a start.

Or I could read the simon source code that is available via Sourceforge. E.g. I could read clientsocket.cpp. But I understand almost nothing.

3. Conclusion: It is a lot of work to focus on these targets.

Tags:
Posted in General, Ubuntu, dictionary | Comments Off

Julius dictionary; PLS: role attribute

Wednesday, January 20th, 2010

This blog post is about (A) Julius dictionary and (B) the import of a PLS dictionary.

A. Obviously, it is possible to import a Julius dictionary:

julius-vocabulary

I didn’t know that this kind of dictionary existed. What are the properties of this format? And what are the advantages?

B. I want to import Ralf's German dictionary (version 0.1.7; October 29, 2009). Great, simon now recognizes the role attribute:

adjektiv-substantiv

1. A few minutes ago, I imported Ralf's German dictionary into simon. I am offering 27 PLS dictionaries for 27 different languages. Choose the dictionary that suits your native language, and import it into simon.

2. Let’s take a look into the shadow dictionary.

3. The word kernchemischen is an Adjektiv. Let’s take a look at the specific entry in Ralf's German dictionary:

<lexeme role=”Adjektiv”>
<grapheme>kernchemischen</grapheme>
<phoneme>kɛʀnçeːmɪʃən</phoneme>
</lexeme>

You can see that the role attribute which is part of the <lexeme> element was imported by simon. Thanks for implementing that feature.

4. The word Kerndurchmesser is a Substantiv. The corresponding entry in the PLS dictionary:

<lexeme role=”Substantiv”>
<grapheme>Kerndurchmesser</grapheme>
<phoneme>kɛʀndʊʀçmɛsɐ</phoneme>
</lexeme>

You can see the strength of the simon import process: The last two letters Kerndurchmesser correspond with one single phoneme kɛʀndʊʀçmɛsɐ. Because such details are implemented, we can get a very good recognition rate as I showed in the video with 200 German words.

Why is Ralf's German dictionary good? Let me explain about the history of this dictionary:

a. The initial steps were done at Voxforge with the development of a German pronunciation dictionary. You can convince yourself: the script espeak2Phones.pl is great because it transforms eSpeak’s cryptic ASCII output into SAMPA. This approach is good for the German language.

b. Later, we used the dictionary acquistion project for the collection of about 8.000 pronunciations. Each single pronunciation was human-controlled. The phoneme concept follows the Wiktionary.

c. I used a German spelling dictionary from OpenOffice.org to get more words for the dictionary (Ubuntu terminal command: unmunch). With eSpeak I created the phonemes. With an XSLT style-sheet (Ubuntu terminal command: saxonb-xslt) I transformed the eSpeak phonemes into IPA phonemes. And I used the XSTL style-sheet for inserting the role attribute (Substantiv, Adjektiv, Zahlwort).

d. The result is the current version of Ralf's German dictionary. It would be nice if someone would help with the improvement. The real difficult work has been done. But it is necessary to fine-tune the dictionary. Let me give you a concrete example:

<lexeme>
<grapheme>stromsparen</grapheme>
<phoneme>ʃtʀɔmʃpaːʀən</phoneme>
</lexeme>
<lexeme>
<grapheme>stromsparend</grapheme>
<phoneme>ʃtʀɔmspaːʀənt</phoneme>
</lexeme>

What is wrong or could be improved? First, the role attribute is missing. stromsparen is a Verb. stromsparend is an Adverb. It would be good if someone added the missing role attributes. Second, there are small phoneme corrections necessary: ʃtʀɔmʃpaːʀən is OK because you speak “schtromschparen”. But ʃtʀɔmspaːʀənt is wrong because you don’t say “schtromsparent”.

You can see that improvements are necessary. Because Ralf's German dictionary is GPLv3, everyone is permitted to improve it.

For good recognition results, things like “schtromsparent” have to be fixed. It is possible that some dialects (e.g. Hamburg) speak “stromsparen” and not “schtromschparen”. I recommend that specific dialect dictionaries should be developed. This would be part of the fine-tuning, too. Ralf's German dictionary covers Standard German. You can use my dictionary for the development of a dialect dictionary that can be used by people who prefer to dictate in their own specific dialect.

By the way, did you notice the following detail? ʃtʀɔmspaːʀənt
 ends with a “t” and not with a “d” because of the “Auslautverhärtung” (which is part of the German pronunciation). Such small details are implemented in the dictionary.

C. Conclusion: I know about the strengths of PLS, but I don’t know which advantages a Julius dictionary would have to offer.

Tags: , , , , ,
Posted in Ubuntu, dictionary | 2 Comments »

Benefitting from eSpeak

Saturday, January 9th, 2010

Is eSpeak good or bad?

“Espeak with it amazingly bad speech synthesis quality and even more amazing popularity. Out-of-date synthesis method doesn’t let it be good with any possible modifications.”

I used eSpeak for the creation of my 27 PLS dictionaries (the phonemes were created with the help of eSpeak). I found out that the phoneme quality for German isn’t that bad. It is usable for speech recognition after I made some adjustments with an XSLT style-sheet.

What about the other languages? To be honest: at the moment, I don’t care. I need the 27 PLS dictionaries mainly for propaganda. It is necessary to involve more people in the development of an open source ASR solution.

A Polish native speaker wants to dictate in the Polish language. Or another user wants to dictate in the Vietnamese language. Or someone wants to dictate in the Greek language. These people could take advantage of PLS dictionaries in their own languages.

This is what I want to do: Build a PLS dictionary in a Chinese language (e.g. Cantonese – eSpeak offers this language as provisional language). I need a GPL word list with Cantonese words. But I didn’t find one in the internet (the description of this word list should be in English because I don’t understand Cantonese).

Is eSpeak’s synthesis method out of date? I don’t know. At least eSpeak creates phonemes that I can implement in my PLS dictionaries. Is there a program available that produces better results than eSpeak? The program has to work out of the box. I can use eSpeak by simply typing “espeak” into the Ubuntu terminal. And eSpeak can interpret SSML mark-up. That worked fine for me.

My PLS dictionaries are in an early state of development. It should be possible to increase the quality substantially with the help of some engaged native speakers.

Things have to work. To be more precise: it should be possible for the user to import a PLS dictionary in his own native language into simon. I made a start by offering 27 PLS dictionaries. At the moment, I am thinking about whether I should offer much more PLS dictionaries. The problem is: I don’t know how I can create the phonemes for the specific language. I will find a work-around for this problem.

Which kind of phonemes should the PLS dictionary contain? There are several possibilities:
- IPA phonemes (like in Ralf's German dictionary), advantage: can easily be edited by linguists;
- eSpeak phonemes (like in Ralf's Polish dictionary), advantage: I didn’t introduce new errors by trying to convert them into IPA;
- SAMPA phonemes (none of my dictionaries uses SAMPA), I don’t see any advantage at the moment.

In my opinion, a good phoneme quality can be achieved by using IPA phonemes. Because IPA phonemes are easy to read by linguists. So what can you learn from this post? If you are a native speaker of Vietnamese, Polish, Greek, you may want to take a closer look at Ralf’s Vietnamese / Polish / Greek dictionary, and think about what you can do to improve the quality.

Which advantage offer Ralf’s PLS dictionaries? They show you a way to make speech recognition work for your native language. As soon as you have a PLS dictionary with acceptable quality for your own language, you can think about using it for training with simon.

You can learn from my blog that you can import
- Ralf's Vietnamese dictionary,
- Ralf's Polish dictionary,
- Ralf's Greek dictionary
into simon. So simon is the target application. If you improve the quality of the specific dictionary, there is a chance that it might work.

And there is another thing that I found out when building / importing each of these 27 PLS dictionaries: the dictionary size should be about 100.000 words (not 1 million words, not 10.000 words). Help is needed to implement a good compression algorithm (like unmunch for OpenOffice.org dictionaries).

The focus should be to
- improve the quality of each PLS dictionary – native speakers should do that;
- integrate an option into simon to automatically download & import each of these PLS dictionaries into simon;
- think about a good compression algorith for PLS dictionaries (like unmunch) – languages like Spanish, Dutch, German, Latin need such a compression algorithm – not necessary for English.

Tags:
Posted in General, dictionary | Comments Off

Tutorial: how to install under Ubuntu

Friday, January 8th, 2010

This tutorial explains how to install simon under Ubuntu, and how to import Ralf’s Portuguese dictionary.

1. Download simon.
2. Double-click on simon-0.2-Linux_i386.deb:

ubuntu-deb

3. Press Install Package:

install-speechrecognition

4. Enter the password that you had chosen during your Ubuntu installation:

administrative-rights

Press the OK button.

5. The installation has been finished. The package simon-0.2-Linux_i386.deb has been installed:

installation-finished

Press the Close button.

6. Select Applications > Universal Access > simon:

universal-access

7. Take a look at the simon main window:

press-wordlist

Press the Wordlist button.

8. The Wordlist tab has opened:

import-dictionary

Press the Import Dictionary button.

9. You have to select the type of the dictionary:

select-dictionary-type

Choose PLS Lexicon, then press the Next button.

Note for simon development team: it would be nice if simon now offered a list of the 27 PLS dictionaries that are available.

10. You can now import one of my 27 PLS dictionaries. In the sidebar of testing simon, you can find a PLS dictionary that you can import:

sidebar-pls

Right-click on Ralf’s Portuguese dictionary, then Save Link As....

11. The dictionary with the name portuguese-dictionary.xml.bz2 will be saved:

save-portuguese-dictionary

It will be saved in the Downloads folder. Press the Save button.

12. It is time to import Ralf's Portuguese dictionary that you have just downloaded:

select-pls-file

Please press the File button to point simon to the downloaded PLS dictionary.

13. Select the Downloads folder:

select-downloads-folder

14. Select portuguese-dictionary.xml.bz2:

select-portuguese-dictionary

On my computer, I didn’t have to press the OK button.

15. simon displays the path to the PLS dictionary:

path-to-dictionary

Note for simon development team: it is pretty complicated to first download, and then import the dictionary. It would be nice if the wizard offered automatic download directly from the internet.

My guess is: the average user begins to lose interest in simon at this point of the installation because he already has invested about 20 minutes of his precious time. It is getting annoying. Don’t annoy the user! Offer automatic PLS dictionary import directly from the internet.

The automatic BOMP import is a great thing. But not everybody is a native German speaker. At the moment, I am offering 27 different languages. An automatic import would make simon much more interesting for a lot of people. E.g. Portuguese is spoken by 200 million native speakers. Recently, someone showed interest in Portuguese at Voxforge. You can imagine that almost all people don’t have a clue where to start, and what is necessary. Helping people from a lot of different countries would be so easy by adding an automatic import function to the wizard.

Press the Next button.

16. simon is now processing the lexicon:

processing-lexicon

What does that mean? It means that simon converts the dictionary from PLS format into HTK format. This process works fine for Ralf's German dictionary. The process is not yet optimized for the other PLS dictionaries. If you are a native speaker of Portuguese (European), you can edit Ralf's Portuguese dictionary with a simple text editor.

17. Ralf's Portuguese dictionary has been imported:

imported-successfully

Press the Finish button.

18. The dictionary is now available:

portuguese-shadow

a. Select Include unused words from the shadow lexicon.
b. Use the scroll bar to get an impression of the Portuguese dictionary.
c. First column: word. Second column: corresponding pronunciation.

19. Let’s finish here. Now you know how to install simon under Ubuntu, and how to import Ralf's Portuguese dictionary into simon.

There are more steps necessary to make it work:
- install HTK;
- record a few training samples;
- define a grammar;
- start ksimond (PDF).

Take a look into the simon handbook to find out more about simon.

Tags: , , , ,
Posted in Ubuntu, dictionary | Comments Off

Import 180.000 Slovak words

Monday, November 16th, 2009

You can import Ralf's Slovak dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

Tags: ,
Posted in dictionary | Comments Off

Import 200.000 Icelandic words

Sunday, November 15th, 2009

Now, you can import Ralf's Icelandic dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

Tags: ,
Posted in dictionary | Comments Off

Ralf’s Vietnamese dictionary

Sunday, November 15th, 2009

You can import Ralf's Vietnamese dictionary (version 0.1; GPLv3) into simon. The dictionary contains about 6.000 words; training is not possible. The phoneme elements contain eSpeak phonemes (not IPA phonemes).

Tags: , , , ,
Posted in dictionary | 1 Comment »

Import 140.000 Russian words

Saturday, November 14th, 2009

Now, you can import Ralf's Russian dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

Tags: ,
Posted in dictionary | Comments Off

Import 300.000 Norwegian words

Saturday, November 14th, 2009

You can import Ralf's Norwegian Bokmål dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

Edit May 2010: License of the source is GPL.

Tags: , ,
Posted in dictionary | Comments Off

Import 380.000 Swedish words

Friday, November 13th, 2009

You can import Ralf's Swedish dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible. The phoneme elements contain eSpeak phonemes (not IPA phonemes).

On my computer, the import of this PLS dictionary took three minutes.

Edit May 2010: License of the source dictionary is GPL.

Tags: , ,
Posted in dictionary | Comments Off

Import 40.000 Swahili words

Thursday, November 12th, 2009

You can import Ralf's Swahili dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible. The phoneme elements contain eSpeak phonemes (not IPA phonemes).

Tags: ,
Posted in dictionary | Comments Off

Import 60.000 Tamil words

Thursday, November 12th, 2009

You can import Ralf's Tamil dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

You can try to record Tamil words with simon:

tamil-record

It is necessary to do further adjustments. This is just a first step to make simon work with the Tamil language.

If your native language is Tamil, you may want to do the following things:

1. Install simon.
2. Install HTK.
3. Import Ralf's Tamil dictionary into simon.
4. Record some Tamil words with simon (see screen-shot).

After you have done that, you can try to improve Ralf's Tamil dictionary.

Tags: , ,
Posted in dictionary | Comments Off

Import 150.000 Romanian words

Thursday, November 12th, 2009

You can import Ralf's Romanian dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak".

Some words about the creation of this dictionary: After getting a spelling dictionary, I generated from the SSML document the phonemes with eSpeak.

Tags: ,
Posted in dictionary | Comments Off

Import 200.000 Croatian words

Wednesday, November 11th, 2009

You can import Ralf's Croatian dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is not possible.

The <phoneme> elements contain eSpeak phonemes (not IPA phonemes) – alphabet="espeak". The eSpeak phonemes are slightly modified: The & has been replaced by &amp;.

Tags: , ,
Posted in dictionary | Comments Off

Ralf’s Polish dictionary

Tuesday, November 10th, 2009

You can import Ralf's Polish dictionary (version 0.1; GPLv3) into simon. Training with this dictionary is currently not possible.

Some details about (the creation of) this dictionary:
- The grapheme elements contain often garbage characters. I tried to fix this issue (with iconv), but unfortunately I wasn’t successful. When the grapheme element contains crap, the phoneme element is crappy, too. These lexeme elements are unusable.
- I didn’t convert the eSpeak phonemes into IPA phonemes. They are still in their original format. This is indicated by the attribute of the lexicon element: alphabet="espeak"
– As source I used a spelling dictionary.

This dictionary is of really bad quality. It is necessary to solve the encoding issue. But I tried three different conversions, and I failed.

At least, you can get an impression how easy it is to import 300.000 Polish words.

Tags: , , , ,
Posted in dictionary | Comments Off

Ralf’s Esperanto dictionary

Saturday, November 7th, 2009

You can import Ralf's Esperanto dictionary (version 0.1; GPLv3) into simon. It should be possible to train simon with this dictionary. The pronunciation of this constructed language should be easy even if you have never spoken this language.

The dictionary contains about 19.000 lexeme elements. Some grapheme elements contain garbage characters. But that doesn’t affect the overall usability of the dictionary (only the specific lexeme elements are affected).

Tags: ,
Posted in dictionary | Comments Off

Import 90.000 Italian words

Saturday, November 7th, 2009

You can import Ralf's Italian dictionary (version 0.1; GPLv3). Training with this dictionary is currently not recommended. Some notes about (how I created) this dictionary:

1. I got an Italian spelling dictonary.

2. The unmunch command produced more than 20 million Italian words. Because simon is not intended to handle very large lexicons, I decided to use the style-sheet create-graphemes-italian.xsl instead. This style-sheet removes the prefix/suffix information from the spelling dictionary it_IT.dic. The result was an SSML file with about 90.000 Italian words.

3. I generated from the SSML file the corresponding phonemes: $ espeak -f italian-audio-o -m -v it -q -x --phonout="italian-espeak"

4. Then I combined the grapheme elements with the phoneme elements.

5. The last step was the conversion from eSpeak phonemes to IPA phonemes with the style-sheet espeak2perfectipa-italian.xsl. Here are some of the Italian specific conversions that are contained in the style-sheet:

replace($sierra, 'dZ:', 'ddʒ')
replace($sierra, 'ts:', 'ddz')
replace($sierra, 't:', 'tt')
replace($sierra, 'd:', 'dd')
replace($sierra, 's:', 'ss')
replace($sierra, 'b:', 'bb')
replace($sierra, 'k:', 'kk')

I tried to follow the IPA for Italian. To make the dictionary work with simon (so that training produces reasonable results), the simon import process has to be adjusted. Effective training is currently not possible.

Now you know that an Italian pronunciation dictionary exists that you can import into simon.

Tags: , ,
Posted in dictionary | 1 Comment »

PLS dictionary: 850.000 Spanish words

Friday, November 6th, 2009

Ralf's Spanish dictionary (version 0.1; GPLv3) contains about 850.000 words. You can import this PLS dictionary into simon. Some remarks about (how I created) this dictionary:

1. I downloaded a spelling dictionary.

2. Then I used this spelling dictionary to produce the content of the grapheme elements:

$ unmunch es_ES.dic es_ES.aff > spanish-wordlist

3. From the word list, I created the content of the phoneme elements:

$ espeak -f spanish-ssml -m -v es -q -x --phonout="spanish-phoneme"

4. I combined the grapheme with the phoneme elements:

$ paste spanish-ssml spanish-phoneme-o > spanish-pls

5. With this style-sheet, I transformed the eSpeak phonemes into IPA phonemes. I am not sure whether I transformed everything correctly. E.g., I am unsure whether the following conversion is correct:

replace($sierra, 'J\^', 'ʎ')
replace($sierra, 'J', 'ʝ')

It might be the other way around.

6. The following phonemes probably will cause problemes when you import the dictionary into simon: β, ð, θ, ɣ, ɾ, ʎ, ʝ

7. On my computer, it was necessary to wait 5 minutes until the import into simon had been finished.

You can see that the import of Ralf's Spanish dictionary is possible. Unfortunately, training with this dictionary is currently almost impossible because of the phoneme issues.

Tags: , , ,
Posted in Ubuntu, dictionary | 1 Comment »

« Older Entries