WiktionaryZ

Last post for this blog ..

2006-12-12T19:38:00.000+01:00

The OmegaWiki project aims to provide information about all words of all languages with a user interface in all languages. It achieves this by using relational database technology extending the wiki paradigm. This project attracted attention from organizations that made the development of relational database technology from inside the MediaWiki software possible.

OmegaWiki originated from the 171 Wiktionary projects of the Wikimedia Foundation. Originally the project was called "Ultimate Wiktionary" it was renamed to "WiktionaryZ" and now it will be known as OmegaWiki to prevent confusion with its Wiktionary sister projects.

The OmegaWiki project will include specialist terminology and will have an ontological component that will allow for the inclusion of specialist thesauri. Due to the experimental nature and to some of the requirements make that the Wikimedia Foundation is not in a position to host the OmegaWiki project.

To allow for the further development and hosting of the OmegaWiki project, the Stichting Open Progress, a foundation based under Dutch law, will assume the responsibility for the project. The "WiktionaryZ commission" will continue to take care of the day-to-day issues for the OmegaWiki project and, it will be renamed to "OmegaWiki commission"

The content of OmegaWiki will continue to be licensed under a combined GFDL and CC-by license. This is to enable what the project has defined as success: "success is when someone finds an application for our data that we did not think of".

The Wikimedia Foundation and Stichting Open Progress will continue to work together to achieve their shared objective; to bring information to all people of this planet. For more information:
We hope and expect that the Wikimedia Foundation and Stichting Open Progress will be able to work together to achieve their shared objective; to bring information to all people of this planet.

For the Stichting Open Progress and the OmegaWiki committee,
Gerard Meijssen

Ahead of the curve

2006-12-06T12:03:00.000+01:00

WiktionaryZ is very much a project in its infancy. Many parts that are necessary to fulfil its promise are lacking. Nothing new here, WiktionaryZ is still pre alpha software. One of the great things of Open Source is that when a project has sufficient attraction people come along willing to collaborate.

WiktionaryZ and the Kamusi project have many similarities as well as differences. The most striking differences are the scope; Kamusi is about the Swahili language and its maturity. Kamusi has been going strong for quite some time. As Kamusi is so much ahead of the curve from where WiktionaryZ is, it is really interesting to look at its strengths.

I was reminded about Kamusi because of its recent newsletter; it offers a rich mix of facts about culture, updates of the project. Merchandise is offered with the Swahili clock being imho the most spectacular item on offer. All this to fund a project that runs as much on passion as it does on money.

Here too is a project that is life while continuously being developed. A project that depends on its community, it currently honours the people who contributed the most content with statistics and like WiktionaryZ it has supporters who are part of what makes Kamusi a reality.

I have met Martin Benjamin, the Kamusi editor at Wikimania 2006 in Boston. It was too short a pleasure. We have talked since using Google talk on several occasions and ast Kamusi is ahead of the curve I have been privileged to learn much from the Kamusi experience.

My hope is that at some stage we will have the maturity of Kamusi. It is likely that we will have this maturity first for some languages and slowly but surely for others as well.. WiktionaryZ is very much a journey, enjoy the ride :)

Thanks,
GerardM

WiktionaryZ is a wiki and not perfect

2006-11-26T22:21:00.000+01:00

One of the contributors of WiktionaryZ who is with us from the start is really into the "minor" languages. There are no newspapers in this language, there are many dialects in this minor language. And now we have this need for a user interface in this language too.

For this stellar worker, this is a problem. As the orthography is not set in stone, what is the correct word for the translations of English, German, Italian? As people will see it all the time, it has to be correct. There is also some reputation involved; when you are seen to be perfect in a language how can you err when everyone will start using what is provided?

I have been many translations in the last few days as well. I have been working hard to add this content on the Dutch Wiktionary. Now I have learned that the names of languages in Russian and Danish are not capitalised. So I will change these when I copy them in. I am not sure for languages like Slovak and Slovene.. I have the choice not to copy them in and leave it for someone else or I can make mistakes and hope that someone cleans up after me.

I am sure that some of the stuff I did needs correction. In the mean time it is an honest effort and, it shows that by making the correction ONCE, we truly benefit from the way WiktionaryZ works.

Thanks,
GerardM

NOW WITH LANGUAGES IN YOUR LANGUAGE

2006-11-20T17:18:00.000+01:00

Well, sometimes advertising the good stuff makes you feel really good. Today, the data that allows us to show languages names in the language of the User Interface has gone life. For WiktionaryZ this is really important new functionality. We want to have a user interface that brings WiktionaryZ in the language of the user. You should not have to know English to get your information about an Urdu word.

The one problem is, we can only show you the languages in your language when we know them. It is therefore that we like you to help us build this part of the User Interface.. This is truly stuff that we will be using all the time..

Thanks,
GerardM

Why Multilingual MediaWiki matters

2006-11-15T08:05:00.000+01:00

You might think that MediaWiki, the software that runs Wikipedia (and WiktionaryZ), is fully multilingual. After all, everyone knows that Wikipedia exists in more than 200 languages -- all of them intricately connected through a network of "interlanguage links" from one edition to the other.

In actual fact, each language edition of Wikipedia runs on a separate database. Wikimedia uses a special setup to share code and configuration files, but that setup is not trivially reproducible. If you want to set up a wiki in multiple languages, your best bet is to set up multiple instances of MediaWiki. As a result, most MediaWiki installations around the world are monolingual. They only accept content in one language.

Of course, users are free to put, say, French language content into an English language wiki. They can even change their user interface language preference to French. But the problems begin quickly:

If multiple languages use the same title to refer to a particular page, you need to disambiguate it. In single database setups, this is typically done by appending the language code manually to the page title. These titles can quickly become messy and inconsistent.
As soon as activity picks up in multiple languages, the list of changes to the wiki quickly becomes cluttered with information that is useless to readers who do not speak the particular languages in which edits are made.
It is impossible to systematically search for pages in a particular language.
Pages about the same content in different languages have to be manually connected to each other, which is often done using templates. The wiki does not facilitate the process of interlanguage linking in a single installation. The interlanguage links which are used on Wikipedia are horribly inefficient, as a separate set of language links has to be maintained for each language.
The first experience for a user who does not speak the default language of the wiki is often negative. Unless the wiki has been specifically built (with policies and interface messages) to encourage multilingual contributions, they are unlikely even if they are theoretically possible.

Take a look at a big wiki hosting site like Wikia -- even though it sets up multilingual wikis on request (by setting up multiple databases), most of its wikis are monolingual by default. English is of course predominant. With hundreds of millions of Internet users who do not speak English but would be happy to contribute to these wikis, this is a tremendous loss of opportunity. Outside a framework like Wikia, with users setting up their own wikis, it gets even worse: very few people go to the effort of structuring their wikis to accept content in multiple languages.

Wikis have become as ubiquitous as forums or blogs. Whether we are talking about documentation, knowledge bases, directories, discussions, experiments in democracy, media archives -- there are millions of potential participants out there, waiting to be invited to contribute. Waiting to feel welcome. We need to reach out to them. It is not just the community that needs to make the decision to "go multilingual". It is the software that should support this decision as much as possible.

Fortunately, there is an answer: Multilingual MediaWiki. This set of specifications describes the changes to the MediaWiki software needed to accept content in multiple languages, to network it effectively, and to build truly multilingual communities. And fortunately, this is more than just a paper: It is being implemented by a very capable programme, with financial support from the University of Bamberg, and another sponsor who shall remain unnamed for now. You can view the first prototype (still very messy :-), which showcases the functionality to a) store content in language "meta-namespaces" and keep it separate, b) connect pages in different languages to each other.

There's still quite some way to go until this becomes part of MediaWiki proper, but we are making steady progress. When the project is completed, thousands of MediaWiki installations across the planet will gradually become fully capable of accepting content in all languages of the world (if their owners want them to). It will be another step in opening up the world of wikis to the global community.

Beyond its very direct impact on wiki users, MLMW is a requirement for WiktionaryZ. Right now, pages about expressions such as AIDS, which exist in many languages, can become very messy. Ideally, the user would, when looking up an expression, always specify which language it is in -- and then only see the DefinedMeanings in that language. However, for this to work effectively, MediaWiki must support looking up pages in a particular language, exactly the functionality that MLMW will provide.

This is one of many examples in which WiktionaryZ development benefits MediaWiki as a whole. Moreover, it reflects our philosophy to structure our work so that milestones can be reached independently whenever possible. We had some initial problems with the MLMW project -- a funding source ran out, and a developer team became unavailable. Fortunately, this did not impact the main WZ development, and work could continue as soon as we found a new source of funding and a new developer.

I'm not aware of any other wiki engine which handles content in multiple languages well. Hopefully, MediaWiki will become the first.

About Definitions

2006-11-14T16:01:00.000+01:00

There is an art to writing lexical definitions. The problem for WiktionaryZ is, that this art has to be relearned. Definitions need to be concise. There is however no reason not to use full sentences. Abbreviations are not needed as there is plenty of space on the hard drives of modern computers.

One accepted practice in the dead wood dictionaries is to define by using synonyms. For WiktionaryZ it leads to circuitous definitions because the definition defines the synonym as much.

Another thing is to use bad constructs like "To do something (to someone)". This looks bad, it is better to just say "To do something to someone". There is no reason why we cannot produce full sentences without constructs that are inherited from existing dictionaries. If they were from existing dictionaries, they might be in line for improvement.

Yet another thing that we should not do is be overly specific and correct. The definitions define; they show what it is about. When this is not defined in "sufficient" detail, it often proves that there is a need for an encyclopaedic article and, that is when we need to refer to the Wikipedia article about the topic.

Thanks,
GerardM

Success can kill

2006-11-10T11:14:00.000+01:00

I am really grateful that Erik was able to reboot WiktionaryZ from Heathrow. It save us from another few hours of downtime. When he finally came home and when we talked, I learned that it is the popularity of WiktionaryZ that brought the system to its knees. When you look at the statistics of WiktionaryZ, you will see an upward trend and as Erik indicates, it is getting at what the current hardware can maximally handle.

It means that we have to upgrade our servers in the very near future.

The development of the software is going ahead strongly. I was told that we are now at the stage where we can start developing attributes on the level of the SynTrans records. This is techno babble for; we are ready to work on things like parts of speech functionality.

Parts of speech is Language specific. This means that two languages will not necessarily have the same parts of speech or inflections, conjugations. It means that the development is in phases.

Indicate that a specific part of speech exists for a language.
Indicate that a word that is among the translations or synonyms is a particular parts of speech
Indicate what conjugations / inflections are possible in a language for a specific parts of speech
Allow for scripts that propose what the conjugations / inflections could look like.

When you read this, it is rather glib. There are many issues that need resolving and all these issues are not mentioned. There are also several opportunities that will make for better functionality that are not indicated. But hey, this is a blog not a specification.

What IS important that this is a good time to talk about this. What I point out is that we are about to conceive of this functionality and that we do welcome comments.

Thanks,
GerardM

Language names

2006-11-08T13:34:00.000+01:00

WiktionaryZ is not far off from the moment when we will be able to have the labels of the languages shown with the translations and synonyms in the language of the user interface (UI). The way I understand it will work is that a DefinedMeaning that is tagged with the corresponding ISO-639-3 code will have its available translations copied to the Language table.

This will be fun to observe.

The codes that we use do not allow for specific versions of for instance English; "English (American)" is coded like eng-US. This is however not an ISO-639-3 code. We will find a solution for that one. For other languages there is this utter confusion of what is meant by a language name; Schwyzerdütsch for instance is also called Alemannisch and it is certainly not the Swiss version of High German. Then there are languages like "Western Mari", Wikipedia considers them a dialect where Ethnologue recognises it as a language.

I hope and expect that people will want to have all the languages in their language on WiktionaryZ. For many languages it will be hard to find a name that fits the orthographic rules of their language.. Haiǁom for instance ..

Thanks,
GerardM

PS It would be nice if these entries are sorted.. Well, this is something for another blog..

When two people do the same ...

2006-11-07T22:07:00.000+01:00

... and you are not aware of it, you can get in a situation where you would like to .... hmmm ... don't know how to translate this into English .... but the sense is: you get somewhat angry with yourself.

While writing the post from some minutes ago I learnt that what Webboy programmed had already been programmed by Leftmost and also submitted. We just were not aware of it.

The positive side of this is: we now know that both are great programmers.

The negative side is: we were not able to manage the programming in such a way that this double effort would have been avoided. That is really not nice for both of them. So: sorry, we will try to find a way in order to avoid this from happening again.

Well yes, we need an open accessible list where to write down our wishes and where programmers can write their note on "is on the way". So: time to think about a solution ... and we shall do that now.

Sorry again Leftmost and Webboy!

Queries ... they are such great ...

2006-11-07T19:41:00.000+01:00

Today I saw one of those queries applied to WiktionaryZ. Imagine you want to know for which expressions of a certain collection there is a defined meaning in a certain language. So you choose the language you want to translate from, choose the one you want to translate in and, if desired you choose a collection. Then you click on that neat button that says "search". What you get is a list of terms where you can work on adding the translations of DefinedMeanings. That is great, right? Well at this stage a big "thank you" to Webboy, who is the programmer of that bit.

Now imagine our OLPC Children's Dictionary ... the same feature only searching on DefinedMeaning level ... that would help us to save really a lot of time - it would not be necessary anymore to say: please care about these 5 words ... you could say: go there, insert English + your target language + select the OPLC Children's Dictionary as a collection and please do some words (even only one word). That would be really great ... I hope we will get this feature soon.

Well there is another thing I would like to mention when it comes to the OLPC Children's Dictionary ... and that is about a dump that was needed to show how we can do things - in a period when the server was down and difficult to access ... Thanks to Leftmost for helping out with that.

Sabine

Hosting as a showstopper

2006-11-03T08:33:00.000+01:00

WiktionaryZ is down. It is probably an index in the database or some such that prevents the system to come up. To make matters worse, Erik the only person with access to the server is on his way to Jamaica from Berlin. This means extended downtime. Extended downtime is not good for a project; it leads to disgruntled users it prevents the further adoption of the project.

WiktionaryZ will in the near future (ie this year) host what some call "professional wikis". They are extensions to specific domains. The people that have this need require certain standards. One of these is 24*7 support, another is back-up support.

There are two points to it; hosting will not be on Erik's server when the new content is merged into WiktionaryZ. The second is that a known single point of failure is only acceptable as long as you are willing to pay the price of failure. It is good that we are increasingly unwilling to pay this price it is gratifying to realise that our hosting WILL be elsewhere at the end of the year.

For those who do not appreciate it; having WZ hosted by Erik was a good thing. It was the option that we had.

Thanks,
GerardM

What to do when a word is not found

2006-10-31T12:37:00.000+01:00

One of the favourite stories of Ronald Beelaard is that many people cannot spell and consequently do not find what they are looking for. The example that he has used often is the word "papegaai", a word many people do not know how to spell.

What Ronald did for the library he was associated with was to write what amounts to an "alternative spelling" function. There are plenty of examples of how this can be done; Google is one of the examples that comes to mind..

For WiktionaryZ, it will not be as straight forward as it is for a monolingual resource. When a word is not found, it is important to know what language this word should be in. Also, WiktionaryZ is lacking many words that you would expect in a resource that want to include all words. This is best demonstrated that of the 1000 most popular words in English, several are still missing..

When a word is not found, it would be cool to have functionality that helps us to understand what the problem is. The first thing would be to aggregate the number of misses. It would be as cool to have the number of hits because it is in the percentages where we will find how well we do.

The words most often not found are the words that are the most relevant to add to WiktionaryZ. This will improve its function as a resource.

Thanks,
GerardM

Some more on non-standard orthography

2006-10-29T14:11:00.000+01:00

Many languages do not have a standardised orthography. The variation that can be found is often quite specific for a specific location and also, the variations that can be found is often shared between different localities.

When there is no single biggest resource of what is considered to be standard for all these variations, we cannot use the strategy in WiktionaryZ like we do for English, French, German.. We need another strategy to make this work.

I had a long talk with Purodha just about this. The idea that we came upon is to accept all spelling variations for a word and have the "locations" subscribe to one variation. Our expectation is that there will be some regularity to be found; rules for writing that are always just so even when this rule is limited in it's scope.

The question we have to ask ourselves is how applicable this is and, it it does not make people come up with how they think "their" language is written. One thing I learned is that accepting this is not always the most reasonable thing to do. Yes, people may come from a certain area but that does not make them necessarily qualified to pronounce on the subject of their language. It often goes wrong because these people cannot and will not take some needed distance from the subject resulting in either a wish for the past or a wish for a new orthography or because of their politics.

Thanks,
GerardM

Dialects any one ?

2006-10-23T09:41:00.000+02:00

When there is a need for new languages in WiktionaryZ, the current procedure is that one of the bureaucrats can and do enable languages for editing content. So far we have added languages and had special considerations for scripts and locales. Scripts are "easy" as we have in the ISO-15924 a list of what are considered scripts, examples of their use is in how we deal with for instance Hausa, Mandarin or Serbian. Locales are more problematic, but so far we have restricted ourselves to using country codes. Country codes are easy too they can be found in the ISO 3166.

Now what to do with dialects. Let me be really practical; I have received a request for several languages by Mark Williamson. His request seems reasonable to me; he indicates clearly what he wants, the languages with their dialects and their scripts. When I try to do some research I find nothing that makes his request unreasonable, it is just that I cannot really judge it.

The biggest stumbling block however is, how to name these dialects as a code. It is one thing to insist on standards but another to find that there is no apparent standard for these dialects. I have been looking around and I subscribed to a few mailinglists to do with linguistics, and I hope that this will get me an answer.

With WiktionaryZ still in "pre-alpha" mode, I have an excellent excuse not to rush into the creation of these languages, but I think it is also fair to let it be known what the issue is. It is a practical one, it is not that we do not want dialects to be part of WiktionaryZ.

Thanks,
GerardM

Persian

2006-10-15T17:32:00.000+02:00

Persian is a right to left language. It also has a different script than what we are used to. The implication of this is that the user interface does not work well because to enter Persian, you have to select the language name in English and add the word in Farsi.

At this moment you have to change keyboardmappings from one to the other. Really inconvenient. We are extremely grateful that we are getting people who contribute to our codebase. One of them is working on showing the language names in the local language.

I can imagine that because of caching purposes we may need to cache versions per language. The work on Mulitlingual MediaWiki has started afresh. It is great to see that these things are coming together at last.. :)

Thanks,
GerardM

A great new tool

2006-10-12T18:55:00.000+02:00

Today Sabine told me about The Locale generator. This represents a great opportunity. It allows people to define what the many settings that are part of the CLDR are. The values defined here are of relevance to things like how you show a date, how to quote etcetera.

Even though it is a great start, there are things inherently problematic. The way languages are referred to are also not necessarily unambiguous. The language names do refer to the ISO-639-1 while the documentation calls them ISO-639-2. ISO-639-2 exists in two flavors ...

The generated XML content is targeted for use with Open Office, they do a good job by including references to what Microsoft does. Sorting out their non-standard ways. Initiatives like this make me really happy. The next step that they could take is to provide the information created in this way to the Unicode people so that they can integrate it in the CLDR itself. When this trickles back into tools like PHP, Python and Java, thing will be really to look up. :)

Thanks,
GerardM

Expressions without semantic content

2006-10-09T15:51:00.000+02:00

A DefinedMeaning is the union between a specific set of signs (a Spelling) in a given Language - an Expression and a paraphrase of the semantic content of this Expression - a Definition. This is a fundamental pillar of WiktionaryZ, and without it WiktionaryZ might well prove to be a house of cards. For the vast majority of Expressions in most languages this pillar is a sound base.

In all languages however there exist a few Expressions (in some languages quite many) that are intrinsically empty of semantic content; their sole function is to serve as glue between the more common semantically heavy Expressions, and relate this either to one another or to the context in some way or another.

There are many examples. To name a few: the adverb of negation in most languages (not), the evidential particles of for example Akha, that serve to ground any declarative sentence in a conceptual framework where the speaker indicates how (s)he has obtained the information given (with the same ease as a speaker of English indicate the temporal framework within which f.ex a declarative sentence is staged), aspectual particles systems of many East Asian languages etc etc.

While there will in due time be space to define the Usage of a given Expression, the question of how to treat the Expressions that have no useful Definition with which to unite into a DefinedMeaning has imo not yet been thoroughly solved.

Some possible ways of dealing with this might be:

Allowing the Definition of a limited set of DefinedMeanings to not express semantic content, but grammatical (in the widest sense of the word) content.
Allowing the Definition of a limited set of DefinedMeanings to be empty, deferring to a Usage note.
Referring these Expressions to a grammar.

I am not too happy with any of these solutions, most especially not with the last one.

WiktionaryZ and RFC 4646

2006-10-07T09:44:00.000+02:00

The RFC 4646 provides many things that I will gladly follow particularly in the way it allows for extentions. I have discussed the issues that I have with Felix Sasaki (W3C) and Gerhard Budin (ISO) among others.

Many recognized languages have to "fit" as a subtag of another language. This is for many people who think "politically" about languages not acceptable. The way it is implemented is also a travesty for people who know about languages. The motivation is to provide backwards compatibility even though ISO-639-2 was dismissed as useless and ISO-639-3 had to be created really quickly.
Some of the things not standardized yet in the ISO codes are worked on as new standards (eg dialects) the proposed codes are not known at this time and this creates a mess of its own.
Orthographies are not supported in the planned for extentions to ISO-639. This is acknowledged as an omission.

Consequently, it is much better to make a clean break while still conforming to standards. The standards complicance existing in WiktionaryZ is better than the standards compliance of the current crop of Wikimedia language codes. They do not conform to a standard because it breaks standards in many places. Some of the language codes used are voted in with a total disregard of what would be valid vis a vis the terms of usage of the ISO-639 codes.

Let me repeat that WiktionaryZ does include a code for Wikimedia language codes; it makes sense to have backwards compatibility. The basis for inclusion at WiktionaryZ for languages at this time are the ISO-639-3 codes. When need be the ISO-639-3 codes are extended using RFC 4646 to provide guidelines on how to do this and we will and do ask people in standards organizations on how to solve issues that are outside what RFC 4646 provides answers for.

We do have a code currently indicated as "ISO-639-2" in our database. We could enter something there that would be compatible with the RFC 4646 once it is figured out what would be a valid code under this regime. Problematic is that many of the ISO-639-2 codes are depreciated in ISO-639-3 and, it would be a nice academic exercise to create these codes where I do not want to deal with the political fall out that this creates.

Thanks,
GerardM

This user is a contributor to the OLPC Children's Dictionary.

2006-10-05T21:22:00.000+02:00

WiktionaryZ is proud to support the OLPC or One Laptop Per Child project. This project intends to innovate education by bringing laptops to kids in countries like Nigeria and Brazil. Huge amounts of computers will be distributed to kids. These computers are exquisite, they are rugged, they are innovative and they will change things one way or another.

In education dictionaries are one of the resources that kids use when they are available. For many languages it is hard to find content. We are currently looking for people who can help us with Igbo, Yoruba, Hausa .. We already have people who can help us with other languages; at WiktionaryZ we have started with the 1000 basic words in English. These we want to expand with as many translations as possible both for the expressions and the definitions.

Sabine is doing the sterling work that we know of her by organizing byte sized packages; this way people help by adding small stitches making this a blanket of communal effort.

Thanks,
GerardM

PS Yes I contribute too

Problems with MediaWiki .. Incompatibilities

2006-09-13T09:53:00.001+02:00

MediaWiki is great software, it works well, it scales like it should, it is probably the software that is localised most often. It is also incompatible with the orthographies of several languages.

We already new about the problem with Neapolitan, they have words like d''a the problem is that '' means that the next bit is to be italic .. or bold. A new problem came up. The N|u language is spoken in South Africa, because of the | it is not possible to create a wiki link to the name of this language.

There is a big move in the MediaWiki software to get WYSIWYG for MediaWiki. I hope people will also take the next step and kill off the current Wikisyntax that prevent us from doing what needs to be done; support all languages.

Thanks,
GerardM

New languages ..

2006-09-12T13:03:00.000+02:00

So far adding new languages to WiktionaryZ was a major pain. Erik had to do this directly in the database AND he is the only one who is allowed to do this. When you combine this with the fact that his time can be better spend in other ways, it was not great.

The good news is, Erik has created a function that allows bureaucrats to add languages. It means that we can slowly but surely add languages to WiktionaryZ. The first thing that we have done is fixed the Chinese problem; what was Chinese is now called "Mandarin (simplified)" and we have added "Mandarin (traditional)". This means that we can now add all the languages that were included in the ISO-639-2 zho code and consider script and other issues. An other thing will be adding English (United Kingdom).

My idea is to add the languages for which we have people on WiktionaryZ.. The good news is, people just need to ask, and after some consideration (scripts etc) we can honour such a request.
Thanks,
GerardM

Alternative spellings

2006-09-11T18:30:00.000+02:00

WiktionaryZ will provide for alternative spellings (though the name for them is yet to be finalized) of Expressions. Each type of alternative spelling will be tagged with a comment explaining its nature.

This could be used to enter currently correct alternative spellings, previously correct spellings, common misspellings, spellings that vary in capitalization etc. (Cf. How to deal with conventions)

Could it be used for more? What about the Swedish verb plural forms that went out of use in the 1950's? Today all persons and numbers use the same verb form, so looking up a verb plural form it makes sense to get the currently used form - that is the form that previously were only used for singular. Is this stretching the usage of an alternative spelling too far?

More new functionality ..

2006-09-11T15:01:00.000+02:00

We are really happy to announce new functionality. These are the first examples of software that helps editors find work that needs to be done. The first one helps to find words in Italian that do not exist in Neapolitan .. (you can change it to other languages by manipulating the URL) .. The second one finds all words that include "casa".

The great thing of open source is that you can publish often; even though the new functionality is still rough around the edges, it does show where we are going. Not only that, it will provide input by having the functionality tested at the differen stages of the development of the new softwar..

Thanks,
GerardM

WiktionaryZ and Machine Translation

2006-09-11T14:52:00.000+02:00

Most of you probably know that Jeffrey V. Merkey is using machine translation to translate the complete English wikipedia to Cherokee and that the output is quite good so that only 5% of it need real corrections.

He agreed to give us the used wordlist under GFDL and so we added it to WiktionaryZ.

Yesterday I tried to work with it and immediately I found some limits for its contents. This is not because his wordlist is wrong, but just because it is so personalized that for other people it becomes somewhat difficult to understand which meaning has been translated to Cherokee and wich meaning than accordingly needs to be translated to other languages, in my case Neapolitan.

I did an estimate of how many words I have to translate to get the work done in a month: 233 words ... that is really a lot ... and I suppose that doing things correctly - I mean creating the English word, adding the defined meanings and translate them into Neapolitan will take me quite a long time and probably I will not be able to do more than an average of 10 words a day.

Now you are asking yourself why I take such a long time ... well: let's say an English word has averagely 3 meanings (well, that's a wish ... there will be words with many more meanings). Then: I am not used to translate from English to Neapolitan - already Italian to Neapolitan is quite difficult. The only dictionary I was able to find here is Neapolitan to Italian ... I did not manage to go to Naples by now and therefore could not try to get one for Italian to Neapolitan there. This means that I have at least to translate from English to Italian - well, then it makes sense to add German as well since I know most translations in German - and then, if I don't know the translation from Neapolitan to Italian I have to guess how it is written in Neapolitan and find poof of it in the dictionary :-) funny right? Now people say: well you speak quite a good part of your day Neapolitan, so why do you need to look it up ... speaking is different from translating ... also writing is different from translating.

The next thing is: Wikipedia articles are of specific domains ... well: we do not have a differentiation of specific domains by now - but when translating for some languages we will need that, because the same English word, according to its use in a specific domain will have different translations.

Considering these points I suppose that we will have approx. 21,000 words in the foreign language starting with the approx. 7,000 words in English.

It will be fun :-)

Countries

2006-09-10T17:44:00.000+02:00

WiktionaryZ is to include all words of all languages. To get a relevant amount of information, there are many ways that this can be achieved. For WiktionaryZ, it will be important to have information that is going to be part of the user interface. In order to include all languages, we will need to have all languages in WiktionaryZ. These lists exist, and it is easiest to start with an import of all these language names.

The same could have been done for countries, however there are less of them and as we have started creating creating portals for countries, it made sense to start creating the common names for countries. Having these portals is important because often people do not appreciate how many languages are spoken in a country.. Australia demonstrates this really well.

When more languages are enabled for editing, the list of translations for the countries becomes less complete. In many ways it is an uphill battle to get the information that we need. It is a good thing that we decided on fall back scenario's; it is relatively easy to provide information in English..

Thanks,
GerardM

WiktionaryZ commission

2006-09-08T09:42:00.000+02:00

WiktionaryZ has a commission. This commission consists of people who are well known in the community. Also they represent different constituencies. As time goes by, the people that make up this commission change. When people are not really active they may be replaced. When new people become important in the community they may be added when they also represent an important community.

We have asked Sannab to join us in the commission. I am really happy she accepted. Sanna is Scandinavian and we have a lot of Scandinavians at the moment but as important is that she studied both Chinese and Yi. This helps us reach out as well to the Chinese languages.

We have asked Barend Mons to join us in the commission. I am really happy he accepted. Barend is a scientist, he is in the field of bio semantics and he has been asked to join in anticipation of the large number of scientists that are likely to join the project once WiktionaryZ starts to include scientifically important data like the UMLS and others. Barend has been instrumental in securing the collaboration necessary to make this happen.

Thanks,
GerardM

Some more numbers

2006-09-05T10:00:00.000+02:00

WiktionaryZ does collect some statistics. We are collecting statistics for the fourth month running and we have two complete months worth of data. Today it is the fifth of September and we have already had more traffic than in the whole month of July.

Now there are "lies, damn lies and statistics" and there is some truth to this as well. The googlebot really likes us. It uses at the moment some 60.29% of the traffic versus 7.26% in July. At this moment it does not hurt us. I wonder what it is doing and how it likes our data.. Maybe it has it's knickers in a twist.

Thanks,
GerardM

Helping editors

2006-09-04T13:01:00.000+02:00

Many people have been working of WiktionaryZ, the amount of new words is staggering. The quality improvements of our first content was profound. But as we now have some 131.000 words and phrases it becomes more difficult to find what needs doing.

It is therefore that we need queries that help us find what we have and help us find what needs doing. There are a few things that come to mind:

All word in a language - with a possibility to start at a given word - with a possibility to select those words where we do not have a translation in another specified language.
All DefinedMeanings in a collection - with a possibility to start at a given word - with a possibility to select those DefinedMeanings where do not have a translation in another specified language.

Obviously these two would benefit our editors a lot. They would also benefit our users ..

Thanks,
GerardM

131.000 records in WiktionaryZ

2006-09-03T22:26:00.000+02:00

Hoi,
WiktionaryZ has some 131.000 records I was told. This is really astonishing. It demonstrates how effective it is to add a translation this way.. The new articles are created at the same time as the new translation is added. When you compare this to the size of the English and French wiktionaries, they are already 170K and 200K big. Many of this content was created by bots. Much of the WiktionaryZ content was created by the import of the GEMET content. This was some 70.000 articles. Given our current 131.000 it means a 61.000 growht .. WAAUW..

Thanks,
GerardM

500 users

2006-08-30T15:32:00.000+02:00

Just a short note: User Olli was the 500th person to subscribe to WiktionaryZ.

The simple wiktionary

2006-08-22T07:47:00.000+02:00

As well as doing things to get WiktionaryZ going, I usually have something like 6 bots running on many wiktionaries. With some regularity I do the smaller wiktionaries too. One wiktionary I really like because of it's crisp user interface is the simple wiktionary. Simple intends to be simply easy to use. It does not want to be difficult; when you want more information that it provides, it just refers you to the English wiktionary. It does not do translations, it does not do etymology but it does inflections and it uses colours and layout to good effect.

When I look at this, I want to have something similar for WiktionaryZ.. Now WiktionaryZ does have translations, it will have etymology and everything under the sun it will have everything and the challenge of having a neat user interface is therefore a challenge.

Today I copied from the simple wiktionary a Basic English alphabetical wordlist. This wordlist originally by Charles Kay Ogden, contains 850 words. There is bound to be a lot of theory behind it, for me it helps me remind me that we need both the simple and those difficult words, the words that people often have difficulties pronouncing like "immunoassay", we need them too, but we will also need a way to define them in such a way that people can understand what is meant.

Thanks,
GerardM

Armenian and Georgian

2006-08-19T10:14:00.000+02:00

Well, yes, I missed two languages. Also

Armenian

and

Georgian

were added :-) As you can see: languages are really getting many, but are still far away from the approx. 7.000 that will be there one day (no, not kidding :-)

It's about new languages again :-)

2006-08-19T03:05:00.000+02:00

I just had a look at a term where I thought I would add a translation and then I saw some more languages ... of course I now had to look which ones were added:

Afrikaans
Akan
Arabic (standard)
Ewe
Haussa (Arabic script)
Haussa (Latin script)

Quite a busy week when it comes to new languages, right? I hope I didn't miss any.

TMX repository

2006-08-17T19:55:00.000+02:00

The first TMX (Term Base eXchange) files were uploaded to WiktionaryZ. Translators use and create these TMX files during their daily work. Of course, confidential information will not be found there, but there are many translations that deal with absolute neutral issues where you don't know who a translation was for and there are plenty of OpenSource and OpenContent projects that can share their translation data in such a way.

This repository can also be very helpful for the localization of Wikipedia contents - imagine all that languages (like many regional languages) - where people have difficulties to write sentences, where alphabetisation level is around 4% ... these can look up how single words were used in other sentences before when they use such files in CAT-Tools like OmegaT. This means that they can imitate correct sentences and improve their language. They will have less hurdles when it comes to write and we obviously can get better quality and more contents in less time.

Then imagine all these countries where people anyway work offline ... they can use the data for their translations and do the same. In many countries, like Africa, online time is very expensive. Working with OmegaT and sharing TMX files they will assure quality and consistent terminology.

I know, we have only two files for now ... I am sure: the repository will grow over time.

Useful .. in some ways yes

2006-08-17T07:57:00.000+02:00

When there is a word that I think should be in WiktionaryZ, I add it. Given that we know of Wikipedia, that people want to know about sex sport and the news, I add words that are in the news. Words like "truce" or "Somaliland" could be found on my favorite news site.

The insanely great thing, is that when I add a translation I know it will stay with that meaning of a word and, that for all the translations I add there is an article to access it for the concept for that language as well. As I want to demonstrate that WiktionaryZ does do other scripts, I have added a fair number of words in Greek, Russian, Hebrew and Japanese. Also a fair number of words in Esperanto but that is a different story.

Having many people add words, means that at some stage you cannot add a word .. it is already there. This means that WiktionaryZ starts to become useful in a VERY modest way. The good news is that the content only gets richer. It is no news that many of the issues with our content will have to be addressed when we get richer functionality..

The thing that truly amazes me, is that we are already at a stage where many ordinary words are there.. in many language..

NB since the last post we added Catalan to the list of languages that are supported :)

Word of the Day

2006-08-15T21:34:00.000+02:00

Today we added the Word of the Day template to the main page. This means that from now on we plan to have a daily change there and will use, if possible, terms that are related to the news. Of course you are invited to have your daily portion of "WZ".

Piedmontese - Ukrainian - Venetian - three new languages added

2006-08-11T17:27:00.000+02:00

As from yesterday evening people on WiktionaryZ now can also work on the languages Piedmontese, Ukrainian and Venetian.

Identical meanings and not ...

2006-08-03T15:49:00.000+02:00

Well, let's take an example that is the easiest way to describe things - the word: cousin

When you look at this word you will find three defined meanings:

- The child of a person's uncle or aunt
- A daughter of a persons's uncle or aunt
- A son of a person's uncle or aunt

Now when translating from English to other languages - let's say German, you have (at least) four translations for the first defined meaning and (at least) two for the other two.

This means in the first case you will for example add the German translations:
Cousin, Vetter, Cousine, Base
and the check box for identical meaning must remain empty.

In the second case you will have: Cousine, Base
And in the third case: Cousin, Vetter

When you look at this now (3 August 2006, 13:49 UTC) you will se that people did not add the translations to the first defined meaning. I suppose this is because it is still not a 100% clear how to handle these situations. (And this is the reason for this note :-)

Definition in UI language, if present in that language

2006-07-28T09:31:00.000+02:00

From today on WiktionaryZ has another nice feature. When your user interface is for example set to Italian and you click on any term (English, German etc.) and there is an Italian definition present you will get that one show up. Only if there is no definition in the language of the user interface you chose an English definition will show up.

What has THAT to do with POV

2006-07-25T21:43:00.000+02:00

WiktionaryZ is a wiki project under development. We want all words of all languages, we want the words of all domains and, we want to collaborate with organizations both companies, universities, NGO's and governmental organizations.

This was easy.. WiktionaryZ is very much a Free/Open project. The software is Open Source, the content is available under a Free license.. Free is very much how we define ourselves. Now when Sabine finds us a resource like this one, we have an important body of computer terminology really rich in it's multilingual content. Consider; 14.000 terms in some 45 languages makes it really important. Given the information about the license we could use it.

Typically we would upload it using the name of the organization responsible for the content.. I think we should, then again some might find this problematic.

Thanks,
GerardM

The Swadesh list

2006-07-24T07:18:00.000+02:00

On WiktionaryZ we are now starting to build our content; we can add new Expressions and new DefinedMeanings. This proves to be an important stage in the development of WiktionaryZ; it is no longer abstract, there are now concrete examples where we need multiple DefinedMeanings like the word German where it is either a person, a man or a woman that is of the German nationality.

With all this functionality, it becomes relevant to give all the activity some focus. For us a Swadesh list is an instrument that helps us to give a clue to the differences between related languages and dialects. This is why it was a first resource to complete. The challenge will be to complete this one list for all the languages that are supported on WiktionaryZ.

There are several reasons for concentrating on a small list of 207 Expressions, it can demonstrate among other things that many concepts share the same expression. When all concepts of all expressions of all languages connected to the Swadesh list are defined as part of this project, I would expect at least a few thousand DefinedMeanings as a result.

Thanks,
GerardM

Adding relations ... for now impossible ...

2006-07-23T01:09:00.000+02:00

Today I corrected a translation into Italian on WiktionaryZ - forbici ... it had to be forbice. Of course, since Italian is not my mother tongue, even if I am sure about a change I check it with other dictionaries. In that way I found several related terms which I then wanted to add as relations. Big surprise: there are only four possible relation types and all are connected to the GEMET multilingual thesaurus that was the first contents of WiktionaryZ. Therefore I could not add these related words, since they simply did not match this scheme.

Then I thought about technical terminology and how a term can be used in different domains ... well I could not even add those right now, since it would require double and triple work afterwards ... or better: consider what happens if we have some 10,000,000 entries and then, for my work I search for a term being my translation of a specific domain - for example the automotive industry. Let's say I get only 50 results for the term I search, all mixed up ... no possibilty to have them filtered by domain ... that would cost me in the worst case 50 clicks to understand if one of the terms matches or not .... considering that clicking through 50 matches, reading the definition will at least take 20 minutes (and that is even fast to my opinion) and considering a price of only 30 EUR/hour it would mean that the search of a single term for a translator costs 10 EUR ... well: that is simply too much - not having domains in WiktionaryZ for practical use means that WiktionaryZ simply would be useless.

Of course there are many other reasons for having domains and I'd say not one reason for not having them ....

I also considered to use semantic web on my wiki ... if these terms were translated with a wikidata structure without the attribution of a domain semantic web would be simply useless for a correct search.

No ... we need it ... we definitely need them asap. And yes, you who are reading this, tell us your opinion - how would domains (like: technics, medicine, biology etc. etc.) be helpful for your use.

Thanks!

What language is H2O

2006-07-18T17:56:00.000+02:00

Everybody with some general knowledge will know that H₂O is the chemical formula for water. This formula is the same in at least all the languages that I speak. Now as there are so many languages that could use the same code, it would be a folly to repeat this and all the other chemical formulae for all the languages. However, it would be equally foolish not to have them at all.

Yesterday I was pleasantly surprised by a WiktionaryZ newbie who informed me about the existence of three special ISO-639-3 codes. They are und for undetermined content, mul for multiple languages and zxx for no linguistic content. As is usual for the language codes, I created a portal page for these codes. The question is; what code to use. I would opt for the zxx code because it is not really linguistic content and it seems to be a universal standard.

I am eager to learn what you think..
Thanks,
GerardM

How to deal with conventions

2006-07-01T10:13:00.000+02:00

There are conventions to indicate certain things. One such is the convention for the the name in Latin of a species; it is customary to capitalize the name of the genus an to have the whole in italics type. This would make it Homo sapiens.

What would we do in WiktionaryZ when we get data from another resource where these names are NOT treated according to the conventions. To me it would be obvious, if a convention is adopted, you do want to move data over in that direction.

Another convention that we are certain to adopt is about the spelling; an expression like MALARIA or Malaria is certainly to be replaced by malaria. In many scientific resources, the first two formats are the convention that is used for expressions. WiktionaryZ will want to maintain these forms in one way or another. One way of doing that is using the table that is currently called MisSpelling. I would not mind when the name of this table changes to become more politically correct.

Thanks,
GerardM

Translation of structural words

2006-06-20T18:40:00.000+02:00

Now WiktionaryZ has some more features ... it is getting closer and closer to what we want. User Sanna started to add definitions + translations to terms like broader terms - and of course so did we ...
Now when it comes to German I don't have too many problems on how to translate these terms, even if sometimes I have some doubts, because they could be translated in different ways ... how do linguists use them? When it comes to Italian, which is not my mother tongue, I have some more problems ... I was not sure how to call these terms ... and decided not to add any translation or definition. Well: we should need a glossary about all these terms :-)
If you happen to know where we can find one, or if you are a linguist and can help us in any language: please tell us and do.
Thanks!
Sabine.

Wordsegmentation .. for Thai

2006-06-10T10:50:00.000+02:00

When you want to work with languages that are not familiar to you, you hear of all kinds of issues with languages that are completely new. The Thai language is one such for me; when you have a sentence in Thai, you do not have spaces between the words and consequently it is not clear to me where I could break a sentence to a new line. This is really relevant for the localisation of MediaWiki; when a text is translated, the translator does not know where it has to fit. It is therefore important to be able to know how to do this.

Luckily there is software , even GPL software that does wordsegmentation for Thai, the next thing is how do we make it available in MediaWiki and, how are we going to use the results.

Thanks,
GerardM

The case of Mandarin

2006-06-03T09:12:00.000+02:00

This is not about the difference between Mandarin or mandarin. Written standardized Mandarin exists in at least two variants they are cmn-Hans and cmn Hant or in human terms traditional and simplified Mandarin. For WiktionaryZ this distinction is crucial because an Expression needs to identified as being one or the other.

For me it becomes more complicated when I find this 普通话/國語/华语 as how to identify Mandarin, I would have 普通话 and 國語 and 华语. This article on the English Wikipedia helps a bit; 普通话 is simplified, 國語 is traditonal script and 华语 is how "standard Mandarin" is called in Malaysia and Singapore (in simplified script, 華語 being the traditional variation). This was "easy" right?

According to this article, Mandarin is linguistically speaking a group of Northern Chinese dialects. The cmn code is associated with this group. This leaves me with a headache. I do not want to use zh or zho for Chinese because that is too broad as it includes languages like Wu and Min-Nan. I can use cmn for standard Mandarin and have additional suffixes to indicate both a specific dialect and its script.

Doing so means that I give the official version of a language the official tag. The problem is that I do not understand the implications when this is made a WiktionaryZ policy.

Thanks,
GerardM

What languages to choose

2006-06-02T09:06:00.000+02:00

I was given the impossible task to select a minimal group of languages that will be part of the first extension of languages that can add translations to WiktionaryZ. Impossible because what languages to select? What argument for selection would be "neutral and objective" and be accepted as such. What exceptions can we make that can be explained and are acceptable?

What we came with is to start with the languages that are have started the localization of their MediaWiki messages. Most of these languages are currently served really well on the BetaWiki. Some languages have not been imported yet here and the ones I know off will be included as well (for instance Thai). Others like Chinese will not be included, Mandarin will be used in stead.

For all the people who want their language included as well, the answer is obvious, at this stage we are truly at the pre-alpha stage we cannot do everything now. When you make the effort and start localizing the MediaWiki software, you provide a powerful incentive to add yet another language .. We will be happy to accommodate you ..

Thanks,
GerardM

Some progress

2006-05-31T11:27:00.000+02:00

It is one thing to mention that an effort is important, but it only is important when you are willing to do something with it. In my previous post on this blog, I mentioned the internationalization and localization effort that is going on for the MediaWiki software.

We have now linked the information about languages and their systemfiles to both Ethnologue and WiktionaryZ. We have also included a category with the languages that have can be understood as languages under ISO-639-3. The next bit will be to link back from WiktionaryZ to this development Wiki.

One of the things that are still to do is deal with the existing localization for languages like Chinese, Arab, Kurdish and Farsi. The codes that are in use in MediaWiki are now in use as macrolanguages. This means that they represent more than one language. Theoretically it is easy to do, just change the codes but practically it is more problematic.

Thanks,
GerardM

The User Interface and project codes

2006-05-30T11:28:00.000+02:00

The user interface (UI) of WiktionaryZ will consist of two parts. One part is specific to WiktionaryZ itself; this part will have data as part of the application itself.. we eat our own dogfood. The second part is the MediaWiki UI. The maintenance has been a big pain, but the last months a lot of great work has been done by Gangleri, Nikerabbit and many others. They have created a great resource where people work on the internationalization and the localization of the MediaWiki software.

For those who do not know the difference, internationalization is making software useful in a multi lingual environment. Given the number of Wikipedia projects, MediaWiki is certainly a project that is on the way to become outstanding in this regard.

There is one problem with respect to the localization; the codes that are used to indicate languages and projects are not the same. Wikipedia uses many ISO-639 codes but also adds codes as and when they feel like it. This has resulted in incompatibility with the ISO-639 code itself.

The correct internationalization and localization is a very important part of creating a multilingual UI for WiktionaryZ. It is therefore gratifying that WiktionaryZ and Ethnologue are now part of the metadata on the implemented languages. The first job will be to link the easy connections, the second will be the problematic connections. The most important job will be to urge people to help with the localization of the MediaWiki messages for their language(s). It will not only improve WiktionaryZ but all MediaWiki projects. It is therefore truly in the interest of all people who use MediaWiki.

Thanks,
GerardM

About splitter and lumpers

2006-05-28T14:27:00.000+02:00

When you are working on the creation of a resource like WiktionaryZ, then one of the biggest issues is the theoretically large amount of different concepts or better DefinedMeanings. There are people (the splitters) who are of the opinion that "clear" distinctions have to be made between the different ways an Expression (word or phrase) is to be understood, another group (the lumpers) are of the opinion that it is "clear" that such a fine grained set of definitions are more confusing than helpful and therefore a reduced set of DefinedMeanings are in order.

This is a recipe for many quarrels; there is an obvious need for an objective way out.

At the LREC 2006 conference, Ed Hovy gave a keynote speech that I really enjoyed and that has in my mind implications on how we can resolve these questions. The key thing is that when a word or phrase has different senses, it should be possible to have a group identify these senses in a corpus and achieve at least 90% agreement. One really nice side effect is, that it proved empirically that typically less definitions proves to achieve better results.

When you have a more limited set of DefinedMeanings, there are also Definitions that are part of either a domain or a resource that has its place in WiktionaryZ. My solution would be to allow these as secondary definitions. They are their to show the different ways a concept is understood and also they provide a link to the resources that are imported in WiktionaryZ.

One thing we will have to experience is how this works out when we consider the multi lingual aspect of WiktionaryZ.. I am fairly confident that by adopting this approach it will help us to reduce the number of instances where the "Identical meaning" flag is turned off.

Thanks,
GerardM

WiktionaryZ Licensing

2006-05-12T08:59:00.000+02:00

GerardM asked me to share my thoughts about the problem of choosing the best licensing option for WiktionaryZ. I have posted them on the wiki and encourage you to add your own feedback.

ISO 639-3 instead of a mixture ....

2006-05-11T16:22:00.000+02:00

Two days ago Gerard asked people on the wikimedia mailing lists if there were objections to a change from ISO 639-2 combined with ISO 639-3 to the use of only ISO 639-3 codes. Well ... we did not receive any objections up to now, so I suppose people agree with us that using only one list of codes makes much more sense and is less confusing. We also talked about this with other people who agreed that the change makes sense. Since doing double work should be avoided it is best to move over asap and this means that we now start to move all WiktionaryZ portals over to ISO 639-3 codes.

Ciao, Sabine

Trolls

2006-05-05T09:25:00.000+02:00

When Trolls become a pest, you do not wish them on the Scandinavians (where they are said to live) . You do not feed them. You make it more difficult to feed on creating an emotional annoyance.

What you do is, you create loads of sysops on WiktionaryZ that can act on the first signs of trolling and, you sadly change the reply functionality of this blog to moderated. It means that someone, me, has to approve the reactions to the blog. I do this gladly, because I do not suffer trolls gladly.

Thanks,
GerardM

The trouble with GFDL

2006-05-03T08:07:00.000+02:00

GFDL includes a clause that says that attribution must be given in order to copy content. It's true in CC-by, too. GFDL, though, requires that the individual contributors be named. For documentation, which was the original purpose of the GFDL license, that makes sense. For projects with more content and more contributors, the requirement to name individual contributors becomes a burden.

One interpretation of GFDL suggests that the use of a particular Wiktionary entry requires attribution of all its contributors. This approach would paralyze the free re-use of the data in applications such as spell-checkers. Another approach would be to handle the attribution en masse, such as by including a single list of contributors to imported data (perhaps with edit counts) without tracking who contributed what.

The GFDL license also does not consider the attribution of organizations. We do not know, for instance, the names of all the individuals who contributed the GEMET data. WiktionaryZ has partners already, and will have many more as it grows.

In order to proceed smoothly, we propose the use of the CC-by license. At this stage, we welcome discussion. If you agree with this direction, please state your agreement to license your own contributions under CC-by on your user pages on WiktionaryZ and your home Wiktionary.

Edit on the relational data of WiktionaryZ

2006-05-01T07:45:00.000+02:00

The first edits on WiktionaryZ have been done. This is an exiting time; not only will we learn how will the (pre-alpha) software works, we will also have to make sure that the concept of the DefinedMeaning is ingrained into the ideas of the editors. This means that the flag that indicates that an "identical meaning" is present in the synonym or translations is turned off when the translation is problematic...

Well, actually at this moment my overriding feeling is one of joy, all these problems will sort themselves. Now we can think in terms of solutions; we can add/edit Definitions and add translations and synonyms. Now we can demonstrate that the relational approach in a Wiki environment makes sense. Now we can make a difference .. :)

The first thing we "solved" is expand the list of languages that can be chosen from. To the list that comes with the Gemet, I asked to have Chinese, Russian and Neapolitan added. For these three languages we have people that will do right. They are just the first few new languages, as you know; we want them all..

Thanks,
GerardM

Competition - Part II

2006-04-27T18:27:00.000+02:00

In the previous post on the blog I wrote about the new competition. It turns out that the organisation behind the competion has competition and when you are that competion, you compete. What I fail to see is the relevance of the latest entry. In contrast to both ProZ and the logosdictionary from Logos, its aims only to be there for translators. Where they ask people to collaborate the registration makes it plain that you have to be a translator to qualify.

I wish them well, I hope they do it for there own good reasons. Collaborate ? Maybe, when they show added value by providing a public service. At this time it seems like a "me to" thing.

Thanks,
GerardM

Competition

2006-04-23T09:48:00.000+02:00

Competition is seen as the mechanism that keeps organisations sharp. It drives innovation and it helps build a healthy ecosystem that drives innovation. WiktionaryZ now has its own competition; ProZ, the organisation of translators, has introduced their new service called Wikiwords. Wikiwords builds on the foundation laid by the KudoZ system; this is a system where translators help each other out with problematic words or phrases. The system has been in place for many years and resulted in a rich resource.

The new system is in its first iteration so it will be interesting to see how they will fix the issues we face in WiktionaryZ. One of the issues I discussed with Henry is the importance of definitions. A recent new concept is the English word Load, it was translated to the Macedonian Вчитува but as there is no definition at all, I do not know if the translation to Dutch of lading or laden is applicable. Another issue is that I would expect the word Load to be written like load.

Obviously, this is the first iteration of the software and I do wish ProZ well with their new project. It is however particularly in the first month that the policies are set and I wish them wisdom in setting the right tone and generating interest.

In the FAQ it is made clear that ProZ has not decided on a license yet. I hope that they will settle on the GFDL, this will allow the exchange of data, it will allow us to duke it out on technology, learning from each others approaches and at the same time collaborate on content.

I really want this friendly technological fight of who can provide the best technology and the best user interface while the data is there to be shared by all.

Thanks,
GerardM

When to try to stop a running train

2006-04-15T18:29:00.000+02:00

WiktionaryZ is about to become editable. Not as in next month, not as in next week (seven days) but as in this week. It is a moment that fills me with anticipation. It is a moment that will teach us what will work and what won't. I want it badly.

Still, I do want Erik to make one change before we can add translations and synonyms. Without it I prefer not to have it...

What is so important that I want to stop the momentum, even if it is for a day or two? It is the flag that indicated that a translation is a good translation; it is what I call the flag for the IndiginousMeaning.

For the understanding of what a DefinedMeaning, this flag is absolutely essential. The biggest struggle for WiktionaryZ will be to make people understand what this means and how it limits what can be done.

It is truly key that we have this from the start. I hope you forgive me and, that you can wait the extra day.

Thanks,
GerardM

Babel templates user level 5

2006-04-12T00:16:00.000+02:00

Today Nautilus, a user of WiktionaryZ left me a note on my talk page. He proposed to change the babel-5 template from something like "This user speaks English on a professional level" to something like "This user makes use of English for professional purposes".
Now maybe you wonder why that ... well, when it comes to translators it happens that a person is 100% proficient in reading highly qualified technical texts and is a great translator for that, but the actual level of the language would be to be cosidered as 3 and not 4 like most people would assume. Well, speak a language on a professional level does not have a specific meaning. So for people who use a language for work the proposal "This user makes use of English for professional purposes" makes a lot of sense. I would say we should adapt this, so that this becomes clear. Maybe we should also have a look at the other templates in order to make sure they do really reflect a standardised meaning.

Thanks,

Sabine

The Swadesh list is becoming relevant for WZ

2006-04-11T10:34:00.000+02:00

The Swadesh list is this list of 207 English Expressions designed by Morris Swadesh to determine the closeness of a pair of languages. As it is a known method of determining exactly this, it makes sense to use it as one of our tools as well.

There are several things to having a Swadesh list; first of all if someone comes along and wants to make a point about a language, a dialect it is a perfect tool to separate the people that do from the people that whine. I know for instance that Westfries is a dialect, I lived in the region of Westfriesland for my younger years but I cannot create a Swadesh list for Westfries. The few words I know, I do not know how to spell and as a consequence I will not champion Westfries; I am the wrong person for that.

When the Swadesh list is translated, you will also find that there are synonyms, Expressions representing multiple DefinedMeanings. These in turn can also be translated and that would be in my opinion be a perfect tool to see how well these in turn are translated to all the other languages. This would be a perfect tool to see how active a language community is.

Now at this time three Swadesh projects on WiktionaryZ have started; one for the Neapolitan language, the Sicilian language and for the Greek language.

Thanks,
GerardM

Babel templates for 255 languages and counting

2006-03-22T23:42:00.000+01:00

In WiktionaryZ we are preparing for when we have editable (relational) content. The Wiki is doing well; we are creating the environment for when we are technically able. When we are, we will slowly but surely invite more and more people to edit.

You can indicate your interest by creating a user and adding the Babel templates for the languages that you are interested in. We use the "Babel" templates that are also used by the English Wikipedia. We do however always allow for "professional users" of a language (think writers, translators and teachers).

We want to use the ISO-639 codes, and even tough we allow for ISO-639-3, we already have situations where we would like to have more languages or good codes for dialects. From this perspective, it would be great to have the "Wiki for Standards" now we have to use our personal contacts and that is much less convenient for many.

Thanks,
GerardM

Using Babel for WiktionaryZ for the coming functionality

2006-03-13T12:46:00.000+01:00

WiktionaryZ is a work in progress. Slowly but surely we will add functionality and content. We have started preparations for the next phase. The next phase is for us to be able to edit the relational content of WiktionaryZ.

Functionally WiktionaryZ is at a pre-alpha phase; this means that we do not have the full functionality available that is in the specifications. It means that when we add functionality it will not be available to everyone. In order to have some control of what happens, a limited group of people will be invited to edit the relational content. We will invite from the people that created a user on the wiki and provided their babel information.

The babel information allows you to give six levels of proficiency:

xx mother tongue
xx-1 basic ability
xx-2 intermediate ability
xx-3 advanced level
xx-4 near-native' level
xx-5 'professional' proficiency

The xx has to be replaced with the codes that you can find on the babel page. At this moment we have the codes for more than 100 languages. This equates to something like 1% of the languages that exist. We do invite you to help us with templates for the missing templates. You can help by checking and or translating the following text :

* This user is able to contribute with a basic level of English.
* This user is able to contribute with an intermediate level of English.
* This user is able to contribute with an advanced level of English.
* This user speaks English at a near native level.
* This user is able to contribute with a professional level of English.
* This user is a native speaker of English.
* These users speak English.

You can help by creating the templates or, by providing us with translations. Translations are best given to us here.

Thanks,
GerardM

Names and Etymology ... and what about old languages?

2006-03-09T22:46:00.000+01:00

I am in Germany with my parents right now - this evening my mom watched "Deutschland Deine Namen" (Germany: your names - I know it is a very literal translation ... maybe it should be "Names of Germany"). But that brought me to the "what to include" question ... and to translations. We will have names of well known people in Wiktionary - only think about the old Romans ... their names have been translated. And today we transliterate names from countries that use other chars than we do. So yes, names must be in WiktionaryZ, because names are still and have been translated in past - and for sure they will be also in future.

But what about the Ethymology? Many names came up about 600 to 700 years ago and refer to old forms of languages they most of all refer to professions of people, places where they lived, habits they had etc.. How do we do that? Just add the Ethymology and that's it or do we then also add the real word these names come from and give it a language name? To me it makes sense to add these languages as I also recall the times when I was at school and we had texts in Middle High German and Old High German ... if you think that knowing German you would be able to understand such texts: hardly or no way. So I would say that integrating such words with definitions makes sense ... but in which language should the definition then be? hmmmm .... not easy, right?

Well these are just two very fast thoughts. I am really quite under time pressure, but I thought it would be good to just write down some lines to remeber that these things need some consideration in future. Please do not consider this to be a 100% error free posting ;-)

Thanks, Sabine.

About sorting or is it collating ?

2006-03-08T22:52:00.000+01:00

When you have a dictionary, you expect words to be sorted in such a way that you can find them. Sorting is obvious right ? You use the alphabet and that is the end of it. According the article about the character Ĳ, the character is to be found in between the X and the Z together with the Y. The best Dutch dictionary Van Dale, has the ij sorted in the i range.

The consequence is that how to sort is not obvious. It does not follow from knowing the alphabet for a language and yes, the Dutch alphabet is different from the French or the English alphabet in the same way as the Farsi alphabet is different from the Arabic, the ....

In my personal blog, I wrote about what I would have a programmer do when I had money to spare. I mentioned this to Gangleri and jested that I would expect his choice to be working on right to left issues and sorting issues. I am really happy that he took it as a challenge and he made me really happy with the template wikivar. I do not understand all the ins and outs of it, but what I do understand is that he is asking people to help with defining the sorting order that makes sense for that resource, for that language.

One thing is, this sorting order only makes sense when you know the language the articles are in. For WiktionaryZ this will be obvious; WiktionaryZ will be language aware. For the current Wiktionary projects this is not possible; Dutch words, German, English and Farsi words are all together to be sorted. The current search routine is not that great, it does not allow for case insensitive sorting a wish that many would really like to see realized.

With [[Multilingual MediaWiki]] it will be possible to make the traditional wiktionaries language aware. This in turn will allow for the sorting of words according to how it is done for a language, for a locale. The question that I have would be; do people have the stomach to go for this. The good news is, that most wiktionaries become more and more structured. This makes it feasible for bots to do a good job.

The irony when it happens, the technology will be courtesy of the WiktionaryZ project. Then again, given its definition of success, it would be considered a success :)

Thanks,
GerardM

About characters ...

2006-03-05T10:57:00.000+01:00

Everyone that has learned the Dutch language knows the alphabet. The Dutch alphabet ends with: ..x, ĳ, z. That means for Dutch is not ..x, y, z. The "Y" or "y" is called the "Griekse ĳ. This is what you need to know to understand this post.

The problem is, when people type, the "ĳ" is not on the keyboard. Typically people use the US-International keyboard. And when you type "ij" It looks the same, it must be the same except for the fact that it is NOT the same. When you use a wordprocessor a sentence would be spell checked to read like "Ijs drijft in de sloot." not "IJs drijft in de sloot."

In WiktionaryZ we do want to do things RIGHT, except everybody DOES use the i and the j not the ĳ. How should we deal with issues like this. I tend to agree that it is a problem and ignore it. At some stage we will create the software that does these things on the fly.

I can have this opinion because it is my language, what if the same question is asked and it is not my language? How will we resolve them? To what extend will we be dogmatic, to what extend pragmatic??

Thanks,
GerardM

Fonts

2006-03-02T15:12:00.000+01:00

This week I had the pleasure to see a number of questionmarks where I should see characters. Where I should have seen the name of a language in that script. It seems so obvious that when you have a computer, that you can see all the information that is available.. It is however much less obvious than it seems.

When you use the English Wikipedia, it is normal to include the names of countries and languages in the script. Many people do not have the necessary fonts to see the characters that are in many languages. They get like I did question marks. For WiktionaryZ we will have a similar situation but on a completely different scale. We want all words in all languages and, a word like water, air, fish, bird can be expected to be translated to any language. This will make it even more relevant for WiktionaryZ to consider fonts.

WiktionaryZ will be in UTF-8 but that only helps in so far as fonts exist in the first place. Also UTF-8 is a living standard, a new version; 5.0 beta 2 represents the latest developments. As we want to have both user interfaces and content in all languages, we sure are going to have our share of issues. With 5.0 we will have better bidi support, that will be a real boon.

It is easy to recognize that people do not have the fonts to see all Wikipedia and Wiktionary content. To solve this, we could provide the best information on the fonts needed for particular languages, scripts. When we start doing this, we will help people. It may also open several cans of worms. Then again, we could go whole hog and offer fonts to download.

PS this article had me start looking into fonts again.. a good and interesting read.

Thanks,
GerardM

Admins or Trusted users

2006-02-20T13:25:00.000+01:00

Given what WiktionaryZ is about, we should appreciate that when you use a word to describe something, existing usage implies all kinds of things. Within the MediaWiki software we have things that are typically called "bureaucrats" and "admins". On the Dutch project the word "moderator" is used in stead of "admin".

The function of an admin is quite specific; an admin is trusted with extra functionality that help to manage the project. This functionality allows for the roll back of changes, for the deletion of content and for the blocking of users that have vandalised the project.

The function of a bureaucrat is also very specific; a bureaucrat can make a user an admin or a bureaucrat and it will also become possible to mark a user as a bot.

Technically, both these tools should be given to people that are trusted. People that are trusted to perform these tools for what they are intended to do. Technically it does not mean at all that any special expertise about subject matter is implied by the status that these titles seem to imply. It is therefore that I am of the opinion that the names of these functions are misnomers, they do not adequately reflect what they are. The name "trusted user" would better reflect what we call an admin. A bureaucrat is nothing but and admin; when consensus has arrived that someone is to be trusted, it is acted upon by the bureaucrat.

Within the wiktionaries and certainly within WiktionaryZ, there is another group that will be more relevant they are the people who have expertise. This expertise is something multi faceted; it does not take much expertise to add many translations to a phrase like "Olympische Winterspelen". I have added some 20 translations, but I do not have the expertise to say that they are correct. I have written a lot about WiktionaryZ so I do have a certain type of expertise about the subject of terminlogy, lexicology and thesauri.

Most of the time when people work on the content of a wiki project, the admins have a good feel to distinguish the good from the bad. Typically it is rather obvious when it is bad. You do not have to be much of an expert to do this; you have to be trusted to give a consistent best effort. That is all.

My proposal is therefore to rename "admin" to "trusted user" and "bureaucrat" to "admin" for WiktionaryZ. When people have a minimum of say 500 or maybe 1000 good edits, a language community can be asked if they have objections. When there is no objection, the admin for that language community just does the deed.

When someone proves that he is not to be trusted as a vandal fighter, the person is first warned and when this proves not to be enough the trust is revoked and the person has to win back the trust.

This does not mean at all that the person is less of an expert in his subject matter, it is just that he is not trusted as a vandal fighter. I am sure that in time we will create functionality that is best trusted in the hands of people that know a language well. These tools are different tools from the current vandal fighting tools, there may be some overlap however ..

So what do you think ?

Thanks,
GerardM

2006-02-19T01:42:00.000+01:00

Elements of a Great Dictionary: Definitions

The definition is what many would consider the heart of a great dictionary.

It is best to write definitions for words that you know, especially at first. To write a definition, I prefer to begin with an example of how the word is used. I may spot this example in the "wild", by reading it somewhere or hearing it in a conversation. I may seek out an example, an easy task in an age of search engines. If I'm fairly familiar with a word, I may just make up a sentence using the word. Regardless of the source, this example gives me some context.

Let's take the word "cakewalk", which has no definition yet, as I write. This word has a few different senses in English. The one that's familiar to me would go in a sentence like, "Since I have studied English for years, that English quiz should be a cakewalk." (If it's a good sentence, I'll use it later, for an example in my article.)

Now, I try to imagine that I am explaining this word to somebody who hasn't heard of it before, especially somebody who is trying to learn English as a second language. I look back at the example sentence. What did I mean by this word, in simple terms? I meant that the quiz would be easy. So we have that definition: something easy.

At this point, I start thinking of some synonyms for this word. Easy and simple both fit this sense. Basic, insulting, and silly are all a bit outside of the sense of this word, but it's okay to feel around for meanings at this point. It's easy to cross off some extras that don't fit later, and the process of thinking of them might lead to other good ideas. You can also think in terms of antonyms, or phrases, such as "not difficult" or "not presenting a challenge."

One thing that also seems to help beginners is to use the word in a sentence: "A cakewalk is something that is not difficult or does not present a challenge." If it helps to think in these terms, please do. When writing the definition itself, though, leave out the part "A cakewalk is..." or "This is a word meaning...". This is a dictionary, so there's no need to reiterate that the meaning is what's to come.

So the definition goes like this, then:
cakewalk - 1. Something that is easy or simple, or does not present a challenge.

Now, the term "cakewalk" has a history, including other, earlier meanings. If you didn't happen to know that, just put in the definition(s) that you can. The beauty of a wiki is that somebody else can expand an article later. I happen to recall that a cakewalk was originally a contest, and is also a form of music, but I never learned the details of those contexts, at least not well enough to write a proper definition. This is where some research might come in handy.

Wikipedia has an article about cakewalk. (Many other sources do, too, but it's important to be careful of copyright status, so I prefer to check free and open references first.) It turns out that it was a dance contest, and the dance and the music associated with it. The prize was a cake. Part of this information goes in additional definitions, and part goes in the etymology.

I will talk about the other parts of this article as I continue this series, but you can see the finished product of this thought process here.

2006-02-17T14:16:00.000+01:00

Glossaries from WiktionaryZ ... and well ... some features that help to improve the contents of WZ

At this moment I am translating the article about the XX Olympic Games from Italian to Neapolitan. A thing that seems to be easy, but that is not ... many terms of sports do not exist in Neapolitan dictionaries and very often I need to search quite a long time to find the term that fits. For translation, in order to allow for an easy update I use OmegaT since it makes sense to re-use the previous translation when I then pass on to update the text online. For small wikipedias it makes sense to work this way, since writing own articles often takes longer. Being a translator makes things easier for me as well, since I simply use the tools I always use.

OmegaT has a glossary function that is already very helpful, but it is not what it could be ... at this moment I am adding new terminology for my glossary here: http://nap.wikipedia.org/wiki/Utente:SabineCretella/lista_di_parole_IT-nap since terms must be approved in some way and available for all to use within OmegaT. Now to integrate them in the actual wiktionary you need to create the pages manually - one by one ... that takes time ... then you need to update your offline glossary, the OmegaT glossary I mean, in order to get the proposals of the terminology to use ... and this means that you have to convert the above page into OmegaT glossary format each time you update. If there are corrections to be made you need to do that in the glossary list and on wiktionary (if terms are created there) ... well WiktionaryZ together with OmegaT can resolve this problem that costs a lot of valuable time that could be used for creating new contents instead of repeating work over and over again. Therefore we thought about a reference implementation for a translation glossary - that would be THE solution. Instead of adding the words manually I just click on source + target and add it to WiktionaryZ and when I go ahead with my translation I can or update my glossary on my local machine or work directly with WiktionaryZ. That would make a huge difference ... loads of time saved, higher quality and coherency in the used terminology ... it would be simply great. Well then there is another feature that would easen work a lot ... the assemble from portions option like DéjàVu, a commercial CAT-Tool, has it. This would mean that while you translate the Wikipedia article OmegaT does not only propose the term to be used, but it already overwrites it within the translation ... if assemble from portions is used effectively it can help you a lot, in particular when translating pages like those of the calendar where you often have parts of prhases like "politician and sociologist" or "musician and componist" and whatever.
Hmmm ... I'd love to have all that now ... it would help so much with creating contents.

Sabine

2006-02-17T05:13:00.000+01:00

Elements of a Great Dictionary: Spelling

Once we have decided that a word belongs in a great dictionary, we must add it by selecting a character or a series of characters that represent that word. That is, we must select a spelling, or more specifically, an orthography for that word. In most cases, this is not a difficult task. However, orthographies vary. In English, there is frequently a difference between US and Commonwealth spellings, for instance colorize/colourise. Spelling may also vary with time. It is no longer common to add a hyphen in "to-day."Since a great dictionary does not lack space, we can treat these simply as separate entries (marked with "archaic", "dated", "UK", "US", "rare," etc. as appropriate), and link them as translations or synonyms, as we like (though this has been the subject of some debate, already.

Orthography also encompasses such entities as diacritics and ligatures. Diacritics can alter the pronunciation and even the meaning of a word. In Spanish, sí means yes, but si means if. English employs few diacritics and ligatures, and those that it does employ tend to vanish over time, as in rôle, coöperate, and encyclopædia. When diacritics create a spelling variation, we should handle them simply as variant spellings.

This issue, though, may be more technological more than linguistic. Computers need character sets installed in order to display or enter special characters, ranging from diacritics to the entire scripts of some languages. Many users have difficulty displaying and entering these special characters, resulting in mojibake. I'm sure it's no coincidence, for example, that the Tamil Wiktionary has a link in English right at the top for help with character display.

For those who don't happen to have a button for æ or û on their keyboard, a great dictionary should have a palette of special characters in software, such as the examples here or here. It should be accessible from the search interface, as well as the edit screen. It should be easy to use, and flexible, so that one doesn't have to hunt past all of the Greek and Cyrillic alphabets just to find á. In fact, the set(s) of characters to display by default should be configurable in the user interface.

A great dictionary should also recognize that users do not always spell a word correctly. Since a free, digital dictionary may be used for spell-checking, we do not want entries for misspellings, unless we have some very clear way to filter them out. Recognizing, though, that one of the function of a dictionary is to provide correct spelling or orthography for those who don't know it, the search function should be broad enough to retrieve "definitely" if a user types "definately", or to retrieve "crème" for "creme" or "créme". Do you spell perfectly, or would you like a little help?

2006-02-16T16:39:00.000+01:00

Elements of a Great Dictionary: Words

Now that we have identified a few items that don't belong in a dictionary, let's start with the basics of what does belong: words. A great dictionary, seeking to catalog all words, must begin with words themselves, or more precisely, with lexemes.

Linguists and writers of dictionaries distinguish between words and lexemes because they are not the same. Most words are lexemes, though a few (such as lickety, hightail and handbasket) have little or no existence outside larger phrases. More commonly, a phrase may comprise a lexeme, when it has a single, idiomatic meaning. When we say in English that something is "old hat," we mean not that it is headgear of some great age, but that it is familiar and well-practiced. Phrases of this sort also need special attention to translation. By contrast, a phrase like "Greek history" merely describes history of or relating to Greece, so we need not define more than the component words, in such a case. Of course, language can be subjective, and the distinction is not always clear.

What shall we do with words hovering on the edge of a language? Most people, I think, would agree that paper, archaic, and horsefeathers are English words that belong in a great dictionary. Some words, though, are not so clear. How widely must the term ginormous, for instance, be used before it is deemed a word? What about l337? In formal writing, the term humongous is rejected as a non-word, yet most native English speakers have heard it and used it colloquially, and would agree that it means "very large". How long must words like metrosexual or astroturfing exist before they merit inclusion?

In some languages, a government-appointed academy decides what words exist, and how to spell them. A great dictionary should have some means of flagging words that are accepted in this manner. In other languages, including English, publishers of dictionaries set standards of word-worthiness by choosing what words to include in the limited space in their dictionaries. Either way, they employ some arbitrary standard, such as how long or how frequently a word has been in use.

Whether academies, publishers, or nobody prescribes standards and proper usage, no language is static. People reuse words for novel meanings, or invent new or combined words, often as slang, at first. New technologies and concepts need names. A great dictionary is inclusive, and it should reflect the shifting language, though it should caution users about questionable words. At the same time, a dictionary should not seek to introduce new words, nor words that are used only by three friends at a certain school.

There must be some basic standard, or at least guideline, by which to judge the existence of a word. Search engines offer an excellent (though by no means certain) tool for determining the how popular or widespread a word is, at least in print. The existing Wiktionaries use a combination of guidelines and consensus. In WiktionaryZ, the community surrounding each language should decide the answers for itself.

2006-02-15T05:21:00.000+01:00

Elements of a Great Dictionary

As a writer of dictionaries, I do a lot of work on definitions. Let's start with this question, then: what is a dictionary? Particularly, what makes a dictionary great, and what should a great dictionary be? We're trying to develop the world's greatest dictionary in WiktionaryZ, so I think these are reasonable questions to ask.

It's one of those questions we all think we know how to answer. Obviously, a dictionary is one of those big, thick books you haul out when children aren't quite tall enough to sit at the dinner table!

On closer inspection, though, the answer is not that obvious, so I'd like to take a closer look at what a dictionary is, and what a dictionary is not. I'll start with a few words about what a dictionary is not. A dictionary is not an encyclopedia. In an encyclopedia article, you should find information about a particular subject, for instance, paper. An encyclopedia might have a history of paper and paper-making, and its influence in world history. It might have information about different types of paper, or standard paper sizes. A dictionary, by contrast, will have information about the word paper, itself: etymology, pronunciation, meanings, and so on. There may very well be overlap. An encyclopedia article may begin with a brief description of what something is, or a word's history. Similarly, a dictionary may contain a few sentences about a subject to clarify a definition.

A great dictionary, furthermore, is not made of paper. Not these days. In an electronic age, a great dictionary should not be limited by the size of a bookshelf or a binding. Rare words should have their place. Complete details about words should have their place. Size and weight are not the only reasons to build a non-paper dictionary. A non-paper dictionary need not contain words in some linear order, and it can organize words by their relationships to one another. It can also retrieve information by an electronic search, either on headwords or on content. The information in it, if correctly structured, can be used for other purposes, such as spell-checking, language exercises, and machine translation. A non-paper dictionary can include multimedia content, too: audio pronunciations, images, and video, to name a few. I'll revisit these topics in subsequent entries.

Finally, a great dictionary is not exclusive. It should not exclude words, even rare or "objectionable" ones. It should not exclude any willing, sincere editors. Perhaps most importantly, it should not exclude users. The Oxford English Dictionary, the largest and most comprehensive dictionary currently existing for the English language, is exclusive in several ways. At $1500 or £850 for the print edition, or $295 for an annual online subscription, the cost is prohibitive to most users. It excludes words that are not English. It is copyrighted, so it is not free for derivative uses. Although scholars from throughout the world have helped to build the OED, it does exclude editors.

Over the next few entries, I'll explore more of what makes a great dictionary. I welcome your comments along the way.

2006-02-07T12:06:00.000+01:00

Translations in WiktionaryZ will seem odd to people. Adding translations will be in many ways be against what people always do; people translate a word from one language to another language and yes, they take into account the meaning of the word in both languages. In WiktionaryZ this is not exactly how it is done.

In WiktionaryZ, a translation is added to a DefinedMeaning. The best way of understanding what the DefinedMeaning means is by looking at its definition. When the Italian word "cavallo" is added as a translation, the person adding the translation may know the word "horse" and "Pferd" and as a consequence put it in with the right DefinedMeaning. When this is associated with the Dutch word "paard", it is relevant to know that paard has multiple meanings and only one is this animal that you can sit on, another is this family of animals that all look like this animal that you can sit on ...

The point that I am making is that it is really relevant to translate the definition of the DefinedMeaning because that is what allows people to link new translations to the right DefinedMeaning. When definitions are newly formulated, there will be descrepancies between the definitions, these will sometimes be enough to warrant multiple DefinedMeanings.

Thanks,
GerardM

2006-02-01T19:03:00.000+01:00

WiktionaryZ will not be the same as any Wiktionary. There are many Wiktionaries and they all are different. They all have their own strong community, policies content and history. The great thing about the Wiktionaries is that they all evolve. In a way this evolution follows paths similar to the ones known from the Wikipedia projects; it starts off with a group of like minded people and over time this group grows, friction grows untill it becomes unpleasant.

WiktionaryZ will be structured and this structure is rather rigid. It is not something that one buzy beaver can do or undo with a thousand edits; information must fit in the structure that is the same for everyone. This does not negate that every language is different, that information of a type specific to a language needs to find its place. It does however mean that conventions used in paper dictionaries are indeed conventions for paper dictionaries. It is more relevant to learn from the TST-Centrale, the treasure trove of the INL, the Institute for Dutch Lexicology. They have mulitiple resources on-line and enrich the content by linking the diverse sources. We will be "stupid" and integrate diverse conten. We will probably regret this tendency at times, but this integration has its own rewards.

WiktionaryZ is intended for all languages, the paradox is that it will take fewer resources in an less well known language for WiktionaryZ to become relevant than it will for a language like Dutch or English. These languages have their great lexicons, WiktionaryZ will get its relevancy for these langages because it will be a digital resource and not so much dead wood.

Thanks,
GerardM

2006-01-31T21:51:00.000+01:00

Microsoft Glossaries and similar g...s

Well today again I read a posting in a forum concerning the Microsoft Glossaries that were available on ftp://ftp.microsoft.com/developr/msdn/newup/Glossary as to the answers from colleagues it really seems that Microsoft has definitely blocked access to their glossaries. Now translators needed them when they did localisations (well they really still do need them) for Windows based software since coherency of terminology within several applications is important.

Now what would be interesting to know: I for example localise software from Italian to German and from English to German and sometimes even German to Italian or English to Italian in co-operation with a colleague. Now we of course maintain our own glossaries. Should it now happen that we need to translate software for Windows we will very likely exchange words - we will ask each other: how is this option called on your system etc. and create our own Windows related glossaries .

What would be very important for me to know is: would there be problems if we attribute a category called "Microsoft" to one of our private made glossaries that we make available to all under GFDL on WiktionaryZ? I mean: Microsoft is a registered trademark, but all translators localising software for Microsoft OS need /create such terminology or should we need to use another name that may not say "Microsoft based terminology" but something like "M-based terminology for software" - how is the legal status for such stuff if we really create the wordlists ourselves?

We will have similar questions over and over again I suppose ... for Machinery (SCADA), Apple etc ...

Comments and hints on how to treat such terminology would be very much appreciated.

Thanks!

Sabine

2006-01-28T18:53:00.000+01:00

Collaboration

WiktionaryZ aims to allow collaboration with other organizations. Even at this early stage, we have made contact with universities, translation organizations, and lexicographic companies. All of these organizations have seen the value of free information and an expanded, worldwide staff of editors.

On the surface, these collaborations are great news. Many of these organizations can offer a wealth of data along with money and manpower to assist with importing it, hosting it, maintaining it, and making necessary modifications to the architecture.

Collaborations will raise some questions which the Wiki community has not yet had occasion to address. Of course, we will make it clear that organizations must contribute, as anyone, under a free license, and must play by the same rules.

For their contributions, organizations will want attribution. Careful observers may note the "partners" link in the left margin of the prototype database. To prevent it from growing too large, the idea is that this section will display links to a handful of recently active partners, perhaps five, at any one time.

Many of these organizations also have professional data to offer. Without going into detail on what constitutes "professional"*, the idea is first to give credit to organizations that contribute data, often data that would require knowledge beyond that of most volunteers, such as unusual languages and dialects. This would probably take the form of a decal with a logo and a link to the organization.

The question this raises is twofold. A logo and link will smack of commercialism to some contributors in the wiki community, but done cautiously, it could be a simple and appropriate concession to organizations who contribute. The other question is a bit trickier, and may require a change to the architecture. By taking responsibility for a particular version of an entry, an organization would give its "stamp of approval" to a particular version of a piece of information. On the one hand, this lends authority and credence to that information. Critics have raised the question of accuracy in a body of data that anyone can edit, a valid concern. The stamp of an organization or of a trusted contributor could serve as a mechanism for review of content. On the other hand, it could intimidate or alienate volunteers, who may feel that an amateur or enthusiast does not have the authority to correct or improve upon a "professional" entry. The perception will persist, despite the fact that many organizations see the distributed editing of WiktionaryZ as a way to improve and expand volumes of data that have grown too large to maintain in other ways.

In my opinion, a healthy WiktionaryZ would include a volunteer community alongside organizations for mutual benefit and greatest volume and accuracy of data. What do you think?

— Dvortygirl

*In a nutshell, "professional" data may be just as incorrect as any other, and Wikipedia has already demonstrated that volunteer efforts can be as conscientious as traditional publications.

2006-01-25T22:12:00.000+01:00

The relation between [[WiktionaryZ]] and the [[Wiktionaries]] is one of those thorny issues. What is the relation.. Do the Wiktionary have to be ended and merged into WiktionaryZ ??

This is a question that comes back time and time again. It is a question that can be answered in many ways and it will probably be answered many times..

The answer from the WiktionaryZ is: It is not for us to decide. It is for the Wiktionary communities to decide what they want to do once the content of their project is merged into WiktionaryZ. It is for the board to decide if they allow new Wiktionary projects once WiktionaryZ has reached its full functionality.

WiktionaryZ will attract people who are currently active in the Wiktionaries. It is each individual persons choise what he or she spends his valuable time on. For the WiktionaryZ project it is extremely important that we will attrackt wiktionarians. Wiktionarians are important because they understand what it is that makes sense to have words and more in a wiki. They understand how to deal with cranks that decide that a word like [[exicornt]] should exist. They are people interested in languages like Bambara, interested in phonetic descriptions, interested in thesauri. There are so many things the current Wiktionarians are interested in, it will keep us on our toes to make sure that we will have interesting content.

Thanks,
GerardM

2006-01-21T22:34:00.000+01:00

Quality in a dictionary is expensive. This is something a Chinese dictionary found in the most negative way. The Xinhua dictionary is facing a suit because one of its readers found more than 3.000 errors. Mr Chen Dingxiang studied the dictionary since 1998; he is looking for some money for the time that he spend on his research and an apology for what he considers a substandard work.

The thing that is sad to me is that a person who has the inclination to work on dictionaries and did so much research did all this work without it leading to improvements in the dictionary. WiktionaryZ will have 3.000 issues, but everyone who recognises one will have the EDIT button in the top left hand side and will be able to make it 2.999 issues.

Thanks,
GerardM

2006-01-19T09:17:00.000+01:00

In all the Wikimedia Foundation project there is one big bias. It is not a bias that we want, but it is there nonetheless. The WMF projects are Anglo centric. In many ways it is a good thing, English is effectively the lingua Franca of these times. In Uganda for instance much of the university education if not all is given in English; English is the official language of Uganda !!

In the WiktionaryZ project we want to have all words of all languages. This is a great idea and it comes with many challenges. In this blog entry I want to focus on Africa.

When WiktionaryZ is to be relevant to Africa, the content of WiktionaryZ has to be relevant. Relevancy can be achieved in many ways; it can be because there is nothing else, it can be because we include glossaries and thesauri for specific topics and it can be because we have a complete dictionary for a language.

The fun thing is that with much content in African languages, we get translations in a Western language, typically English. The consequence is that the African languages are likely to have perpetually less content than many of the Western languages.

One problem with terminological projects is funding.. Many projects do not happen or are abandoned before completion because of lack of funding. The reason for this is obvious; it is very expensive to create quality terminological or lexicological or thesaurus information. One benefit that we hope to bring into the equation is the community and consortium effort we hope and expect that this will bring the cost down.

There will also be a difference in aproach; often people do not publish content because it is incomplete. This is against the Wiki idea; here you publish what you have and together you improve the content. When people learn that WiktionaryZ is there, that it has some relevant content, they will want more relevant content. There are plenty of people in Afica that CAN help.

Thanks,
GerardM

2006-01-17T17:59:00.000+01:00

As Gerard has pointed out, Wikidata and WiktionaryZ development is structured so that we can feed as much functionality as possible directly into the software which runs the Wikimedia projects, MediaWiki. The namespace manager, which is going to be part of MediaWiki 1.6 if I can help it, was the first example of this strategy. Namespaces are essential to structuring data -- they allow us to distinguish different types of content in a single wiki installation.

In WiktionaryZ, for example, we have expressions (which are connected to meanings, synonyms, translations, and so on), but we also have collections (allowing us to identify sets of concepts as being part of a single "body of work", such as the GEMET thesaurus), relation types, attributes, and quite importantly, regular wiki pages: there will be portals, policy and help pages in WiktionaryZ, as well as discussion pages. Namespaces allow us to keep it all neat and separate. These namespaces used to be hardcoded, hackish and unflexible. This has now changed.

The namespace manager is, however, not only beneficial to WiktionaryZ. It is also beneficial to other Wikimedia projects, such as Wikibooks, which can use them to structure large, related sets of pages. It is beneficial, in fact, to any MediaWiki user. And as developers of WiktionaryZ, we have the benefit that our architecture is sound, our work is visible, it is integrated and maintained as part of the official MediaWiki codebase, and that it is tested and reviewed by many users.

I have now specified a new component milestone that will hopefully be similarly beneficial: Multilingual MediaWiki. Currently, MediaWiki has no awareness of languages beyond the user interface level. We have multilingual wikis, such as Wikimedia Commons and Meta-Wiki. However, in these wikis, all languages share a single namespace; if there are title conflicts across languages, they can only be resolved manually. There is no support for linking languages together or creating translations easily.

You may say: "But I thought Wikipedia exists in over 100 languages?" And you're right - it does. But from a technical standpoint, each of these languages is a separate database. Pages in different languages are linked together using something called "Interlanguage Links", a very ugly hack that requires us to have a list of links to the page in all other languages at the bottom of every single language version. In effect, this means that if you have 10 versions of a page, you need to maintain 10*9 interlanguage links (each of the page linking to its 9 corresponding pages in other languages). There is no way to get a list of recent changes from a set of languages either-- MediaWiki knows only about one content language, the one it's set up with.

Still, things are working reasonably well for Wikimedia, and the database split has its advantages, too. But for truly multilingual projects like Meta, Commons and, incidentally, WiktionaryZ, which rely on the presence of a single database, the status quo is badly broken. This is what these specifications seek to address. It is absolutely essential to solve this problem for WiktionaryZ. You can see why by browsing the page index of the GEMET read-only milestone of WiktionaryZ. Here you will notice that pages like "AIDS" have multiple records in the page title index, because they exist in multiple languages. But MediaWiki knows nothing about that -- complicating the situation especially where you have an expression that has different meanings in different languages.

We could be hackish and try to support multiple languages only within WiktionaryZ. But this would bite us in the long run. It makes much more sense to make MediaWiki a truly multilingual application -- and this will benefit its hundreds of users world-wide. Imagine that, thanks to these changes, every single installation in the world will only have to upgrade MediaWiki, and be able to start accepting content in multiple languages if they so desire. Browse the list of sites using MediaWiki. How many of them are multilingual today? How many will be a year from now?

Multilingual MediaWiki also offers some "blue sky" potential. One of those blue sky ideas is the management of translations. When you need a translation of a document quickly, you want to notify all people who are able to provide it to you, and manage the assignment. It would not be too hard to add this functionality: Allow users to specify which languages they are willing and able to translate from, and notify them when a document to be translated appears in one of these languages.

This example shows how important it is to come up with a sound, scalable architecture for Wikidata and WiktionaryZ: it creates synergies. This is what we're trying to do, and this is why things aren't always moving at a rapid pace. Watch this space: One of the next major documents I will publish is a fairly complete set of specifications for Wikidata's versioning engine, multi-language management and schema management.

2006-01-16T12:51:00.000+01:00

Van Dale is the publisher of choice for the Dutch language. They contacted me last year through Kennisnet. Van Dale is interested in exploring what WiktionaryZ will mean to them. They are interested in collaboration and they are interested in business models of commercial organizations that collaborate with the Wikimedia foundation.

One of the great examples of such collaboration manifests itself in the books that are published in Germany by Zenadot Verlagsgesellschaft in Berlin. Together with the German chapter they produce books under the label WIKI press. If you read German I do recommend these books.

Thanks,
GerardM

2006-01-15T21:01:00.000+01:00

Who is going to write the first blog .... that was the question.

Well ... I just thought I could write some notes on WiktionaryZ here.

Almost one and half a year ago there was that Dutch Wiktionarian, GerardM, who posted loads of, at that time, not understandable templates on the Italian wiktionary. Hmmm ... and what does a good wiktionarian do? He/she asks for clarification. Well, so I learnt that those templates where there just to reduce "workload" and that was something I was thinking about myself as well (not knowing templates my thoughts went completely different ways). And that day - it was the 30th of August 2004 was the starting point for this very particular projekt, known with its work name as Ultimate Wiktionary.

WiktionaryZ will provide a unique place for lexicological data, but not only. It is impossible to describe it in just a few words. Imagine all you can imagine in an online dictionary and you will only think about a small percentage of it.

Gerard and me dreamt that dream for some time - started to work on simplifying things ... then definitely Ultimate Wiktionary became a project and Gerard had a meeting with Kennisnet who were ready to pay 5.000 EUR for the programming of UW ... well Erik Moeller accepted :-) from that moment on we were in three.

Now you will ask yourself where we are today ... well, we have a nice name, WiktionaryZ, we have a subset of the software with the GEMET-data online (http://epov.org/wd-gemet/index.php/Main_Page) - it is a read-only version for now, but it is the first "visible" result of Erik's hard work. The database design ... well I let this to Gerard ... it is somewhat toooooooo huge to explain :-)

Being only in three was not enough to assure what needs to be assured (I like this sentence :-) and so there is that commission including some more people who hopefully will introduce themselves here and will write about their view.

Well ... now it is up to you, dear reader, to ask questions ... and it is up to my colleagues to write ... that's why writing the first entry of a blog sometimes is quite a favourable thingie ... you do not need to tell everything since you "must" leave something to others as well.

Thanks for taking the time!

Sabine