Testing KDE 4.3 translation with Lokalize

I created Lokalize script to automate UI translation testing: it compiles currently opened PO file and puts it under ~/.kde where KDE can see it, so that when you start your application next time, it will use your translation.

To use it for your language (you'll need Lokalize coming with KDE 4.3):

cd trunk/l10n-kde4
svn up scripts
dolphin scripts/lokalize

then open Lokalize with project for your language, go to Project -> Configure project -> Scripts and drag-n-drop msgfmt.rc from dolphin to the settings area in Lokalize. Restart Lokalize. You should see a new item under Tools menu.

To make this script available automatically for all your team members copy this msgfmt.rc to l10n-kde4/LANG/lokalize-scripts (they will also need to svn up trunk/l10n-kde4/scripts).


Now a few words on packages.
I have been given write access to Debian KDE packaging repository (thanks to Sune Vuorela), where I was able to specify right dependencies and recommended packages for Lokalize. If any of other distro packagers are reading this, please sip on the spec.

P.S. I installed kde 4.2.96 packages from debian experimental (it conflicts with kdevelop package, so I reinstalled kdevelop manually with dpkg -i kdevelop_3.9.94-1_i386.deb afterwards).

Qt's DOM whitespace specifics

So I've just finished writing xliffmerge scipt that is direct analog of msgmerge@gettext and pomerge@translate-toolkit, but specially for XLIFF, i.e. it leverages all the advantages XLIFF brings, like saving information about each template update, specifying tha phase in which each unit was modified last time, and so on.

I was surprised to find out that translate-toolkit parses XML files into its own internal representation, which means they loose everything they don't support.

So to do it in a nice way, I chose to work directly with DOM representation of XLIFF file (using QDomDocument and friends). In Lokalize XliffStorage class is just a wrapper around QDomDocument.

I enabled preserving the whitespace, but in some cases it added additional whitespace, which means it modified user-editable text. The cases were clear: when no character data is between tags, it 'formats' them: inserts newline character + indent spaces. To override this behaviour I just added insertion of empty text nodes between tags. You can see the code in the end of xliffmerge.py (fixWhiteSpace*())

Scripting Lokalize

I started working on scripting support in Lokalize several months ago, but never really announced this because I wanted to write my own scripts and refine/extend API accordingly along the way.

KDE 4.3 will bring us 'officially' scriptable Lokalize. Among recent additions are new project wizard with anonsvn checkout support (screencast; it also checks out templates/ and scripts/ folders), actions to update .po from template and to compile .po into .mo and place it in local .kde folder so it could be picked up automatically by applications on the next start (see here and here for python sources).

Recently I also extended pology with ability to communicate to Lokalize, so now you can run xml checks against your translations and get erroneous files opened in Lokalize and positioned on appropriate entry.

So how it works: on each project open Lokalize scans PROJECTDIR/lokalize-scripts folder for .rc files and adds them to a 'cache' file called PROJECTDIR/lokalize-scripts/scripts.rc (so you shouldn't generally want to add it to version control system). RC files contain descriptions of scripts, particularly their paths. The path is relative to .rc file folder (or to a system scripts folder - they are tried both). So you for example can specify "../../scripts/lokalize/opensrc.py" to load script from global kde-l10n4 scripts folder (i.e. not specific to your language). Each script is represented with action in application menu (and you may assign a shortcut to it).

I wrote a script (in qtscript/javascript) to do a check specific to Russian language: %1 should be mentioned in _all_ plural forms, because, in Russian, the first form is also used for 21, 31 and so on. You can find it here. The script contains no immediate code, but only a function. The name of the function (fileSaved) is essential, because it matches the signal emitted by Editor object, which allows us to communicate with current editor tab in Lokalize. The function is automatically called on each files save, and if file contains errors, a warning is issued and wrong entries are shown. Scripts are actually don't get loaded until user explicitly triggers their action in the menu, so to override this behaviour RC file specifies <property name="autorun" >true</property>.

Below are links to API references. Everything marked as Q_SCRIPTABLE may be used from scripts
Editor object API reference
Lokalize object API reference
Project object API reference

You are welcome to write your own scripts and polish existing ones.

Lokalize screencast time

I did quite a few things during the last week.

In particular I finished implementation of storing XLIFF inline markup in translation memory and made autosubstitution mechanism more robust. Also I added support for binary units and displaying alternate translations (stored in xliff or from external file -- esp. useful for gettext po files which don't support storing alternate translations in the same file).

A screencast

I'm back

I'm back to Lokalize development with addition of support for storing XLIFF inline markup in translation memory (including mechanism for matching inline tag id's). Gotta make the sprint until KDE 4.3 hard feature freeze (means I'm staying home for May holidays).

Lokalize also got the first real contributor (I only was getting small patches before) - Viesturs Zarins. He rewrote projectmodel in a nicer way because it became broken in KDE 4.2. Now we have statistics calculated automatically for all project dirs in background thread.


I'm packing my beg now for the two-day trip to Mykolayiv (~500 km away from Kiev, 14 hours on the train - I think there's almost no chance that the situation will be improved by the time for UEFA Euro 2012) to attend KDevelop Developer Meeting in hope to take part in bug hunting (gotta few on my list) and overall preparation of KDevelop for 4.0 release. This is going to be my first real world hacking session, so I'm a little nervous ;))

Lokalize enforces workflow

Last week I added workflow phases support to Lokalize (XLIFF only). A person defines his/her role in the project [settings], and then corresponding phase is created automatically when file first changes (an old phase opened by same person is reused if found).

You may list phases, create them manually as well as add notes to them.


Also I added xliff extended states support. I did this in a smart way - there is still two-state button, which selects extended state depending on kind of the current phase (needs-translation/translated for translation phase, needs-translation-review/signed-off for review phase),
with drop-down menu to select specific state manually.

So what for translator is ok (translated), for reviewer is not (i.e. rendered in italics), and needs to be signed-off.



This allows us to achieve something interesting: we now able to see in which phase (and hence by whom) the translation unit was modified last time. This info is shown in Metadata toolview (previously known as 'Message Context').

Fine-grained applications menu

Some time ago I wrote a proposal for reorganizing app menu.

I started implementing it a month ago by writing an alternative applications.menu. As it relies on freedesktop.org spec categories specified in .desktop files, and not all applications are good citizens in that regard, my first steps were to bug application developers to add subcategories to their apps' .desktop files. I also walked through KDE apps and updated their desktop files. All the bugs I reported were fixed, except for the OpenOffice.org one.

Of course, I made a screencast of the results.

I prepared a simple script that you can run [as root] to get same results. The package also contains updated .desktop files (it installs desktop files only for apps present in your system), and it backs up the old kde4-applications.menu (yes, in my debian system it has kde4- prefix).

If anyone likes the alternative layout please let me know.

First results implementing OpenDocument translation workflow

This is how OpenDocument translation workflow will look like in KDE 4.3: Screencast

Don't try to do it yourself right now ;) This requires the latest (unreleased) versions of translate-toolkit trunk and kdelibs trunk.

Implications of achieving this
I plugged kross nicely into Lokalize: I introduced 'project kinds' -- each has it's own set of scripts, so that you have 'Merge into ODF' for OpenDocument translation, and 'Widget Text Capture' for GUI/KDE projects.

These actions, as well as few others (see kde l10n svn and kde ru l10n svn for their plug example) are written in python. I plan to create a separate scripting API docs for Lokalize, but for now you can just rgrep Q_SCRIPTABLE KDE/kdesdk/lokalize to get list of available methods.

This mechanism will make implementing business-logic easy. Examples are action to send your translation to project's editor (translation team leader) and action to commit your work to [svn] repository if you have write access to it.

Thanks
I want to say thanks to Sebastian Sauer for giving me the green light for kross-extending commits, and Wynand Winterbach for commiting the patch to translate-toolkit.
Once I've got the thanks section, I'd also like to mention NLNet Foundation which funds my current efforts and KDevelop team, which makes those efforts efficient (today I opened whole kdelibs in KDevelop, and it took 15 minutes and 0 crashes to scan it completely ;) )

Comment your translations

It it sometimes useful to leave a note for an individual translation (for example when you're unsure about some term). Since some time ago scripty (a daemon which updates KDE translation files to reflect the current state) started to preserve comments in GETTEXT PO files. XLIFF standard supports notes too. So this week I was addding notes support to Lokalize.

Notes mechanism may also be used for bookmarking, as Lokalize now allows quick searching including notes (I added 'options' button to set additional filtering critera). Check this out on the video.


I wrote a simple python-based action to merge translated XLIFF file back to .odt document. Next week I'll be writing patches for traslate-toolkit to support on-the-fly conversion and filling <file origin""> attribute, and now I'm preparing a patch for Kross to help me make organizing project-specific scripts easy.

I also added UI reflection of 'recovered from autosave' state. You'll need kdelibs>=4.2.1 for autosave to work properly though.

Compiling Lokalize from trunk under KDE 4.2

XLIFF markup editing in Lokalize

Please correct me if I'm wrong, but today Lokalize became the first open source tool to support XLIFF inline markup editing.

I tried the latest svn trunk revision of virtaal, but it didn't display markup and mangled my test XLIFF file. AFAIK inline markup editing in proprietary world is only supported by Java Heartsome l10n Suite.

But this of course would mean nothing w/o Translate-Toolkit's (same laboratory as Virtaal's) odf2xliff converter, which will be integrated into Localize thanks to KDE's kross technology ;)

Obligatory screencast.

My plans also include support for inserting markuped text from other entries and translation memory (Lokalize will have to match tag id's for that).

Lokalize screencast

Hi. I'm author of Lokalize computer-aided translation system.

I've played a bit with Wink screencasting/tutorial program, and created an intro tutorial, which show how to open projects, fill translation memory and enable widget text capture feature.

Lokalize intro tutorial for KDE translators