angle-uparrow-clockwisearrow-counterclockwisearrow-down-uparrow-leftatcalendarcard-listchatcheckenvelopefolderhouseinfo-circlepencilpeoplepersonperson-fillperson-plusphoneplusquestion-circlesearchtagtrashx

Flask, Babel and Javascript language files

This post describes a method to generate Javascript language files de.js, en.js, etc. and how to add them to your multilanguage Flask app.

6 January 2020 Updated 9 January 2020
post main image
https://unsplash.com/@goran_ivos

This Flask website is multilanguage. The implementation is described in previous posts. So far all my translations were in the Python code and the HTML templates. On a few places I needed some translations in Javascript and did this by pulling this Javascript code inline in the HTML template. For example, for forms I needed:

	e.target.setCustomValidity('Please fill out this field.');

I pulled this Javascript into the HTML template and changed it to:

	e.target.setCustomValidity("{{ _('Please fill out this field.') }}");

That was easy and works fine.

Language in Javascript files

I knew that one day I had to change this and implement multilanguage for Javascript files. This day came sooner than expected because I wanted to implement the Content Security Policy header. The minimum we should do is remove inline scripts and the eval() function:

Content-Security-Policy: script-src 'self'
This means I no longer can have inline Javascript in the templates, all Javascript code must be moved to files. The solution looks obvious. Generate a language file with translations for every language and include the proper file. The language files are:
de.js
en.js
es.js
fr.js
nl.js
ru.js
Then at the end of the base template we do something like:
<script src="{{ url_for('static', filename='js/mlmanager.js') }}?v={{ et }}"></script>
<script src="{{ url_for('static', filename='js/locales/' + lang_code + '.js') }}?v={{ et }}"></script>
<script src="{{ url_for('static', filename='js/base.js') }}?v={{ et }}"></script>
Note that I add a timestamp to avoid browser caching. Here mlmanager.js holds the object that is used to load and get the languages. The file locales/<language>.js is the file with the translations and base.js is the file with all the code. A first version looks like this:
// mlmanager.js

var ML = function(params){

	this.key2translations = {};
	this.keys = []
	if(params.hasOwnProperty('key2translations')){
		this.key2translations = params.key2translations;
		this.keys = Object.keys(this.key2translations);
	}

	this.t = function(k){
		if(this.keys.indexOf(k) === -1){
			alert('key = ' + k + ' not found');
			return;
		}
		s = this.key2translations[k];
		return s.replace(/"/g,'\"');
	};
};

When creating a new ml object we pass also the translations. Method t is use to get a translation. A translated language file, e.g. de.js, looks like:

// de.js

var ml = new ML({
	'key2translations': {
        'Content item': "Inhaltselement",
        'Please fill out this field.': "Bitte füllen Sie dieses Feld aus.",
	}, 
});

Finally, in the file with the actual Javascript code, base.js, we change the text that must be translated from:

	e.target.setCustomValidity('Please fill out this field.');

to:

	e.target.setCustomValidity( ml.t('Please fill out this field.') );

Problem: how do we generate the Javascript language files de.js, en.js, etc.

The standard Babel documentation only mentions commands like init, extract, update, compile. What we need is a way to:

  • extract the texts to be translated from the javascript files
  • automatically generate the language files de.js, en.js, etc.

Extract the texts to be translated from the Javascript files

I decided not to scan the Javascript files but instead create a new HTML (template) file, jsbase.html, holding all the texts for the Javascript files, example:

var ml = new ML({
	'key2translations': {
		...
        'Content item': "{{ _('Content item') }}",
        'Please fill out this field.': "{{ _('Please fill out this field.') }}",
		...
	}, 
});

We put this file in the templates directory so it will be scanned by Babel when we issue the standard translation commands:

pybabel extract -F babel.cfg -k _l -o messages.pot .

pybabel update -i messages.pot -d app/translations

# do yourself: translate all texts in the po files either manual or automated

pybabel compile -d app/translations

Now we have the translated texts for the Javascript files somewhere in the messages.po files. You can check this e.g. by dumping a messages.po file:

from babel.messages.pofile import read_po
import os

def show_catalog(lc):

    lc_po_file = os.path.join('app_frontend', 'translations', lc, 'LC_MESSAGES', 'messages.po')

    # catalog = read_po(open(lc_po_file, mode='r', encoding='utf-8'))
    # without encoding parameter works if the default encoding of the platform is utf-8
    catalog = read_po(open(lc_po_file, 'r'))
    for message in catalog:
        print('message.id = {}, message.string = {}'.format(message.id, message.string))

show_catalog('de_DE')

This prints a list of message ids and string:

...
message.id = Sub image, message.string = Unterbild
message.id = Sub image text, message.string = Unterbildtext
message.id = Select image, message.string = Bild auswählen
...

Automatically generate the Javascript language files de.js, en.js, etc.

What we need is a way to translate this jsbase.html outside of Flask to our languages and generate the files de.js, en.js, etc. We could use the above code to get the texts from Javascript files and generate the language files de.js, en.js, etc. But this is cumbersome and prone to error.

Then I bounced into a way to render a template outside of Flask, see links below. The idea is to render the jsbase.html template letting Babel put the proper translations into it. Then all we need to do is write the rendered result to language files de.js, en.js, etc. Is it really that easy? Here is the code that does this:

from jinja2 import Environment, FileSystemLoader, select_autoescape
from babel.support import Translations

import os
import sys

def generate_translated_js_file(
        app_translations_dir, 
        language_region_code, 
        app_templates_dir, 
        template_file, 
        js_translation_file):

    template_loader = FileSystemLoader(app_templates_dir)

    # setup environment
    env = Environment(
        loader=template_loader,
        extensions=['jinja2.ext.i18n', 'jinja2.ext.autoescape'],
        autoescape=select_autoescape(['html', 'xml'])
    )

    translations = Translations.load(app_translations_dir, language_region_code)
    env.install_gettext_translations(translations)

    template = env.get_template(template_file)
    rendered_template = template.render()

    with open(js_translation_file, 'w') as f:
         f.write(rendered_template)

This function loads the selected language, uses render() to translate and writes the result as de.js, en.js, etc. Note that I use multiple apps in my setup, app_frontend, app_admin, using DispatcherMiddleware. To generate all Javascript language files for all apps and languages I call the above in another function:

def generate_translated_js_files():

    # app translations directory has subdirectories de_DE, en_US, es_ES, ...
    # lang_code is language code used in the Flask app
    language_region_code2lang_codes = {
        'de_DE': 'de',
        'en_US': 'en',
        'es_ES': 'es',
        'fr_FR': 'fr',
        'nl_NL': 'nl',
        'ru_RU': 'ru',
    }

    template_file = 'jsbase.html'

    for app_name in ['app_frontend', 'app_admin']:

        # app/translations 
        app_translations_dir = os.path.join(app_name, 'translations')

        # app/templates
        app_templates_dir = os.path.join(app_name, 'templates')

        for language_region_code, lang_code in language_region_code2lang_codes.items():

            if not os.path.isdir( os.path.join(app_translations_dir, language_region_code)):
                print('error: not a directory = {}'.format( os.path.isdir( os.path.join(app_translations_dir, language_region_code) )))
                sys.exit()

            # shared/static/js/locales is the directory where we write de.js, en.js, etc.
            js_translation_file = os.path.join('shared', 'static', 'js', 'locales', lang_code + '.js')

            # translate
            generate_translated_js_file(
                    app_translations_dir, 
                    language_region_code, 
                    app_templates_dir, 
                    template_file, 
                    js_translation_file)

# do it
generate_translated_js_files()

Note that this a bit double at the moment because frontend and admin share the same static directory.

Problems

Of course there are problems. When the Javascript code was in the HTML template I added Jinja code:

{% if ... %} 
	...
{% else %} 
	...
{% endif %} 

to use a certain part of the Javascript code. We cannot do this anymore ... :-(. To be more specific, in my case Javascript is calling another page with a url that can exist or not and the url also depends on the language. For example, the link in the Javascript inline code of the HTML template looks like this:

	{% if 'Privacy policy' in app_template_slug %}
	moreLink: '{{ url_for('pages.page_view', slug=app_template_slug['Privacy policy']['slug']) }}',
	{% else %}
	moreLink: '',
	{% endif %}

What I did is a total rewrite of the cookie consent. For now the HTML is no longer generated in the Javascript but in the HTML template. It still needs more work.

Summary

This is a first implementation but it works fine. The magic is using the Babel and Jinja APIs. Possible improvements:

Instead of of having strings as index for the translation:

ml.t('Please fill out this field') 

we may want to use objects:

ml.t( t.Please_fill_out_this_field )

And instead of having a translations Javascript file with Javascript objects we may want to use a JSON file holding the translations only. Anyway, next steps will be to selectively add more custom Javascript files and more translations.

Links / credits

Analyse your HTTP response headers
https://securityheaders.com

Babel documentation
https://readthedocs.org/projects/python-babel/downloads/pdf/latest/

Best practice for localization and globalization of strings and labels [closed]
https://stackoverflow.com/questions/14358817/best-practice-for-localization-and-globalization-of-strings-and-labels/14359147

Content Security Policy - An Introduction
https://scotthelme.co.uk/content-security-policy-an-introduction/

Explore All i18n Advantages of Babel for Your Python App
https://phrase.com/blog/posts/i18n-advantages-babel-python/

Give your JavaScript the ability to speak many languages
https://github.com/airbnb/polyglot.js

Leave a comment

Comment anonymously or log in to comment.

Comments

Leave a reply

Reply anonymously or log in to reply.