We just released WPML 4.3, which includes a completely new way of translating strings. The String Translation admin looks the same but works differently. Instead of reading translations from the database, it generates and loads .mo files. This results in much lighter database usage for string translation.
Why Did We Rewrite String Translation?
The previous version of string translation loaded the strings from the database and attempted to avoid loading .mo files. Some of the large WordPress plugins come with huge .mo files, which translate each and every text in the plugin. For example, WooCommerce .mo files have a bit over 8,000 strings. Other large plugins have similar string counts.
When you run a site in English, WordPress doesn’t load these .mo files. However, when you run a site in any other language (just one language, not multilingual sites), WordPress will load all these .mo files. Loading these files takes anywhere from 100ms to over a second, depending on the server you’re using. These .mo files will load on any page on which the plugin runs (typically, on all the site’s pages).
Until WPML 4.3, we attempted to optimize this mechanism. WPML had an option, which was enabled by default, to prevent the theme and plugins from loading .mo files. Instead, we checked which strings are needed for each page and we loaded these specific strings from the database.
The mechanism worked well in general, but sometimes it backfired. When a plugin hits many strings on every page load, WPML would load a huge list of strings from the database instead of letting the plugin load its own .mo file.
As some plugins grew over time, our optimization mechanism became less and less efficient.
In WPML 4.3 we decided that it’s time to change the approach. We’ve stopped blocking .mo files and we’ve stopped loading strings from the database. Now WPML doesn’t attempt to optimize the translation times of other plugins or themes, and only optimizes its own strings.
How Will This Affect My Site’s Load Time?
This depends. If your site was fast until now (which the majority of the sites using WPML are), you’re not going to feel a significant change in speed. For example, on wpml.org, most pages are a little faster and a few pages take a little more time to load.
The reason why some pages take a little longer to load now is because there are other plugins loading huge .mo files. The speed increase from optimizing string translations is canceled out by the .mo file loading times of other plugins.
However, these fluctuations are very small and we can only measure them with advanced profiling tools. A website that ran fast until now will generally continue running fast with WPML 4.3.
Some websites that suffered from slow load times will now load much faster. Prior to WPML 4.3, some websites had tens of thousands of strings to load from the database. This caused a significant load on the database, consumed a lot of RAM and caused long page load times. These sites will see a very significant improvement in performance.
Any More Performance Optimizations Coming?
Yes. Now that we’ve removed the major performance issues coming from string translation, we’ll have more opportunities for optimization.
We’re expecting to find issues that will require work with the authors of other plugins and themes. Performance improves the most when plugins and themes load fewer strings. Most other plugin and theme authors are happy to work with us and we’re confident that we can resolve almost every performance issue that comes next. The optimization process is almost always the same:
- Split huge .mo files into several smaller files
- Load the right files on the right pages
The trivial segmentation is between strings that are used on the admin and strings that display on the front-end. Deeper optimizations further split front-end strings according to the type of page. For example, on e-commerce sites, there’s no need to load the strings used in the checkout process on every page.
Of course, when there’s an opportunity for optimization within WPML itself, we take care of it immediately.
Update, Measure and Report
WPML 4.3 is being deployed right now. We’re releasing it gradually over one or two weeks. If your sites haven’t received this update yet, give it some time and you’ll see the update in a few days. You can skip the queue and get this update immediately by going to Plugins –> Add New page, clicking the Commercial tab and then using the Check for updates button.
If you feel that any page on your site takes longer to load, we suggest you install the Query Monitor plugin (follow our tutorial on how to analyze performance issues with Query Monitor). Note that in WPML 4.3 we fixed significant compatibility problems with Query Monitor, so the reports that you see are not useful for versions before WPML 4.3.
See if WPML calls are starring in the load reports. If so, please report to us in WPML technical support. Even if you see threads that appear similar, start your own. We’ll need access to your site so it’s better to work on each site separately.
Of course, we’ll be very glad to hear your general updates about performance (it’s equally helpful to report improvements 🙂). Leave your comments here and we’ll get back to you.
Awesome. Two questions:
1) Does the update automatically remove strings from the database that are being read from the .mo file from now on?
2) Does this update also fix the issue that causes the constantly growing icl_string_pages database table?
Thanks
JP
Sorry for the slow reply. I overlooked your comment.
1. No – this release doesn’t automatically clear strings from the database. WPML still needs some of these strings to produce the .mo files. We don’t call the DB on page loads, but we need the DB values to produce our .mo files.
2. Yes. This table is now removed from WPML, so it’s also not going to grow.
Hi,
It is safety remove this table if WPML don’t need this table any more? icl_string_pages
Or WPML do that with next update?
Yes, you can remove that table. As a precaution, we keep tables that we don’t use for one release. WPML will remove that table automatically in the next release, but you can safely remove it already.
Hello Amir
With respect to the second topic, I can still see this table in the database. I’m running WPML 4.3.1.
Is this normal?
Patrick
Hi Patrick,
Indeed, the table is still there.
We didn’t remove it yet, as we plan to do it on the following release.
WPML won’t use the table to store or read data, but there are still some parts in our codebase which may still attempt to access some of the deprecated tables during the upgrade process.
Meanwhile, if you want, you can
TRUNCATE
(do not useDROP
yet) this table as well as theicl_strings_urls
.Unfortunately, the generation of the .mo files apparently requires the WP REST API, unnecessarily opening attack vectors for simple multi-language (corporate) websites that had blocked the REST API because it simply wasn’t needed, as all interaction was already made via the admin web interface.
I’ve been busy with other non-WP-related stuff, and so didn’t have the opportunity to test the beta at the time. Also, I understand and support the desire to remove DB calls; however, I really don’t get why you would need to open a security hole (which is what any endpoint is, if you don’t use it) just to generate an .mo file; you already have access to the file-system, because the plugin is installed…
Cheers
The WordPress editor itself now relies on REST API, so it’s not something that we opened up for ourselves. I understand that some people prefer not to use the WordPress editor and also disable the REST API, but you have to remember that tens of millions of websites use it. So it’s not something that we opened up for ourselves and it’s not something that currently has exploits.
Just because the REST API exists, doesn’t mean you have to use it. In fact, you didn’t need to use it (and I sincerely doubt that using the REST API in this case made your life any easier, but the WPML devs would know that better than me, I guess), because you already have access to the file-system. The REST API was designed for AJAX-oriented scenarios, such as front-end frameworks that needed to interact with the back-end (to fetch HTML data, for example, in single-page websites), not for this kind of scenario.
Decisions such as these should be made according to what is best for your customers, not to serve as showcases for the mentally-handicapped decisions taken lately by the WordPress core devs.
Frankly, I consider this as the WPML team dropping the ball monumentally.
For my part, you are now forcing me to adjust my WPML-supported websites (which were hardened with htaccess rules, which happened to also disable the REST API) to get WPML to work, simply because the team opted to use it because it was shiny.
Most likely, I’ll just downgrade WPML on my test website to a more acceptable version, because you didn’t even remember to include an option to permanently dismiss the dialog box that tells the admin to generate .mo files (a process which doesn’t even work, btw: the REST API is not invoked — checked via Chrome’s dev tools –, and closing the dialog raises an “icl_vars not defined” error. Trying to scan .mo files immediately raises a “g is not a function” javascript error. Who tested this?!?!).
Had I known this upfront a few years ago, I would not have bought a WPML license, and instead gone for something else. Maybe I really will start looking for something else: when I bought a WPML lifetime license, the plugin was great at its job — easily translating WP websites –, but now it’s failing completely in that purpose.
Best regards
Meanwhile, I was able to get the thing to work, by commenting a line that enabled ICL_DONT_LOAD_LANGUAGES_JS.
This kind of thing is a show-stopper; least you could do would be to mention it in the change log, instead of only in a forum which not everyone follows…
https://wpml.org/errata/using-the-constant-icl_dont_load_languages_js-can-lead-to-issues-in-wpml-4-3-with-initial-scan-of-mo-files/
Same thing for me ICL_DONT_LOAD_LANGUAGES_JS was set to true, EVERY click in the backend triggerd a modal popup about mo files. In addition to that the menu editor is unusable and menus are a mess. That broke the website.
I already had tons of problems with WPML updates over the past seven years and am really tired of that. I guess I will use Polylang in the future as it’s a lot less complex and much easier to use.
Update: The broken menu was not WPML’s fault. Instead some ancient code in the theme was messing around with the menu editor.
Thanks for the heads up!
Thanks for the heads up! Happy it’s working well now.
And what happens when you simply remove that constant, which you’ve set manually?
So, we’re now coming to a full circle after 10 years?! Beautiful 🙂
To be honest I think it could be an idea to split these gigantic
.mo
-files into frontend/backend strings, although this would of course mean that plugin/theme developers need to implement that or get the tools to implement that in an easy way.Regarding the REST API issue that João Saraiva brings up above, I would like to check whether for this to function the REST API needs to be open for non-authenticated users?
Actually, WPML does divide the strings into separate .mo files. This depends on how the strings are organized in the themes and plugins. Many plugins set different textdomains for front-end and backend strings. WPML will generate separate .mo files for them. I don’t think that WPML should attempt to further optimize beyond this because we’ll always generate problems with false identification. Then, our attempt to save resources will end up in missing translations.
I’ll check with our developers if we generate the .mo files in all visits or only for logged-in users. I suspect that it’s for all visits, but I’m not sure.
> I suspect that it’s for all visits, but I’m not sure.
But why is that? Aren’t
.mo
files only generated once?And even if it would now be the case that they are only generated when a visitor hits a page, then how hard can it be to crawl the frontend of the site from the backend (therewith done by an authenticated user), generated the
.mo
files and be done with it?Having the REST API wide open for unauthenticated users is a step backwards.
And what will happen on sites where the REST API is blocked for unauthenticated users and they upgrade to WPML 4.3? Generation of
.mo
files will not work, will there be a fallback or are the problems going to be more severe?Every time you update the translation of strings in the backend, WPML needs to update the .mo file for that textdomain that you edited. WPML will generate the .mo file once after every edit (of course, not on every page load).
Like I wrote, we’ll look at the suggestions here and see what we can optimize related to access and privileges.
Great to learn that things are not set in stone!
With latest Divi builder. It asks every time to run so string translation works
Great, but what about a ‘rich text editor’ option to translate strings with html tags?
This is a ‘must’ in a Premium plugin.
Walter.
Yes, this is coming very soon. In the next release, we’re going to have strings translated via WPML’s Advanced Translation Editor. This is a very capable editor which includes full support for fancy HTML markup.
Since the last update, our site is unable to translate into French, Arabic and Portuguese anymore. URLs are not found giving error 404 page not found
From your description, it sounds like a known issue related to Yoast and WPML update, if you are using Yoast please downgrade to the former release. We have already a fix for this issue and we are planning to releases it ASAP
Do you have details regarding the Yoast + WPML issue? I’d like to read more about it since we’re planning to update the plugins, especially WPML to 4.3.2 when that’s released in the next few days.
The detailed information is in the known issue page here – https://wpml.org/errata/yoast-seo-multilingual-causes-404-on-secondary-languages/. The actual issue was caused by a change in both Yoast and Yoast Multilingual – we needed an unfiltered home URL and it was cached, using get_option(‘home’) instead of home_url() and removing the filter works fine, We have a fix and we are testing it. I hope testing will be done and we can have the release today.
After the update to 4.3 we had an annoying popup in the backend. It was not possible to create mo files. With every call in the backend the annoying annoying Poppup. No matter what attitude you chose! We have deactivated the string translation!
Hi, generally speaking, this could happen in one of those cases:
1) Wrong permissions in the folders, in wp-config or in .htaccess — this is actually a WordPress requirement and if this is what you have please follow this document – https://wordpress.org/support/article/changing-file-permissions/
2) define(‘ICL_DONT_LOAD_LANGUAGES_JS’,true) set in wp-config or functions.php — this is a WPML glitch that is being fixed now and will be released with WPML 4.3.2, the workaround is described here – https://wpml.org/errata/using-the-constant-icl_dont_load_languages_js-can-lead-to-issues-in-wpml-4-3-with-initial-scan-of-mo-files/
3) security plugins like wp-security or others that prevent plugins to write to the directory, in that case please disable the plugin that is blocking the writing permissions, let WPML creates the MO files and reactivate the plugin – as stated in point number 1 that permission is a WordPress requirement and you’ll have to repeat that on each update
If what you experience there is different from those cases please open a new report and we’ll debug and see what’s going on.
Are there any plans to implement configurable option to use Memcached (or Redis) as strings storage for better performance?
Hello everyone!
I’m Andrea, lead developer for WPML.
First of all, we have just released WPML 4.3.2 which fixes the
ICL_DONT_LOAD_LANGUAGES_JS
constant preventing the scanning of .mo files and showing errors.This version also fixes other issues and streamlines and improves the performances of jobs handled with the Advanced Translation Editor.
I want to address some of the topics you have risen in your comments.
:: String in tables vs strings in MO Files
John-Pierre
String Translation won’t access the database when trying to fetch the translation: it will only get them from the MO files.
However, the database is still required by String Translation so it can use it to update these MO files when a translation of a string is added or changed.
:: Unused tables
John-Pierre, Yumit, Patrick
With the new version of String Translation, two tables became redundant:
icl_string_pages
andicl_string_urls
.We have decided to not remove them yet as a safety measure.
The next versions of String Translation will take care of removing these tables.
For now, if you don’t want these tables to take space, and you have the required knowledge of MySQL, you can empty them with a
TRUNCATE TABLE
statement (I do not recommend usingDROP TABLE
yet).:: The generation of the MO files happens on every page request
Pieter
No, the generation of these files only happens in back-end pages and only for the time needed to generate these files.
:: REST API can be called by non-authenticated users
João
This statement is not entirely correct.
Our endpoints require capabilities.
And you can only have capabilities if you are an authenticated user.
Moreover, we require specific capabilities: not all authenticated users can consume our endpoints.
:: With disabled REST API (or the enabled “ICL_DONT_LOAD_LANGUAGES_JS” constant), the site doesn’t work
Matthias, Pieter
That’s an issue we have indeed missed.
With disabled REST API, it is not possible to use some functionalities in WPML.
The same happens when the
ICL_DONT_LOAD_LANGUAGES_JS
constant is set to “true”.Moreover, the notice to generate the MO becomes non-dismissible.
The most critical issue is that a Javascript error may affect the functionality of other plugins.
Without it, you won’t be able to use MO files generation, Translation Job, and Advanced Translation Editor, to name a few.
However, WPML shouldn’t even block your site.
We are improving our plugins to block these features more gracefully and, when this happens, WPML will let you know so that you can take the relevant actions.
To fix the more urgent issues, we didn’t include this in WPML 4.3.2, but we are already working on it.
I hope this clarifies most if not all the questions.
For each topic, I’ve mentioned the relevant commenters: forgive me if I missed some of you 🙂
If you have more questions or you need further clarifications, please feel free to ask.
Thanks,
Andrea
Hi! I’m managing a few websites with WPML. Some are quite simple wordpress templates but others are quite more hand made. The basic templates work properly, but the others I see the popup error saying that the .mo files can’t be created… it’s quite annoying, since I see it all the time I change a page. Maybe the designed websites have some issue that prevent the .mo to be created? they all are in the same server, so i don’t think it’s a permission issue.
Anyway, the websites still don’t ask me to update the version, it says I have the 4.3.1 and that it’s updated, will it take some time so I can see the 4.3.2?
thanks!
Hi, not all of our clients got the WPML 4.3.2 update yet if this is your case please go to Plugins -> Add new -> Commercial -> and click on “check for updates”. New versions should be available now in this section, please make sure to update WPML and all of the addons (some of the issues are related to addons)
Hi, great, I could updateto 4.3.2 and now all websites are working properly, and the .mo file is correctly generated. Thanks a lot!