Home›Support›English Support›[Escalated to WPML Developers team] Word calculation with ATE's new Big Segments
[Escalated to WPML Developers team] Word calculation with ATE's new Big Segments
This is the technical support forum for WPML - the multilingual WordPress plugin.
Everyone can read, but only WPML clients can post here. WPML team is replying on the forum 6 days per week, 22 hours per day.
WordPress 6.7 has introduced a new issue that impact translations, please update WooCommerce and WPML to the latest versions before you report issues. More about this here - https://wpml.org/errata/php-error-wp-6-7-notice-function-_load_textdomain_just_in_time-was-called/
Background of the issue:
Hello, after the publication about the new ATE version with translation by paragraph, I wondered how the words/credits are being calculated when you change something in the source content. In the past when you had an ATE translation by sentence, and you change a single word in the source, it would retranslate and calculate all words of the entire sentence. So I thought that if I let the ATE translate by paragraph, then change a single word, it might calculate all words if the entire paragraph. So I did a test. Using the Divi builder, I added a text module with 5 paragraphs to this page hidden link It translated and calculated 610 credits = 305 words (in 5 paragraphs). I then changed "jaren negentig" to "negentiger jaren". It then translated and calculated all 305 words again, so all 5 paragraphs and not just the changed paragraph with 103 words or the changed sentence of 13 words. Then I tested again on this new page with different text hidden link It translated and calculated 360 credits = 180 words (in 5 paragraphs). I changed "restaurantwebsite" into "hotelwebsite". It then translated and calculated 20 credits/10 words, which is the length of the changed sentence (paragraph is 70 words).
Symptoms:
I have two different results and really wonder how it is supposed to work and calculate in case of translation by paragraph, when the source is changed.
Questions:
Is the entire paragraph being retranslated and calculated?
Are unchanged parts in the paragraph taken from the translation memory and only the changed part is retranslated and calculated?
What about my first test that recalculated all 5 paragraphs?
I HATE THIS AI SUMMARY AND DIDN'T CHECK IF IT UNDERSTOOD CORRECTLY. STOP THIS! HERE IS MY ORIGINAL QUESTION
Hello, after the publication about the new ATE version with translation by paragraph, I wondered how the words/credits are being calculated when you change something in the source content.
In the past when you had an ATE translation by sentence, and you change a single word in the source, it would retranslate and calculate all words of the entire sentence.
So I thought that if I let the ATE translate by paragraph, then change a single word, it might calculate all words if the entire paragraph. So I did a test.
Using the Divi builder, I added a text module with 5 paragraphs to this page hidden link
It translated and calculated 610 credits = 305 words (in 5 paragraphs).
I then changed "jaren negentig" to "negentiger jaren".
It then translated and calculated all 305 words again, so all 5 paragraphs and not just the changed paragraph with 103 words or the changed sentence of 13 words.
Then I tested again on this new page with different text hidden link
It translated and calculated 360 credits = 180 words (in 5 paragraphs).
I changed "restaurantwebsite" into "hotelwebsite".
It then translated and calculated 20 credits/10 words, which is the length of the changed sentence (paragraph is 70 words).
Now I have two different results and really wonder how it is supposed to work and calculate in case of translation by paragraph, when the source is changed. Is the entire paragraph being retranslated and calculated? Are unchanged parts in the paragraph taken from the translation memory and only the changed part is retranslated and calculated? And what about my first test the recalculated all 5 paragraphs?
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
Hello!
Can you do a different test and also send me the URL of the page that you tested with so i can look for the timeline of the page translations and understand what's happening.
run this test: make a change again, as you did before and then instead of translating it automatically, open it with the translation editor and see if it appears as RE-translated automatically by default. then just send me the link and the language pair of the page you wanted to translate so i can investigate what happened because what you mention seems off. it should retranslate the entire segment but the other segments should not be touched!
When I tested before, I had automatic translation enabled, and upon review I saw it translated the paragraphs.
For this new test I disabled automatic translation, and in the ATE you now see that it contains sentenced based translations. But when you look at the settings in the ATE, the setting "I prefer bigger segments" is still enabled.
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
Ok, i think the problem is the way that the content is used and probably sent to ATE now. would it be possible for you to copy the content from the text editor in Divi and send it my way? all that content from that text editor.
Even better, could you add it to a page or a post here: hidden link, i just want to see if it happens WITHOUT DIVI and with a classic editor instead (gutenberg would end up segmenting the content into paragraphs and the problem would likely not happen there).
I can already tell you that this should not really be happening, but let me wait on the devs to understand better why it's happening and how we can have it fixed.
In the meantime, if you need to work on other pages, you may want to switch to the old configuration by clicking the gear icon on the top right part of the screen, and select to "switch to the old editor".
Mmm no I think there is a difference between automatic and manual translation. That 5 paragraphs of text are added in a single Divi text module, which is similar to the classic WP editor. When I tested with the automatic translation and reviewed it, the ATE had correctly splitted the text into 5 paragraphs. Not so with the manual translation.
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
I'm afraid i got confused here. when you mean "manual translation" do you mean accessing ATE manually and translating without using any automatic translation? and by "Automatic translation" do you mean when you select the "translate automatically" opton from translation management?
Because that's why i wanted to test with WP's classic editor, since it's just a WYSIWYG editor like the "old" WordPress editor. with gutenberg i doubt it would be recreated since the paragraphs are not seen as "1 block of code" but as a separate paragraph.
Anyways. thanks for the text. i'll continue testing.
do you mean accessing ATE manually and translating without using any automatic translation?
» Going to the ATE manually, but let it translate automatically when you get there
and by "Automatic translation" do you mean when you select the "translate automatically" opton from translation management?
» yes, so automatically after a change is made. Then use the review option.
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
I ran a test on my end and i saw that only the segment with the change got retranslated and charged.
when i checked on your end, for the ate test page, i can see that only 10 words out of the 350 were actually charged which matched the amount on the actual segment that was modified.
Yes, of course the count will say 350 because that's the total count, not taking into account the translation memory but it doesn't seem like you got charged for all the entire 5 paragraphs, only for one segment of the different ones created.
At this point, can you give me the real URL where this happened to you so i can check it out and see if something different happened there?
---------------
By the way, in the sandbox initially they all appeared under 1 huuuuge segment because it didn't contain any HTML in the background, no linebreaks or anything which meant that the entire content was 1 block of text. As soon as i added a linebreak by just formatting in the "visual" editor (so pressing "enter" in the paragraphs, in this case), everything got segmented.
I already gave the real url's where this happened in my original support request. I also added a screenshot with the calculated credits and their uuid.
This huge segment must be a bug with WPML not correctly splitting the paragraphs when they are written in the classic editor. It shouldn't be needed to add extra paragraph breaks as you said in the video. At the start of the video you already see the paragraph breaks in the visual editor, and in the text editor you also see them as a white line. That is exactly how the classic editor works and has always worked.
I did some more testing while recording this video: hidden link
Test 1
Page "Paragraaf". You see I add 4 paragraphs in the classic editor, and let it automatically translate. When viewing the published source text, you see that it really contains 4 paragraphs, but when reviewing the automatic translations, it's one big block of text. However, when you edit the translation, it's all of a sudden split up into single line segments, even though the ATE is set to translated big segments.
Test 2
Page "Paragraaf2". I added a few paragraphs again, and let it automatically translated. When I now review and edit the translation, it's one big translated segment. So no paragraph based segments and no sentence based segments.
To me, this seems like a bug with the ATE and translation text that is added with the classic editor, which is also used by Divi and other page builders.
1) it does not recognize the paragraphs
2) sometimes it translates it as sentence based segments
3) sometimes it translates as a single segment (in stead of paragraphs).
This is probably also what happened when I first tested this and what I described in my original support request. The first time it translated as a whole, the second time as sentences.
So, in my opinion this is a bug that needs to be fixed in the ATE.
And then back to my original support question: how many words is the ATE supposed to calculate when it is set to translated by paragraph in the following example?
If I have one paragraph of 100 words, 4 sentences of 25 words each.
For the first translation is obviously calculates 100 words.
Now I change ONE word in ONE sentence.
- does it calculate 1 word?
- does it calculate 100 words?
- does it calculate 25 words of the changed sentence?
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
1) I think i didn't explain myself better, sorry. When i wrote "real" URL, i meant the one on the production site.
2) You say:
This huge segment must be a bug with WPML not correctly splitting the paragraphs when they are written in the classic editor
I thought i had explained it in the video, that "huge" segment is happening because the content was seen as if it was an INLINE paragraph (if you saw that content in the database it was all in one line, even though VISUALLY it looked "separated). The segmentation now with the new feature "bigger segments" has a different logic than the other. if you have a HUGE inline paragraph, even separated by periods, it could still take it as 1 segment.
Bigger segments are a good help when you want to translate automatically because we use an inhouse AI system + other AI systems to get the context and transform the text into something the translation engines can better understand, increasing the chances of getting a super improved translation. you'll see more of this with WPML 4.7 (what you're seeing now is just the beginning and might seem a bit "useless" even though it has its improvements already).
3) The test in test 1 and test 2 is exactly the same when it comes to the big segment or only segment that i show in the video. you indeed SEE the content separated but in reality it isn't and that's not something coming from WPML, I'm not entirely sure how to recreate it from scratch but it's created ONLY with copy-paste actions. if you wrote that, as i guess you did in divi, then it would not happen. in fact, in the ate-test page you had linked, i saw 2 segments, not 1 (see image attached)
4) You say:
1) it does not recognize the paragraphs
2) sometimes it translates it as sentence based segments
3) sometimes it translates as a single segment (in stead of paragraphs).
This is not a bug, but i would totally understand why you think it is since it's a different logic from the one we've been handling and if you use automatic translation, you'll see it will improve the translation (maybe not so much right now, depending on the content, but wait until WPML 4.7 is released and you'll see a real change there).
You can disable this feature by clicking on the gear icon found on the top right corner of the editor and deactivating "bigger segments".
By the way, another change that you'll start seeing improved in 4.7 is that the "good things" of the Classical translation editor (more organized content, separated by "sections" and then "segments") will also be applied to the Advanced editor. it's just that we're in a transition right now and can't push everything at once 🙁
5) Lastly you say:
how many words is the ATE supposed to calculate when it is set to translated by paragraph in the following example?
This is a difficult question because it all depends how ATE sees it. if it sees that as 1 segment, and it contains 100 words, then it will charge you accordingly. if it separates it into 2 segments, then it will charge you in a different way and only according to the segment translated.
I hidden link">recorded this video for you trying to explain how things work now, in hopes it clears things up. It's a bit long but i tried to create different scenarios and predict them, because the logic is different now.
I did find a bug when deactivating bigger segments, but i have to check if the problem is happening because of this specific text or not which i'll be doing soon, in this same sandbox.
Thank you for explaining in the video, but it's not correct.
2/3. The one big segemnt is not caused by copy and paste. In the classic editor:
- press enter to start a new paragraph » published as <p>
- press shift+enter to start a new line » published as <brp>
Even though you don't see the <p> tags in the text tab, they are there when you look at the page source of the published content, like I showed you in my video. If you look in the database, you also don't see the <p> tags, but when you copy & paste the content from the database into a text editor, you see that it contains character return [CR] and line feed [LF] characters. See the attached screenshot "wpml db.jpg".
Wordpress converts the [CR][LF] characters into <p> tags when publishing and WPML should do the same to get a consistent translation. See the attached screenshot "published.jpg"
It's either a bug or an ommision, but at the moment this new feature doesn't work for text entered in the classic editor. In my opinion this shoud be escalated to the development team.
4. If I have "I prefer bigger segments" enabled in the ATE, and it sometimes translates text as a single paragraph and sometimes as individual sentences, then there is something wrong. If you don't call that a bug, then it's at least not consistent.
5. As for the word count, when you edit translations it means that bigger segments result in higher credit usage, because it counts/calculates all words in the segment rather than the individual sentences. It also means less usage of the translation memory, because with bigger segments there is less chance on a match then with sentences. But I understand how it does it so let's leave this at that.
Languages: English (English )Spanish (Español )Italian (Italiano )
Timezone: Europe/Rome (GMT+01:00)
Oh darn, the last message i had sent didn't go through, sorry about that!
1) The devs are aware of this scenario (CRLF), i escalated it a while ago. I'm almost certain this is not a bug, but it's for them to decide now.
2) With bigger segments, you can't follow the same logic as shorter segments because they have 2 similar goals but with different workflows.
In short,
Shorter Segments
- Have a higher chance of not getting the correct content
- Have a better chance at making you save money because it will probably trigger more translation memory entries
- It's perfect for manual translation if you are translating from the editor.
Bigger Segments
- It's probably more costly because translation memory won't trigger as much as with shorter segments
- It's more comfortable for people that already have a translation and are just copy pasting it (and we have a LOT of people that follow that workflow)
- It will have better chances at translating things correctly because it provies better context (This will be further improved soon in WPML 4.7).
3) You say:
and it sometimes translates text as a single paragraph
This depends greatly on how the content is formatted in HTML. so if you can provide an example of each one you found (maybe just sending the URL and the segment that is separated, etc) i can dig into it and check what's going on.
4) The word count will not tell you how much you WILL spend, but how much you CAN spend. translation memory is not counted there fully (we are working on a different way to solve this, though).
That's also why we decided to leave shorter segments because we know there's not a single solution that will make everyone happy 🙁
1. Thank you for escalating. Please keep me informed on this.
2. That's a perfect and clear explanation, thank you!
3. Sure, here they are:
This page is translated as sentences, and the translation looks right. hidden link
This page is translated a single segement in the ATE, and the translation looks wrong (each sentence on it's own line) hidden link
4. I don't understand. Automatic translations are charged as 1 or 2 credits per word depending on the engine. And as far as I understand, words in unchanged sentences are not calculated because they are taken from the translation memory. My concern was exactly what you explained in #2.
5. Another thing, is the "I prefer bigger segments" setting also going to be added to the WPML settings? I have "Translate Everything Automatically" enabled in WPML, and since the ATE update it translates bigger segments by default. So by the time I go in the ATE to review the translations, it already has translated the bigger segments, before I had a change to flip that switch in the ATE settings. Therefore it would make more sense to me if that's a setting in WPML itself,.
6. Are the ATE settings saved and applied per wpml account, per site or per page? I have 19 websites in my wpml account. Do I have to change these settings only once, for every site or for each page that is being translated?
