Is This Translation Really Weird? Using ChatGPT to Improve Feedback and Quality

Wednesday, 4 PM: The meeting with the product team starts to discuss how the project we’re working on is progressing. Each team gives their update, and questions flow back and forth during the status updates.

Everything is going as it should.

The design team shows a sketch of their mockup to illustrate how the next version of the checkout cart will look.

The developers show which features will be released in the next sprint.

The audio team presents some sample files for the audio that will accompany the marketing campaign.

The Localization team gives an update on the status of translation for different content types and languages.

Everything seems fine—until THAT comment comes. A comment that, even though I’ve heard it many times throughout my career as a Localization professional, still brings a certain sense of frustration. It’s the kind of comment that tests the patience of the entire Localization team every time it’s made. These words, once they leave the stakeholder's lips, create a domino effect, and that comment is …. “This translation sounds weird.”

And there we go. BOOM. Once someone says that, we know things are about to get complicated. I’ve heard this comment so many times, and the outcome is usually the same: it’s up to the Localization team to figure out what “weird” actually means. This often leads to a snowball effect, making the situation even more difficult. If someone says, “This translation sounds weird,” and we have the “bad luck” that someone else adds, “You know, the other day, I was using the app in my language, and there was a text that also sounded a bit odd,” that’s when we know we’re in for a tough time. We’ll need to roll up our sleeves to figure out what’s wrong and defend the work of the translators and Localization team.

When “this translation is weird” refers to grammar mistakes, typos, or something similar, the outcome is simple and obvious: we accept the feedback, fix it, and take the necessary steps to ensure it doesn’t happen again.

The real complexity comes when there are no grammatical errors or glossary issues. When “weird” means “I would have chosen different words,” “I would have used different sentence structures,” or “I would have used different expressions,” things get tricky. In these cases, it’s a matter of preference, style, and subjectivity. Most of the time, there’s no objective way to evaluate whether “this expression” is less weird than another. To do this objectively, we’d need to run an A/B test or conduct a focus group where some users give feedback on the “weird” version, and others provide feedback on another version. However, due to scheduling, budget, and countless other reasons, it’s complicated to do this every time someone says, “this sounds weird.” So, how can we address this situation? How can we evaluate whether a translation is truly “weird”? And one more thing… could ChatGPT help us with this?

There are schools of thought that believe ChatGPT will slowly replace many professionals, while others believe that ChatGPT will help people become more productive and efficient. Is this the case in situations like this? Could ChatGPT be an ally rather than an enemy, helping me analyze the stakeholder’s “this translation sounds weird” comment?

Keep reading because this post is precisely about how ChatGPT can help us determine whether a translation is good or if it indeed needs improvement.

 Understanding What “This Translation is Weird” Means

View fullsize

Click HERE to downlaod the infographic

 Step 1: Look for Evidence

When a stakeholder says this, the first thing we need to do is try to get more specific feedback because “this translation is weird” doesn’t give us enough clues about what’s really happening. So, we must diagnose the situation and gather as much detail as possible. In many cases, the stakeholder won’t be able to articulate what’s wrong, so we can help them with this diagnosis by asking questions like:

• Could you specify which part of the translation feels weird to you?
  This helps narrow down the exact area of concern, whether it’s a specific word, phrase, or structure.

• What would you expect the translation to sound like?
  By asking for suggestions, you can determine whether the feedback is based on personal taste or whether they have a clearer vision that differs from the current translation.

• Is there a specific term or expression you think is incorrect or out of place?
  This allows them to point out if there’s an actual error or if it’s more about the flow or style of the text.

• How does this translation compare to others you’ve liked?
  If they’ve approved translations before, comparing this one to something they liked can help you understand their preferences.

• Is the message clear to you in the translation?
  This question checks if the “weird” feeling is due to confusion about the meaning, which could indicate a problem with clarity.

• Can you explain if it’s more the tone, style, or wording that feels off?
  This helps determine if it’s a stylistic issue (e.g., formal vs. informal tone) or simply different wording that they might not prefer.

• Can you show me an example of another text or translation that matches your expectations?
  This gives them a concrete way to express their preference and allows you to compare and contrast with the current version.

Collecting all this feedback in a spreadsheet or similar document for careful analysis is ideal.

Step 2: Check with the Translators

The Localization team is not the only one frustrated by these “this sounds weird” comments. For a translator, having a client say the translation sounds strange, but being unable to explain why is like being asked to solve a puzzle with missing pieces. That’s why the evidence collected in the previous step is critical for this assessment; the more complete it is, the better. At this stage, it’s also a good idea to reassess or re-share the typical reference materials clients usually provide, such as glossaries and style guides.

Add another column to the document where the translator can leave their impressions, whether they agree or disagree, and provide reasons if they don’t. The more detailed the response, the more credibility it gives to our analysis.

Step 3: Get Another Localization Professional’s Assessment

Sometimes translators can be reluctant to admit their translation has room for improvement; it’s like thinking your child is the most beautiful and not being able to see their flaws. It’s a human reaction, after all. We all become attached to our work, and it can be hard to stay objective. That’s why getting independent feedback is often the best option. In this case, an external language service provider would be the most professional way to approach it. They would look at the original translation, the stakeholder’s comment reporting the issue, and the original translator’s feedback.

Add a new column to the document for the reviewer to leave their feedback, like getting a second opinion, by the way, that’s something that’s not uncommon in fields like medicine. For example, when I needed surgery on my elbow, I consulted three specialists to ensure I knew exactly what type of surgery I’d undergo.

Step 4: ChatGPT as Our Agnostic Advisor?

By the end of steps 2 and 3, we’re accumulating enough knowledge to know if the “translation sounds weird” feedback has merit or is just a matter of preference. But we can go a step further. We can create a custom ChatGPT to help evaluate the translation. The great thing about this is that we can feed it detailed instructions and reference materials like style guides, glossaries, or previous translations. ChatGPT is excellent at analyzing data, so if we have strong reference material and past translations we consider a good benchmark, we can upload them to ensure the bot’s feedback is as thorough as possible.

To do this, it’s best to create a custom GPT ( see the below video to learn how) and use a prompt like this:

-----------

You are a professional translation reviewer. Below, you will find:

1. A glossary and style guide that outline the correct usage of terms and tone.

2. An example of a good translation that meets these standards.

3. The original content and the translated content that a stakeholder has commented on.

4. The stakeholder’s feedback, where they mentioned the translation feels "weird."

 ——

Your task is to review the translation based on the glossary, style guide, and the good example provided. Assess whether the feedback is based on personal preference or if the translation does not align with the style guide and glossary. Provide specific feedback on whether the translation needs improvement or if it is acceptable as it is.

Step 5: Produce the Final Report

I don’t remember who said it or where I read it, but it’s as important to do good work as it is to communicate it—and, I would add, explain how it was done. In this final step, we need to produce a report that explains to our stakeholders all the details of the audit we’ve conducted, including what was done in steps 2, 3, and 4.

We should emphasize the changes we made and explain why we didn’t make certain changes. This level of detail will help the stakeholders feel that we took their feedback seriously (which we did) and help them understand that, in the end, translation and localization are an art form—a way of expressing ideas. Different people will take different approaches to explaining something, and it’s possible that both approaches are correct!

 Conclusion

Handling feedback like "this translation sounds weird" requires us to be armed with patience and resist the temptation to push back, as we might be labeled as defensive. As described in this post, it’s better to take an analytical approach—gathering evidence, consulting with translators, and using tools like ChatGPT for an objective review. By embracing this process with openness and collaboration, we improve the quality of our translations and build stronger relationships with our stakeholders. Do you agree? Or does this suggested approach sound weird to you?