Language translation was one of the earliest applications of natural language processing (NLP). Today’s large language models have the potential to perform accurate translation in a highly contextually adaptive way. But they also have the potential to hallucinate or fail in surprising ways, while trying to convince you of its own accuracy.
My context
- use in church for hospitality (so theological accuracy is important)
- I’m a beginner speaker of the target language (so back-translation and transparency is important)
- need for alignment of section markers
Design principles
- Transparency: checking intermediate steps, back-translation side-by-side, visibility of alternatives, visibility of prompts
- Control: user can provide additional context (whether or not it’s a song, other freeform context)
- Editable intermediate steps: All intermediate outputs are editable, so if the LLM makes an error or the user wants to provide other feedback or guidance at any point, they can simply edit the result of any intermediate step. The app then regenerates any other results that were dependent on that result. The user can also simply choose to regenerate any intermediate step.
- Lightweight structure: Intermediate results are legible, but mark outputs using machine-readable region markers.
- Self-Critique: to avoid sycophancy, prompt suggests that input steps might have errors
- Iterative Improvement: can put the output back into the input
Implementation
I implemented this workflow as a Streamlit app using the Anthropic API. When the user supplies a source text, it executes the following workflow through ordinary sequential code, using Streamlit state management and caching to handle editing and regeneration:
- Analysis: It analyzes theological concepts, metaphors, and (for songs) syllable stress patterns.
- Literal Translation: It generates a word-for-word translation, including alternative translations (with back-translations) where needed.
- Clarity: Refines the literal translation; for songs it additionally considers meter and rhyme scheme. For songs, this stage also generates a syllabification of the translated text.
- Back-translation: It translates the result so far back to English and shows it side-by-side with the input text.
- Review: It generates a review of the translation for accuracy, preservation of theological concepts, and cultural appropriateness, and gives suggestions for improvement. The prompt includes “One or more errors may have been introduced in the translation. Identify and correct them.” The review prompt also includes the back-translation and (for songs) syllabification.
- Final Translation: It generates a final translation based on the review.
The app then shows the final translation side-by-side with the input text for easy comparison.
Example
Suppose we want to translate the song “And Can It Be” into Spanish:
Verse 1
And can it be that I should gain
An interest in the Savior's blood?
Died He for me, who caused His pain?
For me, who Him to death pursued?
Amazing love! How can it be
That Thou my God shouldst die for me?
Amazing love! How can it be
That Thou, my God, shouldst die for me? (For the output below, I had actually provided the entire song.)
Analysis Results
1. Theological Concepts and Terminology:
• Substitutionary atonement ("Died He for me") - Christ's sacrificial death
- Spanish concept: "expiación sustitutiva"
• Divine grace ("So free, so infinite His grace")
- Spanish concept: "gracia infinita"
• Original sin/bondage to sin ("Fast bound in sin and nature's night")
- Spanish concept: "esclavitud al pecado"
• Justification ("clothed in righteousness divine")
- Spanish concept: "justificación"/"vestido de su justicia"
• Adoption/inheritance ("claim the crown")
- Spanish concept: "adopción"/"herencia celestial"
• Scripture references:
- Romans 8:1 ("No condemnation")
- Ephesians 2:8-9 (salvation by grace)
- Colossians 1:18 ("living Head")
2. Cultural References:
• Imagery of royal court/throne
• Prison/dungeon imagery from 18th century England
• References to blood sacrifice, which may need careful handling in some cultures
3. Key Metaphors and Imagery:
• Imprisonment/chains ("imprisoned spirit," "chains fell off")
• Light vs. darkness ("nature's night," "dungeon flamed with light")
• Clothing ("clothed in righteousness")
• Royal/throne imagery
• Blood as payment/interest
4. Potential Translation Challenges:
• Archaic language ("shouldst," "Thou")
• Complex theological concepts
• Maintaining meter while preserving meaning
• Cultural adaptation of prison/dungeon imagery
• Poetic inversions of normal word order
5. Poetic Devices:
• Rhyme scheme: AABB in most verses
• Repetition of "Amazing love! How can it be"
• Parallel structure in many lines
• Internal rhyme and assonance
Stress Pattern Analysis (Verse 1):
"And CAN it BE that I should GAIN"
"An INT'rest IN the SAV-ior's BLOOD"
"DIED He for ME, who CAUSED His PAIN"
"For ME, who HIM to DEATH pur-SUED"
"a-MA-zing LOVE! How CAN it BE"
"That THOU, my GOD, shouldst DIE for ME"
The meter is primarily iambic, with some variations, typically 4 strong beats per line in the verses and 3 in the refrain.
Literal Translation
Verse 1
¿Y puede ser que yo debería ganar
Un interés en la sangre del Salvador?
¿Murió Él por mí, quien causó Su dolor?
¿Por mí, quien Lo persiguió hasta la muerte?
¡Amor asombroso! ¿Cómo puede ser
Que Tú mi Dios debieras morir por mí?
¡Amor asombroso! ¿Cómo puede ser
Que Tú, mi Dios, debieras morir por mí?
[Key terms:
- "interest": "interés" [alternative: "parte" - meaning "share/portion"]
- "pursued to death": "persiguió hasta la muerte" [alternative: "llevó a la muerte" - meaning "led to death"]
- "amazing": "asombroso" [alternatives: "maravilloso", "increíble" - meaning "wonderful", "incredible"]]
Clarity
Verse 1
¿Cómo es posible que yo gané
Una parte en la sangre de mi Rey?
¿Murió por mí, que le causé dolor?
¿Por mí, que lo llevé a morir?
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?It also generates a syllabification, which is surprisingly accurate in this case despite the fact that Claude 3.5 is not character-level (it uses a subword tokenizer).
Verse 1
¿có-mo es po-SI-ble que yo ga-NÉ
u-na PAR-te en la SAN-gre de mi REY?
mu-RIÓ por MÍ, que le cau-SÉ do-LOR?
¿por MÍ, que lo lle-VÉ a mo-RIR?
¡qué GRAN-de a-MOR! ¿có-mo PUE-de SER
que TÚ, mi DIOS, mu-RIE-ras por MÍ?
¡qué GRAN-de a-MOR! ¿có-mo PUE-de SER
que TÚ, mi DIOS, mu-RIE-ras por MÍ?
Back-Translation
Verse 1
How is it possible that I gained
A part in the blood of my King?
Did He die for me, who caused Him pain?
For me, who led Him to die?
What great love! How can it be
That You, my God, would die for me?
What great love! How can it be
That You, my God, would die for me?
Review
Overall Assessment:
The translation successfully captures the theological depth and emotional weight of this classic hymn while maintaining singability. Here's a detailed analysis:
Strengths:
1. Theological accuracy is well-maintained, preserving key concepts like redemption, grace, and justification
2. The emotional progression of the narrative remains intact
3. The Spanish flows naturally while keeping the poetic quality
4. The meter and rhyme schemes work well with the original tune
5. The use of "tú" forms maintains the intimate relationship with God
Areas for Improvement:
1. Verse 1:
- "que yo gané" could be improved to "alcanzar" to better reflect the theological concept of receiving rather than earning grace
- Suggested revision: "¿Cómo es posible alcanzar"
... other verses ...
Recommendations:
1. Consider revising verse 1's opening to use "alcanzar" instead of "gané"
2. Use "Mis cadenas rotas están" in verse 3 for stronger imagery
3. Consider tightening verse 4's ending to match the original's structure more closely
Despite these suggestions, the translation is strong and singable, maintaining both theological accuracy and poetic beauty. The adaptations made serve the target language and culture well while preserving the essential message of the hymn.Note, with some skepticism, the feel-good judgment at the end.
Final Translation
I’ll include the whole song here for anyone interested.
Verse 1
¿Cómo es posible alcanzar
Una parte en la sangre de mi Rey?
¿Murió por mí, que le causé dolor?
¿Por mí, que lo llevé a morir?
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
Verse 2
Dejó Su trono celestial
(¡Tan libre y sin fin es Su amor!)
Se humilló con tanto amor
Y dio Su sangre, el Salvador
¡Qué inmensa gracia y libertad!
¡Me encontró en mi oscuridad!
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
Verse 3
Mi espíritu en prisión yacía,
Atado en noche y perdición;
Tu luz brillante me alcanzó,
Y el calabozo iluminó;
Mis cadenas rotas están;
Me alcé y decidí seguir
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
Verse 4
Ya no hay temor ni condenación;
¡Jesús, y todo en Él, es mío!
En Él yo vivo, mi Salvador,
Vestido en Su justicia estoy,
Con confianza llegaré
Al trono eterno, y por Jesús
La corona obtendré
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?
¡Qué grande amor! ¿Cómo puede ser
Que Tú, mi Dios, murieras por mí?Challenges
- Meter: often made translations that were very metrically off; addressed by adding meter prompt
- Self-Assessment Hallucination: An earlier revision included numeric scores in the review, but I found them to be meaningless. In general we should be skeptical of numeric ratings from LLMs, especially when the rating is generated before the rationale (since it must rationalize the rating it generated).
Next Steps
- Systematic Evaluation
- Simplify flow: do we need to have the “literal translation” step? Can we evaluate whether it helps or hurts?
- Interactivity
- Alternatives are currently displayed as static text; they should be interactive
- Perhaps one could simply have a conversation around the translation, with continuously updating back-translation and contextualized (sidebar comments?) critique?
- The conversation could include tool calls for suggesting translation updates, context updates, etc.
- Comments could have conversations attached to them; there could also be an overall conversation.
- The translation could simply be editable by both the user and the LLM; the LLM could perhaps see a history of what changes the user had made (or maybe both “author”s’ changes show up as “tracked changes” until approved)
- Working with rights holders: Currently, it would only be legal to distribute the outputs of this tool when applied to public-domain songs (of which there are many), but I would like to work with authors and rights-holders of songs to