Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

=============================================================================
     when using a text editor like vim inside xfce4-terminal to create a
     Hebrew text file, the alphabet characters enter in the correct
     order but non-Hebrew characters, like punctuation, appear on the
     wrong end of the line.

     When following the punctuation with another Hebrew letter, they
     then both jump to the correct position.

     There are bidirectional support libraries like libfribidi that
     handle right-to-left characters correctly. mlterm is an example of
     a terminal that uses them so both alphabet and punctuation appear
     in the order they were typed in.

     When using cat in mlterm, the file again appears correct whereas
     cat on the file inside xfce4-terminal has the characters reversed.

     Temporarily, a .vimrc can be added with commands to follow selected
     punctuation marks with a UNICODE character so the order is correct,
     as if punctuation had been followed with a letter.

=============================================================================
*** End of the template - remove these template lines ***

Hello,

With all due respect, please familiarize yourself with the "BiDi in
Terminal Emulators" work at https://terminal-wg.pages.freedesktop.org/bidi/
, as well as BiDi algorithm itself.

Several years ago the situation was that nobody knew how to do BiDi in
terminals. Some folks (including mlterm and vim) did _something_ that might
have been enough for certain particular needs, without seeing and
understanding the whole picture. Others did something else. Other terminals
that claimed to support BiDi (e.g. Konsole, PuTTY) did it differently from
mlterm. Other terminal-based apps that claimed to support BiDi (e.g. Emacs)
did it differently from Vim. Some combinations more or less worked
"accidentally".

(By the way, mlterm already does 3 different things based the command line
option --bimode. How ridiculous is that? Putting on my user hat, I don't
care, I just want things to work, I don't want to restart my terminal if I
encounter an app that needs a different behavior, and I don't want to
remember which app needs which mode. Taking off my user hat and putting on
my developer hat: Which one should I develop my app against???)

The terminal world is pretty complex; simple tools like "cat" and
sophisticad apps like "vim" have fundamentally different needs, I'd argue
that a terminal emulator is the union of at least two utterly different
words (whose BiDi needs are also totally different). In addition, BiDi is
pretty complex, too.

Then someone with notable experience in terminals and notable experience in
BiDi spent years (yes, literally years) in the back of his mind pondering
about the problem space. Once the big picture became clean, he then spent
months studying and evaluating prior work (documents and software),
designing what needs to be done, documenting all of it along with
rationale, and coming up with an implementation (along with extensive
tests) in VTE, the terminal widget used by xfce4-terminal and others. (This
person is me.)

VTE is the first terminal that does not just do _some_ random BiDi, but
understands the whole picture and does a thoroughly designed thing where
finally all the parts _can_ fit together. Notice: _can_ fit together, not
_do_ fit together immediately, that would be impossible. Apps have
different needs against the behavior of the terminal, they need to tell the
terminal which mode to switch to.

Don't worry, this work does not reinvent BiDi shuffling from scratch: it
just connects up the Unicode BiDi Algorithm (UBA) to the world of terminal
emulation, as you also wish it would do.
line

This is one of the possible behaviors of UBA, a behavior you can often
experience in all other contexts as well (e.g. graphical text editors, web
pages, even UBA's online test suite). The key concept here is "base
direction" a.k.a. "paragraph direction". You can programatically configure
it in VTE.

VTE is the first terminal that understands that a single mode of BiDi
operation cannot satisfy the needs of all the apps, therefore offers apps
to choose the mode they wish to have. In fact, VTE is the first terminal
that provides mathematical proof (in the aforementioned documentation) that
a full-blown application, such as vim, cannot be made to function properly
if it's up for the terminal to rearrange the letters. Vim can only work
properly if vim itself does the shuffling, and the terminal lays down the
characters from left to right. Something that VTE can be asked to do.

BiDi and terminals are two extremely complicated things on their own, their
combination is exponentially more complex. What happens in the mlterm + vim
combo cannot be scaled up to work across all the terminal-based apps. Mind
you, I'm absolutely certain it doesn't even work correctly in vim if you
begin to stress test it (e.g. a mixed-direction line's width exceeds the
display width).

VTE is a platform where all apps can implement proper BiDi. It does not
happen magically, apps need to tell VTE which mode to switch to. Please ask
Vim folks to update their software accordingly.

e.

" Quick way to insert RLM (Right-to-Left Mark) after punctuation
inoremap <expr> . ( &keymap == 'He' ? '.<C-V>u200f' : '.' )
inoremap <expr> , ( &keymap == 'He' ? ',<C-V>u200f' : ',' )
inoremap <expr> ? ( &keymap == 'He' ? '?<C-V>u200f' : '?' )
inoremap <expr> ! ( &keymap == 'He' ? '!<C-V>u200f' : '!' )
inoremap <expr> " ( &keymap == 'He' ? '"<C-V>u200f' : '"' )
inoremap <expr> - ( &keymap == 'He' ? '-<C-V>u200f' : '-' )
inoremap <expr> : ( &keymap == 'He' ? ':<C-V>u200f' : ':' )

Sorry for the slow response, I did not get notified about your comment (nor
do I know what I need to do for you to get notified about mine), Debian's
bugtracker has always been a legacy mystery to me.


The proof is not very direct, and surely isn't as nicely formatted as the
proofs you are used to.

"RTL and BiDi Introduction" -> "Why terminals are a truly special story" is
a key point. It elaborates what happens if a non-BiDi-aware app (let's say
Vim) wishes to display (and scroll horizontally) a 100-char wide text file
in an 80-char wide terminal. If the app knows nothing about BiDi and the
terminal shuffles the letters, combined with a heavily bi-directional line,
then the result is catastrophic. The editor sends out the wrong set of
letters, there's no way the terminal could rearrange them properly because
it's not just the arrangement, but the set of characters seen by the
terminal is wrong to begin with.

The editor needs to perform BiDi to be able to perform visual cropping. If
the terminal also does BiDi then it would be a "double BiDi" which is
obviously totally wrong. A way to remedy would be for the application to
perform a "reverse BiDi" to cancel that, resulting in a pretty bad
architecture; but more importantly, as I show in the previous section "RTL
and BiDi text handling in general", such a "reverse BiDi algorithm" doesn't
exist (if BiDi controls are not allowed; if they are allowed then a big
hammer is to place everything between LRO...PDF but then a separate mode
explicitly designed for this kind of operation is cleaner).

Create a text file that contains two 100-char lines. One is fully English,
the other is a complex mixture of the two directionalities. Open it in a
text editor of your choice that does _not_ perform BiDi. Scroll
horizontally in your choice of terminal emulator, one that does BiDi
according to your preference, e.g. mlterm. Notice that what you see is a
total craziness, not resembling at all how you'd slide a larger piece of
paper under a smaller viewing window. You can't fix it in the terminal. You
need to fix it in the app.


Re punctuation marks:

Note that whatever I say here below is utterly irrelevant because it's all
within the "implicit" mode (the terminal does BiDi), whereas, as per the
above, this mode of operation is not suitable for text editors, they need
to switch to "explicit" mode (the terminal does _not_ do BiDi, the app is
expected to).

Open a new LibreOffice text document. Type some RTL text followed by a
question mark, notice that the question mark appears on the "wrong" (i.e.
unexpected for you; on the right) side.

Open a html page as simple as this in your favorite browser (I trired
Firefox and Chromium):

<html><body><textarea></textarea></body></html>

and type the same thing, notice the same behavior again.

Open the UBA test page at https://util.unicode.org/UnicodeJsps/bidi.jsp ,
type the same thing. At the text area you'll see the same behavior again.

Choose "LTR Paragraph Direction", then click "Show Bidi". Notice that under
"Reordered" you get the question mark on the right; again the behavior
you're unhappy with.

VTE implement this exact behavior, specified by the UBA, as its default
behavior.

Now, you might choose "Auto Paragraph Direction" or "RTL Paragraph
direction" on the BiDi test page. You might apply a dir=auto or dir=rtl
attribute to the textarea node in the tiny html example. I'm sure
LibreOffice also has something equivalent. These modify the behavior to
your liking.

In VTE you can also do the same, we have escape sequences for that. First
you need to understand what role "Paragraph (a.k.a. base) direction" plays
in UBA, then pick your preferred behavior (in VTE: via escape sequences; in
mlterm: you're probably out of luck, it only offers one of the behaviors).

One thing we could perhaps argue about is: which one of the possible
behaviors (autodetection or not) should be the terminal's default. One is
better in certain situtations, the other is better in others, it's hard to
pick one. Autodetection (your preferred behavior) would mean that a file
listing (e.g. output of simple "ls") aligns English filenames to the left
edge of the terminal and Arabic/Hebrew filenames to the right edge of the
terminals; for lists I believe it's more desireable to align everything to
the same side. Especially for those who don't really care about BiDi, just
happen to have a single Arabic or Hebrew filename among all the English
ones, this behavior would be very confusing. I had no strong data but a
slight feeling that no autodetection is better suited as the default value
(whenever the terminal is used correctly; running Vim and letting the
terminal rearrange the letters does not count).

But, again, it's all irrelevant for Vim. The autodetected (or hardwired
RTL) paragraph direction within the terminal's implicit BiDi mode cannot
fix its behavior. Only the explicit BiDi mode can fix it. Only Vim knows
the entire string to know where that question mark needs to go, the
terminal potentially only sees a small substring thereof (even a totally
wrong substring if Vim didn't do BiDi), it doesn't have the knowledge to
correctly place it.


VTE implements exactly what the BiDi algorithm says. In fact VTE doesn't
reimplement it, it just calls the corresponding Fribidi methods. The BiDi
algorithm is more complicated than you think. It has multiple possible
behaviors depending on the paragraph direction which is to be passed to it
as an external parameter. One of the possible behaviors places the question
marks at the place you find undesireable, but again, this is what the UBA
tells to do, VTE did not make it up or did not falsely implement UBA. The
paragraph direction (including whether to autodetect it or not) can be
modified according to your liking via escape sequences, as per the
documentation, go and find the mode that suits your use cases the best. But
all this is within "implicit mode" suitable for simple utilities' outputs
where the terminal sees the entire string. For complex utilities where the
terminal potentially only sees a cropped string (e.g. text editors) this
theoretically cannot work at all, the only thing that can work is for the
app (e.g. Vim) to shuffle and the terminal to leave the order unchanged
("explicit mode").


e.