Problem with Devanagari while using combining marks

[Shreeshrii]

I am trying to use the combining marks and characters from Devanagari, DevanagariExtended and VedicExtensions unicode blocks with text in other Indic scripts, eg. Gujarati.

ucharclasses is working fine with xetex (texlive2016) to display the combining marks and characters with another script. however text in Devanagari script is not getting rendered correctly.

An MWE is posted below.

\textbackslash mainfont (Rasa-Regular) is in Gujarati script and also supports Latin range.
\textbackslash vedafont (Siddhanta) supports Devanagari, DevanagariExtended, VedicExtensions and Latin ranges and is defined without specifying a script. \textbackslash devafont is the same font as \textbackslash vedafont but also specifies script=devanagari as part of definition.

\textbackslash ucharclasses with \textbackslash SetTransitionsFor for Devanagari, DevanagariExtended, VedicExtensions combining marks works well with the default Gujarati script. But text in Devnagari script does not render correctly, since the script is NOT specified for \textbackslash vedafont.

Using \textbackslash devafont for transition to devanagari renders the devanagari text correctly but the combining marks break and are displayed with broken circles.

[Shreeshrii]

\documentclass[a5paper,11pt,twoside,onecolumn,openany,final]{memoir}
\usepackage[a5paper,hmargin=0.7in,vmargin=0.7in]{geometry}
\usepackage{multicol,polyglossia,etoolbox,fontspec}
%
\XeTeXgenerateactualtext=1
%
\defaultfontfeatures{Mapping=tex-text}
\setmainlanguage{gujarati}
\setotherlanguage{english}
%
\setmainfont[Scale=1,Script=Gujarati]{Rasa-Regular.ttf}
\newfontfamily{\vedafont}[Scale=1.2]{Siddhanta}
\newfontfamily{\devafont}[Script=Devanagari,Scale=1.2]{Siddhanta}
%
\usepackage[Latin, Devanagari, DevanagariExtended, Gujarati, VedicExtensions]{ucharclasses}
%\setTransitionsFor{Devanagari}{\begingroup\devafont}{\endgroup}
\setTransitionsFor{Devanagari}{\begingroup\vedafont}{\endgroup}
\setTransitionsFor{DevanagariExtended}{\begingroup\vedafont}{\endgroup}
\setTransitionsFor{VedicExtensions}{\begingroup\vedafont}{\endgroup}
%
\begin{document}
This trial is to use the combining marks and characters from Devanagari, DevanagariExtended and VedicExtensions unicode blocks with text in other Indic scripts, eg. Gujarati.
\\
This is the Gujarati text with the vedic accents which are from Devanagari, DevanagariExtended, VedicExtensions blocks.
\\
ૐ ભૂર્ભુવઃ સ્વઃ \\
ૐ ॥ હિર॑ણ્યવર્ણાં॒ હરિ॑ણીં સુ॒વર્ણ॑રજ॒તસ્ર॑જામ્ ।\\
ચ॒ન્દ્રાં હિ॒રણ્મ॑યીં લ॒ક્ષ્મીં જાત॑વેદો મ॒ આવ॑હ ॥ ૧॥\\
\\
તાં મ॒ આવ॑હ॒ જાત॑વેદો લ॒ક્ષ્મીમન॑પગા॒મિની᳚મ્ ।\\
યસ્યાં॒ હિર॑ણ્યં વિ॒ન્દેયં॒ ગામશ્વં॒ પુરુ॑ષાન॒હમ્ ॥ ૨॥\\
\\
This is plain devanagari text.
\\
अस्य श्रीपञ्चमुखिहनुमत्कवचस्तोत्रमन्त्रस्य, ब्रह्मा ऋषिः ।
गायत्री छन्दः । श्रीहनुमान् देवता । रां बीजम् । मं शक्तिः ।
\end{document}

[Shreeshrii]

\newfontfamily{\vedafont}[Scale=1.2]{Siddhanta}
\setTransitionsFor{Devanagari}{\begingroup\vedafont}{\endgroup}

When using the above, devanagari text is not rendered ok. see attached image

[Shreeshrii]

\newfontfamily{\devafont}[Script=Devanagari,Scale=1.2]{Siddhanta}
\setTransitionsFor{Devanagari}{\begingroup\devafont}{\endgroup}

When using the above, the devanagari script text is ok, but the combining marks don't work.
Please see attached image.

[Shreeshrii]

Is there a way to use ucharclasses for both combining marks and devanagari?
Thanks!

[Pomax]

I'm not sure what would cause this, other than if some of the combining characters are treated as being "not part of the same unicode block". Would you happen to have a reduced case with the shortest possible word (or even few non-sense letters that involve the combining characters?)

This might be another instance of #13

[Shreeshrii]

Sorry for the delay in reply. I did some more testing and what I find is that the combining marks (that I tested with) from DevanagariExtended and VedicExtensions unicode blocks do combine correctly with other scripts.

The problem is with the devanagari block which has codepoints for devanagari script but also has certain combining marks which can be used with other Indic scripts.

When I do NOT specify Script=Devanagari then the combining happens correctly, but not otherwise.

A reduced case is attached below.

[Shreeshrii]

\newfontfamily{\devanagarifont}[Scale=1,Color=purple]{Siddhanta}

testchar.pdf

\documentclass[a4paper,11pt,twoside,onecolumn,openany,final]{memoir}
\usepackage[a4paper,hmargin=0.7in,vmargin=0.7in]{geometry}
\usepackage{multicol,polyglossia,etoolbox,fontspec}
\usepackage{xcolor}
%
\XeTeXgenerateactualtext=1
%
\defaultfontfeatures{Mapping=tex-text}
\setmainlanguage{english}
\setotherlanguages{sanskrit,tamil,gujarati}
%
\setmainfont[Color=teal]{FreeSerif}
\newfontfamily{\englishfont}[Color=blue]{FreeSerif}
\newfontfamily{\latinfont}[Scale=1, Color=brown]{FreeSerif}
\newfontfamily{\devanagarifont}[Scale=1,Color=purple]{Siddhanta}
\newfontfamily{\tamilfont}[Script=Tamil]{Lohit Tamil Classical}
\newfontfamily{\gujaratifont}[Path=../fonts/,Scale=1,Script=Gujarati]{Rasa-Regular.ttf}
\newfontfamily{\vedafont}[Scale=1,Color=red]{Siddhanta}
%
\usepackage[Latin, Devanagari, DevanagariExtended, Gujarati, Tamil, VedicExtensions]{ucharclasses}
\setTransitionsForLatin{\latinfont}{}
\setTransitionTo{DevanagariExtended}{\vedafont}
\setTransitionTo{VedicExtensions}{\vedafont}
\setTransitionTo{Devanagari}{\devanagarifont}
\setTransitionTo{Gujarati}{\gujaratifont}
\setTransitionTo{Tamil}{\tamilfont}

\setlength{\parindent}{0pt}
%
\begin{document}
\obeylines
This trial is to use the combining marks and characters from Devanagari, DevanagariExtended and VedicExtensions unicode blocks with text in other Indic scripts, eg. Gujarati, Tamil, etc.
\medskip
"The Devanagari Extended block, as well as the Vedic Extensions block and the Devanagari block, include characters that are used to indicate tone in Vedic Sanskrit. " 
"This block of characters is used chiefly for Vedic Sanskrit, although many of the characters are generic and can be used by other Indic scripts. "
\medskip
Source: http://www.unicode.org/versions/Unicode9.0.0/ch12.pdf
\medskip
Sample text for Devanagari
\hrule
ठ॑  શ્રી॑     ગ॑      த॑  0951    DEVANAGARI STRESS SIGN UDATTA  
ठ॒  શ્રી॒     ગ॒      த॒  0952    DEVANAGARI STRESS SIGN ANUDATTA  
\medskip
Sample text for DevanagariExtended
\hrule
ठ꣱  શ્રી꣱     ગ꣱  த꣱  A8F1    COMBINING DEVANAGARI SIGN AVAGRAHA  
ठꣳ  શ્રીꣳ     ગꣳ      தꣳ  A8F3    DEVANAGARI SIGN CANDRABINDU VIRAMA
\medskip
Sample text for VedicExtensions
\hrule
ठ᳚  શ્રી᳚     ગ᳚  த᳚  1CDA    VEDIC TONE DOUBLE SVARITA 
\medskip

\end{document}

[Shreeshrii]

\newfontfamily{\devanagarifont}[Scale=1,Color=magenta, Script=Devanagari]{Siddhanta}

testchar-broken.pdf

\documentclass[a4paper,11pt,twoside,onecolumn,openany,final]{memoir}
\usepackage[a4paper,hmargin=0.7in,vmargin=0.7in]{geometry}
\usepackage{multicol,polyglossia,etoolbox,fontspec}
\usepackage{xcolor}
%
\XeTeXgenerateactualtext=1
%
\defaultfontfeatures{Mapping=tex-text}
\setmainlanguage{english}
\setotherlanguages{sanskrit,tamil,gujarati}
%
\setmainfont[Color=teal]{FreeSerif}
\newfontfamily{\englishfont}[Color=blue]{FreeSerif}
\newfontfamily{\latinfont}[Scale=1, Color=brown]{FreeSerif}
\newfontfamily{\devanagarifont}[Scale=1,Color=magenta, Script=Devanagari]{Siddhanta}
%\newfontfamily{\devanagarifont}[Scale=1,Color=purple]{Siddhanta}
\newfontfamily{\tamilfont}[Script=Tamil]{Lohit Tamil Classical}
\newfontfamily{\gujaratifont}[Path=../fonts/,Scale=1,Script=Gujarati]{Rasa-Regular.ttf}
\newfontfamily{\vedafont}[Scale=1,Color=red]{Siddhanta}
%
\usepackage[Latin, Devanagari, DevanagariExtended, Gujarati, Tamil, VedicExtensions]{ucharclasses}
\setTransitionsForLatin{\latinfont}{}
\setTransitionTo{DevanagariExtended}{\vedafont}
\setTransitionTo{VedicExtensions}{\vedafont}
\setTransitionTo{Devanagari}{\devanagarifont}
\setTransitionTo{Gujarati}{\gujaratifont}
\setTransitionTo{Tamil}{\tamilfont}

\setlength{\parindent}{0pt}
%
\begin{document}
\obeylines
This trial is to use the combining marks and characters from Devanagari, DevanagariExtended and VedicExtensions unicode blocks with text in other Indic scripts, eg. Gujarati, Tamil, etc.
\medskip
"The Devanagari Extended block, as well as the Vedic Extensions block and the Devanagari block, include characters that are used to indicate tone in Vedic Sanskrit. " 
"This block of characters is used chiefly for Vedic Sanskrit, although many of the characters are generic and can be used by other Indic scripts. "
\medskip
Source: http://www.unicode.org/versions/Unicode9.0.0/ch12.pdf
\medskip
Sample text for Devanagari
\hrule
ठ॑  શ્રી॑     ગ॑      த॑  0951    DEVANAGARI STRESS SIGN UDATTA  
ठ॒  શ્રી॒     ગ॒      த॒  0952    DEVANAGARI STRESS SIGN ANUDATTA  
\medskip
Sample text for DevanagariExtended
\hrule
ठ꣱  શ્રી꣱     ગ꣱  த꣱  A8F1    COMBINING DEVANAGARI SIGN AVAGRAHA  
ठꣳ  શ્રીꣳ     ગꣳ      தꣳ  A8F3    DEVANAGARI SIGN CANDRABINDU VIRAMA
\medskip
Sample text for VedicExtensions
\hrule
ठ᳚  શ્રી᳚     ગ᳚  த᳚  1CDA    VEDIC TONE DOUBLE SVARITA 
\medskip

\end{document}

[Shreeshrii]

Questions:

1. I am using both Polyglossia and ucharclasses. Could using both be causing problems?
2. "U+0951..U+0954 are a set of combining marks used in transcription of Sanskrit texts." Source: Unicode Ch 12

Is it possible to create a informal unicode range for the above in Devanagari range along with DevanagariExtended and VedicExtensions - so as to apply the rules for combining marks to these.

Alternately, is it possible to define just these 4 codepoints as a separate custom range?

Thanks!

[Pomax]

It sounds like the package might need some knowledge on what is "script" and what is script-agnostic diacritics, so that diacritics don't insert block change commands, which would absolutely break up the combining of diacritic with base glyph(s).

[Shreeshrii]

http://unicode.org/charts/PDF/U0900.pdf

​​

Vedic tone marks

0951 $॑ DEVANAGARI STRESS SIGN UDATTA

= Vedic tone svarita

• mostly used for svarita, with rare use for udatta

• used also in Vedic texts written in other scripts

0952 $॒ DEVANAGARI STRESS SIGN ANUDATTA

= Vedic tone anudatta

• used also in Vedic texts written in other scripts

ShreeDevi

---

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

On Thu, Aug 11, 2016 at 11:18 PM, Mike Kamermans notifications@github.com
wrote:

It sounds like the package might need some knowledge on what is "script"
and what is script-agnostic diacritics, so that diacritics don't insert
block change commands, which would absolutely break up the combining of
diacritic with base glyph(s).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_oxyy3Dy_25Sh_C02cy5mKwJOjBAaks5qe2BagaJpZM4Jd_sX
.

[Shreeshrii]

It seems that this issue is font related. When I used a different devanagari font then both the combining marks with different script and regular devanagari script worked. I'll post the tex source and resultant pdf later.

I am guessing - it could be related to new features added in fontspec for texlive2016 - http://ctan.imsc.res.in/macros/latex/contrib/fontspec/README.md

Incorporate "new" font script tags for Indic fonts. E.g., when selecting Script=Bengali, fontspec will first query the font for the bng2 OpenType script, and if not found select beng.

[Shreeshrii]

testuchar.tex.txt
testuchar.log.txt
testuchar.pdf

tex source, resulting pdf and log file are attached, showing that devanagari is being rendered correctly when using a different font .

\documentclass[a4paper,11pt,twoside,onecolumn,openany,final]{memoir}
\usepackage[a4paper,hmargin=0.7in,vmargin=0.7in]{geometry}
\usepackage{multicol,polyglossia,etoolbox,fontspec}
\usepackage{xcolor}
\XeTeXgenerateactualtext=1
%
\defaultfontfeatures{Mapping=tex-text}
\setmainlanguage{english}
\setotherlanguages{sanskrit,tamil,gujarati}
%
\setmainfont[Color=teal]{FreeSerif}
\newfontfamily{\englishfont}[Color=blue]{FreeSerif}
\newfontfamily{\latinfont}[Scale=1]{FreeSerif}
%\newfontfamily{\devanagarifont}[Scale=1,Color=magenta, Script=Devanagari]{Siddhanta}
%\newfontfamily{\devanagarifont}[Scale=1.5,Color=purple]{Siddhanta}
\newfontfamily{\devanagarifont}[Scale=1,Color=purple]{Lohit Devanagari}
\newfontfamily{\tamilfont}[Script=Tamil,Scale=1.2,Color=brown]{Lohit Tamil Classical}
\newfontfamily{\gujaratifont}[Scale=1,Script=Gujarati,Color=gray]{Lohit Gujarati}
\newfontfamily{\vedafont}[Scale=1.2,Color=red]{Siddhanta}
%
\usepackage[Latin, GeneralPunctuation, Devanagari, DevanagariExtended, Gujarati, Tamil, VedicExtensions, SuperscriptsAndSubscripts]{ucharclasses}
\setTransitionsForLatin{\latinfont}{}
\setTransitionTo{GeneralPunctuation}{\englishfont}
\setTransitionTo{SuperscriptsAndSubscripts}{\englishfont}
\setTransitionTo{DevanagariExtended}{\vedafont}
\setTransitionTo{VedicExtensions}{\vedafont}
\setTransitionTo{Devanagari}{\devanagarifont}
\setTransitionTo{Gujarati}{\gujaratifont}
\setTransitionTo{Tamil}{\tamilfont}
%
\setlength{\parindent}{0pt}
%
\begin{document}
\obeylines
This trial is to use the combining marks and characters from Devanagari, DevanagariExtended and VedicExtensions unicode blocks with text in other Indic scripts, eg. Gujarati, Tamil, etc. 
\medskip
This sample also has superscripts for 2, 3 and 4 used with Tamil
\medskip
"The Devanagari Extended block, as well as the Vedic Extensions block and the Devanagari block, include characters that are used to indicate tone in Vedic Sanskrit. " 
"This block of characters is used chiefly for Vedic Sanskrit, although many of the characters are generic and can be used by other Indic scripts. "
\medskip
Source: http://www.unicode.org/versions/Unicode9.0.0/ch12.pdf
\medskip
Sample combining mark from Devanagari
\hrule
ठ॑  શ્રી॑     ગ॑      த॑  0951    DEVANAGARI STRESS SIGN UDATTA  with Devanagari, Gujarati, Gujarati, Tamil Scripts
ठ॒  શ્રી॒     ગ॒      த॒  0952    DEVANAGARI STRESS SIGN ANUDATTA  with Devanagari, Gujarati, Gujarati, Tamil Scripts
\medskip
Sample combining mark from DevanagariExtended
\hrule
ठ꣱  શ્રી꣱     ગ꣱  த꣱  A8F1    COMBINING DEVANAGARI SIGN AVAGRAHA  with Devanagari, Gujarati, Gujarati, Tamil Scripts
ठꣳ  શ્રીꣳ     ગꣳ      தꣳ  A8F3    DEVANAGARI SIGN CANDRABINDU VIRAMA  with Devanagari, Gujarati, Gujarati, Tamil Scripts
\medskip
Sample combining mark from VedicExtensions
\hrule
ठ᳚  શ્રી᳚     ગ᳚  த᳚  1CDA    VEDIC TONE DOUBLE SVARITA  with Devanagari, Gujarati, Gujarati, Tamil Scripts
\medskip
Tamil text ( with superscripts using FreeSerif as Latinfont and Englishfont)
Needed to add GeneralPunctuation unicode block also to cater to invisible space/combining marks.
Otherwise the font change was not happening correctly.
\hrule
ௐ கணாநாம் த்வா கணபதி
ௐ  க³ணாநாம் த்வா க³ணபதி 
ஜ்யேஷ்ட²ராஜம் ப்³ரஹ்மணாம் ப்³
\medskip
Tamil with accents and superscripts 
\hrule
க³॒ம்⁠த⁴॒⁠த்³வா⁠ராம் து³⁠ராத⁴॒ர்ஷாம்॒ நி॒த்யபு⁠ஷ்டாம் கரீ॒ஷிணீ᳚ம் ।
ஈ॒ஶ்வரீ॑ ꣳ  ஸர்வ⁠பூ⁴தா॒நாம்॒ தாமி॒ஹோப॑ஹ்வயே॒ ஶ்ரியம் ॥ 9॥
\medskip
\hrule
devanagari is not correct with Siddhanta and FreeSerif font (without Script=Devanagari) 
but is ok with Lohit Devanagari, Aparajita, Arial Unicode
\medskip
गं॒ध॒द्वा॒रां दु॑राध॒र्षां॒ नि॒त्यपु॑ष्टां करी॒षिणी᳚म् ।
ई॒श्वरी॑ꣳ सर्व॑भूता॒नां॒ तामि॒होप॑ह्वये॒ श्रियम् ॥ ९॥
\end{document}

[Pomax]

hm, given that information, I'm not sure what the best course of action is here... it kind of feels like "using a modern font" solves the issue, in which case this is no longer a ucharclasses-specific issue? How do you feel about that?

[Shreeshrii]

1. Yes, as this is no longer a ucharclasses-specific issue, you can close it.
2. I have forked the project and split the devanagari unicode block into three, it makes it easier for me to apply a different font for the combining marks. Please see https://github.com/Shreeshrii/ucharclasses/blob/master/ucharclasses.sty

I have called the ranges as DevanagariPreMarks, DevanagariMarks and DevanagariPostMarks.

Possibly these could be combined together to form an informal Devanagari group.

1. I would suggest adding an example using Indic fonts as part of the package or on this wiki. I can provide a sample.

Thanks.