"Unknown slot number of character" warning despite it being present in the cfg file

[niruvt]

Description

The microtype package warns about characters already present in font's cfg file (NewComputerModern.cfg in this case).

Relevant for: LuaLaTeX (XeLaTeX works fine!)

Minimal example demonstrating the issue

\documentclass{article}
\usepackage[verbose]{microtype}
\usepackage{linguistix-fonts}

\begin{document}
Hello, bug!
\end{document}

linguistix-fonts is a (newly released) package that loads NewCM fonts for the document by default (I am the author).

Package microtype Warning: Unknown slot number of character
(microtype)                `/uni1FBC.alt'
(microtype)                in font encoding `TU' in protrusion list 
(microtype)                `NCM-default'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/uni1FCC.alt'
(microtype)                in font encoding `TU' in protrusion list 
(microtype)                `NCM-default'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/uni1FFC.alt'
(microtype)                in font encoding `TU' in protrusion list 
(microtype)                `NCM-default'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/a.end'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/e.end'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/k.alt'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/r.end'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/m.end'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.


Package microtype Warning: Unknown slot number of character
(microtype)                `/n.end'
(microtype)                in font encoding `TU' in protrusion list
(microtype)                `NCM-TU'.

[schlcht]

The problem seems to be that you are loading the fonts with the HarfBuzz renderer. Using "Node" or "Base" renderer works fine. Are you sure you really need Harfbuzz? With this renderer, luaotfload seems to get some character codes wrong (see for example here: latex3/luaotfload#268).

[niruvt]

@schlcht

Ah! That's a great catch. I am sure I need HarfBuzz as it is necessary for some combining diacritics. Thank you so much for pointing out the exact bug and the cause of it. As it's nothing related to microtype, I will close it here and report it to luaotfload.

[niruvt]

Oh, just after closing the issue, I saw your edit. It is one of my own old issues. I get the connection now. HarfBuzz people have said that they can't help in this. Do you think we can do anything in microtype in order to solve this? HarfBuzz is essential for seriously many scripts and getting so many warnings with each of the characters will be quite annoying. Sorry, reopening.

[khaledhosny]

What is the expectation here? base and node modes map glyphs to PUA code points (unless the glyphs is mapped from the font’s cmap table), but this conflicts with PUA mapping that might exist in the font, so harf mode takes an alternative approach and maps these glyphs to code points outside valid Unicode range to avoid such conflicts and make a clear distinction between code points and glyph indices (LuaTeX, unfortunately, does not make such a distinction internally).

So what is microtype expectation here? In other words, what is the unknown in “Unknown slot number of character”?

[schlcht]

@khaledhosny The "unknown..." is just because microtype checks whether the number returned by slot_of_name is within the Unicode bounds. What microtype would expect is a number that can be fed to \lpcode (and friends). Here's a plain luahbtex example:

\input luaotfload.sty
\font\x={file:NewCM10-Book.otf:mode=harf} 
\x
\def\y{%
  \directlua{%
    tex.sprint(luaotfload.aux.slot_of_name(
      font.current(),[[n.end]]))}}
\y
\lpcode\x\y = 1000
\bye

[khaledhosny]

\lpcode, \rpcode should be working on glyph indices not characters (a character does not have a protrusion, a specific glyph does), but LuaTeX makes no distinction between characters and glyphs, so they expect a valid code point. This make them of limited use, IMHO. If you want to set the protrusion of glyphs directly, a you can set it using Lua code on the font structure directly without going through character codes. There is even a two luaotfload features that control protrusion and expansion values (protrusion and expansion features, see luaotfload documentation).

[schlcht]

The luaotfload documentation says of aux.slot_of_name:

Translates a name for a glyph in font id to the corresponding glyph slot which can be used e.g. as an argument to \char.

(and I presume the "e.g." would include \lpcode etc.)

This fails with names of glyphs that are encoded outside the Unicode range -- but only when using the harf renderer, whereas everything works fine in node or base mode. Therefore I would consider this a bug in the slot_of_name function, or at least a limitation that should be documented.

The luaotfload font features you mention don't work in harf mode either (see @zauguin's comment here). Instead microtype would have to directly patch the font tables (which is not something that I'm up to right now), but even then, the question remains how to get a slot number given a glyph name.

[zauguin]

Therefore I would consider this a bug in the slot_of_name function, or at least a limitation that should be documented.

It's mainly a documentation bug and a restriction of \char and friends. It does always give the slots, but given that \char does have a limitation that it only accepts the Unicode range it's misleading to say that the values can be used there.

The luaotfload font features you mention don't work in harf mode either.

I just pushed a version to luaotfload's dev branch adding support for these also in harf mode.
If you can it would be great if you could test this already, in any case this will be supported in the next luaotfload release unless some fundamental issue comes up.

but even then, the question remains how to get a slot number given a glyph name.

Well, you use slot_of_name. The function does give you the right slot even if it doesn't work with some TeX commands.

But as an additional feature in harf mode that isn't even needed. The tables you need to pass for the protrusion and expansion features also accept glyph names instead of slots as table indices.

[niruvt]

@schlcht just to check, is there any decision in this regard?