Display a helpful error message if not all pages from the wiki are entered into the dictionary.

[UrsKR]

Observed behaviour:
Using my default example of BattleTechWiki at http://sarna.net:
There is a page called "Storm Crow" at https://www.sarna.net/wiki/Storm_Crow
and another page called "Stormcrow" at https://www.sarna.net/wiki/index.php?title=Stormcrow&redirect=no
with the latter page forwarding to "Stormcrow (Ryoken)" to https://www.sarna.net/wiki/Stormcrow_(Ryoken).
In both Runik's integrated dictionary browser and in Koreader (v25.8) the dictionary only contains an entry for "Storm Crow".

Expected behaviour:
The dictionary contains all the entries or a helpful error message is displayed.

Thoughts:
Is this something about the similarity in the page names?

[UrsKR]

Here is another test case.
In dictionary: https://www.sarna.net/wiki/Timber_Wolf_(disambiguation)
In dictionary: https://www.sarna.net/wiki/Timber_Wolf_(Individual_Invader-class_JumpShip)
In dictionary: https://www.sarna.net/wiki/Timber_Wolf_(Individual_Monolith-class_JumpShip)
Not in dictionary: https://www.sarna.net/wiki/Timber_Wolf_(Mad_Cat)

In both this case and the original case, the missing page is a BattleMech profile page.
Maybe Runik doesn't play well with sarna.net's format for these pages?

[UrsKR]

Test case 3.
In dictionary: https://www.sarna.net/wiki/Joanna
Not in dictionary: https://www.sarna.net/wiki/Joanna_(Jade_Falcon_Guards)
Not in dictionary: https://www.sarna.net/wiki/Joanna_(Peregrine_Galaxy)

These pages describe people, so it is not about that particular type of page.

[UrsKR]

One more note:
Running the builder at CLI
go run . -o ./ -l 80000 generate https://www.sarna.net/wiki/Main_Page
I get

54672/64192 definitions processed
📖 Found 54672 entries 
Successfully built dictionary at BattleTechWiki.json

but no indication what happened to the remaining ~10k pages.

[matteomiceli]

Hey thank you for the examples! I'm still refining the wikitext parser which is custom.

A lot of the entries that don't get parsed are redirects. It is on my list to handle redirects, but one of the challenges I've faced is the redirect syntax is different across different languages which makes it hard to parse (this could also just be a fandom thing).

Other entries that get left out usually just contain tables (currently ignored) and no other text. You are right that some of the examples you provided should be getting parsed. I'll look into why these aren't being included.

[matteomiceli]

I think I narrowed down this mysterious bug to the use of self-closing <ref /> tags in the article. https://github.com/Runik-3/builder now handles self-closing tags in its cleanHtml function. Will be bundled in the next release of Core.

Fixed in Runik-3/builder@746371d

☝️ This solves the missing entries issue, but as for including more messaging, I think I'll kick that one down the road a bit. I'll have to think about the UX around that. This could look like some info in a tooltip, an FAQ on the website, or something more in the user's face.

Happy for any suggestions/PRs in this regard.

[UrsKR]

I will have a look once you update Core, and try to come up with some ideas.
Off of my cuff, FAQ seems to be too well hidden to explain something that might run counter to expectations.

[matteomiceli]

The fix for the missing dictionary entries is in v0.2.5.

I'll remove the bug tag and leave this issue open since it's more about the UX and user feedback.

[UrsKR]

It's your issue tracker, do as you like. 😄 (The tag is still present in my current view.)

In the meantime, thanks!

As for more messaging, in your current UX paradigm, I think a toast on the bottom right hand side would be adequate. For the moment, just seeing the number of entries created next to the number of entries originally discovered could be enough to prompt an investigation.