0.5.5 causing Maximum number of tokens exceeded in DBT queries

[DmytryEmery]

Describe the bug
It seems that DBT core has a dependency on this library, and the latest version 0.5.5 causes some issues with DBT. Specifically larger queries cause the error:
Maximum number of tokens exceeded {number}

I suspect it's this commit:
da67ac1

I realize this is more of a DBT issue than this library, but wanted to give a heads up.

To Reproduce
Using dbt 1.8 or higher, it pulls in this library as dependency.
Not exactly sure, but a fairly large query (sorry I don't have an exact reproducible one I can share)
The limit seemed relatively low.

Expected behavior
Success, these are valid queries that previously had worked.
Aside: terminology of 'tokens' while likely correct gets confusing LLMs being so common these days.

Versions (please complete the following information):

• Python: 3.11.2
• sqlparse: 0.5.5
• dbt: < 1.8 (tested 1.8, 1.9, 1.10)

Additional context
Related DBT issue: dbt-labs/dbt-core#12303

[grainusrainus]

Hi, I also share pain with this token limit. It would be great if this was something we could override with a parameter.