0.5.5 causing Maximum number of tokens exceeded in DBT queries

[DmytryEmery]

Describe the bug
It seems that DBT core has a dependency on this library, and the latest version 0.5.5 causes some issues with DBT. Specifically larger queries cause the error:
Maximum number of tokens exceeded {number}

I suspect it's this commit:
da67ac1

I realize this is more of a DBT issue than this library, but wanted to give a heads up.

To Reproduce
Using dbt 1.8 or higher, it pulls in this library as dependency.
Not exactly sure, but a fairly large query (sorry I don't have an exact reproducible one I can share)
The limit seemed relatively low.

Expected behavior
Success, these are valid queries that previously had worked.
Aside: terminology of 'tokens' while likely correct gets confusing LLMs being so common these days.

Versions (please complete the following information):

• Python: 3.11.2
• sqlparse: 0.5.5
• dbt: < 1.8 (tested 1.8, 1.9, 1.10)

Additional context
Related DBT issue: dbt-labs/dbt-core#12303

[grainusrainus]

Hi, I also share pain with this token limit. It would be great if this was something we could override with a parameter.

[andialbrecht]

Thanks for bringing this up! It shows me, that the recent change in 0.5.5 (issue #827) was correct.

0.5.4 introduced this token and recursion tracking to avoid DoS attacks. Unfortunately instead of raising an exception it returned None when one of the two security guards were triggered. 0.5.5 changed this to raise an exception (what you see now).

The reason is a security-first approach: If you have a valid use-case where you know the statements that are processed by sqlparse it's perfectly fine to disable these constraints (the old behavior before 0.5.4). But if you "just use" sqlparse these constraints cover you from DoS attacks.

The change is documented here: https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations

TL;DR:

import sqlparse.engine.grouping

# Increase limits (use with caution)
sqlparse.engine.grouping.MAX_GROUPING_DEPTH = 200
sqlparse.engine.grouping.MAX_GROUPING_TOKENS = 50000

# Disable limits completely (use with extreme caution)
sqlparse.engine.grouping.MAX_GROUPING_DEPTH = None
sqlparse.engine.grouping.MAX_GROUPING_TOKENS = None