Add timeout to temp table lineage graph node processing

[mohittilala]

Describe your changes:

This PR adds timeout of 30 seconds to each node processing for temporary lineage creation and related tests.

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

---

Summary by Gitar

• Timeout mechanism:
  • Added 30-second timeout to _get_paths_from_subtree using @timeout decorator to prevent indefinite hangs on complex lineage graphs
  • Enhanced timeout logging to include duration in warning messages for better debugging
• Path processing improvement:
  • Modified to collect paths per root node before extending main list, ensuring complete path sets even when timeout occurs
• Performance tracking:
  • Added calculation time tracking for lineage by graph processing to monitor performance
  • Added time tracking for diff parser query masking operations
• Test improvements:
  • Enabled enableTempTableLineage: true in E2E test configuration to properly validate temp table lineage functionality
  • Updated E2E test SQL fixtures and assertions for temp table lineage scenarios
• New test file:
  • test_temp_table_lineage.py (473 lines) with comprehensive graph processing tests covering various topologies, timeout behavior, and edge cases
• E2E validation:
  • Added temp table lineage tests in test_query_lineage.py and SQL fixtures validating lineage traces through deleted temporary tables

_{This will update automatically on new commits.}

---

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.13)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
com.fasterxml.jackson.core:jackson-core	CVE-2025-52999	🚨 HIGH	2.12.7	2.15.0
com.fasterxml.jackson.core:jackson-core	CVE-2025-52999	🚨 HIGH	2.13.4	2.15.0
com.fasterxml.jackson.core:jackson-databind	CVE-2022-42003	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind	CVE-2022-42004	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4
com.google.code.gson:gson	CVE-2022-25647	🚨 HIGH	2.2.4	2.8.9
com.google.protobuf:protobuf-java	CVE-2021-22569	🚨 HIGH	3.3.0	3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java	CVE-2022-3509	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2022-3510	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2024-7254	🚨 HIGH	3.3.0	3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java	CVE-2021-22569	🚨 HIGH	3.7.1	3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java	CVE-2022-3509	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2022-3510	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2024-7254	🚨 HIGH	3.7.1	3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt	CVE-2023-52428	🚨 HIGH	9.8.1	9.37.2
com.squareup.okhttp3:okhttp	CVE-2021-0341	🚨 HIGH	3.12.12	4.9.2
commons-beanutils:commons-beanutils	CVE-2025-48734	🚨 HIGH	1.9.4	1.11.0
commons-io:commons-io	CVE-2024-47554	🚨 HIGH	2.8.0	2.14.0
dnsjava:dnsjava	CVE-2024-25638	🚨 HIGH	2.1.7	3.6.0
io.netty:netty-codec-http2	CVE-2025-55163	🚨 HIGH	4.1.96.Final	4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2	GHSA-xpw8-rcwv-8f8p	🚨 HIGH	4.1.96.Final	4.1.100.Final
io.netty:netty-handler	CVE-2025-24970	🚨 HIGH	4.1.96.Final	4.1.118.Final
net.minidev:json-smart	CVE-2021-31684	🚨 HIGH	1.3.2	1.3.3, 2.4.4
net.minidev:json-smart	CVE-2023-1370	🚨 HIGH	1.3.2	2.4.9
org.apache.avro:avro	CVE-2024-47561	🔥 CRITICAL	1.7.7	1.11.4
org.apache.avro:avro	CVE-2023-39410	🚨 HIGH	1.7.7	1.11.3
org.apache.derby:derby	CVE-2022-46337	🔥 CRITICAL	10.14.2.0	10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy	CVE-2022-46751	🚨 HIGH	2.5.1	2.5.2
org.apache.mesos:mesos	CVE-2018-1330	🚨 HIGH	1.4.3	1.6.0
org.apache.thrift:libthrift	CVE-2019-0205	🚨 HIGH	0.12.0	0.13.0
org.apache.thrift:libthrift	CVE-2020-13949	🚨 HIGH	0.12.0	0.14.0
org.apache.zookeeper:zookeeper	CVE-2023-44981	🔥 CRITICAL	3.6.3	3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server	CVE-2024-13009	🚨 HIGH	9.4.56.v20240826	9.4.57.v20241219
org.lz4:lz4-java	CVE-2025-12183	🚨 HIGH	1.8.0	1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (7)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
apache-airflow	CVE-2025-68438	🚨 HIGH	3.1.5	3.1.6
apache-airflow	CVE-2025-68675	🚨 HIGH	3.1.5	3.1.6
jaraco.context	CVE-2026-23949	🚨 HIGH	5.3.0	6.1.0
starlette	CVE-2025-62727	🚨 HIGH	0.48.0	0.49.1
urllib3	CVE-2025-66418	🚨 HIGH	1.26.20	2.6.0
urllib3	CVE-2025-66471	🚨 HIGH	1.26.20	2.6.0
urllib3	CVE-2026-21441	🚨 HIGH	1.26.20	2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

[github-actions]

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

Vulnerabilities (4)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
libpam-modules	CVE-2025-6020	🚨 HIGH	1.5.2-6+deb12u1	1.5.2-6+deb12u2
libpam-modules-bin	CVE-2025-6020	🚨 HIGH	1.5.2-6+deb12u1	1.5.2-6+deb12u2
libpam-runtime	CVE-2025-6020	🚨 HIGH	1.5.2-6+deb12u1	1.5.2-6+deb12u2
libpam0g	CVE-2025-6020	🚨 HIGH	1.5.2-6+deb12u1	1.5.2-6+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
com.fasterxml.jackson.core:jackson-core	CVE-2025-52999	🚨 HIGH	2.12.7	2.15.0
com.fasterxml.jackson.core:jackson-core	CVE-2025-52999	🚨 HIGH	2.13.4	2.15.0
com.fasterxml.jackson.core:jackson-databind	CVE-2022-42003	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind	CVE-2022-42004	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4
com.google.code.gson:gson	CVE-2022-25647	🚨 HIGH	2.2.4	2.8.9
com.google.protobuf:protobuf-java	CVE-2021-22569	🚨 HIGH	3.3.0	3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java	CVE-2022-3509	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2022-3510	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2024-7254	🚨 HIGH	3.3.0	3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java	CVE-2021-22569	🚨 HIGH	3.7.1	3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java	CVE-2022-3509	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2022-3510	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java	CVE-2024-7254	🚨 HIGH	3.7.1	3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt	CVE-2023-52428	🚨 HIGH	9.8.1	9.37.2
com.squareup.okhttp3:okhttp	CVE-2021-0341	🚨 HIGH	3.12.12	4.9.2
commons-beanutils:commons-beanutils	CVE-2025-48734	🚨 HIGH	1.9.4	1.11.0
commons-io:commons-io	CVE-2024-47554	🚨 HIGH	2.8.0	2.14.0
dnsjava:dnsjava	CVE-2024-25638	🚨 HIGH	2.1.7	3.6.0
io.netty:netty-codec-http2	CVE-2025-55163	🚨 HIGH	4.1.96.Final	4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2	GHSA-xpw8-rcwv-8f8p	🚨 HIGH	4.1.96.Final	4.1.100.Final
io.netty:netty-handler	CVE-2025-24970	🚨 HIGH	4.1.96.Final	4.1.118.Final
net.minidev:json-smart	CVE-2021-31684	🚨 HIGH	1.3.2	1.3.3, 2.4.4
net.minidev:json-smart	CVE-2023-1370	🚨 HIGH	1.3.2	2.4.9
org.apache.avro:avro	CVE-2024-47561	🔥 CRITICAL	1.7.7	1.11.4
org.apache.avro:avro	CVE-2023-39410	🚨 HIGH	1.7.7	1.11.3
org.apache.derby:derby	CVE-2022-46337	🔥 CRITICAL	10.14.2.0	10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy	CVE-2022-46751	🚨 HIGH	2.5.1	2.5.2
org.apache.mesos:mesos	CVE-2018-1330	🚨 HIGH	1.4.3	1.6.0
org.apache.thrift:libthrift	CVE-2019-0205	🚨 HIGH	0.12.0	0.13.0
org.apache.thrift:libthrift	CVE-2020-13949	🚨 HIGH	0.12.0	0.14.0
org.apache.zookeeper:zookeeper	CVE-2023-44981	🔥 CRITICAL	3.6.3	3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server	CVE-2024-13009	🚨 HIGH	9.4.56.v20240826	9.4.57.v20241219
org.lz4:lz4-java	CVE-2025-12183	🚨 HIGH	1.8.0	1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (15)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
Werkzeug	CVE-2024-34069	🚨 HIGH	2.2.3	3.0.3
aiohttp	CVE-2025-69223	🚨 HIGH	3.12.12	3.13.3
aiohttp	CVE-2025-69223	🚨 HIGH	3.13.2	3.13.3
apache-airflow	CVE-2025-68438	🚨 HIGH	3.1.5	3.1.6
apache-airflow	CVE-2025-68675	🚨 HIGH	3.1.5	3.1.6
azure-core	CVE-2026-21226	🚨 HIGH	1.37.0	1.38.0
deepdiff	CVE-2025-58367	🔥 CRITICAL	7.0.1	8.6.1
jaraco.context	CVE-2026-23949	🚨 HIGH	5.3.0	6.1.0
jaraco.context	CVE-2026-23949	🚨 HIGH	5.3.0	6.1.0
pyasn1	CVE-2026-23490	🚨 HIGH	0.6.1	0.6.2
ray	CVE-2025-62593	🔥 CRITICAL	2.47.1	2.52.0
starlette	CVE-2025-62727	🚨 HIGH	0.48.0	0.49.1
urllib3	CVE-2025-66418	🚨 HIGH	1.26.20	2.6.0
urllib3	CVE-2025-66471	🚨 HIGH	1.26.20	2.6.0
urllib3	CVE-2026-21441	🚨 HIGH	1.26.20	2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[gitar-bot]

🔍 CI failure analysis for 53da946: CI failures in Python test jobs are unrelated to the PR's lineage processing changes. The Python 3.10 job has a segmentation fault during cleanup, and the Python 3.11 job has Trino classifier test errors.

Issue

Python 3.10 Job (61103431298) - Latest Run

Segmentation fault during test process cleanup:

• All 520 tests passed, 21 skipped, 1 xfailed
• Tests completed successfully in 1:16:39
• Process crashed with Segmentation fault (core dumped) at cleanup phase
• Error occurred at ingestion/Makefile:91: run_ometa_integration_tests

Python 3.11 Job (61103423588) - Latest Run

7 test setup errors in Trino classifier tests:

• All 7 errors in ingestion/tests/integration/trino/test_classifier.py::test_auto_classification_workflow
• Tests affected: table, titanic, iris, userdata, empty, complex_and_simple, only_complex
• 513 tests passed, 21 skipped, 1 xfailed, 7 errors
• Test run completed in 1:02:52

Root Cause

Python 3.10 Segmentation Fault

Infrastructure/environment issue - not code-related:

• Tests completed successfully (520 passed)
• Segfault occurred during test process cleanup/teardown phase
• This is a runtime crash in the Python interpreter or native extension, not application code
• Segmentation faults during cleanup are typically caused by:
  • Memory corruption in C extensions
  • Issues with database connection cleanup
  • Python interpreter bugs with specific versions
  • Docker container resource constraints

Python 3.11 Classifier Test Errors

AttributeError in test fixture setup:

AttributeError: 'NoneType' object has no attribute 'name'

Occurs in metadata/sampler/processor.py:169 during _copy_service_config:

config_copy.catalog = database.name.root  # type: ignore
                      ^^^^^^^^^^^^^  # database is None

The database object is None when the sampler processor tries to copy service configuration. This is a test fixture setup issue where the database entity is not being properly retrieved or initialized before the classifier workflow runs.

Details

Unrelated to PR Changes:

This PR (#25409) modifies SQL lineage graph processing with performance tracking and timeout handling:

• ingestion/src/metadata/ingestion/lineage/sql_lineage.py - Adds timeout decorators and execution time tracking
• ingestion/src/metadata/ingestion/lineage/masker.py - Adds performance decorators
• ingestion/tests/unit/lineage/test_temp_table_lineage.py - New unit tests for graph processing
• ingestion/tests/unit/lineage/e2e/ - E2E lineage tests (Oracle-specific)

Failing tests are in completely different areas:

1. Python 3.10 Segfault: Runtime crash during test cleanup - no specific failing test, just process crash
2. Python 3.11 Errors: test_classifier.py - Trino auto-classification workflow with sampler processor

There is no code path connecting:

• SQL lineage graph processing (PR changes) ↔️ Sampler processor database configuration
• SQL lineage timeout decorators ↔️ Trino classifier test fixture setup
• Lineage masking performance tracking ↔️ Process segmentation faults

Why These Are Pre-existing/Flaky Issues:

1. Segmentation fault pattern: This is a classic infrastructure flakiness issue:

   • Tests passed but the process crashed during cleanup
   • Indicates resource cleanup issues in test infrastructure
   • Potential memory issues or race conditions in CI environment
   • Not reproducible consistently (Python 3.10 vs 3.11 show different failures)

2. Classifier test errors: The database.name being None indicates:

   • Test fixture dependency issue where database entities aren't being properly created
   • Potential race condition in test setup
   • Environment-specific issue (only affects Python 3.11, not 3.10 in earlier runs)
   • This exact error repeated across multiple commits/runs

Code Review 👍 Approved with suggestions 0 resolved / 1 findings

Solid implementation of timeout mechanism for temp table lineage processing with comprehensive tests. One minor test issue remains where the assertion contradicts the patched value.

More details 💡 1 suggestion

💡 Bug: Test assertion contradicts patched timeout value

📄 ingestion/tests/unit/lineage/test_temp_table_lineage.py:328-333 📄 ingestion/tests/unit/lineage/test_temp_table_lineage.py:483-488

The test test_timeout_logs_warning patches NODE_PROCESSING_TIMEOUT to 1 but then asserts it equals 30. The patch decorator applies to the module-level constant in sql_lineage.py, but the assertion checks NODE_PROCESSING_TIMEOUT which was imported at the top of the test file - that import captured the original value (30) before the patch was applied.

This test doesn't actually validate timeout behavior. The assertion will always pass regardless of whether the patch works correctly.

Suggested fix:
Either:

1. Remove the assertion and add actual timeout behavior testing (e.g., mock a slow operation and verify the warning is logged)
2. Or change the assertion to verify the patched value at the module level:

from metadata.ingestion.lineage import sql_lineage
assert sql_lineage.NODE_PROCESSING_TIMEOUT == 1  # Verify patch applied

What Works Well

Clean implementation of timeout decorator pattern using nested function. Good use of execution time tracking for performance monitoring. Comprehensive test coverage for various graph topologies and edge cases.

Recommendations

Fix the test_timeout_logs_warning test by either removing the @patch decorator (if testing the default value) or adjusting the assertion to match the patched value (if testing the patching mechanism). The current assertion NODE_PROCESSING_TIMEOUT == 30 will always pass because the patch only affects the module-level constant, not the imported name in the test.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}