apunkt/dograh
1
0
Fork 0
mirror of https://github.com/dograh-hq/dograh.git synced 2026-06-25 08:48:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix(qa): tolerate non-dict JSON from QA LLM instead of crashing (#408)

Browse source 
* fix(qa): tolerate non-dict JSON from QA LLM instead of crashing

parse_llm_json is explicitly designed to return a list when the model emits a
top-level JSON array (it has a dedicated test for that). The QA analyzers then
call parsed.get("tags", ...) directly on the result. When parsed is a list,
that raises AttributeError, which is NOT caught by the surrounding
except (json.JSONDecodeError, ValueError) — so a single stray array response
from the QA model crashed the entire QA analysis run instead of degrading to
empty results.

The live variable-extraction path already guards this exact case with an
isinstance(..., dict) check; mirror it in both QA analysis call sites
(_run_qa_analysis per-node and _run_whole_call_qa_analysis fallback) so a
non-dict parse result coerces to {} and the run produces empty defaults.

Adds a regression test that drives the whole-call analyzer with an array
response and asserts empty results rather than a crash.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(qa): log non-object QA JSON responses

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
This commit is contained in:
Mubashir R 2026-06-19 13:40:53 +05:00 • committed by GitHub
parent ed06de73a3
commit 6e194f4b59
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 92 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

20
api/services/workflow/qa/analysis.py
View file

@ -206,6 +206,16 @@ async def run_per_node_qa_analysis(
}
try:
parsed = parse_llm_json(raw_response)
# parse_llm_json can return a list (e.g. when the model emits a
# top-level JSON array); coerce non-dict results so the .get()
# lookups below don't raise AttributeError.
if not isinstance(parsed, dict):
logger.warning(
f"QA LLM returned non-object JSON for node '{node_name}' "
f"on run {workflow_run_id}; got {type(parsed).__name__}, "
"using empty QA result"
)
parsed = {}
node_result["tags"] = parsed.get("tags", [])
node_result["summary"] = parsed.get("summary", "")
node_result["score"] = parsed.get("call_quality_score")
@ -296,6 +306,16 @@ async def _run_whole_call_qa_analysis(
}
try:
parsed = parse_llm_json(raw_response)
# parse_llm_json can return a list (e.g. when the model emits a
# top-level JSON array); coerce non-dict results so the .get()
# lookups below don't raise AttributeError.
if not isinstance(parsed, dict):
logger.warning(
f"QA LLM returned non-object JSON for whole-call QA on run "
f"{workflow_run_id}; got {type(parsed).__name__}, using empty "
"QA result"
)
parsed = {}
node_result["tags"] = parsed.get("tags", [])
node_result["summary"] = parsed.get("summary", "")
node_result["score"] = parsed.get("call_quality_score")