Project Background
This project was built for the ModelEngine community open source contribution competition. The community AgentsHub repository builds a shared ecosystem of agents on top of the Nexent platform. Instead of submitting another chatbot, I wanted to validate a complete engineering path: turning raw medical CSV files and unstructured text into structured data and a knowledge graph, then supporting question answering and analytics on top.
Medical sample data mixes two data forms: structured patient-record tables and unstructured symptom descriptions. That mix amplifies real engineering problems — how cleaning rules cover both inputs, how extraction results land in one graph schema, and which prerequisite artifacts downstream QA and analysis depend on. These problems resemble production far more than single-feature demos do.
Overall Architecture and Task Breakdown
The demo follows the “data to knowledge to insight” split with three tasks plus two cross-cutting layers:
Figure: Data Knowledge Insight Demo overview
- Task 1 (data processing): clean the CSV, normalize text, extract core fields;
- Task 2 (knowledge construction): entity recognition, relation extraction, triple generation and validation;
- Task 3 (data analysis): graph statistics, centrality analysis, SQLite construction, NL2SQL template evaluation, and charts;
- planners layer: a rule-based planner maps natural-language task descriptions to execution plans, with an LLM planner as an enhanced path;
- adapters layer: DataMate-style mappers and Nexent/FastMCP-style tools for migration to real platforms.
Figure: the complete flow from raw data to insight artifacts
The benefit of this split is that every task produces a concrete artifact: clean data, graph files, reports and charts. When something breaks, the failure is localized to a boundary instead of buried inside one large function.
Data Processing Agent
Task 1 consumes two input types. The CSV path handles missing-value cleaning, text-field normalization, and core-field extraction; the text path handles reading, chunking, and structured export. The rule-based planner maps descriptions like “process only the CSV” into different step sequences.
LLMTaskPlanner is the enhancement: a real model generates the plan from the task description. But model output cannot be trusted blindly — invalid structures or unknown steps fall back to the rule-based plan automatically. Each step can also declare retry, required, or skipped behavior through StepExecutionPolicy, and failures accumulate in execution_state.errors instead of aborting the pipeline.
The principle is simple: planning can be flexible, execution must be reliable.
Knowledge Graph Construction and QA
Task 2 extracts diseases, symptoms, and medicines together with their relations, generates triples with validation, and exports three formats: JSON for programs, GraphML for graph tooling, and HTML visualization through pyvis.
Figure: disease-symptom statistics from the graph
On top of the graph sits KGQAAgent, supporting four query types: disease to symptoms, disease to medicines, symptom back to disease, and medicine back to disease. Answers come directly from graph structure with evidence attached, which avoids unverifiable free-form generation — especially important in a medical context.
Data Analysis and NL2SQL
Task 3 turns the graph and cleaned data into insight: node and edge statistics, disease symptom counts, medicine association counts, centrality analysis, matplotlib charts, a SQLite database, and NL2SQL template evaluation over natural-language queries.
Figure: medicine-disease association statistics
I did not aim for free-form text to arbitrary SQL. Templates bound the answerable domain first; evaluation establishes a measurable baseline before any expansion.
Platform Integration: DataMate and Nexent
This part answers a key question: how does the demo migrate to real platforms?
On the DataMate side, I rewrote cleaning logic as sample -> sample mappers (CleanTextDataMateMapper, NormalizeMedicalRecordMapper, and others) and provided a datamate_task1_package directory as a ready-to-move package example. Migration cost drops from rewriting to structural comparison and moving files.
On the Nexent side, FastMCP wraps four business capabilities as remote MCP tools: process_medical_data, build_medical_knowledge_graph, ask_medical_kg, and analyze_medical_data. Tools check prerequisites internally — missing graph files produce a structured missing_prerequisite response instead of a crash.
Figure: configuring the MCP server in Nexent
I verified the integration locally with Docker: the Nexent container reached the host MCP server through host.docker.internal on port 8001, and after creating a medical agent with the four tools selected, natural-language requests triggered data processing, graph construction, QA, and analysis from the UI.
Figure: the medical knowledge assistant inside Nexent
Results and Reflections
The deliverable is a runnable, tested, explainable loop: run_demo.py executes all three tasks, FastAPI exposes five endpoints across health check and three tasks, and 56 pytest tests pass. The project won Third Prize in the ModelEngine Open Source Project Contribution Competition, and the code was merged into the community AgentsHub repository.
Three lessons stand out. First, make the end-to-end loop runnable before optimizing any single part — most interface and artifact-format issues only surfaced once the tasks were connected. Second, keep planning flexible but execution guarded: the fallback mechanism and per-step policies matter more than model quality. Third, design the adapter layer early: isolating adapters turned platform migration into structural comparison instead of invasive rewrite, which is also why the project could join the shared ecosystem.