Skip to content

D
db-tools
Commit 7b4bd4a8 authored by Data Governance Dev's avatar Data Governance Dev
Browse files
Options

feat(web): Phase 3 - Web 端治理平台（FastAPI + Vue 3） 

把工作流包装成浏览器化的产品：填表 → 测试连接 → 一键治理 → 实时进度。

核心架构
- web/app.py         FastAPI 入口 + CORS + 静态资源挂载
- web/start.py       启动前依赖检查（可选 anthropic / dmPython）
- web/start.bat      Windows 一键启动脚本
- web/api/routes.py  12 个 REST 接口（jobs/history/standards/reports...）

调度与运行
- web/core/orchestrator.py 8 步流程调度 + 步骤级日志 + 异常 traceback
- web/core/job_manager.py  后台任务管理 + LogBuffer + SSE 订阅
- web/core/db_adapter.py   MySQL / 达梦 统一连接与查询
- web/core/step_impl/      8 个 Step 的 Web 版实现

LLM 增强（可降级）
- web/core/llm.py          Anthropic / OpenAI / MiniMax 多 provider 适配
- configs/llm.example.yaml provider / api_key / base_url / model 模板

SQL 模板体系（Python 不再写 SQL）
- web/sql/info_schema/     list_columns / list_tables  按 dialect 区分 MySQL / 达梦
- web/sql/verify/          count_table_rows
- web/sql/empty_fields/    count_with_nulls
- web/sql/xzqh/            check_dict_table_exists / find_xzqh_orphans
- web/sql/standards/       sample_field_values
- web/sql/health/          check_connection
- web/sql/loader.py        占位符 + dialect 自动加载

前端（无构建步骤）
- web/static/index.html    Vue 3 单页应用
- web/static/lib/          Vue + Element Plus 静态资源
- web/static/style.css

实时日志 / 历史归档 / 报告下载
- SSE 流（GET /api/jobs/{id}/logs/stream）
- outputs/<db>/<ts>/findings/_all_findings.json
- outputs/<db>/<ts>/reports/{md,docx}
- GET /api/history /api/reports/{db}/{ts}/{file}

调试能力（本次一并落地）
- 三路日志：SSE 实时流 + web/logs/app.log + outputs/<db>/<ts>/run.log
- 每步开始 / 完成带耗时、进度 N/M；异常带 traceback 与异常类型
- 数据库 [DB] 连接 / SQL 执行时间；路由 [POST /api/xxx] 入参摘要
- 启动脚本自带 web/logs/start.log

Why：CLI 工作流需要登录机器；Web 端把治理入口放到任何能开浏览器的设备上，
     配合 SSE 实时反馈，业务方也能直接用。
How：复用 workflow/ 中的 8 步逻辑与 standards/ 国标插件；新增 Pydantic 契约
     + job_manager + SSE，前后端用 REST + 事件流通信。
parent 3747e885
Show whitespace changes
Inline Side-by-side
Showing with 6357 additions and 0 deletions
web/README.md 0 → 100644
View file @ 7b4bd4a8
# 数据治理 Web 工具
> 基于 FastAPI + Vue 3 + Element Plus 的本地浏览器化治理平台
---
## 🚀 快速开始
### 1. 安装依赖（首次运行）
```bash
pip install -r web/requirements.txt
```
依赖包括：
- `fastapi` / `uvicorn` — Web 服务
- `pydantic` / `sse-starlette` — 数据模型 + SSE 流
- `pymysql` / `dmPython` — MySQL / 达梦驱动
- `anthropic` — Claude API（LLM 增强）
- `python-docx` — Word 报告生成
### 2. 配置 LLM Key（可选）
LLM 用于字段注释推测 / 合并建议 / 长度推断 / 空字段分类等智能判断。
未配置时**自动降级**为规则推理，主流程不受影响。
#### 方式 A：写到配置文件（推荐）
编辑 [web/configs/llm.yaml](web/configs/llm.yaml)：
```yaml
provider: minimax # 改为你用的 provider
api_key: sk-请填入你的_MiniMax_API_Key # ← 把这里替换为真实 Key
base_url: https://api.minimaxi.com/anthropic
model: MiniMax-Text-01
```
更多 provider 示例见 [web/configs/llm.example.yaml](web/configs/llm.example.yaml)。
#### 方式 B：环境变量（适合容器/CI）
```bash
# Windows CMD
set LLM_API_KEY=sk-xxxxx
set LLM_PROVIDER=minimax
# Windows PowerShell
$env:LLM_API_KEY="sk-xxxxx"
$env:LLM_PROVIDER="minimax"
# Linux / Git Bash
export LLM_API_KEY=sk-xxxxx
export LLM_PROVIDER=minimax
```
#### 优先级
```
环境变量 > web/configs/llm.yaml > 内置默认值
```
#### 支持的 Provider
| provider | 接入方式 | base_url | 模型示例 |
|----------|---------|----------|---------|
| `minimax` | Anthropic SDK + 自定义 base_url | `https://api.minimaxi.com/anthropic` | `MiniMax-Text-01` / `MiniMax-01` |
| `anthropic` | Anthropic SDK 官方 | (默认) | `claude-sonnet-5` / `claude-haiku-4-5-20251001` |
| `openai` | OpenAI SDK | (默认) | `gpt-4o` / `gpt-4-turbo` |
启动时会打印当前生效的 LLM 配置：
```
[LLM] provider=minimax model=MiniMax-Text-01
[LLM] base_url=https://api.minimaxi.com/anthropic
[LLM] api_key=sk-xxxx...xx（已配置）
```
前端右上角可通过 `GET /api/llm/status` 查看当前配置。
### 3. 启动服务
**方式 A：Python 启动**
```bash
python web/start.py
# 或
python -m web.start
```
**方式 B：Windows 一键脚本**
```cmd
web\start.bat
```
**方式 C：直接 uvicorn**
```bash
python -m uvicorn web.app:app --host 0.0.0.0 --port 8765
```
### 4. 浏览器访问
打开 **http://localhost:8765**
- Web 界面：填表 → 测试连接 → 开始治理
- API 文档：http://localhost:8765/docs
- OpenAPI JSON：http://localhost:8765/openapi.json
---
## 🎯 功能特性
| 特性 | 说明 |
|------|------|
| 🖥️ **Web 界面** | 浏览器访问，Vue 3 + Element Plus 响应式布局 |
| 🔌 **双数据库** | MySQL（PyMySQL）+ 达梦（dmPython）统一接口 |
| 🤖 **LLM 增强** | Claude API 用于字段注释推测 / 合并建议 / 长度推断 / 空字段分类 |
| 📊 **多级表格** | 7 大章节各一张可展开表格，KPI 卡片概览 |
| 📜 **国标展示** | 4 项内置标准可视化展示在页面（无需查询） |
| 📡 **实时日志** | SSE 流式推送治理进度，含 INFO/WARN/ERROR 三色 |
| 📥 **报告下载** | Markdown / Word 双格式，含历史报告检索 |
| 🗂️ **历史归档** | 每次运行按时间戳归档到 `outputs/<db>/<ts>/` |
| 🛡️ **优雅降级** | LLM 失败 / DB 失败不卡死，规则推理兜底 |
---
## 🗂️ 目录结构
```
数据治理/
├── web/ ← Web 工具
│ ├── app.py ← FastAPI 入口
│ ├── start.py ← 启动脚本（带依赖检查）
│ ├── start.bat ← Windows 一键启动
│ ├── requirements.txt
│ ├── README.md ← 本文件
│ ├── sql/ ← 所有 SQL 模板（Python 不再写 SQL）
│ │ ├── loader.py ← 模板加载器
│ │ ├── info_schema/ ← INFORMATION_SCHEMA 查询
│ │ ├── verify/ xzqh/ standards/ empty_fields/ health/
│ ├── core/ ← 核心模块
│ │ ├── db_adapter.py ← MySQL + 达梦 统一接口
│ │ ├── llm.py ← Claude API 封装
│ │ ├── job_manager.py ← 后台任务 + SSE 推送
│ │ ├── orchestrator.py ← 8 步流程调度
│ │ ├── models.py ← Pydantic 契约
│ │ └── step_impl/ ← 8 个 Step 的 Web 版实现
│ ├── api/
│ │ └── routes.py ← 12 个 REST 接口
│ ├── static/
│ │ ├── index.html ← Vue 3 单页前端
│ │ └── style.css
│ ├── configs/
│ │ └── defaults.yaml
│ ├── outputs/ ← 历史治理产出（按 db/ts 分目录）
│ └── logs/
│ └── app.log
│
├── workflow/ ← 原 CLI 工作流（仍可用，独立运行）
│
└── standards/ ← 国标插件（Web 与 CLI 共用）
├── base.py
├── registry.py
├── std_001_id_card.py ← GB 11643-1999
├── std_002_uscc.py ← GB 32100-2015
├── std_003_mobile.py ← YD/T 1313
└── std_004_xzqh.py ← GB/T 2260
```
---
## 🔧 API 端点速查
| 方法 | 路径 | 说明 |
|------|------|------|
| GET | `/api/health` | 健康检查 |
| GET | `/api/steps` | 列出 8 步元信息 |
| GET | `/api/standards` | 列出已注册国标（页面用） |
| POST | `/api/connect/test` | 测试连接（返回表数量） |
| POST | `/api/jobs` | 提交治理任务 |
| GET | `/api/jobs/{id}` | 查询任务状态（含进度百分比） |
| DELETE | `/api/jobs/{id}` | 取消任务 |
| GET | `/api/jobs/{id}/result` | 获取完整结果 |
| GET | `/api/jobs/{id}/logs/stream` | SSE 日志流 |
| GET | `/api/history` | 历史治理记录 |
| GET | `/api/reports/{db}/{ts}/{file}` | 下载报告（md/docx） |
| GET | `/api/history/{db}/{ts}/result` | 历史结果数据 |
---
## 🤖 LLM 增强位置
| Step | LLM 用途 | 失败降级 |
|------|----------|----------|
| Step 2 | 表合并建议文案 + 风险评估 | 模板字符串 |
| Step 4 | 高空字段成因分类 | 跳过 LLM 分类 |
| Step 5 | 无规则命中的字段注释推测 | 仅靠规则映射表 |
| Step 6 | 自定义字段长度推断 | 标记"待业务确认" |
LLM 调用封装在 [web/core/llm.py](web/core/llm.py)，所有方法都设计为**可降级**：
未配置 `ANTHROPIC_API_KEY` 时自动跳过，不阻断主流程。
---
## 🛡️ 双数据库适配说明
通过 [web/core/db_adapter.py](web/core/db_adapter.py) 统一抽象：
| 项 | MySQL | 达梦 |
|----|-------|------|
| 驱动 | `pymysql` | `dmPython` |
| 占位符 | `%s` | `:name` (NamedParam) |
| 标识符引用 | `` ` `` (反引号) | `"` (双引号 ANSI) |
| 字段大小写 | 视配置 | 默认大写 → 统一规范化为小写 |
| 长文本类型 | `text/longtext/...` | `text/clob/longvarchar` |
| `INFORMATION_SCHEMA` | 完全支持 | 完全支持（字段名差异已处理） |
> ⚠️ 达梦驱动 `dmPython` 需要本地先安装达梦客户端，否则 `pip install dmPython` 会失败。
---
## ⚠️ 已知事项
1. **MySQL 兼容**：所有 `INFORMATION_SCHEMA` 查询已用 ANSI 写法 + 适配层规范化，达梦可直接复用
2. **报告 bug 修复**：新版 Web 工具修正了原 `reporter.py` 的多处字段名不匹配（如 `over_provision_ratio`、`risk/suggestion` 等）
3. **单进程**：JobManager 使用进程内 asyncio 队列，并发任务排队执行；如需多任务并行可改为 Celery
4. **数据安全**：数据库密码仅在内存中传递，不写日志；提交前可勾选「启用 LLM」控制是否走外网
---
## 🗃️ SQL 模板管理
所有 SQL 语句都集中在 `web/sql/` 目录下，**Python 代码不再直接拼 SQL**。
### 模板语法（占位符）
| 语法 | 含义 | 示例 |
|------|------|------|
| `${var}` | 简单字符串替换 | `${schema}` |
| `${var \| quote}` | 按当前 dialect 自动加引号 | `${table \| quote}` → MySQL 反引号 / 达梦双引号 |
| `${list \| join:","}` | 列表按分隔符拼接 | `${cols \| join:","}` |
### Dialect 加载规则
加载器优先找 `<name>.<dialect>.sql`，找不到则用 `<name>.sql` 兜底。
### 使用示例
```python
from web.sql.loader import get_sql_loader
loader = get_sql_loader()
sql = loader.render("info_schema/list_columns", dialect="mysql", schema="mydb")
# SELECT ... FROM INFORMATION_SCHEMA.COLUMNS c WHERE c.TABLE_SCHEMA = `mydb`
sql = loader.render("info_schema/list_columns", dialect="dameng", schema="mydb")
# SELECT ... FROM ALL_TAB_COLUMNS c WHERE c.OWNER = "mydb"
```
详细文档：[sql/README.md](sql/README.md)
---
## 📝 后续可扩展
- [ ] 添加更多国标插件（邮箱、银行卡、邮编等）
- [ ] 接入 jaydebeapi 作为达梦 JDBC 替代驱动
- [ ] 报告编辑器（在线编辑 Markdown / 一键导出）
- [ ] 多任务并行（Celery + Redis）
- [ ] 用户系统（FastAPI Users + JWT）
- [ ] Docker 镜像打包
---
## 📚 相关文档
- [docs/DESIGN.md](docs/DESIGN.md) — **完整设计书**（架构、模块、LLM 判断矩阵）
- [../CLAUDE.md](../CLAUDE.md) — 顶层项目目标
\ No newline at end of file
web/__init__.py 0 → 100644
View file @ 7b4bd4a8
"""web 包入口"""
__version__ = "1.0.0"
\ No newline at end of file
web/api/__init__.py 0 → 100644
View file @ 7b4bd4a8
"""web.api 路由模块"""
from .routes import router
__all__ = ["router"]
\ No newline at end of file
web/api/routes.py 0 → 100644
View file @ 7b4bd4a8
"""FastAPI 路由聚合
所有路由集中在此，方便统一 prefix、跨域等配置。
"""
from __future__ import annotations
import logging
from pathlib import Path
from typing import Any, Optional
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel
from ..core.db_adapter import DBConfig, test_connection
from ..core.job_manager import JobManager
from ..core.models import (
ConnectRequest,
JobCreatedResponse,
JobLogsResponse,
JobResultResponse,
JobStatusResponse,
LogEntry,
OverviewStats,
RunHistoryItem,
RunHistoryResponse,
StandardsListResponse,
StandardInfo,
StepsListResponse,
StepInfo,
TestConnectionRequest,
TestConnectionResponse,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api")
# ── 健康检查 ──
@router.get("/health")
async def health():
logger.debug("GET /api/health")
return {"status": "ok", "service": "data-governance-web"}
# ── LLM 状态（前端用来显示当前 provider） ──
@router.get("/llm/status")
async def llm_status():
from ..core.llm import LLMConfig
cfg = LLMConfig.load()
masked = ""
if cfg.api_key and len(cfg.api_key) > 12:
masked = cfg.api_key[:6] + "..." + cfg.api_key[-4:]
elif cfg.api_key:
masked = "***"
return {
"provider": cfg.provider,
"model": cfg.effective_model,
"base_url": cfg.effective_base_url or None,
"api_key_configured": cfg.available,
"api_key_masked": masked,
}
# ── 列出所有步骤 ──
@router.get("/steps", response_model=StepsListResponse)
async def list_steps():
return StepsListResponse(steps=[
StepInfo(num=1, title="获取数据字典",
description="从 information_schema 抓取所有表/字段元数据", requires_db=True),
StepInfo(num=2, title="表合并与冗余字段分析",
description="离线分析：找结构高度相似的表 + 高频字段（LLM 优化建议）", requires_db=False, llm_enhanced=True),
StepInfo(num=3, title="数据验证",
description="连库验证：合并候选实际行数 + 行政区划孤儿 + 数据质量问题", requires_db=True),
StepInfo(num=4, title="大范围空字段扫描",
description="单表全列扫，统计 NULL/空值比例（LLM 分类成因）", requires_db=True, llm_enhanced=True),
StepInfo(num=5, title="缺失注释字段 + 推测",
description="离线：找无注释字段，规则+LLM 推测语义", requires_db=False, llm_enhanced=True),
StepInfo(num=6, title="字段长度检查",
description="离线：识别字段定义长度超出国家标准（LLM 推断自定义字段）", requires_db=False, llm_enhanced=True),
StepInfo(num=7, title="国家标准校验",
description="连库抽样：身份证/USCC/手机号/行政区划符合性", requires_db=True),
StepInfo(num=8, title="报告生成",
description="汇总所有 findings 生成 Markdown + Word 报告", requires_db=False),
])
# ── 列出已注册国标标准 ──
@router.get("/standards", response_model=StandardsListResponse)
async def list_standards():
try:
from standards.registry import list_all_standards
items = list_all_standards()
return StandardsListResponse(standards=[
StandardInfo(
id=x["id"], name=x["name"],
applies_to_fields=x["applies_to_fields"],
description=x["description"],
)
for x in items
])
except Exception as e:
logger.warning(f"加载标准列表失败: {e}")
return StandardsListResponse(standards=[])
# ── 测试连接 ──
@router.post("/connect/test", response_model=TestConnectionResponse)
async def connect_test(req: TestConnectionRequest):
logger.info(f"POST /api/connect/test db_type={req.db_type} host={req.host}:{req.port} db={req.database}")
cfg = DBConfig(
db_type=req.db_type, host=req.host, port=req.port,
user=req.user, password=req.password, database=req.database,
charset=req.charset, connect_timeout=req.connect_timeout,
)
ok, msg = test_connection(cfg)
if not ok:
logger.warning(f"连接测试失败: {msg}")
return TestConnectionResponse(ok=False, message=msg, db_type=req.db_type)
# 顺便取表数量
tables_count = None
try:
from ..core.db_adapter import open_db
with open_db(cfg) as db:
tables_count = len(db.list_tables(req.database))
logger.info(f"连接测试成功: {msg}, 表数 {tables_count}")
except Exception as e:
logger.warning(f"取表数量失败: {e}")
return TestConnectionResponse(
ok=True, message=msg, db_type=req.db_type,
database_name=req.database, tables_count=tables_count,
)
# ── 创建治理任务 ──
@router.post("/jobs", response_model=JobCreatedResponse)
async def create_job(req: ConnectRequest):
logger.info(
f"POST /api/jobs db={req.database}, db_type={req.db_type}, "
f"steps={req.steps or 'all'}, enable_llm={req.enable_llm}"
)
jm = JobManager.get_instance()
job = jm.submit(req)
return JobCreatedResponse(
job_id=job.job_id,
status=job.status,
created_at=job.created_at,
steps_planned=job.steps_planned or list(range(1, 9)),
)
# ── 任务状态 ──
@router.get("/jobs/{job_id}", response_model=JobStatusResponse)
async def get_job(job_id: str):
jm = JobManager.get_instance()
resp = jm.get_status_response(job_id)
if not resp:
logger.warning(f"GET /api/jobs/{job_id} 任务不存在")
raise HTTPException(status_code=404, detail="任务不存在")
logger.debug(f"GET /api/jobs/{job_id} status={resp.status} progress={resp.progress_pct}%")
return resp
# ── 取消任务 ──
@router.delete("/jobs/{job_id}")
async def cancel_job(job_id: str):
logger.info(f"DELETE /api/jobs/{job_id} 请求取消")
jm = JobManager.get_instance()
ok = jm.cancel(job_id)
if not ok:
logger.warning(f"取消任务失败: {job_id}")
raise HTTPException(status_code=400, detail="无法取消（任务不存在或已结束）")
return {"ok": True, "job_id": job_id}
# ── 任务结果 ──
@router.get("/jobs/{job_id}/result", response_model=JobResultResponse)
async def get_job_result(job_id: str):
jm = JobManager.get_instance()
job = jm.get(job_id)
if not job:
logger.warning(f"GET /api/jobs/{job_id}/result 任务不存在")
raise HTTPException(status_code=404, detail="任务不存在")
if job.status != "completed":
logger.warning(f"GET /api/jobs/{job_id}/result 任务未完成（status={job.status}）")
raise HTTPException(status_code=400, detail=f"任务未完成（当前: {job.status}）")
sections = job.result.get("sections", {})
overview_dict = sections.get("overview", {})
overview = OverviewStats(**{k: v for k, v in overview_dict.items() if k in OverviewStats.model_fields})
logger.info(f"GET /api/jobs/{job_id}/result 成功, sections={list(sections.keys())}")
return JobResultResponse(
job_id=job_id,
status=job.status,
overview=overview,
sections=sections,
reports=job.result.get("reports", {}),
)
# ── 日志流（SSE） ──
@router.get("/jobs/{job_id}/logs/stream")
async def stream_logs(job_id: str):
from sse_starlette.sse import EventSourceResponse
jm = JobManager.get_instance()
job = jm.get(job_id)
if not job:
logger.warning(f"GET /api/jobs/{job_id}/logs/stream 任务不存在")
raise HTTPException(status_code=404, detail="任务不存在")
logger.info(f"GET /api/jobs/{job_id}/logs/stream SSE 订阅开始 (历史 {len(job.logs.snapshot())} 条)")
async def event_generator():
async for entry in jm.stream_logs(job_id):
yield {"event": "log", "data": _json_dumps(entry)}
return EventSourceResponse(event_generator())
# ── 历史记录 ──
@router.get("/history", response_model=RunHistoryResponse)
async def list_history(database: Optional[str] = None, limit: int = Query(50, le=200)):
logger.info(f"GET /api/history database={database} limit={limit}")
jm = JobManager.get_instance()
items = jm.list_history(database=database, limit=limit)
logger.debug(f"返回 {len(items)} 条历史记录")
return RunHistoryResponse(items=[
RunHistoryItem(**{k: v for k, v in it.items() if k in RunHistoryItem.model_fields})
for it in items
])
# ── 报告下载 ──
@router.get("/reports/{database}/{run_id}/{filename}")
async def download_report(database: str, run_id: str, filename: str):
"""下载某次运行的报告（md / docx）"""
# 防止路径穿越
if ".." in run_id or ".." in filename or "/" in filename or "\\" in filename:
raise HTTPException(status_code=400, detail="非法文件名")
base = Path(__file__).resolve().parent.parent.parent / "outputs" / database / run_id / "reports"
target = base / filename
if not target.exists():
raise HTTPException(status_code=404, detail="报告不存在")
media_type = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" if filename.endswith(".docx") else "text/markdown"
return FileResponse(target, filename=filename, media_type=media_type)
# ── 历史运行的完整数据（按 run_id） ──
@router.get("/history/{database}/{run_id}/result")
async def get_history_result(database: str, run_id: str):
"""读取历史运行的完整 sections（从 _all_findings.json）"""
if ".." in run_id:
raise HTTPException(status_code=400, detail="非法 run_id")
base = Path(__file__).resolve().parent.parent.parent / "outputs" / database / run_id / "findings"
f = base / "_all_findings.json"
if not f.exists():
raise HTTPException(status_code=404, detail="结果文件不存在")
import json
return json.loads(f.read_text(encoding="utf-8"))
# ── 工具 ──
def _json_dumps(obj: Any) -> str:
import json
from datetime import datetime
return json.dumps(obj, ensure_ascii=False, default=lambda o: o.isoformat() if hasattr(o, "isoformat") else str(o))
\ No newline at end of file
web/app.py 0 → 100644
View file @ 7b4bd4a8
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Web 工具主入口
启动方式：
python -m web.app
# 或：
cd web && python app.py
环境变量：
ANTHROPIC_API_KEY Claude API Key（启用 LLM 增强）
WEB_PORT 端口（默认 8765）
WEB_HOST 主机（默认 0.0.0.0）
"""
from __future__ import annotations
import logging
import os
import sys
from pathlib import Path
# ── 把项目根目录加入 sys.path ──
WEB_DIR = Path(__file__).resolve().parent
PROJECT_ROOT = WEB_DIR.parent
sys.path.insert(0, str(PROJECT_ROOT))
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse
from fastapi.staticfiles import StaticFiles
# ── 日志 ──
LOG_DIR = WEB_DIR / "logs"
LOG_DIR.mkdir(parents=True, exist_ok=True)
logging.basicConfig(
level=os.environ.get("LOG_LEVEL", "INFO"),
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
handlers=[
logging.FileHandler(LOG_DIR / "app.log", encoding="utf-8"),
logging.StreamHandler(),
],
)
logger = logging.getLogger(__name__)
logger.info("=" * 60)
logger.info("数据治理 Web 工具 · 进程启动")
logger.info(f"Python: {sys.version.split()[0]}, 平台: {sys.platform}")
logger.info(f"工作目录: {PROJECT_ROOT}")
logger.info(f"日志目录: {LOG_DIR}")
# ── 创建 FastAPI 应用 ──
app = FastAPI(
title="数据治理 Web 工具",
description="基于 FastAPI + Vue 3 的数据库治理平台（支持 MySQL / 达梦）",
version="1.0.0",
)
logger.info("FastAPI 应用已创建")
# CORS（开发模式）
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
logger.debug("CORS 中间件已挂载（允许所有来源）")
# ── 注册路由 ──
from web.api import router as api_router
app.include_router(api_router)
logger.info(f"已注册 {len(app.routes)} 个路由")
# ── 启动时打印 LLM 配置（方便确认 Key 是否生效） ──
@app.on_event("startup")
async def show_llm_config():
logger.info("FastAPI 应用启动事件触发")
from web.core.llm import LLMConfig
cfg = LLMConfig.load()
if cfg.available:
masked_key = cfg.api_key[:6] + "..." + cfg.api_key[-4:] if len(cfg.api_key) > 12 else "***"
print(f"[LLM] provider={cfg.provider} model={cfg.effective_model}")
print(f"[LLM] base_url={cfg.effective_base_url or '(default)'}")
print(f"[LLM] api_key={masked_key}（已配置）")
logger.info(f"LLM 配置: provider={cfg.provider}, model={cfg.effective_model}")
else:
print("[LLM] 未配置 API Key → LLM 增强不可用，将使用规则推理")
logger.warning("LLM 未配置 API Key → LLM 增强不可用，将使用规则推理")
@app.on_event("shutdown")
async def shutdown_log():
logger.info("FastAPI 应用关闭")
# ── 静态资源 / 入口页 ──
STATIC_DIR = WEB_DIR / "static"
if STATIC_DIR.exists():
app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
@app.get("/", response_class=HTMLResponse)
async def root():
"""服务入口 HTML"""
index = STATIC_DIR / "index.html"
if index.exists():
return HTMLResponse(index.read_text(encoding="utf-8"))
return HTMLResponse("<h1>前端文件未找到</h1><p>请检查 web/static/index.html</p>")
# ── 启动 ──
def main():
import uvicorn
host = os.environ.get("WEB_HOST", "0.0.0.0")
port = int(os.environ.get("WEB_PORT", "8765"))
print("=" * 60)
print(f" 数据治理 Web 工具")
print(f" 访问: http://localhost:{port}")
print(f" API 文档: http://localhost:{port}/docs")
print("=" * 60)
uvicorn.run("web.app:app", host=host, port=port, reload=False, log_level="info")
if __name__ == "__main__":
main()
\ No newline at end of file
web/configs/defaults.yaml 0 → 100644
View file @ 7b4bd4a8
# ============================================================
# Web 工具默认配置
# 用户通过前端表单提交的内容会覆盖这里的连接信息
# ============================================================
# 数据库类型选项
db_types:
- mysql
- dameng
# 采样与阈值（与 CLI 工作流一致）
thresholds:
empty_field:
high: 0.80
mid: 0.50
min_rows_to_check: 5
similarity:
merge_threshold: 0.80
# 标准库
standards:
enabled: true
auto_apply: true
strict_mode: false
# LLM（Claude API）
llm:
model: claude-sonnet-5 # 默认模型
max_tokens: 1024
temperature: 0.2
# API Key 从环境变量 ANTHROPIC_API_KEY 读取，不在此保存
# Web 服务
web:
host: 0.0.0.0
port: 8765
title: "数据治理工具"
\ No newline at end of file
web/configs/llm.example.yaml 0 → 100644
View file @ 7b4bd4a8
# ============================================================
# LLM 配置文件示例 — 多种 provider 用法
# ============================================================
# 复制本文件为 llm.yaml，按需修改。
# ============================================================
# ── 方式 1：MiniMax（推荐本项目用户使用） ──
provider: minimax
api_key: sk-请填入你的_MiniMax_API_Key
base_url: https://api.minimaxi.com/anthropic
model: MiniMax-Text-01
# ── 方式 2：Anthropic Claude ──
# provider: anthropic
# api_key: sk-ant-xxxxx
# model: claude-sonnet-5
# （base_url 留空，用 Anthropic 官方）
# ── 方式 3：OpenAI ──
# provider: openai
# api_key: sk-xxxxx
# model: gpt-4o
# base_url: https://api.openai.com/v1
# ── 方式 4：自部署 / 第三方代理（OpenAI 兼容） ──
# provider: openai
# api_key: 自定义
# base_url: https://your-proxy.com/v1
# model: your-model-name
# 生成参数（通用）
max_tokens: 1024
temperature: 0.2
timeout: 30
\ No newline at end of file
web/core/__init__.py 0 → 100644
View file @ 7b4bd4a8
"""web.core 核心模块
- db_adapter: MySQL / 达梦 适配
- llm: Claude / OpenAI 客户端
- job_manager: 后台任务 + SSE
- orchestrator: 调度 8 步流程
- models: Pydantic 数据模型
"""
from .db_adapter import DBConfig, DBConnection, open_db, test_connection, quote_ident
from .llm import LLMClient, LLMConfig, get_llm_client, LLMUnavailable
__all__ = [
"DBConfig",
"DBConnection",
"open_db",
"test_connection",
"quote_ident",
"LLMClient",
"LLMConfig",
"get_llm_client",
"LLMUnavailable",
]
\ No newline at end of file
web/core/db_adapter.py 0 → 100644
View file @ 7b4bd4a8
"""数据库适配层
提供 MySQL / 达梦 的统一连接与查询接口，屏蔽方言差异：
- 连接管理
- 字段大小写规范化（达梦默认大写，MySQL 视配置而定）
- 反引号 vs 双引号
- information_schema 查询适配
- 标识符引用（防 SQL 注入）
设计原则：
- 暴露类 PEP-249 cursor-like 接口（execute / fetchall / fetchone）
- 列名统一返回小写 snake_case
- 所有动态拼接的标识符走 quote_ident() 函数
"""
from __future__ import annotations
import logging
import re
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Any, Iterator, Optional
import pymysql
from pymysql.cursors import DictCursor
logger = logging.getLogger(__name__)
# ── 数据类 ────────────────────────────────────────────────
@dataclass
class DBConfig:
"""统一的数据库连接配置"""
db_type: str # "mysql" | "dameng"
host: str
port: int
user: str
password: str
database: str # 数据库名/模式名
charset: str = "utf8mb4"
connect_timeout: int = 10
def to_pymysql_kwargs(self) -> dict:
return {
"host": self.host,
"port": self.port,
"user": self.user,
"password": self.password,
"database": self.database,
"charset": self.charset,
"connect_timeout": self.connect_timeout,
"cursorclass": DictCursor,
}
def to_dm_kwargs(self) -> dict:
"""dmPython 关键字参数映射"""
return {
"host": self.host,
"port": self.port,
"user": self.user,
"password": self.password,
"schema": self.database,
"charset": self.charset,
"connect_timeout": self.connect_timeout,
}
# ── 标识符引用 ────────────────────────────────────────────
_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
def quote_ident(name: str, db_type: str) -> str:
"""安全引用 SQL 标识符（表名 / 字段名）。
- MySQL 默认用反引号
- 达梦用双引号（ANSI 标准）
- 仅允许字母数字下划线开头，其余字符拒绝以防注入
"""
if not _IDENT_RE.match(name):
raise ValueError(f"非法标识符: {name!r}")
if db_type == "mysql":
return f"`{name}`"
return f'"{name}"'
def quote_value(value: Any) -> str:
"""把 Python 值转成 SQL 字面量。仅用于已知安全的字典/枚举值。"""
if value is None:
return "NULL"
if isinstance(value, bool):
return "1" if value else "0"
if isinstance(value, (int, float)):
return str(value)
# 字符串转义（仅供受信任字面量使用，业务表名/字段名走 quote_ident）
s = str(value).replace("'", "''")
return f"'{s}'"
# ── 列名规范化 ────────────────────────────────────────────
def normalize_column_name(name: str) -> str:
"""统一列名为小写 + 保持其余内容"""
return name.lower() if isinstance(name, str) else name
def normalize_rows(rows: list[dict] | None) -> list[dict]:
"""把查询结果的列名统一为小写"""
if not rows:
return []
return [{normalize_column_name(k): v for k, v in row.items()} for row in rows]
# ── 信息模式查询（通过 SQL 加载器，无内联 SQL） ──────────
# 所有 SQL 已抽到 web/sql/info_schema/ 目录下的 .sql 模板
# schema 一律走绑定参数（MySQL %s / 达梦 ?），不拼进 SQL 文本
def _render_info_columns_sql(dialect: str) -> str:
from web.sql.loader import get_sql_loader
return get_sql_loader().render("info_schema/list_columns", dialect=dialect)
def _render_info_tables_sql(dialect: str) -> str:
from web.sql.loader import get_sql_loader
return get_sql_loader().render("info_schema/list_tables", dialect=dialect)
def _render_health_sql(dialect: str) -> str:
from web.sql.loader import get_sql_loader
return get_sql_loader().render("health/check_connection", dialect=dialect)
# ── 字符串类型集合 ────────────────────────────────────────
TEXT_TYPES = {
# MySQL
"varchar", "char", "text", "longtext", "mediumtext", "tinytext", "enum", "set",
# 达梦
"varchar2", "nvarchar2", "char", "text", "clob", "longvarchar",
}
# ── 连接管理 ──────────────────────────────────────────────
class DBConnection:
"""统一的数据库连接封装。
用法（SQL 一律走 web/sql/ 模板，不要在 Python 里拼 SQL）：
with DBConnection(cfg) as db:
rows = db.list_columns("mydb") # schema 走绑定参数
# 需要自定义模板时，值用 params 传，标识符才用 ${x | quote}
from web.sql.loader import get_sql_loader
sql = get_sql_loader().render("verify/count_table_rows",
dialect=cfg.db_type, table="t_project")
n = db.fetch_scalar(sql)
"""
def __init__(self, cfg: DBConfig):
self.cfg = cfg
self._conn = None
self._driver = self._select_driver(cfg.db_type)
@staticmethod
def _select_driver(db_type: str):
if db_type == "mysql":
return "pymysql"
if db_type == "dameng":
try:
import dmPython # noqa: F401
except ImportError as e:
raise RuntimeError(
"未安装 dmPython，请执行: pip install dmPython\n"
"（达梦官方驱动需先安装达梦客户端）"
) from e
return "dmPython"
raise ValueError(f"不支持的数据库类型: {db_type!r}（目前支持 mysql / dameng）")
def __enter__(self):
logger.info(
f"[DB] 打开连接: {self.cfg.db_type}://{self.cfg.user}@{self.cfg.host}:"
f"{self.cfg.port}/{self.cfg.database} (driver={self._driver})"
)
self._connect_started = time.monotonic()
try:
if self._driver == "pymysql":
self._conn = pymysql.connect(**self.cfg.to_pymysql_kwargs())
else:
import dmPython
self._conn = dmPython.connect(**self.cfg.to_dm_kwargs())
except Exception as e:
logger.error(f"[DB] 连接失败: {type(e).__name__}: {e}")
raise
elapsed_ms = (time.monotonic() - self._connect_started) * 1000
logger.info(f"[DB] 连接已建立（耗时 {elapsed_ms:.0f}ms）")
return self
def __exit__(self, exc_type, exc, tb):
if self._conn is not None:
try:
self._conn.close()
logger.debug("[DB] 连接已关闭")
except Exception as e:
logger.warning(f"[DB] 关闭连接异常: {e}")
# ── 查询方法（自动规范化列名） ──
def _cursor(self):
if self._conn is None:
raise RuntimeError("DBConnection 未打开，请用 with 语句")
return self._conn.cursor()
def fetchall(self, sql: str, params: tuple | None = None) -> list[dict]:
cur = self._cursor()
started = time.monotonic()
try:
cur.execute(sql, params or ())
rows = cur.fetchall()
elapsed_ms = (time.monotonic() - started) * 1000
logger.debug(f"[SQL] fetchall {len(rows)} rows ({elapsed_ms:.1f}ms): {sql[:120]}...")
return normalize_rows(rows)
finally:
cur.close()
def fetchone(self, sql: str, params: tuple | None = None) -> Optional[dict]:
cur = self._cursor()
started = time.monotonic()
try:
cur.execute(sql, params or ())
row = cur.fetchone()
elapsed_ms = (time.monotonic() - started) * 1000
logger.debug(f"[SQL] fetchone ({elapsed_ms:.1f}ms): {sql[:120]}...")
if row is None:
return None
return {normalize_column_name(k): v for k, v in row.items()}
finally:
cur.close()
def fetch_scalar(self, sql: str, params: tuple | None = None) -> Any:
row = self.fetchone(sql, params)
if not row:
return None
return next(iter(row.values()))
def execute(self, sql: str, params: tuple | None = None) -> int:
cur = self._cursor()
started = time.monotonic()
try:
n = cur.execute(sql, params or ())
elapsed_ms = (time.monotonic() - started) * 1000
logger.debug(f"[SQL] execute {n} rows ({elapsed_ms:.1f}ms): {sql[:120]}...")
return n
finally:
cur.close()
# ── 信息模式便捷方法（schema 走绑定参数，不拼接） ──────
def list_columns(self, schema: str) -> list[dict]:
"""返回指定 schema 下所有列的元数据
SQL 通过 web/sql/info_schema/list_columns.<dialect>.sql 加载，
schema 作为绑定参数传给 cursor，由驱动层负责转义，杜绝 SQL 注入。
"""
sql = _render_info_columns_sql(self.cfg.db_type)
return self.fetchall(sql, (schema,))
def list_tables(self, schema: str) -> list[dict]:
"""返回指定 schema 下所有表的元数据（schema 走绑定参数）"""
sql = _render_info_tables_sql(self.cfg.db_type)
return self.fetchall(sql, (schema,))
def _fetchall_named(self, sql: str, params: dict) -> list[dict]:
"""保留方法以兼容达梦风格的命名占位符（暂未使用，预留）"""
cur = self._cursor()
try:
cur.execute(sql, params)
return normalize_rows(cur.fetchall())
finally:
cur.close()
# ── 上下文管理器快捷方式 ──────────────────────────────────
@contextmanager
def open_db(cfg: DBConfig) -> Iterator[DBConnection]:
with DBConnection(cfg) as conn:
yield conn
def test_connection(cfg: DBConfig) -> tuple[bool, str]:
"""测试连接是否可用，返回 (ok, message)"""
logger.info(
f"[DB] 测试连接: {cfg.db_type}://{cfg.user}@{cfg.host}:{cfg.port}/{cfg.database}"
)
try:
with DBConnection(cfg) as db:
sql = _render_health_sql(cfg.db_type)
v = db.fetch_scalar(sql)
if v == 1:
logger.info("[DB] 测试连接成功")
return True, f"连接成功（{cfg.db_type} @ {cfg.host}:{cfg.port}/{cfg.database}）"
logger.warning(f"[DB] 测试连接返回非预期值: {v}")
return False, "连接成功但查询返回非预期值"
except Exception as e:
logger.error(f"[DB] 测试连接失败: {type(e).__name__}: {e}")
return False, f"连接失败: {e}"
\ No newline at end of file
web/core/job_manager.py 0 → 100644
View file @ 7b4bd4a8
"""后台任务管理（JobManager）
设计目标：
- 单进程内多任务排队（本地工具，不需要 Celery/Redis）
- 每个任务有唯一 job_id，支持 SSE 实时日志流
- 任务状态持久化到 outputs/<db>/<ts>/，方便事后查看与对比
- LLM 失败 / DB 失败都能优雅降级，不让用户卡死
对外接口：
- JobManager.submit(req) 提交一个治理任务
- JobManager.get_status(job_id) 查询状态
- JobManager.subscribe_logs(job_id) 订阅日志（异步迭代器）
- JobManager.get_result(job_id) 获取结构化结果
- JobManager.list_history(db) 历史任务列表
"""
from __future__ import annotations
import asyncio
import json
import logging
import traceback
import uuid
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any, AsyncIterator, Optional
from .db_adapter import DBConfig
from .models import ConnectRequest, JobStatusResponse
from .orchestrator import run_governance_workflow
logger = logging.getLogger(__name__)
# ── 任务状态 ────────────────────────────────────────────
JOB_PENDING = "pending"
JOB_RUNNING = "running"
JOB_COMPLETED = "completed"
JOB_FAILED = "failed"
JOB_CANCELLED = "cancelled"
@dataclass
class LogBuffer:
"""环形日志缓冲 + 异步发布（订阅者可 SSE 流式接收）"""
max_size: int = 5000
def __post_init__(self):
self._entries: list[dict] = []
self._subscribers: list[asyncio.Queue] = []
self._lock = asyncio.Lock()
async def append(self, entry: dict) -> None:
async with self._lock:
self._entries.append(entry)
if len(self._entries) > self.max_size:
self._entries = self._entries[-self.max_size:]
# 推送给所有订阅者
dead = []
for q in self._subscribers:
try:
q.put_nowait(entry)
except asyncio.QueueFull:
dead.append(q)
for q in dead:
try:
self._subscribers.remove(q)
except ValueError:
pass
def snapshot(self) -> list[dict]:
return list(self._entries)
def subscribe(self) -> asyncio.Queue:
q: asyncio.Queue = asyncio.Queue(maxsize=1000)
self._subscribers.append(q)
return q
def unsubscribe(self, q: asyncio.Queue) -> None:
try:
self._subscribers.remove(q)
except ValueError:
pass
@dataclass
class Job:
job_id: str
req: ConnectRequest
status: str = JOB_PENDING
created_at: datetime = field(default_factory=datetime.now)
started_at: Optional[datetime] = None
finished_at: Optional[datetime] = None
current_step: Optional[int] = None
current_step_title: Optional[str] = None
steps_planned: list[int] = field(default_factory=list)
steps_completed: list[int] = field(default_factory=list)
steps_failed: list[int] = field(default_factory=list)
error: Optional[str] = None
db_info: dict = field(default_factory=dict)
logs: LogBuffer = field(default_factory=LogBuffer)
result: dict = field(default_factory=dict) # 各 section 数据
run_dir: Optional[Path] = None # outputs/<db>/<ts>/
_task: Optional[asyncio.Task] = None
_cancel_event: asyncio.Event = field(default_factory=asyncio.Event)
# ── 简易 logger 把日志也写到任务缓冲 ────────────────────
class JobLogHandler(logging.Handler):
def __init__(self, job: Job):
super().__init__()
self.job = job
self.setFormatter(logging.Formatter("%(message)s"))
def emit(self, record: logging.LogRecord) -> None:
try:
msg = self.format(record)
entry = {
"ts": datetime.now().isoformat(),
"level": record.levelname,
"step": getattr(record, "step", None),
"message": msg,
}
# 用线程安全的 put_nowait（log 可能从非 async 上下文触发）
try:
loop = asyncio.get_event_loop()
if loop.is_running():
loop.create_task(self.job.logs.append(entry))
else:
self.job.logs._entries.append(entry)
except RuntimeError:
self.job.logs._entries.append(entry)
except Exception:
self.handleError(record)
# ── JobManager ───────────────────────────────────────────
class JobManager:
"""单例风格的任务管理器"""
_instance: Optional["JobManager"] = None
def __init__(self, base_dir: Path):
self.base_dir = Path(base_dir)
self.base_dir.mkdir(parents=True, exist_ok=True)
self.jobs: dict[str, Job] = {}
@classmethod
def get_instance(cls) -> "JobManager":
if cls._instance is None:
base = Path(__file__).resolve().parent.parent / "outputs"
cls._instance = cls(base)
return cls._instance
# ── 提交 ──
def submit(self, req: ConnectRequest) -> Job:
job_id = datetime.now().strftime("%Y%m%d_%H%M%S_") + uuid.uuid4().hex[:6]
job = Job(job_id=job_id, req=req)
self.jobs[job_id] = job
job._task = asyncio.create_task(self._run_job(job))
logger.info(f"[job {job_id}] 已提交（数据库: {req.database}, db_type={req.db_type}, "
f"steps={req.steps or 'all'}, enable_llm={req.enable_llm}）")
return job
# ── 内部：执行任务 ──
async def _run_job(self, job: Job) -> None:
job.status = JOB_RUNNING
job.started_at = datetime.now()
# 准备运行目录
run_ts = job.started_at.strftime("%Y%m%d_%H%M%S")
run_dir = self.base_dir / job.req.database / run_ts
run_dir.mkdir(parents=True, exist_ok=True)
job.run_dir = run_dir
logger.info(f"[job {job.job_id}] 运行目录: {run_dir}")
# 收集要跑的步骤
all_steps = list(range(1, 9))
steps_planned = job.req.steps if job.req.steps else all_steps
job.steps_planned = steps_planned
logger.info(f"[job {job.job_id}] 计划步骤: {steps_planned}")
cfg = DBConfig(
db_type=job.req.db_type,
host=job.req.host,
port=job.req.port,
user=job.req.user,
password=job.req.password,
database=job.req.database,
charset=job.req.charset,
connect_timeout=job.req.connect_timeout,
)
job.db_info = {
"db_type": cfg.db_type,
"host": cfg.host,
"port": cfg.port,
"user": cfg.user,
"database": cfg.database,
}
logger.info(f"[job {job.job_id}] 数据库配置: {cfg.db_type}://{cfg.user}@{cfg.host}:{cfg.port}/{cfg.database}")
# 起始事件 → SSE 缓冲
await job.logs.append({
"ts": datetime.now().isoformat(),
"level": "INFO",
"step": None,
"message": f"任务启动（job_id={job.job_id}, db={cfg.database}, steps={steps_planned}）",
})
try:
result = await run_governance_workflow(
cfg=cfg,
steps=steps_planned,
run_dir=run_dir,
enable_llm=job.req.enable_llm,
on_log=lambda entry: self._sync_log(job, entry),
on_step_start=lambda num, title: self._sync_step_start(job, num, title),
on_step_done=lambda num, title, ok: self._sync_step_done(job, num, title, ok),
cancel_event=job._cancel_event,
)
job.result = result
job.status = JOB_COMPLETED
job.finished_at = datetime.now()
logger.info(f"[job {job.job_id}] 完成（耗时 {(job.finished_at - job.started_at).total_seconds():.1f}s）")
# 落盘 _all_findings.json
(run_dir / "findings").mkdir(exist_ok=True)
with open(run_dir / "findings" / "_all_findings.json", "w", encoding="utf-8") as f:
json.dump(result.get("sections", {}), f, ensure_ascii=False, indent=2, default=str)
except asyncio.CancelledError:
job.status = JOB_CANCELLED
job.finished_at = datetime.now()
logger.warning(f"[job {job.job_id}] 任务被用户取消")
except Exception as e:
job.error = f"{type(e).__name__}: {e}"
job.status = JOB_FAILED
job.finished_at = datetime.now()
logger.error(f"[job {job.job_id}] 任务失败: {e}")
logger.error(traceback.format_exc())
await job.logs.append({
"ts": datetime.now().isoformat(),
"level": "ERROR",
"step": None,
"message": f"任务失败: {e}",
})
def _sync_log(self, job: Job, entry: dict) -> None:
"""从工作流同步线程推送日志到异步缓冲"""
try:
loop = asyncio.get_event_loop()
if loop.is_running():
loop.create_task(job.logs.append(entry))
else:
job.logs._entries.append(entry)
except RuntimeError:
job.logs._entries.append(entry)
def _sync_step_start(self, job: Job, num: int, title: str) -> None:
job.current_step = num
job.current_step_title = title
def _sync_step_done(self, job: Job, num: int, title: str, ok: bool) -> None:
if ok:
job.steps_completed.append(num)
else:
job.steps_failed.append(num)
# ── 查询 ──
def get(self, job_id: str) -> Optional[Job]:
return self.jobs.get(job_id)
def get_status_response(self, job_id: str) -> Optional[JobStatusResponse]:
job = self.get(job_id)
if not job:
return None
progress = 0.0
if job.steps_planned:
progress = round(len(job.steps_completed) / len(job.steps_planned) * 100, 1)
return JobStatusResponse(
job_id=job.job_id,
status=job.status,
created_at=job.created_at,
started_at=job.started_at,
finished_at=job.finished_at,
current_step=job.current_step,
current_step_title=job.current_step_title,
progress_pct=progress,
steps_planned=job.steps_planned,
steps_completed=job.steps_completed,
steps_failed=job.steps_failed,
error=job.error,
db_info=job.db_info,
)
# ── 取消 ──
def cancel(self, job_id: str) -> bool:
job = self.get(job_id)
if not job or job.status not in (JOB_PENDING, JOB_RUNNING):
return False
job._cancel_event.set()
return True
# ── 日志订阅（SSE 用） ──
async def stream_logs(self, job_id: str) -> AsyncIterator[dict]:
job = self.get(job_id)
if not job:
return
# 先发历史
for entry in job.logs.snapshot():
yield entry
if job.status in (JOB_COMPLETED, JOB_FAILED, JOB_CANCELLED):
return
# 再订阅新日志
q = job.logs.subscribe()
try:
while True:
try:
entry = await asyncio.wait_for(q.get(), timeout=15.0)
yield entry
except asyncio.TimeoutError:
# 发送心跳，让前端知道连接还活着
yield {"ts": datetime.now().isoformat(), "level": "INFO", "step": None, "message": "[heartbeat]"}
if job.status in (JOB_COMPLETED, JOB_FAILED, JOB_CANCELLED):
return
finally:
job.logs.unsubscribe(q)
# ── 历史 ──
def list_history(self, database: str | None = None, limit: int = 50) -> list[dict]:
items: list[dict] = []
db_root = self.base_dir
targets = [db_root / database] if database else sorted(p for p in db_root.iterdir() if p.is_dir())
for db_dir in targets:
if not db_dir.exists():
continue
for run_dir in sorted(db_dir.iterdir(), reverse=True):
if not run_dir.is_dir():
continue
findings_dir = run_dir / "findings"
all_f = findings_dir / "_all_findings.json"
if not all_f.exists():
continue
try:
sections = json.loads(all_f.read_text(encoding="utf-8"))
except Exception:
sections = {}
# 统计
stats: dict[str, Any] = {}
for k, v in sections.items():
if isinstance(v, dict):
if "summary" in v and isinstance(v["summary"], dict):
for sk, sv in v["summary"].items():
if isinstance(sv, (int, float)):
stats[f"{k}_{sk}"] = sv
stats["total_tables"] = sections.get("overview", {}).get("total_tables", 0)
stats["total_fields"] = sections.get("overview", {}).get("total_fields", 0)
reports_dir = run_dir / "reports"
has_md = (reports_dir / f"数据治理报告_{db_dir.name}.md").exists()
has_docx = (reports_dir / f"数据治理报告_{db_dir.name}.docx").exists()
# duration
duration = None
try:
log = (run_dir / "run.log").read_text(encoding="utf-8")
start_lines = [ln for ln in log.splitlines() if "开始" in ln or "Step 1" in ln]
end_lines = [ln for ln in log.splitlines() if "完成" in ln and "流程" in ln]
if start_lines and end_lines:
# 简化：直接用文件 mtime - ctime
pass
dur = run_dir.stat().st_mtime - run_dir.stat().st_ctime
duration = round(dur, 1)
except Exception:
pass
items.append({
"run_id": run_dir.name,
"database": db_dir.name,
"host": "(历史)",
"db_type": "mysql",
"executed_at": datetime.fromtimestamp(run_dir.stat().st_mtime).isoformat(),
"duration_sec": duration,
"stats": stats,
"has_markdown": has_md,
"has_docx": has_docx,
"run_dir": str(run_dir),
})
if len(items) >= limit:
return items
return items
# ── 结果 ──
def get_result(self, job_id: str) -> dict | None:
job = self.get(job_id)
if not job:
return None
return job.result
\ No newline at end of file
web/core/llm.py 0 → 100644
View file @ 7b4bd4a8
"""LLM 客户端封装（支持 MiniMax / Anthropic / OpenAI）
把 Anthropic SDK 包成几个语义清晰的"治理专用方法"，在 Step 中调用。
所有方法都设计为可降级 — 如果 LLM 不可用（无 API Key / 超时 / 解析失败），返回 None 或 fallback 值，
不阻断主流程。
业务方法：
- predict_field_comment(): 推测字段注释（替换 COMMENT_HINTS 硬编码）
- suggest_table_merge(): 表合并建议文案
- classify_empty_field(): 空字段成因分类（业务不需要 / 采集漏了）
- suggest_length_rule(): 自定义字段长度建议
- summarize_findings(): 整份治理报告的执行摘要
支持的 Provider：
- minimax: MiniMax（M2.7 / MiniMax-Text-01 等）— 通过 Anthropic 兼容接口
- anthropic: Claude（官方 Anthropic SDK）
- openai: GPT（OpenAI 兼容协议）
配置优先级（从高到低）：
1. 环境变量（最高）：LLM_PROVIDER / LLM_API_KEY / LLM_BASE_URL / LLM_MODEL
2. 配置文件（中间）：web/configs/llm.yaml
3. 内置默认值（最低）
启动时 LLM 客户端初始化失败会自动降级，治理流程仍可继续。
"""
from __future__ import annotations
import json
import logging
import os
import re
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Optional
logger = logging.getLogger(__name__)
# ── Provider 预设 ────────────────────────────────────────
# 各 Provider 的默认 base_url 与模型，避免每次都填
PROVIDER_PRESETS: dict[str, dict] = {
"minimax": {
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-Text-01",
},
"anthropic": {
"base_url": "", # 不填，使用官方默认
"model": "claude-sonnet-5",
},
"openai": {
"base_url": "", # 不填，使用官方默认
"model": "gpt-4o",
},
}
# ── 配置文件路径 ─────────────────────────────────────────
def default_config_path() -> Path:
"""默认 LLM 配置文件路径：web/configs/llm.yaml
可通过环境变量 LLM_CONFIG_FILE 覆盖：
export LLM_CONFIG_FILE=/path/to/my-llm.yaml
"""
env_path = os.environ.get("LLM_CONFIG_FILE")
if env_path:
return Path(env_path)
return Path(__file__).resolve().parent.parent / "configs" / "llm.yaml"
# ── 配置 ────────────────────────────────────────────────
@dataclass
class LLMConfig:
provider: str = "minimax" # minimax / anthropic / openai
api_key: Optional[str] = None
base_url: Optional[str] = None # 留空则按 provider 预设
model: Optional[str] = None # 留空则按 provider 预设
max_tokens: int = 1024
temperature: float = 0.2
timeout: int = 30
@property
def effective_model(self) -> str:
return self.model or PROVIDER_PRESETS.get(self.provider, {}).get("model", "claude-sonnet-5")
@property
def effective_base_url(self) -> str:
return self.base_url or PROVIDER_PRESETS.get(self.provider, {}).get("base_url", "")
@property
def available(self) -> bool:
return bool(self.api_key)
@classmethod
def load(cls, config_path: Optional[Path] = None) -> "LLMConfig":
"""三段式加载：默认值 < 配置文件 < 环境变量（环境变量最高优先）
优先级说明：
- 文件配置：方便项目级持久化（推荐把 provider / model / base_url 写这里）
- 环境变量：方便临时切换 / 容器化部署 / CI
"""
path = config_path or default_config_path()
cfg = cls() # 默认值
# 1. 文件
file_cfg = _read_yaml_config(path)
if file_cfg:
for k, v in file_cfg.items():
if v is not None and hasattr(cfg, k):
setattr(cfg, k, v)
# 2. 先把 provider 应用（再决定从哪个专用 env 拿 key）
env_provider = os.environ.get("LLM_PROVIDER")
if env_provider:
cfg.provider = env_provider
# 3. 环境变量（最高）
env_overrides: dict[str, Any] = {
"api_key": os.environ.get("LLM_API_KEY"),
"base_url": os.environ.get("LLM_BASE_URL"),
"model": os.environ.get("LLM_MODEL"),
}
if env_overrides["api_key"] is None:
# 兼容各 provider 的专用 env var（以更新后的 provider 为准）
if cfg.provider == "anthropic":
env_overrides["api_key"] = os.environ.get("ANTHROPIC_API_KEY")
elif cfg.provider == "openai":
env_overrides["api_key"] = os.environ.get("OPENAI_API_KEY")
elif cfg.provider == "minimax":
env_overrides["api_key"] = os.environ.get("MINIMAX_API_KEY")
env_max_tokens = os.environ.get("LLM_MAX_TOKENS")
if env_max_tokens:
try:
cfg.max_tokens = int(env_max_tokens)
except (TypeError, ValueError):
pass
for k, v in env_overrides.items():
if v is not None and v != "":
setattr(cfg, k, v)
return cfg
def _read_yaml_config(path: Path) -> Optional[dict]:
"""读取 YAML 配置文件"""
if not path.exists():
return None
try:
import yaml
with open(path, encoding="utf-8") as f:
data = yaml.safe_load(f) or {}
return data if isinstance(data, dict) else None
except ImportError:
logger.warning("未安装 PyYAML，跳过文件配置加载")
return None
except Exception as e:
logger.warning(f"读取 LLM 配置文件失败 {path}: {e}")
return None
# ── 客户端基类 ────────────────────────────────────────────
class LLMUnavailable(Exception):
"""LLM 不可用（无 Key / 网络/解析失败）"""
class LLMClient:
"""统一 LLM 客户端。失败抛 LLMUnavailable，由调用方降级。"""
def __init__(self, cfg: LLMConfig | None = None, config_path: Optional[Path] = None):
self.cfg = cfg or LLMConfig.load(config_path)
self._provider = None
if self.cfg.available:
try:
self._provider = self._init_provider()
logger.info(
f"LLM 客户端就绪: provider={self.cfg.provider}, "
f"model={self.cfg.effective_model}, "
f"base_url={self.cfg.effective_base_url or '默认'}"
)
except Exception as e:
logger.warning(f"LLM 初始化失败: {e}")
else:
logger.info(
f"LLM 未配置 API Key（provider={self.cfg.provider}），"
f"如需启用请在 web/configs/llm.yaml 设置 api_key 或配置环境变量"
)
def _init_provider(self):
"""根据 provider 创建对应 SDK 客户端"""
provider = self.cfg.provider
api_key = self.cfg.api_key
base_url = self.cfg.effective_base_url or None
if provider in ("anthropic", "minimax"):
# MiniMax 用 Anthropic SDK + 自定义 base_url
import anthropic
kwargs: dict[str, Any] = {
"api_key": api_key,
"timeout": self.cfg.timeout,
}
if base_url:
kwargs["base_url"] = base_url
return anthropic.Anthropic(**kwargs)
elif provider == "openai":
import openai
kwargs = {"api_key": api_key, "timeout": self.cfg.timeout}
if base_url:
kwargs["base_url"] = base_url
return openai.OpenAI(**kwargs)
raise ValueError(
f"不支持的 LLM provider: {provider!r}（目前支持 minimax / anthropic / openai）"
)
@property
def available(self) -> bool:
return self._provider is not None
# ── 核心调用 ──
def complete(self, prompt: str, system: str = "", json_mode: bool = False) -> str:
"""调用 LLM，返回纯文本。失败抛 LLMUnavailable。"""
if not self.available:
raise LLMUnavailable("LLM 客户端未配置 API Key")
try:
if self.cfg.provider == "anthropic":
kwargs: dict[str, Any] = {
"model": self.cfg.model,
"max_tokens": self.cfg.max_tokens,
"temperature": self.cfg.temperature,
"messages": [{"role": "user", "content": prompt}],
}
if system:
kwargs["system"] = system
msg = self._provider.messages.create(**kwargs)
return msg.content[0].text
# openai
kwargs = {
"model": self.cfg.model,
"max_tokens": self.cfg.max_tokens,
"temperature": self.cfg.temperature,
"messages": [{"role": "system", "content": system},
{"role": "user", "content": prompt}],
}
if json_mode:
kwargs["response_format"] = {"type": "json_object"}
resp = self._provider.chat.completions.create(**kwargs)
return resp.choices[0].message.content
except Exception as e:
logger.warning(f"LLM 调用失败: {e}")
raise LLMUnavailable(str(e))
# ── JSON 模式（带解析兜底） ──
def complete_json(self, prompt: str, system: str = "") -> dict | list | None:
"""调用 LLM 并要求返回 JSON。解析失败返回 None。"""
json_system = (system + "\n\n" if system else "") + (
"严格返回合法 JSON，不要 Markdown 代码块包裹，不要任何额外文字。"
)
text = self.complete(prompt, system=json_system, json_mode=True)
return _safe_parse_json(text)
# ── 业务方法：字段注释推测 ──
def predict_field_comment(
self,
table_name: str,
table_comment: str,
column_name: str,
data_type: str,
) -> dict | None:
"""推测字段语义，返回 {"comment": str, "confidence": "high|medium|low"} 或 None"""
prompt = f"""你是数据库治理专家。请根据下面的信息，推测一个字段的中文注释。
表名：{table_name}
表注释：{table_comment or '（无）'}
字段名：{column_name}
数据类型：{data_type}
要求：
1. 给出 5-20 字的中文注释
2. 如字段名是拼音首字母缩写（如 xzqhbm），翻译为中文
3. 如字段名是英文组合（如 create_time），翻译为中文
4. 给出置信度（high=非常确定；medium=较确定；low=猜测）
严格返回 JSON 格式：{{"comment": "...", "confidence": "high|medium|low"}}
"""
result = self.complete_json(prompt)
if isinstance(result, dict) and "comment" in result:
return {
"comment": str(result["comment"]).strip(),
"confidence": result.get("confidence", "low"),
}
return None
# ── 业务方法：表合并建议 ──
def suggest_table_merge(
self,
tables: list[dict],
similarity: float,
common_columns: list[str],
diff_columns: list[str],
) -> dict | None:
"""对一组结构高度相似的表给出合并建议。"""
prompt = f"""数据库治理专家：以下 {len(tables)} 张表结构高度相似（相似度 {similarity:.0%}），请评估是否应该合并。
表清单：
{chr(10).join(f'- {t["name"]}: {t.get("comment", "（无注释）")} ({t.get("row_count", "?")} 行)' for t in tables)}
共同字段：{', '.join(common_columns[:15])}
差异字段：{', '.join(diff_columns[:10])}
请评估：
1. 是否真的可以合并（业务上是否同义）
2. 合并策略（保留主表 + 区分字段 / 拆字段命名空间 / 拒绝合并）
3. 合并风险
4. 整改建议（具体 SQL 思路）
严格返回 JSON：
{{"verdict": "merge|partial|reject", "strategy": "...", "risks": ["..."], "suggestion": "..."}}
"""
return self.complete_json(prompt)
# ── 业务方法：空字段成因分类 ──
def classify_empty_field(
self,
table_name: str,
table_comment: str,
column_name: str,
column_comment: str,
data_type: str,
empty_rate: float,
sample_values: list[str],
) -> dict | None:
"""判断高空字段是「业务上不需要」还是「采集漏了」。"""
prompt = f"""数据库治理专家：以下字段高空（空值率 {empty_rate:.0%}），请判断原因。
表：{table_name}（{table_comment or '无注释'}）
字段：{column_name}（{column_comment or '无注释'}，{data_type}）
空值率：{empty_rate:.1%}
非空样本：{sample_values[:5]}
判断维度：
- 「业务不需要」：如历史遗留字段、可选字段、新字段尚未使用
- 「采集漏了」：本应必有数据但空了（数据质量问题）
- 「预留字段」：为将来扩展保留
返回 JSON：{{"classification": "unused|missing|reserved|unknown", "reasoning": "...", "action": "保留|补采|删除|观察"}}
"""
return self.complete_json(prompt)
# ── 业务方法：字段长度建议 ──
def suggest_length_rule(
self,
column_name: str,
column_comment: str,
current_length: int,
table_name: str,
) -> dict | None:
"""对自定义字段推断合理长度。"""
prompt = f"""数据库治理专家：以下字段当前定义长度为 {current_length}，是否合理？
表：{table_name}
字段：{column_name}（{column_comment or '无注释'}）
当前定义：{current_length}
请基于字段语义推断：
- 最小必需长度
- 推荐长度（考虑未来扩展）
- 节省空间的建议
返回 JSON：{{"min_length": int, "recommended": int, "reasoning": "..."}}
"""
return self.complete_json(prompt)
# ── 业务方法：报告摘要 ──
def summarize_findings(self, overview: dict, stats: dict) -> str | None:
"""生成一段 200 字内的执行摘要。"""
prompt = f"""以下是数据库治理的统计结果，请用 200 字内中文写出执行摘要（领导视角，关注最重要的 3 个风险）：
概览：{json.dumps(overview, ensure_ascii=False)}
统计：{json.dumps(stats, ensure_ascii=False)}
直接输出摘要文本，不要 Markdown 标题。"""
try:
return self.complete(prompt)
except LLMUnavailable:
return None
# ── JSON 解析兜底 ────────────────────────────────────────
def _safe_parse_json(text: str) -> Any:
"""从 LLM 返回中提取 JSON（容忍代码块包裹、尾部多余文字）"""
if not text:
return None
# 去掉 Markdown 代码块
text = re.sub(r"^```(?:json)?\s*", "", text.strip())
text = re.sub(r"\s*```$", "", text)
# 找第一个 { 或 [
start = -1
for i, ch in enumerate(text):
if ch in "[{":
start = i
break
if start == -1:
return None
candidate = text[start:].strip()
# 尝试解析，失败则尝试逐步截断尾部字符
try:
return json.loads(candidate)
except json.JSONDecodeError:
pass
# 容错：去掉尾部的非 JSON 字符
for end in range(len(candidate), 0, -1):
try:
return json.loads(candidate[:end])
except json.JSONDecodeError:
continue
return None
# ── 模块级单例 ────────────────────────────────────────────
_client: LLMClient | None = None
def get_llm_client() -> LLMClient:
"""获取全局 LLM 客户端单例（懒初始化）"""
global _client
if _client is None:
_client = LLMClient()
return _client
def reset_llm_client() -> None:
"""重置单例（配置变更时调用）"""
global _client
_client = None
\ No newline at end of file
web/core/models.py 0 → 100644
View file @ 7b4bd4a8
"""Pydantic 数据模型 — Web API 的请求/响应契约"""
from __future__ import annotations
from datetime import datetime
from typing import Any, Optional
from pydantic import BaseModel, Field
# ── 请求 ────────────────────────────────────────────────
class ConnectRequest(BaseModel):
"""前端「新建治理任务」表单提交"""
db_type: str = Field(..., description="mysql / dameng")
host: str
port: int = 3306
user: str
password: str
database: str
charset: str = "utf8mb4"
connect_timeout: int = 10
# 可选：选择要跑的步骤（默认全 8 步）
steps: Optional[list[int]] = None
# 可选：是否启用 LLM 增强（默认 True，缺 Key 时自动降级）
enable_llm: bool = True
# 可选：报告标题
report_title: Optional[str] = None
class TestConnectionRequest(ConnectRequest):
"""只测连通性，不启动任务"""
pass
# ── 响应 ────────────────────────────────────────────────
class TestConnectionResponse(BaseModel):
ok: bool
message: str
db_type: Optional[str] = None
database_name: Optional[str] = None # 改名为避免与 BaseModel.schema() 冲突
tables_count: Optional[int] = None
class JobCreatedResponse(BaseModel):
job_id: str
status: str
created_at: datetime
steps_planned: list[int]
class JobStatusResponse(BaseModel):
job_id: str
status: str # pending / running / completed / failed / cancelled
created_at: datetime
started_at: Optional[datetime] = None
finished_at: Optional[datetime] = None
current_step: Optional[int] = None
current_step_title: Optional[str] = None
progress_pct: float = 0.0 # 0-100
steps_planned: list[int] = []
steps_completed: list[int] = []
steps_failed: list[int] = []
error: Optional[str] = None
db_info: Optional[dict] = None
class StepInfo(BaseModel):
num: int
title: str
description: str
requires_db: bool
llm_enhanced: bool = False
class StepsListResponse(BaseModel):
steps: list[StepInfo]
class StandardInfo(BaseModel):
id: str
name: str
applies_to_fields: list[str]
description: str
class StandardsListResponse(BaseModel):
standards: list[StandardInfo]
# ── 治理结果（前端多级表格消费） ──
class OverviewStats(BaseModel):
database: str
host: str
db_type: str
total_tables: int = 0
total_fields: int = 0
high_empty_fields: int = 0
missing_comments: int = 0
length_issues: int = 0
standard_violations: int = 0
merge_candidates: int = 0
redundancy_fields: int = 0
executed_at: Optional[datetime] = None
class JobResultResponse(BaseModel):
job_id: str
status: str
overview: OverviewStats
# 各章节原始数据（前端按章节渲染多级表格）
sections: dict[str, Any] = {}
# 已生成报告路径
reports: dict[str, Optional[str]] = {}
class LogEntry(BaseModel):
ts: datetime
level: str
step: Optional[str] = None
message: str
class JobLogsResponse(BaseModel):
job_id: str
entries: list[LogEntry]
class RunHistoryItem(BaseModel):
run_id: str # 时间戳目录名
database: str
host: str
db_type: Optional[str] = "mysql"
executed_at: datetime
duration_sec: Optional[float] = None
stats: dict[str, Any] = {}
has_markdown: bool
has_docx: bool
class RunHistoryResponse(BaseModel):
items: list[RunHistoryItem]
\ No newline at end of file
web/core/orchestrator.py 0 → 100644
View file @ 7b4bd4a8
"""治理流程调度器
把 8 个 Step 串起来执行：
1. 数据字典 → 2. 合并/冗余(离线) → 3. 数据验证(连库)
→ 4. 空字段(连库) → 5. 缺注释(离线) → 6. 字段长度(离线)
→ 7. 国标校验(连库) → 8. 报告生成(离线)
特性：
- LLM 失败自动降级
- 每步通过 callback 推送日志
- 支持取消（asyncio.Event）
- 产出统一 sections dict 给前端
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import time
import traceback
from datetime import datetime
from pathlib import Path
from typing import Any, Callable, Optional
from .db_adapter import DBConfig, open_db
from .llm import get_llm_client
logger = logging.getLogger(__name__)
# ── Step 注册表 ────────────────────────────────────────────
# 顺序敏感：前面 Step 的产出可被后面 Step 复用
STEP_REGISTRY: list[tuple[int, str, Callable, bool, bool]] = [
# (num, title, fn, requires_db, llm_enhanced)
(1, "获取数据字典", "_run_step1", True, False),
(2, "表合并与冗余字段分析", "_run_step2", False, True),
(3, "数据验证", "_run_step3", True, False),
(4, "大范围空字段扫描", "_run_step4", True, True),
(5, "缺失注释检查 + 推测", "_run_step5", False, True),
(6, "字段长度检查", "_run_step6", False, True),
(7, "国家标准校验", "_run_step7", True, False),
(8, "报告生成", "_run_step8", False, False),
]
# ── 工具函数 ────────────────────────────────────────────
def _check_cancel(event: asyncio.Event) -> None:
if event.is_set():
raise asyncio.CancelledError()
async def run_governance_workflow(
cfg: DBConfig,
steps: list[int],
run_dir: Path,
enable_llm: bool = True,
on_log: Callable[[dict], None] | None = None,
on_step_start: Callable[[int, str], None] | None = None,
on_step_done: Callable[[int, str, bool], None] | None = None,
cancel_event: Optional[asyncio.Event] = None,
) -> dict:
"""主入口：跑一轮治理，返回结构化结果。
Returns:
{
"sections": {<section_key>: <step_output_dict>},
"overview": {database, host, db_type, total_tables, total_fields, ...},
"reports": {"markdown": str|None, "docx": str|None},
"elapsed_sec": float,
}
"""
if cancel_event is None:
cancel_event = asyncio.Event()
started = datetime.now()
started_ts = time.monotonic()
sections: dict[str, Any] = {}
step_outputs: dict[str, Any] = {} # 原始 step 输出（含 data_dictionary 等）
findings_dir = run_dir / "findings"
findings_dir.mkdir(parents=True, exist_ok=True)
# ── 为本次运行单独挂一个 FileHandler，落到 run_dir/run.log ──
# 注意：只挂到最顶层 'web' logger 一次，避免 propagation 重复写
run_log_path = run_dir / "run.log"
run_handler = logging.FileHandler(run_log_path, encoding="utf-8", mode="w")
run_handler.setFormatter(
logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s")
)
run_handler.setLevel(os.environ.get("LOG_LEVEL", "INFO"))
web_logger = logging.getLogger("web")
web_logger.addHandler(run_handler)
web_logger.setLevel(os.environ.get("LOG_LEVEL", "INFO"))
# 记下挂载前的 level，结束时恢复
_prev_web_level = web_logger.level
# 关键 Step 失败应直接终止流程（其他 Step 依赖其产出）
CRITICAL_STEPS = {1}
llm = get_llm_client() if enable_llm else None
def log(level: str, message: str, step: str | None = None) -> None:
"""统一日志入口：推到 SSE 缓冲 + 写到全局 logger + 本次 run.log"""
entry = {"ts": datetime.now().isoformat(),
"level": level, "step": step, "message": message}
if on_log:
on_log(entry)
logger.log(getattr(logging, level, logging.INFO),
f"[{step or '-'}] {message}" if step else message)
# ── 执行每个 Step ──
log("INFO", f"本次治理配置: db_type={cfg.db_type}, host={cfg.host}:{cfg.port}, db={cfg.database}")
log("INFO", f"计划执行步骤: {steps}")
total_steps = len([n for n, *_ in STEP_REGISTRY if n in steps])
for step_idx, (num, title, fn_name, requires_db, llm_enhanced) in enumerate(STEP_REGISTRY, 1):
if num not in steps:
continue
_check_cancel(cancel_event)
if on_step_start:
on_step_start(num, title)
log("INFO", f"开始 {title}", step=str(num))
if requires_db:
log("INFO", " 需要连接数据库", step=str(num))
step_started = time.monotonic()
# 标记 LLM 状态
llm_note = ""
if llm_enhanced:
if llm and llm.available:
llm_note = f"（LLM 增强已启用: provider={llm.cfg.provider}, model={llm.cfg.effective_model}）"
else:
llm_note = "（LLM 未配置，将使用规则推理）"
if llm_note:
log("INFO", llm_note, step=str(num))
try:
# 在线程池里跑同步步骤（DB 查询 / 文件 IO）
step_fn = globals()[fn_name]
output = await asyncio.get_event_loop().run_in_executor(
None,
lambda: step_fn(
cfg=cfg,
step_outputs=step_outputs,
llm=llm if llm_enhanced else None,
findings_dir=findings_dir,
log=log,
cancel_event=cancel_event,
),
)
# output: {"section_key": ..., "data": ..., "extras": ...}
section_key = output.get("section_key")
if section_key:
sections[section_key] = output["data"]
if output.get("extras"):
step_outputs[step_name(num)] = output["extras"]
# 落盘原始 JSON
step_json = findings_dir / f"step_{step_name(num)}.json"
with open(step_json, "w", encoding="utf-8") as f:
json.dump(output["data"], f, ensure_ascii=False, indent=2, default=str)
elapsed = time.monotonic() - step_started
if on_step_done:
on_step_done(num, title, True)
log("INFO",
f"完成 {title}（耗时 {elapsed:.1f}s, 进度 {step_idx}/{total_steps}）",
step=str(num))
except asyncio.CancelledError:
if on_step_done:
on_step_done(num, title, False)
log("WARN", f"{title} 被取消", step=str(num))
raise
except Exception as e:
if on_step_done:
on_step_done(num, title, False)
log("ERROR", f"{title} 失败: {type(e).__name__}: {e}", step=str(num))
traceback.print_exc()
# 关键 Step 失败 → 直接终止整个流程
if num in CRITICAL_STEPS:
raise RuntimeError(f"关键步骤 Step {num} 失败，终止流程: {e}") from e
# 其他 Step 失败：记录后继续（已写入日志和步骤标记）
# ── 计算 overview ──
overview = _build_overview(sections, cfg)
sections["overview"] = overview
log("INFO", f"概览汇总: 表 {overview.get('total_tables',0)} 张, "
f"字段 {overview.get('total_fields',0)} 个, "
f"高空字段 {overview.get('high_empty_fields',0)}, "
f"缺失注释 {overview.get('missing_comments',0)}, "
f"长度异常 {overview.get('length_issues',0)}, "
f"国标违规 {overview.get('standard_violations',0)}")
# ── Step 8：报告生成（在调度器中独立处理） ──
if 8 in steps:
_check_cancel(cancel_event)
log("INFO", "开始 报告生成", step="8")
report_started = time.monotonic()
try:
md_path, docx_path = await asyncio.get_event_loop().run_in_executor(
None,
lambda: _generate_reports(run_dir, sections, overview, log=log),
)
sections.setdefault("reports", {})
sections["reports"] = {"markdown": md_path, "docx": docx_path}
elapsed = time.monotonic() - report_started
log("INFO", f"报告已生成: md={md_path}, docx={docx_path}（耗时 {elapsed:.1f}s）", step="8")
except Exception as e:
log("ERROR", f"报告生成失败: {type(e).__name__}: {e}", step="8")
traceback.print_exc()
elapsed = time.monotonic() - started_ts
log("INFO", f"治理流程结束，总耗时 {elapsed:.1f}s")
# 卸载本次 run 的 FileHandler，避免重复写
try:
web_logger.removeHandler(run_handler)
except ValueError:
pass
web_logger.setLevel(_prev_web_level)
run_handler.close()
return {
"sections": sections,
"overview": overview,
"elapsed_sec": elapsed,
"reports": sections.get("reports", {}),
}
def step_name(num: int) -> str:
return {
1: "1_data_dict",
2: "2_merge_redundancy",
3: "3_verify",
4: "4_empty_fields",
5: "5_missing_comments",
6: "6_length_check",
7: "7_standards",
8: "8_report",
}[num]
# ── 各 Step 实现（简化版 + LLM 接入点） ──
def _run_step1(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 1: 数据字典"""
from .step_impl.step1_data_dict import run_step1
data = run_step1(cfg, log=log)
return {"section_key": "overview_meta", "data": data, "extras": data}
def _run_step2(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 2: 表合并与冗余（离线）"""
from .step_impl.step2_merge_redundancy import run_step2
dict_data = step_outputs.get("1_data_dict", {})
data = run_step2(dict_data, llm=llm, log=log)
return {"section_key": "merge_candidates", "data": data}
def _run_step3(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 3: 数据验证（连库）"""
from .step_impl.step3_verify import run_step3
s2 = step_outputs.get("2_merge_redundancy", {})
data = run_step3(cfg, dict_data=step_outputs.get("1_data_dict", {}),
merge_candidates=s2.get("merge_candidates", []),
redundancy_fields=s2.get("redundancy_fields", []),
log=log)
return {"section_key": "redundancy_fields", "data": data}
def _run_step4(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 4: 空字段扫描（连库）"""
from .step_impl.step4_empty_fields import run_step4
data = run_step4(cfg, log=log, llm=llm)
return {"section_key": "empty_fields", "data": data}
def _run_step5(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 5: 缺注释 + LLM 推测"""
from .step_impl.step5_missing_comments import run_step5
data = run_step5(dict_data=step_outputs.get("1_data_dict", {}),
llm=llm, log=log)
return {"section_key": "missing_comments", "data": data}
def _run_step6(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 6: 字段长度检查（离线）"""
from .step_impl.step6_length_check import run_step6
data = run_step6(dict_data=step_outputs.get("1_data_dict", {}),
llm=llm, log=log)
return {"section_key": "length_issues", "data": data}
def _run_step7(cfg, step_outputs, llm, findings_dir, log, cancel_event):
"""Step 7: 国标校验（连库）"""
from .step_impl.step7_standards import run_step7
data = run_step7(cfg, dict_data=step_outputs.get("1_data_dict", {}), log=log)
return {"section_key": "standard_violations", "data": data}
# ── 报告生成（Step 8） ──
def _generate_reports(run_dir: Path, sections: dict, overview: dict,
log: Callable | None = None) -> tuple[str | None, str | None]:
"""生成 Markdown + Word 报告"""
from .step_impl.step8_report import generate_reports
return generate_reports(run_dir, sections, overview, log=log)
# ── Overview 构造 ──
def _build_overview(sections: dict, cfg: DBConfig) -> dict:
"""从各 section 汇总概览数据"""
overview_meta = sections.get("overview_meta", {})
empty = sections.get("empty_fields", {}).get("summary", {})
missing = sections.get("missing_comments", {}).get("summary", {})
length = sections.get("length_issues", {}).get("summary", {})
standards = sections.get("standard_violations", {}).get("summary", {})
merge = sections.get("merge_candidates", {})
return {
"database": cfg.database,
"host": cfg.host,
"db_type": cfg.db_type,
"total_tables": overview_meta.get("total_tables", 0),
"total_fields": overview_meta.get("total_fields", 0),
"high_empty_fields": empty.get("high_empty_fields_count", 0),
"mid_empty_fields": empty.get("mid_empty_fields_count", 0),
"missing_comments": missing.get("total_missing_comments", 0),
"length_issues": length.get("total_issues", 0),
"standard_violations": standards.get("violations", 0),
"standards_applied": standards.get("standards_applied", 0),
"merge_candidates": len(merge.get("merge_candidates", [])) if isinstance(merge, dict) else 0,
"redundancy_fields": len(merge.get("redundancy_fields", [])) if isinstance(merge, dict) else 0,
"executed_at": datetime.now().isoformat(),
}
\ No newline at end of file
web/core/step_impl/__init__.py 0 → 100644
View file @ 7b4bd4a8
"""Step 实现包 — Web 版（兼容 MySQL + 达梦，接 LLM）
每个 step_* 函数接收通用 cfg + 上游 step_outputs，返回结构化 dict。
所有 Step 通过 web.core.orchestrator.run_governance_workflow() 调度。
"""
\ No newline at end of file
web/core/step_impl/step1_data_dict.py 0 → 100644
View file @ 7b4bd4a8
"""Step 1: 获取数据字典（MySQL + 达梦 适配版）
通过 db_adapter 统一接口获取：
- data_dictionary: 全部字段元数据
- table_summary: 表级汇总
- overview_meta: 概览统计
"""
from __future__ import annotations
import logging
from typing import Callable
from ..db_adapter import DBConfig, open_db
logger = logging.getLogger(__name__)
def run_step1(cfg: DBConfig, log: Callable | None = None) -> dict:
"""执行 Step 1，返回 data_dictionary + table_summary + overview_meta"""
if log:
log("INFO", f"连接数据库: {cfg.db_type}://{cfg.user}@{cfg.host}:{cfg.port}/{cfg.database}", step="1")
with open_db(cfg) as db:
if log:
log("INFO", "已建立连接，开始读取 information_schema.COLUMNS ...", step="1")
columns = db.list_columns(cfg.database)
if log:
log("INFO", f" · COLUMNS: {len(columns)} 行", step="1")
if log:
log("INFO", "读取 information_schema.TABLES ...", step="1")
tables = db.list_tables(cfg.database)
if log:
log("INFO", f" · TABLES: {len(tables)} 行", step="1")
# 类型规整
fixed_int = 0
for r in columns:
for k in ("char_max_length", "numeric_precision", "numeric_scale", "ordinal_position"):
if r.get(k) is not None:
try:
r[k] = int(r[k])
fixed_int += 1
except (TypeError, ValueError):
pass
if log and fixed_int:
log("DEBUG", f" · 字段类型规整: 转换 {fixed_int} 个数值字段", step="1")
for r in tables:
for tf in ("create_time", "update_time"):
if r.get(tf) is not None and hasattr(r[tf], "strftime"):
r[tf] = r[tf].strftime("%Y-%m-%d %H:%M:%S")
for nf in ("table_rows", "data_length", "index_length"):
if r.get(nf) is not None:
try:
r[nf] = int(r[nf])
except (TypeError, ValueError):
pass
# 概览
table_names = {r["table_name"] for r in columns}
if log:
log("INFO", f"获取完成: {len(table_names)} 张表, {len(columns)} 个字段（覆盖 {len(tables)} 条表信息）", step="1")
overview_meta = {
"database": cfg.database,
"host": cfg.host,
"db_type": cfg.db_type,
"total_tables": len(table_names),
"total_fields": len(columns),
}
return {
"data_dictionary": columns,
"table_summary": tables,
"overview_meta": overview_meta,
}
\ No newline at end of file
web/core/step_impl/step2_merge_redundancy.py 0 → 100644
View file @ 7b4bd4a8
"""Step 2: 表合并候选与冗余字段分析（离线 + LLM 增强）
基于 Step 1 产出，计算：
- merge_candidates: 结构高度相似的表组
- decommission_candidates: 0 行表 + 可能的新表替代
- redundancy_fields: 高频出现的疑似冗余字段
- 每个 merge_candidate 的 suggestion 由 LLM 优化（可选）
"""
from __future__ import annotations
import logging
from collections import Counter, defaultdict
from typing import Callable
from ..llm import LLMClient
logger = logging.getLogger(__name__)
COMMON_FIELDS = {
"id", "create_time", "update_time", "create_by", "update_by",
"remark", "del_flag", "tenant_id", "dept_id", "status",
"sort_order", "version",
}
NEW_PREFIXES = ["t_talent_", "t_ai_", "t_design_"]
OLD_PREFIXES = ["t_", "mall_", "project_"]
def run_step2(dict_data: dict, llm: LLMClient | None = None,
log: Callable | None = None) -> dict:
columns = dict_data.get("data_dictionary", [])
table_summary = dict_data.get("table_summary", [])
if not columns:
if log:
log("WARN", "未获取到任何字段元数据，跳过 Step 2", step="2")
return {"merge_candidates": [], "decommission_candidates": {}, "redundancy_fields": []}
by_table: dict[str, list[dict]] = defaultdict(list)
for row in columns:
by_table[row["table_name"]].append(row)
if log:
log("INFO", f"待分析表数: {len(by_table)}, 字段数: {len(columns)}", step="2")
# 1. 合并候选
if log:
log("INFO", "[1/3] 计算合并候选（相似度阈值 ≥95%）...", step="2")
merge = _find_merge(by_table, llm, log)
if log:
log("INFO", f" · 发现 {len(merge)} 组可合并表", step="2")
# 2. 废弃候选
if log:
log("INFO", "[2/3] 计算废弃候选（0 行表）...", step="2")
rows_by_table = {t["table_name"]: t.get("table_rows", 0) for t in table_summary}
decom = _find_decommission(rows_by_table, by_table.keys())
if log:
log("INFO", f" · 强废弃（有新版替代）{decom['strong']['count']} 张, 弱废弃（无明确替代）{decom['weak']['count']} 张", step="2")
# 3. 高频字段
if log:
log("INFO", "[3/3] 计算高频字段（出现 ≥10 张表）...", step="2")
redundancy = _find_redundancy(by_table)
if log:
log("INFO", f" · 高频字段 {len(redundancy)} 个", step="2")
if log:
log("INFO",
f"合并候选 {len(merge)} 组, 废弃候选 {decom['strong']['count'] + decom['weak']['count']} 张, "
f"高频字段 {len(redundancy)} 个",
step="2")
return {
"merge_candidates": merge,
"decommission_candidates": decom,
"redundancy_fields": redundancy,
}
def _table_columns(cols: list[dict]) -> set[str]:
return {c["column_name"] for c in cols}
def _similarity(a: set[str], b: set[str]) -> float:
if not a or not b:
return 0.0
return len(a & b) / len(a | b)
def _find_merge(by_table: dict, llm: LLMClient | None, log: Callable | None) -> list[dict]:
"""找出结构高度相似的表组（相似度 ≥95%）"""
names = sorted(by_table.keys())
visited = set()
groups = []
for i, t in enumerate(names):
if t in visited:
continue
ci = _table_columns(by_table[t])
cluster = [t]
for j in range(i + 1, len(names)):
t2 = names[j]
if t2 in visited:
continue
cj = _table_columns(by_table[t2])
if _similarity(ci, cj) >= 0.95:
cluster.append(t2)
if len(cluster) >= 2:
visited.update(cluster)
groups.append(cluster)
results = []
for idx, group in enumerate(groups, 1):
ref = _table_columns(by_table[group[0]])
diff = _table_columns(by_table[group[1]]) - ref if len(group) > 1 else set()
# 基础建议（模板化）
suggestion = _template_suggestion(group)
risk = ""
llm_verdict = None
# LLM 增强（如启用）
if llm and llm.available:
try:
if log:
log("INFO", f"调用 LLM 评估合并候选 MERGE-{idx:03d}", step="2")
llm_result = llm.suggest_table_merge(
tables=[
{"name": t, "comment": by_table[t][0].get("table_comment", ""),
"row_count": "?"}
for t in group
],
similarity=0.95,
common_columns=sorted(ref),
diff_columns=sorted(diff),
)
if llm_result:
llm_verdict = llm_result.get("verdict")
if llm_result.get("suggestion"):
suggestion = llm_result["suggestion"]
if llm_result.get("risks"):
risk = "; ".join(llm_result["risks"])
except Exception as e:
if log:
log("WARN", f"LLM 评估失败，使用规则建议: {e}", step="2")
results.append({
"id": f"MERGE-{idx:03d}",
"tables": group,
"common_columns": len(ref),
"similarity": "≥95%",
"similarity_value": 0.95,
"columns": sorted(ref),
"diff_columns": sorted(diff),
"suggestion": suggestion,
"risk": risk,
"verdict": llm_verdict or "merge",
"priority": "P1" if len(group) >= 3 else "P2",
})
return results
def _template_suggestion(group: list[str]) -> str:
if not group:
return "合并候选"
if len(group) == 2:
return f"合并为 {group[0]}，新增 type 字段区分"
if len(group) <= 5:
return f"合并为 {group[0]}，新增 record_type 字段区分"
return f"合并 {len(group)} 张同类表"
def _find_decommission(rows_by_table: dict, all_tables) -> dict:
zero_rows = [t for t, n in rows_by_table.items() if n == 0 and t in all_tables]
strong, weak = [], []
for t in sorted(zero_rows):
replacement = _find_replacement(t, all_tables)
entry = {"table": t, "rows": 0, "replacement": replacement}
(strong if replacement else weak).append(entry)
return {
"strong": {"description": "有新版表替代", "tables": strong, "count": len(strong)},
"weak": {"description": "无明确替代", "tables": weak, "count": len(weak)},
}
def _find_replacement(table: str, all_tables) -> str | None:
for prefix in OLD_PREFIXES:
if table.startswith(prefix):
stem = table[len(prefix):]
for new_prefix in NEW_PREFIXES:
cand = new_prefix + stem
if cand in all_tables and cand != table:
return cand
return None
def _find_redundancy(by_table: dict) -> list[dict]:
counter: Counter = Counter()
for cols in by_table.values():
for c in cols:
counter[c["column_name"]] += 1
logger.debug(f"字段频次统计: 共 {len(counter)} 个不同字段名")
return [
{
"field": f,
"table_count": n,
"risk": "高频出现，需评估是否为业务必要字段" if n >= 30 else "中频出现",
"suggestion": "建议评审是否需要统一到公共字典表" if n >= 30 else "建议评审",
}
for f, n in counter.most_common()
if n >= 10 and f not in COMMON_FIELDS
]
\ No newline at end of file
web/core/step_impl/step3_verify.py 0 → 100644
View file @ 7b4bd4a8
"""Step 3: 数据验证（连库）
连库验证：
- merge_candidates: 对每组前 5 张表跑 SELECT COUNT(*)
- redundancy_fields: 取高频字段，看实际出现在哪些表
- xzqh_orphan_check: 行政区划编码孤儿（如果存在字典表 c_bri_xzqh）
- data_quality_issues: 数据质量问题清单
所有 SQL 走 web/sql/ 模板，通过 db_adapter 调用。
"""
from __future__ import annotations
import logging
from typing import Callable
from ..db_adapter import DBConfig, open_db, quote_ident
from web.sql.loader import get_sql_loader
logger = logging.getLogger(__name__)
def run_step3(cfg: DBConfig, dict_data: dict,
merge_candidates: list, redundancy_fields: list,
log: Callable | None = None) -> dict:
confirmed_merges = []
redundancy_verified = []
xzqh_check = {"orphan_total": 0, "per_table": []}
quality_issues = []
if log:
log("INFO", f"[1/4] 验证合并候选（top {min(5, len(merge_candidates))}）...", step="3")
try:
with open_db(cfg) as db:
loader = get_sql_loader()
# 1. 验证 merge 候选（实际行数）
for idx, mc in enumerate(merge_candidates[:5], 1):
row_counts = []
total = 0
for t in mc.get("tables", []):
try:
sql = loader.render(
"verify/count_table_rows",
dialect=cfg.db_type,
table=t, # 模板内 ${table | quote} 自动加引号
)
n = db.fetch_scalar(sql) or 0
row_counts.append({"table": t, "rows": int(n)})
total += int(n)
except Exception as e:
row_counts.append({"table": t, "error": str(e)})
if log:
log("WARN", f" · 表 {t} 行数统计失败: {e}", step="3")
confirmed_merges.append({
**mc,
"row_counts": row_counts,
"total_rows": total,
})
if log:
log("INFO",
f" · {mc.get('id')} ({idx}/{min(5, len(merge_candidates))}): "
f"{len(mc.get('tables', []))} 张表, 共 {total} 行",
step="3")
# 2. 验证冗余字段（出现表数 + 实际行数）
if log:
log("INFO", f"[2/4] 验证冗余字段（top {min(5, len(redundancy_fields))}）...", step="3")
for rf in redundancy_fields[:5]:
field_name = rf.get("field")
tables = _find_field_in_tables(dict_data.get("data_dictionary", []), field_name)
redundancy_verified.append({
"field": field_name,
"occurrences": len(tables),
"tables_with_field": tables,
})
if log:
log("INFO", f" · 字段 {field_name} 出现在 {len(tables)} 张表", step="3")
# 3. 行政区划孤儿编码检查
if log:
log("INFO", "[3/4] 行政区划孤儿编码检查...", step="3")
xzqh_check = _check_xzqh_orphans(cfg, db, log)
# 4. 数据质量问题
if log:
log("INFO", "[4/4] 数据质量问题检查（占位）...", step="3")
quality_issues = []
except Exception as e:
if log:
log("ERROR", f"Step 3 连库验证失败: {type(e).__name__}: {e}", step="3")
logger.exception("Step 3 详细异常")
if log:
log("INFO",
f"Step 3 完成: 验证合并 {len(confirmed_merges)} 组, "
f"冗余字段 {len(redundancy_verified)} 个, 行政区划孤儿 {xzqh_check.get('orphan_total', 0)} 条",
step="3")
return {
"confirmed_merge_candidates": confirmed_merges,
"redundancy_fields": redundancy_verified,
"xzqh_orphan_check": xzqh_check,
"data_quality_issues": quality_issues,
}
def _find_field_in_tables(columns: list, field_name: str) -> list[str]:
return sorted({r["table_name"] for r in columns if r["column_name"] == field_name})
def _check_xzqh_orphans(cfg: DBConfig, db, log: Callable | None) -> dict:
"""检查行政区划字段是否存在孤儿编码（不在字典表 c_bri_xzqh 中）"""
DICT_TABLE = "c_bri_xzqh"
result = {"dict_table_rows": 0, "orphan_total": 0, "per_table": []}
loader = get_sql_loader()
# 检查字典表是否存在 + 行数
try:
sql = loader.render(
"xzqh/check_dict_table_exists",
dialect=cfg.db_type,
dict_table=DICT_TABLE, # 模板内 ${dict_table | quote} 自动加引号
)
result["dict_table_rows"] = int(db.fetch_scalar(sql) or 0)
except Exception:
if log:
log("INFO", f"字典表 {DICT_TABLE} 不存在，跳过行政区划校验", step="3")
return result
if result["dict_table_rows"] == 0:
return result
if log:
log("INFO", f"字典表 {DICT_TABLE} 存在，共 {result['dict_table_rows']} 行", step="3")
# 这里可扩展：对含 xzqhbm 字段的前 5 张大表跑 LEFT JOIN 找孤儿
# 模板：xzqh/find_xzqh_orphans.sql
return result
\ No newline at end of file
web/core/step_impl/step4_empty_fields.py 0 → 100644
View file @ 7b4bd4a8
"""Step 4: 大范围空字段扫描（连库 + LLM 增强）
对每张表跑动态 SQL 统计：
- COUNT(*) 总行数
- 各列的 NULL/空字符串比例
阈值：
- 空值率 ≥80% → "high" 高空
- 空值率 ≥50% → "mid" 中空
LLM 增强（可选）：对 top 20 高空字段分类「业务不需要 / 采集漏了 / 预留字段」。
SQL 全部走 web/sql/empty_fields/ 模板，Python 端只构造列子句（empty_clauses）。
"""
from __future__ import annotations
import logging
from collections import defaultdict
from typing import Callable
from ..db_adapter import DBConfig, open_db, quote_ident, TEXT_TYPES
from ..llm import LLMClient
from web.sql.loader import get_sql_loader
logger = logging.getLogger(__name__)
# 字段名 → 业务关键性提示（帮助 LLM 更准确分类）
CRITICAL_FIELDS = {
"id", "name", "code", "type", "status", "user_id", "create_time", "update_time",
"amount", "price", "total", "count", "project_id", "user_name",
}
def run_step4(cfg: DBConfig, log: Callable | None = None,
llm: LLMClient | None = None,
min_rows_to_check: int = 5,
high_threshold: float = 0.80,
mid_threshold: float = 0.50,
sample_size: int = 3) -> dict:
"""扫描所有表的空字段情况"""
if log:
log("INFO",
f"Step 4 开始扫描空字段 (high≥{high_threshold:.0%}, mid≥{mid_threshold:.0%}, min_rows={min_rows_to_check})",
step="4")
# Step 1 产出已在 step_outputs 里，通过 cfg 重新拉一遍元数据
from .step1_data_dict import run_step1
dict_data = run_step1(cfg, log=lambda lvl, msg: log(lvl, msg, step="4") if log else None)
columns = dict_data["data_dictionary"]
table_summary = dict_data["table_summary"]
by_table: dict[str, list[dict]] = defaultdict(list)
for row in columns:
by_table[row["table_name"]].append(row)
loader = get_sql_loader()
high_count = 0
mid_count = 0
tables_with_issues = set()
per_table: list[dict] = []
high_fields_all: list[dict] = []
skipped = []
top_tables: list[dict] = []
table_names = sorted(by_table.keys())
total = len(table_names)
if log:
log("INFO", f"将扫描 {total} 张表（已过滤 TEXT_TYPES）", step="4")
try:
with open_db(cfg) as db:
for idx, table in enumerate(table_names, 1):
if log and idx % 10 == 0:
log("INFO", f" 进度 {idx}/{total} - 当前表: {table}", step="4")
elif log and idx == 1:
log("INFO", f" 进度 1/{total} - 当前表: {table}", step="4")
# 拿实际行数（避免 TABLE_ROWS 估算不准）
try:
sql_count = loader.render(
"verify/count_table_rows",
dialect=cfg.db_type,
table=table, # 模板内 ${table | quote} 自动加引号
)
row_count = int(db.fetch_scalar(sql_count) or 0)
except Exception as e:
skipped.append({"table": table, "reason": f"统计行数失败: {e}"})
if log:
log("WARN", f" · {table} 行数统计失败: {e}", step="4")
continue
if row_count < min_rows_to_check:
skipped.append({"table": table, "reason": f"行数 {row_count} < {min_rows_to_check}", "rows": row_count})
logger.debug(f"跳过小表 {table}（行数 {row_count}）")
continue
# 构造动态列子句（Python 只构造这一段，SQL 模板负责拼接）
empty_clauses, cols = _build_empty_clauses(by_table[table], cfg.db_type)
if not cols:
skipped.append({"table": table, "reason": "无可扫描的字符串字段"})
continue
try:
sql = loader.render(
"empty_fields/count_with_nulls",
dialect=cfg.db_type,
empty_clauses=empty_clauses,
table=table, # 模板内 ${table | quote} 自动加引号
)
row = db.fetchone(sql)
except Exception as e:
skipped.append({"table": table, "reason": f"扫描失败: {e}"})
if log:
log("WARN", f" · {table} 空字段扫描失败: {e}", step="4")
continue
if not row:
continue
# 解析每个字段的空值率
high_in_table = []
mid_in_table = []
for col in cols:
empty_count = int(row.get(f"empty_{col['alias']}", 0) or 0)
empty_rate = empty_count / row_count if row_count else 0
field_record = {
"table_name": table,
"table_comment": col.get("table_comment", ""),
"column_name": col["name"],
"data_type": col["data_type"],
"column_type": col["column_type"],
"column_comment": col.get("comment", ""),
"total_rows": row_count,
"empty_count": empty_count,
"empty_rate": round(empty_rate, 4),
"level": "",
}
if empty_rate >= high_threshold:
field_record["level"] = "high"
high_count += 1
high_in_table.append(field_record)
high_fields_all.append(field_record)
elif empty_rate >= mid_threshold:
field_record["level"] = "mid"
mid_count += 1
mid_in_table.append(field_record)
if high_in_table or mid_in_table:
tables_with_issues.add(table)
per_table.append({
"table_name": table,
"table_comment": by_table[table][0].get("table_comment", ""),
"total_rows": row_count,
"high_empty_fields": high_in_table,
"mid_empty_fields": mid_in_table,
"high_count": len(high_in_table),
"mid_count": len(mid_in_table),
})
if len(high_in_table) > 0:
top_tables.append({
"table_name": table,
"high_empty_field_count": len(high_in_table),
})
logger.debug(f"表 {table}: 高空 {len(high_in_table)}, 中空 {len(mid_in_table)}")
except Exception as e:
if log:
log("ERROR", f"Step 4 数据库扫描失败: {type(e).__name__}: {e}", step="4")
logger.exception("Step 4 详细异常")
top_tables.sort(key=lambda x: x["high_empty_field_count"], reverse=True)
summary = {
"total_tables_scanned": total,
"total_fields_checked": sum(len(c) for c in by_table.values()),
"high_empty_fields_count": high_count,
"mid_empty_fields_count": mid_count,
"tables_with_issues_count": len(tables_with_issues),
"skipped_tables_count": len(skipped),
"thresholds": {"high": high_threshold, "mid": mid_threshold},
"top_tables_by_high_empty_fields": top_tables[:10],
"high_empty_fields": high_fields_all[:200],
}
if log:
log("INFO",
f"扫描完成: 高空 {high_count}, 中空 {mid_count}, 涉及 {len(tables_with_issues)} 张表, "
f"跳过 {len(skipped)} 张",
step="4")
# LLM 增强：对 top 20 高空字段分类
if llm and llm.available and high_fields_all:
if log:
log("INFO", f"调用 LLM 分类 top {min(20, len(high_fields_all))} 个高空字段成因", step="4")
classified = []
for idx, fld in enumerate(high_fields_all[:20], 1):
try:
# 抽样非空样本（这里简化为空，仅展示结构）
r = llm.classify_empty_field(
table_name=fld["table_name"],
table_comment=fld["table_comment"],
column_name=fld["column_name"],
column_comment=fld["column_comment"],
data_type=fld["data_type"],
empty_rate=fld["empty_rate"],
sample_values=[],
)
if r:
fld["llm_classification"] = r.get("classification", "unknown")
fld["llm_reasoning"] = r.get("reasoning", "")
fld["llm_action"] = r.get("action", "")
classified.append(fld)
if log:
log("DEBUG",
f" · [{idx}/{min(20, len(high_fields_all))}] "
f"{fld['table_name']}.{fld['column_name']} → {r.get('classification')}",
step="4")
except Exception as e:
if log:
log("WARN", f"LLM 分类失败 ({fld['table_name']}.{fld['column_name']}): {e}", step="4")
summary["llm_classified_count"] = len(classified)
summary["llm_classified_sample"] = classified[:10]
if log:
log("INFO", f"LLM 分类成功 {len(classified)}/{min(20, len(high_fields_all))}", step="4")
return {
"summary": summary,
"per_table": per_table,
"skipped": skipped,
}
def _build_empty_clauses(cols: list[dict], db_type: str) -> tuple[str, list[dict]]:
"""构造 SUM(CASE WHEN ...) 子句列表（不含固定 SQL 部分）
返回:
empty_clauses: 逗号分隔的子句，会被注入 empty_fields/count_with_nulls.sql 的 ${empty_clauses}
valid_cols: 每个子句对应的列元数据（用于解析结果）
"""
parts = []
valid_cols = []
for i, c in enumerate(cols):
dt = (c.get("data_type") or "").lower()
if dt not in TEXT_TYPES:
continue
alias = f"c{i}"
c["alias"] = alias
ident = quote_ident(c["column_name"], db_type)
parts.append(
f"SUM(CASE WHEN {ident} IS NULL OR TRIM({ident}) = '' THEN 1 ELSE 0 END) AS empty_{alias}"
)
valid_cols.append(c)
return ", ".join(parts), valid_cols
\ No newline at end of file
web/core/step_impl/step5_missing_comments.py 0 → 100644
View file @ 7b4bd4a8
"""Step 5: 缺失注释字段检查 + LLM 推测
优先用 LLM 推测无注释字段的语义（替换原硬编码 COMMENT_HINTS）；
LLM 不可用时降级为本地规则。
产出：
- summary: 总数 + 推测命中率
- by_table: 每表缺失注释数量
- predicted_comments: 推测出的注释（含置信度）
- unpredictable_sample: 未推测到的样本
"""
from __future__ import annotations
import logging
from collections import defaultdict
from typing import Callable
from ..llm import LLMClient
logger = logging.getLogger(__name__)
# 兜底规则（LLM 不可用时使用）
FALLBACK_HINTS = {
"id": "主键ID",
"create_time": "创建时间",
"update_time": "更新时间",
"create_by": "创建人",
"update_by": "更新人",
"remark": "备注",
"del_flag": "删除标记",
"tenant_id": "租户ID",
"dept_id": "部门ID",
"project_id": "项目ID",
"project_name": "项目名称",
"project_code": "项目编号",
"site_id": "工地ID",
"site_name": "工地名称",
"status": "状态",
"sort_order": "排序",
"start_time": "开始时间",
"end_time": "结束时间",
"type": "类型",
"code": "编码",
}
def run_step5(dict_data: dict, llm: LLMClient | None = None,
log: Callable | None = None) -> dict:
columns = dict_data.get("data_dictionary", [])
if not columns:
if log:
log("WARN", "未获取到任何字段元数据，跳过 Step 5", step="5")
return {"summary": {}, "by_table": [], "predicted_comments": [], "unpredictable_sample": []}
if log:
log("INFO", f"待检查字段总数: {len(columns)}", step="5")
missing = []
by_table: dict[str, int] = defaultdict(int)
predicted = []
unpredictable = []
# 先用规则快速批匹配
for r in columns:
comment = (r.get("column_comment") or "").strip()
if comment:
continue
by_table[r["table_name"]] += 1
fallback = FALLBACK_HINTS.get(r["column_name"])
entry = {
"table_name": r["table_name"],
"table_comment": r.get("table_comment", ""),
"column_name": r["column_name"],
"data_type": r.get("data_type", ""),
"column_type": r.get("column_type", ""),
"is_nullable": r.get("is_nullable", ""),
"predicted": None,
"confidence": "low",
"reason": "",
}
if fallback:
entry["predicted"] = fallback
entry["confidence"] = "high"
entry["reason"] = "字段名匹配内置规则"
predicted.append(entry)
else:
unpredictable.append(entry)
missing.append(entry)
if log:
log("INFO",
f" · 缺失注释字段: {len(missing)} 个 (覆盖 {len(by_table)} 张表)",
step="5")
log("INFO",
f" · 内置规则命中: {len(predicted)}, 需 LLM 推测: {len(unpredictable)}",
step="5")
# 用 LLM 处理 unpredictable（如果可用）
if llm and llm.available and unpredictable:
if log:
log("INFO", f"[LLM] 调用 LLM 推测 {len(unpredictable)} 个无规则命中的字段注释", step="5")
llm_predicted = []
still_unknown = []
for idx, entry in enumerate(unpredictable, 1):
try:
r = llm.predict_field_comment(
table_name=entry["table_name"],
table_comment=entry["table_comment"],
column_name=entry["column_name"],
data_type=entry["data_type"],
)
if r and r.get("comment"):
entry["predicted"] = r["comment"]
entry["confidence"] = r.get("confidence", "low")
entry["reason"] = "LLM 推测"
llm_predicted.append(entry)
# 从 predicted 列表的视角也算推测成功
predicted.append(entry)
if log and idx % 5 == 0:
log("DEBUG",
f" · LLM 推测进度 {idx}/{len(unpredictable)} "
f"(已成功 {len(llm_predicted)})",
step="5")
else:
still_unknown.append(entry)
except Exception as e:
if log:
log("WARN",
f"LLM 推测失败 ({entry['table_name']}.{entry['column_name']}): {e}",
step="5")
still_unknown.append(entry)
if log:
log("INFO", f"[LLM] 推测成功 {len(llm_predicted)} / {len(unpredictable)}", step="5")
unpredictable = still_unknown
# 按表聚合
by_table_list = sorted(
[{"table_name": t, "missing_count": c} for t, c in by_table.items()],
key=lambda x: x["missing_count"], reverse=True
)[:20]
if log:
log("INFO",
f"缺失注释 {len(missing)} 个字段，覆盖 {len(by_table)} 张表；"
f"推测成功 {len(predicted)}（规则 {sum(1 for p in predicted if p.get('reason') == '字段名匹配内置规则')}, "
f"LLM {sum(1 for p in predicted if p.get('reason') == 'LLM 推测')}）",
step="5")
return {
"summary": {
"total_missing_comments": len(missing),
"tables_affected": len(by_table),
"predicted_count": len(predicted),
"predicted_by_rules": sum(1 for p in predicted if p.get("reason") == "字段名匹配内置规则"),
"predicted_by_llm": sum(1 for p in predicted if p.get("reason") == "LLM 推测"),
"unpredicted_count": len(unpredictable),
},
"by_table": by_table_list,
"predicted_comments": predicted[:50],
"unpredictable_sample": unpredictable[:50],
}
\ No newline at end of file
web/core/step_impl/step6_length_check.py 0 → 100644
View file @ 7b4bd4a8
"""Step 6: 字段长度检查（离线 + LLM 增强）
识别字段定义长度超出标准所需（如身份证号 18 位用 varchar(50) 存储）。
基础规则覆盖常见字段；自定义字段由 LLM 推断建议长度。
产出：
- summary: 总异常数
- issues: 每条具体异常（含浪费字节估算）
"""
from __future__ import annotations
import logging
import re
from typing import Callable
from ..llm import LLMClient
logger = logging.getLogger(__name__)
# 字段名模式 → 期望长度的规则
# 注意：期望长度需严格匹配国家标准
LENGTH_RULES: list[tuple[re.Pattern, int, str, str]] = [
# (pattern, expected_length, standard, description)
(re.compile(r"^(id_?card|id_?number|identity_?card)$", re.I), 18, "GB 11643-1999", "身份证号 18 位"),
(re.compile(r"^(id_?card_?no|_?sfz_?hm)$", re.I), 18, "GB 11643-1999", "身份证号 18 位"),
(re.compile(r"^(uscc|credit_?code|social_?credit_?code)$", re.I), 18, "GB 32100-2015", "统一社会信用代码 18 位"),
(re.compile(r"^mobile(_?phone)?$", re.I), 11, "YD/T 1313", "手机号 11 位"),
(re.compile(r"^phone(_?no)?$", re.I), 11, "YD/T 1313", "手机号 11 位"),
(re.compile(r"^tel(ephone)?$", re.I), 11, "YD/T 1313", "手机号 11 位"),
(re.compile(r"^(xzqhbm|adcode|district_?code)$", re.I), 6, "GB/T 2260", "行政区划代码 6 位"),
(re.compile(r"^province_?code$", re.I), 2, "GB/T 2260", "省级代码 2 位"),
(re.compile(r"^city_?code$", re.I), 4, "GB/T 2260", "市级代码 4 位"),
(re.compile(r"^region_?code$", re.I), 6, "GB/T 2260", "区县级代码 6 位"),
(re.compile(r"^zip_?code$", re.I), 6, "GB/T 23703", "邮政编码 6 位"),
(re.compile(r"^email$", re.I), 50, "RFC 5321", "电子邮件 ≤254 位，常用 ≤50"),
(re.compile(r"^bank_?card(_?no)?$", re.I), 19, "JR/T 0002", "银行卡号 ≤19 位"),
(re.compile(r"^post_?code$", re.I), 6, "GB/T 23703", "邮政编码 6 位"),
]
def run_step6(dict_data: dict, llm: LLMClient | None = None,
log: Callable | None = None) -> dict:
columns = dict_data.get("data_dictionary", [])
if not columns:
if log:
log("WARN", "未获取到任何字段元数据，跳过 Step 6", step="6")
return {"summary": {"total_issues": 0}, "issues": []}
if log:
log("INFO",
f"内置长度规则: {len(LENGTH_RULES)} 条, 待扫描字段: {len(columns)}",
step="6")
issues = []
rule_match_count = 0
for r in columns:
col_name = r["column_name"]
dt = (r.get("data_type") or "").lower()
max_len = r.get("char_max_length")
if dt not in ("varchar", "char") or not max_len:
continue
for pat, expected, std, desc in LENGTH_RULES:
if pat.match(col_name):
rule_match_count += 1
if max_len > expected:
issues.append({
"table_name": r["table_name"],
"table_comment": r.get("table_comment", ""),
"column_name": col_name,
"data_type": dt,
"column_type": r.get("column_type", ""),
"actual_length": max_len,
"required_length": expected,
"wasted_bytes_per_row": max_len - expected,
"standard": std,
"rule": desc,
"issue": f"定义 {max_len} 位超出标准 {expected} 位",
"suggestion": f"改为 VARCHAR({expected})",
"column_comment": r.get("column_comment", ""),
})
break
if log:
log("INFO",
f"[1/2] 规则匹配 {rule_match_count} 次，发现异常 {len(issues)} 条",
step="6")
# 自定义字段（如命名不符合常见模式但 varchar 过大） — 简单启发式
# 仅对没有匹配规则且长度 ≥100 的字段提醒
custom_count = 0
for r in columns:
col_name = r["column_name"]
dt = (r.get("data_type") or "").lower()
max_len = r.get("char_max_length")
if dt not in ("varchar",) or not max_len or max_len < 100:
continue
if any(pat.match(col_name) for pat, *_ in LENGTH_RULES):
continue
if any(i["table_name"] == r["table_name"] and i["column_name"] == col_name for i in issues):
continue
# 大字段且无规则命中 → 候选给 LLM
issues.append({
"table_name": r["table_name"],
"table_comment": r.get("table_comment", ""),
"column_name": col_name,
"data_type": dt,
"column_type": r.get("column_type", ""),
"actual_length": max_len,
"required_length": None,
"wasted_bytes_per_row": None,
"standard": "（自定义）",
"rule": "未匹配任何标准规则",
"issue": f"字段长度 {max_len} 较大，未匹配任何标准",
"suggestion": "需业务确认或 LLM 推断",
"column_comment": r.get("column_comment", ""),
"needs_llm": True,
})
custom_count += 1
if log:
log("INFO",
f"[2/2] 自定义大字段 {custom_count} 条（候选给 LLM）",
step="6")
# LLM 处理 needs_llm 标记的字段（前 20 个）
if llm and llm.available:
to_llm = [i for i in issues if i.get("needs_llm")][:20]
if to_llm and log:
log("INFO", f"[LLM] 调用 LLM 推断 {len(to_llm)} 个自定义字段的合理长度", step="6")
llm_ok = 0
for idx, issue in enumerate(to_llm, 1):
try:
r = llm.suggest_length_rule(
column_name=issue["column_name"],
column_comment=issue["column_comment"],
current_length=issue["actual_length"],
table_name=issue["table_name"],
)
if r:
rec = r.get("recommended")
if rec and rec < issue["actual_length"]:
issue["required_length"] = rec
issue["suggestion"] = f"建议改为 VARCHAR({rec})（{r.get('reasoning', '')})"
issue["llm_recommended"] = rec
issue["llm_reasoning"] = r.get("reasoning", "")
issue["wasted_bytes_per_row"] = issue["actual_length"] - rec
llm_ok += 1
if log:
log("DEBUG",
f" · [{idx}/{len(to_llm)}] "
f"{issue['table_name']}.{issue['column_name']}: "
f"{issue['actual_length']} → {rec}",
step="6")
except Exception as e:
if log:
log("WARN",
f"LLM 长度推断失败 ({issue['table_name']}.{issue['column_name']}): {e}",
step="6")
if to_llm and log:
log("INFO", f"[LLM] 推断完成: 采纳 {llm_ok}/{len(to_llm)}", step="6")
if log:
log("INFO",
f"字段长度异常 {len(issues)} 条（其中 {custom_count} 条自定义大字段）",
step="6")
return {
"summary": {
"total_issues": len(issues),
"rules_applied": len(LENGTH_RULES),
"wasted_bytes_total": sum(i.get("wasted_bytes_per_row") or 0 for i in issues),
},
"issues": issues[:200],
}
\ No newline at end of file
web/core/step_impl/step7_standards.py 0 → 100644
View file @ 7b4bd4a8
"""Step 7: 国家标准字段校验（连库）
按字段名匹配 standards 插件，抽样校验实际数据：
- 身份证号（GB 11643）
- 统一社会信用代码（GB 32100）
- 手机号（YD/T 1313）
- 行政区划代码（GB/T 2260）
抽样 SQL 走 web/sql/standards/sample_field_values.sql 模板。
"""
from __future__ import annotations
import logging
from typing import Callable
from ..db_adapter import DBConfig, open_db, quote_ident
from standards.registry import discover_standards, list_all_standards
from standards.base import BaseStandard, ValidationResult
from web.sql.loader import get_sql_loader
logger = logging.getLogger(__name__)
SAMPLE_LIMIT = 500 # 每字段最多抽样
MAX_TABLES_PER_FIELD = 5 # 每字段最多检查前 N 张表
def run_step7(cfg: DBConfig, dict_data: dict, log: Callable | None = None) -> dict:
"""执行 Step 7：国标校验"""
columns = dict_data.get("data_dictionary", [])
if not columns:
return {"summary": {}, "violations_by_standard": [], "violations": []}
# 字段名 → 标准实例的映射
standards_by_field = discover_standards()
all_standards_info = list_all_standards()
# 按字段名索引列
by_name: dict[str, list[dict]] = {}
for r in columns:
by_name.setdefault(r["column_name"], []).append(r)
by_standard: dict[str, dict] = {}
total_violations = 0
total_fields_checked = 0
violations_flat: list[dict] = []
if not standards_by_field:
if log:
log("WARN", "未发现任何标准插件", step="7")
return {
"summary": {"standards_applied": 0, "fields_checked": 0, "violations": 0},
"violations_by_standard": [],
"violations": [],
}
if log:
log("INFO",
f"已加载 {len(all_standards_info)} 个标准插件, "
f"匹配字段 {len(standards_by_field)} 个, "
f"最大每字段抽样 {SAMPLE_LIMIT} 行 / 每字段最多 {MAX_TABLES_PER_FIELD} 张表",
step="7")
loader = get_sql_loader()
try:
with open_db(cfg) as db:
for std_idx, (std_field, std_instance) in enumerate(standards_by_field.items(), 1):
if std_field not in by_name:
if log:
log("DEBUG",
f" · [{std_idx}/{len(standards_by_field)}] 标准字段 {std_field} 未出现在任何表中, 跳过",
step="7")
continue
tables_with_field = by_name[std_field][:MAX_TABLES_PER_FIELD]
std_id = std_instance.standard_id
std_record = by_standard.setdefault(std_id, {
"standard": std_id,
"standard_name": std_instance.standard_name,
"fields": [],
"total_invalid": 0,
"total_sampled": 0,
})
if log:
log("INFO",
f" · [{std_idx}/{len(standards_by_field)}] "
f"{std_id} ({std_field}) → {len(tables_with_field)} 张表",
step="7")
for col_record in tables_with_field:
table = col_record["table_name"]
field_name = col_record["column_name"]
total_fields_checked += 1
try:
sql = loader.render(
"standards/sample_field_values",
dialect=cfg.db_type,
table=table, # 模板内 ${table | quote} 自动加引号
field=field_name,
limit=SAMPLE_LIMIT,
)
rows = db.fetchall(sql)
except Exception as e:
std_record["fields"].append({
"table": table,
"field": field_name,
"error": str(e),
"violations": 0,
"sampled": 0,
})
if log:
log("WARN",
f" - {table}.{field_name} 抽样失败: {e}",
step="7")
continue
sampled = len(rows)
invalid_samples = []
invalid_count = 0
for row in rows:
val = row.get("val")
if val is None:
continue
result: ValidationResult = std_instance.validate(str(val))
if not result.valid:
invalid_count += 1
if len(invalid_samples) < 10:
invalid_samples.append({
"value": str(val)[:50],
"reason": result.reason,
})
violations_flat.append({
"table_name": table,
"column_name": field_name,
"rule_type": std_id,
"standard_name": std_instance.standard_name,
"column_type": col_record.get("column_type", ""),
"value": str(val)[:50],
"error": result.reason,
})
total_violations += invalid_count
std_record["total_invalid"] += invalid_count
std_record["total_sampled"] += sampled
std_record["fields"].append({
"table": table,
"field": field_name,
"sampled": sampled,
"violations": invalid_count,
"violation_rate": round(invalid_count / sampled, 4) if sampled else 0,
"samples": invalid_samples,
})
logger.debug(
f"{table}.{field_name}: 抽样 {sampled}, 违规 {invalid_count}"
)
except Exception as e:
if log:
log("ERROR", f"Step 7 连库校验失败: {type(e).__name__}: {e}", step="7")
logger.exception("Step 7 详细异常")
by_standard_list = []
for std_id, rec in by_standard.items():
by_standard_list.append({
"standard": rec["standard"],
"standard_name": rec["standard_name"],
"fields_count": len(rec["fields"]),
"total_invalid": rec["total_invalid"],
"total_sampled": rec["total_sampled"],
"violation_rate": round(rec["total_invalid"] / rec["total_sampled"], 4) if rec["total_sampled"] else 0,
"details": rec["fields"],
})
if log:
log("INFO", f"校验完成: {len(by_standard_list)} 个标准, {total_fields_checked} 个字段, {total_violations} 条违规", step="7")
return {
"summary": {
"standards_applied": len(all_standards_info),
"standards_used": len(by_standard_list),
"fields_checked": total_fields_checked,
"violations": total_violations,
},
"violations_by_standard": by_standard_list,
"violations": violations_flat[:500], # 给前端表格用
"standards_info": all_standards_info,
}
\ No newline at end of file
web/core/step_impl/step8_report.py 0 → 100644
View file @ 7b4bd4a8
"""Step 8: 报告生成（Markdown + Word）
读取前 7 步的 sections 汇总，生成：
- {dbname}.md - Markdown 报告（可读、可编辑）
- {dbname}.docx - Word 报告（业务方用）
"""
from __future__ import annotations
import logging
from datetime import datetime
from pathlib import Path
from typing import Any, Callable
logger = logging.getLogger(__name__)
def generate_reports(run_dir: Path, sections: dict, overview: dict,
log: Callable | None = None) -> tuple[str | None, str | None]:
"""生成 Markdown + docx 报告。返回 (md_path, docx_path)。"""
reports_dir = run_dir / "reports"
reports_dir.mkdir(parents=True, exist_ok=True)
db_name = overview.get("database", "report")
md_path = reports_dir / f"数据治理报告_{db_name}.md"
docx_path = reports_dir / f"数据治理报告_{db_name}.docx"
if log:
log("INFO", f"输出目录: {reports_dir}", step="8")
log("INFO", f"[1/2] 生成 Markdown 报告 → {md_path.name}", step="8")
# 1. Markdown
md_text = _build_markdown(sections, overview)
md_path.write_text(md_text, encoding="utf-8")
logger.info(f"Markdown 报告已写入: {md_path} ({len(md_text)} 字符)")
# 2. docx
docx_result = None
try:
if log:
log("INFO", f"[2/2] 生成 Word 报告 → {docx_path.name}", step="8")
from docx import Document
from docx.shared import Pt, Cm, RGBColor
from docx.enum.text import WD_ALIGN_PARAGRAPH
doc = Document()
_render_docx(doc, sections, overview)
doc.save(str(docx_path))
docx_result = str(docx_path)
logger.info(f"Word 报告已写入: {docx_path}")
except ImportError:
logger.warning("未安装 python-docx，跳过 Word 报告生成")
if log:
log("WARN", "未安装 python-docx，跳过 Word 报告生成", step="8")
except Exception as e:
logger.exception("Word 报告生成失败")
if log:
log("ERROR", f"Word 报告生成失败: {e}", step="8")
return str(md_path), docx_result
def _build_markdown(sections: dict, overview: dict) -> str:
"""拼装完整 Markdown 报告"""
lines: list[str] = []
# 标题
lines.append(f"# 数据治理报告 — {overview.get('database', '')}")
lines.append("")
lines.append(f"> 生成时间: {overview.get('executed_at', datetime.now().isoformat())}")
lines.append(f"> 数据库类型: {overview.get('db_type', '-')}")
lines.append("")
# 概览
lines.append("## 一、概览")
lines.append("")
lines.append("| 项目 | 值 |")
lines.append("|------|-----|")
lines.append(f"| 数据库名 | {overview.get('database', '-')} |")
lines.append(f"| 数据库类型 | {overview.get('db_type', '-')} |")
lines.append(f"| 主机 | {overview.get('host', '-')} |")
lines.append(f"| 表总数 | {overview.get('total_tables', 0)} |")
lines.append(f"| 字段总数 | {overview.get('total_fields', 0)} |")
lines.append(f"| 大范围空字段（≥80%） | {overview.get('high_empty_fields', 0)} |")
lines.append(f"| 中范围空字段（≥50%） | {overview.get('mid_empty_fields', 0)} |")
lines.append(f"| 缺失注释字段 | {overview.get('missing_comments', 0)} |")
lines.append(f"| 字段长度异常 | {overview.get('length_issues', 0)} |")
lines.append(f"| 国标违规记录 | {overview.get('standard_violations', 0)} |")
lines.append(f"| 表合并候选 | {overview.get('merge_candidates', 0)} |")
lines.append(f"| 冗余字段 | {overview.get('redundancy_fields', 0)} |")
lines.append("")
# 表合并
_section_merge(lines, sections.get("merge_candidates", {}))
# 冗余字段
_section_redundancy(lines, sections.get("redundancy_fields", {}))
# 空字段
_section_empty(lines, sections.get("empty_fields", {}))
# 缺失注释
_section_missing(lines, sections.get("missing_comments", {}))
# 字段长度
_section_length(lines, sections.get("length_issues", {}))
# 国标违规
_section_standards(lines, sections.get("standard_violations", {}))
return "\n".join(lines)
def _section_merge(lines: list[str], data: dict):
lines.append("## 二、表合并候选")
cands = data.get("merge_candidates", [])
if not cands:
lines.append("（未发现可合并表）")
lines.append("")
return
lines.append("| ID | 优先级 | 表 | 相似度 | 建议 |")
lines.append("|----|--------|------|--------|------|")
for c in cands:
tables = ", ".join(c.get("tables", []))
lines.append(f"| {c.get('id','')} | {c.get('priority','')} | {tables} | "
f"{c.get('similarity', '')} | {c.get('suggestion', '')[:60]} |")
lines.append("")
decom = data.get("decommission_candidates", {})
strong = decom.get("strong", {}).get("tables", [])
if strong:
lines.append("### 废弃候选（有新版替代）")
lines.append("| 旧表 | 建议替代 |")
lines.append("|------|----------|")
for d in strong:
lines.append(f"| {d.get('table','')} | {d.get('replacement','')} |")
lines.append("")
def _section_redundancy(lines: list[str], data: dict):
lines.append("## 三、冗余字段")
fields = data.get("redundancy_fields", [])
if not fields:
lines.append("（未发现明显冗余字段）")
lines.append("")
return
lines.append("| 字段 | 出现表数 | 风险 | 建议 |")
lines.append("|------|----------|------|------|")
for f in fields:
lines.append(f"| {f.get('field','')} | {f.get('table_count','')} | "
f"{f.get('risk','')} | {f.get('suggestion','')[:60]} |")
lines.append("")
def _section_empty(lines: list[str], data: dict):
summary = data.get("summary", {})
lines.append("## 四、大范围空字段")
lines.append(f"扫描表数: **{summary.get('total_tables_scanned', 0)}**, "
f"高空字段: **{summary.get('high_empty_fields_count', 0)}**, "
f"中空字段: **{summary.get('mid_empty_fields_count', 0)}**")
lines.append("")
top = summary.get("top_tables_by_high_empty_fields", [])[:10]
if top:
lines.append("### 高空字段最多的表 Top 10")
lines.append("| 表名 | 高空字段数 |")
lines.append("|------|------------|")
for t in top:
lines.append(f"| {t.get('table_name','')} | {t.get('high_empty_field_count', 0)} |")
lines.append("")
def _section_missing(lines: list[str], data: dict):
summary = data.get("summary", {})
lines.append("## 五、缺失注释的字段")
lines.append(f"缺失注释字段: **{summary.get('total_missing_comments', 0)}**, "
f"涉及表数: **{summary.get('tables_affected', 0)}**, "
f"已推测: **{summary.get('predicted_count', 0)}**")
lines.append("")
samples = data.get("predicted_comments", [])[:20]
if samples:
lines.append("### 推测注释样本")
lines.append("| 表 | 字段 | 推测注释 | 置信度 | 依据 |")
lines.append("|----|------|----------|--------|------|")
for p in samples:
lines.append(f"| {p.get('table_name','')} | {p.get('column_name','')} | "
f"{p.get('predicted','')} | {p.get('confidence','')} | {p.get('reason','')} |")
lines.append("")
def _section_length(lines: list[str], data: dict):
summary = data.get("summary", {})
lines.append("## 六、字段长度异常")
lines.append(f"异常字段数: **{summary.get('total_issues', 0)}**, "
f"应用规则数: **{summary.get('rules_applied', 0)}**")
lines.append("")
issues = data.get("issues", [])[:20]
if issues:
lines.append("### 异常样本")
lines.append("| 表 | 字段 | 实际 | 标准 | 规则 | 建议 |")
lines.append("|----|------|------|------|------|------|")
for i in issues:
lines.append(f"| {i.get('table_name','')} | {i.get('column_name','')} | "
f"{i.get('actual_length','')} | "
f"{i.get('required_length','?')} | {i.get('rule','')} | "
f"{i.get('suggestion','')[:60]} |")
lines.append("")
def _section_standards(lines: list[str], data: dict):
summary = data.get("summary", {})
lines.append("## 七、不符合国家标准的字段")
lines.append(f"应用标准数: **{summary.get('standards_applied', 0)}**, "
f"校验字段数: **{summary.get('fields_checked', 0)}**, "
f"违规记录数: **{summary.get('violations', 0)}**")
lines.append("")
by_std = data.get("violations_by_standard", [])
if by_std:
lines.append("### 按标准统计")
lines.append("| 标准 | 标准名 | 字段数 | 违规数 | 违规率 |")
lines.append("|------|--------|--------|--------|--------|")
for v in by_std:
lines.append(f"| {v.get('standard','')} | {v.get('standard_name','')[:30]} | "
f"{v.get('fields_count',0)} | {v.get('total_invalid',0)} | "
f"{v.get('violation_rate', 0):.2%} |")
lines.append("")
def _render_docx(doc, sections: dict, overview: dict):
"""简易 docx 渲染（标题 + 段落 + 表格）"""
from docx.shared import Pt, Cm, RGBColor
from docx.enum.text import WD_ALIGN_PARAGRAPH
# 标题
h = doc.add_heading(f"数据治理报告 — {overview.get('database', '')}", level=0)
h.alignment = WD_ALIGN_PARAGRAPH.CENTER
doc.add_paragraph(f"生成时间: {overview.get('executed_at', '')}")
doc.add_paragraph(f"数据库类型: {overview.get('db_type', '-')}")
# 概览表格
doc.add_heading("一、概览", level=1)
overview_table = doc.add_table(rows=1, cols=2)
overview_table.style = "Light Grid Accent 1"
overview_table.rows[0].cells[0].text = "项目"
overview_table.rows[0].cells[1].text = "值"
for k, v in [
("数据库", overview.get("database", "-")),
("数据库类型", overview.get("db_type", "-")),
("主机", overview.get("host", "-")),
("表总数", str(overview.get("total_tables", 0))),
("字段总数", str(overview.get("total_fields", 0))),
("高空字段（≥80%）", str(overview.get("high_empty_fields", 0))),
("中空字段（≥50%）", str(overview.get("mid_empty_fields", 0))),
("缺失注释字段", str(overview.get("missing_comments", 0))),
("字段长度异常", str(overview.get("length_issues", 0))),
("国标违规记录", str(overview.get("standard_violations", 0))),
("表合并候选", str(overview.get("merge_candidates", 0))),
("冗余字段", str(overview.get("redundancy_fields", 0))),
]:
row = overview_table.add_row().cells
row[0].text = k
row[1].text = str(v)
# 简单添加各章节标题（详细表格由 Web 前端展示，Word 只给概览）
for sec_title, sec_key in [
("二、表合并候选", "merge_candidates"),
("三、冗余字段", "redundancy_fields"),
("四、大范围空字段", "empty_fields"),
("五、缺失注释的字段", "missing_comments"),
("六、字段长度异常", "length_issues"),
("七、不符合国家标准的字段", "standard_violations"),
]:
doc.add_heading(sec_title, level=1)
data = sections.get(sec_key, {})
summary = data.get("summary", {})
if summary:
doc.add_paragraph(f"本章节发现: {summary}")
doc.add_paragraph("（完整明细请查看 Web 页面或 Markdown 报告）")
\ No newline at end of file
web/docs/DESIGN.md 0 → 100644
View file @ 7b4bd4a8
# 数据治理 Web 工具 · 设计书
> 版本: v1.0 · 日期: 2026-08-05 · 范围: `web/` 目录
---
## 一、项目概述
### 1.1 目标
把已有的 CLI 版数据治理工作流（[workflow/](../workflow/)）改造为 **浏览器可访问、本地可发布** 的 Web 应用。Web 版必须满足：
| 维度 | 要求 |
|------|------|
| 部署 | 单机本地启动，浏览器访问 |
| 适配 | MySQL + 达梦（DM）双数据库 |
| 智能化 | 把"需要大模型判断"的内容封装为方法，在治理流程中调用 |
| 可视化 | 最终结果以**多级表格**呈现 |
| 透明度 | 页面显式展示**已注册的国标校验标准** |
| 可维护 | 后端用 Python，方便调大模型 + 复用原工作流代码 |
### 1.2 与原 CLI 工作流的关系
```
┌─────────────────────────────────────────────────────────────┐
│ 原 CLI 工作流 (workflow/) │
│ python workflow/run_governance.py --db smart-build │
└─────────────────────────────────────────────────────────────┘
│
│ 复用：standards/ 4 个国标插件
│ 重写：Step 业务逻辑改为 web 版
▼
┌─────────────────────────────────────────────────────────────┐
│ 新 Web 工具 (web/) │
│ python web/start.py → http://localhost:8765 │
└─────────────────────────────────────────────────────────────┘
```
- **`standards/`** 完全共享 — 插件化设计无侵入
- **`workflow/steps/`** 不再直接复用，而是参考其业务逻辑**重新实现**在 `web/core/step_impl/`
- 原因：CLI 版用 PyMySQL 直连，硬编码反引号，迁达梦要改多处
- Web 版走 `db_adapter` 抽象层 + LLM 增强，更适合浏览器侧体验
- **`workflow/reporter.py`** 的字段命名 bug 在 Web 版已修
---
## 二、架构总览
### 2.1 整体架构
```
┌────────────────────────────────────────────────────────────────────┐
│ 浏览器（前端） │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Vue 3 单页应用 (static/index.html + style.css) │ │
│ │ ├─ Element Plus（UI 组件库） │ │
│ │ ├─ 5 个卡片区块（表单/国标/进度/结果/历史） │ │
│ │ └─ 多级表格 + KPI 卡片 + SSE 日志流 │ │
│ └──────────────────────────────────────────────────────────────┘ │
└──────────────────────────┬─────────────────────────────────────────┘
│ HTTP + JSON / SSE
▼
┌────────────────────────────────────────────────────────────────────┐
│ FastAPI 后端 (web/app.py) │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 路由层 (api/routes.py) — 12 个端点 │ │
│ └──────────────────────┬───────────────────────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 任务层 (core/job_manager.py) │ │
│ │ ├─ 异步提交 + 进度跟踪 │ │
│ │ ├─ 日志环形缓冲 + SSE 发布 │ │
│ │ └─ 历史归档 (outputs/<db>/<ts>/) │ │
│ └──────────────────────┬───────────────────────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 调度层 (core/orchestrator.py) │ │
│ │ 8 步流程顺序执行，关键 Step 失败立即终止 │ │
│ └──────┬───────────┬───────────┬───────────┬─────────────────────┘ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Step 1 数据│ │Step 2 合并│ │Step 4 空 │ │Step 7 国标│ ... 8 步 │
│ │ 字典 │ │ + 冗余 │ │ 字段扫描│ │ 校验 │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ └────────────┴────────────┴────────────┘ │
│ │ │
│ ┌────────────────┼────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ db_adapter │ │ llm.py │ │ standards/ │ │
│ │ MySQL+达梦 │ │ Claude API │ │ 4 个国标 │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└────────────────────────────────────────────────────────────────────┘
```
### 2.2 技术选型
| 层 | 选型 | 理由 |
|----|------|------|
| 后端框架 | **FastAPI** | 原生 async、SSE 友好、自动 OpenAPI 文档 |
| ASGI 服务器 | **uvicorn** | FastAPI 官方推荐 |
| 数据校验 | **Pydantic v2** | FastAPI 配套；契约明确 |
| 数据库驱动 | **PyMySQL**（MySQL）+ **dmPython**（达梦） | 达梦官方驱动 |
| LLM | **Anthropic Claude API**（`anthropic` SDK） | 你正在用的 Claude 系列 |
| 前端框架 | **Vue 3**（CDN，无构建） | 单 HTML 即可运行 |
| UI 库 | **Element Plus**（CDN） | 多级表格 + 树形 + 进度条开箱即用 |
| 实时推送 | **Server-Sent Events**（`sse-starlette`） | 比 WebSocket 轻量；浏览器原生支持 |
| 报告生成 | **python-docx** | CLI 版沿用 |
---
## 三、目录结构
```
web/
├── app.py ← FastAPI 入口（uvicorn 启动）
├── start.py ← 启动脚本（依赖检查 + uvicorn）
├── start.bat ← Windows 一键启动
├── requirements.txt
│
├── core/ ← 后端核心（业务逻辑层）
│ ├── db_adapter.py ← 数据库适配（MySQL + 达梦）
│ ├── llm.py ← LLM 客户端 + 业务方法封装
│ ├── job_manager.py ← 后台任务 + SSE 日志
│ ├── orchestrator.py ← 8 步流程调度
│ ├── models.py ← Pydantic 请求/响应模型
│ └── step_impl/ ← 8 个 Step 的 Web 版实现
│ ├── step1_data_dict.py
│ ├── step2_merge_redundancy.py
│ ├── step3_verify.py
│ ├── step4_empty_fields.py
│ ├── step5_missing_comments.py
│ ├── step6_length_check.py
│ ├── step7_standards.py
│ └── step8_report.py
│
├── api/ ← 路由层
│ └── routes.py ← 12 个 REST 端点
│
├── static/ ← 前端静态资源
│ ├── index.html ← Vue 3 单页应用
│ └── style.css
│
├── configs/
│ └── defaults.yaml ← 默认配置
│
├── outputs/ ← 治理产出（按 db/ts 分目录）
└── logs/
└── app.log
```
---
## 四、后端设计
### 4.1 数据库适配层（`core/db_adapter.py`）
#### 设计动机
CLI 版 Step 1 用 PyMySQL 直连，硬编码：
```sql
SELECT c.TABLE_NAME FROM INFORMATION_SCHEMA.COLUMNS c WHERE c.TABLE_SCHEMA = %s
```
- 表/字段名用 PyMySQL 风格的 `%s` 占位符
- 标识符引用用反引号（MySQL 专属）
- 依赖 PyMySQL 的 DictCursor 返回列名
迁达梦需全部改写，**抽适配层一劳永逸**。
#### 统一接口
```python
class DBConfig:
db_type: str # "mysql" | "dameng"
host: str; port: int; user: str; password: str
database: str; charset: str; connect_timeout: int
class DBConnection:
def __enter__() → self
def __exit__()
def fetchall(sql, params) → list[dict] # 列名已规范化为小写
def fetchone(sql, params) → dict | None
def fetch_scalar(sql, params) → Any
def list_columns(schema) → list[dict] # 适配两种库
def list_tables(schema) → list[dict]
def quote_ident(name, db_type) → str # MySQL用反引号，达梦用双引号
def test_connection(cfg) → (ok, msg)
```
#### 适配点对照
| 项 | MySQL | 达梦 | 适配方式 |
|----|-------|------|----------|
| 驱动 | `pymysql` | `dmPython` | 运行时按 db_type 选择 import |
| 占位符 | `%s` 位置参数 | `:name` 命名参数 | `fetchall()` 内部统一改写 |
| 标识符引用 | `` ` `` | `"` | `quote_ident()` 函数 |
| 大文本类型 | `text/longtext/...` | `text/clob/longvarchar` | `TEXT_TYPES` 集合并集 |
| `INFORMATION_SCHEMA` | 列名小写 | 列名大写 | `normalize_rows()` 全转小写 |
| 字典游标 | DictCursor | 字典类 cursor | 应用层统一取 `.items()` |
> **安全要点**：`quote_ident()` 用正则 `[A-Za-z_][A-Za-z0-9_]*` 白名单校验，**杜绝 SQL 注入**。
---
### 4.2 LLM 客户端（`core/llm.py`）
#### 设计原则
1. **封装业务方法**：不暴露原始 Prompt，让 Step 只调 `predict_field_comment()` 这样语义清晰的方法
2. **优雅降级**：LLM 不可用（无 Key / 超时 / 解析失败）→ 返回 `None` 或 fallback，**不阻断主流程**
3. **JSON 解析兜底**：容忍 LLM 返回 Markdown 代码块、尾部多余文字
4. **三段式配置**：默认值 < 配置文件 < 环境变量（环境变量最高）
#### 业务方法清单
| 方法 | 用途 | 失败降级 |
|------|------|----------|
| `predict_field_comment()` | 推测字段中文注释 | 返回 `None`，规则兜底 |
| `suggest_table_merge()` | 表合并建议 + 风险 | 返回 `None`，用模板字符串 |
| `classify_empty_field()` | 高空字段成因分类 | 返回 `None`，跳过 LLM 标签 |
| `suggest_length_rule()` | 自定义字段长度推断 | 返回 `None`，标记"待确认" |
| `summarize_findings()` | 报告执行摘要 | 返回 `None`，略过该段 |
#### 支持的 Provider
| provider | 接入方式 | 默认 base_url | 默认模型 |
|----------|---------|---------------|----------|
| `minimax` | Anthropic SDK + 自定义 base_url | `https://api.minimaxi.com/anthropic` | `MiniMax-Text-01` |
| `anthropic` | Anthropic SDK 官方 | (空) | `claude-sonnet-5` |
| `openai` | OpenAI SDK | (空) | `gpt-4o` |
> **MiniMax 的关键点**：用 Anthropic SDK 调用，仅需传入 `base_url` 走兼容协议，**无需单独的 SDK**。
#### 配置加载（三段式优先级）
```
优先级从低到高：
1. 内置默认值（provider=minimax, model=MiniMax-Text-01 等）
2. web/configs/llm.yaml 文件（推荐持久化配置）
3. 环境变量（适合容器/CI/临时切换）
```
**配置文件路径**（`web/configs/llm.yaml`）：
```yaml
provider: minimax # minimax / anthropic / openai
api_key: sk-请填入你的_MiniMax_API_Key # 替换为真实 Key
base_url: https://api.minimaxi.com/anthropic
model: MiniMax-Text-01
max_tokens: 1024
temperature: 0.2
timeout: 30
```
**环境变量覆盖**：
| 变量 | 说明 | 默认专用变量 |
|------|------|--------------|
| `LLM_PROVIDER` | provider 名 | — |
| `LLM_API_KEY` | 通用 key | 也可 `MINIMAX_API_KEY` / `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` |
| `LLM_BASE_URL` | 自定义 base_url | — |
| `LLM_MODEL` | 模型名 | — |
| `LLM_MAX_TOKENS` | 最大 token | — |
| `LLM_CONFIG_FILE` | 配置文件路径（默认 `web/configs/llm.yaml`） | — |
> 启动时打印当前生效的 LLM 配置（Key 脱敏）：
> ```
> [LLM] provider=minimax model=MiniMax-Text-01
> [LLM] base_url=https://api.minimaxi.com/anthropic
> [LLM] api_key=sk-xxxx...xx（已配置）
> ```
> 前端可通过 `GET /api/llm/status` 查看。
---
### 4.3 后台任务调度（`core/job_manager.py`）
#### 数据结构
```python
@dataclass
class Job:
job_id: str # 时间戳 + uuid 前 6 位
req: ConnectRequest # 前端表单
status: str # pending / running / completed / failed / cancelled
created_at / started_at / finished_at: datetime
current_step / current_step_title: int | str
steps_planned / completed / failed: list[int]
logs: LogBuffer # 环形缓冲 + 异步发布
result: dict # 各 section 数据
run_dir: Path # outputs/<db>/<ts>/
_task: asyncio.Task
_cancel_event: asyncio.Event
class LogBuffer:
_entries: list[dict]
_subscribers: list[asyncio.Queue] # SSE 订阅者
```
#### SSE 日志流
```
前端 EventSource 后端 stream_logs()
│ │
│──── GET /api/jobs/<id>/logs/stream ──▶│
│ │ yield 历史日志
│◀── event: log data: {...} ─────────│
│◀── event: log data: {...} ─────────│
│ │ 每 15s 心跳
│◀── event: log data: {heartbeat} ──│
│ │ 任务完成 → 关闭流
│◀── (close) ──────────────────────│
```
#### 关键路径
1. 用户提交 `POST /api/jobs`
2. `submit()` 创建 `Job` + 启动 `asyncio.create_task(_run_job)`
3. `_run_job` 调用 `orchestrator.run_governance_workflow()`
4. 每步通过 `on_log` callback 把日志推入 `LogBuffer`
5. 任务完成/失败 → 状态更新 + `_all_findings.json` 落盘
6. 取消 → `_cancel_event.set()`，Step 内 `_check_cancel` 检测
#### 关键 Step 失败处理（`orchestrator.py`）
```python
CRITICAL_STEPS = {1} # Step 1 是基础，失败必须终止
try:
output = await run_in_executor(step_fn, ...)
...
except Exception as e:
log("ERROR", f"{title} 失败: {e}")
if num in CRITICAL_STEPS:
raise RuntimeError(f"关键步骤 Step {num} 失败，终止流程") from e
# 其他 Step 失败：继续（不阻断）
```
---
### 4.4 调度器（`core/orchestrator.py`）
```
run_governance_workflow(cfg, steps, run_dir, enable_llm, callbacks)
│
├─ 加载 LLM 客户端（按 enable_llm 决定）
│
├─ for step_num in [1,2,3,4,5,6,7,8]:
│ ├─ if step_num not in steps: 跳过
│ ├─ log("开始 Step X")
│ ├─ run_in_executor(step_fn) ← 同步步骤扔到线程池
│ ├─ 写 step_X_xxx.json 到 findings/
│ ├─ 累积到 sections dict
│ └─ log("完成 Step X")
│
├─ 构造 overview（从各 section 汇总）
│
└─ Step 8: 生成 Markdown + Word 报告
└─ 写入 reports/ 数据治理报告_<db>.md/.docx
```
每个 Step 的 `run_*()` 函数：
```python
def run_stepX(cfg, step_outputs, llm, findings_dir, log, cancel_event) -> dict:
"""返回 {"section_key": "...", "data": {...}, "extras": {...}}"""
```
---
### 4.5 路由设计（`api/routes.py`）
| 方法 | 路径 | 用途 |
|------|------|------|
| GET | `/api/health` | 健康检查 |
| GET | `/api/steps` | 列出 8 步元信息（含是否连库、是否 LLM） |
| GET | `/api/standards` | 列出已注册国标（页面初始化用） |
| POST | `/api/connect/test` | 测试连接（不启动任务） |
| POST | `/api/jobs` | 提交治理任务 |
| GET | `/api/jobs/{id}` | 任务状态 + 进度 |
| DELETE | `/api/jobs/{id}` | 取消任务 |
| GET | `/api/jobs/{id}/result` | 完整结果 |
| GET | `/api/jobs/{id}/logs/stream` | **SSE** 日志流 |
| GET | `/api/history` | 历史治理记录 |
| GET | `/api/history/{db}/{ts}/result` | 历史结果数据 |
| GET | `/api/reports/{db}/{ts}/{file}` | 下载报告 |
所有路径走 `/api` 前缀；CORS 全开（本地工具）。
---
## 五、前端设计（`static/index.html`）
### 5.1 单页结构
整个页面是 5 个 `<el-card>` 区块垂直堆叠：
```
┌─ ① 配置数据库连接 ────────────────────────┐
│ db_type [MySQL|达梦] host port │
│ user password database charset │
│ 执行步骤 (checkbox group) LLM 开关 │
│ [测试连接] [开始治理] [取消任务] │
└───────────────────────────────────────────┘
┌─ ② 已注册的国标校验标准 ───────────────────┐
│ el-table: id / name / applies_to / desc │
└───────────────────────────────────────────┘
┌─ ③ 治理进度 (v-if="currentJob") ──────────┐
│ el-progress + 当前步骤 + 完成数/总数 │
│ 可折叠实时日志（SSE 推送） │
└───────────────────────────────────────────┘
┌─ ④ 治理结果 (v-if="jobResult") ───────────┐
│ 8 个 KPI 卡片 │
│ el-tabs (7 个): │
│ ├ 表合并候选（可展开看 row_counts + columns）│
│ ├ 冗余字段 │
│ ├ 大范围空字段（含空值率进度条 + LLM 分类） │
│ ├ 缺失注释字段（含置信度 tag） │
│ ├ 字段长度异常 │
│ └ 国标违规（嵌套表：按标准 + 违规明细） │
└───────────────────────────────────────────┘
┌─ ⑤ 历史治理记录 ─────────────────────────┐
│ el-table: run_id / db / 时间 / 耗时 / 下载 │
└───────────────────────────────────────────┘
```
### 5.2 关键交互
#### 多级表格（Element Plus `el-table`）
```html
<el-table :data="mergeCandidates">
<el-table-column type="expand">
<template #default="{ row }">
<!-- 第二级：合并细节（共同字段、差异字段、行数） -->
<el-table :data="row.row_counts">...</el-table>
<el-tag v-for="c in row.columns">{{ c }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="tables" label="涉及表">
<template #default="{ row }">
<!-- 同行多 tag 显示 -->
<el-tag v-for="t in row.tables">{{ t }}</el-tag>
</template>
</el-table-column>
...
</el-table>
```
#### SSE 实时日志
```js
function subscribeLogs(jobId) {
logEventSource = new EventSource(`/api/jobs/${jobId}/logs/stream`);
logEventSource.addEventListener('log', (ev) => {
const entry = JSON.parse(ev.data);
logs.value.push(entry);
});
}
```
#### KPI 卡片网格
```css
.kpi-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
gap: 16px;
}
.kpi-card { border-left: 4px solid #1890ff; } /* 不同指标不同颜色 */
```
### 5.3 状态管理（无 Vuex）
页面状态用 `ref()` / `reactive()` 管理：
```js
const form = reactive({ db_type, host, port, ... }); // 表单
const currentJob = ref(null); // 当前任务 ID
const jobStatus = ref({ status, progress_pct, ... }); // 轮询更新
const logs = ref([]); // SSE 推送
const jobResult = ref(null); // 任务完成后加载
```
无外部依赖、无构建步骤，**直接打开 `index.html` 也能跑**（CDN 引入 Vue/Element）。
---
## 六、LLM 判断 vs 固定代码判断（重点）
这是设计书的核心：明确每个 Step 的判断来源，方便后续维护时知道**哪些可以放心改、哪些要小心动**。
### 6.1 决策矩阵
| Step | 业务点 | 判断方式 | 失败降级 |
|------|--------|----------|----------|
| **1** | 取数据字典 | **固定代码**（SQL） | N/A |
| **2** | 找结构相似表 | **固定代码**（Jaccard 相似度 ≥95%） | N/A |
| 2 | 合并建议文案 | **LLM**（看表注释+字段给自然语言建议） | 模板字符串 |
| 2 | 合并风险 | **LLM** | 留空 |
| **3** | 跑 COUNT(*) 验证 | **固定代码**（SQL） | 跳过该表 |
| 3 | 行政区划孤儿判断 | **固定代码**（字典表 LEFT JOIN） | N/A |
| **4** | 字段空值率统计 | **固定代码**（动态 SQL） | N/A |
| 4 | 高空字段成因分类 | **LLM**（业务不需要 / 采集漏了 / 预留） | 跳过 LLM 标签 |
| **5** | 字段注释推测（命中规则） | **固定代码**（内置 18 条字段名→注释映射） | N/A |
| 5 | 字段注释推测（未命中） | **LLM**（看字段名+表注释+类型） | 留空 |
| **6** | 字段长度检查（命中规则） | **固定代码**（14 条规则对应 GB/YD 标准） | N/A |
| 6 | 自定义字段长度推断 | **LLM** | 标记"待业务确认" |
| **7** | 国标字段校验 | **固定代码**（4 个 standards/ 插件） | N/A |
| **8** | 报告章节拼装 | **固定代码**（Markdown 模板） | N/A |
### 6.2 各 Step 判断来源详解
#### Step 1：100% 固定代码
```sql
SELECT c.TABLE_NAME, c.COLUMN_NAME, c.COLUMN_TYPE, ...
FROM INFORMATION_SCHEMA.COLUMNS c
WHERE c.TABLE_SCHEMA = %s
```
- 完全无 LLM 参与
- 跨数据库由 `db_adapter` 屏蔽差异
#### Step 2：固定 + LLM 混合
**固定代码**（核心逻辑）：
```python
def _similarity(a, b):
return len(a & b) / len(a | b) # Jaccard 系数
groups = []
for t in tables:
for t2 in remaining:
if similarity(cols[t], cols[t2]) >= 0.95:
groups.append((t, t2))
```
**LLM 增强**（文案/风险）：
```python
if llm and llm.available:
llm_result = llm.suggest_table_merge(
tables=[...], similarity=0.95,
common_columns=[...], diff_columns=[...]
)
# 用 LLM 生成的 suggestion 覆盖模板
```
#### Step 4：固定 + LLM 混合
**固定代码**（核心统计）：
```sql
SELECT
COUNT(*) AS total,
SUM(CASE WHEN col IS NULL OR TRIM(col) = '' THEN 1 ELSE 0 END) AS empty_c0,
SUM(CASE WHEN col2 IS NULL ... ) AS empty_c1,
...
FROM table
```
**LLM 增强**（仅对 top 20 高空字段分类成因）：
```python
for fld in high_fields_all[:20]:
llm.classify_empty_field(
table_name=..., column_name=..., empty_rate=...,
)
# 返回 {"classification": "missing|unused|reserved",
# "reasoning": "...", "action": "补采|删除|保留"}
```
#### Step 5：固定 + LLM 混合（LLM 增强比例最高）
**固定代码**（快速规则匹配）：
```python
FALLBACK_HINTS = {
"id": "主键ID",
"create_time": "创建时间",
"mobile": "手机号",
... # 18 条
}
if column_name in FALLBACK_HINTS:
return FALLBACK_HINTS[column_name], "high", "规则匹配"
```
**LLM 增强**（处理拼音首字母缩写等）：
```python
# 字段名 gzlb → LLM 推测 "工种类别"
llm.predict_field_comment(
table_name="t_worker",
table_comment="工人表",
column_name="gzlb",
data_type="varchar",
)
# 返回 {"comment": "工种类别", "confidence": "high"}
```
#### Step 6：固定 + LLM 混合
**固定代码**（14 条已知标准）：
```python
LENGTH_RULES = [
(re.compile(r"id_card"), 18, "GB 11643-1999", "身份证号"),
(re.compile(r"mobile"), 11, "YD/T 1313", "手机号"),
...
]
```
**LLM 增强**（自定义字段）：
```python
# varchar(500) 的 remark 字段 → LLM 推断合理长度
llm.suggest_length_rule(
column_name="remark",
column_comment="备注",
current_length=500,
)
# 返回 {"min_length": 50, "recommended": 200, "reasoning": "..."}
```
#### Step 7：100% 固定代码（standards 插件）
```python
# 注册：standards/registry.py 自动扫描 std_001_id_card.py 等
std = standards_by_field["id_card"] # IdCardStandard 实例
result = std.validate("110101199003078819")
# ValidationResult(valid=False, reason="校验码错误，应为 X", ...)
```
- 完全是确定性算法（身份证 MOD 11-2、USCC 31 字符集加权、手机号正则等）
- **不应**让 LLM 判断合规性（必须是确定性规则）
### 6.3 LLM 使用纪律
1. **核心判断必须是固定代码**：合并阈值、空值率、合规校验都是确定规则
2. **LLM 只做"语义增强"**：推测、分类、文案生成
3. **LLM 永远可降级**：无 Key / 失败 → 不报错，跳过 LLM 部分
4. **每次 LLM 调用要限制**：批量处理（如推测注释），单字段 ≤1 次；Top N 限制
5. **JSON 解析容错**：`_safe_parse_json()` 处理代码块包裹、尾部多余字符
---
## 七、启动方式
### 7.1 前置条件
| 条件 | 说明 |
|------|------|
| Python | 3.10+（开发环境 3.11.9） |
| 网络 | MySQL/达梦可达；LLM 需外网（可选） |
| 依赖 | 见 `web/requirements.txt` |
### 7.2 首次安装
```bash
# 在项目根目录
pip install -r web/requirements.txt
# 如需达梦数据库（需先装达梦客户端）
pip install dmPython
# 如需 LLM 增强
pip install anthropic
```
### 7.3 启动
#### 方式 A：Python（推荐）
```bash
python web/start.py
```
启动脚本会自动检查依赖并打印状态：
```
==============================================================
数据治理 Web 工具 · 启动检查
==============================================================
✓ 关键依赖齐全
可选组件状态：
· anthropic: 0.39.0
· dmPython: 未安装（达梦数据库不可用）
· ANTHROPIC_API_KEY: 已配置（sk-ant-...）
启动配置：
· 主机: 0.0.0.0
· 端口: 8765
==============================================================
```
#### 方式 B：Windows 一键
```cmd
web\start.bat
```
自动检查 Python + 自动 pip install + 启动。
#### 方式 C：直接 uvicorn（开发模式）
```bash
python -m uvicorn web.app:app --host 0.0.0.0 --port 8765 --reload
```
### 7.4 环境变量
| 变量 | 默认 | 说明 |
|------|------|------|
| `WEB_HOST` | `0.0.0.0` | 监听地址 |
| `WEB_PORT` | `8765` | 监听端口 |
| `LLM_PROVIDER` | `minimax` | LLM 提供方（minimax/anthropic/openai） |
| `LLM_API_KEY` | （从文件） | LLM API Key |
| `LLM_BASE_URL` | （从文件） | 自定义 base_url |
| `LLM_MODEL` | （从文件） | 模型名 |
| `LLM_MAX_TOKENS` | `1024` | 单次响应最大 token |
| `LLM_CONFIG_FILE` | `web/configs/llm.yaml` | 配置文件路径 |
| `MINIMAX_API_KEY` | （无） | MiniMax 专用 Key（备选） |
| `ANTHROPIC_API_KEY` | （无） | Anthropic 专用 Key（备选，兼容旧版） |
| `LOG_LEVEL` | `INFO` | 日志级别 |
### 7.5 验证启动
```bash
# 浏览器访问
open http://localhost:8765
# 或 curl 测试
curl http://localhost:8765/api/health
# {"status":"ok","service":"data-governance-web"}
# OpenAPI 文档
open http://localhost:8765/docs
```
---
## 八、配置项
### 8.1 默认配置（`web/configs/defaults.yaml`）
```yaml
db_types: [mysql, dameng]
thresholds:
empty_field: { high: 0.80, mid: 0.50, min_rows_to_check: 5 }
similarity: { merge_threshold: 0.80 }
standards:
enabled: true
auto_apply: true
strict_mode: false
llm:
model: claude-sonnet-5
max_tokens: 1024
temperature: 0.2
```
### 8.2 前端表单覆盖
页面表单（`static/index.html`）填写的连接信息**直接覆盖** YAML 配置，无需修改文件。
---
## 九、扩展指南
### 9.1 新增国标校验
在 `standards/` 下新建 `std_005_xxx.py`：
```python
from .base import BaseStandard, ValidationResult
import re
class EmailStandard(BaseStandard):
standard_id = "STD-005"
standard_name = "RFC 5322 电子邮件"
applies_to_fields = ["email", "e_mail"]
description = "标准邮箱格式"
_REGEX = re.compile(r"^[\w.+-]+@[\w-]+\.[\w.-]+$")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if not self._REGEX.match(value):
return ValidationResult(False, "", value, "格式错误", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
```
无需改主流程，`registry.discover_standards()` 会自动发现，Step 7 自动应用。
### 9.2 新增自定义 Step
在 `web/core/step_impl/step9_xxx.py` 写实现函数，然后在 `orchestrator.py` 的 `STEP_REGISTRY` 加一行：
```python
STEP_REGISTRY = [
...
(9, "新增检查", "_run_step9", False, False),
]
```
### 9.3 接入更多数据库
1. 在 `db_adapter.py` 加驱动 import 逻辑
2. 实现 `to_xxx_kwargs()` 方法
3. 修改 `quote_ident()` 标识符引用规则
4. 调整 `TEXT_TYPES` 类型集合
### 9.4 切换 LLM 提供方
```python
# web/core/llm.py 的 _init_provider 加分支
elif self.cfg.provider == "openai":
import openai
return openai.OpenAI(api_key=self.cfg.api_key)
```
设置环境变量：
```bash
export LLM_PROVIDER=openai
export OPENAI_API_KEY=sk-xxx
```
---
## 十、安全与限制
### 10.1 数据安全
| 项 | 措施 |
|----|------|
| 数据库密码 | 仅在内存中传递，不写日志；前端表单→Pydantic→cfg，不持久化 |
| SQL 注入 | `quote_ident()` 强制白名单（`[A-Za-z_][A-Za-z0-9_]*`） |
| 路径穿越 | `download_report()` 检查 `..`、分隔符 |
| CORS | 全开（本地工具，公网部署需收紧） |
| LLM 数据外发 | 字段名+表注释+少量样本值；用户可关闭 LLM 开关 |
### 10.2 性能与并发
- **单进程**：JobManager 用 asyncio 队列，**串行执行任务**（防止 DB 连接竞争）
- **进度感知**：Step 4（最重）每 10 张表打日志；前端 SSE 实时看到
- **大表保护**：Step 4 跳过行数 < 5 的小表（`min_rows_to_check`）
- **抽样限制**：Step 7 每字段最多抽 500 行 × 5 张表
### 10.3 已知限制
1. **达梦驱动依赖客户端**：`dmPython` 需本地装达梦客户端才能编译
2. **LLM Token 成本**：批量推测注释时按字段数计费，建议对 100+ 字段场景显式开关
3. **JobManager 不持久化**：服务重启后 in-memory job 丢失，但 `outputs/` 下 JSON 仍在
4. **前端无构建**：大表格（668 条违规）卡顿可考虑分页/虚拟滚动
---
## 十一、与历史产出的关系
`web/outputs/<db>/<ts>/` 与原 `outputs/<db>/<ts>/` **目录结构兼容**：
```
outputs/smart-build/20260805_120000/
├── findings/
│ ├── step_1_data_dict.json
│ ├── step_2_merge_redundancy.json
│ ├── ...
│ ├── step_8_report.json
│ └── _all_findings.json ← Web 工具额外写入
└── reports/
├── 数据治理报告_smart-build.md ← Markdown
└── 数据治理报告_smart-build.docx
```
历史 `outputs/` 中的 CLI 跑过的产物仍可读（前端 `/api/history` 会列出）。
---
## 十二、附录
### 12.1 接口契约示例
```json
POST /api/jobs
{
"db_type": "mysql",
"host": "192.168.20.10",
"port": 3306,
"user": "root",
"password": "****",
"database": "smart-build",
"charset": "utf8mb4",
"connect_timeout": 10,
"steps": [1,2,3,4,5,6,7,8],
"enable_llm": true
}
→ 201 Created
{
"job_id": "20260805_120000_a1b2c3",
"status": "pending",
"created_at": "2026-08-05T12:00:00",
"steps_planned": [1,2,3,4,5,6,7,8]
}
```
### 12.2 字段命名约定
| 前缀 | 含义 |
|------|------|
| `total_*` | 计数 |
| `*_count` | 同上 |
| `*_rate` | 比率（0-1） |
| `*_pct` | 百分比（0-100） |
| `is_*` / `has_*` | 布尔 |
| `*_at` | 时间戳 |
| `*_by` | 操作人 |
| `llm_*` | LLM 推断的字段 |
### 12.3 错误码
| 场景 | HTTP | 说明 |
|------|------|------|
| 测试连接失败 | 200 | `ok: false`，message 含 PyMySQL 错误 |
| 任务不存在 | 404 | `{detail: "任务不存在"}` |
| 任务未完成取结果 | 400 | `{detail: "任务未完成（当前: ...）"}` |
| 非法路径 | 400 | `{detail: "非法文件名"}` |
| 报告不存在 | 404 | `{detail: "报告不存在"}` |
---
**文档版本**：v1.0 · **最后更新**：2026-08-05 · **维护者**：项目所有者
\ No newline at end of file
web/requirements.txt 0 → 100644
View file @ 7b4bd4a8
# ============================================================
# 数据治理 Web 工具 — Python 依赖
# ============================================================
# Web 框架
fastapi>=0.110.0
uvicorn[standard]>=0.27.0
pydantic>=2.5.0
# 数据库
pymysql>=1.1.0 # MySQL 驱动
dmPython>=2.5.0 # 达梦数据库驱动（需本地有达梦客户端）
# LLM
anthropic>=0.39.0 # Claude API 官方 SDK
# 报告生成
python-docx>=1.1.0
# 配置
pyyaml>=6.0.1
# 异步流式输出（可选，FastAPI 自带 sse-starlette 替代）
sse-starlette>=2.1.0
\ No newline at end of file
web/sql/README.md 0 → 100644
View file @ 7b4bd4a8
# SQL 模板目录
所有 SQL 语句集中在此目录，Python 代码不再直接拼 SQL。
## 目录结构
```
sql/
├── loader.py ← 加载器（get_sql_loader() 单例）
├── info_schema/
│ ├── list_columns.mysql.sql ← INFORMATION_SCHEMA.COLUMNS (MySQL)
│ ├── list_columns.dameng.sql ← ALL_TAB_COLUMNS (达梦)
│ ├── list_tables.mysql.sql
│ └── list_tables.dameng.sql
├── verify/
│ └── count_table_rows.sql ← SELECT COUNT(*) FROM tbl
├── xzqh/
│ ├── check_dict_table_exists.sql
│ └── find_xzqh_orphans.sql
├── empty_fields/
│ └── count_with_nulls.sql ← 动态列空值统计
└── standards/
└── sample_field_values.sql ← 字段抽样（国标校验用）
```
## 占位符语法
| 语法 | 含义 | 示例 |
|------|------|------|
| `${var}` | 简单字符串替换 | `${schema}` |
| `${var \| quote}` | 用 `quote_ident()` 加引号（按当前 dialect 自动选反引号或双引号） | `${table \| quote}` |
| `${list \| join:","}` | 列表按分隔符拼接 | `${cols \| join:","}` |
| `${var \| upper}` | 转大写 | `${schema \| upper}` |
| `${var \| lower}` | 转小写 | `${schema \| lower}` |
## Dialect 加载规则
加载器按以下优先级找模板：
1. `<name>.<dialect>.sql`（如 `list_columns.mysql.sql`）
2. `<name>.sql`（跨方言通用版）
## 在 Python 中使用
```python
from web.sql.loader import get_sql_loader
loader = get_sql_loader()
sql = loader.render("info_schema/list_columns", dialect="mysql", schema="mydb")
# SELECT ... FROM INFORMATION_SCHEMA.COLUMNS c WHERE c.TABLE_SCHEMA = `mydb`
```
## 添加新 SQL 模板
1. 在合适子目录下创建 `<name>.sql`（或 `<name>.<dialect>.sql`）
2. 用 `${...}` 占位符标记变量
3. Python 代码调 `loader.render("子目录/name", dialect=..., **params)`
## 注意事项
- 所有标识符（表名/字段名）必须用 `${var | quote}`，由加载器自动按方言加引号
- 字面量（数字、字符串）可直接替换 `${var}`
- 列表类占位符用 `${list | join:"sep"}`
- 模板中可写 SQL 注释（`--`），方便维护
\ No newline at end of file
web/sql/empty_fields/count_with_nulls.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 统计单表多个列的 NULL/空值数量 + 总行数
-- 调用方：web/core/step_impl/step4_empty_fields.py
-- 参数：
-- ${table} 表名（自动加引号）
-- ${empty_clauses} 已构造好的 SUM(CASE WHEN...) 子句列表（逗号分隔）
-- 说明：
-- ${empty_clauses} 由 Python 端根据 data_type 过滤后动态构造，例如：
-- SUM(CASE WHEN "col1" IS NULL OR TRIM("col1") = '' THEN 1 ELSE 0 END) AS empty_c0,
-- SUM(CASE WHEN "col2" IS NULL OR TRIM("col2") = '' THEN 1 ELSE 0 END) AS empty_c1
-- ============================================================================
SELECT COUNT(*) AS total, ${empty_clauses}
FROM ${table | quote}
\ No newline at end of file
web/sql/health/check_connection.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 健康检查：确认连接可用，返回固定值 1
-- 调用方：web/core/db_adapter.py → test_connection()
-- 跨方言通用
-- ============================================================================
SELECT 1
\ No newline at end of file
web/sql/info_schema/list_columns.dameng.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 列出指定 schema 下所有列的元数据 (达梦方言)
-- 调用方：web/core/db_adapter.py → list_columns()
-- 参数：schema 数据库/模式名（绑定参数，由调用方通过 params 传入，勿拼接）
-- 说明：达梦的 ALL_TAB_COLUMNS.COMMENTS 列名与 MySQL 的 COLUMN_COMMENT 不同
-- 达梦的 OWNER 通常为大写
-- ============================================================================
SELECT
c.TABLE_NAME AS table_name,
c.COLUMN_NAME AS column_name,
c.COLUMN_ID AS ordinal_position,
c.DATA_TYPE || CASE WHEN c.DATA_LENGTH IS NOT NULL THEN '(' || c.DATA_LENGTH || ')' END AS column_type,
c.DATA_TYPE AS data_type,
c.DATA_LENGTH AS char_max_length,
c.DATA_PRECISION AS numeric_precision,
c.DATA_SCALE AS numeric_scale,
c.NULLABLE AS is_nullable,
c.DATA_DEFAULT AS column_default,
c.COMMENTS AS column_comment,
NULL AS extra
FROM ALL_TAB_COLUMNS c
WHERE c.OWNER = ?
ORDER BY c.TABLE_NAME, c.COLUMN_ID
\ No newline at end of file
web/sql/info_schema/list_columns.mysql.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 列出指定 schema 下所有列的元数据 (MySQL 方言)
-- 调用方：web/core/db_adapter.py → list_columns()
-- 参数：schema 数据库/模式名（绑定参数，由调用方通过 params 传入，勿拼接）
-- 注意：本文件不得出现其它百分号字面量，否则会被驱动误认为占位符
-- 返回字段：table_name, column_name, ordinal_position, column_type,
-- data_type, char_max_length, numeric_precision, numeric_scale,
-- is_nullable, column_default, column_comment, extra
-- ============================================================================
SELECT
c.TABLE_NAME AS table_name,
c.COLUMN_NAME AS column_name,
c.ORDINAL_POSITION AS ordinal_position,
c.COLUMN_TYPE AS column_type,
c.DATA_TYPE AS data_type,
c.CHARACTER_MAXIMUM_LENGTH AS char_max_length,
c.NUMERIC_PRECISION AS numeric_precision,
c.NUMERIC_SCALE AS numeric_scale,
c.IS_NULLABLE AS is_nullable,
c.COLUMN_DEFAULT AS column_default,
c.COLUMN_COMMENT AS column_comment,
c.EXTRA AS extra
FROM INFORMATION_SCHEMA.COLUMNS c
WHERE c.TABLE_SCHEMA = %s
ORDER BY c.TABLE_NAME, c.ORDINAL_POSITION
\ No newline at end of file
web/sql/info_schema/list_tables.dameng.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 列出指定 schema 下所有表 (达梦方言)
-- 调用方：web/core/db_adapter.py → list_tables()
-- 参数：schema 数据库/模式名（绑定参数，由调用方通过 params 传入，勿拼接）
-- ============================================================================
SELECT TABLE_NAME,
'BASE TABLE' AS TABLE_TYPE,
NULL AS ENGINE,
NUM_ROWS AS TABLE_ROWS,
BYTES AS DATA_LENGTH,
0 AS INDEX_LENGTH,
COMMENTS AS TABLE_COMMENT,
CREATED AS CREATE_TIME,
LAST_DDL_TIME AS UPDATE_TIME
FROM ALL_TABLES
WHERE OWNER = UPPER(?)
ORDER BY TABLE_NAME
\ No newline at end of file
web/sql/info_schema/list_tables.mysql.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 列出指定 schema 下所有表 (MySQL 方言)
-- 调用方：web/core/db_adapter.py → list_tables()
-- 参数：schema 数据库/模式名（绑定参数，由调用方通过 params 传入，勿拼接）
-- 注意：本文件不得出现其它百分号字面量，否则会被驱动误认为占位符
-- ============================================================================
SELECT TABLE_NAME, TABLE_TYPE, ENGINE, TABLE_ROWS,
DATA_LENGTH, INDEX_LENGTH, TABLE_COMMENT,
CREATE_TIME, UPDATE_TIME
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = %s
ORDER BY TABLE_NAME
\ No newline at end of file
web/sql/loader.py 0 → 100644
View file @ 7b4bd4a8
"""SQL 模板加载器
把所有 SQL 语句集中到 web/sql/ 目录下，Python 代码不再直接拼 SQL。
模板使用 ${var} 占位符（避免与 SQL 自身的 {} 冲突）。
支持的占位符：
- ${var_name} 简单字符串替换
- ${list | join:","} 列表拼接（默认分隔符 ","）
- ${list | join:" AND "} 自定义分隔符
Dialect 加载规则：
- 优先加载 <name>.<dialect>.sql
- 回退到 <name>.sql（通用版本）
- 都没有则报错
注意：${x | quote} 只用于**标识符**（表名/字段名）。
WHERE 子句里比较的**值**（如 schema 名）请走驱动的绑定参数，
不要用 quote —— 它产生的是标识符引用（反引号/双引号），不是字符串字面量。
示例：
loader = get_sql_loader()
sql = loader.render("verify/count_table_rows", dialect="mysql", table="t_project")
"""
from __future__ import annotations
import logging
import re
from pathlib import Path
from threading import Lock
from typing import Any, Iterable
logger = logging.getLogger(__name__)
# ── 占位符语法 ────────────────────────────────────────────
# 支持：${var} / ${var | filter} / ${var | filter:"sep"}
# 不支持嵌套 ${${a}} 这种（SQL 模板里不需要）
_PLACEHOLDER_RE = re.compile(
r"\$\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*"
r"(?:\|\s*([A-Za-z_]+)\s*(?::\s*\"([^\"]*)\")?\s*)?\}"
)
def _render_placeholders(template: str, params: dict) -> str:
"""把 ${var} / ${var | filter} 替换为 params 里的值
内置 filter：
- quote: quote_ident 单个值（需要 dialect 参数）
- join[:"<sep>"]: 把列表按 sep 拼接，默认 ","
- upper: 转大写
- lower: 转小写
"""
def _replace(m: re.Match) -> str:
var_name = m.group(1)
filt = m.group(2)
sep = m.group(3) # filter 的字符串参数
if var_name not in params:
raise KeyError(
f"SQL 模板参数缺失: {var_name!r}（提供: {list(params.keys())}）"
)
value = params[var_name]
# 应用 filter
if filt == "quote":
from web.core.db_adapter import quote_ident
dialect = params.get("_dialect", "mysql")
return quote_ident(str(value), dialect)
elif filt == "join":
if not isinstance(value, (list, tuple)):
raise TypeError(f"join filter 需要 list/tuple，{var_name} 是 {type(value).__name__}")
return sep.join(str(v) for v in value)
elif filt == "upper":
return str(value).upper()
elif filt == "lower":
return str(value).lower()
elif filt is None:
return str(value)
else:
raise ValueError(f"未知 filter: {filt!r}（{var_name}）")
return _PLACEHOLDER_RE.sub(_replace, template)
# ── 加载器 ────────────────────────────────────────────────
class SQLLoader:
"""SQL 模板加载器（单例）"""
def __init__(self, base_dir: Path | None = None):
self.base_dir = base_dir or (Path(__file__).resolve().parent)
self._cache: dict[str, str] = {}
self._lock = Lock()
# ── 加载 ──
def _find_template_file(self, name: str, dialect: str) -> Path:
"""优先找 .<dialect>.sql，回退 .sql"""
dsp = self.base_dir / f"{name}.{dialect}.sql"
if dsp.exists():
return dsp
common = self.base_dir / f"{name}.sql"
if common.exists():
return common
raise FileNotFoundError(
f"SQL 模板不存在: {name}（dialect={dialect}）；"
f"已查找: {dsp.relative_to(self.base_dir)} 和 {common.relative_to(self.base_dir)}"
)
def _load_text(self, name: str, dialect: str) -> str:
cache_key = f"{dialect}::{name}"
with self._lock:
if cache_key in self._cache:
return self._cache[cache_key]
path = self._find_template_file(name, dialect)
text = path.read_text(encoding="utf-8")
self._cache[cache_key] = text
logger.debug(f"加载 SQL 模板: {path.relative_to(self.base_dir)}")
return text
# ── 渲染 ──
def render(self, name: str, dialect: str = "mysql", **params) -> str:
"""渲染 SQL 模板
Args:
name: 模板路径（不含 .sql 后缀），如 "info_schema/list_columns"
dialect: 数据库方言 "mysql" / "dameng"
**params: 占位符参数
Returns:
渲染后的 SQL 字符串
"""
text = self._load_text(name, dialect)
params_with_dialect = {**params, "_dialect": dialect}
try:
return _render_placeholders(text, params_with_dialect)
except KeyError as e:
raise KeyError(f"SQL 模板 {name!r} (dialect={dialect}): {e}") from None
# ── 列出所有模板（调试用） ──
def list_templates(self) -> list[dict]:
"""列出所有可用模板（按分组）"""
results: list[dict] = []
for path in sorted(self.base_dir.rglob("*.sql")):
rel = path.relative_to(self.base_dir)
dialect = None
stem = rel.stem
if "." in stem:
# 可能含 dialect 后缀：list_columns.mysql
base, _, maybe_dialect = stem.rpartition(".")
if maybe_dialect in ("mysql", "dameng"):
stem = base
dialect = maybe_dialect
results.append({
"name": str(rel.with_name(stem))[:-4], # 去 .sql
"dialect": dialect,
"path": str(rel),
})
return results
# ── 清缓存（热加载调试用） ──
def clear_cache(self) -> None:
with self._lock:
self._cache.clear()
# ── 全局单例 ──────────────────────────────────────────────
_instance: SQLLoader | None = None
def get_sql_loader() -> SQLLoader:
"""获取 SQL 加载器单例"""
global _instance
if _instance is None:
# web/sql/loader.py → web/sql/ 是模板根目录
_instance = SQLLoader(Path(__file__).resolve().parent)
return _instance
def reset_sql_loader() -> None:
"""重置（测试用）"""
global _instance
_instance = None
\ No newline at end of file
web/sql/standards/sample_field_values.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 抽样某表某字段的非空非空白值（用于国标校验）
-- 调用方：web/core/step_impl/step7_standards.py
-- 参数：
-- ${table} 表名（自动加引号）
-- ${field} 字段名（自动加引号）
-- ${limit} 抽样上限（数字）
-- ============================================================================
SELECT ${field | quote} AS val
FROM ${table | quote}
WHERE ${field | quote} IS NOT NULL
AND TRIM(${field | quote}) <> ''
LIMIT ${limit}
\ No newline at end of file
web/sql/verify/count_table_rows.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 统计单表的实际行数
-- 调用方：web/core/step_impl/step3_verify.py
-- 参数：${table} 表名（自动加引号：MySQL 反引号 / 达梦双引号）
-- ============================================================================
SELECT COUNT(*) AS n FROM ${table | quote}
\ No newline at end of file
web/sql/xzqh/check_dict_table_exists.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 检查行政区划字典表是否存在
-- 调用方：web/core/step_impl/step3_verify.py
-- 参数：${dict_table} 字典表名（自动加引号）
-- 返回：单行单列（行数；>0 表示表存在且非空）
-- ============================================================================
SELECT COUNT(*) FROM ${dict_table | quote}
\ No newline at end of file
web/sql/xzqh/find_xzqh_orphans.sql 0 → 100644
View file @ 7b4bd4a8
-- ============================================================================
-- 找出业务表中存在但字典表中不存在的行政区划编码（孤儿）
-- 调用方：web/core/step_impl/step3_verify.py
-- 参数：
-- ${table} 业务表名（自动加引号）
-- ${xzqh_field} 行政区划字段名（自动加引号）
-- ${dict_table} 字典表名（自动加引号）
-- ${dict_code} 字典表编码字段名（自动加引号）
-- ${sample_limit} 抽样上限
-- ============================================================================
SELECT t.${xzqh_field | quote} AS orphan_code, COUNT(*) AS occurrences
FROM ${table | quote} t
LEFT JOIN ${dict_table | quote} d ON t.${xzqh_field | quote} = d.${dict_code | quote}
WHERE t.${xzqh_field | quote} IS NOT NULL
AND TRIM(t.${xzqh_field | quote}) <> ''
AND d.${dict_code | quote} IS NULL
GROUP BY t.${xzqh_field | quote}
ORDER BY occurrences DESC
LIMIT ${sample_limit}
\ No newline at end of file
web/start.bat 0 → 100644
View file @ 7b4bd4a8
@echo off
REM ============================================================
REM Data Governance Web Tool - Windows one-click launcher
REM (REM lines are ASCII-only to avoid cmd.exe encoding issues)
REM ============================================================
REM Switch to project root (parent of this script's directory)
cd /d "%~dp0\.."
echo.
echo ============================================================
echo Data Governance Web Tool
echo ============================================================
echo.
REM Check Python
python --version >nul 2>&1
if errorlevel 1 (
echo [ERROR] Python not found. Please install Python 3.10+ first.
pause
exit /b 1
)
REM Check fastapi dependency; install if missing
python -c "import fastapi" 2>nul
if errorlevel 1 (
echo [INFO] Installing dependencies...
python -m pip install -r web/requirements.txt
if errorlevel 1 (
echo [ERROR] Failed to install dependencies.
pause
exit /b 1
)
echo [INFO] Dependencies installed.
)
REM Launch the web tool
python web/start.py
pause
\ No newline at end of file
web/start.py 0 → 100644
View file @ 7b4bd4a8
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Web 工具启动脚本（Windows / 跨平台通用）
用法：
python web/start.py
# 或
python -m web.start
环境变量（可选）：
WEB_HOST=0.0.0.0
WEB_PORT=8765
ANTHROPIC_API_KEY=sk-ant-xxxxx （启用 LLM 增强）
LOG_LEVEL=DEBUG （默认 INFO）
"""
from __future__ import annotations
import logging
import os
import sys
from pathlib import Path
WEB_DIR = Path(__file__).resolve().parent
PROJECT_ROOT = WEB_DIR.parent
sys.path.insert(0, str(PROJECT_ROOT))
# ── 启动脚本自身的日志：写到 web/logs/start.log ──
_START_LOG = WEB_DIR / "logs"
_START_LOG.mkdir(parents=True, exist_ok=True)
logging.basicConfig(
level=os.environ.get("LOG_LEVEL", "INFO"),
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
handlers=[
logging.FileHandler(_START_LOG / "start.log", encoding="utf-8"),
logging.StreamHandler(),
],
)
logger = logging.getLogger("web.start")
def check_dependencies():
"""启动前检查关键依赖"""
missing = []
for pkg, name in [
("fastapi", "FastAPI"),
("uvicorn", "Uvicorn"),
("pydantic", "Pydantic"),
("sse_starlette", "sse-starlette"),
("pymysql", "PyMySQL"),
("docx", "python-docx"),
("yaml", "PyYAML"),
]:
try:
__import__(pkg)
logger.debug(f"依赖 {name} ({pkg}) 已安装")
except ImportError:
missing.append((pkg, name))
logger.warning(f"依赖缺失: {name} ({pkg})")
return missing
def check_optional():
"""检查可选依赖（LLM、达梦）"""
info = {}
try:
import anthropic
info["anthropic"] = anthropic.__version__
logger.debug(f"anthropic SDK 版本: {anthropic.__version__}")
except ImportError:
info["anthropic"] = "未安装（LLM 增强不可用）"
logger.info("anthropic SDK 未安装（LLM 增强不可用）")
try:
import dmPython
info["dmPython"] = "已安装"
logger.info("dmPython 已安装（达梦数据库可用）")
except ImportError:
info["dmPython"] = "未安装（达梦数据库不可用）"
logger.info("dmPython 未安装（达梦数据库不可用）")
api_key = os.environ.get("ANTHROPIC_API_KEY")
if api_key:
info["ANTHROPIC_API_KEY"] = f"已配置（{api_key[:8]}...）"
logger.info(f"ANTHROPIC_API_KEY 已配置（前缀: {api_key[:8]}...）")
else:
info["ANTHROPIC_API_KEY"] = "未配置"
logger.info("ANTHROPIC_API_KEY 未配置")
return info
def main():
print("=" * 70)
print(" 数据治理 Web 工具 · 启动检查")
print("=" * 70)
logger.info("=" * 60)
logger.info("数据治理 Web 工具 · 启动检查")
logger.info(f"Python: {sys.version.split()[0]}, 平台: {sys.platform}")
logger.info(f"工作目录: {PROJECT_ROOT}")
# 1. 依赖检查
missing = check_dependencies()
if missing:
msg = f"缺少关键依赖: {', '.join(n for _, n in missing)}"
print(f"\n✗ {msg}")
print(f"\n请执行: pip install -r {WEB_DIR / 'requirements.txt'}")
logger.error(msg)
sys.exit(1)
print("✓ 关键依赖齐全")
logger.info("关键依赖齐全")
# 2. 可选依赖
print("\n可选组件状态：")
logger.info("可选组件状态：")
for k, v in check_optional().items():
print(f" · {k}: {v}")
logger.info(f" · {k}: {v}")
# 3. 配置
host = os.environ.get("WEB_HOST", "0.0.0.0")
port = int(os.environ.get("WEB_PORT", "8765"))
print(f"\n启动配置：")
print(f" · 主机: {host}")
print(f" · 端口: {port}")
logger.info(f"启动配置: host={host}, port={port}")
# 4. 启动
import uvicorn
print("\n" + "=" * 70)
print(f" 数据治理 Web 工具")
print(f" 本机访问: http://localhost:{port}")
print(f" API 文档: http://localhost:{port}/docs")
print(f" 按 Ctrl+C 停止")
print("=" * 70 + "\n")
logger.info(f"启动 uvicorn: web.app:app @ http://{host}:{port}")
logger.info(f"日志级别: {os.environ.get('LOG_LEVEL', 'info').upper()}")
uvicorn.run(
"web.app:app",
host=host,
port=port,
reload=False,
log_level=os.environ.get("LOG_LEVEL", "info"),
)
if __name__ == "__main__":
main()
\ No newline at end of file
web/static/index.html 0 → 100644
View file @ 7b4bd4a8
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>数据治理工具</title>
<!-- 前端依赖全部本地化（避免 CDN 被墙/慢） -->
<!-- Element Plus CSS -->
<link rel="stylesheet" href="/static/lib/element-plus/index.css">
<!-- Vue 3 -->
<script src="/static/lib/vue.global.prod.js"></script>
<!-- Element Plus -->
<script src="/static/lib/element-plus/index.full.min.js"></script>
<!-- Element Plus Icons (IIfe bundle, 直接挂在 window.ElementPlusIconsVue) -->
<script src="/static/lib/@element-plus/icons-vue/index.iife.min.js"></script>
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<div id="app">
<el-container>
<!-- 顶栏 -->
<el-header class="app-header">
<div class="header-inner">
<div class="brand">
<el-icon><DataAnalysis /></el-icon>
<span class="brand-title">数据治理工具</span>
<el-tag size="small" type="info" effect="plain">v1.0</el-tag>
</div>
<div class="header-actions">
<el-button @click="loadHistory" :icon="Refresh">历史记录</el-button>
<el-button @click="loadStandards" :icon="Medal">国标清单</el-button>
</div>
</div>
</el-header>
<el-main class="app-main">
<!-- 步骤 1：连接数据库 -->
<el-card class="section-card" shadow="hover">
<template #header>
<div class="card-header">
<el-icon><Connection /></el-icon>
<span class="card-title">① 配置数据库连接</span>
</div>
</template>
<el-form :model="form" label-width="100px" :inline="false">
<el-row :gutter="20">
<el-col :span="12">
<el-form-item label="数据库类型">
<el-radio-group v-model="form.db_type">
<el-radio-button label="mysql">MySQL</el-radio-button>
<el-radio-button label="dameng">达梦 (DM)</el-radio-button>
</el-radio-group>
</el-form-item>
</el-col>
<el-col :span="12">
<el-form-item label="主机 IP">
<el-input v-model="form.host" placeholder="192.168.20.10" />
</el-form-item>
</el-col>
</el-row>
<el-row :gutter="20">
<el-col :span="8">
<el-form-item label="端口">
<el-input-number v-model="form.port" :min="1" :max="65535" style="width: 100%" />
</el-form-item>
</el-col>
<el-col :span="8">
<el-form-item label="用户名">
<el-input v-model="form.user" placeholder="root" />
</el-form-item>
</el-col>
<el-col :span="8">
<el-form-item label="密码">
<el-input v-model="form.password" type="password" show-password placeholder="（不回显）" />
</el-form-item>
</el-col>
</el-row>
<el-row :gutter="20">
<el-col :span="12">
<el-form-item label="数据库名">
<el-input v-model="form.database" placeholder="smart-build" />
</el-form-item>
</el-col>
<el-col :span="12">
<el-form-item label="字符集">
<el-select v-model="form.charset" style="width: 100%">
<el-option label="utf8mb4" value="utf8mb4" />
<el-option label="utf8" value="utf8" />
<el-option label="gbk" value="gbk" />
</el-select>
</el-form-item>
</el-col>
</el-row>
<el-row :gutter="20">
<el-col :span="12">
<el-form-item label="执行步骤">
<el-checkbox-group v-model="form.steps">
<el-checkbox v-for="s in stepsList" :key="s.num" :label="s.num">
Step {{ s.num }}: {{ s.title }}
</el-checkbox>
</el-checkbox-group>
</el-form-item>
</el-col>
<el-col :span="12">
<el-form-item label="启用 LLM">
<el-switch v-model="form.enable_llm" active-text="启用" inactive-text="关闭" />
<span class="form-hint">（缺 Key 时自动降级为规则推理）</span>
</el-form-item>
</el-col>
</el-row>
<el-form-item>
<el-button type="primary" @click="testConnection" :loading="testing">
<el-icon><CircleCheck /></el-icon>测试连接
</el-button>
<el-button type="success" @click="startJob" :disabled="!connected" :loading="submitting">
<el-icon><VideoPlay /></el-icon>开始治理
</el-button>
<el-button v-if="currentJob" @click="cancelJob" type="danger" plain>
<el-icon><Close /></el-icon>取消任务
</el-button>
</el-form-item>
<el-alert v-if="testResult" :title="testResult.msg" :type="testResult.ok ? 'success' : 'error'" :closable="false" show-icon />
</el-form>
</el-card>
<!-- 步骤 2：国标清单 -->
<el-card class="section-card" shadow="hover">
<template #header>
<div class="card-header">
<el-icon><Medal /></el-icon>
<span class="card-title">② 已注册的国标校验标准（{{ standards.length }} 项）</span>
</div>
</template>
<el-table :data="standards" stripe size="small">
<el-table-column prop="id" label="标准编号" width="120" />
<el-table-column prop="name" label="标准名称" width="240" />
<el-table-column prop="description" label="校验说明" />
<el-table-column label="适用字段名">
<template #default="{ row }">
<el-tag v-for="f in row.applies_to_fields" :key="f" size="small" type="info" effect="plain" style="margin-right: 4px">{{ f }}</el-tag>
</template>
</el-table-column>
</el-table>
</el-card>
<!-- 步骤 3：进度 + 日志 -->
<el-card v-if="currentJob" class="section-card" shadow="hover">
<template #header>
<div class="card-header">
<el-icon><Loading /></el-icon>
<span class="card-title">③ 治理进度</span>
<el-tag :type="statusTagType">{{ jobStatus.status }}</el-tag>
</div>
</template>
<el-progress :percentage="jobStatus.progress_pct" :status="jobStatus.status === 'failed' ? 'exception' : (jobStatus.status === 'completed' ? 'success' : '')" />
<div class="progress-info">
<div>当前: {{ jobStatus.current_step_title || '—' }}</div>
<div>已完成: {{ jobStatus.steps_completed.length }} / {{ jobStatus.steps_planned.length }}</div>
<div v-if="jobStatus.error" class="error-info">错误: {{ jobStatus.error }}</div>
</div>
<el-collapse v-model="logOpen" class="log-collapse">
<el-collapse-item title="实时日志（点击展开）" name="logs">
<div class="log-container">
<div v-for="(log, idx) in logs" :key="idx" :class="['log-line', `log-${log.level.toLowerCase()}`]">
<span class="log-ts">{{ log.ts.substring(11, 19) }}</span>
<span class="log-level">{{ log.level }}</span>
<span v-if="log.step" class="log-step">Step {{ log.step }}</span>
<span class="log-msg">{{ log.message }}</span>
</div>
</div>
</el-collapse-item>
</el-collapse>
</el-card>
<!-- 步骤 4：结果展示（多级表格） -->
<el-card v-if="jobResult" class="section-card" shadow="hover">
<template #header>
<div class="card-header">
<el-icon><DataLine /></el-icon>
<span class="card-title">④ 治理结果</span>
<div>
<el-button v-if="jobResult.reports.markdown" size="small" @click="downloadReport('md')">
<el-icon><Download /></el-icon>Markdown
</el-button>
<el-button v-if="jobResult.reports.docx" size="small" type="primary" @click="downloadReport('docx')">
<el-icon><Download /></el-icon>Word
</el-button>
</div>
</div>
</template>
<!-- 概览 KPI 卡片 -->
<div class="kpi-grid">
<div class="kpi-card kpi-tables">
<div class="kpi-value">{{ jobResult.overview.total_tables }}</div>
<div class="kpi-label">表总数</div>
</div>
<div class="kpi-card kpi-fields">
<div class="kpi-value">{{ jobResult.overview.total_fields }}</div>
<div class="kpi-label">字段总数</div>
</div>
<div class="kpi-card kpi-empty">
<div class="kpi-value">{{ jobResult.overview.high_empty_fields }}</div>
<div class="kpi-label">高空字段</div>
</div>
<div class="kpi-card kpi-missing">
<div class="kpi-value">{{ jobResult.overview.missing_comments }}</div>
<div class="kpi-label">缺注释字段</div>
</div>
<div class="kpi-card kpi-length">
<div class="kpi-value">{{ jobResult.overview.length_issues }}</div>
<div class="kpi-label">字段长度异常</div>
</div>
<div class="kpi-card kpi-std">
<div class="kpi-value">{{ jobResult.overview.standard_violations }}</div>
<div class="kpi-label">国标违规</div>
</div>
<div class="kpi-card kpi-merge">
<div class="kpi-value">{{ jobResult.overview.merge_candidates }}</div>
<div class="kpi-label">合并候选</div>
</div>
<div class="kpi-card kpi-redundancy">
<div class="kpi-value">{{ jobResult.overview.redundancy_fields }}</div>
<div class="kpi-label">冗余字段</div>
</div>
</div>
<!-- 多级结果 -->
<el-tabs v-model="activeTab" type="border-card">
<!-- 表合并候选 -->
<el-tab-pane label="表合并候选" name="merge">
<el-table :data="jobResult.sections.merge_candidates?.merge_candidates || []" stripe size="small">
<el-table-column type="expand">
<template #default="{ row }">
<div style="padding: 10px">
<p><strong>合并建议：</strong>{{ row.suggestion }}</p>
<p v-if="row.risk"><strong>合并风险：</strong>{{ row.risk }}</p>
<p><strong>共同字段（{{ row.common_columns }}）：</strong></p>
<el-tag v-for="c in row.columns" :key="c" size="small" style="margin-right: 4px">{{ c }}</el-tag>
<p v-if="row.diff_columns?.length"><strong>差异字段：</strong></p>
<el-tag v-for="c in row.diff_columns" :key="c" size="small" type="warning" style="margin-right: 4px">{{ c }}</el-tag>
<p v-if="row.row_counts"><strong>实际行数：</strong></p>
<el-table v-if="row.row_counts" :data="row.row_counts" size="small" border style="margin-top: 6px">
<el-table-column prop="table" label="表名" />
<el-table-column label="行数">
<template #default="{ row: rc }">{{ rc.rows ?? rc.error }}</template>
</el-table-column>
</el-table>
</div>
</template>
</el-table-column>
<el-table-column prop="id" label="ID" width="100" />
<el-table-column label="优先级" width="80">
<template #default="{ row }">
<el-tag :type="row.priority === 'P1' ? 'danger' : 'warning'" size="small">{{ row.priority }}</el-tag>
</template>
</el-table-column>
<el-table-column label="涉及表" width="300">
<template #default="{ row }">
<el-tag v-for="t in row.tables" :key="t" size="small" style="margin-right: 4px">{{ t }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="common_columns" label="共同字段数" width="100" />
<el-table-column prop="similarity" label="相似度" width="100" />
<el-table-column prop="suggestion" label="建议" />
</el-table>
</el-tab-pane>
<!-- 冗余字段 -->
<el-tab-pane label="冗余字段" name="redundancy">
<el-table :data="jobResult.sections.merge_candidates?.redundancy_fields || []" stripe size="small">
<el-table-column prop="field" label="字段名" width="200" />
<el-table-column prop="table_count" label="出现表数" width="120" sortable />
<el-table-column prop="risk" label="风险" width="200" />
<el-table-column prop="suggestion" label="建议" />
</el-table>
</el-tab-pane>
<!-- 大范围空字段 -->
<el-tab-pane label="大范围空字段" name="empty">
<el-alert :title="`扫描 ${jobResult.sections.empty_fields?.summary?.total_tables_scanned || 0} 张表，发现 ${jobResult.sections.empty_fields?.summary?.high_empty_fields_count || 0} 个高空字段`" type="info" :closable="false" />
<el-table :data="jobResult.sections.empty_fields?.summary?.high_empty_fields || []" stripe size="small" style="margin-top: 12px">
<el-table-column prop="table_name" label="表名" width="180" fixed />
<el-table-column prop="column_name" label="字段名" width="160" fixed />
<el-table-column prop="data_type" label="类型" width="100" />
<el-table-column prop="column_comment" label="注释" width="160" />
<el-table-column prop="total_rows" label="总行数" width="100" sortable />
<el-table-column label="空值数" width="100" sortable>
<template #default="{ row }">{{ row.empty_count }}</template>
</el-table-column>
<el-table-column label="空值率" width="120" sortable>
<template #default="{ row }">
<el-progress :percentage="Math.round(row.empty_rate * 100)" :stroke-width="12" :status="row.empty_rate >= 0.8 ? 'exception' : 'warning'" />
</template>
</el-table-column>
<el-table-column label="LLM 分类" width="160">
<template #default="{ row }">
<el-tag v-if="row.llm_classification" :type="row.llm_classification === 'missing' ? 'danger' : 'success'" size="small">
{{ row.llm_classification }}
</el-tag>
<span v-else class="muted">—</span>
</template>
</el-table-column>
<el-table-column prop="llm_reasoning" label="LLM 判断依据" />
</el-table>
</el-tab-pane>
<!-- 缺失注释字段 -->
<el-tab-pane label="缺失注释字段" name="missing">
<el-alert :title="`${jobResult.sections.missing_comments?.summary?.total_missing_comments || 0} 个字段缺注释，已推测 ${jobResult.sections.missing_comments?.summary?.predicted_count || 0} 个（其中 LLM 推测 ${jobResult.sections.missing_comments?.summary?.predicted_by_llm || 0} 个）`" type="info" :closable="false" />
<el-table :data="jobResult.sections.missing_comments?.predicted_comments || []" stripe size="small" style="margin-top: 12px">
<el-table-column prop="table_name" label="表名" width="180" fixed />
<el-table-column prop="column_name" label="字段名" width="160" fixed />
<el-table-column prop="data_type" label="类型" width="100" />
<el-table-column prop="predicted" label="推测注释" width="200" />
<el-table-column label="置信度" width="100">
<template #default="{ row }">
<el-tag :type="row.confidence === 'high' ? 'success' : (row.confidence === 'medium' ? 'warning' : 'info')" size="small">
{{ row.confidence }}
</el-tag>
</template>
</el-table-column>
<el-table-column prop="reason" label="依据" />
</el-table>
</el-tab-pane>
<!-- 字段长度异常 -->
<el-tab-pane label="字段长度异常" name="length">
<el-alert :title="`${jobResult.sections.length_issues?.summary?.total_issues || 0} 条异常，应用 ${jobResult.sections.length_issues?.summary?.rules_applied || 0} 条规则`" type="warning" :closable="false" />
<el-table :data="jobResult.sections.length_issues?.issues || []" stripe size="small" style="margin-top: 12px">
<el-table-column prop="table_name" label="表名" width="180" fixed />
<el-table-column prop="column_name" label="字段名" width="160" fixed />
<el-table-column prop="data_type" label="类型" width="80" />
<el-table-column prop="actual_length" label="实际长度" width="100" />
<el-table-column prop="required_length" label="标准长度" width="100" />
<el-table-column prop="wasted_bytes_per_row" label="每行浪费字节" width="120" />
<el-table-column prop="standard" label="国家标准" width="160" />
<el-table-column prop="rule" label="规则" width="200" />
<el-table-column prop="suggestion" label="建议" />
</el-table>
</el-tab-pane>
<!-- 国标违规 -->
<el-tab-pane label="国标违规" name="standards">
<el-alert :title="`${jobResult.sections.standard_violations?.summary?.standards_applied || 0} 个标准，${jobResult.sections.standard_violations?.summary?.violations || 0} 条违规`" type="error" :closable="false" />
<!-- 按标准汇总 -->
<h4 style="margin-top: 16px">按标准统计</h4>
<el-table :data="jobResult.sections.standard_violations?.violations_by_standard || []" stripe size="small">
<el-table-column type="expand">
<template #default="{ row }">
<el-table :data="row.details" size="small" border>
<el-table-column prop="table" label="表" width="180" />
<el-table-column prop="field" label="字段" width="160" />
<el-table-column prop="sampled" label="抽样数" width="100" />
<el-table-column prop="violations" label="违规数" width="100" />
<el-table-column label="违规率" width="120">
<template #default="{ row: d }">{{ (d.violation_rate * 100).toFixed(1) }}%</template>
</el-table-column>
<el-table-column label="样本">
<template #default="{ row: d }">
<div v-for="(s, idx) in d.samples" :key="idx" style="font-size: 12px">
<code>{{ s.value }}</code> → {{ s.reason }}
</div>
<div v-if="d.error" style="color: red">{{ d.error }}</div>
</template>
</el-table-column>
</el-table>
</template>
</el-table-column>
<el-table-column prop="standard" label="标准" width="100" />
<el-table-column prop="standard_name" label="标准名称" />
<el-table-column prop="fields_count" label="字段数" width="100" />
<el-table-column prop="total_sampled" label="抽样数" width="100" />
<el-table-column prop="total_invalid" label="违规数" width="100" />
<el-table-column label="违规率" width="120">
<template #default="{ row }">{{ (row.violation_rate * 100).toFixed(1) }}%</template>
</el-table-column>
</el-table>
<!-- 违规明细 -->
<h4 style="margin-top: 16px">违规明细（前 200 条）</h4>
<el-table :data="(jobResult.sections.standard_violations?.violations || []).slice(0, 200)" stripe size="small">
<el-table-column prop="table_name" label="表" width="180" />
<el-table-column prop="column_name" label="字段" width="160" />
<el-table-column prop="rule_type" label="标准" width="100" />
<el-table-column prop="standard_name" label="标准名" width="180" />
<el-table-column prop="value" label="值" width="160">
<template #default="{ row }"><code>{{ row.value }}</code></template>
</el-table-column>
<el-table-column prop="error" label="违规原因" />
</el-table>
</el-tab-pane>
</el-tabs>
</el-card>
<!-- 历史记录 -->
<el-card class="section-card" shadow="hover">
<template #header>
<div class="card-header">
<el-icon><Histogram /></el-icon>
<span class="card-title">⑤ 历史治理记录（{{ history.length }} 次）</span>
</div>
</template>
<el-table :data="history" stripe size="small">
<el-table-column prop="run_id" label="运行 ID" width="200" />
<el-table-column prop="database" label="数据库" width="160" />
<el-table-column prop="executed_at" label="执行时间" width="200" />
<el-table-column prop="duration_sec" label="耗时 (秒)" width="100" />
<el-table-column label="概览">
<template #default="{ row }">
<el-tag size="small" type="info">{{ row.stats.total_tables || 0 }} 表 / {{ row.stats.total_fields || 0 }} 字段</el-tag>
<el-tag size="small" type="warning" style="margin-left: 4px">{{ row.stats.high_empty_fields || 0 }} 高空</el-tag>
<el-tag size="small" type="danger" style="margin-left: 4px">{{ row.stats.standard_violations || 0 }} 违规</el-tag>
</template>
</el-table-column>
<el-table-column label="下载">
<template #default="{ row }">
<el-button v-if="row.has_markdown" size="small" link @click="downloadHistoryReport(row, 'md')">MD</el-button>
<el-button v-if="row.has_docx" size="small" link type="primary" @click="downloadHistoryReport(row, 'docx')">Word</el-button>
<span v-if="!row.has_markdown && !row.has_docx" class="muted">—</span>
</template>
</el-table-column>
</el-table>
</el-card>
</el-main>
</el-container>
</div>
<script>
const { createApp, ref, reactive, onMounted, computed, watch } = Vue;
const App = {
setup() {
// ── 表单 ──
const form = reactive({
db_type: 'mysql',
host: '192.168.20.10',
port: 3306,
user: 'root',
password: '',
database: 'smart-build',
charset: 'utf8mb4',
connect_timeout: 10,
steps: [1, 2, 3, 4, 5, 6, 7, 8],
enable_llm: true,
});
const testing = ref(false);
const submitting = ref(false);
const connected = ref(false);
const testResult = ref(null);
// ── 步骤 & 标准 ──
const stepsList = ref([]);
const standards = ref([]);
// ── 任务状态 ──
const currentJob = ref(null);
const jobStatus = ref({
status: 'pending',
progress_pct: 0,
steps_planned: [],
steps_completed: [],
steps_failed: [],
current_step_title: '',
error: null,
});
const logs = ref([]);
const logOpen = ref(false);
let logEventSource = null;
let pollTimer = null;
const statusTagType = computed(() => ({
pending: 'info', running: 'warning', completed: 'success',
failed: 'danger', cancelled: 'info',
}[jobStatus.value.status] || 'info'));
// ── 结果 ──
const jobResult = ref(null);
const activeTab = ref('merge');
// ── 历史 ──
const history = ref([]);
// ── 加载步骤 ──
async function loadSteps() {
try {
const r = await fetch('/api/steps').then(r => r.json());
stepsList.value = r.steps;
} catch (e) { console.error(e); }
}
// ── 加载标准 ──
async function loadStandards() {
try {
const r = await fetch('/api/standards').then(r => r.json());
standards.value = r.standards;
} catch (e) { console.error(e); }
}
// ── 测试连接 ──
async function testConnection() {
testing.value = true;
testResult.value = null;
try {
const r = await fetch('/api/connect/test', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(form),
}).then(r => r.json());
testResult.value = { ok: r.ok, msg: r.message };
connected.value = r.ok;
} catch (e) {
testResult.value = { ok: false, msg: '请求失败: ' + e };
} finally {
testing.value = false;
}
}
// ── 启动治理 ──
async function startJob() {
submitting.value = true;
jobResult.value = null;
logs.value = [];
try {
const r = await fetch('/api/jobs', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(form),
}).then(r => r.json());
currentJob.value = r.job_id;
subscribeLogs(r.job_id);
pollStatus(r.job_id);
loadHistory();
} catch (e) {
ElementPlus.ElMessage.error('启动失败: ' + e);
} finally {
submitting.value = false;
}
}
// ── 订阅日志 ──
function subscribeLogs(jobId) {
if (logEventSource) logEventSource.close();
logEventSource = new EventSource(`/api/jobs/${jobId}/logs/stream`);
logEventSource.addEventListener('log', (ev) => {
try {
const entry = JSON.parse(ev.data);
logs.value.push(entry);
if (logs.value.length > 1000) logs.value.shift();
} catch (e) {}
});
logEventSource.onerror = () => {
// SSE 断线不报错（任务结束时正常关闭）
};
}
// ── 轮询状态 ──
function pollStatus(jobId) {
if (pollTimer) clearInterval(pollTimer);
pollTimer = setInterval(async () => {
try {
const r = await fetch(`/api/jobs/${jobId}`).then(r => r.json());
jobStatus.value = r;
if (r.status === 'completed') {
clearInterval(pollTimer);
await loadResult(jobId);
ElementPlus.ElMessage.success('治理完成');
loadHistory();
} else if (r.status === 'failed' || r.status === 'cancelled') {
clearInterval(pollTimer);
ElementPlus.ElMessage.warning(`任务${r.status === 'failed' ? '失败' : '已取消'}`);
if (r.status === 'completed') loadResult(jobId);
}
} catch (e) { console.error(e); }
}, 1500);
}
async function loadResult(jobId) {
try {
const r = await fetch(`/api/jobs/${jobId}/result`).then(r => r.json());
jobResult.value = r;
} catch (e) { console.error(e); }
}
async function cancelJob() {
if (!currentJob.value) return;
try {
await fetch(`/api/jobs/${currentJob.value}`, { method: 'DELETE' });
ElementPlus.ElMessage.info('已请求取消');
} catch (e) {}
}
// ── 下载报告 ──
function downloadReport(fmt) {
if (!jobResult.value || !jobResult.value.reports) return;
const url = jobResult.value.reports[fmt];
if (!url) return;
window.open(url.replace(/^.*outputs/, '/api/reports'), '_blank');
}
async function downloadHistoryReport(row, fmt) {
const ext = fmt === 'docx' ? 'docx' : 'md';
const filename = `数据治理报告_${row.database}.${ext}`;
window.open(`/api/reports/${row.database}/${row.run_id}/${filename}`, '_blank');
}
// ── 历史 ──
async function loadHistory() {
try {
const r = await fetch('/api/history?limit=50').then(r => r.json());
history.value = r.items;
} catch (e) {}
}
onMounted(() => {
loadSteps();
loadStandards();
loadHistory();
});
return {
form, testing, submitting, connected, testResult,
stepsList, standards,
currentJob, jobStatus, statusTagType,
logs, logOpen,
jobResult, activeTab,
history,
testConnection, startJob, cancelJob,
downloadReport, downloadHistoryReport,
loadHistory, loadStandards,
};
}
};
const app = createApp(App);
app.use(ElementPlus);
for (const [key, comp] of Object.entries(ElementPlusIconsVue)) {
app.component(key, comp);
}
app.mount('#app');
</script>
</body>
</html>
\ No newline at end of file
web/static/lib/@element-plus/icons-vue/index.iife.min.js 0 → 100644
View file @ 7b4bd4a8
This source diff could not be displayed because it is too large. You can view the blob instead.
web/static/lib/element-plus/index.css 0 → 100644
View file @ 7b4bd4a8
This source diff could not be displayed because it is too large. You can view the blob instead.
web/static/lib/element-plus/index.full.min.js 0 → 100644
View file @ 7b4bd4a8
This source diff could not be displayed because it is too large. You can view the blob instead.
web/static/lib/vue.global.prod.js 0 → 100644
View file @ 7b4bd4a8
This source diff could not be displayed because it is too large. You can view the blob instead.
web/static/style.css 0 → 100644
View file @ 7b4bd4a8
/* 数据治理工具样式 */
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC",
"Hiragino Sans GB", "Microsoft YaHei", sans-serif;
background: #f5f7fa;
color: #303133;
}
.app-header {
background: linear-gradient(135deg, #1890ff 0%, #722ed1 100%);
color: white;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
height: 64px !important;
display: flex;
align-items: center;
}
.header-inner {
width: 100%;
display: flex;
justify-content: space-between;
align-items: center;
}
.brand {
display: flex;
align-items: center;
gap: 12px;
font-size: 18px;
}
.brand .el-icon {
font-size: 28px;
}
.brand-title {
font-weight: 600;
}
.header-actions .el-button {
color: white;
background-color: rgba(255, 255, 255, 0.15);
border-color: rgba(255, 255, 255, 0.5);
}
.header-actions .el-button:hover,
.header-actions .el-button:focus {
color: white;
background-color: rgba(255, 255, 255, 0.28);
border-color: rgba(255, 255, 255, 0.8);
}
.header-actions .el-button:active {
color: white;
background-color: rgba(255, 255, 255, 0.35);
}
/* Element Plus 把文字包在 span 里，确保 span 也是白色 */
.header-actions .el-button span {
color: white;
}
.app-main {
max-width: 1400px;
margin: 0 auto;
padding: 20px;
}
.section-card {
margin-bottom: 20px;
border-radius: 8px;
}
.card-header {
display: flex;
align-items: center;
gap: 10px;
font-size: 16px;
font-weight: 600;
}
.card-header .el-icon {
font-size: 20px;
color: #1890ff;
}
.card-title {
flex: 1;
}
.form-hint {
color: #909399;
font-size: 12px;
margin-left: 8px;
}
/* ── 进度条 ── */
.progress-info {
display: flex;
gap: 24px;
margin: 12px 0;
font-size: 14px;
color: #606266;
}
.error-info {
color: #f56c6c;
font-weight: 500;
}
.log-collapse {
margin-top: 12px;
}
.log-container {
max-height: 400px;
overflow-y: auto;
background: #1e1e1e;
color: #d4d4d4;
font-family: "Consolas", "Monaco", monospace;
font-size: 12px;
padding: 12px;
border-radius: 4px;
}
.log-line {
padding: 2px 0;
display: flex;
gap: 8px;
align-items: baseline;
}
.log-ts {
color: #858585;
flex-shrink: 0;
}
.log-level {
flex-shrink: 0;
font-weight: 600;
width: 60px;
}
.log-step {
color: #dcdcaa;
flex-shrink: 0;
font-weight: 600;
}
.log-msg {
flex: 1;
word-break: break-all;
}
.log-info .log-level { color: #569cd6; }
.log-warn .log-level { color: #dcdcaa; }
.log-error .log-level { color: #f48771; }
.log-error { background: rgba(244, 135, 113, 0.1); }
/* ── KPI 卡片 ── */
.kpi-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
gap: 16px;
margin-bottom: 20px;
}
.kpi-card {
background: white;
border-radius: 8px;
padding: 16px;
text-align: center;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.06);
border-left: 4px solid #1890ff;
transition: all 0.2s;
}
.kpi-card:hover {
transform: translateY(-2px);
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
}
.kpi-card.kpi-tables { border-left-color: #1890ff; }
.kpi-card.kpi-fields { border-left-color: #13c2c2; }
.kpi-card.kpi-empty { border-left-color: #faad14; }
.kpi-card.kpi-missing { border-left-color: #fa8c16; }
.kpi-card.kpi-length { border-left-color: #f5222d; }
.kpi-card.kpi-std { border-left-color: #eb2f96; }
.kpi-card.kpi-merge { border-left-color: #722ed1; }
.kpi-card.kpi-redundancy { border-left-color: #52c41a; }
.kpi-value {
font-size: 32px;
font-weight: 700;
color: #303133;
line-height: 1.2;
}
.kpi-label {
color: #909399;
font-size: 13px;
margin-top: 4px;
}
.muted {
color: #c0c4cc;
}
code {
background: #f5f7fa;
padding: 2px 6px;
border-radius: 3px;
font-family: "Consolas", "Monaco", monospace;
font-size: 12px;
}
/* Element Plus 表格紧凑模式 */
.el-table {
font-size: 13px;
}
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment