Skip to content

D
db-tools
Commit 5c37925e authored by Data Governance Dev's avatar Data Governance Dev
Browse files
Options

feat(standards): 国标分析拆分为 24 个细粒度 indicator，并注册为独立 step 

国标分析改造两阶段，合并提交：

【阶段 1】拆分 + 新增 5 类分析对象
- STD-001 身份证 → IND-001-a/b/c/d (格式/校验位/出生日期/15位兼容)
- STD-002 USCC  → IND-002-a/b   (格式/校验位)
- STD-003 手机   → IND-003-a/b   (格式/号段)
- STD-004 区划   → IND-004-a/b   (格式/编码存在)
- 新增 IND-101 证件类型枚举（5 值）
- 新增 IND-201~204 4 种证件号码（身份证/护照/港澳台居住证/外国人永居）
- 新增 IND-301 跨字段 (证件类型+号码一致性)
- 新增 IND-302 跨表 (证件唯一性，GROUP BY HAVING)
- 新增 IND-401/402 姓名 (字符集/长度)
- 新增 IND-501/502 出生日期 (格式/年龄范围, 性别推断退休年龄)
- 新增 IND-601/602/603 区划/民族/婚姻

基础设施：
- BaseStandard 加 group/severity/depends_on 类属性 + validate_rows() 默认实现
- registry.list_all_standards / list_standards_by_group
- data_loader.py 惰性加载 data/*.json + extracted_standards.json
- 新增 data/{id_type,marriage,gender,retirement_age}.json
- 旧 STD-001~004 标 [已拆分], applies_to_fields=[] (保留兼容)

【阶段 2】每个 indicator 注册为独立 step
- registry.py 新增 load_standard_class / indicator_step_id /
  step_id_to_indicator_id / list_step_ids 工具函数
- orchestrator.py 动态按 group 顺序注册 24 个 indicator step
  (国标字段规范=100, 证件信息=200, 身份信息=300, 民政信息=400)
- step7_standards.py 新增 run_step7_for_indicator，每个 indicator
  独立 section_key (standards_ind_XXX)，独立 tab
- 旧 run_step7() 保留 (向后兼容: 跑全部按 4 group 输出)

校验层：
- IND-301 用 sample_field_pairs.sql
- IND-302 用 sample_field_groups.sql
- 84 单测覆盖所有 indicator (tests/test_indicators.py)

协议层：
- 沿用上轮的 tab_protocol；每个 indicator 一个独立 tab
  (1 KPI 违规字段 + 1 KPI 检查字段 + 2 表: 指标汇总 + 违规明细)

详细工作记录见 docs/WORKLOG.md。
parent 714f714a
Hide whitespace changes
Inline Side-by-side
Showing with 3379 additions and 320 deletions
docs/TAB_PROTOCOL.md
View file @ 5c37925e
......@@ -340,4 +340,108 @@ register_step(
## 9. 协议版本
- 当前：`1.0`
- 后续扩展方向：chart（`render.kind="chart"`）、inline-edit、`custom` 特判
\ No newline at end of file
- 后续扩展方向：chart（`render.kind="chart"`）、inline-edit、`custom` 特判
---
## 10. 多 tab by group 示例（step7 国标校验 v2）
step7 国标校验从「单 tab 4 个国标」重构为「4 tab × 24 indicator」的版本。每个 tab 对应一个分析主题（group），tab 内仍走「按 indicator 聚合 + 违规明细 + 字段联动」三件套。
### 后端：按 group 拆 sections
```python
# web/core/step_impl/step7_standards.py
from standards.registry import list_standards_by_group
_TABS_BY_GROUP = {
"国标字段规范": _make_group_tab(key="standards_gb_format", title="国标字段规范", ...),
"证件信息": _make_group_tab(key="standards_document", title="证件信息", ...),
"身份信息": _make_group_tab(key="standards_personal", title="身份信息", ...),
"民政信息": _make_group_tab(key="standards_civil", title="民政信息", ...),
}
def _wrap(per_group):
"""per_group = {group_name: data_dict} → 多 tab 注入。"""
sections = {}
for group, data in per_group.items():
tab = _TABS_BY_GROUP.get(group)
sections[group] = inject_protocol(data, step_id="standards", tabs=[tab])
return {"sections": sections}
```
`per_group` 字典的 key 就是 sections 的 key，前端 `displayedTabs` 会自动为每个 section 生成一个 tab。
### 前端：零改动
`web/static/index.html` 的 `displayedTabs` 计算属性已经在迭代 `sections[*]._protocol.tabs`，新增 tab 自动出现。无需改前端代码。
### Indicator 元数据：group / severity / depends_on
`BaseStandard` 加了 3 个类属性（[standards/base.py](standards/base.py)）：
```python
class BaseStandard(ABC):
standard_id: str = ""
applies_to_fields: list[str] | None = None
description: str = ""
# ── 多 tab 拆分的元数据 ──
group: str = "通用" # 国标字段规范 / 证件信息 / 身份信息 / 民政信息 / 通用
severity: str = "warning" # warning | error
depends_on: list[str] | None = None # 跨字段依赖（如 IND-301 → [IND-201, IND-202]）
```
子类按 group 归类即可：
```python
class IdCardFormatIndicator(BaseStandard):
standard_id = "IND-001-a"
group = "国标字段规范"
severity = "warning"
# applies_to_fields = [...] 命中数据库字段
```
### 跨字段 / 跨行校验（IND-301 / IND-302）
单值校验走 `validate(value)`。跨字段校验覆写 `validate_rows(rows)`：
```python
class IdTypeNumberConsistencyIndicator(BaseStandard):
standard_id = "IND-301"
applies_to_fields = [] # ← 不参与单字段自动发现
group = "证件信息"
depends_on = ["IND-101", "IND-201", "IND-202", "IND-203", "IND-204"]
def validate(self, value):
# 单值模式不适用
return ValidationResult(True, "", value, "IND-301 仅支持跨字段校验", self.standard_id)
def validate_rows(self, rows):
# rows 每条含 {id_type, id_no}
out = []
for row in rows:
...
if not indicator.validate(row["id_no"]).valid:
out.append({"value": row["id_no"], "reason": "..."})
return out
```
step7 检测到 indicator 的 `validate` 是「no-op」就走第二轮 SQL 抽样同表多字段，调 `validate_rows`。
跨表聚合（IND-302）走 `GROUP BY HAVING` SQL：
```sql
-- web/sql/standards/sample_field_groups.sql
SELECT ${type_col}, ${no_col}, COUNT(*) AS dup_count
FROM ${table}
WHERE ${type_col} IS NOT NULL AND ${no_col} IS NOT NULL
GROUP BY ${type_col}, ${no_col}
HAVING COUNT(*) >= ${min_count}
ORDER BY dup_count DESC
LIMIT 200
```
### 验证
- **单测**：`tests/test_indicators.py` —— 84 个测试（每 indicator 1-2 合规 + 1-2 违规 + 协议形状）
- **前端编译**：`scripts/verify_template.cjs` —— 验证 `<div id="app">` 在 Vue 3 prod build 下编译产物合法 JS
- **端到端**：勾选「国家标准校验」step，分析结果应出现 4 个 tab（国标字段规范 / 证件信息 / 身份信息 / 民政信息）
\ No newline at end of file
docs/WORKLOG.md
View file @ 5c37925e
......@@ -2,6 +2,76 @@
> 任务做完一次记一次。最近的在最上面。
## 2026-08-07 · 国标分析：拆分细粒度 indicator + 新增 5 类分析对象
### 用户原话
> 把现有的国标分析拆分车一个个的小的分析指标，然后再加上下面这些分析对象：
> 1. 证件类型（5 值枚举，不可自定义）
> 2. 证件号码（身份证为核心：18位正则+校验位+出生日期；护照/港澳台居住证/永居证）
> 3. 姓名（汉字+少数民族点分隔，长度2-20）
> 4. 出生日期（YYYYMMDD + 16-法定退休年龄）
> 5. 户籍所在地 / 民族 / 婚姻状况
### 改动总览
| 类别 | 内容 |
|------|------|
| **拆分现有 4 个** | STD-001 身份证 → IND-001-a/b/c/d（格式/校验位/出生日期/15位兼容）<br>STD-002 USCC → IND-002-a/b（格式/校验位）<br>STD-003 手机号 → IND-003-a/b（格式/号段）<br>STD-004 区划 → IND-004-a/b（格式/编码存在）|
| **新增 14 个** | IND-101 证件类型枚举<br>IND-201~204 4 种证件号码（身份证/护照/港澳台居住证/外国人永居）<br>IND-301 跨字段：证件类型+号码一致性<br>IND-302 跨表：证件唯一性<br>IND-401/402 姓名（字符集/长度）<br>IND-501/502 出生日期（格式/年龄范围）<br>IND-601/602/603 区划/民族/婚姻 |
| **拆分后总数** | 24 indicator + 4 deprecated（占位保留）= 28 标准插件 |
| **数据文件** | 新增 `standards/data/{id_type,marriage,gender,retirement_age}.json`<br>复用 `national_standards/extracted_standards.json`（GB/T 2260 / 3304 / 6565 / ISO 3166） |
| **执行模式** | 单值 `validate(value)` + 跨字段 `validate_rows(rows)`（IND-301）+ 跨表 SQL `GROUP BY HAVING`（IND-302）|
| **协议层** | `_STANDARDS_TAB` 单 tab → 4 个 group tab（国标字段规范 / 证件信息 / 身份信息 / 民政信息）|
| **执行轮次** | Round 1 单值校验 → Round 2 跨字段（IND-301）→ Round 2 跨表聚合（IND-302）|
| **测试** | `tests/test_indicators.py` —— 84 个测试，全绿 |
### 关键文件
- [standards/base.py:29-78](standards/base.py#L29-L78) — `BaseStandard` 加 `group` / `severity` / `depends_on` 类属性 + `validate_rows(rows)` 默认实现
- [standards/registry.py:46-86](standards/registry.py#L46-L86) — `list_all_standards()` 返回值加 group/severity；新增 `list_standards_by_group()`
- [standards/data_loader.py](standards/data_loader.py) — 惰性加载 `data/*.json` + `extracted_standards.json`
- [standards/std_001~004_*.py](standards/) — 旧 4 个标 `[已拆分]` + `applies_to_fields=[]`
- [standards/ind_001a_*.py ~ ind_603_*.py](standards/) — 24 个新 indicator
- [web/sql/standards/sample_field_pairs.sql](web/sql/standards/sample_field_pairs.sql) — IND-301 用
- [web/sql/standards/sample_field_groups.sql](web/sql/standards/sample_field_groups.sql) — IND-302 用
- [web/core/step_impl/step7_standards.py](web/core/step_impl/step7_standards.py) — 4 个 group tab + Round 1/2 编排
- [tests/test_indicators.py](tests/test_indicators.py) — 84 测试
### 调研备忘（影响算法决策）
| 调研项 | 结论 | 影响 |
|--------|------|------|
| GB 11643-1999 身份证校验位 | ISO 7064 MOD 11-2；Wi = 2^(18-i) mod 11；映射表 `1,0,X,9,8,7,6,5,4,3,2` | IND-001-b / IND-201 沿用 |
| GB 32100-2015 USCC | MOD 31-3；31 字符集（不含 I/O/Z/S/V） | IND-002-b 沿用 |
| GB/T 2260-2007 行政区划 | 6 位省 2+地 2+县 2；港澳台前缀 71/81/82（83 不在表内）| IND-004-b / IND-601 |
| 港澳台居民居住证 | 地址码 810000/820000/830000（83 由国务院台办配；GB/T 2260 只含 81/82）| IND-203 接受 83 前缀 |
| 中国护照 | 9 位：G+8 / E+8 / E+1字母+7 / PE/SE/DE+7；字母排除 I/O；无公开校验位 | IND-202 格式校验不做算法 |
| 外国人永居证 | 旧 15 位 LLL+12数字 / 新 18 位 9+17数字；官方未公开校验位 | IND-204 格式校验 |
| 港澳居民来往内地通行证（回乡证）| 9 位 H/M+8 位数字（旧版 11 位含换证次数）| **未纳入本轮**（不在用户列出的 5 类内）|
| 台湾居民来往大陆通行证（台胞证）| 8 位数字（2008-起）；旧版 10+括号 | **未纳入本轮**（同上）|
| GB/T 3304-1991 民族 | 01-56 + 补充码 97/98；81 穿青人非标准正文 | IND-602 接受 97/98 |
| GB/T 2261.2-2003 婚姻 | 10/20/21/22/23/30/40/90（GB/T 4761 是家庭关系 ≠ 婚姻）| IND-603 |
| GB/T 2261.1-2003 性别 | 0/1/2/9（与 ISO/IEC 5218 对齐）| 未纳入 indicator（暂不需要单值校验；IND-502 用性别推断退休年龄）|
| 国发〔1978〕104 号 退休 | 男 60 / 女工 50 / 女干部 55 | IND-502 |
| GA/T 2000.156-2016 证件类型 | 三位代码表；用户给简化 5 值 | IND-101 接受用户给定 5 值 |
### 验证
```
$ python -m pytest tests/test_indicators.py -q
84 passed in 0.17s
$ node scripts/verify_template.cjs
✓ compiled #0 (69296 chars) is valid JS
2 passed, 0 failed
$ node scripts/verify_render.cjs
12 passed, 0 failed
```
### 注意点 / 后续工作
1. IND-301 当前实现 `validate_rows()` 按 (id_type, id_no) 配对扫描 —— 同表多对字段时遍历所有组合；性能上 OK，500 行内抽样足矣
2. IND-302 的 GROUP BY 没考虑跨表（不同表里出现相同 (type, no) 不会合并）；按用户需求「禁止重复开户」严格说应跨表 —— 留作下一轮
3. IND-502 的性别推断依赖「同表有 gender 字段」；若不同表需手动配对（可后续做关联字段配置）
4. IND-401 接受 ASCII 字母（拼音名）；严格模式可改为「仅汉字」
## 2026-08-07 · 数据字典浏览器：表名 / 表注释 拆成两列
### 问题
......@@ -442,5 +512,88 @@ $ node scripts/verify_template.cjs
2. **v-else 在长 v-else-if 链末尾要谨慎**：超过 ~5 层以后，Vue 3.5.x 的编译器会输出 `: ()`。要么用 `v-else-if="!cond"` 显式收尾，要么减少分支数。
3. **Function 劫持是个便宜的验证手段**：想要捕获「模板编译 → render 函数构造」这一步的 SyntaxError，劫持 `window.Function` 抓大于 1KB 的代码体即可。
---
## [未提交] indicator 拆分为独立 step（2026-08-07）
### 背景
上一轮把粗粒度「国家标准校验」拆成 24 个细粒度 indicator + 4 个 group tab 显示，但用户反馈：
> 我没看到拆分，我的意思是这一个个小的分析指标都是可选。我知道这里会变得很长，之后会修改为树形结构，先拆分成一个个小的分析指标。
所以这轮要把 UI 的「要跑的检查」从 1 个 checkbox（国家标准校验）变成 **24 个 indicator checkbox**，每个用户可以单独勾选。
### 改动
#### 1. 每个 indicator 注册为独立 orchestrator step
`standards/registry.py` 新增 3 个工具函数：
- `load_standard_class(standard_id)`：按 "IND-001-a" 找对应类
- `indicator_step_id(standard_id)`：`IND-001-a` → `std_ind_001_a`
- `step_id_to_indicator_id(step_id)`：反向（保留大小写：IND-001-a 而非 IND-001-A）
- `list_step_ids()`：列出所有 indicator 的 (step_id, metadata)
`web/core/orchestrator.py` 在基础 4 个 step 注册后，**动态循环注册每个 indicator**：
```python
_GROUP_ORDER_START = {"国标字段规范": 100, "证件信息": 200, "身份信息": 300, "民政信息": 400, "通用": 500}
def _register_indicator_steps():
prev_group = None; same_group_count = 0; base_order = 0
for sid, meta in _INDICATOR_STEPS:
if meta["id"].startswith("STD-"): # 跳过已 deprecated 的旧插件
continue
group = meta.get("group", "通用")
if group != prev_group:
same_group_count = 0
base_order = _GROUP_ORDER_START[group]
prev_group = group
same_group_count += 1
order = base_order + same_group_count
register_step(step_id=sid, title=f"{meta['id']} · {meta['name']}", ...)
```
注册结果：
- 4 个基础 step（merge_redundancy / empty_fields / missing_comments / length_check）
- **24 个 indicator step**（IND-001-a ~ IND-603，跨 4 个 group）
- 已 deprecated 的 STD-001~004 不注册（避免 UI 出现冗余）
#### 2. step7_standards.py 拆为 per-indicator 执行
新加 `run_step7_for_indicator(standard_id, cfg, dict_data, log, table_filter)`：
- 单值类（22 个 indicator）→ `_run_round1_one(cfg, by_name, std_instance, log)`
- 跨字段 IND-301 → `_run_round2_cross_field_one(cfg, by_name, log)`
- 跨表聚合 IND-302 → `_run_round2_uniqueness_one(cfg, by_name, log)`
- 未知 ID → `_empty_indicator_step_result(standard_id, reason)`
返回 `{"section_key": "standards_ind_XXX", "data": {...}}`，每个 indicator 独立 section / 独立 tab。
向后兼容：`run_step7(cfg, dict_data, log, table_filter)` 仍可调（跑全部 + 按 group 输出 4 tab，老逻辑走 `_run_round1` + `_run_round2_cross_field` + `_run_round2_uniqueness`）。
#### 3. 每个 indicator tab 协议
`_make_single_indicator_tab(key, title)` —— 1 KPI「违规字段」+ 1 KPI「检查字段」 + 2 表（指标汇总 + 违规明细）。
没有跨表联动（每个 indicator 独立 tab，不需要联动）。
### 验证
```
$ python -m pytest tests/test_indicators.py -v
=== 84 passed in 0.13s ===
$ python -c "from web.core.orchestrator import get_step_defs; print(len(get_step_defs()))"
28 # 4 基础 + 24 indicator（无 deprecated）
$ curl /api/steps
Total: 28 steps # 4 基础 + 24 indicator checkbox
```
每个 indicator 可单独勾选，结果以独立 section 写入 sections，按 section_key 渲染成独立 tab。
### 不在本轮范围
- **树形 UI**：用户提到后续改成树形结构；本次先满足"独立勾选"
- **section 跨 indicator 合并**：本轮 24 个 step 各自写独立 section；如需合并回 4 个 group tab 需 orchestrator 写合并逻辑（待评估）
- **STD-001~004 完全删除**：保留文件但 `applies_to_fields=[]`，不出现在 UI 里（用户历史数据可能引用）
### 未提交
按用户要求，待 review 后再提交。
standards/base.py
View file @ 5c37925e
......@@ -4,7 +4,7 @@ from __future__ import annotations
import re
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from dataclasses import dataclass
@dataclass
......@@ -30,24 +30,74 @@ class BaseStandard(ABC):
"""标准插件基类
子类需实现：
- standard_id: 标准编号，如 "STD-001"
- standard_id: 标准编号，如 "IND-001-a"
- standard_name: 标准中文名
- applies_to_fields: 适用的字段名（list）
- validate(value): 单值校验逻辑
- describe(): 给人看的描述（可选）
扩展元数据（可选覆盖）：
- group: "国标字段规范" / "证件信息" / "身份信息" / "民政信息" / "通用"
- severity: "warning" | "error" —— 影响前端 KPI 染色
- depends_on: 列表，声明跨字段依赖（如 IND-301 依赖 IND-201/202）
注意：本类不是 @dataclass（与 ABC 抽象方法不兼容），所以 list 字段
不能用 field(default_factory=list) —— 用 None + __init_subclass__ 兜底，
或干脆让子类各自覆写。基类取值时做 None → [] 的兼容。
"""
standard_id: str = ""
standard_name: str = ""
applies_to_fields: list[str] = field(default_factory=list)
applies_to_fields: list[str] | None = None
description: str = ""
# ── 扩展元数据（向后兼容 —— 缺省走「通用 / warning」） ──
group: str = "通用"
severity: str = "warning"
depends_on: list[str] | None = None
def __init__(self):
# 子类可省略 __init__；基类负责把 None 兜底为 []
if self.applies_to_fields is None:
self.applies_to_fields = []
if self.depends_on is None:
self.depends_on = []
# ── 抽象方法 ──
@abstractmethod
def validate(self, value: str) -> ValidationResult:
"""校验单个值，返回 ValidationResult"""
...
# ── 可选覆写：跨字段 / 跨行校验 ──
# 默认实现：rows 每条独立调 validate(value)
# 跨字段校验器（如 IND-301 类型+号码一致性）可覆写此方法
# 跨表聚合校验器（如 IND-302 唯一性）走 SQL，不走这里
def validate_rows(self, rows: list[dict]) -> list[dict]:
"""批量校验多行，每行至少含 indicator 的若干字段。
返回违规明细 dict 列表（与 validate(value).to_dict() 同结构，
但允许带额外字段如 inconsistency_type）。
默认实现：每行取「主字段」（=applies_to_fields[0]）调 validate(value)。
"""
if not self.applies_to_fields:
return []
primary = self.applies_to_fields[0]
out = []
for row in rows:
v = row.get(primary)
if v is None or v == "":
continue
r = self.validate(str(v))
if not r.valid:
out.append({
"value": str(v)[:50],
"reason": r.reason,
"row_keys": list(row.keys()),
})
return out
# ── 辅助：批量校验多个值，统计违规率 ──
def validate_batch(self, values: list[str], field_name: str = "") -> dict:
if not values:
......
standards/data/gender_codes.json 0 → 100644
View file @ 5c37925e
{
"_doc": "人的性别代码 —— GB/T 2261.1-2003《个人基本信息分类与代码 第1部分：人的性别代码》",
"std_id": "GB/T 2261.1-2003",
"enum": {
"0": "未知的性别",
"1": "男性",
"2": "女性",
"9": "未说明的性别"
}
}
\ No newline at end of file
standards/data/id_type_enum.json 0 → 100644
View file @ 5c37925e
{
"_doc": "证件类型枚举 —— 用户业务场景下整理。完整 GA/T 2000.156-2016 三位代码表较繁，本项目按用户给定的 5 值落地。",
"enum": {
"01": {"name": "居民身份证", "std": "GB 11643-1999"},
"02": {"name": "军官证", "std": "（建议核实）"},
"03": {"name": "护照", "std": "ICAO Doc 9303 / 中华人民共和国Passport法"},
"04": {"name": "外国人永久居留身份证", "std": "公安部令第118号 / 2023-12-01新版五星卡"},
"05": {"name": "港澳台居民居住证", "std": "国办发〔2018〕81号"}
}
}
\ No newline at end of file
standards/data/marriage_codes.json 0 → 100644
View file @ 5c37925e
{
"_doc": "婚姻状况代码 —— GB/T 2261.2-2003《个人基本信息分类与代码 第2部分：婚姻状况代码》",
"std_id": "GB/T 2261.2-2003",
"enum": {
"10": "未婚",
"20": "已婚",
"21": "初婚",
"22": "再婚",
"23": "复婚",
"30": "丧偶",
"40": "离婚",
"90": "未说明的婚姻状况"
}
}
\ No newline at end of file
standards/data/retirement_age.json 0 → 100644
View file @ 5c37925e
{
"_doc": "法定退休年龄 —— 国务院《国务院关于工人退休、退职的暂行办法》(国发〔1978〕104号) 及后续干部/职工分类调整",
"rules": {
"default": {"age": 60, "note": "兜底（找不到性别字段时用）"},
"male": {"age": 60, "note": "男职工/干部 60 岁"},
"female_worker": {"age": 50, "note": "女工人 50 岁"},
"female_cadre": {"age": 55, "note": "女干部 55 岁"}
},
"policy_notes": "1978 年国发〔1978〕104 号确立男 60 / 女 50、女干部 55 的基本框架；近年渐进式延迟退休政策（2025-2026 实施）正在调整，本表暂以原框架为准。"
}
\ No newline at end of file
standards/data_loader.py 0 → 100644
View file @ 5c37925e
"""数据加载器 —— 惰性加载 + 缓存 standards/data/ 下的枚举 JSON。
设计目标：
- 启动时不读文件，第一次访问时才 load
- 同一份数据多次访问走缓存
- 加载失败抛 RuntimeError 并给出明确路径
"""
from __future__ import annotations
import json
from functools import lru_cache
from pathlib import Path
_DATA_DIR = Path(__file__).parent / "data"
_EXTRACTED_PATH = Path(__file__).resolve().parents[1] / "national_standards" / "extracted_standards.json"
def _load_json(path: Path) -> dict:
if not path.exists():
raise RuntimeError(f"数据文件不存在: {path}")
# extracted_standards.json 是 GBK 编码（内嵌 UTF-8 BOM？实际是 GBK）
# 其他 data/*.json 是 UTF-8
if path.name == "extracted_standards.json":
# 含 4 字节 GBK 序列；用 utf-8 errors=replace 兜底
with open(path, "r", encoding="utf-8", errors="replace") as f:
return json.load(f)
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
# ── 单文件加载 ──
@lru_cache(maxsize=8)
def load_id_type_enum() -> dict[str, dict]:
"""证件类型枚举（id_type_enum.json → {code: {name, std}}）"""
return _load_json(_DATA_DIR / "id_type_enum.json")["enum"]
@lru_cache(maxsize=8)
def load_marriage_codes() -> dict[str, str]:
"""GB/T 2261.2-2003 婚姻状况枚举 → {code: name}"""
return _load_json(_DATA_DIR / "marriage_codes.json")["enum"]
@lru_cache(maxsize=8)
def load_gender_codes() -> dict[str, str]:
"""GB/T 2261.1-2003 性别枚举 → {code: name}"""
return _load_json(_DATA_DIR / "gender_codes.json")["enum"]
@lru_cache(maxsize=8)
def load_retirement_age_rules() -> dict[str, dict]:
"""法定退休年龄规则 → {rule_key: {age, note}}"""
return _load_json(_DATA_DIR / "retirement_age.json")["rules"]
# ── 复用 national_standards/extracted_standards.json ──
@lru_cache(maxsize=4)
def load_extracted_standards() -> dict:
"""整份 extracted_standards.json（GB/T 2260 / 3304 / 4658 / 6565 / ISO 3166 / ISO 4217 / GB/T 4754）"""
return _load_json(_EXTRACTED_PATH)
@lru_cache(maxsize=8)
def load_district_codes() -> set[str]:
"""GB/T 2260-2007 区划代码集合（6 位）"""
extracted = load_extracted_standards()
return {r["code"] for r in extracted["GB_T_2260_2007"]["records"]}
@lru_cache(maxsize=8)
def load_district_province_codes() -> set[str]:
"""省级代码集合（前 2 位）"""
return {code[:2] for code in load_district_codes()}
@lru_cache(maxsize=8)
def load_ethnicity_codes() -> set[str]:
"""GB/T 3304-1991 民族代码集合（2 位数字）"""
extracted = load_extracted_standards()
return {r["code"] for r in extracted["GB_T_3304_1991"]["records"]}
@lru_cache(maxsize=8)
def load_iso_country_codes_3() -> set[str]:
"""ISO 3166-1 alpha-3 国家代码集合（3 位字母，外国人永居证国别码用）"""
extracted = load_extracted_standards()
return {r["alpha3"] for r in extracted["ISO_3166_1"]["records"] if r.get("alpha3")}
@lru_cache(maxsize=8)
def load_iso_country_codes_alpha2() -> set[str]:
"""ISO 3166-1 alpha-2 国家代码集合（2 位字母）"""
extracted = load_extracted_standards()
return {r["alpha2"] for r in extracted["ISO_3166_1"]["records"] if r.get("alpha2")}
@lru_cache(maxsize=8)
def load_iso_country_codes_numeric() -> set[str]:
"""ISO 3166-1 numeric-3 国家代码集合（3 位数字，零填充）"""
extracted = load_extracted_standards()
return {r["numeric"] for r in extracted["ISO_3166_1"]["records"] if r.get("numeric")}
# ── 验证快捷接口 ──
def is_valid_id_type(code: str) -> bool:
return code in load_id_type_enum()
def is_valid_marriage(code: str) -> bool:
return code in load_marriage_codes()
def is_valid_gender(code: str) -> bool:
return code in load_gender_codes()
def is_valid_district_6(code: str) -> bool:
return code in load_district_codes()
def is_valid_district_prefix_2(code: str) -> bool:
return code[:2] in load_district_province_codes()
def is_valid_ethnicity(code: str) -> bool:
return code in load_ethnicity_codes()
# ── 调试入口 ──
if __name__ == "__main__":
print("id_type_enum:", len(load_id_type_enum()), "codes")
print("marriage_codes:", list(load_marriage_codes().keys()))
print("district_codes:", len(load_district_codes()))
print("district_province_codes:", sorted(load_district_province_codes()))
print("ethnicity_codes count:", len(load_ethnicity_codes()))
\ No newline at end of file
standards/ind_001a_id_card_format.py 0 → 100644
View file @ 5c37925e
"""IND-001-a 身份证号格式（18 位 + 正则）
拆分自 STD-001。只检查「长度 + 6+8+3+1 结构」，不做校验位 / 出生日期。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class IdCardFormatIndicator(BaseStandard):
standard_id = "IND-001-a"
standard_name = "GB 11643-1999 身份证号格式"
applies_to_fields = ["id_card", "id_card_no", "id_number", "identity_card"]
description = "长度 = 18；6 位地址 + 8 位出生日期 + 3 位顺序码 + 1 位校验位"
group = "国标字段规范"
severity = "warning"
_REGEX = re.compile(r"^[1-9]\d{5}(19|20)\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}[\dXx]$")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 18:
return ValidationResult(False, "", value, f"长度 {len(value)} ≠ 18", self.standard_id)
if not self._REGEX.match(value):
return ValidationResult(False, "", value, "不符合 6+8+3+1 字符结构", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_001b_id_card_checksum.py 0 → 100644
View file @ 5c37925e
"""IND-001-b 身份证号校验位（ISO 7064 MOD 11-2）
拆分自 STD-001。假定格式已通过（由 IND-001-a 保证），仅校验最后一位。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
class IdCardChecksumIndicator(BaseStandard):
standard_id = "IND-001-b"
standard_name = "GB 11643-1999 身份证校验位"
applies_to_fields = ["id_card", "id_card_no", "id_number", "identity_card"]
description = "ISO 7064 MOD 11-2 加权因子：7,9,10,5,8,4,2,1,6,3,7,9,10,5,8,4,2"
group = "国标字段规范"
severity = "warning"
_WEIGHTS = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]
_CHECKSUM_MAP = ["1", "0", "X", "9", "8", "7", "6", "5", "4", "3", "2"]
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 18:
return ValidationResult(
False, "", value,
"校验位要求 18 位（请先确认 IND-001-a 格式）",
self.standard_id,
)
expected = self._calc(value[:17])
if value[17].upper() != expected:
return ValidationResult(
False, "", value,
f"校验位错误，应为 {expected}",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
def _calc(self, seventeen: str) -> str:
total = sum(int(c) * w for c, w in zip(seventeen, self._WEIGHTS))
return self._CHECKSUM_MAP[total % 11]
\ No newline at end of file
standards/ind_001c_id_card_birthdate.py 0 → 100644
View file @ 5c37925e
"""IND-001-c 身份证号出生日期合法性
拆分自 STD-001。仅校验 value[6:14] 是真实日期（含闰年）。
"""
from __future__ import annotations
from datetime import date
from .base import BaseStandard, ValidationResult
class IdCardBirthdateIndicator(BaseStandard):
standard_id = "IND-001-c"
standard_name = "GB 11643-1999 身份证出生日期"
applies_to_fields = ["id_card", "id_card_no", "id_number", "identity_card"]
description = "value[6:14] 必须能解析为合法日期（含闰年 2-29）"
group = "国标字段规范"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 18:
return ValidationResult(
False, "", value,
"出生日期要求 18 位完整号码",
self.standard_id,
)
try:
y, m, d = int(value[6:10]), int(value[10:12]), int(value[12:14])
date(y, m, d)
except ValueError:
return ValidationResult(
False, "", value,
f"出生日期 {value[6:14]} 不合法（含闰年 / 月日范围）",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_001d_id_card_15_legacy.py 0 → 100644
View file @ 5c37925e
"""IND-001-d 身份证 15 位老证兼容（warning 提示）
GB 11643-1989 老证为 15 位（6 位地址 + 6 位出生年月（YYMMDD）+ 3 位顺序码）。
本 indicator 命中即提示「存在 15 位存量」，不视为 error（可能为历史数据）。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class IdCard15LegacyIndicator(BaseStandard):
standard_id = "IND-001-d"
standard_name = "GB 11643-1989 身份证 15 位老证提示"
applies_to_fields = ["id_card", "id_card_no", "id_number", "identity_card"]
description = (
"命中即提示存在 15 位老证（已自 1999-07-01 停发）。"
" 仅 warning，不视作违规 —— 业务侧需评估是否升级为 18 位。"
)
group = "国标字段规范"
severity = "warning"
_REGEX_15 = re.compile(r"^[1-9]\d{5}\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}$")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) == 15 and self._REGEX_15.match(value):
return ValidationResult(
False, "", value,
"15 位老证，建议升级为 18 位（GB 11643-1999）",
self.standard_id,
)
# 非 15 位不算违规 —— 交给 IND-001-a/b/c
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_002a_uscc_format.py 0 → 100644
View file @ 5c37925e
"""IND-002-a USCC 格式（18 位 + 字符集 + 结构）
拆分自 STD-002。校验「长度 + 字符集 + 1+1+6+9+1 结构」。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class UsccFormatIndicator(BaseStandard):
standard_id = "IND-002-a"
standard_name = "GB 32100-2015 USCC 格式"
applies_to_fields = ["uscc", "credit_code", "social_credit_code", "unified_social_credit_code"]
description = "长度 18；31 字符集（不含 I/O/Z/S/V）；登记管理机关(1)+机构类别(1)+区划(6)+主体标识码(9)+校验码(1)"
group = "国标字段规范"
severity = "warning"
_REGEX = re.compile(r"^[1-9A-HJ-NP-RT-Y]\d[0-9A-HJ-NP-RT-Y]{6}[0-9A-HJ-NP-RT-Y]{9}[0-9A-HJ-NP-RT-Y]$")
_FORBIDDEN = set("IOZSV")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 18:
return ValidationResult(False, "", value, f"长度 {len(value)} ≠ 18", self.standard_id)
bad = set(value) & self._FORBIDDEN
if bad:
return ValidationResult(False, "", value, f"包含禁用字符 {''.join(sorted(bad))}", self.standard_id)
if not self._REGEX.match(value):
return ValidationResult(False, "", value, "不符合 1+1+6+9+1 结构", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_002b_uscc_checksum.py 0 → 100644
View file @ 5c37925e
"""IND-002-b USCC 校验位（MOD 31-3）
拆分自 STD-002。假定格式已通过（IND-002-a），仅校验末位。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
class UsccChecksumIndicator(BaseStandard):
standard_id = "IND-002-b"
standard_name = "GB 32100-2015 USCC 校验位"
applies_to_fields = ["uscc", "credit_code", "social_credit_code", "unified_social_credit_code"]
description = "MOD 31-3：Σ Ci × 3^i mod 31，取字符集倒索引"
group = "国标字段规范"
severity = "warning"
_CHARSET = "0123456789ABCDEFGHJKLMNPQRTUWXY" # 31 个字符
_CHAR_MAP = {c: i for i, c in enumerate(_CHARSET)}
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 18:
return ValidationResult(
False, "", value,
"校验位要求 18 位（请先确认 IND-002-a 格式）",
self.standard_id,
)
try:
expected = self._calc(value[:17])
except ValueError as e:
return ValidationResult(False, "", value, str(e), self.standard_id)
if value[17] != expected:
return ValidationResult(
False, "", value,
f"校验位错误，应为 {expected}",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
def _calc(self, seventeen: str) -> str:
total = 0
for i, c in enumerate(seventeen):
if c not in self._CHAR_MAP:
raise ValueError(f"字符 {c} 不在 GB 32100 字符集内")
total += self._CHAR_MAP[c] * (3 ** i)
idx = (31 - total % 31) % 31
return self._CHARSET[idx]
\ No newline at end of file
standards/ind_003a_mobile_format.py 0 → 100644
View file @ 5c37925e
"""IND-003-a 手机号格式（11 位 + 1[3-9] 开头 + 清洗）
拆分自 STD-003。仅做宽松长度 / 头部校验。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class MobileFormatIndicator(BaseStandard):
standard_id = "IND-003-a"
standard_name = "工信部 手机号格式"
applies_to_fields = [
"mobile", "phone", "mobile_phone", "tel", "telephone",
"contact_phone", "customer_mobile", "legal_person_mobile",
"notify_phone", "maintainer_phone", "manager_phone", "receiver_phone",
]
description = "11 位；以 1 开头，第 2 位 3-9；自动清洗 +86 / 空格 / 横线"
group = "国标字段规范"
severity = "warning"
_REGEX = re.compile(r"^1[3-9]\d{9}$")
@staticmethod
def _clean(raw: str) -> str:
v = re.sub(r"[\s\-]", "", raw)
if v.startswith("+86"):
v = v[3:]
elif v.startswith("86") and len(v) == 13:
v = v[2:]
return v
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = self._clean(value)
if len(v) != 11:
return ValidationResult(False, "", value, f"清洗后长度 {len(v)} ≠ 11", self.standard_id)
if not self._REGEX.match(v):
return ValidationResult(False, "", value, "不符合 1[3-9]XXXXXXXXX 格式", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_003b_mobile_segment.py 0 → 100644
View file @ 5c37925e
"""IND-003-b 手机号号段（更严格 —— 命中已知号段）
拆分自 STD-003。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class MobileSegmentIndicator(BaseStandard):
standard_id = "IND-003-b"
standard_name = "工信部 手机号号段"
applies_to_fields = [
"mobile", "phone", "mobile_phone", "tel", "telephone",
"contact_phone", "customer_mobile", "legal_person_mobile",
"notify_phone", "maintainer_phone", "manager_phone", "receiver_phone",
]
description = "在 IND-003-a 通过的前提下，进一步校验号段正则"
group = "国标字段规范"
severity = "warning"
_DETAILED_REGEX = re.compile(
r"^1(3\d|4[5-9]|5[0-35-9]|6[2567]|7[0-8]|8\d|9[0-35-9])\d{8}$"
)
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = re.sub(r"[\s\-]", "", value)
if v.startswith("+86"):
v = v[3:]
elif v.startswith("86") and len(v) == 13:
v = v[2:]
if not self._DETAILED_REGEX.match(v):
return ValidationResult(False, "", value, "号段不在已知号段表内", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_004a_xzqh_format.py 0 → 100644
View file @ 5c37925e
"""IND-004-a 行政区划代码格式（6 位 / 12 位 + 省级代码合法性）
拆分自 STD-004。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
from .data_loader import load_district_province_codes
class XzqhFormatIndicator(BaseStandard):
standard_id = "IND-004-a"
standard_name = "GB/T 2260 行政区划代码格式"
applies_to_fields = ["xzqhbm", "xzqhjb", "adcode", "district_code", "province_code", "city_code", "region_code"]
description = "6 位标准版或 12 位扩展版；前 2 位必须 ∈ GB/T 2260 省级代码表"
group = "民政信息" # 行政区划本身属于民政
severity = "warning"
_REGEX_6 = re.compile(r"^\d{6}$")
_REGEX_12 = re.compile(r"^\d{12}$")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if not (self._REGEX_6.match(value) or self._REGEX_12.match(value)):
return ValidationResult(False, "", value, "长度必须为 6 或 12 位数字", self.standard_id)
province = value[:2]
if province not in load_district_province_codes():
return ValidationResult(
False, "", value,
f"省级代码 {province} 不在 GB/T 2260 表内",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_004b_xzqh_exists.py 0 → 100644
View file @ 5c37925e
"""IND-004-b 行政区划编码存在性（6 位必须在 GB/T 2260-2007 表内）
拆分自 STD-004。在 IND-004-a 通过的前提下，进一步查 6 位代码是否真实存在。
12 位扩展版本 indicator 仅做长度合法性，不查全表（行政区划代码表只有 6 位版）。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import load_district_codes
class XzqhExistsIndicator(BaseStandard):
standard_id = "IND-004-b"
standard_name = "GB/T 2260 行政区划编码存在性"
applies_to_fields = ["xzqhbm", "xzqhjb", "adcode", "district_code", "province_code", "city_code", "region_code"]
description = "6 位代码必须在 GB/T 2260-2007 表内（共 3406 条）"
group = "民政信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
if len(value) != 6:
return ValidationResult(
False, "", value,
"仅校验 6 位标准版；12 位扩展版交由 IND-004-a 兜底",
self.standard_id,
)
if value not in load_district_codes():
return ValidationResult(
False, "", value,
"6 位代码不在 GB/T 2260-2007 表内",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_101_id_type.py 0 → 100644
View file @ 5c37925e
"""IND-101 证件类型枚举校验
值必须在 {01,02,03,04,05} 内（用户给定 5 值枚举，详见 data/id_type_enum.json）。
- 01=居民身份证
- 02=军官证
- 03=护照
- 04=外国人永久居留身份证
- 05=港澳台居民居住证
注：完整 GA/T 2000.156-2016 三位代码表更细（含军官证细分等），
本项目按用户业务侧给定的 5 值落地。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import is_valid_id_type
class IdTypeEnumIndicator(BaseStandard):
standard_id = "IND-101"
standard_name = "证件类型枚举"
applies_to_fields = ["id_type", "cert_type", "document_type", "zjlb"]
description = "代码必须在 {01 居民身份证 / 02 军官证 / 03 护照 / 04 永居证 / 05 港澳台居住证} 内"
group = "证件信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if not is_valid_id_type(v):
return ValidationResult(
False, "", value,
f"证件类型 {v!r} 不在标准枚举内（01/02/03/04/05）",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_201_id_number_id_card.py 0 → 100644
View file @ 5c37925e
"""IND-201 公民身份号码（18 位 GB 11643 + 兼容 15 位老证）
复用了 IND-001-a/b/c 的逻辑，但作为「以身份证为核心」的证件号码校验：
- 命中字段同时包含身份证 + 其他证件号时，由 IND-201~204 分流
- 这里与 IND-001-a/b/c 是有意重叠的（用于「证件号码」语义场景）
未做 ID 类型分发（type=01 → IND-201）—— 那由 IND-301 跨字段校验器处理。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class IdNumberIdCard18Indicator(BaseStandard):
standard_id = "IND-201"
standard_name = "证件号码-居民身份证 18 位（GB 11643-1999）"
applies_to_fields = ["id_no", "cert_no", "zjhm", "document_no"]
description = (
"18 位身份证：长度 + 正则 + 校验位 + 出生日期 全部通过。"
" 适用字段含 id_no / cert_no / zjhm / document_no（与 id_card 等并列，"
" 由前端/业务侧按语义挑选哪个 indicator 命中）"
)
group = "证件信息"
severity = "warning"
_REGEX_18 = re.compile(r"^[1-9]\d{5}(19|20)\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}[\dXx]$")
_REGEX_15 = re.compile(r"^[1-9]\d{5}\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}$")
_WEIGHTS = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]
_CHECKSUM_MAP = ["1", "0", "X", "9", "8", "7", "6", "5", "4", "3", "2"]
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if len(v) == 15:
if self._REGEX_15.match(v):
return ValidationResult(
False, "", value,
"15 位老证，请升级为 18 位（GB 11643-1999）",
self.standard_id,
)
return ValidationResult(False, "", value, "15 位格式错误", self.standard_id)
if len(v) != 18:
return ValidationResult(False, "", value, f"长度 {len(v)} ≠ 18", self.standard_id)
if not self._REGEX_18.match(v):
return ValidationResult(False, "", value, "不符合 18 位 6+8+3+1 结构", self.standard_id)
# 出生日期
from datetime import date
try:
date(int(v[6:10]), int(v[10:12]), int(v[12:14]))
except ValueError:
return ValidationResult(False, "", value, f"出生日期 {v[6:14]} 不合法", self.standard_id)
# 校验位
total = sum(int(c) * w for c, w in zip(v[:17], self._WEIGHTS))
expected = self._CHECKSUM_MAP[total % 11]
if v[17].upper() != expected:
return ValidationResult(
False, "", value,
f"校验位错误，应为 {expected}",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_202_id_number_passport.py 0 → 100644
View file @ 5c37925e
"""IND-202 证件号码-中国护照
号码格式（基于 2026-08-07 网络调研结论）：
- 普通护照（旧）：G + 8 位数字 = 9 字符
- 普通护照（早期电子）：E + 8 位数字 = 9 字符（2012-05-15 ~ 2017-04-04）
- 普通护照（电子扩容）：E + 1 位字母（排除 I/O）+ 7 位数字 = 9 字符（2017-04-05 起）
- 公务普通：PE + 7 位数字 = 9 字符
- 公务：SE + 7 位数字 = 9 字符
- 外交：DE + 7 位数字 = 9 字符
不在本 indicator 范围内：
- 旧版 P/S/D（未查证到权威起止日期，二手资料冲突 7 vs 8 位）—— 仅 warning 不拒
- ICAO Doc 9303 机读码校验位（属于 MRZ 区，不在可视区号码字符内）
参考资料：
- gov.cn 护照法 https://www.gov.cn/flfg/2006-04/29/content_270216.htm
- ICAO Doc 9303 Part 4
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class IdNumberPassportIndicator(BaseStandard):
standard_id = "IND-202"
standard_name = "证件号码-中国护照"
applies_to_fields = ["passport_no", "hz_no", "huzhao_no"]
description = (
"9 字符：G+8 / E+8 / E+1 字母（排除I/O）+7 / PE+7 / SE+7 / DE+7。"
" 不强校验位算法（公开材料未查到）。"
)
group = "证件信息"
severity = "warning"
# 主流正则（推荐用）
_REGEX = re.compile(
r"^("
r"G[0-9]{8}|" # 普通护照旧版
r"E[0-9]{8}|" # 电子普通护照初期
r"E[A-HJ-NP-Z][0-9]{7}|" # 电子普通护照扩容（字母排除 I/O）
r"PE[0-9]{7}|SE[0-9]{7}|DE[0-9]{7}" # 公务普通 / 公务 / 外交
r")$"
)
# 旧 P/S/D 版本（公开资料冲突 —— 仅作 info 命中，不拒）
_LEGACY_REGEX = re.compile(r"^[PSD][0-9]{7,8}$")
_FORBIDDEN_CHARS = set("'\"`~!@#$%^&*()+=[]{}|\\:;<>?,./")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip().upper()
# 非法字符
if any(c in self._FORBIDDEN_CHARS for c in v):
return ValidationResult(
False, "", value,
"护照号包含非法特殊字符",
self.standard_id,
)
# 长度区间
if not (5 <= len(v) <= 20):
return ValidationResult(
False, "", value,
f"长度 {len(v)} 不在 5-20 区间",
self.standard_id,
)
# 主流正则
if self._REGEX.match(v):
return ValidationResult(True, "", value, "", self.standard_id)
# 旧 P/S/D 兼容 —— 不报错，仅 warning 提示
if self._LEGACY_REGEX.match(v):
return ValidationResult(
False, "", value,
"旧 P/S/D 公务 / 外交护照格式（未查证到权威起止日期，请人工确认）",
self.standard_id,
)
return ValidationResult(
False, "", value,
"不符合已知护照号格式（普通 G+8 / E+8 / E+字母+7 / 公务 PE/SE/DE+7）",
self.standard_id,
)
\ No newline at end of file
standards/ind_203_id_number_hkmacao_residence.py 0 → 100644
View file @ 5c37925e
"""IND-203 证件号码-港澳台居民居住证（18 位公民身份号码框架）
依据：国办发〔2018〕81 号《港澳台居民居住证申领发放办法》
- 18 位：6 位地址码 + 8 位出生日期 + 3 位顺序码 + 1 位校验码
- 地址码：810000=香港 / 820000=澳门 / 830000=台湾
- 与居民身份证共用 GB 11643-1999 校验算法
注意：GB/T 2260-2007 现行版本只含 810000 / 820000（不含 830000）；
830000 由国务院台办配套文件给出。本 indicator 在地址码段位放宽，
接受 81/82/83 三段前缀。
"""
from __future__ import annotations
import re
from datetime import date
from .base import BaseStandard, ValidationResult
class IdNumberHKMacaoResidenceIndicator(BaseStandard):
standard_id = "IND-203"
standard_name = "证件号码-港澳台居民居住证（国办发〔2018〕81 号）"
applies_to_fields = [
"hk_residence_no", "mo_residence_no", "tw_residence_no",
"gat_residence_no", "gatjzz_no",
]
description = (
"18 位 + 地址码 ∈ {810000 香港, 820000 澳门, 830000 台湾} + GB 11643 校验位"
)
group = "证件信息"
severity = "warning"
_REGEX = re.compile(r"^8[123]\d{4}(19|20)\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}[\dXx]$")
_WEIGHTS = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]
_CHECKSUM_MAP = ["1", "0", "X", "9", "8", "7", "6", "5", "4", "3", "2"]
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if len(v) != 18:
return ValidationResult(False, "", value, f"长度 {len(v)} ≠ 18", self.standard_id)
if not self._REGEX.match(v):
return ValidationResult(
False, "", value,
"不符合 81/82/83 前缀 + 18 位 GB 11643 结构",
self.standard_id,
)
try:
date(int(v[6:10]), int(v[10:12]), int(v[12:14]))
except ValueError:
return ValidationResult(False, "", value, f"出生日期 {v[6:14]} 不合法", self.standard_id)
total = sum(int(c) * w for c, w in zip(v[:17], self._WEIGHTS))
expected = self._CHECKSUM_MAP[total % 11]
if v[17].upper() != expected:
return ValidationResult(False, "", value, f"校验位错误，应为 {expected}", self.standard_id)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_204_id_number_foreign_residence.py 0 → 100644
View file @ 5c37925e
"""IND-204 证件号码-外国人永久居留身份证
两种版本共存：
- 旧版（2017-）：3 位大写字母（ISO 3166-1 alpha-3 国别码）+ 12 位数字 = 15 位
- 新版「五星卡」（2023-12-01 起）：18 位，首位固定 9
仅做格式校验；官方未公开校验位算法（不在二手 MOD 11 推测上做强校验）。
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
class IdNumberForeignResidenceIndicator(BaseStandard):
standard_id = "IND-204"
standard_name = "证件号码-外国人永久居留身份证"
applies_to_fields = ["foreign_residence_no", "yd_no", "ywjzj_no", "prc_residence_no"]
description = "旧版 15 位（3 字母 ISO 3166-1 alpha-3 + 12 数字）/ 新版 18 位（首位 9）"
group = "证件信息"
severity = "warning"
_REGEX_OLD = re.compile(r"^[A-Z]{3}\d{12}$")
_REGEX_NEW = re.compile(r"^9\d{16}[\dX]$")
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip().upper()
if self._REGEX_NEW.match(v):
return ValidationResult(True, "", value, "（2023 版五星卡，格式通过）", self.standard_id)
if self._REGEX_OLD.match(v):
return ValidationResult(True, "", value, "（2017 旧版 15 位，格式通过）", self.standard_id)
return ValidationResult(
False, "", value,
"不符合永居证格式（旧 15 位 LLL+12 数字 / 新 18 位 9+17 数字+X）",
self.standard_id,
)
\ No newline at end of file
standards/ind_301_id_type_number_consistency.py 0 → 100644
View file @ 5c37925e
"""IND-301 证件类型 + 证件号 一致性（跨字段校验器）
按证件类型分发到对应 indicator：
- 01 居民身份证 → 复用 IND-201
- 03 护照 → 复用 IND-202
- 05 港澳台居住证 → 复用 IND-203
- 04 永居证 → 复用 IND-204
- 02 军官证 → 当前跳过（无公开权威格式文档）
通过覆写 validate_rows() 实现：rows 元素含 {id_type, id_no} 两个键。
step7 检测到 indicator 实现 validate_rows 时走「跨字段」轮，
SQL 模板走 web/sql/standards/sample_field_pairs.sql。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import is_valid_id_type
from .ind_201_id_number_id_card import IdNumberIdCard18Indicator
from .ind_202_id_number_passport import IdNumberPassportIndicator
from .ind_203_id_number_hkmacao_residence import IdNumberHKMacaoResidenceIndicator
from .ind_204_id_number_foreign_residence import IdNumberForeignResidenceIndicator
# 类型 → indicator 映射
_DISPATCH = {
"01": IdNumberIdCard18Indicator(),
"03": IdNumberPassportIndicator(),
"04": IdNumberForeignResidenceIndicator(),
"05": IdNumberHKMacaoResidenceIndicator(),
}
class IdTypeNumberConsistencyIndicator(BaseStandard):
standard_id = "IND-301"
standard_name = "证件类型 + 证件号 一致性"
# applies_to_fields 留空 —— 本 indicator 由跨字段 SQL 触发，不走单字段自动注册
applies_to_fields = []
description = (
"type=01 → 必须通过 IND-201（18 位 GB 11643）；"
"type=03 → IND-202（护照）；type=04 → IND-204（永居证）；"
"type=05 → IND-203（港澳台居住证）。"
" SQL 模板：web/sql/standards/sample_field_pairs.sql"
)
group = "证件信息"
severity = "warning"
depends_on = ["IND-101", "IND-201", "IND-202", "IND-203", "IND-204"]
def validate(self, value: str) -> ValidationResult:
# 单值模式不适用 —— 跨字段校验必须走 validate_rows
return ValidationResult(True, "", value, "IND-301 仅支持跨字段校验（validate_rows）", self.standard_id)
def validate_rows(self, rows: list[dict]) -> list[dict]:
out = []
for row in rows:
t = (row.get("id_type") or "").strip()
n = (row.get("id_no") or "").strip()
if not t or not n:
continue
if not is_valid_id_type(t):
out.append({
"value": n,
"id_type": t,
"reason": f"证件类型 {t!r} 不在标准枚举（IND-101 也会命中）",
"rule": "type_invalid",
})
continue
indicator = _DISPATCH.get(t)
if indicator is None:
# 02 军官证等暂跳过
continue
r = indicator.validate(n)
if not r.valid:
out.append({
"value": n,
"id_type": t,
"reason": r.reason,
"rule": f"type_{t}_mismatch",
})
return out
\ No newline at end of file
standards/ind_302_id_uniqueness.py 0 → 100644
View file @ 5c37925e
"""IND-302 证件唯一性（跨表聚合校验器 —— GROUP BY HAVING）
按 (id_type, id_no) 维度跨表去重，疑似重复开户：
- 在多张表里出现相同 (type, no) → 报「疑似重复开户」
- 在同一张表里出现多次 → 报「同表重复」
走 SQL `GROUP BY HAVING COUNT(*) > N`，SQL 模板：
web/sql/standards/sample_field_groups.sql
indicator 类本身不参与单字段自动发现，由 step7 第二轮显式调用。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
class IdUniquenessIndicator(BaseStandard):
standard_id = "IND-302"
standard_name = "证件唯一性（跨表去重）"
applies_to_fields = []
description = (
"GROUP BY (id_type, id_no) HAVING COUNT > N 检出重复。"
" SQL 模板：web/sql/standards/sample_field_groups.sql"
)
group = "证件信息"
severity = "warning"
depends_on = ["IND-101"]
def validate(self, value: str) -> ValidationResult:
# 单值模式不适用
return ValidationResult(True, "", value, "IND-302 仅支持跨表聚合（GROUP BY HAVING）", self.standard_id)
\ No newline at end of file
standards/ind_401_name_charset.py 0 → 100644
View file @ 5c37925e
"""IND-401 姓名字符集
校验规则（基于 GA/T 2000.27 等行业惯例 + 网络调研）：
- 允许字符：
- 汉字（基本区 0x4E00-0x9FA5 + 扩展 A 区 0x3400-0x4DBF）
- 维吾尔族等少数民族的「·」间隔符（U+00B7，中点）
- 半角空格（U+0020）
- ASCII 字母（部分业务系统支持拼音名）
- 禁止：
- 纯数字 / 纯特殊符号
- 控制字符 / Emoji 等奇异字符
"""
from __future__ import annotations
import re
from .base import BaseStandard, ValidationResult
_MIDDLE_DOT = "·" # 维吾尔族等少数民族「·」间隔符
_FULLWIDTH_DOT = "．" # 全角点（兼容录入）
def _is_han(c: str) -> bool:
return (
"一" <= c <= "鿿" # CJK 基本
or "㐀" <= c <= "䶿" # CJK 扩展 A
)
def _is_allowed(c: str) -> bool:
if _is_han(c):
return True
if c in (_MIDDLE_DOT, _FULLWIDTH_DOT, " "):
return True
if "A" <= c <= "Z" or "a" <= c <= "z":
return True
return False
class NameCharsetIndicator(BaseStandard):
standard_id = "IND-401"
standard_name = "姓名字符集"
applies_to_fields = [
"name", "real_name", "customer_name", "user_name", "xm",
"user_real_name", "person_name",
]
description = (
"允许：汉字 / 维吾尔等少数民族的「·」/ 全角点「．」/ 半角空格 / ASCII 字母。"
" 禁止：纯数字 / 纯特殊符号 / 控制字符 / Emoji"
)
group = "身份信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if not v:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
has_han = any(_is_han(c) for c in v)
has_letter = bool(re.search(r"[A-Za-z]", v))
if not (has_han or has_letter):
return ValidationResult(
False, "", value,
"姓名不含任何汉字或字母（疑似纯数字 / 纯特殊符号）",
self.standard_id,
)
bad = [c for c in v if not _is_allowed(c)]
if bad:
return ValidationResult(
False, "", value,
f"姓名包含不规范字符 {''.join(bad[:5])!r}",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_402_name_length.py 0 → 100644
View file @ 5c37925e
"""IND-402 姓名长度（2-20 字符）
业务侧惯例；公安部《姓名登记条例（征求意见稿）》及实际办理经验。
少数民族姓名可能较长（如「额尔德木图·巴特巴依尔」13 字），20 字留余。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
class NameLengthIndicator(BaseStandard):
standard_id = "IND-402"
standard_name = "姓名长度"
applies_to_fields = [
"name", "real_name", "customer_name", "user_name", "xm",
"user_real_name", "person_name",
]
description = "字符数 ∈ [2, 20]"
group = "身份信息"
severity = "warning"
_MIN = 2
_MAX = 20
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
n = len(v)
if n < self._MIN:
return ValidationResult(
False, "", value,
f"姓名长度 {n} < {self._MIN}",
self.standard_id,
)
if n > self._MAX:
return ValidationResult(
False, "", value,
f"姓名长度 {n} > {self._MAX}",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_501_birth_date_format.py 0 → 100644
View file @ 5c37925e
"""IND-501 出生日期格式（8 位 YYYYMMDD + 合法日期）
兼容 ISO 8601（YYYY-MM-DD）、YYYY/MM/DD、YYYY.MM.DD —— 实际业务常见。
"""
from __future__ import annotations
import re
from datetime import date
from .base import BaseStandard, ValidationResult
_RE = re.compile(r"^(\d{4})[-./]?(\d{2})[-./]?(\d{2})$")
class BirthDateFormatIndicator(BaseStandard):
standard_id = "IND-501"
standard_name = "出生日期格式"
applies_to_fields = ["birth_date", "birthday", "csrq", "date_of_birth"]
description = "8 位 YYYYMMDD 或带 - / . 分隔；日期必须真实存在（含闰年）"
group = "身份信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
m = _RE.match(v)
if not m:
return ValidationResult(
False, "", value,
"不符合 8 位 YYYYMMDD 格式（或含 - / . 分隔）",
self.standard_id,
)
try:
y_, m_, d_ = int(m.group(1)), int(m.group(2)), int(m.group(3))
date(y_, m_, d_)
except ValueError:
return ValidationResult(
False, "", value,
f"日期 {v} 不合法（含闰年 / 月日范围）",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_502_birth_date_age_range.py 0 → 100644
View file @ 5c37925e
"""IND-502 出生日期年龄范围（16 ~ 法定退休年龄）
策略：
- 最低 16 岁（劳动法最低就业年龄）
- 上限按性别字段推断：男 60 / 女 50 / 女干部 55 —— 详见 retirement_age.json
- 找不到性别字段时退化为 60
- 假定出生日期格式已通过（IND-501）
实现为单值校验（需要外部传入性别）—— 不便跨字段，于是放宽为：
- 若同表有 gender 字段，step7 在调用前做 JOIN 拼出 (birth, gender) 后调用本类；
- 默认降级为 60 岁上限（无论性别）。
"""
from __future__ import annotations
from datetime import date
from typing import Iterable
from .base import BaseStandard, ValidationResult
from .data_loader import load_retirement_age_rules
_RE_YMD = __import__("re").compile(r"^(\d{4})[-./]?(\d{2})[-./]?(\d{2})$")
class BirthDateAgeRangeIndicator(BaseStandard):
standard_id = "IND-502"
standard_name = "出生日期年龄范围"
applies_to_fields = ["birth_date", "birthday", "csrq", "date_of_birth"]
description = (
"年龄 ∈ [16, 法定退休年龄]。退休年龄按性别：男 60 / 女（工人）50 / 女（干部）55；"
" 默认 60。"
)
group = "身份信息"
severity = "warning"
_MIN_AGE = 16
def validate(self, value: str) -> ValidationResult:
# 默认按「无性别字段」处理：上限 60
return self._validate_with_gender(value, gender=None)
def validate_with_gender(self, value: str, gender: str | None) -> ValidationResult:
"""step7 入口在能拿到性别字段时显式调用本方法。"""
return self._validate_with_gender(value, gender)
def _validate_with_gender(self, value: str, gender: str | None) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
m = _RE_YMD.match(v)
if not m:
return ValidationResult(
False, "", value,
"年龄校验要求先通过 IND-501 格式",
self.standard_id,
)
try:
bd = date(int(m.group(1)), int(m.group(2)), int(m.group(3)))
except ValueError:
return ValidationResult(False, "", value, "日期不合法", self.standard_id)
today = date.today()
age = today.year - bd.year - ((today.month, today.day) < (bd.month, bd.day))
if age < self._MIN_AGE:
return ValidationResult(
False, "", value,
f"年龄 {age} < {self._MIN_AGE}（劳动法最低就业年龄）",
self.standard_id,
)
# 退休年龄上限
rules = load_retirement_age_rules()
upper = rules.get("default", {}).get("age", 60)
if gender in ("2", "F", "f", "女"):
# 简化处理：女工 50 / 女干部 55 —— 默认按工人 50
upper = rules.get("female_worker", {}).get("age", 50)
elif gender in ("1", "M", "m", "男"):
upper = rules.get("male", {}).get("age", 60)
if age > upper:
return ValidationResult(
False, "", value,
f"年龄 {age} > {upper}（{gender or '未知'}），疑似超龄（贷款业务尤其注意）",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_601_district_code.py 0 → 100644
View file @ 5c37925e
"""IND-601 行政区划代码（GB/T 2260-2007）
6 位代码必须在 GB/T 2260-2007 表内（3406 条）。
与 IND-004-a/b 字段重叠 —— 本 indicator 适用于「以行政区划编码为核心语义」字段（如户籍所在地代码）。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import is_valid_district_6
class DistrictCodeIndicator(BaseStandard):
standard_id = "IND-601"
standard_name = "GB/T 2260-2007 行政区划代码"
applies_to_fields = [
"hjszd", "hjsxdm", "hukou_xzqh", "registered_district",
"native_district_code", "current_district_code",
]
description = "6 位代码 ∈ GB/T 2260-2007（3406 条）"
group = "民政信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if len(v) != 6 or not v.isdigit():
return ValidationResult(
False, "", value,
f"行政区划代码须为 6 位数字（实际 {len(v)} 位）",
self.standard_id,
)
if not is_valid_district_6(v):
return ValidationResult(
False, "", value,
"6 位代码不在 GB/T 2260-2007 表内",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/ind_602_ethnicity.py 0 → 100644
View file @ 5c37925e
"""IND-602 民族代码（GB/T 3304-1991）
2 位数字 ∈ {01..56, 97, 98}
- 01-56：正文 56 民族（extracted_standards.json 提供）
- 97：其他（公安部补充码，未在 extracted_standards.json 中）
- 98：外国血统中国籍人士（同上）
- 81（穿青人）是公安部扩展码，未在标准正文 —— 不强校验
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import is_valid_ethnicity
class EthnicityIndicator(BaseStandard):
standard_id = "IND-602"
standard_name = "GB/T 3304-1991 民族代码"
applies_to_fields = ["ethnicity", "nation", "mz", "minzu", "ethnic"]
description = "2 位数字 ∈ {01..56, 97, 98}（GB/T 3304-1991 正文 56 民族 + 补充码 97/98）"
group = "民政信息"
severity = "warning"
# extracted_standards.json 只含正文 01-56；97/98 是补充码，手动接受
_EXTRA = {"97", "98"}
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip().zfill(2) # 接受 "1" → "01"
if is_valid_ethnicity(v) or v in self._EXTRA:
return ValidationResult(True, "", value, "", self.standard_id)
return ValidationResult(
False, "", value,
f"民族代码 {v!r} 不在 GB/T 3304-1991 表内（01-56 + 97/98）",
self.standard_id,
)
\ No newline at end of file
standards/ind_603_marriage_status.py 0 → 100644
View file @ 5c37925e
"""IND-603 婚姻状况代码（GB/T 2261.2-2003）
2 位 ∈ {10, 20, 21, 22, 23, 30, 40, 90}
- 10 未婚 / 20 已婚 / 21 初婚 / 22 再婚 / 23 复婚
- 30 丧偶 / 40 离婚 / 90 未说明
注意：
- GB/T 4761 是「家庭关系代码」，不是婚姻状况 —— 别用错标准。
- 旧 GB/T 4766-1984 已被本标准代替。
"""
from __future__ import annotations
from .base import BaseStandard, ValidationResult
from .data_loader import is_valid_marriage
class MarriageStatusIndicator(BaseStandard):
standard_id = "IND-603"
standard_name = "GB/T 2261.2-2003 婚姻状况代码"
applies_to_fields = ["marital_status", "marriage", "hyzk"]
description = "2 位 ∈ {10,20,21,22,23,30,40,90}（GB/T 2261.2-2003）"
group = "民政信息"
severity = "warning"
def validate(self, value: str) -> ValidationResult:
if not value:
return ValidationResult(True, "", value, "空值跳过", self.standard_id)
v = str(value).strip()
if not is_valid_marriage(v):
return ValidationResult(
False, "", value,
f"婚姻状况代码 {v!r} 不在 GB/T 2261.2-2003 表内（10/20/21/22/23/30/40/90）",
self.standard_id,
)
return ValidationResult(True, "", value, "", self.standard_id)
\ No newline at end of file
standards/registry.py
View file @ 5c37925e
......@@ -44,14 +44,20 @@ def discover_standards() -> dict[str, BaseStandard]:
def list_all_standards() -> list[dict]:
"""返回所有标准的基本信息（用于报告/调试）"""
"""返回所有标准的基本信息（用于报告/调试）
返回字段：
- id, name, applies_to_fields, description: 同原
- group: "国标字段规范" / "证件信息" / "身份信息" / "民政信息" / "通用"
- severity: "warning" | "error"
"""
import standards
out = []
seen: set[type] = set()
package_path = Path(__file__).parent
for module_info in pkgutil.iter_modules([str(package_path)]):
if module_info.name in ("base", "registry"):
if module_info.name in ("base", "registry", "data_loader"):
continue
module = importlib.import_module(f"standards.{module_info.name}")
for attr_name in dir(module):
......@@ -69,5 +75,91 @@ def list_all_standards() -> list[dict]:
"name": inst.standard_name,
"applies_to_fields": inst.applies_to_fields,
"description": inst.description,
"group": getattr(inst, "group", "通用"),
"severity": getattr(inst, "severity", "warning"),
})
return out
def list_standards_by_group() -> dict[str, list[dict]]:
"""按 group 分组返回所有标准 —— step7 拆 tab 用"""
by_group: dict[str, list[dict]] = {}
for s in list_all_standards():
by_group.setdefault(s["group"], []).append(s)
return by_group
def load_standard_class(standard_id: str) -> type[BaseStandard] | None:
"""按 standard_id（如 "IND-001-a"）找到对应的标准子类。
返回类本身（不是实例），调用方负责实例化。
用于 orchestrator 按 step_id 找到对应 indicator 类后分发执行。
"""
import standards
seen: set[type] = set()
package_path = Path(__file__).parent
for module_info in pkgutil.iter_modules([str(package_path)]):
if module_info.name in ("base", "registry", "data_loader"):
continue
module = importlib.import_module(f"standards.{module_info.name}")
for attr_name in dir(module):
attr = getattr(module, attr_name)
if (
isinstance(attr, type)
and issubclass(attr, BaseStandard)
and attr is not BaseStandard
and attr not in seen
):
seen.add(attr)
if attr.standard_id == standard_id:
return attr
return None
# 缓存 indicator 的 prefix（IND- 或 STD-），用于 step_id ↔ indicator_id 双向还原
_PREFIX_RE = __import__("re").compile(r"^(IND|STD)-")
def indicator_step_id(standard_id: str) -> str:
"""IND-001-a → std_ind_001_a；STD-001 → std_std_001（保留 prefix 信息）。
标准 ID 形如 "IND-NNN-x" / "IND-NNN" / "STD-NNN"；取 prefix + 尾段拼到 step_id 里。
"""
m = _PREFIX_RE.match(standard_id)
prefix = (m.group(1) if m else "IND").lower() # "ind" / "std"
tail = standard_id[m.end():] if m else standard_id # "001-a" / "001"
return f"std_{prefix}_{tail.lower().replace('-', '_')}"
def step_id_to_indicator_id(step_id: str) -> str | None:
"""std_ind_001_a → IND-001-a；std_std_001 → STD-001；非 indicator step 返回 None。"""
m = __import__("re").match(r"^std_(ind|std)_(.+)$", step_id)
if not m:
return None
prefix = m.group(1).upper()
raw = m.group(2)
if "_" in raw:
num, suffix = raw.rsplit("_", 1)
# 标准 ID 后缀保持小写（IND-001-a / IND-001-b ...）
return f"{prefix}-{num}-{suffix.lower()}"
return f"{prefix}-{raw}"
def list_step_ids() -> list[tuple[str, dict]]:
"""列出所有 indicator 的 (step_id, metadata)（给 orchestrator 循环注册用）。
返回顺序按 group（国标字段规范 → 证件信息 → 身份信息 → 民政信息 → 通用），
同 group 内按 standard_id 字典序。
metadata 字段：standard_id / standard_name / group / severity / applies_to_fields
"""
items = list_all_standards()
items.sort(key=lambda s: (
{"国标字段规范": 0, "证件信息": 1, "身份信息": 2, "民政信息": 3, "通用": 4}.get(s["group"], 99),
s["id"],
))
out = []
for s in items:
sid = indicator_step_id(s["id"])
out.append((sid, s))
return out
standards/std_001_id_card.py
View file @ 5c37925e
"""GB 11643-1999 公民身份号码"""
"""GB 11643-1999 公民身份号码 —— 已拆分为 IND-001-a/b/c/d，本类保留为 deprecated 占位。
本类已不再自动注册到字段映射（applies_to_fields=[]）。字段命中由
IND-001-a（格式）/ IND-001-b（校验位）/ IND-001-c（出生日期）/
IND-001-d（15 位兼容）四个细粒度 indicator 接管。
保留本文件仅为向后兼容旧引用 + 国标清单页面调试用。
"""
from __future__ import annotations
......@@ -9,10 +16,15 @@ from .base import BaseStandard, ValidationResult
class IdCardStandard(BaseStandard):
standard_id = "STD-001"
standard_name = "GB 11643-1999 公民身份号码"
applies_to_fields = ["id_card", "id_card_no", "id_number", "identity_card"]
description = "18位：6位地址码 + 8位出生日期 + 3位顺序码 + 1位校验码（ISO 7064 MOD 11-2）"
standard_id = "STD-001" # 保留旧 ID 不变（list_all_standards 仍会列出，标 deprecated）
standard_name = "GB 11643-1999 公民身份号码 [已拆分]"
applies_to_fields = [] # 关键：置空 → discover_standards 不再自动注册
description = (
"18位：6位地址码 + 8位出生日期 + 3位顺序码 + 1位校验码（ISO 7064 MOD 11-2）。"
" 本类已拆分 → 字段命中请用 IND-001-a/b/c/d"
)
group = "通用"
severity = "warning"
_REGEX = re.compile(r"^[1-9]\d{5}(19|20)\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}[\dXx]$")
_WEIGHTS = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]
......
standards/std_002_uscc.py
View file @ 5c37925e
"""GB 32100-2015 法人和其他组织统一社会信用代码"""
"""GB 32100-2015 法人和其他组织统一社会信用代码 —— 已拆分为 IND-002-a/b，本类保留为 deprecated 占位。"""
from __future__ import annotations
......@@ -9,9 +9,14 @@ from .base import BaseStandard, ValidationResult
class UsccStandard(BaseStandard):
standard_id = "STD-002"
standard_name = "GB 32100-2015 统一社会信用代码"
applies_to_fields = ["uscc", "credit_code", "social_credit_code", "unified_social_credit_code"]
description = "18位：登记管理机关(1) + 机构类别(1) + 行政区划(6) + 主体标识码(9) + 校验码(1)"
standard_name = "GB 32100-2015 统一社会信用代码 [已拆分]"
applies_to_fields = []
description = (
"18位：登记管理机关(1) + 机构类别(1) + 行政区划(6) + 主体标识码(9) + 校验码(1)。"
" 本类已拆分 → 字段命中请用 IND-002-a/b"
)
group = "通用"
severity = "warning"
# 排除 I、O、Z、S、V
_REGEX = re.compile(r"^[1-9A-HJ-NP-RT-Y]\d[0-9A-HJ-NP-RT-Y]{6}[0-9A-HJ-NP-RT-Y]{9}[0-9A-HJ-NP-RT-Y]$")
......
standards/std_003_mobile.py
View file @ 5c37925e
"""手机号校验（工信部《电信网编号计划》）"""
"""手机号校验（工信部《电信网编号计划》）—— 已拆分为 IND-003-a/b，本类保留为 deprecated 占位。"""
from __future__ import annotations
......@@ -9,13 +9,13 @@ from .base import BaseStandard, ValidationResult
class MobileStandard(BaseStandard):
standard_id = "STD-003"
standard_name = "工信部《电信网编号计划》手机号"
applies_to_fields = [
"mobile", "phone", "mobile_phone", "tel", "telephone",
"contact_phone", "customer_mobile", "legal_person_mobile",
"notify_phone", "maintainer_phone", "manager_phone", "receiver_phone",
]
description = "11位：以 1 开头，第 2 位 3-9"
standard_name = "工信部《电信网编号计划》手机号 [已拆分]"
applies_to_fields = []
description = (
"11位：以 1 开头，第 2 位 3-9。本类已拆分 → 字段命中请用 IND-003-a/b"
)
group = "通用"
severity = "warning"
_REGEX = re.compile(r"^1[3-9]\d{9}$")
# 详细号段正则（更严格）
......
standards/std_004_xzqh.py
View file @ 5c37925e
"""GB/T 2260 行政区划代码
仅做格式校验（6位或12位全数字），具体编码合法性需对照官方编码表。
"""
"""GB/T 2260 行政区划代码 —— 已拆分为 IND-004-a/b，本类保留为 deprecated 占位。"""
from __future__ import annotations
......@@ -12,9 +9,13 @@ from .base import BaseStandard, ValidationResult
class XzqhStandard(BaseStandard):
standard_id = "STD-004"
standard_name = "GB/T 2260 行政区划代码"
applies_to_fields = ["xzqhbm", "xzqhjb", "adcode", "district_code", "province_code", "city_code", "region_code"]
description = "6位标准版或12位扩展版，全数字"
standard_name = "GB/T 2260 行政区划代码 [已拆分]"
applies_to_fields = []
description = (
"6位标准版或12位扩展版。本类已拆分 → 字段命中请用 IND-004-a（格式）/ IND-004-b（编码存在）"
)
group = "通用"
severity = "warning"
_REGEX_6 = re.compile(r"^\d{6}$")
_REGEX_12 = re.compile(r"^\d{12}$")
......
tests/test_indicators.py 0 → 100644
View file @ 5c37925e
"""24 个 indicator 的单测 —— 合规 + 违规 case 各 1-2 个
运行方式：
cd 项目根目录
python -m pytest tests/test_indicators.py -v
"""
from __future__ import annotations
import pytest
import standards.ind_001a_id_card_format as m001a
import standards.ind_001b_id_card_checksum as m001b
import standards.ind_001c_id_card_birthdate as m001c
import standards.ind_001d_id_card_15_legacy as m001d
import standards.ind_002a_uscc_format as m002a
import standards.ind_002b_uscc_checksum as m002b
import standards.ind_003a_mobile_format as m003a
import standards.ind_003b_mobile_segment as m003b
import standards.ind_004a_xzqh_format as m004a
import standards.ind_004b_xzqh_exists as m004b
import standards.ind_101_id_type as m101
import standards.ind_201_id_number_id_card as m201
import standards.ind_202_id_number_passport as m202
import standards.ind_203_id_number_hkmacao_residence as m203
import standards.ind_204_id_number_foreign_residence as m204
import standards.ind_301_id_type_number_consistency as m301
import standards.ind_401_name_charset as m401
import standards.ind_402_name_length as m402
import standards.ind_501_birth_date_format as m501
import standards.ind_502_birth_date_age_range as m502
import standards.ind_601_district_code as m601
import standards.ind_602_ethnicity as m602
import standards.ind_603_marriage_status as m603
# ── IND-001-a 身份证号格式 ──
class TestIdCardFormat:
def test_valid_18(self):
assert m001a.IdCardFormatIndicator().validate("11010519491231002X").valid
def test_invalid_length(self):
assert not m001a.IdCardFormatIndicator().validate("12345").valid
def test_invalid_chars(self):
assert not m001a.IdCardFormatIndicator().validate("12345678901234567a").valid
def test_empty_skipped(self):
assert m001a.IdCardFormatIndicator().validate("").valid
# ── IND-001-b 身份证校验位 ──
class TestIdCardChecksum:
def test_valid(self):
assert m001b.IdCardChecksumIndicator().validate("11010519491231002X").valid
def test_bad_checksum(self):
assert not m001b.IdCardChecksumIndicator().validate("110105194912310020").valid
# ── IND-001-c 身份证出生日期 ──
class TestIdCardBirthdate:
def test_valid(self):
assert m001c.IdCardBirthdateIndicator().validate("11010519491231002X").valid
def test_invalid_date(self):
# 02-30 不存在
assert not m001c.IdCardBirthdateIndicator().validate("110105199002300012").valid
# ── IND-001-d 15 位老证 ──
class TestIdCard15Legacy:
def test_15_legacy_hit(self):
# 15 位应被命中（warning）
assert not m001d.IdCard15LegacyIndicator().validate("110105491231002").valid
def test_18_legacy_no_hit(self):
# 18 位不算违规
assert m001d.IdCard15LegacyIndicator().validate("11010519491231002X").valid
# ── IND-002-a USCC 格式 ──
class TestUsccFormat:
def test_valid(self):
assert m002a.UsccFormatIndicator().validate("91110000600037341L").valid
def test_invalid_length(self):
assert not m002a.UsccFormatIndicator().validate("91110000600037341").valid
def test_forbidden_char(self):
# I 是禁用字符
assert not m002a.UsccFormatIndicator().validate("9I110000600037341L").valid
# ── IND-002-b USCC 校验位 ──
class TestUsccChecksum:
def test_valid(self):
assert m002b.UsccChecksumIndicator().validate("91110000600037341L").valid
def test_bad_checksum(self):
assert not m002b.UsccChecksumIndicator().validate("911100006000373410").valid
# ── IND-003-a 手机号格式 ──
class TestMobileFormat:
def test_valid(self):
assert m003a.MobileFormatIndicator().validate("13800138000").valid
def test_invalid_prefix(self):
assert not m003a.MobileFormatIndicator().validate("12800138000").valid
def test_invalid_length(self):
assert not m003a.MobileFormatIndicator().validate("1380013800").valid
def test_clean_with_country_code(self):
assert m003a.MobileFormatIndicator().validate("+8613800138000").valid
# ── IND-003-b 手机号号段 ──
class TestMobileSegment:
def test_valid_segment(self):
assert m003b.MobileSegmentIndicator().validate("13800138000").valid
def test_invalid_segment(self):
# 11x、12x、14[0-4]、15[4] 等不在已知号段表内
# 11111111111：第二位 1 不在任何合法号段
assert not m003b.MobileSegmentIndicator().validate("11111111111").valid
# ── IND-004-a 行政区划格式 ──
class TestXzqhFormat:
def test_valid_6(self):
assert m004a.XzqhFormatIndicator().validate("110000").valid
def test_invalid_length(self):
assert not m004a.XzqhFormatIndicator().validate("12345").valid
def test_invalid_province(self):
# 99 不是省级代码
assert not m004a.XzqhFormatIndicator().validate("990000").valid
# ── IND-004-b 行政区划存在性 ──
class TestXzqhExists:
def test_valid(self):
assert m004b.XzqhExistsIndicator().validate("110000").valid
def test_invalid_code(self):
# 不存在的 6 位代码（合法但表内无）
assert not m004b.XzqhExistsIndicator().validate("110999").valid
def test_skip_12_digit(self):
# 12 位版本不强校验
assert not m004b.XzqhExistsIndicator().validate("110000110000").valid # 标"仅校验 6 位"
# ── IND-101 证件类型枚举 ──
class TestIdTypeEnum:
def test_valid_01(self):
assert m101.IdTypeEnumIndicator().validate("01").valid
def test_invalid_99(self):
assert not m101.IdTypeEnumIndicator().validate("99").valid
# ── IND-201 居民身份证 18 位 ──
class TestIdNumberIdCard18:
def test_valid_18(self):
assert m201.IdNumberIdCard18Indicator().validate("11010519491231002X").valid
def test_invalid_15(self):
# 15 位老证应被报（建议升级）
assert not m201.IdNumberIdCard18Indicator().validate("110105491231002").valid
def test_invalid_checksum(self):
assert not m201.IdNumberIdCard18Indicator().validate("110105194912310020").valid
# ── IND-202 中国护照 ──
class TestIdNumberPassport:
def test_valid_electronic(self):
# E + 字母 + 7 位数字
assert m202.IdNumberPassportIndicator().validate("EA1234567").valid
def test_valid_old_G(self):
assert m202.IdNumberPassportIndicator().validate("G12345678").valid
def test_valid_diplomatic_SE(self):
assert m202.IdNumberPassportIndicator().validate("SE1234567").valid
def test_invalid_length(self):
assert not m202.IdNumberPassportIndicator().validate("E12345").valid
def test_invalid_special_chars(self):
assert not m202.IdNumberPassportIndicator().validate("E12345678!").valid
# ── IND-203 港澳台居民居住证 ──
class TestHKMacaoResidence:
def test_valid_hk(self):
# 81xxxx + 19900101 + 顺序 + 校验
# 正确校验位：810000199001010000（mod11=1 → map='0'）
assert m203.IdNumberHKMacaoResidenceIndicator().validate(
"810000199001010000"
).valid
def test_invalid_wrong_prefix(self):
# 84xxxx 不是居住证前缀
assert not m203.IdNumberHKMacaoResidenceIndicator().validate("840000199001010001").valid
# ── IND-204 外国人永居证 ──
class TestForeignResidence:
def test_valid_old_15(self):
assert m204.IdNumberForeignResidenceIndicator().validate("USA123456789012").valid
def test_valid_new_18(self):
assert m204.IdNumberForeignResidenceIndicator().validate("912345678901234567").valid
def test_invalid(self):
assert not m204.IdNumberForeignResidenceIndicator().validate("X12345").valid
# ── IND-301 证件类型 + 号码一致性（跨字段）──
class TestIdTypeNumberConsistency:
def test_consistent_id_card(self):
rows = [{"id_type": "01", "id_no": "11010519491231002X"}]
out = m301.IdTypeNumberConsistencyIndicator().validate_rows(rows)
assert len(out) == 0
def test_inconsistent_passport(self):
# 03 护照 → IND-202 校验
rows = [{"id_type": "03", "id_no": "X12345"}] # 不符合护照格式
out = m301.IdTypeNumberConsistencyIndicator().validate_rows(rows)
assert len(out) == 1
def test_invalid_type(self):
rows = [{"id_type": "99", "id_no": "X"}]
out = m301.IdTypeNumberConsistencyIndicator().validate_rows(rows)
assert len(out) == 1
assert out[0]["rule"] == "type_invalid"
# ── IND-401 姓名字符集 ──
class TestNameCharset:
def test_valid_chinese(self):
assert m401.NameCharsetIndicator().validate("张三").valid
def test_valid_with_minzu_dot(self):
assert m401.NameCharsetIndicator().validate("额尔德木图·巴特巴依尔").valid
def test_invalid_pure_digits(self):
assert not m401.NameCharsetIndicator().validate("123456").valid
def test_invalid_special_chars(self):
assert not m401.NameCharsetIndicator().validate("张三<>").valid
def test_empty_skipped(self):
assert m401.NameCharsetIndicator().validate("").valid
# ── IND-402 姓名长度 ──
class TestNameLength:
def test_valid(self):
assert m402.NameLengthIndicator().validate("张三").valid
def test_too_short(self):
assert not m402.NameLengthIndicator().validate("张").valid
def test_too_long(self):
assert not m402.NameLengthIndicator().validate("a" * 21).valid
# ── IND-501 出生日期格式 ──
class TestBirthDateFormat:
def test_valid_ymd(self):
assert m501.BirthDateFormatIndicator().validate("19900101").valid
def test_valid_with_dash(self):
assert m501.BirthDateFormatIndicator().validate("1990-01-01").valid
def test_invalid(self):
assert not m501.BirthDateFormatIndicator().validate("1990-13-01").valid
# ── IND-502 出生日期年龄范围 ──
class TestBirthDateAgeRange:
def test_valid_adult(self):
# 1990-01-01 → 36 岁
result = m502.BirthDateAgeRangeIndicator().validate("19900101")
assert result.valid
def test_too_young(self):
# 2020-01-01 → ~6 岁（< 16）
assert not m502.BirthDateAgeRangeIndicator().validate("20200101").valid
def test_with_gender_male(self):
# 男 60 退休 —— 1980-01-01 ≈ 46 岁，应合规
result = m502.BirthDateAgeRangeIndicator().validate_with_gender("19800101", "1")
assert result.valid
def test_overage_male(self):
# 男 1960-01-01 ≈ 66 岁 > 60（男退休年龄），应违规
result = m502.BirthDateAgeRangeIndicator().validate_with_gender("19600101", "1")
assert not result.valid
def test_overage_female_worker(self):
# 女工 1965-01-01 ≈ 61 岁 > 50，应违规
result = m502.BirthDateAgeRangeIndicator().validate_with_gender("19650101", "2")
assert not result.valid
# ── IND-601 行政区划代码 ──
class TestDistrictCode:
def test_valid(self):
assert m601.DistrictCodeIndicator().validate("110000").valid
def test_invalid_format(self):
assert not m601.DistrictCodeIndicator().validate("12345").valid
def test_invalid_code(self):
assert not m601.DistrictCodeIndicator().validate("110999").valid
# ── IND-602 民族代码 ──
class TestEthnicity:
def test_valid_han(self):
assert m602.EthnicityIndicator().validate("01").valid
def test_valid_zhuang(self):
assert m602.EthnicityIndicator().validate("08").valid
def test_valid_other_97(self):
assert m602.EthnicityIndicator().validate("97").valid
def test_invalid(self):
assert not m602.EthnicityIndicator().validate("99").valid
# ── IND-603 婚姻状况 ──
class TestMarriageStatus:
def test_valid_unmarried(self):
assert m603.MarriageStatusIndicator().validate("10").valid
def test_valid_married(self):
assert m603.MarriageStatusIndicator().validate("20").valid
def test_invalid(self):
assert not m603.MarriageStatusIndicator().validate("11").valid
# ── 注册器 / 协议层 ──
class TestRegistryAndProtocol:
def test_all_indicators_listed(self):
from standards.registry import list_all_standards
all_std = list_all_standards()
# 至少 24 个 indicator（不含 4 个 deprecated）
new_inds = [s for s in all_std if s["id"].startswith("IND-")]
assert len(new_inds) == 24, f"期望 24 个 indicator，实际 {len(new_inds)}"
def test_all_groups_present(self):
from standards.registry import list_standards_by_group
by_group = list_standards_by_group()
assert "国标字段规范" in by_group
assert "证件信息" in by_group
assert "身份信息" in by_group
assert "民政信息" in by_group
def test_deprecated_have_empty_fields(self):
from standards.registry import list_all_standards
deprecated = [
s for s in list_all_standards()
if s["id"] in ("STD-001", "STD-002", "STD-003", "STD-004")
]
assert len(deprecated) == 4
for s in deprecated:
assert s["applies_to_fields"] == []
def test_discover_skips_deprecated(self):
from standards.registry import discover_standards
discovered = discover_standards()
# 不应包含已 deprecated 的 4 个（其 applies_to_fields=[]）
for s_id in ("STD-001", "STD-002", "STD-003", "STD-004"):
assert s_id not in str(discovered.values())
def test_base_standard_has_group_severity(self):
from standards.base import BaseStandard
# 类属性（不要实例化 —— validate 是抽象方法）
assert BaseStandard.group == "通用"
assert BaseStandard.severity == "warning"
# depends_on 是 dataclass field；通过子类实例验证能正确求值
from standards.ind_001a_id_card_format import IdCardFormatIndicator
inst = IdCardFormatIndicator()
assert list(inst.depends_on) == []
def test_base_standard_validate_rows_default(self):
from standards.base import BaseStandard
from standards.ind_001a_id_card_format import IdCardFormatIndicator
ind = IdCardFormatIndicator()
# 默认实现：每行取 applies_to_fields[0]
rows = [
{"id_card": "11010519491231002X"},
{"id_card": "bad"},
]
violations = ind.validate_rows(rows)
assert len(violations) == 1
# ── Data loader ──
class TestDataLoader:
def test_marriage_codes(self):
from standards.data_loader import is_valid_marriage, load_marriage_codes
assert is_valid_marriage("10")
assert is_valid_marriage("90")
assert not is_valid_marriage("11")
assert len(load_marriage_codes()) == 8
def test_id_type_enum(self):
from standards.data_loader import is_valid_id_type
for c in ("01", "02", "03", "04", "05"):
assert is_valid_id_type(c)
assert not is_valid_id_type("06")
def test_district_codes_loaded(self):
from standards.data_loader import load_district_codes
codes = load_district_codes()
assert "110000" in codes
assert "710000" in codes # 台湾
assert "810000" in codes # 香港
assert "820000" in codes # 澳门
def test_ethnicity_codes(self):
from standards.data_loader import is_valid_ethnicity
for c in ("01", "08", "56"):
assert is_valid_ethnicity(c)
assert not is_valid_ethnicity("99")
def test_iso_country_alpha3(self):
from standards.data_loader import load_iso_country_codes_3
codes = load_iso_country_codes_3()
assert "USA" in codes
assert "CHN" in codes
# ── Step 7 协议形状 ──
class TestStep7Protocol:
def test_empty_fallback_4_tabs(self):
from web.core.step_impl.step7_standards import _EMPTY_FALLBACK
result = _EMPTY_FALLBACK()
sections = result["sections"]
assert len(sections) == 4
for group, sec in sections.items():
assert "_protocol" in sec
assert "tabs" in sec["_protocol"]
assert len(sec["_protocol"]["tabs"]) == 1
def test_each_tab_has_required_fields(self):
from web.core.step_impl.step7_standards import _TABS_BY_GROUP
for group, tab in _TABS_BY_GROUP.items():
assert tab.key.startswith("standards_")
assert tab.title == group
assert len(tab.kpis) >= 1
assert len(tab.tables) == 2 # 按 indicator 聚合 + 违规明细
assert tab.linkage is not None
assert tab.linkage.target_table == 1
if __name__ == "__main__":
pytest.main([__file__, "-v"])
\ No newline at end of file
web/core/orchestrator.py
View file @ 5c37925e
......@@ -106,7 +106,20 @@ def _run_length_check(*, cfg, dict_data, llm, log, cancel_event, table_filter):
return {"section_key": "length_issues", "data": data}
def _run_standards(*, cfg, dict_data, llm, log, cancel_event, table_filter):
def _run_standards_one(standard_id: str):
"""返回单个 indicator step 的 runner（捕获 standard_id 闭包）。"""
def _runner(*, cfg, dict_data, llm, log, cancel_event, table_filter):
from .step_impl.step7_standards import run_step7_for_indicator
return run_step7_for_indicator(
standard_id, cfg, dict_data=dict_data, log=log, table_filter=table_filter,
)
return _runner
def _run_standards(*, cfg, dict_data, llm, log, cancel_event, table_filter): # noqa: ARG001
"""旧版「国家标准校验」聚合入口 —— 已被 per-indicator 模式取代，保留以兼容历史调用。
实际 orchestrator 不再注册此 step；用户勾选的是 std_ind_* 单 indicator。
"""
from .step_impl.step7_standards import run_step7
data = run_step7(cfg, dict_data=dict_data, log=log, table_filter=table_filter)
return {"section_key": "standard_violations", "data": data}
......@@ -141,13 +154,69 @@ register_step(
requires_db=False, llm_mode="none", required=False, order=4,
fn=_run_length_check,
)
register_step(
step_id="standards",
title="国家标准校验",
description="连库抽样：身份证/USCC/手机号/行政区划符合性",
requires_db=True, llm_mode="none", required=False, order=5,
fn=_run_standards,
)
# ── Indicator 步骤：从 standards 包动态注册 24+ 个 ──────────────
# 每个 indicator 是 1 个 step（用户可单独勾选）。
# order：基础步骤 1-4 → 国标字段规范 100-199 → 证件信息 200-299 → 身份信息 300-399 → 民政信息 400-499
_GROUP_ORDER_START = {
"国标字段规范": 100,
"证件信息": 200,
"身份信息": 300,
"民政信息": 400,
"通用": 500,
}
# 在 import time 触发 standards 包扫描，拿到所有 indicator 元数据
try:
from standards.registry import list_step_ids as _list_std_steps
_INDICATOR_STEPS: list[tuple[str, dict]] = _list_std_steps()
except Exception as _e: # pragma: no cover
logger.warning("加载 indicator 步骤列表失败: %s", _e)
_INDICATOR_STEPS = []
def _register_indicator_steps() -> None:
"""动态注册每个 indicator 为一个 orchestrator step。
顺序：先按 group（国标字段规范 → 证件信息 → 身份信息 → 民政信息），
同 group 内按 standard_id 字典序。
过滤：
- 跳过 std_id 以 "STD-" 开头的已拆分旧插件（applies_to_fields=[]）
- 保留 IND-301 / IND-302 等无 applies_to_fields 的跨字段 indicator
"""
prev_group: str | None = None
same_group_count = 0
base_order: int = 0
for sid, meta in _INDICATOR_STEPS:
std_id = meta["id"]
# 跳过已 deprecated 的旧插件（STD-*，已拆分为 IND-* 子指标）
if std_id.startswith("STD-"):
logger.info(f"跳过已 deprecated 旧插件: {std_id}（已被 IND-* 拆分）")
continue
group = meta.get("group", "通用")
if group != prev_group:
same_group_count = 0
base_order = _GROUP_ORDER_START.get(group, 900)
prev_group = group
same_group_count += 1
order = base_order + same_group_count
title = f"{std_id} · {meta['name']}"
description = (
f"分组: {group} | 适用字段: {', '.join(meta.get('applies_to_fields', [])) or '(跨字段/跨表)'} | "
f"{meta.get('description', '')}"
)
register_step(
step_id=sid,
title=title,
description=description,
requires_db=True, llm_mode="none", required=False, order=order,
fn=_run_standards_one(std_id),
)
_register_indicator_steps()
# ── 关键 Step（失败直接终止任务） ──────────────────────────────
......
web/core/step_impl/step7_standards.py
View file @ 5c37925e
"""国家标准字段校验（连库）
"""国家标准字段校验（连库）—— 每个 indicator 注册为独立 step
按字段名匹配 standards 插件，抽样校验实际数据：
- 身份证号（GB 11643）
- 统一社会信用代码（GB 32100）
- 手机号（YD/T 1313）
- 行政区划代码（GB/T 2260）
按 standards/registry 自动发现所有 BaseStandard 子类，**每个 indicator 单独注册为
一个 orchestrator step**（step_id 形如 `std_ind_001a`）。用户在「要跑的检查」里
可以单独勾选任意 indicator；结果按 indicator 输出独立 section / 独立 tab。
抽样 SQL 走 web/sql/standards/sample_field_values.sql 模板。
执行路径分三类：
- 单值类（22 个 indicator）：按 `applies_to_fields` 自动发现，单字段抽样 + validate(value)
- 跨字段类（IND-301）：按字段配对抽样 + validate_rows
- 跨表聚合类（IND-302）：GROUP BY HAVING 检出重复
向后兼容：旧版 `run_step7()` 仍可调（跑所有 indicator 并按 group 输出 4 tab）。
抽样 SQL：
- 单值：web/sql/standards/sample_field_values.sql
- 跨字段：web/sql/standards/sample_field_pairs.sql
- 跨表聚合：web/sql/standards/sample_field_groups.sql
"""
from __future__ import annotations
......@@ -16,164 +24,413 @@ from typing import Callable
from ..db_adapter import DBConfig, open_db, quote_ident
from ..tab_protocol import (
TabProtocol, TableSpec, ColumnSpec, RenderSpec, TagStyle, Threshold,
TabProtocol, TableSpec, ColumnSpec, RenderSpec, Threshold,
KpiSpec, SummarySpec, LinkageSpec, inject_protocol,
)
from standards.registry import discover_standards, list_all_standards
from standards.registry import (
discover_standards, list_all_standards, list_standards_by_group,
load_standard_class, indicator_step_id, step_id_to_indicator_id, list_step_ids,
)
from standards.base import BaseStandard, ValidationResult
from web.sql.loader import get_sql_loader
logger = logging.getLogger(__name__)
# ── 协议 ──────────────────────────────────────────────
# 单 tab「国家标准校验」，含 2 张子表 + 跨表联动：
# 1. violations_by_standard —— 每个标准一行，展开看每字段样本 + 点击字段联动到 violations
# 2. violations —— 全部违规值的扁平表（默认前 200 条，选中字段后按 (table,field) 过滤）
# 联动规则：
# - 在 table 0 行的展开区点击 details_with_violations 中的某个 {table, field}
# - 前端把 {table, field} 写入 selectedField
# - filteredViolations = this.violations.filter(v =>
# v[linkage.field_map['table']] === selectedField.table
# && v[linkage.field_map['field']] === selectedField.field)
# - 未选中时：默认按 default_filter 截断 (limit 200)
_STANDARDS_TAB = TabProtocol(
key="standards",
title="国家标准校验",
kpis=(
KpiSpec(css_class="kpi-standards", label="违规字段",
value="this.summary.violations"),
KpiSpec(css_class="kpi-standards-applied", label="已应用标准",
value="this.summary.standards_used"),
),
summary=SummarySpec(
type="error",
template=(
"应用 {{applied}} 个标准, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
# ── Step 排序常量 ──────────────────────────────────────
# Indicator group 的 UI 显示顺序；同 group 内按标准 ID 字典序排
_GROUP_ORDER = {
"国标字段规范": 0,
"证件信息": 1,
"身份信息": 2,
"民政信息": 3,
"通用": 4,
}
# ── Per-indicator 单 tab 协议模板 ────────────────────────
def _make_single_indicator_tab(
*, key: str, title: str,
) -> TabProtocol:
"""每个 indicator 一个 tab —— 走「字段汇总 + 违规明细」最小两表结构。"""
return TabProtocol(
key=key,
title=title,
kpis=(
KpiSpec(css_class=f"kpi-{key}", label="违规字段",
value="this.summary.violations"),
KpiSpec(css_class=f"kpi-{key}-fields", label="检查字段",
value="this.summary.fields_checked"),
),
vars={
"applied": "this.summary.standards_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
tables=(
# 表 1（按标准聚合）
TableSpec(
title="按标准聚合",
source="this.violations_by_standard",
default_sort={"prop": "violation_rate", "order": "descending"},
max_height=560,
columns=(
ColumnSpec(type="expand", template=(
"<div class=\"expand-block\">"
"<p><strong>有违规字段（点击下方字段联动到右侧违规明细）：</strong></p>"
"<div class=\"field-link-list\">"
"{{ details_with_violations | field_links }}"
"</div>"
"<p class=\"muted\" style=\"margin-top: 8px; font-size: 12px\">"
"取数口径: sample_limit={{ sample_limit }}, "
"max_tables_per_field={{ max_tables_per_field }}"
"</p>"
)),
ColumnSpec(prop="standard", label="标准代号", width=140,
render=RenderSpec(kind="code")),
ColumnSpec(prop="standard_name", label="标准名称", width=240,
show_overflow_tooltip=True),
ColumnSpec(prop="fields_count", label="检查字段数", width=110,
align="center"),
ColumnSpec(prop="violating_fields_count", label="违规字段数",
width=120, align="center"),
ColumnSpec(
label="违规率 (违规/抽样)", prop="violation_rate", width=170,
align="center",
render=RenderSpec(
kind="percent",
value_path="violation_rate",
precision=2,
thresholds=(
Threshold(gt=0.5, color="#f56c6c"),
Threshold(gt=0.2, color="#e6a23c"),
Threshold(gt=0.0, color="#67c23a"),
summary=SummarySpec(
type="warning",
template=(
"应用 {{applied}} 个指标, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
),
vars={
"applied": "this.summary.indicators_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
tables=(
# 表 1：按 indicator 聚合（每 indicator 一行）
TableSpec(
title="指标汇总",
source="this.violations_by_indicator",
default_sort={"prop": "violation_rate", "order": "descending"},
max_height=400,
columns=(
ColumnSpec(prop="indicator", label="指标编号", width=140,
render=RenderSpec(kind="code")),
ColumnSpec(prop="indicator_name", label="指标名称", min_width=200,
show_overflow_tooltip=True),
ColumnSpec(prop="fields_count", label="检查字段数", width=110,
align="center"),
ColumnSpec(prop="violating_fields_count", label="违规字段数",
width=120, align="center"),
ColumnSpec(
label="违规率 (违规/抽样)", prop="violation_rate", width=170,
align="center",
render=RenderSpec(
kind="percent",
value_path="violation_rate",
precision=2,
thresholds=(
Threshold(gt=0.5, color="#f56c6c"),
Threshold(gt=0.2, color="#e6a23c"),
Threshold(gt=0.0, color="#67c23a"),
),
),
),
),
),
# 表 2：违规明细
TableSpec(
title="违规明细",
source="this.violations",
default_sort=None,
max_height=520,
columns=(
ColumnSpec(
label="表名 / 表注释", width=260,
render=RenderSpec(
kind="two_line",
main_path="table_name",
sub_path="table_comment",
sub_empty_text="（无表注释）",
),
),
ColumnSpec(
label="字段名 / 字段注释", width=240,
render=RenderSpec(
kind="two_line",
main_path="column_name",
sub_path="column_comment",
sub_empty_text="（无字段注释）",
),
),
ColumnSpec(prop="rule_type", label="规则", width=160,
render=RenderSpec(kind="code")),
ColumnSpec(prop="value", label="违规值", width=180,
render=RenderSpec(kind="code")),
ColumnSpec(prop="error", label="违规原因", min_width=240,
show_overflow_tooltip=True),
),
),
),
# 表 2（违规明细，受联动过滤）
TableSpec(
title="违规明细",
source="this.violations",
default_sort=None, # 由 linkage 接管过滤
max_height=560,
columns=(
ColumnSpec(
label="表名 / 表注释", width=260,
render=RenderSpec(
kind="two_line",
main_path="table_name",
sub_path="table_comment",
sub_empty_text="（无表注释）",
)
def _wrap_single_indicator(indicator: BaseStandard, data: dict) -> dict:
"""包装单个 indicator 结果为 1 section / 1 tab。"""
std_id = indicator.standard_id
step_id_str = indicator_step_id(std_id) # "std_ind_001_a"
safe_id = step_id_str[len("std_"):] # "ind_001_a"
key = f"standards_{safe_id}" # "standards_ind_001_a"
tab = _make_single_indicator_tab(key=key, title=f"{std_id} · {indicator.standard_name}")
data = dict(data)
data["_protocol"] = {
"step_id": indicator_step_id(std_id),
"tabs": [{
"key": tab.key,
"title": tab.title,
"kpis": [
{"css_class": k.css_class, "label": k.label, "value": k.value}
for k in tab.kpis
],
"summary": {
"type": tab.summary.type,
"template": tab.summary.template,
"vars": dict(tab.summary.vars),
},
"tables": [
{
"title": t.title,
"source": t.source,
"default_sort": t.default_sort,
"max_height": t.max_height,
"columns": [
{
"prop": c.prop, "label": c.label, "type": c.type,
"width": c.width, "min_width": c.min_width,
"fixed": c.fixed, "align": c.align,
"show_overflow_tooltip": c.show_overflow_tooltip,
"sortable": c.sortable,
"render": (
{
"kind": c.render.kind,
"value_path": c.render.value_path,
"status_from_value_path": c.render.status_from_value_path,
"precision": c.render.precision,
"thresholds": (
[{"gt": th.gt, "color": th.color} for th in c.render.thresholds]
if c.render.thresholds else []
),
"main_path": c.render.main_path,
"sub_path": c.render.sub_path,
"sub_empty_text": c.render.sub_empty_text,
}
if c.render else None
),
"template": c.template,
}
for c in t.columns
],
}
for t in tab.tables
],
}],
}
return data
# ── 4 个 group tab 协议 ──────────────────────────────────────────
# 每个 tab 内部仍走「按 indicator 聚合 + 扁平违规明细 + 联动」三件套结构。
def _make_group_tab(
*,
key: str,
title: str,
kpi_label_violations: str,
kpi_label_indicators: str,
summary_template: str,
summary_vars: dict[str, str],
) -> TabProtocol:
"""4 个 group tab 共用同一结构 —— 用工厂避免 4 段重复代码。"""
return TabProtocol(
key=key,
title=title,
kpis=(
KpiSpec(css_class=f"kpi-{key}", label=kpi_label_violations,
value="this.summary.violations"),
KpiSpec(css_class=f"kpi-{key}-applied", label=kpi_label_indicators,
value="this.summary.indicators_used"),
),
summary=SummarySpec(
type="error",
template=summary_template,
vars=summary_vars,
),
tables=(
# 表 1（按 indicator 聚合）
TableSpec(
title="按 indicator 聚合",
source="this.violations_by_indicator",
default_sort={"prop": "violation_rate", "order": "descending"},
max_height=560,
columns=(
ColumnSpec(type="expand", template=(
"<div class=\"expand-block\">"
"<p><strong>有违规字段（点击下方字段联动到右侧违规明细）：</strong></p>"
"<div class=\"field-link-list\">"
"{{ details_with_violations | field_links }}"
"</div>"
"<p class=\"muted\" style=\"margin-top: 8px; font-size: 12px\">"
"取数口径: sample_limit={{ sample_limit }}, "
"max_tables_per_field={{ max_tables_per_field }}"
"</p>"
)),
ColumnSpec(prop="indicator", label="指标编号", width=140,
render=RenderSpec(kind="code")),
ColumnSpec(prop="indicator_name", label="指标名称", width=240,
show_overflow_tooltip=True),
ColumnSpec(prop="fields_count", label="检查字段数", width=110,
align="center"),
ColumnSpec(prop="violating_fields_count", label="违规字段数",
width=120, align="center"),
ColumnSpec(
label="违规率 (违规/抽样)", prop="violation_rate", width=170,
align="center",
render=RenderSpec(
kind="percent",
value_path="violation_rate",
precision=2,
thresholds=(
Threshold(gt=0.5, color="#f56c6c"),
Threshold(gt=0.2, color="#e6a23c"),
Threshold(gt=0.0, color="#67c23a"),
),
),
),
),
ColumnSpec(
label="字段名 / 字段注释", width=240,
render=RenderSpec(
kind="two_line",
main_path="column_name",
sub_path="column_comment",
sub_empty_text="（无字段注释）",
),
# 表 2（违规明细，受联动过滤）
TableSpec(
title="违规明细",
source="this.violations",
default_sort=None,
max_height=560,
columns=(
ColumnSpec(
label="表名 / 表注释", width=260,
render=RenderSpec(
kind="two_line",
main_path="table_name",
sub_path="table_comment",
sub_empty_text="（无表注释）",
),
),
ColumnSpec(
label="字段名 / 字段注释", width=240,
render=RenderSpec(
kind="two_line",
main_path="column_name",
sub_path="column_comment",
sub_empty_text="（无字段注释）",
),
),
ColumnSpec(prop="rule_type", label="规则", width=160,
render=RenderSpec(kind="code")),
ColumnSpec(prop="value", label="违规值", width=180,
render=RenderSpec(kind="code")),
ColumnSpec(prop="error", label="违规原因", min_width=240,
show_overflow_tooltip=True),
),
ColumnSpec(prop="rule_type", label="规则", width=160,
render=RenderSpec(kind="code")),
ColumnSpec(prop="value", label="违规值", width=180,
render=RenderSpec(kind="code")),
ColumnSpec(prop="error", label="违规原因", min_width=240,
show_overflow_tooltip=True),
),
),
linkage=LinkageSpec(
source_table=0,
clickable_path="details_with_violations",
clickable_id_props=("table", "field"),
target_table=1,
field_map={"table": "table_name", "field": "column_name"},
default_filter_kind="limit",
default_filter_value=200,
),
)
_TABS_BY_GROUP = {
"国标字段规范": _make_group_tab(
key="standards_gb_format",
title="国标字段规范",
kpi_label_violations="违规字段",
kpi_label_indicators="已应用指标",
summary_template=(
"应用 {{applied}} 个指标, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
),
summary_vars={
"applied": "this.summary.indicators_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
linkage=LinkageSpec(
source_table=0,
clickable_path="details_with_violations",
clickable_id_props=("table", "field"),
target_table=1,
field_map={"table": "table_name", "field": "column_name"},
default_filter_kind="limit",
default_filter_value=200,
"证件信息": _make_group_tab(
key="standards_document",
title="证件信息",
kpi_label_violations="违规字段",
kpi_label_indicators="已应用指标",
summary_template=(
"应用 {{applied}} 个指标, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
),
summary_vars={
"applied": "this.summary.indicators_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
)
"身份信息": _make_group_tab(
key="standards_personal",
title="身份信息",
kpi_label_violations="违规字段",
kpi_label_indicators="已应用指标",
summary_template=(
"应用 {{applied}} 个指标, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
),
summary_vars={
"applied": "this.summary.indicators_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
"民政信息": _make_group_tab(
key="standards_civil",
title="民政信息",
kpi_label_violations="违规字段",
kpi_label_indicators="已应用指标",
summary_template=(
"应用 {{applied}} 个指标, 检查 {{fields}} 个字段, "
"发现 {{violations}} 条违规"
),
summary_vars={
"applied": "this.summary.indicators_applied",
"fields": "this.summary.fields_checked",
"violations": "this.summary.violations",
},
),
}
def _wrap(data: dict) -> dict:
return inject_protocol(data, step_id="standards", tabs=[_STANDARDS_TAB])
def _wrap(per_group: dict[str, dict]) -> dict:
"""包装协议：per_group = {group_name: data_dict} → 多 tab 注入。"""
sections = {}
for group, data in per_group.items():
tab = _TABS_BY_GROUP.get(group)
if tab is None:
# 未知 group —— 仍走协议（兜底 tab）
tab = _make_group_tab(
key=f"standards_{group}",
title=group,
kpi_label_violations="违规字段",
kpi_label_indicators="已应用指标",
summary_template="应用 {{applied}} 个指标, 发现 {{violations}} 条违规",
summary_vars={
"applied": "this.summary.indicators_applied",
"violations": "this.summary.violations",
},
)
sections[group] = inject_protocol(data, step_id="standards", tabs=[tab])
return {"sections": sections}
def _EMPTY_FALLBACK() -> dict:
"""无字段/无标准插件时的兜底 —— 仍走协议。"""
return _wrap({
def _empty_group_data() -> dict:
return {
"summary": {
"standards_applied": 0,
"standards_used": 0,
"indicators_applied": 0,
"indicators_used": 0,
"fields_checked": 0,
"violations": 0,
},
"violations_by_standard": [],
"violations_by_indicator": [],
"violations": [],
"standards_info": [],
})
"indicators_info": [],
}
def _EMPTY_FALLBACK() -> dict:
"""无字段/无标准插件时的兜底 —— 每个 group 走一个空 tab。"""
return _wrap({g: _empty_group_data() for g in _TABS_BY_GROUP.keys()})
SAMPLE_LIMIT = 500 # 每字段最多抽样
MAX_TABLES_PER_FIELD = 5 # 每字段最多检查前 N 张表
# 取数口径列表 —— 一个标准可对应多条抽样 SQL（按 spec_id 区分）
# 当前所有标准都共享同一默认口径；后续若不同标准需不同抽样策略，
# 在此处按 standard_id 配置多条即可。
# 单字段抽样 SQL
SAMPLE_SPECS: list[dict] = [
{
"id": "default",
......@@ -190,12 +447,537 @@ SAMPLE_SPECS: list[dict] = [
]
def _empty_indicator_data() -> dict:
"""单 indicator 的空数据。"""
return {
"summary": {
"indicators_applied": 1,
"indicators_used": 0,
"fields_checked": 0,
"violations": 0,
},
"violations_by_indicator": [],
"violations": [],
"indicators_info": [],
}
def _run_round1_one(
cfg: DBConfig,
by_name: dict[str, list[dict]],
std_instance: BaseStandard,
log: Callable | None,
) -> dict:
"""单值校验：跑一个 indicator 实例（适用于 22 个单值类 indicator）。
返回单 indicator 数据结构（适用 _wrap_single_indicator）。
若该 indicator 在数据字典里没命中任何字段 → 返回 _empty_indicator_data()。
"""
loader = get_sql_loader()
bucket = _empty_indicator_data()
std_id = std_instance.standard_id
if log:
log("INFO",
f" · {std_id} ({getattr(std_instance, 'group', '通用')}) 启动",
step=indicator_step_id(std_id))
# 找该 indicator 命中的所有字段（applies_to_fields 中任意一个）
applicable = list(getattr(std_instance, "applies_to_fields", []) or [])
fields_to_check: list[dict] = []
for f in applicable:
if f in by_name:
for c in by_name[f][:MAX_TABLES_PER_FIELD]:
# 标记命中的字段名（一个 indicator 可能命中多列）
fields_to_check.append({**c, "_hit_field": f})
if not fields_to_check:
if log:
log("DEBUG",
f" · {std_id} 未匹配任何字段（适用 {applicable}），跳过",
step=indicator_step_id(std_id))
return bucket
rec = {
"indicator": std_id,
"indicator_name": std_instance.standard_name,
"fields": [],
"total_invalid": 0,
"total_sampled": 0,
}
try:
with open_db(cfg) as db:
for col_record in fields_to_check:
table = col_record["table_name"]
field_name = col_record["column_name"]
bucket["summary"]["fields_checked"] += 1
try:
sql = loader.render(
"standards/sample_field_values",
dialect=cfg.db_type,
table=table,
field=field_name,
limit=SAMPLE_LIMIT,
)
rows = db.fetchall(sql)
sample_sql_rendered = sql
except Exception as e:
rec["fields"].append({
"table": table,
"field": field_name,
"error": str(e),
"violations": 0,
"sampled": 0,
"sample_spec_id": "default",
"sample_sql": f"SELECT `{field_name}` ... -- 渲染失败: {e}",
})
if log:
log("WARN",
f" - {table}.{field_name} 抽样失败: {e}",
step=indicator_step_id(std_id))
continue
sampled = len(rows)
invalid_samples = []
invalid_count = 0
for row in rows:
val = row.get("val")
if val is None:
continue
result: ValidationResult = std_instance.validate(str(val))
if not result.valid:
invalid_count += 1
if len(invalid_samples) < 10:
invalid_samples.append({
"value": str(val)[:50],
"reason": result.reason,
})
bucket["violations"].append({
"table_name": table,
"table_comment": col_record.get("table_comment", ""),
"column_name": field_name,
"column_comment": col_record.get("column_comment", ""),
"rule_type": std_id,
"standard_name": std_instance.standard_name,
"column_type": col_record.get("column_type", ""),
"value": str(val)[:50],
"error": result.reason,
})
rec["total_invalid"] += invalid_count
rec["total_sampled"] += sampled
rec["fields"].append({
"table": table,
"field": field_name,
"sampled": sampled,
"violations": invalid_count,
"violation_rate": round(invalid_count / sampled, 4) if sampled else 0,
"samples": invalid_samples,
"sample_spec_id": "default",
"sample_sql": sample_sql_rendered,
})
logger.debug(
f"{table}.{field_name}: 抽样 {sampled}, 违规 {invalid_count}"
)
except Exception as e:
if log:
log("ERROR", f"连库校验失败: {type(e).__name__}: {e}",
step=indicator_step_id(std_id))
logger.exception("详细异常")
if rec["total_invalid"] > 0:
violating_fields = [d for d in rec["fields"] if d.get("violations", 0) > 0]
bucket["violations_by_indicator"].append({
"indicator": rec["indicator"],
"indicator_name": rec["indicator_name"],
"fields_count": len(rec["fields"]),
"violating_fields_count": len(violating_fields),
"total_invalid": rec["total_invalid"],
"total_sampled": rec["total_sampled"],
"violation_rate": (
round(rec["total_invalid"] / rec["total_sampled"], 4)
if rec["total_sampled"] else 0
),
"details": rec["fields"],
"details_with_violations": violating_fields,
"sample_specs": [dict(s) for s in SAMPLE_SPECS],
"sample_limit": SAMPLE_LIMIT,
"max_tables_per_field": MAX_TABLES_PER_FIELD,
})
bucket["summary"]["indicators_used"] = len(bucket["violations_by_indicator"])
bucket["summary"]["violations"] = len(bucket["violations"])
bucket["violations"] = bucket["violations"][:500]
bucket["indicators_info"] = [rec]
if log:
log("INFO",
f" · {std_id} 完成: {bucket['summary']['violations']} 条违规, "
f"{bucket['summary']['fields_checked']} 个字段",
step=indicator_step_id(std_id))
return bucket
def _run_round1(
cfg: DBConfig,
by_name: dict[str, list[dict]],
standards_by_field: dict[str, BaseStandard],
log: Callable | None,
) -> dict:
"""Round 1：单值校验（按 group 聚合结果）—— 跑全部 indicator，按 group 输出。"""
per_group: dict[str, dict] = {
g: {
"summary": {"indicators_applied": 0, "indicators_used": 0,
"fields_checked": 0, "violations": 0},
"violations_by_indicator": [],
"violations": [],
"indicators_info": [],
}
for g in _TABS_BY_GROUP.keys()
}
seen_instances: dict[str, BaseStandard] = {}
for std_instance in standards_by_field.values():
sid = std_instance.standard_id
if sid not in seen_instances:
seen_instances[sid] = std_instance
if log:
log("INFO",
f"Round 1 启动: 共 {len(seen_instances)} 个 indicator, "
f"按 applies_to_fields 命中 {len(standards_by_field)} 个字段",
step="standards")
for std_id, std_instance in seen_instances.items():
single = _run_round1_one(cfg, by_name, std_instance, log)
group = getattr(std_instance, "group", "通用")
bucket = per_group.setdefault(group, _empty_group_data())
# 合并：violations_by_indicator + violations
bucket["violations_by_indicator"].extend(single["violations_by_indicator"])
bucket["violations"].extend(single["violations"])
bucket["summary"]["fields_checked"] += single["summary"]["fields_checked"]
bucket["indicators_info"].extend(single["indicators_info"])
# ── 填 summary ──
for group, bucket in per_group.items():
bucket["summary"]["indicators_applied"] = sum(
1 for v in bucket["indicators_info"]
)
bucket["summary"]["indicators_used"] = len(bucket["violations_by_indicator"])
bucket["summary"]["violations"] = len(bucket["violations"])
bucket["violations"] = bucket["violations"][:500]
if log:
log("INFO",
f"Round 1 完成: {sum(b['summary']['violations'] for b in per_group.values())} 条违规, "
f"{sum(b['summary']['fields_checked'] for b in per_group.values())} 个字段",
step="standards")
return per_group
def _run_round2_cross_field_one(
cfg: DBConfig,
by_name: dict[str, list[dict]],
log: Callable | None,
) -> dict:
"""跨字段校验 IND-301：证件类型 + 证件号 一致性 —— 单 indicator 输出。"""
from standards.ind_301_id_type_number_consistency import IdTypeNumberConsistencyIndicator
loader = get_sql_loader()
bucket = _empty_indicator_data()
ind301 = IdTypeNumberConsistencyIndicator()
std_id = ind301.standard_id
# 找同时含 id_type + id_no 字段的表
pairs: list[tuple[str, str, str]] = []
for table, cols in by_name.items():
type_cols = [
c["column_name"] for c in cols
if c["column_name"] in ("id_type", "cert_type", "document_type", "zjlb")
]
no_cols = [
c["column_name"] for c in cols
if c["column_name"] in ("id_no", "cert_no", "zjhm", "document_no")
]
for tc in type_cols:
for nc in no_cols:
pairs.append((table, tc, nc))
if not pairs:
if log:
log("INFO", f" · {std_id} 未找到 (id_type, id_no) 字段配对，跳过",
step=indicator_step_id(std_id))
return bucket
if log:
log("INFO",
f" · {std_id} (跨字段): {len(pairs)} 对 (id_type, id_no) 字段",
step=indicator_step_id(std_id))
indicator_rec = {
"indicator": ind301.standard_id,
"indicator_name": ind301.standard_name,
"fields": [],
"total_invalid": 0,
"total_sampled": 0,
}
try:
with open_db(cfg) as db:
for table, tc, nc in pairs:
try:
sql = loader.render(
"standards/sample_field_pairs",
dialect=cfg.db_type,
table=table,
type_col=tc,
no_col=nc,
limit=SAMPLE_LIMIT,
)
rows = db.fetchall(sql)
except Exception as e:
if log:
log("WARN",
f" - {table}.({tc},{nc}) 抽样失败: {e}",
step=indicator_step_id(std_id))
continue
sampled = len(rows)
violations = ind301.validate_rows(rows)
bucket["summary"]["fields_checked"] += 1
indicator_rec["total_invalid"] += len(violations)
indicator_rec["total_sampled"] += sampled
indicator_rec["fields"].append({
"table": table,
"field": f"({tc}, {nc})",
"sampled": sampled,
"violations": len(violations),
"violation_rate": (
round(len(violations) / sampled, 4) if sampled else 0
),
"samples": violations[:10],
"sample_spec_id": "round2_cross_field",
"sample_sql": sql,
})
for v in violations[:50]:
bucket["violations"].append({
"table_name": table,
"table_comment": "",
"column_name": nc,
"column_comment": "",
"rule_type": ind301.standard_id,
"standard_name": ind301.standard_name,
"column_type": "",
"value": (v.get("value") or "")[:50],
"error": v.get("reason", ""),
})
except Exception as e:
if log:
log("ERROR", f"{std_id} 跨字段校验失败: {e}",
step=indicator_step_id(std_id))
if indicator_rec["total_invalid"] > 0:
violating = [d for d in indicator_rec["fields"] if d["violations"] > 0]
bucket["violations_by_indicator"].append({
"indicator": ind301.standard_id,
"indicator_name": ind301.standard_name,
"fields_count": len(indicator_rec["fields"]),
"violating_fields_count": len(violating),
"total_invalid": indicator_rec["total_invalid"],
"total_sampled": indicator_rec["total_sampled"],
"violation_rate": (
round(indicator_rec["total_invalid"] / indicator_rec["total_sampled"], 4)
if indicator_rec["total_sampled"] else 0
),
"details": indicator_rec["fields"],
"details_with_violations": violating,
"sample_specs": [{
"id": "round2_cross_field",
"name": "抽样 (id_type, id_no) 同行配对",
"description": "IND-301 跨字段校验",
"sql_template": (
"SELECT `${type_col}` AS id_type, `${no_col}` AS id_no "
"FROM `${table}` WHERE ... LIMIT ${limit}"
),
}],
"sample_limit": SAMPLE_LIMIT,
"max_tables_per_field": MAX_TABLES_PER_FIELD,
})
bucket["summary"]["indicators_used"] = len(bucket["violations_by_indicator"])
bucket["summary"]["violations"] = len(bucket["violations"])
bucket["violations"] = bucket["violations"][:500]
bucket["indicators_info"] = [indicator_rec]
return bucket
def _run_round2_uniqueness_one(
cfg: DBConfig,
by_name: dict[str, list[dict]],
log: Callable | None,
) -> dict:
"""跨表聚合 IND-302：证件唯一性 GROUP BY HAVING —— 单 indicator 输出。"""
from standards.ind_302_id_uniqueness import IdUniquenessIndicator
loader = get_sql_loader()
bucket = _empty_indicator_data()
ind302 = IdUniquenessIndicator()
std_id = ind302.standard_id
no_cols = ("id_no", "cert_no", "zjhm", "document_no")
tables_with_no: list[tuple[str, str]] = []
for table, cols in by_name.items():
for c in cols:
if c["column_name"] in no_cols:
tables_with_no.append((table, c["column_name"]))
if not tables_with_no:
if log:
log("INFO", f" · {std_id} 未找到证件号字段，跳过",
step=indicator_step_id(std_id))
return bucket
if log:
log("INFO",
f" · {std_id} (跨表聚合): {len(tables_with_no)} 张表含证件号字段",
step=indicator_step_id(std_id))
indicator_rec = {
"indicator": ind302.standard_id,
"indicator_name": ind302.standard_name,
"groups": [],
"total_invalid": 0,
"total_sampled": 0,
}
try:
with open_db(cfg) as db:
for table, nc in tables_with_no:
try:
sql = loader.render(
"standards/sample_field_groups",
dialect=cfg.db_type,
table=table,
type_col="id_type",
no_col=nc,
min_count=2,
)
rows = db.fetchall(sql)
except Exception as e:
if log:
log("WARN",
f" - {table}.{nc} GROUP BY 抽样失败: {e}",
step=indicator_step_id(std_id))
continue
bucket["summary"]["fields_checked"] += 1
for r in rows:
indicator_rec["total_invalid"] += 1
indicator_rec["groups"].append({
"table": table,
"field": nc,
"id_type": r.get("id_type", ""),
"id_no": r.get("id_no", "")[:50],
"dup_count": r.get("dup_count", 0),
"sample_sql": sql,
})
bucket["violations"].append({
"table_name": table,
"table_comment": "",
"column_name": nc,
"column_comment": "",
"rule_type": ind302.standard_id,
"standard_name": ind302.standard_name,
"column_type": "",
"value": str(r.get("id_no", ""))[:50],
"error": (
f"疑似重复开户：证件类型 {r.get('id_type', '')} "
f"在本表出现 {r.get('dup_count', 0)} 次"
),
})
except Exception as e:
if log:
log("ERROR", f"{std_id} 跨表聚合失败: {e}",
step=indicator_step_id(std_id))
if indicator_rec["total_invalid"] > 0:
bucket["violations_by_indicator"].append({
"indicator": ind302.standard_id,
"indicator_name": ind302.standard_name,
"fields_count": len(tables_with_no),
"violating_fields_count": len(indicator_rec["groups"]),
"total_invalid": indicator_rec["total_invalid"],
"total_sampled": sum(g["dup_count"] for g in indicator_rec["groups"]),
"violation_rate": 1.0,
"details": [{
"table": g["table"],
"field": g["field"],
"sampled": g["dup_count"],
"violations": g["dup_count"],
"violation_rate": 1.0,
"samples": [{
"value": g["id_no"],
"reason": g["dup_count"],
"id_type": g["id_type"],
}],
"sample_spec_id": "round2_uniqueness",
"sample_sql": g["sample_sql"],
} for g in indicator_rec["groups"]],
"details_with_violations": [
{"table": g["table"], "field": g["field"]}
for g in indicator_rec["groups"]
],
"sample_specs": [{
"id": "round2_uniqueness",
"name": "GROUP BY HAVING 重复值",
"description": "IND-302 跨表聚合",
"sql_template": (
"SELECT `${type_col}`, `${no_col}`, COUNT(*) AS dup_count "
"FROM `${table}` GROUP BY ... HAVING COUNT(*) >= ${min_count}"
),
}],
"sample_limit": 200,
"max_tables_per_field": MAX_TABLES_PER_FIELD,
})
bucket["summary"]["indicators_used"] = len(bucket["violations_by_indicator"])
bucket["summary"]["violations"] = len(bucket["violations"])
bucket["violations"] = bucket["violations"][:500]
bucket["indicators_info"] = [indicator_rec]
return bucket
# ── 旧版全量执行入口（向后兼容）──
def _run_round2_cross_field(
cfg: DBConfig,
by_name: dict[str, list[dict]],
log: Callable | None,
) -> dict:
"""向后兼容：返回 IND-301 单 indicator 数据（兼容旧 _wrap 用 group bucket 形态）。"""
return _run_round2_cross_field_one(cfg, by_name, log)
def _run_round2_uniqueness(
cfg: DBConfig,
by_name: dict[str, list[dict]],
log: Callable | None,
) -> dict:
"""向后兼容：返回 IND-302 单 indicator 数据。"""
return _run_round2_uniqueness_one(cfg, by_name, log)
def run_step7(cfg: DBConfig, dict_data: dict, log: Callable | None = None,
table_filter: set[str] | None = None) -> dict:
"""执行国标校验
"""旧版全量入口（向后兼容）：跑所有 indicator，按 group 输出 4 tab。
Args:
table_filter: 前端勾选的表名集合（None/空 = 全部）。
新代码应直接调 run_step7_for_indicator(standard_id, ...) 跑单个 indicator。
"""
columns = dict_data.get("data_dictionary", [])
if table_filter:
......@@ -207,20 +989,13 @@ def run_step7(cfg: DBConfig, dict_data: dict, log: Callable | None = None,
if not columns:
return _EMPTY_FALLBACK()
# 字段名 → 标准实例的映射
standards_by_field = discover_standards()
all_standards_info = list_all_standards()
# 按字段名索引列
by_name: dict[str, list[dict]] = {}
for r in columns:
by_name.setdefault(r["column_name"], []).append(r)
by_standard: dict[str, dict] = {}
total_violations = 0
total_fields_checked = 0
violations_flat: list[dict] = []
if not standards_by_field:
if log:
log("WARN", "未发现任何标准插件", step="standards")
......@@ -233,154 +1008,119 @@ def run_step7(cfg: DBConfig, dict_data: dict, log: Callable | None = None,
f"最大每字段抽样 {SAMPLE_LIMIT} 行 / 每字段最多 {MAX_TABLES_PER_FIELD} 张表",
step="standards")
loader = get_sql_loader()
# ── Round 1 单值校验 ──
per_group = _run_round1(cfg, by_name, standards_by_field, log)
try:
with open_db(cfg) as db:
for std_idx, (std_field, std_instance) in enumerate(standards_by_field.items(), 1):
if std_field not in by_name:
if log:
log("DEBUG",
f" · [{std_idx}/{len(standards_by_field)}] 标准字段 {std_field} 未出现在任何表中, 跳过",
step="standards")
continue
tables_with_field = by_name[std_field][:MAX_TABLES_PER_FIELD]
std_id = std_instance.standard_id
std_record = by_standard.setdefault(std_id, {
"standard": std_id,
"standard_name": std_instance.standard_name,
"fields": [],
"total_invalid": 0,
"total_sampled": 0,
})
# ── Round 2 跨字段（IND-301）──
cross_field = _run_round2_cross_field(cfg, by_name, log)
if cross_field["summary"]["violations"] > 0:
group_name = "证件信息"
if group_name not in per_group:
per_group[group_name] = _empty_group_data()
for k in ("violations_by_indicator", "violations"):
per_group[group_name][k].extend(cross_field[k])
per_group[group_name]["summary"]["violations"] = len(
per_group[group_name]["violations"]
)
per_group[group_name]["summary"]["indicators_used"] = sum(
1 for x in per_group[group_name]["violations_by_indicator"]
if x.get("violations", 0) > 0 or x.get("total_invalid", 0) > 0
)
per_group[group_name]["violations"] = per_group[group_name]["violations"][:500]
if log:
log("INFO",
f" · [{std_idx}/{len(standards_by_field)}] "
f"{std_id} ({std_field}) → {len(tables_with_field)} 张表",
step="standards")
for col_record in tables_with_field:
table = col_record["table_name"]
field_name = col_record["column_name"]
total_fields_checked += 1
try:
sql = loader.render(
"standards/sample_field_values",
dialect=cfg.db_type,
table=table, # 模板内 ${table | quote} 自动加引号
field=field_name,
limit=SAMPLE_LIMIT,
)
rows = db.fetchall(sql)
# 真正执行过的 SQL，落库展示给审计
sample_sql_rendered = sql
sample_spec_id = "default"
except Exception as e:
std_record["fields"].append({
"table": table,
"field": field_name,
"error": str(e),
"violations": 0,
"sampled": 0,
"sample_spec_id": "default",
"sample_sql": f"SELECT `${field_name}` ... -- 渲染失败: {e}",
})
if log:
log("WARN",
f" - {table}.{field_name} 抽样失败: {e}",
step="standards")
continue
# ── Round 2 跨表聚合（IND-302）──
uniqueness = _run_round2_uniqueness(cfg, by_name, log)
if uniqueness["summary"]["violations"] > 0:
group_name = "证件信息"
if group_name not in per_group:
per_group[group_name] = _empty_group_data()
for k in ("violations_by_indicator", "violations"):
per_group[group_name][k].extend(uniqueness[k])
per_group[group_name]["summary"]["violations"] = len(
per_group[group_name]["violations"]
)
per_group[group_name]["summary"]["indicators_used"] = sum(
1 for x in per_group[group_name]["violations_by_indicator"]
if x.get("violations", 0) > 0 or x.get("total_invalid", 0) > 0
)
per_group[group_name]["violations"] = per_group[group_name]["violations"][:500]
sampled = len(rows)
invalid_samples = []
invalid_count = 0
for row in rows:
val = row.get("val")
if val is None:
continue
result: ValidationResult = std_instance.validate(str(val))
if not result.valid:
invalid_count += 1
if len(invalid_samples) < 10:
invalid_samples.append({
"value": str(val)[:50],
"reason": result.reason,
})
violations_flat.append({
"table_name": table,
"table_comment": col_record.get("table_comment", ""),
"column_name": field_name,
"column_comment": col_record.get("column_comment", ""),
"rule_type": std_id,
"standard_name": std_instance.standard_name,
"column_type": col_record.get("column_type", ""),
"value": str(val)[:50],
"error": result.reason,
})
total_violations += invalid_count
std_record["total_invalid"] += invalid_count
std_record["total_sampled"] += sampled
std_record["fields"].append({
"table": table,
"field": field_name,
"sampled": sampled,
"violations": invalid_count,
"violation_rate": round(invalid_count / sampled, 4) if sampled else 0,
"samples": invalid_samples,
"sample_spec_id": sample_spec_id,
"sample_sql": sample_sql_rendered,
})
logger.debug(
f"{table}.{field_name}: 抽样 {sampled}, 违规 {invalid_count}"
)
except Exception as e:
return _wrap(per_group)
# ── Per-indicator 入口（orchestrator 每个 step 调一次） ──────────────
def run_step7_for_indicator(
standard_id: str,
cfg: DBConfig,
dict_data: dict,
log: Callable | None = None,
table_filter: set[str] | None = None,
) -> dict:
"""跑单个 indicator（orchestrator 每个 step 调一次）。
Args:
standard_id: 例如 "IND-001-a" / "IND-301" / "IND-302"
cfg / dict_data / log / table_filter: 同 run_step7
Returns:
{section_key, data} —— data 已注入 _protocol（1 tab）
"""
columns = dict_data.get("data_dictionary", [])
if table_filter:
columns = [c for c in columns if c["table_name"] in table_filter]
cls = load_standard_class(standard_id)
if cls is None:
if log:
log("ERROR", f"连库校验失败: {type(e).__name__}: {e}", step="standards")
logger.exception("详细异常")
log("WARN", f"未知 indicator: {standard_id}", step=indicator_step_id(standard_id))
return _empty_indicator_step_result(standard_id, reason=f"未找到 indicator {standard_id}")
std_instance = cls()
by_name: dict[str, list[dict]] = {}
for r in columns:
by_name.setdefault(r["column_name"], []).append(r)
if not columns:
return _empty_indicator_step_result(standard_id)
# 按 indicator 类型分发执行路径
step_id_str = indicator_step_id(standard_id) # "std_ind_001_a"
safe_id = step_id_str[len("std_"):] # "ind_001_a"
section_key = f"standards_{safe_id}" # "standards_ind_001_a"
by_standard_list = []
for std_id, rec in by_standard.items():
# 过滤掉零违规的标准
if rec["total_invalid"] == 0:
if standard_id == "IND-301":
data = _run_round2_cross_field_one(cfg, by_name, log)
elif standard_id == "IND-302":
data = _run_round2_uniqueness_one(cfg, by_name, log)
else:
# 单值类（含 22 个有 applies_to_fields 的 indicator）
if not std_instance.applies_to_fields:
if log:
log("INFO",
f" · 标准 {rec['standard']} 抽样 {rec['total_sampled']} 条全部合规，跳过",
step="standards")
continue
violating_fields = [d for d in rec["fields"] if d.get("violations", 0) > 0]
by_standard_list.append({
"standard": rec["standard"],
"standard_name": rec["standard_name"],
"fields_count": len(rec["fields"]),
"violating_fields_count": len(violating_fields),
"total_invalid": rec["total_invalid"],
"total_sampled": rec["total_sampled"],
"violation_rate": round(rec["total_invalid"] / rec["total_sampled"], 4) if rec["total_sampled"] else 0,
"details": rec["fields"],
# 字段清单只展示有违规的，过滤掉 0 违规的占位项
"details_with_violations": violating_fields,
# 取数口径：每条标准附带一份规格清单（一个标准可对应多条）
"sample_specs": [dict(s) for s in SAMPLE_SPECS],
"sample_limit": SAMPLE_LIMIT,
"max_tables_per_field": MAX_TABLES_PER_FIELD,
})
log("WARN",
f"{standard_id} 没有 applies_to_fields 也没匹配 IND-301/302 模板",
step=indicator_step_id(standard_id))
return _empty_indicator_step_result(standard_id)
data = _run_round1_one(cfg, by_name, std_instance, log)
if log:
log("INFO", f"校验完成: {len(by_standard_list)} 个标准有违规, {total_fields_checked} 个字段, {total_violations} 条违规", step="standards")
return {"section_key": section_key, "data": _wrap_single_indicator(std_instance, data)}
return _wrap({
"summary": {
"standards_applied": len(all_standards_info),
"standards_used": len(by_standard_list),
"fields_checked": total_fields_checked,
"violations": total_violations,
},
"violations_by_standard": by_standard_list,
"violations": violations_flat[:500], # 给前端表格用
"standards_info": all_standards_info,
})
\ No newline at end of file
def _empty_indicator_step_result(standard_id: str, reason: str = "") -> dict:
"""指标未命中字段 / 未知 ID 时，返回的兜底 section。"""
step_id_str = indicator_step_id(standard_id)
safe_id = step_id_str[len("std_"):]
section_key = f"standards_{safe_id}"
cls = load_standard_class(standard_id)
inst = cls() if cls else None
data = _empty_indicator_data()
if reason:
data["note"] = reason
if inst is not None:
data = _wrap_single_indicator(inst, data)
else:
data["_protocol"] = {
"step_id": indicator_step_id(standard_id),
"tabs": [],
}
return {"section_key": section_key, "data": data}
\ No newline at end of file
web/sql/standards/sample_field_groups.sql 0 → 100644
View file @ 5c37925e
-- ============================================================================
-- 跨表去重 —— GROUP BY (id_type, id_no) HAVING COUNT(*) > N（IND-302 用）
-- 调用方：web/core/step_impl/step7_standards.py
-- 参数：
-- ${table} 表名（自动加引号）
-- ${type_col} 证件类型字段名（自动加引号）
-- ${no_col} 证件号字段名（自动加引号）
-- ${min_count} 阈值（默认 2），出现次数 >= 该值才纳入疑似重复
-- ============================================================================
SELECT `${type_col}` AS id_type, `${no_col}` AS id_no, COUNT(*) AS dup_count
FROM `${table}`
WHERE `${type_col}` IS NOT NULL AND TRIM(`${type_col}`) <> ''
AND `${no_col}` IS NOT NULL AND TRIM(`${no_col}`) <> ''
GROUP BY `${type_col}`, `${no_col}`
HAVING COUNT(*) >= ${min_count}
ORDER BY dup_count DESC
LIMIT 200
\ No newline at end of file
web/sql/standards/sample_field_pairs.sql 0 → 100644
View file @ 5c37925e
-- ============================================================================
-- 抽样「证件类型 + 证件号」同行配对（IND-301 跨字段一致性用）
-- 调用方：web/core/step_impl/step7_standards.py
-- 参数：
-- ${table} 表名（自动加引号）
-- ${type_col} 证件类型字段名（自动加引号）
-- ${no_col} 证件号字段名（自动加引号）
-- ${limit} 抽样上限
-- ============================================================================
SELECT `${type_col}` AS id_type, `${no_col}` AS id_no
FROM `${table}`
WHERE `${type_col}` IS NOT NULL AND TRIM(`${type_col}`) <> ''
AND `${no_col}` IS NOT NULL AND TRIM(`${no_col}`) <> ''
LIMIT ${limit}
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment