refactor: prepare_label_studio.py 的内容改动(补上 a3e08a9 漏掉的部分)
a3e08a9 只提交了 git mv 的改名,内容修改因 git add 的路径报错没有进入暂存区。
本提交补上:
- 缺省输出 runs/<source>_label_studio/,不再写到 data/;该目录同时是 Label Studio
的 DOCUMENT_ROOT,打印的导入步骤里提醒复核期间不要删除或 --overwrite。
- 总是生成 tasks.json 与 labeling config,去掉 --label-studio 开关。
- 不再写 data.yaml:复核材料不是训练集,没有 data.yaml 就不会被误拿去训练,
原先"train 与 val 相同"的告警也随之移除(detect.py 里的同类告警保留)。
- detect.py、configs/detect/film.yaml、README 里对旧脚本的引用改为新流程。
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a3e08a9bbc
commit
891f930de2
@@ -589,15 +589,18 @@ runs/<name>/
|
||||
|
||||
## 检测模型训练(YOLO)
|
||||
|
||||
上一步的产出按类别分目录,方便人工核对;YOLO 的类别写在标签第一列,不靠目录名。
|
||||
所以先拍平整理,导入 Label Studio 复核,再训练:
|
||||
`generate_boxes.py` 产出的是弱监督框,要先在 Label Studio 里人工修正,再划分 train/val 训练:
|
||||
|
||||
```bash
|
||||
# 1) 拍平类别目录、统一 PNG、按类别重命名,硬链接到 data/;同时生成 Label Studio 任务
|
||||
uv run python scripts/prepare_yolo_dataset.py --source runs/boxes_shap --label-studio
|
||||
# 1) 生成 Label Studio 复核材料到 runs/boxes_shap_label_studio/
|
||||
uv run python scripts/prepare_label_studio.py --source runs/boxes_shap
|
||||
# 先看计划加 --dry-run
|
||||
|
||||
# 2) 训练(训练前需要先划分,见下文"不划分 train/val")
|
||||
# 2) 在 Label Studio 里修正标签(导入步骤见下文,脚本结束时也会打印)
|
||||
|
||||
# 3) 导出修好的标签,划分 train/val 到 data/boxes_shap/(导出脚本待写)
|
||||
|
||||
# 4) 训练
|
||||
uv run python detect.py --config configs/detect/film.yaml
|
||||
|
||||
# 只评估 / 续训
|
||||
@@ -606,27 +609,26 @@ uv run python detect.py --config configs/detect/film.yaml --eval-only \
|
||||
uv run python detect.py --config configs/detect/film.yaml --resume
|
||||
```
|
||||
|
||||
产出:
|
||||
### 准备 Label Studio 复核材料
|
||||
|
||||
```
|
||||
data/boxes_shap/
|
||||
runs/boxes_shap_label_studio/ # 同时也是 Label Studio 的 DOCUMENT_ROOT
|
||||
├── images/<类别>_<序号>.png # 一律 PNG:PNG 原图硬链接,BMP 等无损转换
|
||||
├── labels/<类别>_<序号>.txt
|
||||
├── classes.txt # 每行一个类别名,行号即 class_id
|
||||
├── name_map.csv # 新文件名 ↔ 源文件路径
|
||||
├── data.yaml # train 与 val 都指向 images/
|
||||
├── summary.json # 逐类数量、空标签处置、落盘方式
|
||||
├── tasks.json # 仅 --label-studio:每张图一个任务,框作为 predictions
|
||||
└── tasks.label_config.xml # 仅 --label-studio:各类别的 RectangleLabels
|
||||
├── tasks.json # 每张图一个任务,框作为 predictions
|
||||
└── tasks.label_config.xml # 各类别的 RectangleLabels
|
||||
```
|
||||
|
||||
**不划分 train/val。** 这份数据集先导入 Label Studio 复核,划分留到复核之后。`data.yaml`
|
||||
的 `train` 与 `val` 只能指向同一个 `images/`,ultralytics 能读,但验证指标算的是训练集本身,
|
||||
mAP 会虚高,早停与 `best.pt` 的选择也随之失真。脚本结束时会告警,`detect.py` 检测到
|
||||
train 与 val 相同时也会告警。正式训练前要先划分。
|
||||
**不写 `data.yaml`、不划分 train/val。** 这是给人复核的材料,不是训练集,没有 `data.yaml`
|
||||
也就不会被误拿去训练。复核期间 Label Studio 直接从这个目录读图,**不要删除或 `--overwrite` 它**。
|
||||
`detect.py` 若检测到 `data.yaml` 的 train 与 val 相同(`generate_boxes.py` 的产出就是这样),
|
||||
也会告警:验证指标算的是训练集本身,mAP 虚高,早停与 `best.pt` 的选择失真。
|
||||
|
||||
**图片统一成 PNG。** pearlescent / blue / uv 的原始数据是 BMP(共 10,814 张),
|
||||
`generate_boxes.py` 按原后缀链接,于是 YOLO 数据集里会混着两种格式。这一步按**文件头**
|
||||
`generate_boxes.py` 按原后缀链接,于是产出里混着两种格式。这一步按**文件头**
|
||||
(不是后缀)判断:真 PNG 照旧硬链接不占磁盘,其余无损转 PNG——不缩放像素,归一化的框坐标
|
||||
原样有效;CMYK 这类 PNG 存不下的模式转 RGB。转换无法硬链接,会占额外磁盘,`--dry-run`
|
||||
会先报出要转多少张。统一后缀后 `a.png` 与另一目录的 `a.bmp` 会撞名,脚本直接报错。
|
||||
@@ -634,16 +636,13 @@ train 与 val 相同时也会告警。正式训练前要先划分。
|
||||
**按类别重新编号。** 源文件名五花八门(`反面纵黑点_1_550mmX1561.71m_20260416160441_0.png`、
|
||||
`pearlescent_404.bmp`……),输出统一改成 `<类别>_<序号>.png`,如 `blackdot_0042.png`。
|
||||
类别以标签第一列为准,文件名永远不会与标签矛盾;序号在类内按源路径排序,同一份来源重跑,
|
||||
同一张图拿到的名字不变。原文件名里的正反面、位置、
|
||||
时间戳等信息都在 `name_map.csv` 里(带 BOM,Excel 直接打开中文不乱码)。想保留原名加 `--keep-names`。
|
||||
同一张图拿到的名字不变。原文件名里的正反面、位置、时间戳等信息都在 `name_map.csv` 里
|
||||
(带 BOM,Excel 直接打开中文不乱码)。想保留原名加 `--keep-names`。
|
||||
|
||||
**classes.txt** 是给 LabelImg / X-AnyLabeling 这类标注工具用的,人工复核、修改框时
|
||||
靠它把 class_id 映射回类别名;ultralytics 训练本身只读 `data.yaml`,两者内容严格一致。
|
||||
**classes.txt** 是 label-studio-converter 读类别用的(也兼容 LabelImg / X-AnyLabeling),
|
||||
行号就是 class_id,内容与来源 `data.yaml` 严格一致。
|
||||
|
||||
### 导入 Label Studio 人工复核
|
||||
|
||||
弱监督框需要人工把关。加 `--label-studio` 会在数据集根目录额外生成 `tasks.json` 与
|
||||
`tasks.label_config.xml`,数据集根目录本身就是 Label Studio 的 `DOCUMENT_ROOT`。
|
||||
### 导入 Label Studio
|
||||
|
||||
脚本结束时会打印带实际路径的导入步骤:设置 `LABEL_STUDIO_LOCAL_FILES_*` 环境变量启动、
|
||||
粘贴 labeling config、添加 Local files 存储(**只保存不要 Sync**,否则每张图多出一个空任务)、
|
||||
@@ -652,7 +651,7 @@ train 与 val 相同时也会告警。正式训练前要先划分。
|
||||
几个取舍:
|
||||
|
||||
- **为什么是扁平结构**:`label-studio-converter import yolo` 只认 `<input>/{images,labels,classes.txt}`,
|
||||
且 `os.listdir` 不递归,看不见子目录。数据集不划分、直接扁平,正好就是它要的结构。
|
||||
且 `os.listdir` 不递归,看不见子目录。
|
||||
- **硬链接而不是软链接**:Label Studio 读本地文件时会检查路径是否在 `DOCUMENT_ROOT` 之下,
|
||||
软链接解析后可能落到外面。
|
||||
- **框作为 predictions 而不是 annotations**:它们是模型生成的。审核员修改提交后才成为 annotation,
|
||||
@@ -686,7 +685,7 @@ Grad-CAM 没框出东西的有 339 张(占 3%)。它们**默认保留为负
|
||||
等于在教模型抑制真实缺陷。用 `--empty-policy` 按来源类别分别处置:
|
||||
|
||||
```bash
|
||||
uv run python scripts/prepare_yolo_dataset.py --source runs/boxes_gradcam \
|
||||
uv run python scripts/prepare_label_studio.py --source runs/boxes_gradcam \
|
||||
--empty-policy drop normal=keep black=full edge_clean=full
|
||||
```
|
||||
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# 在 Grad-CAM / SHAP 生成的弱监督框上训练 YOLO 检测模型。
|
||||
# 在人工修正过的弱监督框上训练 YOLO 检测模型。
|
||||
#
|
||||
# 前置步骤:
|
||||
# uv run python scripts/generate_boxes.py --config ... --method gradcam # 产出 runs/boxes_gradcam
|
||||
# uv run python scripts/prepare_yolo_dataset.py --source runs/boxes_gradcam
|
||||
# uv run python scripts/generate_boxes.py --method shap --out runs/boxes_shap # 生成弱监督框
|
||||
# uv run python scripts/prepare_label_studio.py --source runs/boxes_shap # 复核材料
|
||||
# 在 Label Studio 里修正标签,导出并划分 train/val 到 data/boxes_shap/
|
||||
#
|
||||
# 用法:
|
||||
# uv run python detect.py --config configs/detect/film.yaml
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
uv run python detect.py --config configs/detect/film.yaml --eval-only \
|
||||
--checkpoint runs/detect/film/weights/best.pt
|
||||
|
||||
数据是 scripts/prepare_yolo_dataset.py 的产物(train/val 已划分好)。
|
||||
数据是在 Label Studio 里修正过的标签,导出并划分 train/val 之后放在 data/ 下的数据集。
|
||||
框来自 Grad-CAM / SHAP 的弱监督标注,不是人工标注——指标该按"与热力区域的一致性"
|
||||
来读,而不是当成真值上的检测精度。
|
||||
"""
|
||||
@@ -64,13 +64,13 @@ def main() -> int:
|
||||
data_yaml = Path(cfg["data"]["yaml"])
|
||||
if not data_yaml.exists():
|
||||
logger.error(
|
||||
"找不到 %s;先跑 scripts/prepare_yolo_dataset.py 把 runs/ 下的产出链接到 data/", data_yaml
|
||||
"找不到 %s;先把 Label Studio 里修好的标签导出、划分 train/val 到 data/ 下", data_yaml
|
||||
)
|
||||
return 1
|
||||
meta = yaml.safe_load(data_yaml.read_text(encoding="utf-8"))
|
||||
logger.info("数据集: %s(%d 类)", data_yaml, meta["nc"])
|
||||
if meta.get("train") == meta.get("val"):
|
||||
# prepare_yolo_dataset.py / generate_boxes.py 的产出都不划分,两者指向同一目录
|
||||
# generate_boxes.py 的产出就不划分,两者指向同一目录
|
||||
logger.warning(
|
||||
"data.yaml 的 train 与 val 相同(%s):验证集就是训练集,mAP 会虚高,"
|
||||
"早停与 best.pt 的选择也因此失真。正式训练前请先划分数据集",
|
||||
|
||||
@@ -1,45 +1,39 @@
|
||||
#!/usr/bin/env python
|
||||
"""把 generate_boxes.py 的产出整理成扁平的 YOLO 数据集,放到 data/ 下。
|
||||
"""把 generate_boxes.py 的产出整理成 Label Studio 复核材料,输出放在 runs/ 下。
|
||||
|
||||
uv run python scripts/prepare_yolo_dataset.py --source runs/boxes_gradcam
|
||||
uv run python scripts/prepare_label_studio.py --source runs/boxes_shap
|
||||
|
||||
generate_boxes.py 的产出是**按类别**分目录的(便于人工核对):
|
||||
generate_boxes.py 生成的是弱监督框,需要人工在 Label Studio 里修正之后才能拿去训练 YOLO。
|
||||
它的产出按**类别**分目录(便于人工核对),而 label-studio-converter 的 YOLO 导入只认扁平的
|
||||
`<input>/{images,labels,classes.txt}`,且 os.listdir 不递归。这一步做三件事:
|
||||
|
||||
```
|
||||
runs/boxes_gradcam/
|
||||
├── images/<类别>/<图名>.png
|
||||
└── labels/<类别>/<图名>.txt
|
||||
```
|
||||
|
||||
而 YOLO 的类别信息在标签文件的第一列里,不靠目录名。这一步做三件事:
|
||||
|
||||
1. 把按类别分的目录**拍平**,不划分 train/val——这份数据集先导入 Label Studio 复核,
|
||||
划分留到复核之后;
|
||||
1. 把按类别分的目录**拍平**;
|
||||
2. 图片**统一成 PNG**:本来就是 PNG 的硬链接过去,不占额外磁盘;BMP 等其他格式
|
||||
(pearlescent / blue / uv 原始数据都是 BMP)无损转成 PNG。不缩放像素,
|
||||
归一化的框坐标原样有效。
|
||||
|
||||
归一化的框坐标原样有效;
|
||||
3. 文件**按类别重新编号**:`<类别>_<序号>.png`,序号在类内按源路径排序,重跑结果一致。
|
||||
原文件名里的正反面、位置、时间戳等信息记在 `name_map.csv` 里,不会丢。
|
||||
想保留原名加 `--keep-names`。
|
||||
|
||||
然后在输出目录上调用 label-studio-converter,生成可直接导入的任务:
|
||||
|
||||
```
|
||||
data/<name>/
|
||||
runs/<source>_label_studio/ # 同时也是 Label Studio 的 DOCUMENT_ROOT
|
||||
├── images/<类别>_<序号>.png
|
||||
├── labels/<类别>_<序号>.txt
|
||||
├── classes.txt # 每行一个类别名,行号即 class_id
|
||||
├── name_map.csv # 新名 ↔ 源文件对照
|
||||
├── data.yaml # train 与 val 都指向 images/,不能直接用于评估
|
||||
├── summary.json
|
||||
├── tasks.json # 仅 --label-studio:框作为 predictions
|
||||
└── tasks.label_config.xml # 仅 --label-studio
|
||||
├── tasks.json # 每张图一个任务,框作为 predictions
|
||||
└── tasks.label_config.xml
|
||||
```
|
||||
|
||||
这个结构正是 label-studio-converter 的 YOLO 导入要求的 `<input>/{images,labels,classes.txt}`,
|
||||
所以 `--label-studio` 直接在数据集根目录上转换,DOCUMENT_ROOT 就是数据集根目录。
|
||||
这里**不写 data.yaml、不划分 train/val**:这份材料是给人复核的,不是训练集。
|
||||
修好标签后,再从 Label Studio 导出、划分到 data/ 下给 YOLO 训练。
|
||||
复核期间 Label Studio 直接从这个目录读图,**不要删除或 --overwrite 它**。
|
||||
|
||||
没有框的图(标签为空)默认保留:YOLO 训练里它们是负样本,有助于压低误检。但**空标签的
|
||||
语义是"这张图什么都没有"**,不是"有缺陷但没定位出来"。来自缺陷类别的空标签如果照搬,
|
||||
没有框的图(标签为空)默认保留,导入后是一个没有预标注框的任务。但**空标签在 YOLO 里的语义是
|
||||
"这张图什么都没有"**,不是"有缺陷但没定位出来"。来自缺陷类别的空标签如果原样走到训练,
|
||||
等于在教模型抑制真实缺陷,所以可以用 `--empty-policy` 按来源类别分别处置:
|
||||
|
||||
--empty-policy drop normal=keep black=full
|
||||
@@ -67,7 +61,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
import yaml # noqa: E402
|
||||
|
||||
from datasets.yolo import IMAGE_SUFFIXES, write_classes_txt, write_data_yaml # noqa: E402
|
||||
from datasets.yolo import IMAGE_SUFFIXES, write_classes_txt # noqa: E402
|
||||
from utils import setup_logging # noqa: E402
|
||||
|
||||
logger = None
|
||||
@@ -210,7 +204,7 @@ def assign_names(samples: Sequence[Sample], names: Sequence[str]) -> dict[Path,
|
||||
|
||||
|
||||
def build_label_studio_tasks(out_dir: Path, num_images: int) -> dict:
|
||||
"""在数据集根目录上调用 label-studio-converter,生成 tasks.json 与 labeling config。
|
||||
"""在输出目录上调用 label-studio-converter,生成 tasks.json 与 labeling config。
|
||||
|
||||
converter 的 YOLO 导入只认 `<input>/{images,labels,classes.txt}`,且 os.listdir 不递归——
|
||||
这正是本脚本产出的扁平结构。图片是硬链接而不是软链接也有讲究:Label Studio 本地存储
|
||||
@@ -222,7 +216,7 @@ def build_label_studio_tasks(out_dir: Path, num_images: int) -> dict:
|
||||
try:
|
||||
from label_studio_sdk.converter.imports.yolo import convert_yolo_to_ls
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("--label-studio 需要 label-studio(uv sync 后可用)") from exc
|
||||
raise RuntimeError("需要 label-studio(uv sync 后可用)") from exc
|
||||
|
||||
tasks_file = out_dir / "tasks.json"
|
||||
# converter 会往 stdout 打一份缺了环境变量的导入说明,还会刷一串 root logger 的 INFO;
|
||||
@@ -323,15 +317,16 @@ def main() -> int:
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
parser.add_argument("--source", required=True, help="generate_boxes.py 的产出目录")
|
||||
parser.add_argument("--out", default=None, help="输出目录;缺省 data/<source 目录名>")
|
||||
parser.add_argument("--data-root", default="data", help="自动推导时的根目录,默认 data")
|
||||
parser.add_argument(
|
||||
"--out", default=None, help="输出目录;缺省 runs/<source 目录名>_label_studio",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--empty-policy", nargs="*", default=[], metavar="POLICY|类别=POLICY",
|
||||
help=(
|
||||
"怎么处置没有框的图(标签为空)。可写一个全局默认,再按类别覆盖,如 "
|
||||
"`--empty-policy drop normal=keep black=full`。"
|
||||
"keep=当负样本保留(默认)、drop=丢弃、full=退化成整图框。"
|
||||
"注意空标签在 YOLO 里的意思是『此图无缺陷』,缺陷类的空标签照搬会教模型抑制真实缺陷"
|
||||
"注意空标签在 YOLO 里的意思是『此图无缺陷』,缺陷类的空标签走到训练会教模型抑制真实缺陷"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
@@ -339,15 +334,14 @@ def main() -> int:
|
||||
help="保留源文件名,不按 <类别>_<序号> 重新编号(源文件名需去掉后缀后全局唯一)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--label-studio", action="store_true",
|
||||
help="额外在数据集根目录生成 tasks.json 与 labeling config,可直接导入 Label Studio 复核",
|
||||
"--overwrite", action="store_true",
|
||||
help="输出目录已存在时先清空。Label Studio 复核期间从这里读图,复核中不要用",
|
||||
)
|
||||
parser.add_argument("--overwrite", action="store_true", help="输出目录已存在时先清空")
|
||||
parser.add_argument("--dry-run", action="store_true", help="只打印计划")
|
||||
args = parser.parse_args()
|
||||
|
||||
source = Path(args.source)
|
||||
out_dir = Path(args.out) if args.out else Path(args.data_root) / source.name
|
||||
out_dir = Path(args.out) if args.out else Path("runs") / f"{source.name}_label_studio"
|
||||
logger = setup_logging()
|
||||
|
||||
try:
|
||||
@@ -383,7 +377,7 @@ def main() -> int:
|
||||
)
|
||||
if as_negative:
|
||||
logger.warning(
|
||||
"以上 %s 的空标签会以『此图无缺陷』的身份进入训练。"
|
||||
"以上 %s 的空标签会以『此图无缺陷』的身份导入,复核时不补框的话会这样进入训练。"
|
||||
"如果其中某些类其实是有缺陷、只是没能定位出来,它们会教模型抑制真实缺陷——"
|
||||
"用 --empty-policy 逐类改成 drop 或 full。",
|
||||
"/".join(as_negative),
|
||||
@@ -408,8 +402,7 @@ def main() -> int:
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
if args.label_studio:
|
||||
logger.info("--label-studio:会在 %s 下生成 tasks.json", out_dir)
|
||||
logger.info("输出: %s", out_dir)
|
||||
logger.info("--dry-run:未写入任何文件")
|
||||
return 0
|
||||
|
||||
@@ -452,12 +445,7 @@ def main() -> int:
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_text(f"{sample.full_box} 0.5 0.5 1.0 1.0\n", encoding="utf-8")
|
||||
|
||||
# 不划分时 train 与 val 只能指向同一目录:ultralytics 能读,但验证指标算的是训练集本身
|
||||
write_data_yaml(out_dir / "data.yaml", names, root=out_dir.resolve(), train="images", val="images")
|
||||
write_classes_txt(out_dir / "classes.txt", names)
|
||||
|
||||
label_studio = None
|
||||
if args.label_studio:
|
||||
try:
|
||||
label_studio = build_label_studio_tasks(out_dir, len(samples))
|
||||
except RuntimeError as exc:
|
||||
@@ -480,7 +468,6 @@ def main() -> int:
|
||||
"empty_policy": {"default": default_policy, **per_class_policy},
|
||||
"renamed": renamed is not None,
|
||||
"label_studio": label_studio,
|
||||
"split": False,
|
||||
"per_class": dict(counts),
|
||||
"link_modes": dict(modes),
|
||||
},
|
||||
@@ -491,11 +478,6 @@ def main() -> int:
|
||||
)
|
||||
logger.info("图片落盘方式: %s(hard=硬链接, copy=跨盘复制, convert=转成 PNG)", dict(modes))
|
||||
logger.info("完成 -> %s", out_dir)
|
||||
logger.warning(
|
||||
"数据集未划分 train/val,data.yaml 的 train 与 val 都指向 images/。"
|
||||
"直接拿去训练的话验证指标算的是训练集本身,会虚高;训练前需要另行划分。"
|
||||
)
|
||||
if label_studio:
|
||||
# label-studio-converter 自带的提示漏了环境变量(它在循环里改写了 image_root_url,
|
||||
# 与默认值比较时就不相等了),这里给出带实际路径的完整步骤
|
||||
logger.info(
|
||||
@@ -506,9 +488,10 @@ def main() -> int:
|
||||
" 2) 新建项目 -> Labeling Setup -> Code,粘贴 %s\n"
|
||||
" 3) Settings -> Cloud Storage -> Add Source Storage -> Local files,\n"
|
||||
" 绝对路径填 %s/images。只保存,不要点 Sync(会给每张图再建一个空任务)\n"
|
||||
" 4) 导入 %s",
|
||||
" 4) 导入 %s\n"
|
||||
"复核期间 Label Studio 从 %s 读图,不要删除或 --overwrite 这个目录",
|
||||
label_studio["tasks"], label_studio["document_root"], label_studio["label_config"],
|
||||
label_studio["document_root"], label_studio["tasks_file"],
|
||||
label_studio["document_root"], label_studio["tasks_file"], label_studio["document_root"],
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""把 generate_boxes.py 的产出整理成 YOLO 训练集。"""
|
||||
"""把 generate_boxes.py 的产出整理成 Label Studio 复核材料。"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
@@ -14,7 +14,7 @@ import yaml # noqa: E402
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
sys.path.insert(0, "scripts")
|
||||
import prepare_yolo_dataset as pyd # noqa: E402
|
||||
import prepare_label_studio as pls # noqa: E402
|
||||
|
||||
CLASSES = ["grease", "normal", "whitedot"]
|
||||
|
||||
@@ -55,17 +55,17 @@ def output_of(out, source_stem):
|
||||
|
||||
|
||||
def run(source, out, *extra):
|
||||
argv = ["prepare_yolo_dataset.py", "--source", str(source), "--out", str(out), *extra]
|
||||
argv = ["prepare_label_studio.py", "--source", str(source), "--out", str(out), *extra]
|
||||
original, sys.argv = sys.argv, argv
|
||||
try:
|
||||
return pyd.main()
|
||||
return pls.main()
|
||||
finally:
|
||||
sys.argv = original
|
||||
|
||||
|
||||
def test_flat_layout_without_split(source, tmp_path):
|
||||
"""类别目录被拍平、不划分 train/val:images/ 与 labels/ 下直接是文件,一一对应。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
def test_flat_layout(source, tmp_path):
|
||||
"""类别目录被拍平:images/ 与 labels/ 下直接是文件,一一对应(converter 不递归)。"""
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
|
||||
for kind in ("images", "labels"):
|
||||
@@ -74,12 +74,11 @@ def test_flat_layout_without_split(source, tmp_path):
|
||||
labels = sorted(p.stem for p in (out / "labels").iterdir())
|
||||
assert images == labels
|
||||
assert len(images) == 13
|
||||
assert json.loads((out / "summary.json").read_text(encoding="utf-8"))["split"] is False
|
||||
|
||||
|
||||
def test_hard_links_share_inode(source, tmp_path):
|
||||
"""用硬链接而不是复制,不占额外磁盘。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
original = source / "images" / "grease" / "grease_0.png"
|
||||
linked = output_of(out, "grease_0")
|
||||
@@ -87,21 +86,15 @@ def test_hard_links_share_inode(source, tmp_path):
|
||||
|
||||
|
||||
def test_per_class_counts_in_summary(source, tmp_path):
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
per_class = json.loads((out / "summary.json").read_text(encoding="utf-8"))["per_class"]
|
||||
assert per_class == {"grease": 6, "normal": 4, "(无框)": 1, "whitedot": 2}
|
||||
|
||||
|
||||
def test_warns_that_data_is_not_split(source, tmp_path, capsys):
|
||||
"""train 与 val 指向同一目录,直接训练验证指标会虚高,必须提醒。"""
|
||||
assert run(source, tmp_path / "data" / "boxes") == 0
|
||||
assert "未划分" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_empty_labels_kept_as_negatives_by_default(source, tmp_path):
|
||||
"""无框的图默认保留——YOLO 里它们是压低误检的负样本。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
summary = json.loads((out / "summary.json").read_text(encoding="utf-8"))
|
||||
assert summary["empty_labels"] == {"normal": {"policy": "keep", "count": 1}}
|
||||
@@ -109,7 +102,7 @@ def test_empty_labels_kept_as_negatives_by_default(source, tmp_path):
|
||||
|
||||
|
||||
def test_empty_policy_drop_removes_the_image(source, tmp_path):
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--empty-policy", "drop") == 0
|
||||
summary = json.loads((out / "summary.json").read_text(encoding="utf-8"))
|
||||
assert summary["num_images"] == 12
|
||||
@@ -119,7 +112,7 @@ def test_empty_policy_drop_removes_the_image(source, tmp_path):
|
||||
|
||||
def test_empty_policy_is_per_class(source, tmp_path):
|
||||
"""缺陷类的空标签丢掉,normal 的留作负样本——两种处置能共存。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--empty-policy", "drop", "normal=keep") == 0
|
||||
summary = json.loads((out / "summary.json").read_text(encoding="utf-8"))
|
||||
assert summary["empty_labels"]["normal"]["policy"] == "keep"
|
||||
@@ -128,7 +121,7 @@ def test_empty_policy_is_per_class(source, tmp_path):
|
||||
|
||||
def test_empty_policy_full_writes_a_whole_image_box(source, tmp_path):
|
||||
"""整图属性的类别退化成整图框:标签要重写,不能链接源文件(内容不同)。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--empty-policy", "normal=full") == 0
|
||||
|
||||
image = output_of(out, "normal_0")
|
||||
@@ -156,17 +149,25 @@ def test_empty_policy_rejects_typos(source, tmp_path, capsys, tokens, expected):
|
||||
assert expected in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_data_yaml_uses_absolute_path_and_source_names(source, tmp_path):
|
||||
"""data.yaml 要能被 ultralytics 从任意工作目录读到,类别顺序沿用来源。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
def test_no_data_yaml_so_it_cannot_be_trained_on_by_mistake(source, tmp_path):
|
||||
"""复核材料不是训练集:不写 data.yaml,免得没修过的弱监督框被直接拿去训练。"""
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
meta = yaml.safe_load((out / "data.yaml").read_text(encoding="utf-8"))
|
||||
assert Path(meta["path"]).is_absolute()
|
||||
assert meta["nc"] == 3
|
||||
assert [meta["names"][i] for i in sorted(meta["names"])] == CLASSES
|
||||
# 不划分:两者都指向扁平的 images/
|
||||
assert meta["train"] == meta["val"] == "images"
|
||||
assert (Path(meta["path"]) / meta["train"]).is_dir()
|
||||
assert not (out / "data.yaml").exists()
|
||||
assert not (out / "images" / "train").exists()
|
||||
|
||||
|
||||
def test_default_output_goes_under_runs(source, tmp_path, monkeypatch):
|
||||
"""缺省输出是 runs/<source>_label_studio,不再写到 data/。"""
|
||||
pytest.importorskip("label_studio_sdk.converter.imports.yolo")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
original, sys.argv = sys.argv, ["prepare_label_studio.py", "--source", str(source)]
|
||||
try:
|
||||
assert pls.main() == 0
|
||||
finally:
|
||||
sys.argv = original
|
||||
assert (tmp_path / "runs" / "boxes_label_studio" / "tasks.json").is_file()
|
||||
assert not (tmp_path / "data").exists()
|
||||
|
||||
|
||||
def test_name_collision_after_flattening_is_reported(source, tmp_path, capsys):
|
||||
@@ -175,13 +176,13 @@ def test_name_collision_after_flattening_is_reported(source, tmp_path, capsys):
|
||||
Image.fromarray(np.zeros((32, 32, 3), dtype=np.uint8)).save(duplicate)
|
||||
(source / "labels" / "normal" / "grease_0.txt").write_text("1 0.5 0.5 0.2 0.2\n", encoding="utf-8")
|
||||
|
||||
assert run(source, tmp_path / "data" / "boxes", "--keep-names") != 0
|
||||
assert run(source, tmp_path / "runs" / "boxes_label_studio", "--keep-names") != 0
|
||||
assert "冲突" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_refuses_to_clobber_existing_output(source, tmp_path):
|
||||
"""输出目录非空时要求显式 --overwrite。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
out.mkdir(parents=True)
|
||||
(out / "keep.txt").write_text("x", encoding="utf-8")
|
||||
assert run(source, out) != 0
|
||||
@@ -197,7 +198,7 @@ def test_missing_label_is_an_error(source, tmp_path):
|
||||
|
||||
|
||||
def test_dry_run_writes_nothing(source, tmp_path):
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--dry-run") == 0
|
||||
assert not out.exists()
|
||||
|
||||
@@ -225,13 +226,13 @@ def test_all_output_images_are_png(source, tmp_path):
|
||||
"""BMP / JPEG 来源都要落成真正的 PNG——后缀和文件头都得是。"""
|
||||
_add_sample(source, "grease", "bmp_img", ".bmp", 0, fmt="BMP")
|
||||
_add_sample(source, "normal", "jpg_img", ".jpg", 1, fmt="JPEG")
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
|
||||
images = _all_outputs(out, "images")
|
||||
assert len(images) == 15
|
||||
assert {p.suffix for p in images} == {".png"}
|
||||
assert all(pyd.is_png(p) for p in images)
|
||||
assert all(pls.is_png(p) for p in images)
|
||||
# 标签跟着改名后的图片走,一一对应
|
||||
assert sorted(p.stem for p in images) == sorted(p.stem for p in _all_outputs(out, "labels"))
|
||||
|
||||
@@ -239,7 +240,7 @@ def test_all_output_images_are_png(source, tmp_path):
|
||||
def test_bmp_conversion_is_lossless(source, tmp_path):
|
||||
"""转换不能缩放或有损压缩,否则归一化的框就不再对应原来的像素。"""
|
||||
_, original = _add_sample(source, "grease", "bmp_img", ".bmp", 0, fmt="BMP")
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
|
||||
converted = output_of(out, "bmp_img")
|
||||
@@ -251,7 +252,7 @@ def test_bmp_conversion_is_lossless(source, tmp_path):
|
||||
def test_png_sources_are_still_hard_linked(source, tmp_path):
|
||||
"""只有非 PNG 才转换;PNG 照旧硬链接,不占额外磁盘。"""
|
||||
bmp, _ = _add_sample(source, "grease", "bmp_img", ".bmp", 0, fmt="BMP")
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
|
||||
summary = json.loads((out / "summary.json").read_text(encoding="utf-8"))
|
||||
@@ -264,18 +265,18 @@ def test_png_sources_are_still_hard_linked(source, tmp_path):
|
||||
def test_png_suffix_with_non_png_content_is_converted(source, tmp_path):
|
||||
"""按文件头判断而不是后缀:写着 .png 其实是 JPEG 的也要转。"""
|
||||
fake, _ = _add_sample(source, "grease", "fake", ".png", 0, fmt="JPEG")
|
||||
assert not pyd.is_png(fake)
|
||||
out = tmp_path / "data" / "boxes"
|
||||
assert not pls.is_png(fake)
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
output = output_of(out, "fake")
|
||||
assert pyd.is_png(output)
|
||||
assert pls.is_png(output)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["L", "RGBA", "CMYK"])
|
||||
def test_conversion_handles_pixel_modes(source, tmp_path, mode):
|
||||
"""灰度 / 带透明度原样保留;PNG 存不下的 CMYK 转成 RGB。"""
|
||||
_add_sample(source, "grease", "moded", ".tif", 0, fmt="TIFF", mode=mode)
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
output = output_of(out, "moded")
|
||||
with Image.open(output) as image:
|
||||
@@ -286,7 +287,7 @@ def test_conversion_handles_pixel_modes(source, tmp_path, mode):
|
||||
def test_same_stem_different_suffix_collides(source, tmp_path, capsys):
|
||||
"""grease_0.png 与另一目录的 grease_0.bmp 统一后缀后会撞成同一个文件,必须报错。"""
|
||||
_add_sample(source, "normal", "grease_0", ".bmp", 1, fmt="BMP")
|
||||
assert run(source, tmp_path / "data" / "boxes", "--keep-names") != 0
|
||||
assert run(source, tmp_path / "runs" / "boxes_label_studio", "--keep-names") != 0
|
||||
assert "冲突" in capsys.readouterr().out
|
||||
|
||||
|
||||
@@ -296,13 +297,13 @@ def test_dry_run_reports_conversion_count(source, tmp_path, capsys):
|
||||
assert "1 张不是 PNG" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_classes_txt_matches_data_yaml(source, tmp_path):
|
||||
"""classes.txt 的行号就是 class_id,必须与 data.yaml 的 names 严格一致。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
def test_classes_txt_matches_source_data_yaml(source, tmp_path):
|
||||
"""classes.txt 的行号就是 class_id,必须与来源 data.yaml 的 names 严格一致。"""
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
|
||||
lines = (out / "classes.txt").read_text(encoding="utf-8").splitlines()
|
||||
meta = yaml.safe_load((out / "data.yaml").read_text(encoding="utf-8"))
|
||||
meta = yaml.safe_load((source / "data.yaml").read_text(encoding="utf-8"))
|
||||
assert lines == CLASSES
|
||||
assert lines == [meta["names"][i] for i in range(meta["nc"])]
|
||||
assert (out / "classes.txt").read_text(encoding="utf-8").endswith("\n")
|
||||
@@ -323,7 +324,7 @@ def _stems(out):
|
||||
|
||||
def test_files_renamed_by_class(source, tmp_path):
|
||||
"""默认按 <类别>_<序号> 重新编号,序号在类内从 1 连续编到该类总数。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
assert _stems(out) == (
|
||||
[f"grease_{i:04d}" for i in range(1, 7)]
|
||||
@@ -335,7 +336,7 @@ def test_files_renamed_by_class(source, tmp_path):
|
||||
|
||||
def test_renamed_file_class_matches_its_label(source, tmp_path):
|
||||
"""文件名里的类别必须与标签第一列一致,否则改名反倒制造了误导。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
for label in _all_outputs(out, "labels"):
|
||||
text = label.read_text(encoding="utf-8").split()
|
||||
@@ -345,7 +346,7 @@ def test_renamed_file_class_matches_its_label(source, tmp_path):
|
||||
|
||||
def test_name_map_traces_back_to_source(source, tmp_path):
|
||||
"""name_map.csv 覆盖每一张输出,且源路径真实存在、内容一致。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
with (out / "name_map.csv").open(encoding="utf-8-sig", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
@@ -368,7 +369,7 @@ def test_names_stable_across_reruns(source, tmp_path):
|
||||
def test_keep_names_preserves_source_stems(source, tmp_path):
|
||||
"""--keep-names 保留原名,只是后缀统一成 .png。"""
|
||||
_add_sample(source, "grease", "原名_中文", ".bmp", 0, fmt="BMP")
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--keep-names") == 0
|
||||
assert "原名_中文" in _stems(out)
|
||||
assert output_of(out, "原名_中文").name == "原名_中文.png"
|
||||
@@ -376,7 +377,7 @@ def test_keep_names_preserves_source_stems(source, tmp_path):
|
||||
|
||||
def test_kept_empty_labels_named_after_source_folder(source, tmp_path):
|
||||
"""空标签没有 class_id,按来源目录名编号;与同类有框的图共用序号,不会撞名。"""
|
||||
out = tmp_path / "data" / "boxes"
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out) == 0
|
||||
assert output_of(out, "normal_0").stem.startswith("normal_")
|
||||
assert len([s for s in _stems(out) if s.startswith("normal_")]) == 5
|
||||
@@ -385,8 +386,8 @@ def test_kept_empty_labels_named_after_source_folder(source, tmp_path):
|
||||
@pytest.fixture
|
||||
def label_studio_out(source, tmp_path):
|
||||
pytest.importorskip("label_studio_sdk.converter.imports.yolo")
|
||||
out = tmp_path / "data" / "boxes"
|
||||
assert run(source, out, "--empty-policy", "normal=keep", "--label-studio") == 0
|
||||
out = tmp_path / "runs" / "boxes_label_studio"
|
||||
assert run(source, out, "--empty-policy", "normal=keep") == 0
|
||||
return out
|
||||
|
||||
|
||||
@@ -394,8 +395,8 @@ def _tasks(out):
|
||||
return json.loads((out / "tasks.json").read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def test_label_studio_files_at_dataset_root(label_studio_out):
|
||||
"""tasks.json 与 labeling config 直接放在数据集根目录,不再有 label_studio/ 子目录。"""
|
||||
def test_label_studio_files_at_output_root(label_studio_out):
|
||||
"""tasks.json 与 labeling config 总是生成,直接放在输出根目录(即 DOCUMENT_ROOT)。"""
|
||||
assert (label_studio_out / "tasks.json").is_file()
|
||||
assert (label_studio_out / "tasks.label_config.xml").is_file()
|
||||
assert not (label_studio_out / "label_studio").exists()
|
||||
@@ -442,8 +443,11 @@ def test_label_studio_config_and_summary(label_studio_out):
|
||||
assert Path(summary["label_studio"]["document_root"]) == label_studio_out.resolve()
|
||||
|
||||
|
||||
def test_label_studio_not_built_by_default(source, tmp_path):
|
||||
out = tmp_path / "data" / "boxes"
|
||||
assert run(source, out) == 0
|
||||
assert not (out / "tasks.json").exists()
|
||||
assert json.loads((out / "summary.json").read_text(encoding="utf-8"))["label_studio"] is None
|
||||
def test_prints_import_steps_with_real_paths(label_studio_out, capsys):
|
||||
"""converter 自带提示漏了环境变量,脚本要给出带实际路径的完整步骤。"""
|
||||
capsys.readouterr()
|
||||
source = label_studio_out.parent / "boxes"
|
||||
assert run(source, label_studio_out, "--overwrite") == 0
|
||||
out = capsys.readouterr().out
|
||||
assert f"LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT={label_studio_out.resolve()}" in out
|
||||
assert "不要点 Sync" in out
|
||||
|
||||
Reference in New Issue
Block a user