diff --git a/palladium-monitor/.gitignore b/palladium-monitor/.gitignore new file mode 100644 index 0000000..2d160cf --- /dev/null +++ b/palladium-monitor/.gitignore @@ -0,0 +1,33 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +*.egg-info/ +.pytest_cache/ +.mypy_cache/ + +# Virtual env +venv/ +env/ +.venv/ + +# IDE +.vscode/ +.idea/ +*.swp +.DS_Store + +# 项目数据 (含真实快照, 不上传) +instance/*.db +instance/*.sqlite +instance/*.sqlite3 + +# 日志 +*.log +logs/ + +# 测试 fixture +test_fixtures/ +/tmp/ diff --git a/palladium-monitor/README.md b/palladium-monitor/README.md new file mode 100644 index 0000000..943865f --- /dev/null +++ b/palladium-monitor/README.md @@ -0,0 +1,287 @@ +# Palladium Z1 Monitor + +LD 逻辑板使用监控系统 — 解析 `test_server -short` 输出，可视化展示 Palladium Z1 仿真器的逻辑板状态、逻辑板利用率、作业信息和 T-Pod 可用性。 + +## 功能 + +- **逻辑板状态可视化** — Rack / Cluster / Board 三级结构，每块 LD 板的 8 个逻辑板 (D0-D7) 以色块展示 + - 🟢 可用 (`-`) + - 🔵 已加载/预留 (``) + - 🔴 已禁用 (`D`) + - ⬜ 不存在 (`X`) +- **实时指标** — 逻辑板总数、在线/离线、逻辑板利用率、活跃作业数 +- **作业信息** — 用户、PID、T-Pod、设计名、已运行时间 +- **T-Pod 可用性** — 按 Rack 展示 HDSB / T-Pod 的可用/锁定/预留/不可用状态 +- **历史趋势** — 利用率折线图 (Chart.js)，支持历史快照查看和清理 +- **自动刷新** — 可选 15s/30s/1m/5m 间隔自动刷新仪表板 +- **多数据来源** — 手动粘贴、API 推送、远程采集脚本 + +## 快速开始 + +```bash +cd palladium-monitor +pip install -r requirements.txt --break-system-packages +python3 app.py +``` + +浏览器访问 `http://localhost:5100` + +## 数据采集方式 + +### 方式一：手动粘贴 (Web 界面) + +1. 在 Palladium 主机执行 `test_server -short` +2. 复制完整输出 +3. 在 Web 界面点击「📥 粘贴数据」按钮 +4. 粘贴输出到文本框，点击「解析并存储」 + +### 方式二：API 推送 (collect.py + 一键安装) + +在 Palladium Z1 主机上部署采集 agent，**推荐**用一键脚本（处理 env 加载、crontab、依赖）： + +```bash +# 把 palladium-monitor/ 整个目录 scp 到 Z1 主机, 然后: +sudo ./install_z1_agent.sh \ + --url http://<监控服务器IP>:5100/api/collect \ + --host scmp03 \ + --env /edatools/vxe/VXE23.03.001/bin/env.csh \ + --interval 30 +``` + +脚本会：建 `/opt/palladium-monitor/`、装 `requests`、备份原 crontab、写入 `*/30 * * * *` 任务、跑一次 dry-run 验证。 + +**手动部署**（不跑安装器）： + +```bash +# 安装 requests +pip install requests + +# 手动推送一次 +python3 collect.py --url http://<监控服务器IP>:5100/api/collect --host scmp03 + +# 配合 crontab 定时推送 (每30分钟) +*/30 * * * * /opt/palladium-monitor/collect.py --url http://<监控服务器IP>:5100/api/collect --host scmp03 --env /edatools/vxe/VXE23.03.001/bin/env.csh --log /var/log/palladium-collect.log >> /var/log/palladium-collect.cron.log 2>&1 +``` + +#### collect.py 参数 + +| 参数 | 默认值 | 说明 | +|------|--------|------| +| `--url` | `http://127.0.0.1:5000/api/collect` | 监控服务器 API 地址 | +| `--cmd` | `test_server -short` | 采集命令 | +| `--host` | `hostname` | 本机标签，会写入 `source` 字段 (如 `collect.py@scmp03`) | +| `--env` | — | VXE env.csh/env.sh 路径，脚本自动 `source` 后再执行采集命令 | +| `--timeout` | `300` | 采集命令超时秒数 | +| `--retries` | `3` | 推送失败重试次数 | +| `--retry-delay` | `5` | 重试间隔秒数 | +| `--log` | — | 追加写日志文件路径 | +| `--dry-run` | — | 只采集不推送，打印输出 | + +#### 退出码 + +| Code | 含义 | +|------|------| +| 0 | 成功 | +| 1 | 采集命令无输出 / 失败 | +| 2 | 推送失败 (已重试) | +| 130 | 用户中断 (Ctrl-C) | + +cron 收到非 0 时会通过 `MAILTO` 触发告警。 + +### 方式三：直接 API 调用 + +```bash +curl -X POST http://<服务器>:5100/api/collect \ + -H "Content-Type: application/json" \ + -d '{"raw_output": ""}' +``` + +## API 端点 + +| 方法 | 路径 | 说明 | +|------|------|------| +| GET | `/` | 仪表板页面 | +| GET | `/history` | 历史快照列表 | +| GET | `/snapshot/` | 快照详情 | +| POST | `/paste` | 手动粘贴数据 (表单提交) | +| POST | `/api/collect` | 接收原始输出 (JSON) | +| GET | `/api/latest` | 最新快照 JSON | +| GET | `/api/history` | 历史趋势 JSON | +| GET | `/api/export` | 导出历史 (CSV/JSON, 支持时间范围 + 来源过滤) | +| POST | `/api/cleanup?days=N` | 清理 N 天前数据 | +| POST | `/api/snapshot//delete` | 删除指定快照 | + +### 导出历史数据 + +`/api/export` 端点支持 CSV (Excel 友好, 带 UTF-8 BOM) 和 JSON 两种格式，可按时间范围和来源过滤。 + +``` +GET /api/export?format=csv # 默认: CSV 全部 +GET /api/export?format=json # JSON 全部 +GET /api/export?days=7 # 最近 7 天 +GET /api/export?since=2026-07-01&until=2026-07-31 # 自定义范围 +GET /api/export?source=collect.py # 按来源模糊匹配 +GET /api/export?format=json&days=30&source=scmp03 # 组合 +``` + +参数说明: +- `format`: `csv` (默认) 或 `json` +- `days`: 限制最近 N 天 (0/缺省 = 全部) +- `since` / `until`: `YYYY-MM-DD` 或 ISO 时间戳 +- `source`: 模糊匹配 `source` 字段 (如 `collect.py`、`scmp03`) +- `limit`: 最多返回条数 (默认 10000, 上限 100000) + +CSV 文件名: `palladium_snapshots_YYYYMMDD_HHMMSS.csv`，包含 19 列（时间本地+UTC、仿真器/硬件/ConfigMgr、系统状态、来源、板/域计数、利用率、作业数等）。 + +Web 界面入口：历史记录页右上角「📤 导出」按钮。 + +## 解析器支持的输出格式 + +``` +Emulator: sc01_emu Hardware: Palladium Z1 Configmgr: V21.02.102.s002System Status: PARTIAL +Rack 0 has 2 clusters +Cluster 0 has 6 boards CCD: ONLINE + LD Status D0 D1 D2 D3 D4 D5 D6 D7 + 0 ONLINE - - - - - - - - + ... +``` + +逻辑板状态图例： +- `-` → 可用 (Domain is available) +- `X` → 不存在 (Domain does not exist) +- `D` → 已禁用 (Domain is disabled) +- `` → 已加载/预留 (Domain is downloaded or reserved) + +## 项目结构 + +``` +palladium-monitor/ +├── app.py # Flask 应用: 解析器 + 数据模型 + 路由 + API +├── collect.py # 远程数据采集推送脚本 (Z1 主机跑) +├── install_z1_agent.sh # Z1 主机一键部署脚本 (env 加载 + crontab + 依赖) +├── requirements.txt # Flask, Flask-SQLAlchemy +├── instance/ # SQLite 数据库 +├── templates/ +│ ├── base.html # 基础布局 (导航 + 粘贴弹窗) +│ ├── dashboard.html # 主仪表板 +│ ├── history.html # 历史记录列表 +│ └── snapshot.html # 快照详情页 +└── static/ + └── style.css # 暗色主题样式 +``` + +## 生产部署 + +### 监控服务器端 + +```bash +cd palladium-monitor +pip install -r requirements.txt --break-system-packages +gunicorn --workers 2 --bind 0.0.0.0:5100 app:app +``` + +### 邮件功能配置 (环境变量) + +监控服务启动时会自动读以下环境变量, 无需 Web UI。所有变量都是可选的 (不配就相当于关闭邮件功能)。 + +| 变量 | 必填 | 默认 | 说明 | +|------|------|------|------| +| `MAIL_ENABLED` | ❌ | `false` | 是否启用邮件功能, 设为 `true`/`1`/`yes` 开启 | +| `SMTP_HOST` | 启用时必填 | — | SMTP 服务器地址 (如 `smtp.gmail.com`、`smtp.exmail.qq.com`) | +| `SMTP_PORT` | ❌ | `587` | SMTP 端口 (587=STARTTLS, 465=SSL, 25=明文) | +| `SMTP_USER` | 启用时必填 | — | SMTP 登录用户名 (通常等于邮箱地址) | +| `SMTP_PASSWORD` | 启用时必填 | — | SMTP 密码 (Gmail 用 App Password, 16 位) | +| `SMTP_USE_TLS` | ❌ | `true` | 是否用 STARTTLS (端口 587 时设为 `true`) | +| `SMTP_USE_SSL` | ❌ | `false` | 是否用隐式 SSL (端口 465 时设为 `true`) | +| `MAIL_FROM` | 启用时必填 | — | 发件人邮箱 | +| `MAIL_FROM_NAME` | ❌ | `Palladium Z1 Monitor` | 发件人显示名 | +| `MAIL_TO` | 启用时必填 | — | 收件人列表, 英文逗号分隔 (如 `alice@x.com,bob@y.com`) | +| `MAIL_SEND_HOUR` | ❌ | `9` | 每天发送时间-小时 (0-23) | +| `MAIL_SEND_MINUTE` | ❌ | `0` | 每天发送时间-分钟 (0-59) | +| `MONITOR_BASE_URL` | ❌ | `http://127.0.0.1:5100` | 监控页 URL (用于邮件里"查看完整仪表板"链接) | +| `ALLOW_EMAIL_TRIGGER` | ❌ | — | 设为 `true` 允许通过 `POST /api/email/send` 手动触发 (默认禁止) | + +#### 启动方式 + +**方式 1: 直接 export** +```bash +export MAIL_ENABLED=true +export SMTP_HOST=smtp.gmail.com +export SMTP_PORT=587 +export SMTP_USER=monitor@gmail.com +export SMTP_PASSWORD=**** # Gmail 用 App Password +export MAIL_FROM=monitor@gmail.com +export MAIL_TO=team-lead@company.com,ops@company.com +gunicorn --workers 2 --bind 0.0.0.0:5100 app:app +``` + +**方式 2: systemd EnvironmentFile (推荐)** +```ini +# /etc/palladium-monitor/mail.env +MAIL_ENABLED=true +SMTP_HOST=smtp.exmail.qq.com +SMTP_PORT=587 +SMTP_USER=monitor@company.com +SMTP_PASSWORD=**** # 或从 secret manager 注入 +MAIL_FROM=monitor@company.com +MAIL_TO=team@company.com +MAIL_SEND_HOUR=8 +MAIL_SEND_MINUTE=30 +``` + +```ini +# /etc/systemd/system/palladium-monitor.service +[Service] +EnvironmentFile=/etc/palladium-monitor/mail.env +ExecStart=/usr/bin/gunicorn --workers 2 --bind 0.0.0.0:5100 app:app +``` + +`systemctl daemon-reload && systemctl restart palladium-monitor` + +#### 邮件 API 端点 + +| 方法 | 路径 | 说明 | +|------|------|------| +| GET | `/api/email/status` | 查看当前邮件配置 (不泄露密码) | +| GET | `/api/email/preview` | 渲染报告内容 (不发送, 浏览器内联展示) | +| POST | `/api/email/send` | 手动触发一次 (需 `ALLOW_EMAIL_TRIGGER=true`) | + +**注意**: scheduler 在进程内启动, **每个 gunicorn worker 都会尝试启动**; 但用 `_scheduler_started` 锁保证实际只有一个线程在跑, 其余的只是 `time.sleep(86400)` 等下一天, 不影响。 + +#### 邮件内容示例 + +每日报告包含: +- **主题**: `[Palladium Z1] 每日报告 YYYY-MM-DD — 利用率 X.X%` +- **正文 (HTML + 纯文本)**: 系统信息 / 当前指标卡片 / 近 24h 趋势统计 + 样本柱状图 / 离线板列表 / 活跃作业列表 +- **多部分**: 同时发 HTML 和纯文本, 邮件客户端自动选最合适的渲染 + +### Z1 主机端（采集 agent） + +把整个 `palladium-monitor/` 目录 `scp` 到 Z1 主机，然后一键安装： + +```bash +sudo ./install_z1_agent.sh \ + --url http://<监控服务器IP>:5100/api/collect \ + --host <本机标签, e.g. scmp03> \ + --env /edatools/vxe/VXE23.03.001/bin/env.csh \ + --interval 30 +``` + +卸载：`sudo ./install_z1_agent.sh --uninstall`（保留日志）。 + +### 故障排查 + +| 症状 | 检查 | +|------|------| +| 仪表板一直显示示例数据 | `curl http://:5100/api/latest` 看最新 snapshot 的 `source` 字段，应为 `collect.py@` 而不是 `sample` | +| Z1 端推送失败 (exit=2) | `tail -50 /var/log/palladium-collect.log`；若 `test_server` 命令找不到，加 `--env /edatools/vxe/VXE23.03.001/bin/env.csh` | +| cron 没跑 | `crontab -l \| grep palladium`；`grep CRON /var/log/syslog`（Linux） | +| 推送成功但数据没更新 | 监控服务器 SQLite 写入是同步的，先看 `/api/latest` 的 `timestamp`，再 `tail /var/log/palladium-monitor-stderr.log`（gunicorn 日志） | +| 想清理老数据 | `curl -X POST 'http://:5100/api/cleanup?days=30'` | +| 多台 Z1 一起推 | 给每台用不同的 `--host` 标签，仪表板即可区分来源 | + +## 技术栈 + +- **后端**: Flask 3.0 + Flask-SQLAlchemy + SQLite +- **前端**: Jinja2 模板 + 原生 CSS (暗色主题) + Chart.js (趋势图) +- **解析**: Python re 正则表达式解析 `test_server -short` 输出 diff --git a/palladium-monitor/app.py b/palladium-monitor/app.py new file mode 100644 index 0000000..e20dadd --- /dev/null +++ b/palladium-monitor/app.py @@ -0,0 +1,1189 @@ +#!/usr/bin/env python3 +""" +Palladium Z1 Monitor — LD逻辑板使用监控系统 + +解析 `test_server -short` 输出，展示 Palladium Z1 仿真器的 +逻辑板 (LD) 状态、逻辑板利用率、作业信息和 T-Pod 可用性。 + +支持两种数据来源: + 1. 手动粘贴 — 在 Web 界面粘贴 test_server -short 输出 + 2. API 推送 — 在 Palladium 主机运行 collect.py 远程推送 +""" + +import os +import re +import io +import csv +import json +import subprocess +import threading +import time +from datetime import datetime, timezone, timedelta +from flask import Flask, render_template, request, jsonify, redirect, url_for, Response +from flask_sqlalchemy import SQLAlchemy + +# ═══════════════════════════════════════════════════════════════════════════ +# Config +# ═══════════════════════════════════════════════════════════════════════════ +BASE_DIR = os.path.dirname(os.path.abspath(__file__)) +INSTANCE_DIR = os.path.join(BASE_DIR, "instance") +os.makedirs(INSTANCE_DIR, exist_ok=True) + +app = Flask(__name__) +app.config["SECRET_KEY"] = os.environ.get("PALLADIUM_SECRET", "palladium-z1-monitor-2026") +app.config["SQLALCHEMY_DATABASE_URI"] = f"sqlite:///{os.path.join(INSTANCE_DIR, 'palladium.db')}" +app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False + +db = SQLAlchemy(app) + +# ═══════════════════════════════════════════════════════════════════════════ +# Domain State Constants +# ═══════════════════════════════════════════════════════════════════════════ +DOMAIN_AVAILABLE = "available" +DOMAIN_DOWNLOADED = "downloaded" +DOMAIN_DISABLED = "disabled" +DOMAIN_NOT_EXIST = "not_exist" + +DOMAIN_COLORS = { + DOMAIN_AVAILABLE: "#28a745", + DOMAIN_DOWNLOADED: "#007bff", + DOMAIN_DISABLED: "#dc3545", + DOMAIN_NOT_EXIST: "#4a4a5a", +} + +DOMAIN_LABELS = { + DOMAIN_AVAILABLE: "可用", + DOMAIN_DOWNLOADED: "已加载", + DOMAIN_DISABLED: "已禁用", + DOMAIN_NOT_EXIST: "不存在", +} + +# ═══════════════════════════════════════════════════════════════════════════ +# Models +# ═══════════════════════════════════════════════════════════════════════════ +class ScanSnapshot(db.Model): + __tablename__ = "snapshots" + id = db.Column(db.Integer, primary_key=True) + timestamp = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc), index=True) + emulator = db.Column(db.String(100), default="") + hardware = db.Column(db.String(100), default="") + configmgr = db.Column(db.String(200), default="") + system_status = db.Column(db.String(20), default="") + source = db.Column(db.String(50), default="manual") + raw_output = db.Column(db.Text) + # Summary stats (denormalised for quick queries) + total_boards = db.Column(db.Integer, default=0) + online_boards = db.Column(db.Integer, default=0) + offline_boards = db.Column(db.Integer, default=0) + total_domains = db.Column(db.Integer, default=0) + available_domains = db.Column(db.Integer, default=0) + downloaded_domains = db.Column(db.Integer, default=0) + disabled_domains = db.Column(db.Integer, default=0) + not_exist_domains = db.Column(db.Integer, default=0) + active_jobs = db.Column(db.Integer, default=0) + used_boards = db.Column(db.Integer, default=0) + utilization = db.Column(db.Float, default=0.0) + + boards = db.relationship("BoardStatus", backref="snapshot", cascade="all, delete-orphan", lazy=True) + jobs = db.relationship("JobInfo", backref="snapshot", cascade="all, delete-orphan", lazy=True) + pods = db.relationship("PodInfo", backref="snapshot", cascade="all, delete-orphan", lazy=True) + + +class BoardStatus(db.Model): + __tablename__ = "board_statuses" + id = db.Column(db.Integer, primary_key=True) + snapshot_id = db.Column(db.Integer, db.ForeignKey("snapshots.id"), nullable=False, index=True) + rack = db.Column(db.Integer) + cluster = db.Column(db.Integer) + ld_index = db.Column(db.Integer) + status = db.Column(db.String(20)) + ccd = db.Column(db.String(20)) + d0 = db.Column(db.String(20), default="-") + d1 = db.Column(db.String(20), default="-") + d2 = db.Column(db.String(20), default="-") + d3 = db.Column(db.String(20), default="-") + d4 = db.Column(db.String(20), default="-") + d5 = db.Column(db.String(20), default="-") + d6 = db.Column(db.String(20), default="-") + d7 = db.Column(db.String(20), default="-") + + +class JobInfo(db.Model): + __tablename__ = "job_infos" + id = db.Column(db.Integer, primary_key=True) + snapshot_id = db.Column(db.Integer, db.ForeignKey("snapshots.id"), nullable=False, index=True) + job_index = db.Column(db.Integer) + owner = db.Column(db.String(100), default="") + pid = db.Column(db.String(200), default="") + t_pod = db.Column(db.String(200), default="") + design = db.Column(db.String(200), default="") + elap_time = db.Column(db.String(20), default="") + reserved_key = db.Column(db.String(100), default="") + + +class PodInfo(db.Model): + __tablename__ = "pod_infos" + id = db.Column(db.Integer, primary_key=True) + snapshot_id = db.Column(db.Integer, db.ForeignKey("snapshots.id"), nullable=False, index=True) + rack = db.Column(db.Integer) + all_hdsb = db.Column(db.Text, default="") # JSON: {"5": "T4", "30": "T5"} + all_t_pods = db.Column(db.Text, default="") # JSON: {"5": "T0, T1, ..."} + available_hdsb = db.Column(db.Text, default="") + available_t_pods = db.Column(db.Text, default="") + locked_hdsb = db.Column(db.Text, default="") + locked_t_pods = db.Column(db.Text, default="") + reserved_hdsb = db.Column(db.Text, default="") + reserved_t_pods = db.Column(db.Text, default="") + unavailable_hdsb = db.Column(db.Text, default="") + unavailable_t_pods = db.Column(db.Text, default="") + + +# ═══════════════════════════════════════════════════════════════════════════ +# Parser — test_server -short 输出解析 +# ═══════════════════════════════════════════════════════════════════════════ +def parse_test_server_output(raw_text: str) -> dict: + """ + 解析 `test_server -short` 命令输出，返回结构化数据。 + + 返回结构: + { + "system": {emulator, hardware, configmgr, system_status}, + "racks": [ + {"rack": 0, "clusters": [ + {"cluster": 0, "ccd": "ONLINE", "boards": [ + {"ld": 0, "status": "ONLINE", "domains": ["-","-","-","-","-","-","-","-"]} + ]} + ]} + ], + "jobs": [{"index", "owner", "pid", "t_pod", "design", "elap_time", "reserved_key"}], + "pods": [{"rack", "all_hdsb", "all_t_pods", "available_hdsb", ...}], + } + """ + result = { + "system": {}, + "racks": [], + "jobs": [], + "pods": [], + } + + # ─── 系统信息 ────────────────────────────────────────────── + # 注意: Configmgr 值可能与 "System Status" 拼接在一起没有空格 + emu_match = re.search(r"Emulator:\s*(\S+)", raw_text) + # Hardware 名称含空格 (如 "Palladium Z1"), 匹配到 Configmgr 为止 + hw_match = re.search(r"Hardware:\s*(.+?)\s+Configmgr:", raw_text) + # 非贪婪匹配直到 "System Status" 或行尾 + cfg_match = re.search(r"Configmgr:\s*(.+?)(?:System\s*Status|$)", raw_text) + status_match = re.search(r"System\s*Status:\s*(\w+)", raw_text) + + result["system"] = { + "emulator": emu_match.group(1).strip() if emu_match else "", + "hardware": hw_match.group(1).strip() if hw_match else "", + "configmgr": cfg_match.group(1).strip() if cfg_match else "", + "system_status": status_match.group(1).strip() if status_match else "", + } + + # ─── Rack / Cluster / Board 结构 ─────────────────────────── + lines = raw_text.split("\n") + current_rack = None + + # LD 行: 序号 + 状态(ONLINE/OFFLINE) + 8个逻辑板值 + board_re = re.compile( + r"^\s*(\d+)\s+(ONLINE|OFFLINE)\s+" + r"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+" + r"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s*$" + ) + + for line in lines: + # Rack 头: "Rack 0 has 2 clusters" + rack_match = re.match(r"Rack\s+(\d+)\s+has\s+(\d+)\s+clusters", line) + if rack_match: + current_rack = int(rack_match.group(1)) + result["racks"].append({ + "rack": current_rack, + "clusters": [], + }) + continue + + # Cluster 头: "Cluster 0 has 6 boards CCD: ONLINE" + cluster_match = re.match( + r"Cluster\s+(\d+)\s+has\s+(\d+)\s+boards\s+CCD:\s*(\w+)", line + ) + if cluster_match and result["racks"]: + current_cluster = int(cluster_match.group(1)) + ccd = cluster_match.group(3) + result["racks"][-1]["clusters"].append({ + "cluster": current_cluster, + "ccd": ccd, + "boards": [], + }) + continue + + # LD 板行 + board_match = board_re.match(line) + if board_match and result["racks"] and result["racks"][-1]["clusters"]: + ld_idx = int(board_match.group(1)) + status = board_match.group(2) + domains = [board_match.group(i) for i in range(3, 11)] + result["racks"][-1]["clusters"][-1]["boards"].append({ + "ld": ld_idx, + "status": status, + "domains": domains, + }) + continue + + # ─── 作业信息 ────────────────────────────────────────────── + job_section = re.search( + r"Job Information:.*?(?=Target Pod Information:|Exit code:|$)", + raw_text, re.DOTALL, + ) + if job_section: + job_lines = job_section.group(0).split("\n") + for jline in job_lines: + jline = jline.strip() + if not jline or jline.startswith("Job Information") or jline.startswith("Index"): + continue + # 分割后从两端取值: 前3个是 index/owner/pid, 后3个是 design/elaptime/reservedkey + # 中间的是 T-Pod (可能含空格, 如 "-- --") + parts = jline.split() + if len(parts) >= 7: + idx = int(parts[0]) + owner = parts[1] + pid = parts[2] + reserved_key = parts[-1] + elap_time = parts[-2] + design = parts[-3] + t_pod = " ".join(parts[3:-3]) + result["jobs"].append({ + "index": idx, + "owner": owner, + "pid": pid, + "t_pod": t_pod, + "design": design, + "elap_time": elap_time, + "reserved_key": reserved_key, + }) + + # ─── T-Pod 信息 ──────────────────────────────────────────── + pod_section = re.search( + r"Target Pod Information:.*?(?=Exit code:|$)", + raw_text, re.DOTALL, + ) + if pod_section: + pod_text = pod_section.group(0) + # 按 "Rack N:" 分割 + rack_pod_re = re.compile(r"Rack\s+(\d+):(.*?)(?=Rack\s+\d+:|Exit code:|$)", re.DOTALL) + for m in rack_pod_re.finditer(pod_text): + rack_num = int(m.group(1)) + rack_text = m.group(2) + + all_hdsb = {} + all_t_pods = {} + fields = { + "available_hdsb": "", + "available_t_pods": "", + "locked_hdsb": "", + "locked_t_pods": "", + "reserved_hdsb": "", + "reserved_t_pods": "", + "unavailable_hdsb": "", + "unavailable_t_pods": "", + } + + for rline in rack_text.split("\n"): + rline = rline.strip() + if not rline: + continue + + # "All length 5 HDSB: T4" / "All length 30 HDSB: T5" + all_h_match = re.match(r"All length (\d+) HDSB:\s*(.+)", rline) + if all_h_match: + all_hdsb[all_h_match.group(1)] = all_h_match.group(2).strip() + continue + + # "All length 5 T-Pods: T0, T1, ..." + all_t_match = re.match(r"All length (\d+) T-Pods:\s*(.+)", rline) + if all_t_match: + all_t_pods[all_t_match.group(1)] = all_t_match.group(2).strip() + continue + + # 标准字段 — 用 startswith 精确匹配避免 "Available" vs "Unavailable" 混淆 + for field_key in fields: + label = field_key.replace("_", " ").title() + # 修正大小写和连字符: "Hdsb"→"HDSB", "T Pods"→"T-Pods" + label = label.replace("Hdsb", "HDSB").replace("T Pods", "T-Pods") + if rline.startswith(label + ":"): + val = rline.split(":", 1)[1].strip() + fields[field_key] = val + break + + result["pods"].append({ + "rack": rack_num, + "all_hdsb": json.dumps(all_hdsb), + "all_t_pods": json.dumps(all_t_pods), + **fields, + }) + + return result + + +def classify_domain(raw_value: str) -> str: + """将域原始值分类为状态常量。""" + v = raw_value.strip() + if v == "-": + return DOMAIN_AVAILABLE + if v == "X": + return DOMAIN_NOT_EXIST + if v == "D": + return DOMAIN_DISABLED + if v.isdigit(): + return DOMAIN_DOWNLOADED + return DOMAIN_AVAILABLE + + +def calculate_stats(parsed: dict) -> dict: + """从解析结果计算汇总统计。""" + total_boards = online = offline = used = 0 + total_dom = avail = dl = dis = ne = 0 + + for rack in parsed["racks"]: + for cluster in rack["clusters"]: + for board in cluster["boards"]: + total_boards += 1 + if board["status"] == "ONLINE": + online += 1 + else: + offline += 1 + board_used = False + for d in board["domains"]: + state = classify_domain(d) + total_dom += 1 + if state == DOMAIN_AVAILABLE: + avail += 1 + elif state == DOMAIN_DOWNLOADED: + dl += 1 + board_used = True + elif state == DOMAIN_DISABLED: + dis += 1 + elif state == DOMAIN_NOT_EXIST: + ne += 1 + if board_used: + used += 1 + + util = round(used / total_boards * 100, 1) if total_boards > 0 else 0.0 + + return { + "total_boards": total_boards, + "online_boards": online, + "offline_boards": offline, + "used_boards": used, + "total_domains": total_dom, + "available_domains": avail, + "downloaded_domains": dl, + "disabled_domains": dis, + "not_exist_domains": ne, + "active_jobs": len(parsed["jobs"]), + "utilization": util, + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# Template Filters +# ═══════════════════════════════════════════════════════════════════════════ +@app.template_filter("domain_state") +def domain_state_filter(raw_value): + return classify_domain(raw_value) + + +@app.template_filter("domain_color") +def domain_color_filter(raw_value): + return DOMAIN_COLORS.get(classify_domain(raw_value), "#888") + + +@app.template_filter("domain_label") +def domain_label_filter(raw_value): + return DOMAIN_LABELS.get(classify_domain(raw_value), "未知") + + +@app.template_filter("fmt_time") +def fmt_time_filter(dt): + if not dt: + return "—" + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + local = dt.astimezone() + return local.strftime("%Y-%m-%d %H:%M:%S") + + +@app.template_filter("fmt_time_short") +def fmt_time_short_filter(dt): + if not dt: + return "—" + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + local = dt.astimezone() + return local.strftime("%H:%M:%S") + + +@app.template_filter("from_json") +def from_json_filter(s): + if not s: + return {} + try: + return json.loads(s) + except (json.JSONDecodeError, TypeError): + return {} + + +# ═══════════════════════════════════════════════════════════════════════════ +# Routes +# ═══════════════════════════════════════════════════════════════════════════ +@app.route("/") +def dashboard(): + """主仪表板 — 显示最新快照数据。""" + snapshot = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).first() + stats = None + racks_data = [] + jobs_data = [] + pods_data = [] + history = [] + + if snapshot: + stats = { + "total_boards": snapshot.total_boards, + "online_boards": snapshot.online_boards, + "offline_boards": snapshot.offline_boards, + "total_domains": snapshot.total_domains, + "available_domains": snapshot.available_domains, + "downloaded_domains": snapshot.downloaded_domains, + "disabled_domains": snapshot.disabled_domains, + "not_exist_domains": snapshot.not_exist_domains, + "active_jobs": snapshot.active_jobs, + "used_boards": snapshot.used_boards, + "utilization": snapshot.utilization, + } + # 重建 racks 结构 + boards = BoardStatus.query.filter_by(snapshot_id=snapshot.id).order_by( + BoardStatus.rack, BoardStatus.cluster, BoardStatus.ld_index + ).all() + rack_map = {} + for b in boards: + if b.rack not in rack_map: + rack_map[b.rack] = {} + if b.cluster not in rack_map[b.rack]: + rack_map[b.rack][b.cluster] = { + "cluster": b.cluster, + "ccd": b.ccd or "UNKNOWN", + "boards": [], + } + rack_map[b.rack][b.cluster]["boards"].append({ + "ld": b.ld_index, + "status": b.status, + "domains": [b.d0, b.d1, b.d2, b.d3, b.d4, b.d5, b.d6, b.d7], + }) + for rack_num in sorted(rack_map): + racks_data.append({ + "rack": rack_num, + "clusters": [rack_map[rack_num][c] for c in sorted(rack_map[rack_num])], + }) + + jobs_data = JobInfo.query.filter_by(snapshot_id=snapshot.id).all() + pods_data = PodInfo.query.filter_by(snapshot_id=snapshot.id).order_by(PodInfo.rack).all() + + # 历史趋势 (最近50条) + history = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).limit(50).all() + history = list(reversed(history)) # 时间正序 + + return render_template( + "dashboard.html", + snapshot=snapshot, + stats=stats, + racks=racks_data, + jobs=jobs_data, + pods=pods_data, + history=history, + ) + + +@app.route("/history") +def history(): + """历史快照列表。""" + page = request.args.get("page", 1, type=int) + per_page = 30 + pagination = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).paginate( + page=page, per_page=per_page, error_out=False + ) + return render_template("history.html", pagination=pagination) + + +@app.route("/snapshot/") +def snapshot_detail(sid): + """单条快照详情。""" + snapshot = db.get_or_404(ScanSnapshot, sid) + stats = { + "total_boards": snapshot.total_boards, + "online_boards": snapshot.online_boards, + "offline_boards": snapshot.offline_boards, + "total_domains": snapshot.total_domains, + "available_domains": snapshot.available_domains, + "downloaded_domains": snapshot.downloaded_domains, + "disabled_domains": snapshot.disabled_domains, + "not_exist_domains": snapshot.not_exist_domains, + "active_jobs": snapshot.active_jobs, + "used_boards": snapshot.used_boards, + "utilization": snapshot.utilization, + } + boards = BoardStatus.query.filter_by(snapshot_id=sid).order_by( + BoardStatus.rack, BoardStatus.cluster, BoardStatus.ld_index + ).all() + rack_map = {} + for b in boards: + if b.rack not in rack_map: + rack_map[b.rack] = {} + if b.cluster not in rack_map[b.rack]: + rack_map[b.rack][b.cluster] = { + "cluster": b.cluster, + "ccd": b.ccd or "UNKNOWN", + "boards": [], + } + rack_map[b.rack][b.cluster]["boards"].append({ + "ld": b.ld_index, + "status": b.status, + "domains": [b.d0, b.d1, b.d2, b.d3, b.d4, b.d5, b.d6, b.d7], + }) + racks_data = [] + for rack_num in sorted(rack_map): + racks_data.append({ + "rack": rack_num, + "clusters": [rack_map[rack_num][c] for c in sorted(rack_map[rack_num])], + }) + jobs_data = JobInfo.query.filter_by(snapshot_id=sid).all() + pods_data = PodInfo.query.filter_by(snapshot_id=sid).order_by(PodInfo.rack).all() + return render_template( + "snapshot.html", + snapshot=snapshot, + stats=stats, + racks=racks_data, + jobs=jobs_data, + pods=pods_data, + ) + + +@app.route("/paste", methods=["POST"]) +def paste_data(): + """手动粘贴 test_server -short 输出。""" + raw_output = request.form.get("raw_output", "").strip() + if not raw_output: + return jsonify({"error": "无输出内容"}), 400 + sid = store_snapshot(raw_output, source="paste") + return redirect(url_for("dashboard")) + + +@app.route("/api/collect", methods=["POST"]) +def api_collect(): + """API 端点 — 接收 test_server -short 原始输出。""" + data = request.get_json(silent=True) + if not data or "raw_output" not in data: + return jsonify({"error": "缺少 raw_output 字段"}), 400 + raw_output = data["raw_output"] + source = data.get("source", "api") + sid = store_snapshot(raw_output, source=source) + snapshot = db.session.get(ScanSnapshot, sid) + return jsonify({ + "ok": True, + "snapshot_id": sid, + "stats": { + "total_boards": snapshot.total_boards, + "online_boards": snapshot.online_boards, + "offline_boards": snapshot.offline_boards, + "total_domains": snapshot.total_domains, + "available_domains": snapshot.available_domains, + "downloaded_domains": snapshot.downloaded_domains, + "disabled_domains": snapshot.disabled_domains, + "not_exist_domains": snapshot.not_exist_domains, + "active_jobs": snapshot.active_jobs, + "used_boards": snapshot.used_boards, + "utilization": snapshot.utilization, + }, + }) + + +@app.route("/api/latest") +def api_latest(): + """返回最新快照 JSON。""" + snapshot = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).first() + if not snapshot: + return jsonify({"error": "无数据"}), 404 + boards = BoardStatus.query.filter_by(snapshot_id=snapshot.id).all() + jobs = JobInfo.query.filter_by(snapshot_id=snapshot.id).all() + pods = PodInfo.query.filter_by(snapshot_id=snapshot.id).all() + return jsonify({ + "snapshot": { + "id": snapshot.id, + "timestamp": snapshot.timestamp.isoformat() if snapshot.timestamp else None, + "emulator": snapshot.emulator, + "hardware": snapshot.hardware, + "configmgr": snapshot.configmgr, + "system_status": snapshot.system_status, + "source": snapshot.source, + }, + "stats": { + "total_boards": snapshot.total_boards, + "online_boards": snapshot.online_boards, + "offline_boards": snapshot.offline_boards, + "total_domains": snapshot.total_domains, + "available_domains": snapshot.available_domains, + "downloaded_domains": snapshot.downloaded_domains, + "disabled_domains": snapshot.disabled_domains, + "not_exist_domains": snapshot.not_exist_domains, + "active_jobs": snapshot.active_jobs, + "used_boards": snapshot.used_boards, + "utilization": snapshot.utilization, + }, + "boards": [{"rack": b.rack, "cluster": b.cluster, "ld": b.ld_index, + "status": b.status, "ccd": b.ccd, + "domains": [b.d0, b.d1, b.d2, b.d3, b.d4, b.d5, b.d6, b.d7]} + for b in boards], + "jobs": [{"index": j.job_index, "owner": j.owner, "pid": j.pid, + "t_pod": j.t_pod, "design": j.design, + "elap_time": j.elap_time, "reserved_key": j.reserved_key} + for j in jobs], + "pods": [{"rack": p.rack, + "all_hdsb": json.loads(p.all_hdsb) if p.all_hdsb else {}, + "all_t_pods": json.loads(p.all_t_pods) if p.all_t_pods else {}, + "available_hdsb": p.available_hdsb, "available_t_pods": p.available_t_pods, + "locked_hdsb": p.locked_hdsb, "locked_t_pods": p.locked_t_pods, + "reserved_hdsb": p.reserved_hdsb, "reserved_t_pods": p.reserved_t_pods, + "unavailable_hdsb": p.unavailable_hdsb, "unavailable_t_pods": p.unavailable_t_pods} + for p in pods], + }) + + +@app.route("/api/history") +def api_history(): + """历史趋势 JSON (用于图表)。""" + snapshots = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).limit(100).all() + snapshots = list(reversed(snapshots)) + return jsonify({ + "data": [{ + "timestamp": s.timestamp.isoformat() if s.timestamp else None, + "utilization": s.utilization, + "online_boards": s.online_boards, + "offline_boards": s.offline_boards, + "downloaded_domains": s.downloaded_domains, + "available_domains": s.available_domains, + "active_jobs": s.active_jobs, + } for s in snapshots] + }) + + +@app.route("/api/snapshot//delete", methods=["POST"]) +def delete_snapshot(sid): + """删除指定快照。""" + snapshot = db.get_or_404(ScanSnapshot, sid) + db.session.delete(snapshot) + db.session.commit() + return redirect(url_for("history")) + + +@app.route("/api/cleanup", methods=["POST"]) +def cleanup_old(): + """清理 N 天前的快照。""" + days = request.args.get("days", 30, type=int) + cutoff = datetime.now(timezone.utc) - timedelta(days=days) + old = ScanSnapshot.query.filter(ScanSnapshot.timestamp < cutoff).all() + count = len(old) + for s in old: + db.session.delete(s) + db.session.commit() + return jsonify({"ok": True, "deleted": count}) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Email Routes (延迟导入 mailer, 避免在 mailer 缺失时整个 app 起不来) +# ═══════════════════════════════════════════════════════════════════════════ +def _get_mailer(): + try: + import mailer + return mailer + except ImportError as e: + return None + + +@app.route("/api/email/status") +def email_status(): + """返回当前邮件配置状态 (不泄露密码)。""" + ml = _get_mailer() + if not ml: + return jsonify({"ok": False, "error": "mailer.py 不存在或导入失败"}), 500 + cfg = ml.get_config() + errs = ml.config_errors(cfg) + return jsonify({ + "ok": True, + "enabled": cfg["enabled"], + "configured": len(errs) == 0, + "config_errors": errs, + "smtp_host": cfg["smtp_host"], + "smtp_port": cfg["smtp_port"], + "smtp_use_tls": cfg["smtp_use_tls"], + "smtp_use_ssl": cfg["smtp_use_ssl"], + "from_addr": cfg["from_addr"], + "from_name": cfg["from_name"], + "recipients": cfg["to_addrs"], + "send_at": f"{cfg['send_hour']:02d}:{cfg['send_minute']:02d}", + "monitor_base_url": cfg["monitor_base_url"], + }) + + +@app.route("/api/email/preview") +def email_preview(): + """渲染报告到文件 (不发送), 返回文件路径供下载预览。""" + import tempfile + ml = _get_mailer() + if not ml: + return jsonify({"ok": False, "error": "mailer.py 不可用"}), 500 + fd, path = tempfile.mkstemp(prefix="palladium-mail-preview-", suffix=".eml", dir="/tmp") + os.close(fd) + ml.write_preview(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + try: + os.unlink(path) + except OSError: + pass + return Response(content, mimetype="text/plain; charset=utf-8", + headers={"Content-Disposition": 'inline; filename="mail-preview.txt"'}) + + +@app.route("/api/email/send", methods=["POST"]) +def email_send(): + """ + 手动触发一次发送 (用于测试)。 + + 安全: 除非 ALLOW_EMAIL_TRIGGER=true, 否则禁止 (防止误触发给真实收件人发邮件)。 + """ + ml = _get_mailer() + if not ml: + return jsonify({"ok": False, "error": "mailer.py 不可用"}), 500 + if os.environ.get("ALLOW_EMAIL_TRIGGER", "").strip().lower() not in ("1", "true", "yes"): + return jsonify({ + "ok": False, + "error": "未授权: 设置环境变量 ALLOW_EMAIL_TRIGGER=true 后才能手动发送" + }), 403 + result = ml.send_report() + status = 200 if result["ok"] else 500 + return jsonify(result), status + + +# ═══════════════════════════════════════════════════════════════════════════ +# Email Scheduler (后台 daemon thread, 每天指定时间触发) +# ═══════════════════════════════════════════════════════════════════════════ +_scheduler_started = False +_scheduler_lock = threading.Lock() + + +def _seconds_until_next(hour: int, minute: int) -> int: + """距离下一次 hh:mm 还有多少秒 (0 ~ 86400)。""" + now = datetime.now() + target = now.replace(hour=hour, minute=minute, second=0, microsecond=0) + if target <= now: + target += timedelta(days=1) + return int((target - now).total_seconds()) + + +def _scheduler_loop(): + """后台线程: 每天到点调一次 send_report(), 错误也不退出, 1 分钟后自愈。""" + import mailer + print("[scheduler] 邮件调度线程已启动", flush=True) + while True: + try: + cfg = mailer.get_config() + if not cfg["enabled"]: + # 关闭, 每小时看一眼 + time.sleep(3600) + continue + wait = _seconds_until_next(cfg["send_hour"], cfg["send_minute"]) + print(f"[scheduler] 下次发送: {wait}s 后 " + f"({cfg['send_hour']:02d}:{cfg['send_minute']:02d})", flush=True) + time.sleep(wait) + # 到点了, 触发一次 + print("[scheduler] 触发每日报告发送...", flush=True) + result = mailer.send_report(cfg) + if result["ok"]: + print(f"[scheduler] 发送成功: {result['subject']} -> {result['recipients']}", + flush=True) + else: + print(f"[scheduler] 发送失败: {result['error']}", flush=True) + except Exception as e: + # 出错不退出, 1 分钟后重试 + print(f"[scheduler] 异常: {type(e).__name__}: {e}", flush=True) + time.sleep(60) + + +def start_email_scheduler(): + """启动后台调度线程 (只启动一次)。""" + global _scheduler_started + with _scheduler_lock: + if _scheduler_started: + return + # 延迟导入, 避免 mailer 缺失时启动失败 + ml = _get_mailer() + if ml is None: + print("[scheduler] mailer.py 不可用, 调度器未启动", flush=True) + return + # 检查配置: 至少 SMTP_HOST 设置了才启动 + cfg = ml.get_config() + if not cfg["smtp_host"]: + print("[scheduler] SMTP_HOST 未设置, 调度器未启动 " + "(设了之后 gunicorn 重启生效)", flush=True) + return + t = threading.Thread(target=_scheduler_loop, name="email-scheduler", daemon=True) + t.start() + _scheduler_started = True + + +# ═══════════════════════════════════════════════════════════════════════════ +# Export — 历史快照导出 (CSV / JSON) +# ═══════════════════════════════════════════════════════════════════════════ +EXPORT_FIELDS = [ + ("id", "快照 ID"), + ("timestamp_local", "时间(本地)"), + ("timestamp_utc", "时间(UTC)"), + ("emulator", "仿真器"), + ("hardware", "硬件"), + ("configmgr", "ConfigMgr"), + ("system_status", "系统状态"), + ("source", "数据来源"), + ("total_boards", "板总数"), + ("online_boards", "在线板"), + ("offline_boards", "离线板"), + ("used_boards", "使用板"), + ("utilization", "利用率%"), + ("total_domains", "域总数"), + ("available_domains", "可用域"), + ("downloaded_domains", "已加载域"), + ("disabled_domains", "已禁用域"), + ("not_exist_domains", "不存在域"), + ("active_jobs", "活跃作业"), +] + + +def _build_export_query(): + """ + 解析导出过滤参数, 返回构建好的 query。 + 参数: + - days: 限制最近 N 天 (0/缺省=全部) + - since: ISO 日期 (YYYY-MM-DD 或 YYYY-MM-DDTHH:MM:SS) + - until: 同上 + - source: 模糊匹配 source 字段 (如 "collect.py" 或 "scmp03") + - limit: 最多返回 N 条 (默认 10000, 上限 100000) + """ + q = ScanSnapshot.query + + # ── 时间范围 ── + days = request.args.get("days", type=int) + if days and days > 0: + cutoff = datetime.now(timezone.utc) - timedelta(days=days) + q = q.filter(ScanSnapshot.timestamp >= cutoff) + + def _parse_dt(s: str): + """接受 'YYYY-MM-DD' 或完整 ISO, 返回带 tz 的 datetime。""" + if not s: + return None + s = s.strip() + for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"): + try: + dt = datetime.strptime(s, fmt) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + except ValueError: + continue + return None + + since = _parse_dt(request.args.get("since", "")) + if since is not None: + q = q.filter(ScanSnapshot.timestamp >= since) + until = _parse_dt(request.args.get("until", "")) + if until is not None: + q = q.filter(ScanSnapshot.timestamp <= until) + + # ── 来源过滤 ── + source = request.args.get("source", "").strip() + if source: + q = q.filter(ScanSnapshot.source.like(f"%{source}%")) + + # ── 排序 + limit ── + limit = min(request.args.get("limit", 10000, type=int), 100000) + q = q.order_by(ScanSnapshot.timestamp.asc()).limit(limit) + return q + + +def _snapshot_to_row(snap: ScanSnapshot) -> dict: + """把 ORM 对象转成可序列化的 dict (供 CSV/JSON 共享)。""" + ts = snap.timestamp + if ts and ts.tzinfo is None: + ts = ts.replace(tzinfo=timezone.utc) + return { + "id": snap.id, + "timestamp_utc": ts.isoformat() if ts else "", + "timestamp_local": ts.astimezone().strftime("%Y-%m-%d %H:%M:%S") if ts else "", + "emulator": snap.emulator or "", + "hardware": snap.hardware or "", + "configmgr": snap.configmgr or "", + "system_status": snap.system_status or "", + "source": snap.source or "", + "total_boards": snap.total_boards, + "online_boards": snap.online_boards, + "offline_boards": snap.offline_boards, + "used_boards": snap.used_boards, + "utilization": snap.utilization, + "total_domains": snap.total_domains, + "available_domains": snap.available_domains, + "downloaded_domains": snap.downloaded_domains, + "disabled_domains": snap.disabled_domains, + "not_exist_domains": snap.not_exist_domains, + "active_jobs": snap.active_jobs, + } + + +@app.route("/api/export") +def api_export(): + """ + 导出历史快照。 + + Query: + - format: csv | json (默认 csv) + - days: 限制最近 N 天 + - since: 起始时间 (YYYY-MM-DD 或 ISO) + - until: 截止时间 + - source: 模糊匹配 source 字段 + - limit: 最多 N 条 (默认 10000, 上限 100000) + + CSV 文件名: palladium_snapshots_YYYYMMDD_HHMMSS.csv + JSON 文件名: palladium_snapshots_YYYYMMDD_HHMMSS.json + """ + q = _build_export_query() + rows = [_snapshot_to_row(s) for s in q.all()] + + fmt = request.args.get("format", "csv").lower() + ts_str = datetime.now().strftime("%Y%m%d_%H%M%S") + + if fmt == "json": + body = { + "exported_at": datetime.now(timezone.utc).isoformat(), + "count": len(rows), + "rows": rows, + } + resp = jsonify(body) + resp.headers["Content-Disposition"] = ( + f'attachment; filename="palladium_snapshots_{ts_str}.json"' + ) + return resp + + # ── CSV (默认) ── + buf = io.StringIO() + # 写 BOM 让 Excel 直接识别 UTF-8 (否则中文乱码) + buf.write("\ufeff") + writer = csv.writer(buf) + writer.writerow([label for _, label in EXPORT_FIELDS]) + for r in rows: + writer.writerow([r.get(key, "") for key, _ in EXPORT_FIELDS]) + csv_data = buf.getvalue() + + resp = Response(csv_data, mimetype="text/csv") + resp.headers["Content-Disposition"] = ( + f'attachment; filename="palladium_snapshots_{ts_str}.csv"' + ) + return resp + + +# ═══════════════════════════════════════════════════════════════════════════ +# Storage Helper +# ═══════════════════════════════════════════════════════════════════════════ +def store_snapshot(raw_output: str, source: str = "manual") -> int: + """解析、存储一条快照，返回 snapshot id。""" + parsed = parse_test_server_output(raw_output) + stats = calculate_stats(parsed) + sysinfo = parsed["system"] + + snap = ScanSnapshot( + emulator=sysinfo.get("emulator", ""), + hardware=sysinfo.get("hardware", ""), + configmgr=sysinfo.get("configmgr", ""), + system_status=sysinfo.get("system_status", ""), + source=source, + raw_output=raw_output, + total_boards=stats["total_boards"], + online_boards=stats["online_boards"], + offline_boards=stats["offline_boards"], + total_domains=stats["total_domains"], + available_domains=stats["available_domains"], + downloaded_domains=stats["downloaded_domains"], + disabled_domains=stats["disabled_domains"], + not_exist_domains=stats["not_exist_domains"], + active_jobs=stats["active_jobs"], + used_boards=stats["used_boards"], + utilization=stats["utilization"], + ) + db.session.add(snap) + db.session.flush() # 获取 snap.id + + for rack in parsed["racks"]: + for cluster in rack["clusters"]: + for board in cluster["boards"]: + domains = board["domains"] + db.session.add(BoardStatus( + snapshot_id=snap.id, + rack=rack["rack"], + cluster=cluster["cluster"], + ld_index=board["ld"], + status=board["status"], + ccd=cluster["ccd"], + d0=domains[0], d1=domains[1], d2=domains[2], d3=domains[3], + d4=domains[4], d5=domains[5], d6=domains[6], d7=domains[7], + )) + + for job in parsed["jobs"]: + db.session.add(JobInfo( + snapshot_id=snap.id, + job_index=job["index"], + owner=job["owner"], + pid=job["pid"], + t_pod=job["t_pod"], + design=job["design"], + elap_time=job["elap_time"], + reserved_key=job["reserved_key"], + )) + + for pod in parsed["pods"]: + db.session.add(PodInfo( + snapshot_id=snap.id, + rack=pod["rack"], + all_hdsb=pod.get("all_hdsb", ""), + all_t_pods=pod.get("all_t_pods", ""), + available_hdsb=pod.get("available_hdsb", ""), + available_t_pods=pod.get("available_t_pods", ""), + locked_hdsb=pod.get("locked_hdsb", ""), + locked_t_pods=pod.get("locked_t_pods", ""), + reserved_hdsb=pod.get("reserved_hdsb", ""), + reserved_t_pods=pod.get("reserved_t_pods", ""), + unavailable_hdsb=pod.get("unavailable_hdsb", ""), + unavailable_t_pods=pod.get("unavailable_t_pods", ""), + )) + + db.session.commit() + return snap.id + + +# ═══════════════════════════════════════════════════════════════════════════ +# Seed Sample Data (首次运行) +# ═══════════════════════════════════════════════════════════════════════════ +SAMPLE_OUTPUT = r"""Emulator: sc01_emu Hardware: Palladium Z1 Configmgr: V21.02.102.s002System Status: PARTIAL +Rack 0 has 2 clusters +Cluster 0 has 6 boards CCD: ONLINE + LD Status D0 D1 D2 D3 D4 D5 D6 D7 + 0 ONLINE - - - - - - - - + 1 ONLINE - - - - - - - - + 2 ONLINE - - - - - - - - + 3 ONLINE - - - - - - - - + 4 ONLINE - - - - - - - - + 5 ONLINE - - - - - - - - + +Cluster 1 has 6 boards CCD: ONLINE + LD Status D0 D1 D2 D3 D4 D5 D6 D7 + 6 ONLINE 1 1 1 1 1 1 1 1 + 7 ONLINE 1 1 1 1 1 1 1 1 + 8 ONLINE - - - - - - - - + 9 OFFLINE X X X X X X X X + 10 ONLINE - - - - - - - - + 11 ONLINE - - - - - - - - + +Rack 1 has 2 clusters +Cluster 3 has 6 boards CCD: ONLINE + LD Status D0 D1 D2 D3 D4 D5 D6 D7 + 18 ONLINE - - - - - - - - + 19 ONLINE - - - - - - - - + 20 ONLINE - - - - - - - - + 21 ONLINE - - - - - - - - + 22 ONLINE - - - - - - - - + 23 ONLINE - - - - - - - - + +Cluster 4 has 6 boards CCD: ONLINE + LD Status D0 D1 D2 D3 D4 D5 D6 D7 + 24 ONLINE - - - - - - - - + 25 ONLINE - - - - - - - - + 26 ONLINE - - - - - - - - + 27 ONLINE - - - - - - - - + 28 ONLINE - - - - - - - - + 29 ONLINE - - - - - - - - + + +X: Domain does not exist +D: Domain is disabled +-: Domain is available +: Domain is downloaded or reserved + +Job Information: +Index Owner PID T-Pod Design ElapTime ReservedKey + 1 pzuser101 scmp03:106144 -- -- top 00:35:39 -- + +Target Pod Information: +Rack 0: +All length 5 HDSB: T4 +All length 5 T-Pods: T0, T1, T2, T3, T5, T6, T7, T8, T9 +Available HDSB: T4 +Available T-Pods: T0, T1, T2, T3, T5 +Locked HDSB: NONE +Locked T-Pods: NONE +Reserved HDSB: NONE +Reserved T-Pods: NONE +Unavailable HDSB: NONE +Unavailable T-Pods: T6, T7, T8, T9 + +Rack 1: +All length 5 HDSB: T4 +All length 30 HDSB: T5 +Available HDSB: T4, T5 +Available T-Pods: NONE +Locked HDSB: NONE +Locked T-Pods: NONE +Reserved HDSB: NONE +Reserved T-Pods: NONE +Unavailable HDSB: NONE +Unavailable T-Pods: NONE + +Exit code: 0 +Deinitializing IB - 0 - 0 +""" + + +def seed_sample_data(): + """首次运行时写入示例数据。""" + if ScanSnapshot.query.first(): + return + store_snapshot(SAMPLE_OUTPUT, source="sample") + print("[seed] 示例数据已写入") + + +# ═══════════════════════════════════════════════════════════════════════════ +# Gunicorn hook: 让 gunicorn worker 也能启动调度器 (每个 worker 各 1 个) +# 生产用法: --preload 时只有 master 会触发, --workers N 时 N 个 daemon 都会启动 +# daemon 内部有 _scheduler_started 锁, 实际只跑一个; 其余 sleep 等下一天的空跑无害 +# ═══════════════════════════════════════════════════════════════════════════ +def _gunicorn_post_fork(server, worker): # noqa: ARG001 + start_email_scheduler() + + +# ═══════════════════════════════════════════════════════════════════════════ +# Main +# ═══════════════════════════════════════════════════════════════════════════ +if __name__ == "__main__": + with app.app_context(): + db.create_all() + seed_sample_data() + start_email_scheduler() + print("════════════════════════════════════════════════════════") + print(" Palladium Z1 Monitor") + print(" http://0.0.0.0:5100") + print("════════════════════════════════════════════════════════") + app.run(host="0.0.0.0", port=5100, debug=True) diff --git a/palladium-monitor/collect.py b/palladium-monitor/collect.py new file mode 100644 index 0000000..27f8d29 --- /dev/null +++ b/palladium-monitor/collect.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +""" +collect.py — Palladium Z1 远程数据采集推送脚本 + +在 Palladium Z1 主机上运行，执行 test_server -short 并将结果推送到监控服务器。 + +用法: + python3 collect.py # 默认配置 + python3 collect.py --url http://monitor:5100/api/collect + python3 collect.py --host scmp03 --env /edatools/vxe/VXE23.03.001/bin/env.csh + python3 collect.py --dry-run # 只采集不推送 + python3 collect.py --log /var/log/palladium-collect.log # 追加写日志 + +可配合 crontab 定时执行 (每 30 分钟): + */30 * * * * /opt/palladium-monitor/collect.py --host scmp03 --url http://MONITOR:5100/api/collect --log /var/log/palladium-collect.log >> /var/log/palladium-collect.cron.log 2>&1 +""" + +import argparse +import os +import socket +import subprocess +import sys +import time +from datetime import datetime + +try: + import requests +except ImportError: + print("ERROR: requests 未安装，请执行 pip install requests", file=sys.stderr) + sys.exit(1) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Defaults +# ═══════════════════════════════════════════════════════════════════════════ +DEFAULT_URL = "http://127.0.0.1:5000/api/collect" +DEFAULT_CMD = "test_server -short" +DEFAULT_TIMEOUT = 300 +DEFAULT_RETRIES = 3 +DEFAULT_RETRY_DELAY = 5 + + +# ═══════════════════════════════════════════════════════════════════════════ +# Logging +# ═══════════════════════════════════════════════════════════════════════════ +class Logger: + """同时输出到 stderr 和可选的日志文件。""" + + def __init__(self, log_file: str | None = None): + self.log_file = log_file + + def _emit(self, level: str, msg: str, ts: str) -> None: + line = f"{ts} [{level}] {msg}" + print(line, file=sys.stderr) + if self.log_file: + try: + os.makedirs(os.path.dirname(self.log_file), exist_ok=True) + with open(self.log_file, "a", encoding="utf-8") as f: + f.write(line + "\n") + except OSError as e: + print(f"{ts} [WARN] 无法写入日志文件 {self.log_file}: {e}", file=sys.stderr) + + def info(self, msg: str) -> None: + self._emit("INFO", msg, datetime.now().strftime("%Y-%m-%d %H:%M:%S")) + + def warn(self, msg: str) -> None: + self._emit("WARN", msg, datetime.now().strftime("%Y-%m-%d %H:%M:%S")) + + def error(self, msg: str) -> None: + self._emit("ERROR", msg, datetime.now().strftime("%Y-%m-%d %H:%M:%S")) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Command Execution +# ═══════════════════════════════════════════════════════════════════════════ +def build_shell_cmd(cmd: str, env_script: str | None) -> str: + """ + 如果指定了 env_script，把 source env 之后再执行 cmd 拼成一个 shell 串。 + 不指定则直接返回 cmd。 + """ + if not env_script: + return cmd + # env 文件可能是 csh 或 sh 后缀, 用 bash -c 强制走 sh 语义, 兼容大多数 vendor + # 假设 vendor 提供的 env.* 是 POSIX 兼容 (setenv → export) + return f"source {shlex_quote(env_script)} >/dev/null 2>&1; {cmd}" + + +def shlex_quote(s: str) -> str: + """最小 shell 单引号转义 (用于 env_script 路径里有空格的情况)。""" + return "'" + s.replace("'", "'\\''") + "'" + + +def run_test_server(cmd: str, timeout: int, env_script: str | None) -> tuple[str, int, str]: + """ + 执行 test_server -short, 返回 (stdout, returncode, stderr)。 + timeout 触发时 subprocess.run 不会抛 (>=3.12 subprocessing 时可能抛), 这里显式处理。 + """ + full_cmd = build_shell_cmd(cmd, env_script) + try: + result = subprocess.run( + full_cmd, + shell=True, + executable="/bin/bash", + capture_output=True, + timeout=timeout, + ) + except subprocess.TimeoutExpired as e: + return (e.stdout.decode("utf-8", errors="replace") if e.stdout else "", -1, + f"命令超时 (>={timeout}s)") + except FileNotFoundError as e: + return ("", 127, f"shell 不可用: {e}") + + stdout = result.stdout.decode("utf-8", errors="replace") if result.stdout else "" + stderr = result.stderr.decode("utf-8", errors="replace") if result.stderr else "" + return (stdout, result.returncode, stderr) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Push to monitor +# ═══════════════════════════════════════════════════════════════════════════ +def push_to_monitor(url: str, raw_output: str, source: str, + retries: int, retry_delay: int, + log: Logger) -> tuple[bool, dict | None, str]: + """带重试的 HTTP 推送, 返回 (ok, json_body_or_None, error_msg)。""" + payload = {"raw_output": raw_output, "source": source} + last_err = "" + for attempt in range(1, retries + 1): + try: + resp = requests.post(url, json=payload, timeout=30) + except requests.exceptions.ConnectionError as e: + last_err = f"无法连接 {url}: {e}" + except requests.exceptions.Timeout as e: + last_err = f"推送超时: {e}" + except requests.exceptions.RequestException as e: + last_err = f"请求异常: {e}" + else: + if resp.status_code == 200: + try: + return (True, resp.json(), "") + except ValueError: + return (True, None, "") + last_err = f"HTTP {resp.status_code}: {resp.text[:200]}" + + if attempt < retries: + log.warn(f"推送失败 (第 {attempt}/{retries} 次): {last_err} — {retry_delay}s 后重试") + time.sleep(retry_delay) + return (False, None, last_err) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Main +# ═══════════════════════════════════════════════════════════════════════════ +def main() -> int: + parser = argparse.ArgumentParser( + description="Palladium Z1 数据采集推送 (cron 友好, 带重试和日志)", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + parser.add_argument("--url", default=DEFAULT_URL, + help=f"监控服务器 API 地址 (默认: {DEFAULT_URL})") + parser.add_argument("--cmd", default=DEFAULT_CMD, + help=f"采集命令 (默认: {DEFAULT_CMD})") + parser.add_argument("--host", default=socket.gethostname(), + help="本机标签, 会写入 source 字段 (默认: hostname)") + parser.add_argument("--env", default=None, + help="VXE env.csh/env.sh 路径, 自动 source 后再执行命令") + parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT, + help=f"命令超时秒数 (默认: {DEFAULT_TIMEOUT})") + parser.add_argument("--retries", type=int, default=DEFAULT_RETRIES, + help=f"推送失败重试次数 (默认: {DEFAULT_RETRIES})") + parser.add_argument("--retry-delay", type=int, default=DEFAULT_RETRY_DELAY, + help=f"重试间隔秒数 (默认: {DEFAULT_RETRY_DELAY})") + parser.add_argument("--log", default=None, + help="追加写日志的文件路径 (例如 /var/log/palladium-collect.log)") + parser.add_argument("--dry-run", action="store_true", + help="只采集不推送, 打印解析结果") + args = parser.parse_args() + + log = Logger(args.log) + source = f"collect.py@{args.host}" + log.info(f"开始采集 — host={args.host} cmd={args.cmd!r} env={args.env!r}") + + # ─── 执行采集命令 ────────────────────────────────────────── + stdout, rc, stderr = run_test_server(args.cmd, args.timeout, args.env) + if rc != 0: + log.warn(f"命令退出码 {rc} (stderr: {stderr.strip()[:300]})") + if not stdout.strip(): + log.error("命令无输出, 终止") + return 1 + + log.info(f"采集成功 — {len(stdout)} 字节, rc={rc}") + + # ─── dry-run 模式 ────────────────────────────────────────── + if args.dry_run: + print("─" * 60) + print(stdout) + print("─" * 60) + log.info("dry-run 模式, 不推送") + return 0 + + # ─── 推送到监控服务器 ────────────────────────────────────── + log.info(f"推送到 {args.url} (source={source})") + ok, body, err = push_to_monitor( + args.url, stdout, source, + args.retries, args.retry_delay, log, + ) + + if not ok: + log.error(f"推送失败 (已重试 {args.retries} 次): {err}") + return 2 + + # 成功 — 打印 stats 摘要 + snapshot_id = body.get("snapshot_id") if body else "?" + stats = body.get("stats") if body else None + if stats: + log.info( + f"推送成功 — snapshot #{snapshot_id} | " + f"板 {stats['total_boards']} (在线 {stats['online_boards']}/离线 {stats['offline_boards']}) | " + f"利用率 {stats['utilization']}% (使用 {stats['used_boards']}/{stats['total_boards']}) | " + f"作业 {stats['active_jobs']}" + ) + else: + log.info(f"推送成功 — snapshot #{snapshot_id} (无 stats 字段)") + + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except KeyboardInterrupt: + print("\n[!] 用户中断", file=sys.stderr) + sys.exit(130) diff --git a/palladium-monitor/install_z1_agent.sh b/palladium-monitor/install_z1_agent.sh new file mode 100755 index 0000000..7931029 --- /dev/null +++ b/palladium-monitor/install_z1_agent.sh @@ -0,0 +1,217 @@ +#!/bin/bash +#═══════════════════════════════════════════════════════════════════════════════ +# install_z1_agent.sh — Palladium Z1 主机端 collect.py 一键部署脚本 +#═══════════════════════════════════════════════════════════════════════════════ +# 用法: +# sudo ./install_z1_agent.sh --url http://MONITOR_HOST:5100/api/collect --host scmp03 +# sudo ./install_z1_agent.sh --uninstall +# +# 关键点: +# - 默认 30 分钟一次 (cron */30) +# - 自动检测 env.csh / env.sh (如不存在则提示用户提供 --env) +# - 幂等: 重复运行会先卸载旧的 +# - 保险模式: 不会覆盖用户已有 crontab, 也不会 rm 任何文件 +#═══════════════════════════════════════════════════════════════════════════════ + +set -euo pipefail + +# ─── Defaults ──────────────────────────────────────────────────────────────── +INSTALL_DIR="/opt/palladium-monitor" +LOG_DIR="/var/log" +LOG_FILE="$LOG_DIR/palladium-collect.log" +CRON_TAG="palladium-monitor-collect" # crontab 注释标记, 便于后续识别 +DEFAULT_TIMEOUT=300 +DEFAULT_RETRIES=3 +DEFAULT_RETRY_DELAY=5 + +# ─── CLI ───────────────────────────────────────────────────────────────────── +usage() { + cat <&2; usage; exit 1 ;; + esac +done + +# ─── Helpers ────────────────────────────────────────────────────────────────── +log_info() { echo "[INFO] $*"; } +log_warn() { echo "[WARN] $*" >&2; } +log_error() { echo "[ERROR] $*" >&2; } + +require_root() { + if [[ $EUID -ne 0 ]]; then + log_error "请用 root 运行: sudo $0 $*" + exit 1 + fi +} + +# 备份当前 crontab +backup_crontab() { + local bak="/tmp/crontab.backup.$(date +%Y%m%d_%H%M%S)" + if crontab -l >/dev/null 2>&1; then + crontab -l > "$bak" + log_info "已备份现有 crontab → $bak" + else + : > "$bak" + log_info "当前用户无 crontab, 备份为空文件 $bak" + fi +} + +# 移除已存在的同标记行 (幂等) +remove_existing_cron() { + if crontab -l 2>/dev/null | grep -q "$CRON_TAG"; then + log_info "检测到旧任务, 先移除 (幂等)" + crontab -l | grep -v "$CRON_TAG" | grep -v "^#" | crontab - + fi +} + +# ─── Uninstall ──────────────────────────────────────────────────────────────── +do_uninstall() { + require_root + log_info "卸载: 移除 crontab 任务 (标签: $CRON_TAG)" + remove_existing_cron + if [[ -f "$INSTALL_DIR/collect.py" ]]; then + log_info "移除脚本: $INSTALL_DIR/collect.py" + rm -f "$INSTALL_DIR/collect.py" + fi + log_info "卸载完成. 日志保留在 $LOG_FILE (如不再需要请手动清理)" + exit 0 +} + +[[ "$UNINSTALL" -eq 1 ]] && do_uninstall + +# ─── Install ────────────────────────────────────────────────────────────────── +require_root + +if [[ -z "$URL" ]]; then + log_error "--url 是必选参数" + usage + exit 1 +fi + +# 校验 URL 格式 (基本) +if ! [[ "$URL" =~ ^https?://[^:]+:[0-9]+/.* ]]; then + log_warn "URL 格式异常: $URL (期望 http://host:port/path)" + read -r -p "仍要继续? [y/N] " ans + [[ "$ans" =~ ^[Yy]$ ]] || { log_info "已取消"; exit 1; } +fi + +# 校验 env 脚本 (如果指定) +if [[ -n "$ENV_SCRIPT" ]]; then + if [[ ! -f "$ENV_SCRIPT" ]]; then + log_error "env 脚本不存在: $ENV_SCRIPT" + exit 1 + fi + log_info "env 脚本校验通过: $ENV_SCRIPT" +fi + +# 创建安装目录 +log_info "创建安装目录: $INSTALL_DIR" +mkdir -p "$INSTALL_DIR" + +# 复制 collect.py +SCRIPT_SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/collect.py" +if [[ ! -f "$SCRIPT_SRC" ]]; then + log_error "未找到 collect.py, 请把脚本和本安装器放在同一目录" + exit 1 +fi +install -m 0755 "$SCRIPT_SRC" "$INSTALL_DIR/collect.py" +log_info "已安装: $INSTALL_DIR/collect.py" + +# 确保 requests 可用 +if ! python3 -c "import requests" 2>/dev/null; then + log_warn "python3 'requests' 未安装, 尝试安装..." + if command -v pip3 >/dev/null; then + pip3 install requests + elif command -v pip >/dev/null; then + pip install requests + else + log_error "找不到 pip/pip3, 请手动安装 requests 后重试" + exit 1 + fi +fi +log_info "Python 'requests' 模块可用" + +# 准备日志 +touch "$LOG_FILE" 2>/dev/null || { log_error "无法写入 $LOG_FILE"; exit 1; } +chmod 0644 "$LOG_FILE" + +# 安装 crontab +log_info "安装 crontab — 每 $INTERVAL 分钟, 标签: $CRON_TAG" +backup_crontab +remove_existing_cron + +# 构造 cron 行 +CRON_LINE="*/${INTERVAL} * * * * /usr/bin/python3 ${INSTALL_DIR}/collect.py --url ${URL} --host ${HOST}" +if [[ -n "$ENV_SCRIPT" ]]; then + CRON_LINE="${CRON_LINE} --env ${ENV_SCRIPT}" +fi +CRON_LINE="${CRON_LINE} --timeout ${TIMEOUT} --retries ${RETRIES} --log ${LOG_FILE} >> ${LOG_FILE}.cron 2>&1 # ${CRON_TAG}" + +# 写入新 crontab (保留旧任务) +{ crontab -l 2>/dev/null; echo "$CRON_LINE"; } | crontab - +log_info "crontab 已更新" + +# ─── 验证安装 ──────────────────────────────────────────────────────────────── +log_info "── 当前 crontab 中本任务相关行 ──" +crontab -l | grep "$CRON_TAG" || log_warn "未找到新行, 请检查" + +log_info "── 立即跑一次 dry-run 验证脚本可执行 ──" +"$INSTALL_DIR/collect.py" \ + --url "$URL" --host "$HOST" \ + ${ENV_SCRIPT:+--env "$ENV_SCRIPT"} \ + --timeout 30 --retries 1 --retry-delay 1 \ + --dry-run 2>&1 | tail -20 || log_warn "dry-run 失败 (test_server 可能在 VXE 工具链环境外, 这是正常的)" + +cat < bool: + v = os.environ.get(name, "").strip().lower() + if v in ("1", "true", "yes", "y", "on"): + return True + if v in ("0", "false", "no", "n", "off", ""): + return False + return default + + +def _int(name: str, default: int) -> int: + try: + return int(os.environ.get(name, "").strip()) + except (TypeError, ValueError): + return default + + +def get_config() -> dict: + """读取并返回当前生效的邮件配置 (用于 /api/email/status 显示)。""" + return { + "enabled": _bool("MAIL_ENABLED", False), + "smtp_host": os.environ.get("SMTP_HOST", "").strip(), + "smtp_port": _int("SMTP_PORT", 587), + "smtp_user": os.environ.get("SMTP_USER", "").strip(), + "smtp_password_set": bool(os.environ.get("SMTP_PASSWORD", "").strip()), + "smtp_use_tls": _bool("SMTP_USE_TLS", True), + "smtp_use_ssl": _bool("SMTP_USE_SSL", False), + "from_addr": os.environ.get("MAIL_FROM", "").strip(), + "from_name": os.environ.get("MAIL_FROM_NAME", "Palladium Z1 Monitor").strip(), + "to_addrs": [a.strip() for a in os.environ.get("MAIL_TO", "").split(",") if a.strip()], + "send_hour": _int("MAIL_SEND_HOUR", 9), # 每天 09:00 + "send_minute": _int("MAIL_SEND_MINUTE", 0), + "monitor_base_url": os.environ.get("MONITOR_BASE_URL", "http://127.0.0.1:5100").strip(), + } + + +def config_errors(cfg: dict) -> list[str]: + """检查配置是否完整可发送。返回缺失/错误项的列表 (空=OK)。""" + errs = [] + if not cfg["enabled"]: + return errs # 禁用时不做任何检查 + if not cfg["smtp_host"]: + errs.append("SMTP_HOST 未设置") + if not cfg["smtp_user"]: + errs.append("SMTP_USER 未设置") + if not cfg["smtp_password_set"]: + errs.append("SMTP_PASSWORD 未设置") + if not cfg["from_addr"]: + errs.append("MAIL_FROM 未设置") + if not cfg["to_addrs"]: + errs.append("MAIL_TO 未设置 (逗号分隔多个地址)") + if not (0 <= cfg["send_hour"] <= 23): + errs.append(f"MAIL_SEND_HOUR 越界: {cfg['send_hour']}") + if not (0 <= cfg["send_minute"] <= 59): + errs.append(f"MAIL_SEND_MINUTE 越界: {cfg['send_minute']}") + return errs + + +# ═══════════════════════════════════════════════════════════════════════════ +# 数据收集 +# ═══════════════════════════════════════════════════════════════════════════ +def collect_report_data() -> dict: + """ + 拉取邮件所需的所有数据。 + 返回 dict 包含: latest, boards_offline, jobs_active, stats_24h, samples_24h。 + """ + now = datetime.now(timezone.utc) + cutoff_24h = now - timedelta(hours=24) + cutoff_yesterday = now - timedelta(hours=48) + + # ── 最新快照 ── + latest = ScanSnapshot.query.order_by(ScanSnapshot.timestamp.desc()).first() + if not latest: + return { + "latest": None, + "boards_offline": [], + "jobs_active": [], + "stats_24h": None, + "samples_24h": [], + "now_utc": now, + "now_local": datetime.now(), + } + + # ── 离线板列表 ── + boards_offline = BoardStatus.query.filter_by( + snapshot_id=latest.id, status="OFFLINE" + ).order_by(BoardStatus.rack, BoardStatus.cluster, BoardStatus.ld_index).all() + + # ── 活跃作业 ── + jobs_active = JobInfo.query.filter_by(snapshot_id=latest.id).order_by(JobInfo.job_index).all() + + # ── 24h 统计 ── + snaps_24h = ScanSnapshot.query.filter( + ScanSnapshot.timestamp >= cutoff_24h + ).order_by(ScanSnapshot.timestamp.asc()).all() + + if snaps_24h: + utils = [s.utilization for s in snaps_24h] + used_counts = [s.used_boards for s in snaps_24h] + stats_24h = { + "count": len(snaps_24h), + "util_min": min(utils), + "util_max": max(utils), + "util_avg": round(sum(utils) / len(utils), 1), + "used_min": min(used_counts), + "used_max": max(used_counts), + "used_avg": round(sum(used_counts) / len(used_counts), 1), + "first_ts": snaps_24h[0].timestamp, + "last_ts": snaps_24h[-1].timestamp, + } + # 趋势样本: 最多 12 个等距点 (避免邮件太长) + if len(snaps_24h) > 12: + step = len(snaps_24h) // 12 + samples = snaps_24h[::step] + if samples[-1] != snaps_24h[-1]: + samples.append(snaps_24h[-1]) + else: + samples = snaps_24h + samples_24h = samples + else: + stats_24h = None + samples_24h = [] + + return { + "latest": latest, + "boards_offline": boards_offline, + "jobs_active": jobs_active, + "stats_24h": stats_24h, + "samples_24h": samples_24h, + "now_utc": now, + "now_local": datetime.now(), + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# 模板渲染 +# ═══════════════════════════════════════════════════════════════════════════ +def _fmt_ts(ts, fmt: str = "%Y-%m-%d %H:%M:%S") -> str: + if ts is None: + return "—" + if ts.tzinfo is None: + ts = ts.replace(tzinfo=timezone.utc) + return ts.astimezone().strftime(fmt) + + +def _util_color_class(util: float) -> str: + """根据利用率返回 CSS class 名 (用于 HTML 颜色条)。""" + if util >= 80: + return "util-high" + if util >= 50: + return "util-mid" + return "util-low" + + +def _render_text(data: dict, base_url: str) -> str: + """纯文本版本 (HTML 不支持时回退)。""" + latest = data["latest"] + if not latest: + return "Palladium Z1 Monitor — 当前无快照数据, 无法生成报告。\n" + + lines = [] + title_date = _fmt_ts(latest.timestamp, "%Y-%m-%d") + lines.append(f"Palladium Z1 每日报告 — {title_date}") + lines.append("=" * 60) + lines.append(f"生成时间: {_fmt_ts(data['now_utc'])} UTC") + lines.append(f"快照时间: {_fmt_ts(latest.timestamp)}") + lines.append("") + + # 系统信息 + lines.append("【系统信息】") + lines.append(f" 仿真器: {latest.emulator or '—'}") + lines.append(f" 硬件: {latest.hardware or '—'}") + lines.append(f" ConfigMgr:{latest.configmgr or '—'}") + lines.append(f" 系统状态: {latest.system_status or '—'}") + lines.append(f" 数据来源: {latest.source or '—'}") + lines.append("") + + # 当前指标 + lines.append("【当前指标】") + lines.append(f" 逻辑板总数: {latest.total_boards}") + lines.append(f" 在线: {latest.online_boards}") + lines.append(f" 离线: {latest.offline_boards}") + lines.append(f" 利用率: {latest.utilization:.1f}% (使用 {latest.used_boards}/{latest.total_boards})") + lines.append(f" 活跃作业: {latest.active_jobs}") + lines.append("") + + # 24h 趋势 + if data["stats_24h"]: + s = data["stats_24h"] + lines.append("【近 24 小时趋势】") + lines.append(f" 快照数: {s['count']}") + lines.append(f" 利用率: min={s['util_min']:.1f}% max={s['util_max']:.1f}% avg={s['util_avg']:.1f}%") + lines.append(f" 使用板: min={s['used_min']} max={s['used_max']} avg={s['used_avg']:.1f}") + lines.append("") + lines.append(" 趋势样本 (最多 12 个):") + for s in data["samples_24h"]: + bar = "█" * int(round(s.utilization / 5)) # 1 个 █ = 5% + lines.append(f" {_fmt_ts(s.timestamp, '%m-%d %H:%M')} {s.utilization:5.1f}% {bar}") + lines.append("") + + # 离线板 + if data["boards_offline"]: + lines.append(f"【离线板列表】 ({len(data['boards_offline'])} 块)") + for b in data["boards_offline"]: + lines.append(f" Rack {b.rack} / Cluster {b.cluster} / LD {b.ld_index} CCD={b.ccd or '?'}") + lines.append("") + else: + lines.append("【离线板列表】 无 (全部在线)") + lines.append("") + + # 活跃作业 + if data["jobs_active"]: + lines.append(f"【活跃作业】 ({len(data['jobs_active'])} 个)") + for j in data["jobs_active"]: + t_pod = j.t_pod or "—" + lines.append(f" #{j.job_index} {j.owner} {j.pid} T-Pod: {t_pod} {j.design or '—'} ({j.elap_time or '—'})") + lines.append("") + + lines.append("─" * 60) + lines.append(f"查看完整仪表板: {base_url}/") + return "\n".join(lines) + + +def _render_html(data: dict, base_url: str) -> str: + """HTML 版本 (multipart/alternative 中优先使用)。""" + latest = data["latest"] + if not latest: + return ( + '' + '

Palladium Z1 Monitor

' + '

当前无快照数据, 无法生成报告。

' + '' + ) + + title_date = _fmt_ts(latest.timestamp, "%Y-%m-%d") + util_class = _util_color_class(latest.utilization) + util_bar_width = f"{min(latest.utilization, 100):.1f}%" + + # 离线板 HTML + if data["boards_offline"]: + offline_rows = "".join( + f"Rack {b.rack}Cluster {b.cluster}" + f"LD {b.ld_index}{b.ccd or '—'}" + for b in data["boards_offline"] + ) + offline_html = f""" + + + + + + + + {offline_rows} +
RackClusterLDCCD
""" + else: + offline_html = '

✓ 无离线板, 全部在线

' + + # 作业 HTML + if data["jobs_active"]: + job_rows = "".join( + f"#{j.job_index}{j.owner}{j.pid}" + f"{j.t_pod or '—'}{j.design or '—'}{j.elap_time or '—'}" + for j in data["jobs_active"] + ) + jobs_html = f""" + + + + + + + + + + {job_rows} +
#用户PIDT-Pod设计已运行
""" + else: + jobs_html = '

无活跃作业

' + + # 24h 趋势 HTML + if data["stats_24h"]: + s = data["stats_24h"] + sample_rows = "".join( + f'{_fmt_ts(smp.timestamp, "%m-%d %H:%M")}' + f'{smp.utilization:.1f}%' + f'' + f'
' + f'
' + f'
' + for smp in data["samples_24h"] + ) + trend_html = f""" + + + + + + +
快照数{s['count']}
利用率最小/最大/平均{s['util_min']:.1f}% / {s['util_max']:.1f}% / {s['util_avg']:.1f}%
使用板数最小/最大/平均{s['used_min']} / {s['used_max']} / {s['used_avg']:.1f}
+

趋势样本

+ + + + + + + {sample_rows} +
时间利用率图示
""" + else: + trend_html = '

近 24 小时无快照数据

' + + return f""" + + + +
+ + +
+

⚡ Palladium Z1 每日报告

+
{title_date}
+
+ +
+ + +

📋 系统信息

+ + + + + + + +
仿真器{latest.emulator or '—'}
硬件{latest.hardware or '—'}
ConfigMgr{latest.configmgr or '—'}
系统状态{latest.system_status or '—'}
数据来源{latest.source or '—'}
快照时间{_fmt_ts(latest.timestamp)}
+ + +

📊 当前指标

+
+
+
逻辑板总数
+
{latest.total_boards}
+
在线 {latest.online_boards}
+
离线 {latest.offline_boards}
+
+
+
逻辑板利用率
+
{latest.utilization:.1f}%
+
+
+
+
使用 {latest.used_boards} / 空闲 {latest.online_boards - latest.used_boards}
+
+
+
活跃作业
+
{latest.active_jobs}
+
+
+ + +

📈 近 24 小时趋势

+ {trend_html} + + +

🔴 离线板

+ {offline_html} + + +

⚙️ 活跃作业

+ {jobs_html} + +
+ + +
+ 由 Palladium Z1 Monitor 自动生成 · + {_fmt_ts(data['now_utc'], '%Y-%m-%d %H:%M:%S')} UTC +
+ +
+ +""" + + +# ═══════════════════════════════════════════════════════════════════════════ +# 发送 +# ═══════════════════════════════════════════════════════════════════════════ +def render_report(data: dict, cfg: dict) -> tuple[str, str, str]: + """返回 (subject, html, text)。""" + latest = data["latest"] + if latest: + title_date = _fmt_ts(latest.timestamp, "%Y-%m-%d") + else: + title_date = _fmt_ts(data["now_utc"], "%Y-%m-%d") + util = f"{latest.utilization:.1f}%" if latest else "—" + subject = f"[Palladium Z1] 每日报告 {title_date} — 利用率 {util}" + html = _render_html(data, cfg["monitor_base_url"]) + text = _render_text(data, cfg["monitor_base_url"]) + return subject, html, text + + +def _send_smtp(cfg: dict, subject: str, html: str, text: str) -> tuple[bool, str]: + """实际 SMTP 发送, 返回 (ok, error_msg)。""" + msg = MIMEMultipart("alternative") + msg["Subject"] = subject + msg["From"] = formataddr((cfg["from_name"], cfg["from_addr"])) + msg["To"] = ", ".join(cfg["to_addrs"]) + msg["Date"] = format_datetime(datetime.now(timezone.utc)) + msg.attach(MIMEText(text, "plain", "utf-8")) + msg.attach(MIMEText(html, "html", "utf-8")) + + try: + if cfg["smtp_use_ssl"]: + # SMTP_SSL (port 465) + ctx = ssl.create_default_context() + with smtplib.SMTP_SSL(cfg["smtp_host"], cfg["smtp_port"], context=ctx, timeout=30) as s: + s.login(cfg["smtp_user"], os.environ.get("SMTP_PASSWORD", "")) + s.sendmail(cfg["from_addr"], cfg["to_addrs"], msg.as_string()) + else: + # STARTTLS (port 587) 或 明文 (port 25) + with smtplib.SMTP(cfg["smtp_host"], cfg["smtp_port"], timeout=30) as s: + s.ehlo() + if cfg["smtp_use_tls"]: + s.starttls() + s.ehlo() + s.login(cfg["smtp_user"], os.environ.get("SMTP_PASSWORD", "")) + s.sendmail(cfg["from_addr"], cfg["to_addrs"], msg.as_string()) + return (True, "") + except smtplib.SMTPAuthenticationError as e: + # Python 3.12+: smtp_error is str; older: bytes + smtp_err = e.smtp_error.decode() if isinstance(e.smtp_error, bytes) else (e.smtp_error or "") + return (False, f"SMTP 认证失败: {e.smtp_code} {smtp_err}") + except smtplib.SMTPException as e: + return (False, f"SMTP 错误: {e}") + except (socket.timeout, ConnectionError, OSError) as e: + return (False, f"网络错误: {e}") + except Exception as e: + return (False, f"{type(e).__name__}: {e}") + + +def send_report(cfg: dict | None = None) -> dict: + """ + 收集数据 → 渲染 → 发送。 + 返回: {"ok": bool, "subject": str, "recipients": [...], "error": str, "preview_text": str} + """ + cfg = cfg or get_config() + if not cfg["enabled"]: + return {"ok": False, "error": "邮件功能未启用 (MAIL_ENABLED 未设 true)", "subject": "", "recipients": []} + errs = config_errors(cfg) + if errs: + return {"ok": False, "error": "配置不完整: " + "; ".join(errs), "subject": "", "recipients": []} + + with app.app_context(): + data = collect_report_data() + subject, html, text = render_report(data, cfg) + + ok, err = _send_smtp(cfg, subject, html, text) + return { + "ok": ok, + "subject": subject, + "recipients": cfg["to_addrs"], + "error": err, + "preview_text": text[:500] + ("..." if len(text) > 500 else ""), + "bytes": len(text) + len(html), + } + + +def write_preview(output_path: str) -> str: + """把渲染好的报告写到文件 (不发送, 仅供调试)。返回写入路径。""" + cfg = get_config() + with app.app_context(): + data = collect_report_data() + subject, html, text = render_report(data, cfg) + with open(output_path, "w", encoding="utf-8") as f: + f.write(f"Subject: {subject}\n") + f.write(f"To: {', '.join(cfg['to_addrs'])}\n") + f.write(f"From: {cfg['from_name']} <{cfg['from_addr']}>\n") + f.write("\n--- TEXT VERSION ---\n") + f.write(text) + f.write("\n\n--- HTML VERSION ---\n") + f.write(html) + return output_path diff --git a/palladium-monitor/requirements.txt b/palladium-monitor/requirements.txt new file mode 100644 index 0000000..c34608a --- /dev/null +++ b/palladium-monitor/requirements.txt @@ -0,0 +1,2 @@ +Flask==3.0.3 +Flask-SQLAlchemy==3.1.1 diff --git a/palladium-monitor/static/style.css b/palladium-monitor/static/style.css new file mode 100644 index 0000000..cb53b73 --- /dev/null +++ b/palladium-monitor/static/style.css @@ -0,0 +1,701 @@ +/* ═══════════════════════════════════════════════════════════════ + Palladium Z1 Monitor — Dark Theme Dashboard + ═══════════════════════════════════════════════════════════════ */ +:root { + --bg-primary: #0d1117; + --bg-secondary: #161b22; + --bg-card: #1c2330; + --bg-card-hover: #232b3a; + --border: #30363d; + --border-light: #3a4250; + --text-primary: #e6edf3; + --text-secondary: #8b949e; + --text-muted: #6e7681; + --accent: #58a6ff; + --green: #3fb950; + --red: #f85149; + --orange: #d29922; + --blue: #58a6ff; + --purple: #bc8cff; + --gray: #6e7681; + --radius: 8px; + --shadow: 0 2px 8px rgba(0,0,0,0.3); +} + +* { margin: 0; padding: 0; box-sizing: border-box; } + +body { + font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", + Arial, "Noto Sans SC", "Microsoft YaHei", sans-serif; + background: var(--bg-primary); + color: var(--text-primary); + font-size: 14px; + line-height: 1.5; +} + +/* ─── Top Nav ─── */ +.topnav { + display: flex; + align-items: center; + justify-content: space-between; + padding: 0 24px; + height: 56px; + background: var(--bg-secondary); + border-bottom: 1px solid var(--border); + position: sticky; + top: 0; + z-index: 100; +} +.nav-brand { + display: flex; + align-items: center; + gap: 8px; +} +.nav-logo { font-size: 22px; } +.nav-title { + font-size: 17px; + font-weight: 700; + color: var(--text-primary); +} +.nav-links { + display: flex; + align-items: center; + gap: 4px; +} +.nav-links a { + padding: 8px 16px; + color: var(--text-secondary); + text-decoration: none; + border-radius: var(--radius); + transition: all 0.15s; + font-weight: 500; +} +.nav-links a:hover { color: var(--text-primary); background: var(--bg-card); } +.nav-links a.active { color: var(--accent); background: rgba(88,166,255,0.1); } +.nav-btn { + padding: 8px 16px; + background: var(--accent); + color: #fff; + border: none; + border-radius: var(--radius); + cursor: pointer; + font-weight: 600; + font-size: 13px; + transition: opacity 0.15s; +} +.nav-btn:hover { opacity: 0.85; } + +/* ─── Container ─── */ +.container { + max-width: 1600px; + margin: 0 auto; + padding: 20px 24px 60px; +} + +/* ─── System Info Bar ─── */ +.system-bar { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 24px; + padding: 12px 20px; + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + margin-bottom: 16px; +} +.sys-item { + display: flex; + flex-direction: column; + gap: 2px; +} +.sys-label { + font-size: 11px; + color: var(--text-muted); + text-transform: uppercase; + letter-spacing: 0.5px; +} +.sys-value { + font-size: 14px; + font-weight: 600; + color: var(--text-primary); + font-family: "SF Mono", "Fira Code", Consolas, monospace; +} +.refresh-controls { + margin-left: auto; + display: flex; + align-items: center; + gap: 8px; + flex-direction: row; +} +.refresh-controls label { + display: flex; + align-items: center; + gap: 4px; + font-size: 13px; + color: var(--text-secondary); + cursor: pointer; +} +.refresh-controls select { + background: var(--bg-primary); + color: var(--text-primary); + border: 1px solid var(--border); + border-radius: 4px; + padding: 3px 8px; + font-size: 12px; +} +.btn-sm { + padding: 4px 8px; + font-size: 12px; + background: var(--bg-card-hover); + color: var(--text-primary); + border: 1px solid var(--border); + border-radius: 4px; + cursor: pointer; + text-decoration: none; + transition: all 0.15s; +} +.btn-sm:hover { background: var(--border-light); } +.btn-sm.btn-danger { color: var(--red); border-color: var(--red); } +.btn-sm.btn-danger:hover { background: rgba(248,81,73,0.15); } + +/* ─── Status Badges ─── */ +.status-badge { + display: inline-block; + padding: 2px 10px; + border-radius: 12px; + font-size: 12px; + font-weight: 700; + letter-spacing: 0.3px; +} +.status-ok { background: rgba(63,185,80,0.15); color: var(--green); } +.status-warning { background: rgba(210,153,34,0.15); color: var(--orange); } +.status-error { background: rgba(248,81,73,0.15); color: var(--red); } +.status-unknown { background: rgba(110,118,129,0.15); color: var(--gray); } +.status-online { background: rgba(63,185,80,0.15); color: var(--green); } +.status-offline { background: rgba(248,81,73,0.15); color: var(--red); } + +/* ─── Metrics Grid ─── */ +.metrics-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); + gap: 12px; + margin-bottom: 20px; +} +.metric-card { + display: flex; + align-items: center; + gap: 14px; + padding: 16px; + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + transition: border-color 0.15s; +} +.metric-card:hover { border-color: var(--border-light); } +.metric-icon { font-size: 28px; } +.metric-label { + font-size: 12px; + color: var(--text-muted); + text-transform: uppercase; + letter-spacing: 0.5px; +} +.metric-value { + font-size: 28px; + font-weight: 700; + line-height: 1.2; + color: var(--text-primary); +} +.metric-value.metric-warn { color: var(--orange); } +.metric-sub { + font-size: 12px; + color: var(--text-secondary); + margin-top: 2px; +} + +/* ─── Section Headers ─── */ +.section-header { + display: flex; + align-items: center; + justify-content: space-between; + margin: 24px 0 12px; +} +.section-header h2 { + font-size: 18px; + font-weight: 700; + color: var(--text-primary); +} + +/* ─── Legend ─── */ +.legend { + display: flex; + gap: 16px; + font-size: 12px; + color: var(--text-secondary); +} +.legend-item { + display: flex; + align-items: center; + gap: 4px; +} + +/* ─── Rack Card ─── */ +.rack-card { + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + margin-bottom: 12px; + overflow: hidden; +} +.rack-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 10px 16px; + background: var(--bg-secondary); + border-bottom: 1px solid var(--border); +} +.rack-title { + font-size: 15px; + font-weight: 700; + color: var(--text-primary); +} +.rack-count { + font-size: 12px; + color: var(--text-muted); +} + +/* ─── Cluster Section ─── */ +.cluster-section { + padding: 12px 16px; + border-bottom: 1px solid var(--border); +} +.cluster-section:last-child { border-bottom: none; } +.cluster-header { + display: flex; + align-items: center; + gap: 12px; + margin-bottom: 10px; + font-size: 14px; + font-weight: 600; + color: var(--text-secondary); +} +.ccd-badge { + padding: 1px 8px; + border-radius: 10px; + font-size: 11px; + font-weight: 600; +} +.ccd-online { background: rgba(63,185,80,0.15); color: var(--green); } +.ccd-offline { background: rgba(248,81,73,0.15); color: var(--red); } + +/* ─── Boards Grid ─── */ +.boards-grid { + display: grid; + grid-template-columns: repeat(auto-fill, minmax(170px, 1fr)); + gap: 8px; +} + +/* ─── Board Card ─── */ +.board-card { + background: var(--bg-primary); + border: 1px solid var(--border); + border-radius: 6px; + padding: 8px 10px; + transition: all 0.15s; +} +.board-card:hover { border-color: var(--border-light); transform: translateY(-1px); } +.board-card.board-offline { + border-color: rgba(248,81,73,0.4); + background: rgba(248,81,73,0.05); +} +.board-top { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: 6px; +} +.board-ld { + font-size: 13px; + font-weight: 700; + color: var(--text-primary); +} +.board-status-badge { + padding: 1px 6px; + border-radius: 8px; + font-size: 10px; + font-weight: 600; +} + +/* ─── Domain Cells ─── */ +.domains-row { + display: grid; + grid-template-columns: repeat(8, 1fr); + gap: 2px; +} +.domain-cell { + text-align: center; + font-size: 10px; + font-weight: 700; + padding: 4px 0; + border-radius: 3px; + min-width: 18px; + transition: transform 0.1s; + cursor: default; + font-family: "SF Mono", "Fira Code", Consolas, monospace; +} +.domain-cell:hover { transform: scale(1.15); z-index: 5; position: relative; } +.domain-available { + background: rgba(63,185,80,0.15); + color: var(--green); + border: 1px solid rgba(63,185,80,0.3); +} +.domain-downloaded { + background: rgba(88,166,255,0.2); + color: var(--blue); + border: 1px solid rgba(88,166,255,0.4); +} +.domain-disabled { + background: rgba(248,81,73,0.15); + color: var(--red); + border: 1px solid rgba(248,81,73,0.3); +} +.domain-not_exist { + background: rgba(110,118,129,0.1); + color: var(--gray); + border: 1px solid rgba(110,118,129,0.2); +} + +/* ─── Card (generic) ─── */ +.card { + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + overflow: hidden; + margin-bottom: 12px; +} +.card.empty-state { + padding: 24px; + text-align: center; + color: var(--text-muted); +} + +/* ─── Data Table ─── */ +.data-table { + width: 100%; + border-collapse: collapse; + font-size: 13px; +} +.data-table thead { + background: var(--bg-secondary); +} +.data-table th { + padding: 10px 14px; + text-align: left; + font-weight: 600; + color: var(--text-muted); + text-transform: uppercase; + font-size: 11px; + letter-spacing: 0.5px; + border-bottom: 1px solid var(--border); + white-space: nowrap; +} +.data-table td { + padding: 8px 14px; + border-bottom: 1px solid rgba(48,54,61,0.5); + color: var(--text-primary); +} +.data-table tbody tr:hover { background: var(--bg-card-hover); } +.data-table tbody tr:last-child td { border-bottom: none; } +.text-mono { + font-family: "SF Mono", "Fira Code", Consolas, monospace; + font-size: 12px; +} +.text-green { color: var(--green); } +.text-red { color: var(--red); } +.text-blue { color: var(--blue); } +.text-gray { color: var(--gray); } + +/* ─── Pods Grid ─── */ +.pods-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(320px, 1fr)); + gap: 12px; + margin-bottom: 12px; +} +.pod-card { + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + overflow: hidden; +} +.pod-header { + padding: 8px 14px; + background: var(--bg-secondary); + border-bottom: 1px solid var(--border); + font-weight: 700; + font-size: 14px; +} +.pod-table { + width: 100%; + border-collapse: collapse; + font-size: 12px; +} +.pod-table td { + padding: 5px 14px; + border-bottom: 1px solid rgba(48,54,61,0.4); +} +.pod-table tr:last-child td { border-bottom: none; } +.pod-label { + color: var(--text-secondary); + white-space: nowrap; + width: 40%; +} +.pod-sublabel { + color: var(--text-muted); + font-size: 11px; + padding-left: 24px !important; +} +.pod-value { + font-family: "SF Mono", "Fira Code", Consolas, monospace; + color: var(--text-primary); +} +.pod-row-available td { color: var(--green); } +.pod-row-locked td { color: var(--orange); } +.pod-row-reserved td { color: var(--blue); } +.pod-row-unavailable td { color: var(--red); } + +/* ─── Utilization Bar (history table) ─── */ +.util-bar { + position: relative; + width: 120px; + height: 20px; + background: var(--bg-primary); + border-radius: 10px; + overflow: hidden; + border: 1px solid var(--border); +} +.util-fill { + height: 100%; + background: linear-gradient(90deg, var(--green), var(--blue)); + border-radius: 10px; + transition: width 0.3s; +} +.util-text { + position: absolute; + top: 0; left: 0; right: 0; bottom: 0; + display: flex; + align-items: center; + justify-content: center; + font-size: 11px; + font-weight: 700; + color: var(--text-primary); + text-shadow: 0 1px 2px rgba(0,0,0,0.5); +} + +/* ─── Pagination ─── */ +.pagination { + display: flex; + align-items: center; + justify-content: center; + gap: 16px; + margin-top: 16px; +} +.page-btn { + padding: 6px 14px; + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + color: var(--text-primary); + text-decoration: none; + font-size: 13px; + transition: all 0.15s; +} +.page-btn:hover { border-color: var(--accent); color: var(--accent); } +.page-info { + font-size: 13px; + color: var(--text-secondary); +} + +/* ─── Tags ─── */ +.tag { + display: inline-block; + padding: 1px 8px; + background: rgba(88,166,255,0.1); + color: var(--accent); + border-radius: 4px; + font-size: 11px; + font-weight: 600; +} + +/* ─── Modal ─── */ +.modal-overlay { + position: fixed; + top: 0; left: 0; right: 0; bottom: 0; + background: rgba(0,0,0,0.7); + display: flex; + align-items: center; + justify-content: center; + z-index: 200; +} +.modal { + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: 12px; + width: 90%; + max-width: 800px; + max-height: 85vh; + display: flex; + flex-direction: column; + box-shadow: 0 8px 32px rgba(0,0,0,0.5); +} +.modal-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 16px 20px; + border-bottom: 1px solid var(--border); +} +.modal-header h2 { font-size: 16px; font-weight: 700; } +.modal-close { + background: none; + border: none; + color: var(--text-secondary); + font-size: 18px; + cursor: pointer; + padding: 4px 8px; + border-radius: 4px; +} +.modal-close:hover { background: var(--bg-card-hover); color: var(--text-primary); } +.modal-body { + padding: 16px 20px; + overflow-y: auto; + flex: 1; +} +.modal-hint { + margin-bottom: 10px; + color: var(--text-secondary); + font-size: 13px; +} +.modal-hint code { + background: var(--bg-primary); + padding: 1px 6px; + border-radius: 3px; + font-size: 12px; + color: var(--accent); +} +.paste-textarea { + width: 100%; + background: var(--bg-primary); + color: var(--text-primary); + border: 1px solid var(--border); + border-radius: var(--radius); + padding: 12px; + font-family: "SF Mono", "Fira Code", Consolas, monospace; + font-size: 12px; + resize: vertical; + line-height: 1.6; +} +.paste-textarea:focus { + outline: none; + border-color: var(--accent); +} +.modal-footer { + display: flex; + justify-content: flex-end; + gap: 8px; + padding: 12px 20px; + border-top: 1px solid var(--border); +} +.btn { + padding: 8px 16px; + border: none; + border-radius: var(--radius); + cursor: pointer; + font-weight: 600; + font-size: 13px; + text-decoration: none; + transition: all 0.15s; +} +.btn-primary { background: var(--accent); color: #fff; } +.btn-primary:hover { background: #4493f8; } +.btn-secondary { background: var(--bg-card-hover); color: var(--text-primary); border: 1px solid var(--border); } +.btn-secondary:hover { border-color: var(--border-light); } +.btn-lg { padding: 12px 24px; font-size: 15px; } + +/* ─── Empty State (no data) ─── */ +.empty-state-full { + text-align: center; + padding: 80px 20px; +} +.empty-icon { font-size: 64px; margin-bottom: 16px; } +.empty-state-full h2 { font-size: 22px; margin-bottom: 8px; } +.empty-state-full p { color: var(--text-secondary); margin-bottom: 20px; } +.empty-actions { margin-bottom: 16px; } +.empty-hint { + color: var(--text-muted) !important; + font-size: 13px; +} +.empty-hint code { + background: var(--bg-card); + padding: 2px 8px; + border-radius: 4px; + color: var(--accent); +} + +/* ─── Snapshot Header ─── */ +.snapshot-header { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: 16px; +} +.snapshot-meta { + font-size: 13px; + color: var(--text-secondary); + margin-top: 4px; +} + +/* ─── Raw Output ─── */ +.raw-output-section { + margin-top: 24px; + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + padding: 12px 16px; +} +.raw-output-section summary { + cursor: pointer; + font-weight: 600; + color: var(--text-secondary); + padding: 4px 0; +} +.raw-output { + margin-top: 12px; + background: var(--bg-primary); + border: 1px solid var(--border); + border-radius: var(--radius); + padding: 12px; + font-family: "SF Mono", "Fira Code", Consolas, monospace; + font-size: 11px; + color: var(--text-secondary); + overflow-x: auto; + white-space: pre-wrap; + max-height: 500px; + overflow-y: auto; +} + +/* ─── Chart container ─── */ +.card canvas { + padding: 16px; +} + +/* ─── Responsive ─── */ +@media (max-width: 768px) { + .topnav { padding: 0 12px; } + .container { padding: 12px; } + .system-bar { gap: 12px; } + .refresh-controls { margin-left: 0; } + .boards-grid { grid-template-columns: 1fr; } + .metrics-grid { grid-template-columns: 1fr; } +} diff --git a/palladium-monitor/templates/base.html b/palladium-monitor/templates/base.html new file mode 100644 index 0000000..e45419e --- /dev/null +++ b/palladium-monitor/templates/base.html @@ -0,0 +1,61 @@ + + + + + + {% block title %}Palladium Z1 Monitor{% endblock %} + + {% block head %}{% endblock %} + + + + +
+ {% block content %}{% endblock %} +
+ + + + + + {% block scripts %}{% endblock %} + + diff --git a/palladium-monitor/templates/dashboard.html b/palladium-monitor/templates/dashboard.html new file mode 100644 index 0000000..873ecc3 --- /dev/null +++ b/palladium-monitor/templates/dashboard.html @@ -0,0 +1,394 @@ +{% extends "base.html" %} +{% block title %}仪表板 — Palladium Z1 Monitor{% endblock %} + +{% block head %} + +{% endblock %} + +{% block content %} +{% if snapshot %} + + +
+
+ 仿真器 + {{ snapshot.emulator }} +
+
+ 硬件 + {{ snapshot.hardware }} +
+
+ ConfigMgr + {{ snapshot.configmgr }} +
+
+ 系统状态 + {% set ss = snapshot.system_status.upper() %} + {% if ss == 'OK' %} + ● {{ snapshot.system_status }} + {% elif 'PARTIAL' in ss %} + ● {{ snapshot.system_status }} + {% elif 'DOWN' in ss or 'ERROR' in ss %} + ● {{ snapshot.system_status }} + {% else %} + ● {{ snapshot.system_status }} + {% endif %} +
+
+ 最后更新 + {{ snapshot.timestamp | fmt_time }} +
+
+ 数据来源 + {{ snapshot.source }} +
+
+ + + +
+
+ + +
+
+
🖥️
+
+
逻辑板总数
+
{{ stats.total_boards }}
+
+ 在线 {{ stats.online_boards }} · + 离线 {{ stats.offline_boards }} +
+
+
+
+
📊
+
+
逻辑板利用率
+
{{ stats.utilization }}%
+
+ 使用 {{ stats.used_boards }} / 空闲 {{ stats.online_boards - stats.used_boards }} +
+
+
+
+
🔲
+
+
逻辑板状态
+
{{ stats.total_boards }}
+
+ ● {{ stats.online_boards - stats.used_boards }} 空闲 + ● {{ stats.used_boards }} 使用 + ● {{ stats.offline_boards }} 离线 +
+
+
+
+
⚙️
+
+
活跃作业
+
{{ stats.active_jobs }}
+
+ {% if jobs %} + {{ jobs[0].owner }} · {{ jobs[0].elap_time }} + {% else %} + 无活跃作业 + {% endif %} +
+
+
+
+ + +
+

🏗️ 逻辑板状态

+
+ - 可用 + 1 已加载 + D 已禁用 + X 不存在 +
+
+ +{% for rack in racks %} +
+
+ Rack {{ rack.rack }} + {{ rack.clusters | length }} clusters +
+ {% for cluster in rack.clusters %} +
+
+ Cluster {{ cluster.cluster }} + {% set ccd = cluster.ccd.upper() %} + {% if ccd == 'ONLINE' %} + CCD: {{ cluster.ccd }} + {% else %} + CCD: {{ cluster.ccd }} + {% endif %} +
+
+ {% for board in cluster.boards %} +
+
+ LD {{ board.ld }} + {{ board.status }} +
+
+ {% for d in board.domains %} +
{{ d }}
+ {% endfor %} +
+
+ {% endfor %} +
+
+ {% endfor %} +
+{% endfor %} + + +
+

📋 作业信息

+
+{% if jobs %} +
+ + + + + + + + + + + + + + {% for job in jobs %} + + + + + + + + + + {% endfor %} + +
#用户PIDT-Pod设计名已运行预留Key
{{ job.job_index }}{{ job.owner }}{{ job.pid }}{{ job.t_pod }}{{ job.design }}{{ job.elap_time }}{{ job.reserved_key }}
+
+{% else %} +
当前无活跃作业
+{% endif %} + + +
+

🎯 T-Pod 可用性

+
+
+ {% for pod in pods %} +
+
Rack {{ pod.rack }}
+ + {% set all_h = pod.all_hdsb | from_json %} + {% set all_t = pod.all_t_pods | from_json %} + {% if all_h %} + + + + {% for length, val in all_h.items() %} + + + + + {% endfor %} + {% endif %} + {% if all_t %} + + + + {% for length, val in all_t.items() %} + + + + + {% endfor %} + {% endif %} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
全部 HDSB
Length {{ length }}{{ val }}
全部 T-Pods
Length {{ length }}{{ val }}
可用 HDSB{{ pod.available_hdsb }}
可用 T-Pods{{ pod.available_t_pods }}
锁定 HDSB{{ pod.locked_hdsb }}
锁定 T-Pods{{ pod.locked_t_pods }}
预留 HDSB{{ pod.reserved_hdsb }}
预留 T-Pods{{ pod.reserved_t_pods }}
不可用 HDSB{{ pod.unavailable_hdsb }}
不可用 T-Pods{{ pod.unavailable_t_pods }}
+
+ {% endfor %} +
+ + +{% if history | length > 1 %} +
+

📈 利用率趋势

+
+
+ +
+{% endif %} + +{% else %} + +
+
📊
+

暂无监控数据

+

请通过以下方式导入数据：

+
+ +
+

或在 Palladium 主机运行：python3 collect.py --url http://<监控服务器>:5100/api/collect

+
+{% endif %} +{% endblock %} + +{% block scripts %} +{% if snapshot and history | length > 1 %} + +{% endif %} + + +{% endblock %} diff --git a/palladium-monitor/templates/history.html b/palladium-monitor/templates/history.html new file mode 100644 index 0000000..01e53e7 --- /dev/null +++ b/palladium-monitor/templates/history.html @@ -0,0 +1,196 @@ +{% extends "base.html" %} +{% block title %}历史记录 — Palladium Z1 Monitor{% endblock %} + +{% block content %} +
+

📅 历史快照

+
+ + +
+
+ +{% if pagination.items %} +
+ + + + + + + + + + + + + + + + + {% for snap in pagination.items %} + + + + + + + + + + + + + {% endfor %} + +
ID时间仿真器系统状态逻辑板逻辑板利用率使用/总数作业数来源操作
{{ snap.id }}{{ snap.timestamp | fmt_time }}{{ snap.emulator }} + {% set ss = snap.system_status.upper() %} + {% if ss == 'OK' %} + {{ snap.system_status }} + {% elif 'PARTIAL' in ss %} + {{ snap.system_status }} + {% elif 'DOWN' in ss or 'ERROR' in ss %} + {{ snap.system_status }} + {% else %} + {{ snap.system_status }} + {% endif %} + + {{ snap.total_boards }} + ({{ snap.online_boards }}/{{ snap.offline_boards }}) + +
+
+ {{ snap.utilization }}% +
+ 		{{ snap.used_boards }} / {{ snap.total_boards }}{{ snap.active_jobs }}{{ snap.source }} + 查看 +
+ +
+
+
+ + + + +{% else %} +
暂无历史数据
+{% endif %} + + + + + + +{% endblock %} diff --git a/palladium-monitor/templates/snapshot.html b/palladium-monitor/templates/snapshot.html new file mode 100644 index 0000000..976b0f7 --- /dev/null +++ b/palladium-monitor/templates/snapshot.html @@ -0,0 +1,183 @@ +{% extends "base.html" %} +{% block title %}快照 #{{ snapshot.id }} — Palladium Z1 Monitor{% endblock %} + +{% block content %} +
+
+

📸 快照 #{{ snapshot.id }}

+
+ {{ snapshot.timestamp | fmt_time }} · 来源: {{ snapshot.source }} · + {{ snapshot.emulator }} / {{ snapshot.hardware }} +
+
+ ← 返回仪表板 +
+ + +
+
+ 仿真器 + {{ snapshot.emulator }} +
+
+ 硬件 + {{ snapshot.hardware }} +
+
+ ConfigMgr + {{ snapshot.configmgr }} +
+
+ 系统状态 + {% set ss = snapshot.system_status.upper() %} + {% if ss == 'OK' %} + {{ snapshot.system_status }} + {% elif 'PARTIAL' in ss %} + {{ snapshot.system_status }} + {% elif 'DOWN' in ss or 'ERROR' in ss %} + {{ snapshot.system_status }} + {% else %} + {{ snapshot.system_status }} + {% endif %} +
+
+ + +
+
+
🖥️
+
+
逻辑板
+
{{ stats.total_boards }}
+
+ 在线 {{ stats.online_boards }} · + 离线 {{ stats.offline_boards }} +
+
+
+
+
📊
+
+
逻辑板利用率
+
{{ stats.utilization }}%
+
使用 {{ stats.used_boards }} / 空闲 {{ stats.online_boards - stats.used_boards }}
+
+
+
+
⚙️
+
+
活跃作业
+
{{ stats.active_jobs }}
+
+
+
+ + +
+

🏗️ 逻辑板状态

+
+ - 可用 + 1 已加载 + D 已禁用 + X 不存在 +
+
+ +{% for rack in racks %} +
+
+ Rack {{ rack.rack }} +
+ {% for cluster in rack.clusters %} +
+
+ Cluster {{ cluster.cluster }} + {% set ccd = cluster.ccd.upper() %} + {% if ccd == 'ONLINE' %} + CCD: {{ cluster.ccd }} + {% else %} + CCD: {{ cluster.ccd }} + {% endif %} +
+
+ {% for board in cluster.boards %} +
+
+ LD {{ board.ld }} + {{ board.status }} +
+
+ {% for d in board.domains %} +
{{ d }}
+ {% endfor %} +
+
+ {% endfor %} +
+
+ {% endfor %} +
+{% endfor %} + + +

📋 作业信息

+{% if jobs %} +
+ + + + + + {% for job in jobs %} + + + + + + + + + + {% endfor %} + +
#用户PIDT-Pod设计名已运行预留Key
{{ job.job_index }}{{ job.owner }}{{ job.pid }}{{ job.t_pod }}{{ job.design }}{{ job.elap_time }}{{ job.reserved_key }}
+
+{% else %} +
该快照无活跃作业
+{% endif %} + + +

🎯 T-Pod 可用性

+
+ {% for pod in pods %} +
+
Rack {{ pod.rack }}
+ + {% set all_h = pod.all_hdsb | from_json %} + {% set all_t = pod.all_t_pods | from_json %} + {% for length, val in all_h.items() %} + + {% endfor %} + {% for length, val in all_t.items() %} + + {% endfor %} + + + + + + + + +
HDSB L{{ length }}{{ val }}
T-Pods L{{ length }}{{ val }}
可用 HDSB{{ pod.available_hdsb }}
可用 T-Pods{{ pod.available_t_pods }}
锁定 HDSB{{ pod.locked_hdsb }}
锁定 T-Pods{{ pod.locked_t_pods }}
预留 HDSB{{ pod.reserved_hdsb }}
预留 T-Pods{{ pod.reserved_t_pods }}
不可用 HDSB{{ pod.unavailable_hdsb }}
不可用 T-Pods{{ pod.unavailable_t_pods }}
+
+ {% endfor %} +
+ + +
+ 📄 原始 test_server 输出 + 
{{ snapshot.raw_output }}
+
+{% endblock %}