commit 4a1d949e412d1532d307286521d2125d84919824 Author: Hermes Agent Date: Wed Jul 15 10:19:21 2026 +0800 feat: 完整 Squid Web Manager 平台 - P0-P3 全功能 完整 Web 管理平台,涵盖: P0 生产加固: - 会话超时(空闲+绝对) + CSRF 防护 + 登录失败限流 - HTTPS/TLS 证书管理(自签生成/上传/删除/过期提醒) - SSL Bump (HTTPS 透明代理) 可视化配置 - 配置文件 diff 预览 + 二次确认才写入 - 服务控制二次确认 + 操作期间按钮锁定 P1 运维能力: - 日志轮转管理(logrotate) + 日志状态监控 - 告警规则(5xx/命中率/磁盘/客户端流量) - 多 Squid 实例管理 + 一键切换 - squidclient mgr 实时性能指标 - 流量异常检测(突增/大文件/高频小包/新客户端) P2 日志分析增强: - 日志持久化到 SQLite + 历史时间范围查询 - 导出 CSV/JSON(日志/KPI/异常/审计) - GeoIP 地图(Leaflet + ip-api.com 离线可选 GeoLite2) - 每客户端 24h 趋势图 - 自定义 Squid logformat 解析器 P3 体验锦上添花: - 暗/亮主题切换(cookie 持久化) - WebSSH 终端(xterm.js + paramiko) - 完整审计日志 + 用户管理 技术栈: Flask 3.0 + SQLAlchemy 2.0 + SQLite + Chart.js + Leaflet 总代码量: ~16500 行 (15 Python 模块 + 34 模板 + CSS) 路由数: 73 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2ddf137 --- /dev/null +++ b/.gitignore @@ -0,0 +1,57 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +*.egg +*.egg-info/ +dist/ +build/ +.eggs/ +.installed.cfg +*.egg + +# Virtual envs +venv/ +env/ +.venv/ +ENV/ + +# pytest / coverage +.pytest_cache/ +.coverage +htmlcov/ +.tox/ + +# IDE +.idea/ +.vscode/ +*.swp +*.swo +.DS_Store + +# Logs +*.log + +# Local instance (sqlite db, uploaded files, generated certs) +instance/ +uploads/ +backups/ +ssl_certs/ +alerts/ +instances/ +log_storage/ +security/ + +# Sample artifacts +*.pid +*.sock +*.pid.lock + +# Local config overrides +.env +.env.local + +# OS +Thumbs.db diff --git a/README.md b/README.md new file mode 100644 index 0000000..56179af --- /dev/null +++ b/README.md @@ -0,0 +1,284 @@ +# Squid Web Manager + +一个基于 Flask 的 Squid 代理服务器 Web 管理平台。 + +## 功能 + +### 1. 全 Web 化配置管理 +- **主配置**: HTTP/HTTPS/ICP/SNMP 端口、缓存内存、对象大小、超时、DNS、日志路径、管理员邮箱等 +- **ACL 规则表**: 完整 ACL CRUD，支持 28+ 类型 (src, dst, dstdomain, port, method, time, url_regex, proxy_auth, ...) +- **访问控制**: http_access / https_access / icp_access / snmp_access 等所有访问指令的可视化管理 +- **缓存目录**: cache_dir 多目录管理 (ufs / aufs / diskd / rock) +- **刷新规则**: refresh_pattern 正则/百分比/最大时间 +- **认证配置**: basic / digest / ntlm / negotiate 全套 auth_param 管理 +- **缓存对等**: cache_peer 父代理/兄弟节点配置 +- **原始编辑**: 直接编辑 squid.conf，支持语法校验 + +每次保存自动备份当前配置，可选启用 `squid -k parse` 语法校验。 + +### 2. 日志分析 +解析 Squid 原生 access.log，提供： + +- **KPI 卡片**: 总请求数、缓存命中率、总流量、平均响应、拒绝率、中断率、4xx/5xx 错误率 +- **每小时流量图**: 请求数 + 流量双 Y 轴折线图 +- **结果码分布**: TCP_HIT / TCP_MISS / TCP_TUNNEL / TCP_DENIED 等 (含中文释义) +- **HTTP 状态码分布**: 2xx/3xx/4xx/5xx 分类 +- **方法分布**: GET / POST / CONNECT / ... +- **请求分类**: Hit / Miss / Tunnel / Denied / Aborted / Error +- **Top 客户端**: 按 IP 排序，显示请求数和流量 +- **Top 目标主机**: 按域名排序 +- **Top URL**: 访问最多的资源 +- **层级代码**: HIER_DIRECT / HIER_PARENT_HIT / ... +- **Content-Type 分布** +- **日志查询**: 多维度过滤 (IP / 方法 / 结果码 / 主机 / URL / 大小范围 / 时间范围) +- **实时日志**: 自动刷新的 Live Tail 视图 +- **日志导入**: 离线分析一次性粘贴的日志 + +### 3. 服务控制 +- 启动 / 停止 / 重启 / 平滑重载 (reload，失败回退 restart) +- 服务状态查看 (PID, 版本, 启动时间, unit 文件) +- 关键路径一览 + +### 4. 运维功能 +- **配置备份**: 自动备份历史，可下载/恢复 +- **审计日志**: 所有配置变更、服务操作、登录登出全程留痕 +- **修改密码**: admin/admin 默认账号，强制可改 + +## 技术栈 + +| 组件 | 选型 | +|------|------| +| 后端 | Python 3 + Flask + SQLAlchemy | +| 数据库 | SQLite (instance/squid_mgr.db) | +| 前端 | 原生 HTML + CSS + 原生 JS | +| 图表 | Chart.js 4.4 (CDN) | +| WSGI | gunicorn | +| 进程管理 | systemd (可选) | + +## 快速开始 + +### 安装依赖 +```bash +cd /root/squid-manager +pip install -r requirements.txt --break-system-packages +``` + +### 直接运行 (开发) +```bash +python3 app.py +# 默认监听 0.0.0.0:5200，调试模式 +``` + +### 生产部署 (gunicorn + systemd) +```bash +# 初始化数据库 +python3 -c "from app import app, db, seed_admin; \ + app.app_context().push(); \ + db.create_all(); seed_admin()" + +# 启动 gunicorn +gunicorn --workers 2 --bind 0.0.0.0:5200 wsgi:app +``` + +### systemd 单元 (推荐) +`/etc/systemd/system/squid-manager.service`: +```ini +[Unit] +Description=Squid Web Manager +After=network.target + +[Service] +Type=exec +User=root +WorkingDirectory=/root/squid-manager +ExecStart=/usr/bin/gunicorn --workers 2 --bind 0.0.0.0:5200 wsgi:app +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +```bash +systemctl daemon-reload +systemctl enable --now squid-manager +``` + +## 默认账号 + +- 用户名: `admin` +- 密码: `admin` + +**首次登录后请立即修改密码** (左侧导航 → 修改密码)。 + +## 配置路径 + +默认假设 Squid 安装在本机: +- 配置文件: `/etc/squid/squid.conf` +- 访问日志: `/var/log/squid/access.log` +- 缓存日志: `/var/log/squid/cache.log` +- 二进制: `squid` +- 服务名: `squid` + +如需管理远程 Squid 或路径不同，在 +**路径设置** 页面修改 (路径需在本平台可访问的位置)。 + +## 项目结构 + +``` +squid-manager/ +├── app.py # Flask 应用 (模型、路由、业务逻辑) +├── squid_config.py # squid.conf 解析 / 生成 / 校验 +├── log_parser.py # access.log 解析与统计 +├── wsgi.py # gunicorn 入口 +├── requirements.txt +├── README.md +├── instance/ # SQLite 数据库 (自动创建) +│ └── squid_mgr.db +├── backups/ # squid.conf 自动备份 +├── templates/ # Jinja2 模板 +│ ├── base.html +│ ├── login.html +│ ├── dashboard.html +│ ├── logs.html +│ ├── logs_import.html +│ ├── logs_live.html +│ ├── service.html +│ ├── config_main.html +│ ├── config_acls.html +│ ├── config_rules.html +│ ├── config_cache.html +│ ├── config_refresh.html +│ ├── config_auth.html +│ ├── config_peers.html +│ ├── config_raw.html +│ ├── config_paths.html +│ ├── audit.html +│ ├── backups.html +│ ├── change_password.html +│ └── error.html +└── static/ + └── style.css +``` + +## 日志格式支持 + +标准 Squid access.log 格式 (默认): +``` +time elapsed client action/code size method URL ident hierarchy content-type +``` + +示例: +``` +1783841223.438 2488 172.16.12.232 TCP_MISS_ABORTED/000 0 GET http://169.254.169.254/metadata/instance/compute - HIER_DIRECT/169.254.169.254 - +1783841225.683 4564 172.16.237.9 TCP_TUNNEL/200 10197 CONNECT default.exp-tas.com:443 - HIER_DIRECT/13.107.5.93 - +``` + +支持的结果码: +- TCP_HIT, TCP_MEM_HIT, TCP_NEGATIVE_HIT, TCP_IMS_HIT, TCP_REFRESH_HIT (命中) +- TCP_MISS, TCP_REFRESH_MISS, TCP_CLIENT_REFRESH_MISS (未命中) +- TCP_TUNNEL (CONNECT 隧道) +- TCP_DENIED, TCP_DENIED_REPLY (拒绝) +- TCP_MISS_ABORTED (中断) +- TCP_REDIRECT (重定向) +- UDP_* (ICP 协议) +- NONE (错误) + +## 安全建议 + +1. **修改默认密码** - admin/admin 仅为示例 +2. **绑定内网 IP** - 不要暴露到公网，或前置 nginx + Basic Auth +3. **限制文件权限** - 配置文件含敏感信息，确保 600 +4. **审计日志** - 定期查看 `/audit` 页面 +5. **TLS** - 生产环境建议 nginx 反代 + HTTPS + +## 已知限制 + +- 服务控制 (start/stop/reload) 仅在 Squid 与本平台同机时有效 +- `squid -k parse` 校验需要本机有 squid 二进制 +- 多 gunicorn worker 下 SQLite 写入有竞态,已用 try/except 兜底 +- access.log 解析按需读取 (默认末尾 10000 行),不持久化 + +## TLS / HTTPS 部署指南 + +本 Web 平台本身默认监听 HTTP。在公网或需要 HTTPS 的内网场景下,**强烈不要把本平台直接暴露** +请用 `nginx` 做反向代理 + TLS 终结。 + +Web UI 提供 **🔐 TLS 证书管理** 页面 (`/config/tls`): + +### 1. 自签名证书 (仅用于测试 / 内网) + +> 进入 **配置管理 → TLS 证书**,填入 Common Name (如 `proxy.example.local`),点击"生成自签名证书"。 +> 生成的文件保存在 `ssl_certs/` 目录下,名为 `CN_YYYYMMDD_xxxxxxxx.pem`, +> 内含 cert + key,可直接用于 nginx 自测。 +> +> 浏览器会提示"不受信任",仅用于内网 / 自测 / `curl -k`。 + +### 2. 生产部署: nginx 反代 + Let's Encrypt + 强制 HTTPS + +最小化的 `nginx` 配置示例 (假设本平台监听 `127.0.0.1:5200`): + +```nginx +# HTTP -> HTTPS 强制重定向 +server { + listen 80; + server_name proxy.example.com; + return 301 https://$host$request_uri; +} + +# HTTPS 终结 +server { + listen 443 ssl http2; + server_name proxy.example.com; + + # Let's Encrypt 签发的证书 + ssl_certificate /etc/letsencrypt/live/proxy.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/proxy.example.com/privkey.pem; + + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 1d; + + # 转发到本平台 + location / { + proxy_pass http://127.0.0.1:5200; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_buffering off; + } +} +``` + +申请 Let's Encrypt 证书: + +```bash +# 一次性安装 certbot (Debian/Ubuntu) +apt install -y certbot +# 仅申请证书,不需要 nginx 插件 (webroot 模式) +certbot certonly --webroot -w /var/www/html \ + -d proxy.example.com \ + --email admin@example.com --agree-tos -n +# 证书自动存放在 /etc/letsencrypt/live/proxy.example.com/ +# 推荐加上 cron 自动续期 +echo "0 3 * * * /usr/bin/certbot renew --quiet --post-hook 'systemctl reload nginx'" \ + | sudo crontab - +``` + +也可以从本平台 **上传** 页面手动上传 `.pem` / `.crt` / `.key` (上限 1 MB): + +| 字段 | 用法 | +|------|------| +| 主题 (CN) | 证书绑定的域名,需与 nginx `server_name` 一致 | +| 状态 | 绿色=有效 ≥30 天 / 黄色=<30 天 / 红色=已过期 | +| SHA-256 指纹 | 审计追踪用,只记录指纹,**绝不写私钥内容** | + +### 3. 常见误区 + +- 不要把 `ssl_certs/*.pem` 加入 git 或公网备份,文件含**私钥** +- 不要把 80 / 443 直接绑到 `app.py` 运行端口,务必过 nginx (WAF / 限速 / 防护) +- Let's Encrypt 单证书有效期 90 天,务必配置自动续期 +- 反代场景下,`app.py` 的 `SECRET_KEY` 必须固定 (环境变量),否则每次重启会话失效 + diff --git a/alerts.py b/alerts.py new file mode 100644 index 0000000..07abe2c --- /dev/null +++ b/alerts.py @@ -0,0 +1,435 @@ +"""alerts.py - Alert rule engine for Squid Web Manager (P1-2). + +Provides: +- AlertRule dataclass - configurable metric threshold rule +- evaluate_rules(entries, rules, now=None) - run all enabled rules against + the current access.log stats and return a list of triggered alerts. +- save_rules / load_rules - persist rule list as JSON +- default_rules() - opinionated starting set that covers the common SRE cases + +All metric computation uses only Python stdlib + existing log_parser statistics +- no new dependencies. Audit writes use a lazy import of `app` so this module +can be imported without forcing a circular dependency on app.py at startup. +""" +from __future__ import annotations + +import json +import os +import shutil +import time +from dataclasses import asdict, dataclass, field +from typing import Any, Iterable + +# --- Supported metrics ---------------------------------------------------- +# Each entry documents the operator semantics. Values are floats unless noted. +METRICS: dict[str, str] = { + "5xx_rate": "HTTP 5xx 错误占比 (0.0 - 1.0)", + "4xx_rate": "HTTP 4xx 错误占比 (0.0 - 1.0)", + "hit_ratio": "缓存命中率 (0.0 - 1.0)", + "denied_rate": "ACL 拒绝请求占比 (0.0 - 1.0)", + "disk_usage_pct": "缓存目录磁盘使用率 (0.0 - 100.0)", + "client_bytes_per_min": "Top 客户端每分钟流量 (字节/分钟)", +} + +OPERATORS: dict[str, str] = { + "gt": ">", + "lt": "<", + "gte": ">=", + "lte": "<=", +} + +SEVERITIES = ("info", "warning", "critical") + + +# --- Rule dataclass ------------------------------------------------------- + +@dataclass +class AlertRule: + """One alert rule. Persisted as JSON via save_rules / load_rules.""" + + name: str + metric: str + operator: str + threshold: float + window_minutes: int = 5 + enabled: bool = True + cooldown_minutes: int = 30 + last_triggered: float = 0.0 + severity: str = "warning" + notify_webhook: str = "" + + def to_dict(self) -> dict: + d = asdict(self) + # JSON doesn't love bare floats like 0.0 - keep it simple. + return d + + @classmethod + def from_dict(cls, d: dict) -> "AlertRule": + """Build from a dict (e.g. one loaded from JSON). Missing keys fall + back to dataclass defaults so old config files keep working.""" + if not isinstance(d, dict): + raise ValueError("AlertRule.from_dict expects a dict") + # Only forward the keys we know about; ignore extras. + allowed = {f for f in cls.__dataclass_fields__.keys()} # type: ignore[attr-defined] + kwargs = {k: d[k] for k in d.keys() & allowed} + # Coerce numerics - if the file is hand-edited we don't want to crash. + for key in ("threshold", "last_triggered"): + if key in kwargs: + try: + kwargs[key] = float(kwargs[key]) + except (TypeError, ValueError): + kwargs[key] = 0.0 + for key in ("window_minutes", "cooldown_minutes"): + if key in kwargs: + try: + kwargs[key] = int(kwargs[key]) + except (TypeError, ValueError): + kwargs[key] = 5 if key == "window_minutes" else 30 + kwargs["enabled"] = bool(kwargs.get("enabled", True)) + # Severity / operator / metric - fall back if hand-edited to junk. + sev = kwargs.get("severity", "warning") + if sev not in SEVERITIES: + sev = "warning" + kwargs["severity"] = sev + op = kwargs.get("operator", "gt") + if op not in OPERATORS: + op = "gt" + kwargs["operator"] = op + return cls(**kwargs) + + +# --- Metric evaluation --------------------------------------------------- + +def _apply_operator(value: float, op: str, threshold: float) -> bool: + if op == "gt": + return value > threshold + if op == "lt": + return value < threshold + if op == "gte": + return value >= threshold + if op == "lte": + return value <= threshold + return False + + +def _compute_metric( + metric: str, + stats: dict, + cache_dir: str | None, + window_minutes: int, +) -> float: + """Compute the current value of a metric from pre-aggregated stats. + + For most metrics this is just `stats[metric]`. For client_bytes_per_min we + fold in the time window (which is configurable on the rule, so we can't + pre-bake it into log_parser). disk_usage_pct needs shutil.disk_usage. + """ + if metric == "5xx_rate": + return float(stats.get("errors_5xx_rate") or 0.0) + if metric == "4xx_rate": + return float(stats.get("errors_4xx_rate") or 0.0) + if metric == "hit_ratio": + return float(stats.get("hit_ratio") or 0.0) + if metric == "denied_rate": + return float(stats.get("denied_rate") or 0.0) + if metric == "disk_usage_pct": + target = cache_dir or "/var/spool/squid" + try: + total, used, free = shutil.disk_usage(target) + except (FileNotFoundError, PermissionError, OSError): + return 0.0 + if not total: + return 0.0 + return round(used * 100.0 / total, 2) + if metric == "client_bytes_per_min": + tops = stats.get("top_clients") or [] + if not tops: + return 0.0 + max_bytes = max((c.get("bytes") or 0) for c in tops) + # Approximate bytes-per-minute for the top client over the rule window. + # If the time span is < window_minutes we still divide by window so + # the operator gets a comparable value (long window = low rate). + w = max(1, int(window_minutes)) + return float(max_bytes) * 60.0 / w + return 0.0 + + +def _humanise_metric(metric: str, value: float) -> str: + if metric.endswith("_rate"): + return f"{value * 100:.2f}%" + if metric == "hit_ratio": + return f"{value * 100:.1f}%" + if metric == "disk_usage_pct": + return f"{value:.1f}%" + if metric == "client_bytes_per_min": + # Display in KB/MB/GB - keep parity with log_parser.format_bytes shape. + n = float(value) + for unit in ("B/s", "KB/s", "MB/s", "GB/s"): + if abs(n) < 1024.0: + if unit == "B/s": + return f"{n:.0f} {unit}" + return f"{n:.1f} {unit}" + n /= 1024.0 + return f"{n:.2f} TB/s" + return f"{value:.4f}" + + +def _format_message(rule: AlertRule, value: float, window_minutes: int) -> str: + op = OPERATORS.get(rule.operator, rule.operator) + human = _humanise_metric(rule.metric, value) + metric_desc = METRICS.get(rule.metric, rule.metric) + return ( + f"{rule.name}: {metric_desc} 当前值 {human} {op} 阈值 " + f"{_humanise_metric(rule.metric, rule.threshold)} " + f"(窗口 {window_minutes}m, 严重度 {rule.severity})" + ) + + +# --- Audit (lazy import to avoid circular reference) --------------------- + +def _audit_trigger(rule: AlertRule, value: float, message: str): + """Try to write a row into the audit_log. Silently swallow on failure + so a missing app context never breaks evaluation.""" + try: + from app import audit, db # type: ignore + audit( + "alert_triggered", + f"name={rule.name} metric={rule.metric} " + f"value={value:.6f} threshold={rule.threshold} " + f"severity={rule.severity} msg={message[:300]}", + ) + db.session.commit() + except Exception: + # Never let an audit failure mask a real alert. + try: + from app import db # type: ignore + db.session.rollback() + except Exception: + pass + + +def _post_webhook(rule: AlertRule, payload: dict): + """Fire-and-forget webhook POST. We import urllib lazily; if the URL is + bad or the import fails we just log silently - alerts mustn't crash + the calling request.""" + url = (rule.notify_webhook or "").strip() + if not url: + return + try: + import urllib.request + import urllib.error + body = json.dumps(payload).encode("utf-8") + req = urllib.request.Request( + url, + data=body, + headers={"Content-Type": "application/json"}, + method="POST", + ) + urllib.request.urlopen(req, timeout=5).read() + except Exception: + # We deliberately don't surface webhook errors - delivery is + # best-effort and the audit row is the source of truth. + pass + + +# --- Main entry point ---------------------------------------------------- + +def evaluate_rules( + entries: list | None, + rules: Iterable[AlertRule], + *, + cache_dir: str | None = None, + now: float | None = None, +) -> list[dict]: + """Evaluate all enabled rules against the current access.log parsing. + + Parameters + ---------- + entries : list of log_parser.LogEntry (may be empty / None). + rules : iterable of AlertRule to check. + cache_dir : path to the Squid cache dir. Used for disk_usage_pct. If + None, defaults to /var/spool/squid. + now : epoch seconds to treat as "now". Defaults to time.time(). + + Returns + ------- + list of dict: + {rule, value, severity, message, ts, triggered} + Each dict contains: + - rule : the AlertRule instance that matched + - value : the float value that triggered the rule + - severity : copy of rule.severity + - message : human-readable summary + - ts : epoch seconds when this was evaluated + - triggered : bool - True if outside cooldown, False if suppressed + The list always contains one entry per *evaluated* rule; rows with + triggered=False are useful for the UI to show "currently OK / cooldown". + """ + ts = float(now if now is not None else time.time()) + stats = _safe_stats(entries) + out: list[dict] = [] + rules_list = list(rules) + for rule in rules_list: + if not getattr(rule, "enabled", True): + continue + metric = getattr(rule, "metric", "") + op = getattr(rule, "operator", "gt") + threshold = float(getattr(rule, "threshold", 0.0) or 0.0) + window = max(1, int(getattr(rule, "window_minutes", 5) or 5)) + try: + value = _compute_metric(metric, stats, cache_dir, window) + except Exception: + value = 0.0 + triggered = _apply_operator(value, op, threshold) + in_cooldown = False + if triggered: + last = float(getattr(rule, "last_triggered", 0.0) or 0.0) + cooldown = max(0, int(getattr(rule, "cooldown_minutes", 30) or 30)) + if last and (ts - last) < cooldown * 60: + # Suppress but still report so the UI can show "cooling down". + triggered = False + in_cooldown = True + else: + rule.last_triggered = ts + msg = _format_message(rule, value, window) + if triggered: + _audit_trigger(rule, value, msg) + _post_webhook( + rule, + { + "name": rule.name, + "metric": rule.metric, + "value": value, + "threshold": threshold, + "operator": op, + "severity": rule.severity, + "message": msg, + "ts": ts, + }, + ) + out.append({ + "rule": rule, + "value": value, + "threshold": threshold, + "operator": op, + "severity": rule.severity, + "message": msg, + "ts": ts, + "triggered": triggered, + "in_cooldown": in_cooldown, + }) + return out + + +def _safe_stats(entries: list | None) -> dict: + """Run log_parser.aggregate_stats if available, otherwise return {}. + Import log_parser lazily so unit tests / minimal contexts don't crash.""" + if not entries: + return {} + try: + import log_parser # type: ignore + return log_parser.aggregate_stats(entries) or {} + except Exception: + return {} + + +# --- Persistence -------------------------------------------------------- + +def save_rules(rules: list[AlertRule], path: str) -> None: + """Write the rule list to `path` as pretty-printed JSON.""" + payload = [r.to_dict() for r in (rules or [])] + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, ensure_ascii=False, indent=2) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def load_rules(path: str) -> list[AlertRule]: + """Load rules from a JSON file. Missing / corrupt files yield the + `default_rules()` set; that way the UI always has something to show.""" + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except FileNotFoundError: + return default_rules() + except (json.JSONDecodeError, OSError, ValueError): + return default_rules() + if not isinstance(data, list): + return default_rules() + out: list[AlertRule] = [] + for item in data: + try: + out.append(AlertRule.from_dict(item)) + except Exception: + # Skip individual bad rows instead of nuking the whole file. + continue + return out or default_rules() + + +def default_rules() -> list[AlertRule]: + """A balanced starter set: catches the common SRE cases without being + noisy. Operators can tune thresholds or disable rules from the UI.""" + return [ + AlertRule( + name="5xx 错误率突增", + metric="5xx_rate", + operator="gt", + threshold=0.05, # > 5% 5xx + window_minutes=5, + cooldown_minutes=30, + severity="critical", + ), + AlertRule( + name="缓存命中率下跌", + metric="hit_ratio", + operator="lt", + threshold=0.20, # < 20% + window_minutes=5, + cooldown_minutes=60, + severity="warning", + ), + AlertRule( + name="磁盘空间不足", + metric="disk_usage_pct", + operator="gte", + threshold=90.0, # >= 90% + window_minutes=5, + cooldown_minutes=60, + severity="critical", + ), + AlertRule( + name="客户端流量异常", + metric="client_bytes_per_min", + operator="gt", + threshold=200 * 1024 * 1024, # > 200 MB/min + window_minutes=5, + cooldown_minutes=15, + severity="warning", + ), + AlertRule( + name="拒绝率过高", + metric="denied_rate", + operator="gt", + threshold=0.50, # > 50% + window_minutes=5, + cooldown_minutes=30, + severity="warning", + ), + ] + + +# --- CLI smoke test ------------------------------------------------------ + +if __name__ == "__main__": # pragma: no cover - manual sanity check + import log_parser as _lp + sample = os.path.join(os.path.dirname(__file__), "sample_access.log") + text = open(sample, "r", encoding="utf-8").read() if os.path.exists(sample) else "" + entries = _lp.parse_lines(text) if text else [] + triggered = evaluate_rules(entries, default_rules()) + print(f"triggered: {len(triggered)}") + for t in triggered: + marker = "*" if t["triggered"] else " " + print(f" {marker} {t['rule'].name}: value={t['value']:.4f} threshold={t['threshold']} ({t['message']})") diff --git a/anomaly.py b/anomaly.py new file mode 100644 index 0000000..ad49dc6 --- /dev/null +++ b/anomaly.py @@ -0,0 +1,552 @@ +"""P1-5: 流量异常检测 (traffic anomaly detection). + +Reads ``list[LogEntry]`` from :mod:`log_parser` and flags: + +1. **Sudden traffic spikes** per client + Compare a baseline window (first ``BASELINE_RATIO`` of entries) to the + current window (last ``CURRENT_RATIO`` of entries) and flag clients whose + request-per-minute rate grew more than ``SPIKE_WARN_RATIO`` (default 3x). +2. **Abnormally large responses** + Single responses larger than ``size_threshold_mb`` (default 100 MB). +3. **High-frequency, small payload clients** + Clients that emit lots of tiny responses (< ``size_threshold_kb``). +4. **First-time-seen clients** + Clients whose first request happens in the current window. + +All helpers defensively swallow exceptions and return safe empty values so a +bad log line never breaks the dashboard. +""" +from __future__ import annotations + +import statistics +from collections import Counter, defaultdict +from datetime import datetime +from typing import Iterable + +# --- Default thresholds (overridable via kwargs) ---------------------------- + +#: Fraction of entries used to compute the baseline (the "past") +BASELINE_RATIO = 0.8 +#: Fraction of entries used to compute the current state (the "now") +CURRENT_RATIO = 0.2 +#: Spike ratio above which a client is flagged as anomaly (warning) +SPIKE_WARN_RATIO = 3.0 +#: Spike ratio above which a client is flagged as critical +SPIKE_CRIT_RATIO = 10.0 +#: Minimum samples required for baseline; below this the result is unreliable +MIN_SAMPLES = 5 + +# Worst case fallback so we don't return monstrous lists for giant logs +DEFAULT_LARGE_LIMIT = 100 +DEFAULT_SMALL_LIMIT = 50 +DEFAULT_NEW_LIMIT = 50 + + +# --- Internal helpers -------------------------------------------------------- + +def _safe_float(value, default: float = 0.0) -> float: + """Return ``value`` as float, or ``default`` on conversion failure.""" + try: + return float(value) + except (TypeError, ValueError): + return default + + +def _entry_ts(entry) -> float | None: + """Pull a numeric timestamp from a LogEntry, falling back to float(time).""" + try: + ts = entry.get("timestamp") if isinstance(entry, dict) else None + if isinstance(ts, datetime): + return ts.timestamp() + if ts is not None: + return _safe_float(ts) + return _safe_float(entry.get("time"), 0.0) if isinstance(entry, dict) else None + except Exception: + return None + + +def _entry_field(entry, key: str, default=""): + """Dict-or-attribute getter that never raises.""" + try: + if isinstance(entry, dict): + return entry.get(key, default) + return getattr(entry, key, default) + except Exception: + return default + + +def _window_split(entries: list) -> tuple[list, list]: + """Split ``entries`` into (baseline, current). + + First BASELINE_RATIO of entries go to the baseline, the remainder into the + current window. The current window is dropped entirely for tiny logs. + """ + if not entries: + return [], [] + try: + n = len(entries) + cut = max(1, int(n * BASELINE_RATIO)) + baseline = entries[:cut] + current = entries[cut:] + # ensure both windows have a minimum sample size to be meaningful + if len(current) < max(2, MIN_SAMPLES // 2): + return baseline, [] + return baseline, current + except Exception: + return [], [] + + +def _window_duration(window: list) -> float: + """Compute the spanned seconds in ``window`` using timestamp field. + + Returns 0.0 when timestamps are unavailable so callers can avoid div-by-zero. + """ + if not window: + return 0.0 + try: + timestamps: list[float] = [] + for e in window: + ts = _entry_ts(e) + if ts is not None and ts > 0: + timestamps.append(ts) + if len(timestamps) < 2: + return 0.0 + span = max(timestamps) - min(timestamps) + return span if span > 0 else 0.0 + except Exception: + return 0.0 + + +def _rate_per_minute(count: int, duration_sec: float) -> float: + """Convert ``count over duration_sec`` to requests-per-minute.""" + try: + if duration_sec <= 0: + return float(count) + return count / (duration_sec / 60.0) + except Exception: + return 0.0 + + +def _client_counts(window: Iterable) -> Counter: + """Build a Counter {client: count} for ``window``.""" + out: Counter = Counter() + if not window: + return out + try: + for e in window: + client = _entry_field(e, "client", "") + if not client: + continue + out[client] += 1 + except Exception: + return out + return out + + +# --- Public API -------------------------------------------------------------- + +def detect_client_anomalies( + entries: list, + baseline_window: list | None = None, + current_window: list | None = None, +) -> list[dict]: + """Detect clients whose recent request rate spikes vs. their own past. + + If ``baseline_window`` / ``current_window`` are omitted we split the given + ``entries`` using :data:`BASELINE_RATIO` / :data:`CURRENT_RATIO`. + + Returns a list of dicts sorted by severity (critical first), then by ratio:: + + { + "client": "172.16.x.y", + "baseline_rpm": 12.3, # requests/min in baseline window + "current_rpm": 180.0, # requests/min in current window + "baseline_count": 50, + "current_count": 200, + "ratio": 14.6, # current / baseline + "is_anomaly": True, + "severity": "critical" | "warning" | None, + "baseline_window_sec": 244.0, + "current_window_sec": 66.5, + } + """ + try: + if baseline_window is None or current_window is None: + baseline_window, current_window = _window_split(entries) + + if not baseline_window or not current_window: + return [] + + base_counts = _client_counts(baseline_window) + cur_counts = _client_counts(current_window) + + if not base_counts or not cur_counts: + return [] + + base_dur = _window_duration(baseline_window) + cur_dur = _window_duration(current_window) + + anomalies: list[dict] = [] + # only consider clients that appear in BOTH windows for ratio math + for client, cur_n in cur_counts.items(): + try: + base_n = base_counts.get(client, 0) + if base_n < MIN_SAMPLES: + # New client; show as anomaly but skip ratio math + anomalies.append({ + "client": client, + "baseline_rpm": 0.0, + "current_rpm": _rate_per_minute(cur_n, cur_dur), + "baseline_count": 0, + "current_count": cur_n, + "ratio": float("inf"), + "is_anomaly": True, + "severity": "warning", + "baseline_window_sec": round(base_dur, 2), + "current_window_sec": round(cur_dur, 2), + }) + continue + + base_rpm = _rate_per_minute(base_n, base_dur) + cur_rpm = _rate_per_minute(cur_n, cur_dur) + # avoid div-by-zero: if base_rpm is zero, treat as infinite ratio + if base_rpm <= 0: + ratio = float("inf") if cur_rpm > 0 else 0.0 + else: + ratio = cur_rpm / base_rpm + + if ratio > SPIKE_WARN_RATIO: + severity = "critical" if ratio > SPIKE_CRIT_RATIO else "warning" + anomalies.append({ + "client": client, + "baseline_rpm": round(base_rpm, 2), + "current_rpm": round(cur_rpm, 2), + "baseline_count": base_n, + "current_count": cur_n, + "ratio": round(ratio, 2) if ratio != float("inf") else ratio, + "is_anomaly": True, + "severity": severity, + "baseline_window_sec": round(base_dur, 2), + "current_window_sec": round(cur_dur, 2), + }) + except Exception: + # single client failure doesn't kill the whole result + continue + + # sort: critical first, then highest ratio + def _sort_key(a: dict): + sev_rank = 0 if a.get("severity") == "critical" else 1 + ratio = a.get("ratio", 0) + ratio_key = -ratio if isinstance(ratio, (int, float)) and ratio != float("inf") else -1e18 + return (sev_rank, ratio_key) + + try: + anomalies.sort(key=_sort_key) + except Exception: + pass + return anomalies + except Exception: + return [] + + +def detect_large_requests( + entries: list, + size_threshold_mb: float = 100, + limit: int = DEFAULT_LARGE_LIMIT, +) -> list[dict]: + """Find individual responses larger than ``size_threshold_mb`` MB. + + Returned dicts are sorted by size descending and capped at ``limit``:: + + { + "time": "2024-01-01 12:34:56", + "timestamp": datetime(...), + "client": "172.16.x.y", + "url": "https://...", + "host": "example.com", + "method": "GET", + "result": "TCP_MISS/200", + "size_bytes": 157286400, + "size_mb": 150.0, + } + """ + try: + if not entries: + return [] + threshold_bytes = float(size_threshold_mb) * 1024.0 * 1024.0 + out: list[dict] = [] + for e in entries: + try: + size = _safe_float(_entry_field(e, "size", 0), 0.0) + if size <= threshold_bytes: + continue + ts = _entry_ts(e) + dt = None + try: + dt = _entry_field(e, "timestamp", None) + if not isinstance(dt, datetime): + dt = datetime.fromtimestamp(ts) if ts else None + except Exception: + dt = None + out.append({ + "time": dt.strftime("%Y-%m-%d %H:%M:%S") if dt else "", + "timestamp": dt.isoformat() if dt else None, + "client": _entry_field(e, "client", ""), + "url": _entry_field(e, "url", ""), + "host": _entry_field(e, "host", ""), + "method": _entry_field(e, "method", ""), + "result": _entry_field(e, "result", ""), + "result_code": _entry_field(e, "result_code", ""), + "http_code": _entry_field(e, "http_code", ""), + "size_bytes": int(size), + "size_mb": round(size / (1024.0 * 1024.0), 2), + }) + except Exception: + continue + + try: + out.sort(key=lambda x: x.get("size_bytes", 0), reverse=True) + except Exception: + pass + return out[: max(1, int(limit))] + except Exception: + return [] + + +def detect_high_freq_small( + entries: list, + size_threshold_kb: float = 1, + min_requests: int = 100, + limit: int = DEFAULT_SMALL_LIMIT, +) -> list[dict]: + """Find clients that emit lots of tiny responses (< ``size_threshold_kb``). + + Tiny-response spam is often a fingerprint of polling bots, beaconing + implants, or chatty telemetry agents. + + Returns dicts sorted by request count desc:: + + { + "client": "172.16.x.y", + "request_count": 540, + "total_bytes": 420000, + "avg_size": 778, + "sample_url": "https://...", + "sample_host": "example.com", + } + """ + try: + if not entries: + return [] + threshold_bytes = float(size_threshold_kb) * 1024.0 + counts: Counter = Counter() + total_bytes: dict[str, int] = defaultdict(int) + sample_url: dict[str, str] = {} + sample_host: dict[str, str] = {} + for e in entries: + try: + size = _safe_float(_entry_field(e, "size", 0), 0.0) + if size >= threshold_bytes: + continue + client = _entry_field(e, "client", "") + if not client: + continue + counts[client] += 1 + total_bytes[client] += int(size) + if client not in sample_url: + sample_url[client] = _entry_field(e, "url", "") + sample_host[client] = _entry_field(e, "host", "") + except Exception: + continue + + out: list[dict] = [] + for client, n in counts.most_common(): + if n < max(1, int(min_requests)): + continue + tb = total_bytes.get(client, 0) + avg = tb / n if n else 0 + out.append({ + "client": client, + "request_count": n, + "total_bytes": tb, + "avg_size": round(avg, 1), + "sample_url": sample_url.get(client, ""), + "sample_host": sample_host.get(client, ""), + }) + + try: + out.sort(key=lambda x: x.get("request_count", 0), reverse=True) + except Exception: + pass + return out[: max(1, int(limit))] + except Exception: + return [] + + +def detect_new_clients( + entries: list, + known_clients: set[str] | list[str] | None = None, + limit: int = DEFAULT_NEW_LIMIT, +) -> list[dict]: + """Find clients that appear for the first time within ``entries``. + + "First time" means: their first LogEntry timestamp is among the entries + we are inspecting AND they aren't in ``known_clients`` (when supplied). + + Returns:: + + { + "client": "172.16.x.y", + "first_seen": "2024-...", + "first_ts": float, + "request_count": int, + "sample_url": "...", + } + """ + try: + if not entries: + return [] + if known_clients is None: + known = set() + elif isinstance(known_clients, set): + known = known_clients + else: + known = set(known_clients or []) + + # find earliest timestamp per client + first_ts: dict[str, float] = {} + first_dt: dict[str, datetime] = {} + counts: Counter = Counter() + sample_url: dict[str, str] = {} + for e in entries: + try: + client = _entry_field(e, "client", "") + if not client: + continue + ts = _entry_ts(e) or 0.0 + dt = _entry_field(e, "timestamp", None) + if ts and (client not in first_ts or ts < first_ts[client]): + first_ts[client] = ts + if isinstance(dt, datetime) and (client not in first_dt or dt < first_dt[client]): + first_dt[client] = dt + counts[client] += 1 + if client not in sample_url: + sample_url[client] = _entry_field(e, "url", "") + except Exception: + continue + + # min timestamp across all entries = baseline cutoff + all_ts = [t for t in first_ts.values() if t > 0] + if not all_ts: + return [] + global_min_ts = min(all_ts) + out: list[dict] = [] + for client, ts in first_ts.items(): + try: + # first-seen in entries exactly equal to global min means + # they've never been seen before — anything <= min qualifies + # as a "new" arrival within this log slice + if ts > global_min_ts: + continue + if client in known: + continue + dt = first_dt.get(client) + out.append({ + "client": client, + "first_seen": dt.strftime("%Y-%m-%d %H:%M:%S") if dt else "", + "first_ts": ts, + "request_count": counts.get(client, 1), + "sample_url": sample_url.get(client, ""), + }) + except Exception: + continue + + try: + out.sort(key=lambda x: x.get("first_ts", 0)) + except Exception: + pass + return out[: max(1, int(limit))] + except Exception: + return [] + + +def get_anomaly_summary(entries: list) -> dict: + """Run all four detectors and bundle the results. + + Layout:: + + { + "anomalous_clients": [...], # detect_client_anomalies + "large_requests": [...], # detect_large_requests + "high_freq_small": [...], # detect_high_freq_small + "new_clients": [...], # detect_new_clients + "total_anomalies": int, + "thresholds": {...}, + } + """ + try: + anomalous = detect_client_anomalies(entries) + except Exception: + anomalous = [] + try: + large = detect_large_requests(entries) + except Exception: + large = [] + try: + small = detect_high_freq_small(entries) + except Exception: + small = [] + try: + new = detect_new_clients(entries) + except Exception: + new = [] + + total = len(anomalous) + len(large) + len(small) + len(new) + + # quick stats over the (current) window for the page header + header_stats: dict = {} + try: + if entries: + n = len(entries) + cut = max(1, int(n * CURRENT_RATIO)) + current = entries[cut:] + header_stats = { + "current_window_count": len(current), + "current_window_clients": len({_entry_field(e, "client", "") for e in current}), + } + except Exception: + pass + + return { + "anomalous_clients": anomalous, + "large_requests": large, + "high_freq_small": small, + "new_clients": new, + "total_anomalies": total, + "thresholds": { + "spike_warn_ratio": SPIKE_WARN_RATIO, + "spike_crit_ratio": SPIKE_CRIT_RATIO, + "large_size_mb": 100, + "small_size_kb": 1, + "min_small_requests": 100, + }, + "header_stats": header_stats, + } + + +# --- Tiny self-test when run directly -------------------------------------- + +if __name__ == "__main__": # pragma: no cover + import json + import log_parser + + sample_path = "/root/squid-manager/sample_access.log" + try: + with open(sample_path, encoding="utf-8") as f: + text = f.read() + entries = log_parser.parse_lines(text) + except OSError: + entries = [] + + summary = get_anomaly_summary(entries) + print(json.dumps(summary, default=str, ensure_ascii=False, indent=2)[:2000]) diff --git a/app.py b/app.py new file mode 100644 index 0000000..367477a --- /dev/null +++ b/app.py @@ -0,0 +1,4185 @@ +"""Squid Web Manager - main Flask application. + +Provides a web UI for managing squid.conf and analyzing access.log. + +Architecture: +- Flask + SQLite for users / app config / audit log +- access.log is parsed on demand (with in-memory cache) - not stored in DB +- Configured via /config page; squid paths default to /etc/squid, /var/log/squid +- Service control via systemctl (squid.service) - graceful when squid missing +""" +from __future__ import annotations + +import os +import re +import shlex +import shutil +import socket +import subprocess +import threading +import time +import json +from collections import deque +from datetime import datetime, timezone, timedelta +from functools import wraps + +import log_parser +import squid_config +import security +import squid_mgr +import geoip_lookup +from flask import ( + Flask, Response, abort, flash, g, jsonify, redirect, render_template, + request, send_from_directory, session, url_for, +) +from flask_sqlalchemy import SQLAlchemy +from werkzeug.security import check_password_hash, generate_password_hash + +# --------------------------------------------------------------------------- + +BASE_DIR = os.path.dirname(os.path.abspath(__file__)) +INSTANCE_DIR = os.path.join(BASE_DIR, "instance") +BACKUP_DIR = os.path.join(BASE_DIR, "backups") +UPLOAD_DIR = os.path.join(BASE_DIR, "uploads") +for d in (INSTANCE_DIR, BACKUP_DIR, UPLOAD_DIR): + os.makedirs(d, exist_ok=True) + +app = Flask(__name__, static_folder="static", template_folder="templates") +app.config.update( + SECRET_KEY=os.environ.get("SECRET_KEY", "squid-mgr-change-me-in-prod"), + SQLALCHEMY_DATABASE_URI=f"sqlite:///{os.path.join(INSTANCE_DIR, 'squid_mgr.db')}", + SQLALCHEMY_TRACK_MODIFICATIONS=False, + MAX_CONTENT_LENGTH=20 * 1024 * 1024, + SQUID_BINARY=os.environ.get("SQUID_BINARY", "squid"), + SYSTEMCTL=os.environ.get("SYSTEMCTL", "systemctl"), + # security + SESSION_COOKIE_HTTPONLY=True, + SESSION_COOKIE_SAMESITE="Lax", + PERMANENT_SESSION_LIFETIME=28800, # absolute ceiling + SECURITY_SESSION_IDLE_TIMEOUT=int(os.environ.get("SQUIDMGR_IDLE_TO", "1800")), + SECURITY_SESSION_ABSOLUTE_TIMEOUT=int(os.environ.get("SQUIDMGR_ABS_TO", "28800")), + SECURITY_LOGIN_THROTTLE_WINDOW=int(os.environ.get("SQUIDMGR_THROTTLE_WIN", "300")), + SECURITY_LOGIN_THROTTLE_MAX=int(os.environ.get("SQUIDMGR_THROTTLE_MAX", "5")), + SECURITY_LOGIN_THROTTLE_LOCKOUT=int(os.environ.get("SQUIDMGR_THROTTLE_LOCK", "900")), + SECURITY_CSRF_ENABLED=os.environ.get("SQUIDMGR_CSRF", "1") == "1", +) +db = SQLAlchemy(app) + +# Wire security: enforce session lifetime + CSRF on every request +@app.before_request +def _security_gate(): + security.enforce_session_lifetime() + security.csrf_protect() + +# Make CSRF token + security state available to all templates +@app.context_processor +def _security_globals(): + return { + # csrf_token as a function (for `{{ csrf_token() }}` calls) + "csrf_token": security.generate_csrf_token, + "password_strength": security.password_strength, + "session_idle_to": app.config["SECURITY_SESSION_IDLE_TIMEOUT"], + "current_username": session.get("username", ""), + "current_user_id": session.get("user_id"), + } + + +@app.template_global() +def csrf_input(): + """Render for use in Jinja forms.""" + return f'' + + +@app.template_global() +def csrf_meta(): + """Render for AJAX use.""" + return f'' + + +# --------------------------------------------------------------------------- +# Models +# --------------------------------------------------------------------------- + +class User(db.Model): + __tablename__ = "users" + id = db.Column(db.Integer, primary_key=True) + username = db.Column(db.String(80), unique=True, nullable=False) + password_hash = db.Column(db.String(255), nullable=False) + role = db.Column(db.String(20), default="admin") + created_at = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc)) + + def set_password(self, pw): + # pbkdf2:sha256 to avoid scrypt portability issues (see flask skill) + self.password_hash = generate_password_hash(pw, method="pbkdf2:sha256") + + def check_password(self, pw): + try: + return check_password_hash(self.password_hash, pw) + except Exception: + return False + + +class AppConfig(db.Model): + """Key-value store for app-level settings (squid paths, log behavior).""" + __tablename__ = "app_config" + id = db.Column(db.Integer, primary_key=True) + key = db.Column(db.String(60), unique=True, nullable=False) + value = db.Column(db.Text, default="") + updated_at = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc), + onupdate=lambda: datetime.now(timezone.utc)) + + +class AuditLog(db.Model): + __tablename__ = "audit_log" + id = db.Column(db.Integer, primary_key=True) + username = db.Column(db.String(80), default="system") + action = db.Column(db.String(120), nullable=False) + detail = db.Column(db.Text, default="") + timestamp = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc), + index=True) + + @property + def timestamp_local(self): + if not self.timestamp: + return "" + # stored UTC, render as UTC for consistency + return self.timestamp.strftime("%Y-%m-%d %H:%M:%S UTC") + + +class SquidInstance(db.Model): + """A managed Squid instance (local or remote via SSH). + + Multiple rows can coexist; one row is marked is_active=True (or selected + via session['active_instance_id']). All pages (config/*, logs, service) + resolve paths/service/binary through get_config() which prefers the + active instance over the global AppConfig table. + """ + __tablename__ = "squid_instances" + id = db.Column(db.Integer, primary_key=True) + name = db.Column(db.String(80), unique=True, nullable=False) + description = db.Column(db.String(200), default="") + squid_conf_path = db.Column(db.String(500), default="/etc/squid/squid.conf") + access_log_path = db.Column(db.String(500), default="/var/log/squid/access.log") + cache_log_path = db.Column(db.String(500), default="/var/log/squid/cache.log") + squid_binary = db.Column(db.String(100), default="squid") + systemctl = db.Column(db.String(100), default="systemctl") + squid_service = db.Column(db.String(100), default="squid") + ssh_host = db.Column(db.String(200), default="") + ssh_user = db.Column(db.String(80), default="") + ssh_port = db.Column(db.Integer, default=22) + ssh_key_path = db.Column(db.String(500), default="") + is_local = db.Column(db.Boolean, default=True) + is_active = db.Column(db.Boolean, default=False) + created_at = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc)) + + +class LogEntry(db.Model): + """A parsed squid access.log line, persisted to SQLite. + + Rows are written by :mod:`log_storage` (via :func:`sync_log_to_db`) and + read by the logs UI, statistics page, alerts, anomaly detection and the + real-time dashboard. Indexes are tuned for the common access patterns: + + - single-column indexes on ``log_time``, ``client``, ``result_code``, + ``method``, ``host`` to keep filter-by-X queries fast. + - composite ``(log_time, client)`` and ``(log_time, host)`` indexes for + "top clients in time window T" / "top hosts in time window T" style + aggregations done via ``GROUP BY``. + - we do *not* add a unique constraint at the SQL layer (SQLite would + reject bulk inserts that violate it, and we want ON CONFLICT DO + NOTHING to gracefully drop duplicates). Dedup is enforced by an + in-memory ``(instance_id, log_time, client, raw_line)`` check inside + :func:`log_storage.sync_log_to_db`. + """ + __tablename__ = "log_entries" + id = db.Column(db.Integer, primary_key=True) + instance_id = db.Column(db.Integer, default=0, index=True) # SquidInstance.id, 0=default + log_time = db.Column(db.Float, nullable=False, index=True) # unix timestamp + elapsed_ms = db.Column(db.Integer, default=0) + client = db.Column(db.String(64), default='', index=True) + result_code = db.Column(db.String(32), default='', index=True) + http_code = db.Column(db.String(8), default='') + size_bytes = db.Column(db.BigInteger, default=0) + method = db.Column(db.String(16), default='', index=True) + url = db.Column(db.Text, default='') + host = db.Column(db.String(256), default='', index=True) + hier_code = db.Column(db.String(32), default='') + content_type = db.Column(db.String(128), default='') + raw_line = db.Column(db.Text, default='') + created_at = db.Column(db.DateTime, default=lambda: datetime.now(timezone.utc)) + + __table_args__ = ( + db.Index('idx_log_time_client', 'log_time', 'client'), + db.Index('idx_log_time_host', 'log_time', 'host'), + ) + + +# --------------------------------------------------------------------------- +# Default config +# --------------------------------------------------------------------------- + +DEFAULTS = { + "squid_conf_path": "/etc/squid/squid.conf", + "squid_binary": "squid", + "systemctl": "systemctl", + "squid_service": "squid", + "access_log_path": "/var/log/squid/access.log", + "cache_log_path": "/var/log/squid/cache.log", + "log_parse_limit": "10000", # max lines to parse from access.log + "log_cache_seconds": "30", # in-memory parse cache TTL + "live_tail_lines": "100", # default for /logs live view + "auto_refresh_seconds": "30", + # P2-1: persistent log storage switch. 1 = read from SQLite LogEntry + # table (imported via /logs/sync); 0 = legacy behaviour - read tail of + # access.log on every request. Default-on so existing single-user + # installs get history retention for free. + "log_use_persistent": "1", + # P2-1: auto-sync interval (seconds). 0 = manual sync only. + "log_sync_interval": "0", + # P1-4: squidclient mgr: connection settings for the real-time dashboard + "squidclient_binary": "squidclient", + "squid_mgr_host": "127.0.0.1", + "squid_mgr_port": "3128", + "squid_mgr_password": "", + "squid_mgr_timeout": "5", +} + +# Keys whose value comes from the active SquidInstance (if any) rather than +# the global AppConfig table. All other keys continue to be resolved from +# AppConfig → DEFAULTS as before, so multi-instance support is purely +# additive and app-level knobs (log parsing, refresh, etc.) stay global. +_INSTANCE_KEYS = { + "squid_conf_path", + "squid_binary", + "systemctl", + "squid_service", + "access_log_path", + "cache_log_path", +} + + +def _resolve_active_instance(): + """Return the active SquidInstance row for this request, or None. + + Resolution order: + 1. session['active_instance_id'] (if set and still exists) + 2. SquidInstance.is_active == True (first one) + 3. None (no instances configured / none active) + + Cached on flask.g for the lifetime of the request so template renders + don't repeatedly hit the DB. + """ + if "active_instance" in g.__dict__: + return g.active_instance + inst = None + sess_id = session.get("active_instance_id") + if sess_id is not None: + try: + inst = db.session.get(SquidInstance, int(sess_id)) + except (TypeError, ValueError): + inst = None + if inst is None: + inst = SquidInstance.query.filter_by(is_active=True).first() + if inst is not None: + # materialise the choice in the session so subsequent requests + # stay sticky even when multiple instances are flagged active + session["active_instance_id"] = inst.id + g.active_instance = inst + return inst + + +def active_instance_name() -> str: + """Display name of the active instance, or '(default)' if none.""" + inst = _resolve_active_instance() + if inst is None: + return "(default)" + return inst.name or f"#{inst.id}" + + +def get_config(key: str, default: str = "") -> str: + # 1. Active instance overrides AppConfig for instance-specific keys + if key in _INSTANCE_KEYS: + inst = _resolve_active_instance() + if inst is not None: + val = getattr(inst, key, None) + if val not in (None, ""): + return str(val) + # 2. AppConfig table (per-key override) + row = AppConfig.query.filter_by(key=key).first() + if row and row.value is not None and row.value != "": + return row.value + # 3. Built-in defaults + return DEFAULTS.get(key, default) + + +def get_all_config() -> dict: + out = dict(DEFAULTS) + for row in AppConfig.query.all(): + if row.value is not None and row.value != "": + out[row.key] = row.value + # Active instance wins over AppConfig for instance-specific keys + inst = _resolve_active_instance() + if inst is not None: + for k in _INSTANCE_KEYS: + val = getattr(inst, k, None) + if val not in (None, ""): + out[k] = str(val) + return out + + +def set_config(key: str, value: str): + row = AppConfig.query.filter_by(key=key).first() + if row: + row.value = value + else: + db.session.add(AppConfig(key=key, value=value)) + + +def audit(action: str, detail: str = ""): + db.session.add(AuditLog( + username=session.get("username", "system"), + action=action, detail=detail[:2000], + )) + + +# --------------------------------------------------------------------------- +# Auth helpers +# --------------------------------------------------------------------------- + +def current_user(): + uid = session.get("user_id") + if not uid: + return None + return db.session.get(User, uid) + + +def login_required(func): + @wraps(func) + def wrapper(*a, **kw): + if not session.get("user_id"): + return redirect(url_for("login", next=request.path)) + return func(*a, **kw) + return wrapper + + +@app.context_processor +def inject_globals(): + cu = current_user() + # Expose instance list + active name so the sidebar switcher and the + # /instances page can render without each route re-querying. Querying + # outside an auth context (e.g. login page) returns an empty list. + instances = [] + active_id = None + active_name = "(default)" + if session.get("user_id"): + try: + instances = SquidInstance.query.order_by(SquidInstance.id.asc()).all() + except Exception: + instances = [] + try: + inst = _resolve_active_instance() + if inst is not None: + active_id = inst.id + active_name = inst.name or f"#{inst.id}" + except Exception: + pass + return { + "current_user": cu, + "all_config": get_all_config(), + "instances": instances, + "active_instance_id": active_id, + "active_instance_name": active_name, + } + + +# --------------------------------------------------------------------------- +# Jinja filters +# --------------------------------------------------------------------------- + +@app.template_filter("thousands") +def thousands_filter(n): + """Format integer with thousands separator.""" + try: + return f"{int(n):,}" + except (TypeError, ValueError): + return str(n) + + +# --------------------------------------------------------------------------- +# Squid paths & service control +# --------------------------------------------------------------------------- + +def squid_conf_path() -> str: + return get_config("squid_conf_path") + + +def access_log_path() -> str: + return get_config("access_log_path") + + +def cache_log_path() -> str: + return get_config("cache_log_path") + + +def run_cmd(cmd: list[str] | str, timeout: int = 10) -> tuple[int, str, str]: + """Run command, return (exit_code, stdout, stderr).""" + shell = isinstance(cmd, str) + try: + r = subprocess.run(cmd, shell=shell, capture_output=True, + text=True, timeout=timeout) + return r.returncode, (r.stdout or "").strip(), (r.stderr or "").strip() + except subprocess.TimeoutExpired: + return 124, "", f"Command timed out after {timeout}s" + except FileNotFoundError as e: + return 127, "", str(e) + except Exception as e: + return 1, "", f"{type(e).__name__}: {e}" + + +def service_status() -> dict: + """Get squid service status via systemctl.""" + systemctl = get_config("systemctl", "systemctl") + svc = get_config("squid_service", "squid") + rc, out, err = run_cmd([systemctl, "is-active", svc]) + active = (rc == 0) + state = out.strip() if out else "unknown" + if state == "inactive": + state = "stopped" + # fetch version + binary = get_config("squid_binary", "squid") + rc_v, out_v, _ = run_cmd([binary, "--version"]) + version = "" + if rc_v == 0 and out_v: + m = re.search(r"Version\s+(\S+)", out_v) + version = m.group(1) if m else out_v.split("\n")[0][:80] + # uptime / pid + rc_s, out_s, _ = run_cmd([systemctl, "show", svc, + "--property=MainPID,ActiveEnterTimestamp,FragmentPath"]) + meta = {} + if rc_s == 0 and out_s: + for kv in out_s.split("\n"): + if "=" in kv: + k, _, v = kv.partition("=") + meta[k] = v + return { + "active": active, + "state": state, + "version": version, + "pid": meta.get("MainPID", ""), + "since": meta.get("ActiveEnterTimestamp", ""), + "unit_file": meta.get("FragmentPath", ""), + "binary_available": bool(shutil.which(binary)), + } + + +def service_control(action: str) -> tuple[bool, str]: + """start / stop / restart / reload / reconfigure.""" + systemctl = get_config("systemctl", "systemctl") + svc = get_config("squid_service", "squid") + valid = {"start", "stop", "restart", "reload", "try-reload-or-restart"} + if action not in valid: + return False, f"Unknown action: {action}" + # reload for sysv-compat services may fail; try reload then restart if needed + rc, out, err = run_cmd([systemctl, action, svc], timeout=30) + msg = (out + " " + err).strip() + time.sleep(1.5) + if action in ("reload", "try-reload-or-restart"): + rc2, _, _ = run_cmd([systemctl, "is-active", svc]) + if rc2 != 0: + # fall back to restart + rc3, out3, err3 = run_cmd([systemctl, "restart", svc], timeout=30) + msg += " | fallback restart: " + (out3 + " " + err3).strip() + rc = rc3 + if rc == 0: + return True, msg or "OK" + return False, msg or f"{systemctl} {action} {svc} failed (exit {rc})" + + +# --------------------------------------------------------------------------- +# squid.conf read/write +# --------------------------------------------------------------------------- + +def read_squid_conf() -> tuple[str, str]: + """Return (content, error_message).""" + path = squid_conf_path() + if not os.path.isfile(path): + return "", f"配置文件不存在: {path}" + try: + with open(path, "r", encoding="utf-8", errors="replace") as f: + return f.read(), "" + except OSError as e: + return "", f"读取失败: {e}" + + +def backup_conf(reason: str = "manual") -> str | None: + """Back up current squid.conf to backups/ dir, return backup path.""" + path = squid_conf_path() + if not os.path.isfile(path): + return None + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + safe_reason = re.sub(r"[^A-Za-z0-9_]", "_", reason)[:40] + backup_name = f"squid.conf.{ts}.{safe_reason}" + backup_path = os.path.join(BACKUP_DIR, backup_name) + try: + shutil.copy2(path, backup_path) + return backup_path + except OSError: + return None + + + +def _show_config_preview(pending_content: str, return_to: str): + """Store pending content in session, redirect to GET /config/preview for confirmation.""" + import base64 + session["_pending_conf"] = base64.b64encode(pending_content.encode()).decode() + session["_pending_return"] = return_to + return redirect(url_for("config_preview")) + +def write_squid_conf(content: str, validate: bool = True) -> tuple[bool, str]: + """Write new content to squid.conf. Validates first if requested.""" + binary = get_config("squid_binary", "squid") + if validate: + ok, msg = squid_config.validate_conf(content, squid_binary=binary) + if not ok: + return False, f"配置验证失败: {msg}" + path = squid_conf_path() + backup = backup_conf("before-write") + try: + # write to temp then rename + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + f.write(content) + os.replace(tmp, path) + return True, f"已写入 {path}" + (f" (备份: {os.path.basename(backup)})" if backup else "") + except OSError as e: + return False, f"写入失败: {e}" + + +# --------------------------------------------------------------------------- +# Log reading & parsing +# --------------------------------------------------------------------------- + +_log_cache_lock = threading.Lock() +_log_cache: dict[str, tuple[float, list[log_parser.LogEntry]]] = {} + + +def read_access_log_tail(max_lines: int = 10000) -> str: + """Read the last N lines of access.log.""" + path = access_log_path() + if not os.path.isfile(path): + return "" + try: + # Use deque for efficient tail + with open(path, "rb") as f: + try: + f.seek(0, 2) + size = f.tell() + # Read backwards in chunks for efficiency + chunks: list[bytes] = [] + lines_found = 0 + block_size = 65536 + pos = size + while pos > 0 and lines_found <= max_lines: + read_size = min(block_size, pos) + pos -= read_size + f.seek(pos) + chunk = f.read(read_size) + chunks.append(chunk) + lines_found += chunk.count(b"\n") + data = b"".join(reversed(chunks)) + except OSError: + f.seek(0) + data = f.read() + text = data.decode("utf-8", errors="replace") + lines = text.splitlines() + if len(lines) > max_lines: + lines = lines[-max_lines:] + return "\n".join(lines) + except OSError: + return "" + + +def get_parsed_logs(force: bool = False) -> list[log_parser.LogEntry]: + """Get parsed access.log entries. + + Behaviour is driven by ``log_use_persistent`` in :class:`AppConfig`: + + - ``1`` (default): sync the access.log tail into SQLite on demand + (with a small per-request throttle so we don't re-import the same + chunk repeatedly), then read from the persistent table via + :func:`log_storage.query_logs`. The result is cached in-memory using + the legacy ``_log_cache`` so subsequent page renders are fast. + - ``0``: legacy behaviour - tail the access.log file via + :func:`read_access_log_tail` and parse in-memory. Used as an + escape hatch if the persistent backend has issues. + + The return shape is always :class:`log_parser.LogEntry` (a dict + subclass) with ``time``, ``client``, ``size`` etc. so consumers + (templates, alert rules, anomaly detection) keep working unmodified. + """ + import log_storage + + use_persistent = get_config("log_use_persistent", "1") not in ("0", "false", "False") + if not use_persistent: + # legacy path + limit = int(get_config("log_parse_limit", "10000")) + ttl = int(get_config("log_cache_seconds", "30")) + key = f"access:legacy:{limit}" + now = time.time() + with _log_cache_lock: + cached = _log_cache.get(key) + if cached and not force and (now - cached[0]) < ttl: + return cached[1] + text = read_access_log_tail(max_lines=limit) + entries = log_parser.parse_lines(text) + with _log_cache_lock: + _log_cache[key] = (now, entries) + return entries + + # persistent path: pull from SQLite + instance_id = 0 + try: + inst = _resolve_active_instance() + if inst is not None: + instance_id = inst.id + except Exception: + instance_id = 0 + + # Best-effort auto-sync (throttled by the file mtime + last sync stamp) + if force or _should_auto_sync(instance_id): + try: + path = access_log_path() + if os.path.isfile(path): + log_storage.sync_log_to_db( + path=path, instance_id=instance_id, + batch_size=500, max_lines=100000, + ) + except Exception: + # never break a request because of a sync hiccup + pass + + # cache the query result like before so template rendering is cheap + limit = int(get_config("log_parse_limit", "10000")) + ttl = int(get_config("log_cache_seconds", "30")) + key = f"access:db:{instance_id}:{limit}" + now = time.time() + with _log_cache_lock: + cached = _log_cache.get(key) + if cached and not force and (now - cached[0]) < ttl: + return cached[1] + rows = log_storage.query_logs({ + "instance_id": instance_id, + "limit": limit, + }) + # Wrap as LogEntry so getattr(d, "category") etc. keep working + wrapped = [log_parser.LogEntry(**r) for r in rows] + # reverse so chronological order (oldest first) matches legacy + wrapped = list(reversed(wrapped)) + with _log_cache_lock: + _log_cache[key] = (now, wrapped) + return wrapped + + +def _should_auto_sync(instance_id: int) -> bool: + """Decide whether to re-sync the access.log tail for ``instance_id``. + + Cheap heuristic: check the file's mtime against the last sync stamp + kept in :class:`AppConfig`. Always returns False when + ``log_sync_interval`` is 0 (manual sync only). + """ + try: + interval = int(get_config("log_sync_interval", "0")) + if interval <= 0: + return False + path = access_log_path() + if not os.path.isfile(path): + return False + mtime = os.path.getmtime(path) + last_sync = log_storage.get_last_sync_at() or "" + if not last_sync: + return True + try: + last_dt = datetime.fromisoformat(last_sync.replace("Z", "+00:00")) + except ValueError: + return True + # re-sync if mtime is newer than last_sync, but throttle to once + # per ``interval`` seconds between requests + now = time.time() + if now - _auto_sync_last_call < interval: + return False + _auto_sync_last_call = now + return mtime > last_dt.timestamp() + except Exception: + return False + + +_auto_sync_last_call = 0.0 + + +def get_stats() -> dict: + """Return aggregate stats. Drives off :func:`log_storage.get_log_stats` + when the persistent backend is enabled, otherwise falls back to the + legacy ``log_parser.aggregate_stats(get_parsed_logs())`` path. + """ + import log_storage + use_persistent = get_config("log_use_persistent", "1") not in ("0", "false", "False") + if not use_persistent: + entries = get_parsed_logs() + return log_parser.aggregate_stats(entries) + + instance_id = 0 + try: + inst = _resolve_active_instance() + if inst is not None: + instance_id = inst.id + except Exception: + instance_id = 0 + return log_storage.get_log_stats(instance_id=instance_id) + + +# --------------------------------------------------------------------------- +# Routes - auth +# --------------------------------------------------------------------------- + +@app.route("/login", methods=["GET", "POST"]) +def login(): + if request.method == "POST": + # throttle check + throttled, secs = security.is_throttled() + if throttled: + flash(f"登录失败次数过多,请等待 {secs} 秒后再试", "error") + audit("login_throttled", f"ip={security.client_ip()}") + db.session.commit() + return render_template("login.html", next=request.args.get("next", ""), + throttle_remaining=secs) + username = request.form.get("username", "").strip() + password = request.form.get("password", "") + nxt = request.form.get("next") or url_for("dashboard") + user = User.query.filter_by(username=username).first() + if user and user.check_password(password): + session.clear() + session["user_id"] = user.id + session["username"] = user.username + session["_csrf"] = security.generate_csrf_token() + session["_logged_in_at"] = time.time() + session["_last_seen"] = time.time() + security.record_successful_login() + audit("login", f"user={username} ip={security.client_ip()}") + db.session.commit() + return redirect(nxt) + # failed login + locked, lockout = security.record_failed_login() + if locked: + flash(f"连续登录失败,已锁定 {lockout} 秒", "error") + audit("login_locked", f"user={username} ip={security.client_ip()}") + else: + flash("用户名或密码错误", "error") + audit("login_fail", f"user={username} ip={security.client_ip()}") + db.session.commit() + return render_template("login.html", next=request.args.get("next", ""), + throttle_remaining=0) + + +@app.route("/logout") +def logout(): + audit("logout", f"user={session.get('username', '')}") + db.session.commit() + session.clear() + return redirect(url_for("login")) + + +@app.route("/change-password", methods=["GET", "POST"]) +@login_required +def change_password(): + if request.method == "POST": + old = request.form.get("old_password", "") + new = request.form.get("new_password", "") + confirm = request.form.get("confirm_password", "") + u = current_user() + if not u.check_password(old): + flash("旧密码错误", "error") + elif len(new) < 6: + flash("新密码至少 6 位", "error") + elif new != confirm: + flash("两次输入不一致", "error") + else: + u.set_password(new) + audit("change_password") + db.session.commit() + flash("密码已更新", "ok") + return redirect(url_for("dashboard")) + return render_template("change_password.html") + + +# --------------------------------------------------------------------------- +# Routes - dashboard (log analytics) +# --------------------------------------------------------------------------- + +@app.route("/") +@login_required +def dashboard(): + stats = get_stats() + entries = get_parsed_logs() + cfg = get_all_config() + # P1-2: surface any active alert rules at the top of the dashboard so + # operators see them without having to navigate to /alerts. + try: + import alerts as _alerts # local import - keeps top imports stable + _rules = _alerts.load_rules(_alerts_path()) + statuses = _alerts.evaluate_rules(entries, _rules) + active_alerts = [s for s in statuses if s["triggered"]] + if active_alerts: + flash( + "当前有 %d 条活跃告警，详情见" + "告警规则" + % (len(active_alerts), url_for("alerts_view")), + "warning", + ) + except Exception: + active_alerts = [] + return render_template("dashboard.html", stats=stats, entries_count=len(entries), + cfg=cfg, format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration, + active_alerts=active_alerts) + + +@app.route("/api/stats") +@login_required +def api_stats(): + return jsonify(get_stats()) + + +@app.route("/api/logs/refresh", methods=["POST"]) +@login_required +def api_logs_refresh(): + """Force re-read of access.log.""" + entries = get_parsed_logs(force=True) + # P2-1: when persistent mode is on, the legacy aggregate_stats path + # is *not* equivalent to get_stats() (which uses SQL aggregation). + # Return whichever the user has configured. + use_persistent = get_config("log_use_persistent", "1") not in ("0", "false", "False") + if use_persistent: + stats = get_stats() + return jsonify({"ok": True, "count": len(entries), "stats": stats}) + stats = log_parser.aggregate_stats(entries) + return jsonify({"ok": True, "count": len(entries), "stats": stats}) + + +# --------------------------------------------------------------------------- +# Routes - logs (search, filter, live tail, import) +# --------------------------------------------------------------------------- + +@app.route("/logs") +@login_required +def logs_view(): + """Logs search page. + + With P2-1 persistent storage enabled (``log_use_persistent=1``), this + route pushes every filter down to the SQLite layer so we get accurate + counts and pagination over multi-MB / multi-day ranges. The legacy + file-tail + in-Python filter path is still available as a fallback + when the user explicitly disables persistence. + """ + cfg = get_all_config() + import log_storage + use_persistent = cfg.get("log_use_persistent", "1") not in ("0", "false", "False") + + # ---- filters from query string ---------------------------------------- + f_client = request.args.get("client", "").strip() + f_method = request.args.get("method", "").strip() + f_result = request.args.get("result_code", "").strip() + f_host = request.args.get("host", "").strip() + f_url = request.args.get("url", "").strip() + f_min_size = request.args.get("min_size", type=int) + f_max_size = request.args.get("max_size", type=int) + f_min_elapsed = request.args.get("min_elapsed", type=int) + f_max_elapsed = request.args.get("max_elapsed", type=int) + f_start = request.args.get("start_time", "").strip() + f_end = request.args.get("end_time", "").strip() + page = max(1, request.args.get("page", 1, type=int)) + per_page = 100 + + instance_id = 0 + try: + inst = _resolve_active_instance() + if inst is not None: + instance_id = inst.id + except Exception: + instance_id = 0 + + base_filters = { + "instance_id": instance_id, + "client": f_client or None, + "method": f_method or None, + "result_code": f_result or None, + "host": f_host or None, + "url": f_url or None, + "min_size": f_min_size, + "max_size": f_max_size, + "min_elapsed": f_min_elapsed, + "max_elapsed": f_max_elapsed, + "start_time": f_start or None, + "end_time": f_end or None, + } + + if use_persistent: + # best-effort sync before serving (skipped if the file is unchanged) + try: + if _should_auto_sync(instance_id): + path = access_log_path() + if os.path.isfile(path): + log_storage.sync_log_to_db(path=path, instance_id=instance_id) + except Exception: + pass + + # count first for accurate totals, then fetch the page slice + total = log_storage.count_logs(base_filters) + page_items = log_storage.query_logs({ + **base_filters, + "limit": per_page, + "offset": (page - 1) * per_page, + }) + # query_logs returns newest-first, so no re-reverse needed + entries = page_items + else: + entries = get_parsed_logs() + f = dict(base_filters) + # legacy filter util uses positional kwargs, not a dict + filtered = log_parser.filter_entries( + entries, + client=f_client or None, method=f_method or None, + result_code=f_result or None, host=f_host or None, url=f_url or None, + min_size=f_min_size, max_size=f_max_size, + min_elapsed=f_min_elapsed, max_elapsed=f_max_elapsed, + limit=10000, + ) + filtered = list(reversed(filtered)) + total = len(filtered) + start = (page - 1) * per_page + entries = filtered[start:start + per_page] + + return render_template("logs.html", + entries=entries, total=total, page=page, + per_page=per_page, + total_pages=(total + per_page - 1) // per_page, + filters={ + "client": f_client, "method": f_method, + "result_code": f_result, "host": f_host, "url": f_url, + "min_size": f_min_size or "", "max_size": f_max_size or "", + "min_elapsed": f_min_elapsed or "", "max_elapsed": f_max_elapsed or "", + "start_time": f_start, "end_time": f_end, + }, + cfg=cfg, + last_sync=log_storage.get_last_sync_at(), + last_sync_path=log_storage.get_last_sync_path(), + use_persistent=use_persistent, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration) + + +@app.route("/logs/sync", methods=["GET", "POST"]) +@login_required +def logs_sync(): + """Manually trigger access.log -> SQLite sync. + + GET: redirect to /logs (avoids accidental GET-driven state change) + POST: imports new lines from access.log and flashes a summary. The + ``?full=1`` query parameter forces a full-history import (10M-line + ceiling) instead of the incremental tail import. + """ + import log_storage + if request.method == "GET": + return redirect(url_for("logs_view")) + full = request.args.get("full") == "1" or request.form.get("full") == "1" + instance_id = 0 + try: + inst = _resolve_active_instance() + if inst is not None: + instance_id = inst.id + except Exception: + instance_id = 0 + path = access_log_path() + if full: + inserted, skipped, err = log_storage.import_full_history( + path=path, instance_id=instance_id, + ) + audit("logs_sync_full", f"path={path} inserted={inserted} skipped={skipped} err={err or ''}") + else: + inserted, skipped, err = log_storage.sync_log_to_db( + path=path, instance_id=instance_id, + ) + audit("logs_sync", f"path={path} inserted={inserted} skipped={skipped} err={err or ''}") + try: + db.session.commit() + except Exception: + pass + if err: + flash(f"同步失败: {err}", "error") + else: + scope = "全量导入" if full else "增量同步" + flash(f"{scope}完成: 新增 {inserted} 条,跳过 {skipped} 条。", "ok") + # If posted via AJAX, return JSON; otherwise redirect back to /logs + if request.headers.get("X-Requested-With") == "XMLHttpRequest": + return jsonify({ + "ok": err is None, + "inserted": inserted, + "skipped": skipped, + "error": err, + "full": full, + "last_sync": log_storage.get_last_sync_at(), + }) + return redirect(url_for("logs_view")) + + +@app.route("/logs/import", methods=["GET", "POST"]) +@login_required +def logs_import(): + if request.method == "POST": + text = request.form.get("log_text", "") + if not text.strip(): + flash("请粘贴日志内容", "error") + return redirect(url_for("logs_import")) + entries = log_parser.parse_lines(text) + if not entries: + flash("未能解析任何有效日志行，请检查格式", "error") + return redirect(url_for("logs_import")) + stats = log_parser.aggregate_stats(entries) + audit("logs_import", f"lines={len(text.splitlines())} parsed={len(entries)}") + db.session.commit() + return render_template("logs_import.html", imported=True, + entries=entries, stats=stats, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration, + raw_lines=text.splitlines()) + return render_template("logs_import.html", imported=False, + entries=None, stats=None, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration) + + +@app.route("/logs/live") +@login_required +def logs_live(): + """Live tail view (auto-refresh).""" + cfg = get_all_config() + n = int(cfg.get("live_tail_lines", 100)) + entries = get_parsed_logs(force=False)[-n:] + entries = list(reversed(entries)) + return render_template("logs_live.html", entries=entries, + cfg=cfg, n=n, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration) + + +# --------------------------------------------------------------------------- +# Routes - service control +# --------------------------------------------------------------------------- + +@app.route("/service") +@login_required +def service_view(): + status = service_status() + return render_template("service.html", status=status, + conf=squid_conf_path(), + access_log=access_log_path(), + cache_log=cache_log_path()) + + +@app.route("/api/service/", methods=["POST"]) +@login_required +def api_service_control(action): + # P0-5: verify per-action confirmation token for dangerous operations + expected = session.pop(f'_svc_token_{action}', None) + sent = request.form.get("confirm_token", "") + if action in ('stop', 'restart') and (not expected or sent != expected): + audit(f"service_{action}_confirm_fail", f"ip={security.client_ip()}") + db.session.commit() + return jsonify({"ok": False, "message": "确认码无效或已过期,请刷新页面"}), 400 + t0 = time.time() + ok, msg = service_control(action) + elapsed = time.time() - t0 + audit(f"service_{action}", f"elapsed={elapsed:.2f}s msg={msg[:300]}") + db.session.commit() + return jsonify({"ok": ok, "message": msg, "status": service_status(), + "elapsed": round(elapsed, 2)}) + + +@app.route("/api/service/status") +@login_required +def api_service_status(): + return jsonify(service_status()) + + +# --------------------------------------------------------------------------- +# Routes - config: paths / app settings +# --------------------------------------------------------------------------- + +@app.route("/config/paths", methods=["GET", "POST"]) +@login_required +def config_paths(): + if request.method == "POST": + keys = ["squid_conf_path", "squid_binary", "systemctl", "squid_service", + "access_log_path", "cache_log_path", + "log_parse_limit", "log_cache_seconds", + "live_tail_lines", "auto_refresh_seconds", + "log_use_persistent", "log_sync_interval"] + for k in keys: + v = request.form.get(k, "").strip() + set_config(k, v) + # Flush in-memory log cache so the next page render reflects the + # new log_use_persistent / log_sync_interval value. + with _log_cache_lock: + _log_cache.clear() + audit("config_paths", "updated paths & app settings") + db.session.commit() + flash("应用配置已保存", "ok") + return redirect(url_for("config_paths")) + return render_template("config_paths.html", cfg=get_all_config()) + + +# --------------------------------------------------------------------------- +# Routes - squid.conf config editor (full structured editor) +# --------------------------------------------------------------------------- + +def get_struct() -> dict: + """Read and parse current squid.conf. Returns default struct on failure.""" + text, err = read_squid_conf() + if err or not text.strip(): + return squid_config.default_struct() + return squid_config.parse_conf(text) + + +def save_struct(struct: dict, raw_extra: str = "") -> tuple[bool, str, str]: + """Generate squid.conf from struct. + Returns (ok, message, content). Does NOT write to disk. + Writing is done via /config/preview + /config/confirm. + """ + try: + content = squid_config.generate_conf(struct, raw_extra) + return True, "OK", content + except Exception as e: + return False, f"生成配置失败: {e}", "" + + +@app.route("/config/main", methods=["GET", "POST"]) +@login_required +def config_main(): + if request.method == "POST": + struct = get_struct() + simple_keys = ["http_port", "https_port", "icp_port", "snmp_port", + "visible_hostname", "unique_hostname", + "cache_mem", "maximum_object_size", + "minimum_object_size", "maximum_object_size_in_memory", + "cache_replacement_policy", "memory_replacement_policy", + "cache_swap_low", "cache_swap_high", + "connect_timeout", "read_timeout", + "client_lifetime", "shutdown_lifetime", + "cache_mgr", "err_html_language", "ftp_user", + "access_log", "cache_log", "cache_store_log", + "pid_filename", "logformat", "debug_options", + "log_ip_on_direct", "buffered_logs", + "dns_nameservers", "dns_timeout", "hosts_file", + "dns_defnames"] + new_simple = {} + for k in simple_keys: + v = request.form.get(k, "").strip() + if v: + # split on whitespace + new_simple[k] = [(v.split(), "")] + else: + new_simple[k] = [] + struct["simple"] = new_simple + raw_extra = request.form.get("raw_extra", "") + ok, msg, content = save_struct(struct, raw_extra) + if not ok: + audit("config_main_preview_fail", msg[:500]) + db.session.commit() + flash(msg, "error") + return redirect(url_for("config_main")) + audit("config_main_preview", f"size={len(content)}") + db.session.commit() + return _show_config_preview(content, "config_main") + struct = get_struct() + simple = {k: " ".join(args[0]) if struct["simple"].get(k) else "" + for k, args_data in struct["simple"].items() + for args in [args_data]} if False else {} + # simpler approach + simple = {} + for k in ["http_port", "https_port", "icp_port", "snmp_port", + "visible_hostname", "unique_hostname", + "cache_mem", "maximum_object_size", "minimum_object_size", + "maximum_object_size_in_memory", "cache_replacement_policy", + "memory_replacement_policy", "cache_swap_low", "cache_swap_high", + "connect_timeout", "read_timeout", "client_lifetime", + "shutdown_lifetime", "cache_mgr", "err_html_language", "ftp_user", + "access_log", "cache_log", "cache_store_log", "pid_filename", + "logformat", "debug_options", "log_ip_on_direct", "buffered_logs", + "dns_nameservers", "dns_timeout", "hosts_file", "dns_defnames"]: + lst = struct["simple"].get(k, []) + if lst and isinstance(lst[0], tuple) and lst[0][0]: + simple[k] = " ".join(lst[0][0]) + else: + simple[k] = "" + return render_template("config_main.html", simple=simple, + squid_conf_path=squid_conf_path()) + + +@app.route("/config/acls", methods=["GET", "POST"]) +@login_required +def config_acls(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "delete": + name = request.form.get("name", "") + struct["acls"] = [a for a in struct["acls"] if a.name != name] + elif action == "add": + name = request.form.get("name", "").strip() + atype = request.form.get("type", "").strip() + values_raw = request.form.get("values", "").strip() + comment = request.form.get("comment", "").strip() + if not name or not atype: + flash("ACL 名称和类型必填", "error") + return redirect(url_for("config_acls")) + values = values_raw.split() + if not values: + flash("至少需要一个值", "error") + return redirect(url_for("config_acls")) + struct["acls"].append(squid_config.Acl( + name=name, type=atype, values=values, comment=comment)) + elif action == "save": + # rebuild ACL list from form fields (id, name, type, values, comment) + ids = request.form.getlist("id[]") + names = request.form.getlist("name[]") + types = request.form.getlist("type[]") + values_list = request.form.getlist("values[]") + comments = request.form.getlist("comment[]") + new_acls = [] + for i in range(len(ids)): + if i < len(names) and names[i].strip(): + vals = (values_list[i] if i < len(values_list) else "").split() + cmnt = comments[i] if i < len(comments) else "" + new_acls.append(squid_config.Acl( + name=names[i].strip(), + type=types[i] if i < len(types) else "src", + values=vals, comment=cmnt)) + struct["acls"] = new_acls + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_acls")) + # Preview before write + return _show_config_preview(content_preview, "config_acls") + struct = get_struct() + return render_template("config_acls.html", acls=struct["acls"], + acl_types=squid_config.ACL_TYPES) + + +@app.route("/config/rules", methods=["GET", "POST"]) +@login_required +def config_rules(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "add": + directive = request.form.get("directive", "http_access") + act = request.form.get("action_type", "allow") + acls_raw = request.form.get("acls", "").strip() + comment = request.form.get("comment", "").strip() + if act not in ("allow", "deny"): + flash("动作必须是 allow 或 deny", "error") + return redirect(url_for("config_rules")) + acls = acls_raw.split() + struct["rules"].append(squid_config.Rule( + action=act, acls=acls, comment=comment, directive=directive)) + elif action == "save": + ids = request.form.getlist("id[]") + directives = request.form.getlist("directive[]") + acts = request.form.getlist("action_type[]") + acls_list = request.form.getlist("acls[]") + comments = request.form.getlist("comment[]") + new_rules = [] + for i in range(len(ids)): + if i < len(directives): + acl_vals = (acls_list[i] if i < len(acls_list) else "").split() + cmnt = comments[i] if i < len(comments) else "" + act = acts[i] if i < len(acts) else "allow" + if act not in ("allow", "deny"): + act = "allow" + new_rules.append(squid_config.Rule( + action=act, acls=acl_vals, comment=cmnt, + directive=directives[i])) + struct["rules"] = new_rules + elif action == "delete": + idx = int(request.form.get("index", "-1")) + if 0 <= idx < len(struct["rules"]): + del struct["rules"][idx] + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_rules")) + # Preview before write + return _show_config_preview(content_preview, "config_rules") + struct = get_struct() + # group rules by directive for display + acls_by_name = {a.name: a for a in struct["acls"]} + return render_template("config_rules.html", rules=struct["rules"], + rule_directives=squid_config.RULE_DIRECTIVES, + acls_by_name=acls_by_name) + + +@app.route("/config/cache", methods=["GET", "POST"]) +@login_required +def config_cache(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "add_dir": + cd_type = request.form.get("type", "ufs") + path = request.form.get("path", "/var/spool/squid").strip() + size = int(request.form.get("size_mb", "100") or "100") + l1 = int(request.form.get("l1", "16") or "16") + l2 = int(request.form.get("l2", "256") or "256") + struct["cache_dirs"].append(squid_config.CacheDir( + type=cd_type, path=path, size_mb=size, l1=l1, l2=l2)) + elif action == "save": + ids = request.form.getlist("id[]") + types = request.form.getlist("type[]") + paths = request.form.getlist("path[]") + sizes = request.form.getlist("size_mb[]") + l1s = request.form.getlist("l1[]") + l2s = request.form.getlist("l2[]") + new_dirs = [] + for i in range(len(ids)): + if i < len(paths) and paths[i].strip(): + new_dirs.append(squid_config.CacheDir( + type=types[i] if i < len(types) else "ufs", + path=paths[i].strip(), + size_mb=int(sizes[i]) if i < len(sizes) and sizes[i].isdigit() else 100, + l1=int(l1s[i]) if i < len(l1s) and l1s[i].isdigit() else 16, + l2=int(l2s[i]) if i < len(l2s) and l2s[i].isdigit() else 256, + )) + struct["cache_dirs"] = new_dirs + elif action == "delete_dir": + idx = int(request.form.get("index", "-1")) + if 0 <= idx < len(struct["cache_dirs"]): + del struct["cache_dirs"][idx] + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_cache")) + # Preview before write + return _show_config_preview(content_preview, "config_cache") + struct = get_struct() + return render_template("config_cache.html", cache_dirs=struct["cache_dirs"]) + + +@app.route("/config/refresh", methods=["GET", "POST"]) +@login_required +def config_refresh(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "add": + rp = squid_config.RefreshPattern() + rp.case_insensitive = request.form.get("case_insensitive") == "on" + rp.regex = request.form.get("regex", ".").strip() + try: + rp.min_minutes = int(request.form.get("min_minutes", "0") or "0") + rp.percent = int(request.form.get("percent", "20") or "20") + rp.max_minutes = int(request.form.get("max_minutes", "4320") or "4320") + except ValueError: + flash("数值格式错误", "error") + return redirect(url_for("config_refresh")) + struct["refresh_patterns"].append(rp) + elif action == "save": + ids = request.form.getlist("id[]") + cis = request.form.getlist("case_insensitive[]") + regexes = request.form.getlist("regex[]") + mins = request.form.getlist("min_minutes[]") + pcts = request.form.getlist("percent[]") + maxs = request.form.getlist("max_minutes[]") + new_rps = [] + for i in range(len(ids)): + if i < len(regexes) and regexes[i].strip(): + rp = squid_config.RefreshPattern( + case_insensitive=(str(i) in cis), + regex=regexes[i].strip(), + min_minutes=int(mins[i]) if i < len(mins) and mins[i].lstrip("-").isdigit() else 0, + percent=int(pcts[i].rstrip("%")) if i < len(pcts) and pcts[i].rstrip("%").lstrip("-").isdigit() else 20, + max_minutes=int(maxs[i]) if i < len(maxs) and maxs[i].lstrip("-").isdigit() else 4320, + ) + new_rps.append(rp) + struct["refresh_patterns"] = new_rps + elif action == "delete": + idx = int(request.form.get("index", "-1")) + if 0 <= idx < len(struct["refresh_patterns"]): + del struct["refresh_patterns"][idx] + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_refresh")) + # Preview before write + return _show_config_preview(content_preview, "config_refresh") + struct = get_struct() + return render_template("config_refresh.html", patterns=struct["refresh_patterns"]) + + +@app.route("/config/auth", methods=["GET", "POST"]) +@login_required +def config_auth(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "add": + scheme = request.form.get("scheme", "basic") + program = request.form.get("program", "").strip() + realm = request.form.get("realm", "Squid Proxy").strip() + try: + children = int(request.form.get("children", "5") or "5") + credentialsttl = int(request.form.get("credentialsttl", "2") or "2") + except ValueError: + children, credentialsttl = 5, 2 + struct["auth_params"].append(squid_config.AuthParam( + scheme=scheme, program=program, realm=realm, + children=children, credentialsttl=credentialsttl)) + elif action == "save": + ids = request.form.getlist("id[]") + schemes = request.form.getlist("scheme[]") + programs = request.form.getlist("program[]") + realms = request.form.getlist("realm[]") + childrens = request.form.getlist("children[]") + ttls = request.form.getlist("credentialsttl[]") + new_aps = [] + for i in range(len(ids)): + if i < len(schemes) and schemes[i].strip(): + try: + ch = int(childrens[i]) if i < len(childrens) and childrens[i].isdigit() else 5 + tt = int(ttls[i]) if i < len(ttls) and ttls[i].isdigit() else 2 + except ValueError: + ch, tt = 5, 2 + new_aps.append(squid_config.AuthParam( + scheme=schemes[i].strip(), + program=(programs[i] if i < len(programs) else "").strip(), + realm=(realms[i] if i < len(realms) else "Squid Proxy").strip(), + children=ch, credentialsttl=tt)) + struct["auth_params"] = new_aps + elif action == "delete": + idx = int(request.form.get("index", "-1")) + if 0 <= idx < len(struct["auth_params"]): + del struct["auth_params"][idx] + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_auth")) + # Preview before write + return _show_config_preview(content_preview, "config_auth") + struct = get_struct() + return render_template("config_auth.html", auth_params=struct["auth_params"]) + + +@app.route("/config/peers", methods=["GET", "POST"]) +@login_required +def config_peers(): + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + if action == "add": + host = request.form.get("host", "").strip() + if not host: + flash("Peer 主机必填", "error") + return redirect(url_for("config_peers")) + ptype = request.form.get("type", "parent") + try: + http_port = int(request.form.get("http_port", "3128") or "3128") + icp_port = int(request.form.get("icp_port", "0") or "0") + except ValueError: + http_port, icp_port = 3128, 0 + options = request.form.get("options", "").strip().split() + struct["cache_peers"].append(squid_config.CachePeer( + host=host, type=ptype, http_port=http_port, + icp_port=icp_port, options=options)) + elif action == "save": + ids = request.form.getlist("id[]") + hosts = request.form.getlist("host[]") + types = request.form.getlist("type[]") + http_ports = request.form.getlist("http_port[]") + icp_ports = request.form.getlist("icp_port[]") + options_list = request.form.getlist("options[]") + new_peers = [] + for i in range(len(ids)): + if i < len(hosts) and hosts[i].strip(): + new_peers.append(squid_config.CachePeer( + host=hosts[i].strip(), + type=types[i] if i < len(types) else "parent", + http_port=int(http_ports[i]) if i < len(http_ports) and http_ports[i].isdigit() else 3128, + icp_port=int(icp_ports[i]) if i < len(icp_ports) and icp_ports[i].isdigit() else 0, + options=(options_list[i] if i < len(options_list) else "").split(), + )) + struct["cache_peers"] = new_peers + elif action == "delete": + idx = int(request.form.get("index", "-1")) + if 0 <= idx < len(struct["cache_peers"]): + del struct["cache_peers"][idx] + content_preview = msg if ok else "" + if not ok: + flash(msg, "error") + return redirect(url_for("config_peers")) + # Preview before write + return _show_config_preview(content_preview, "config_peers") + struct = get_struct() + return render_template("config_peers.html", peers=struct["cache_peers"]) + + +@app.route("/config/raw", methods=["GET", "POST"]) +@login_required +def config_raw(): + if request.method == "POST": + content = request.form.get("content", "") + # Validate first if user opted in + if request.form.get("validate") == "on": + binary = get_config("squid_binary", "squid") + ok, vmsg = squid_config.validate_conf(content, squid_binary=binary) + if not ok: + flash(f"校验失败: {vmsg}", "error") + audit("config_raw_validate_fail", vmsg[:500]) + db.session.commit() + return redirect(url_for("config_raw")) + # Go through preview before writing + return _show_config_preview(content, "config_raw") + content, err = read_squid_conf() + return render_template("config_raw.html", content=content, error=err, + squid_conf_path=squid_conf_path()) + + +@app.route("/api/validate", methods=["POST"]) +@login_required +def api_validate(): + text = request.form.get("content") or (request.get_json(silent=True) or {}).get("content", "") + if not text: + return jsonify({"ok": False, "message": "空内容"}) + binary = get_config("squid_binary", "squid") + ok, msg = squid_config.validate_conf(text, squid_binary=binary) + return jsonify({"ok": ok, "message": msg}) + + +@app.route("/api/reload", methods=["POST"]) +@login_required +def api_reload(): + ok, msg = service_control("reload") + audit("service_reload", msg[:500]) + db.session.commit() + return jsonify({"ok": ok, "message": msg}) + + +# --------------------------------------------------------------------------- +# Audit log +# --------------------------------------------------------------------------- + +@app.route("/audit") +@login_required +def audit_view(): + page = max(1, request.args.get("page", 1, type=int)) + per_page = 100 + q = AuditLog.query.order_by(AuditLog.timestamp.desc()) + total = q.count() + items = q.offset((page - 1) * per_page).limit(per_page).all() + return render_template("audit.html", items=items, total=total, page=page, + per_page=per_page, + total_pages=(total + per_page - 1) // per_page) + + +# --------------------------------------------------------------------------- +# Backups +# --------------------------------------------------------------------------- + +@app.route("/backups") +@login_required +def backups_view(): + files = [] + if os.path.isdir(BACKUP_DIR): + for name in sorted(os.listdir(BACKUP_DIR), reverse=True): + if not name.startswith("squid.conf"): + continue + p = os.path.join(BACKUP_DIR, name) + if os.path.isfile(p): + st = os.stat(p) + files.append({ + "name": name, + "size": st.st_size, + "mtime": datetime.fromtimestamp(st.st_mtime).strftime("%Y-%m-%d %H:%M:%S"), + "path": p, + }) + return render_template("backups.html", files=files) + + +@app.route("/backups/") +@login_required +def backup_download(name): + if "/" in name or ".." in name: + abort(404) + return send_from_directory(BACKUP_DIR, name, as_attachment=True) + + +@app.route("/backups/restore/", methods=["POST"]) +@login_required +def backup_restore(name): + if "/" in name or ".." in name: + abort(404) + src = os.path.join(BACKUP_DIR, name) + if not os.path.isfile(src): + flash("备份不存在", "error") + return redirect(url_for("backups_view")) + # back up current before restore + backup_conf("before-restore") + try: + shutil.copy2(src, squid_conf_path()) + audit("backup_restore", name) + db.session.commit() + flash(f"已从备份 {name} 恢复", "ok") + except OSError as e: + flash(f"恢复失败: {e}", "error") + return redirect(url_for("config_raw")) + + +# --------------------------------------------------------------------------- +# Static uploads (none yet, but reserved) +# --------------------------------------------------------------------------- + + +# --------------------------------------------------------------------------- +# Seed / init +# --------------------------------------------------------------------------- + +def seed_admin(): + if not User.query.filter_by(username="admin").first(): + a = User(username="admin", role="admin") + a.set_password("admin") + db.session.add(a) + db.session.commit() + print("[init] seeded admin user (admin/admin)") + + +@app.errorhandler(404) +def not_found(e): + return render_template("error.html", code=404, message="页面不存在"), 404 + + +@app.errorhandler(403) +def forbidden(e): + return render_template("error.html", code=403, message="无权限"), 403 + + +@app.errorhandler(500) +def server_error(e): + return render_template("error.html", code=500, message="服务器内部错误"), 500 + + +# --------------------------------------------------------------------------- +# Entry points +# --------------------------------------------------------------------------- + +if __name__ == "__main__": + with app.app_context(): + db.create_all() + seed_admin() + app.run(host="0.0.0.0", port=5200, debug=True) + + +# --------------------------------------------------------------------------- +# P0-4: config diff preview +# --------------------------------------------------------------------------- + +import diff_tool + +@app.route("/config/preview", methods=["GET", "POST"]) +@login_required +def config_preview(): + """Show unified diff of the proposed config vs current. User must POST again to confirm.""" + import base64 + if request.method == "POST": + pending = request.form.get("_pending_content", "") + if pending: + session["_pending_conf"] = base64.b64encode(pending.encode()).decode() + session["_pending_return"] = request.form.get("return_to", url_for("dashboard")) + pending_b64 = session.pop("_pending_conf", "") + if not pending_b64: + flash("无待保存内容或会话已过期", "error") + return redirect(url_for("dashboard")) + try: + pending = base64.b64decode(pending_b64).decode("utf-8") + except Exception: + flash("内容解析失败", "error") + return redirect(url_for("dashboard")) + return_to = session.pop("_pending_return", url_for("dashboard")) + current, err = read_squid_conf() + if err: + flash(err, "error") + return redirect(return_to) + diff = diff_tool.unified_diff(current, pending) + summary = diff_tool.summarize(diff) + html = diff_tool.render_html(diff) + return render_template("config_preview.html", diff_html=html, summary=summary, + pending_b64=pending_b64, return_to=return_to) + + +@app.route("/config/confirm", methods=["POST"]) +@login_required +def config_confirm(): + """Actually write the config that the user previewed.""" + import base64 + pending_b64 = request.form.get("pending", "") + try: + content = base64.b64decode(pending_b64).decode("utf-8") + except Exception: + flash("内容解析失败", "error") + return redirect(url_for("dashboard")) + return_to = request.form.get("return_to", url_for("dashboard")) + validate = request.form.get("validate") == "1" + ok, msg = write_squid_conf(content, validate=validate) + audit("config_write", msg[:500]) + db.session.commit() + if ok: + flash("配置已应用: " + msg, "ok") + else: + flash("写入失败: " + msg, "error") + return redirect(return_to) + + +# --------------------------------------------------------------------------- +# P0-2: TLS / HTTPS certificate management +# --------------------------------------------------------------------------- +# Imports deferred to avoid touching the existing import block; ssl_certs only +# uses the standard library + the openssl CLI binary. + +import ssl_certs # noqa: E402 (intentionally at end of module per task spec) +import hashlib # noqa: E402 (used only by TLS audit fingerprints) + +SSL_CERT_DIR = os.path.join(BASE_DIR, "ssl_certs") + + +@app.route("/config/tls", methods=["GET"]) +@login_required +def config_tls(): + """List all certificates stored under ``ssl_certs/`` with metadata.""" + ok, msg, items = ssl_certs.list_certificates(SSL_CERT_DIR) + if not ok: + flash(f"无法读取证书目录: {msg}", "error") + items = [] + return render_template( + "config_tls.html", + certs=items, + cert_dir=SSL_CERT_DIR, + openssl_info=ssl_certs.is_openssl_available(), + ) + + +@app.route("/config/tls/generate", methods=["POST"]) +@login_required +def config_tls_generate(): + """Generate a self-signed certificate via openssl.""" + common_name = request.form.get("common_name", "").strip() + days_raw = request.form.get("days", "365").strip() or "365" + try: + days = int(days_raw) + except ValueError: + flash("有效期必须是整数", "error") + return redirect(url_for("config_tls")) + + ok, msg, data = ssl_certs.generate_self_signed(common_name, days, SSL_CERT_DIR) + if ok: + # Audit record: name + SHA-256 only, NEVER private key contents. + fp = data.get("fingerprint", "") or hashlib.sha256( + data.get("name", "").encode() + ).hexdigest() + cn_audit = data.get("cn", common_name) or common_name + days_audit = data.get("days", days) + name_audit = data.get("name", "") + audit( + "tls_generate", + f'name={name_audit} cn={cn_audit} days={days_audit} sha256={fp}', + ) + db.session.commit() + flash( + f"已生成: {msg} ({cn_audit}, {days_audit} 天)", + "ok", + ) + else: + audit("tls_generate_fail", f"cn={common_name} days={days} err={msg[:120]}") + db.session.commit() + flash(f"生成失败: {msg}", "error") + return redirect(url_for("config_tls")) + + +@app.route("/config/tls/upload", methods=["POST"]) +@login_required +def config_tls_upload(): + """Accept an uploaded PEM/CRT/KEY file and save it under ssl_certs/.""" + f = request.files.get("certfile") + if not f: + flash("未选择文件", "error") + return redirect(url_for("config_tls")) + ok, msg, data = ssl_certs.upload_certificate(f, SSL_CERT_DIR) + if ok: + up_name = data.get("name", "") + up_size = data.get("size", 0) + up_kind = data.get("kind", "") + up_sha = data.get("sha256", "") + audit( + "tls_upload", + f'name={up_name} size={up_size} kind={up_kind} sha256={up_sha}', + ) + db.session.commit() + flash(msg, "ok") + else: + fname = getattr(f, "filename", "") or "" + audit("tls_upload_fail", f"filename={fname} err={msg[:120]}") + db.session.commit() + flash(f"上传失败: {msg}", "error") + return redirect(url_for("config_tls")) + + +@app.route("/config/tls/delete", methods=["POST"]) +@login_required +def config_tls_delete(): + """Delete a single cert / key file by name.""" + name = (request.form.get("name") or "").strip() + ok, msg, data = ssl_certs.delete_certificate(name, SSL_CERT_DIR) + if ok: + del_name = data.get("name", name) or name + del_size = data.get("size", 0) + del_sha = data.get("sha256", "") + audit( + "tls_delete", + f'name={del_name} size={del_size} sha256={del_sha}', + ) + db.session.commit() + flash(msg, "ok") + else: + audit("tls_delete_fail", f"name={name} err={msg[:120]}") + db.session.commit() + flash(f"删除失败: {msg}", "error") + return redirect(url_for("config_tls")) + + +# --------------------------------------------------------------------------- +# P0-5: Service control confirmation tokens +# --------------------------------------------------------------------------- + +import secrets as _secrets + +def confirm_tokens(): + """Generate per-action confirmation tokens for the service page. + + Stores them in the session so /api/service/ can verify + that the user actually came through the rendered service page + (mitigates trivial CSRF / script-kiddie attempts). + """ + tokens = {} + for action in ('start', 'stop', 'restart', 'reload'): + tokens[action] = _secrets.token_urlsafe(8) + session[f'_svc_token_{action}'] = tokens[action] + return tokens + + +@app.context_processor +def _service_confirm_globals(): + return { + "confirm_tokens": confirm_tokens, + } + + +# --------------------------------------------------------------------------- +# P0-3: SSL Bump (HTTPS interception) configuration UI +# --------------------------------------------------------------------------- + +SSL_BUMP_SIMPLE_KEYS = ( + "sslcrtd_program", + "sslcrtd_children", + "sslproxy_ca_certfile", + "sslproxy_ca_keyfile", +) + + +def _ssl_bump_simple_to_form(struct: dict) -> dict: + """Convert struct['simple'] ssl_bump entries into form-string form. + + Returns dict like {key: "arg1 arg2 arg3", ...}. + """ + out = {} + for k in SSL_BUMP_SIMPLE_KEYS: + lst = struct.get("simple", {}).get(k, []) + if lst and isinstance(lst[0], tuple) and lst[0][0]: + out[k] = " ".join(lst[0][0]) + else: + out[k] = "" + return out + + +def _ssl_bump_form_to_simple(form_values: dict) -> dict: + """Parse the four sslcrtd/sslproxy_ca form fields into struct simple dict. + + Returns: {key: [(args_list, ""), ...] or [] } + """ + out = {} + for k in SSL_BUMP_SIMPLE_KEYS: + raw = (form_values.get(k) or "").strip() + if raw: + out[k] = [(raw.split(), "")] + else: + out[k] = [] + return out + + +@app.route("/config/ssl_bump", methods=["GET", "POST"]) +@login_required +def config_ssl_bump(): + """SSL Bump (HTTPS interception) rule editor. + + GET - render the rules table + add form + sslcrtd settings form + POST action=add - append a new SslBumpRule + POST action=save - replace entire list from form rows + POST action=delete - drop a rule by index + POST action=generate-cert - stub: emit a self-signed CA into ssl_certs/ + """ + if request.method == "POST": + action = request.form.get("action", "save") + struct = get_struct() + + if action == "add": + act = request.form.get("action_type", "peek").strip() + if act not in squid_config.SSL_BUMP_ACTIONS: + flash(f"未知动作: {act}（合法值: {', '.join(squid_config.SSL_BUMP_ACTIONS)}）", "error") + return redirect(url_for("config_ssl_bump")) + acls_raw = request.form.get("acls", "").strip() + comment = request.form.get("comment", "").strip() + acls = acls_raw.split() + struct.setdefault("ssl_bump_rules", []) + struct["ssl_bump_rules"].append(squid_config.SslBumpRule( + action=act, + acls=acls, + step=len(struct["ssl_bump_rules"]) + 1, + comment=comment, + )) + audit("config_sslbump_add", f"action={act} acls={acls}") + db.session.commit() + + elif action == "save": + # Replace ssl_bump_rules list using form arrays, AND persist the + # four sslcrtd/sslproxy_ca_* simple fields alongside the rules. + ids = request.form.getlist("id[]") + actions = request.form.getlist("action_type[]") + acls_list = request.form.getlist("acls[]") + comments = request.form.getlist("comment[]") + new_rules = [] + for i in range(len(ids)): + act = (actions[i] if i < len(actions) else "peek").strip() or "peek" + if act not in squid_config.SSL_BUMP_ACTIONS: + act = "peek" + acl_vals = (acls_list[i] if i < len(acls_list) else "").split() + cmnt = comments[i] if i < len(comments) else "" + new_rules.append(squid_config.SslBumpRule( + action=act, + acls=acl_vals, + step=i + 1, + comment=cmnt, + )) + struct["ssl_bump_rules"] = new_rules + + # simple sslcrtd / sslproxy_ca_* fields + new_simple_updates = _ssl_bump_form_to_simple(request.form) + simple = struct.setdefault("simple", {}) + simple.update(new_simple_updates) + audit("config_sslbump_save", f"rules={len(new_rules)}") + db.session.commit() + + elif action == "delete": + try: + idx = int(request.form.get("index", "-1")) + except (TypeError, ValueError): + idx = -1 + rules = struct.setdefault("ssl_bump_rules", []) + if 0 <= idx < len(rules): + del rules[idx] + # renumber steps + for i, r in enumerate(rules, start=1): + r.step = i + audit("config_sslbump_delete", f"index={idx}") + db.session.commit() + else: + flash("无效的规则索引", "error") + return redirect(url_for("config_ssl_bump")) + + elif action == "generate-cert": + # Optional: emit a self-signed CA cert under ssl_certs/ that admins + # can later wire in via sslproxy_ca_certfile / sslproxy_ca_keyfile. + cn = (request.form.get("common_name") or "").strip() or "Squid Bump CA" + try: + days = int(request.form.get("days", "3650")) + except (TypeError, ValueError): + days = 3650 + if days < 1: + days = 1 + if days > 36500: + days = 36500 + try: + import ssl_certs as _ssl_certs + ok, msg, data = _ssl_certs.generate_self_signed( + common_name=cn, days=days, + cert_dir=os.path.join(BASE_DIR, "ssl_certs"), + ) + except Exception as e: # pragma: no cover - defensive + ok, msg = False, f"证书生成异常: {e}" + if ok: + audit("config_sslbump_gencert", + f"cn={cn} days={days} cert={data.get('cert_path','')}") + db.session.commit() + flash(f"CA 证书已生成: {data.get('cert_path','')} (CN={cn})", "ok") + else: + audit("config_sslbump_gencert_fail", f"cn={cn} msg={msg[:160]}") + db.session.commit() + flash(f"证书生成失败: {msg}", "error") + return redirect(url_for("config_ssl_bump")) + + # Persist via the standard preview path (3-tuple from save_struct). + ok, msg, content = save_struct(struct) + if not ok: + audit("config_sslbump_fail", msg[:500]) + db.session.commit() + flash(msg, "error") + return redirect(url_for("config_ssl_bump")) + audit("config_sslbump_preview", f"size={len(content)}") + db.session.commit() + return _show_config_preview(content, "config_ssl_bump") + + struct = get_struct() + acls_by_name = {a.name: a for a in struct.get("acls", [])} + simple = _ssl_bump_simple_to_form(struct) + return render_template( + "config_ssl_bump.html", + rules=struct.get("ssl_bump_rules", []), + simple=simple, + actions=squid_config.SSL_BUMP_ACTIONS, + default_example=squid_config.SSL_BUMP_DEFAULT_EXAMPLE, + acls_by_name=acls_by_name, + ) + + +# --------------------------------------------------------------------------- +# Routes - P1-1: log status & logrotate management +# --------------------------------------------------------------------------- + +# Local import to keep the existing top-of-file imports untouched. +import log_rotate # noqa: E402 + + +def _collect_log_paths() -> list[str]: + """Pull the log paths Squid Manager knows about, in display order. + + Reads from get_all_config() so we stay in sync with the path-settings UI. + Empty / blank entries are skipped. cache_store_log is included if set. + """ + cfg = get_all_config() + paths: list[str] = [] + for key in ("access_log_path", "cache_log_path", "cache_store_log"): + v = (cfg.get(key) or "").strip() + if v: + paths.append(v) + return paths + + +@app.route("/ops/logs") +@login_required +def ops_logs(): + """Show size/mtime/age/writability for every known Squid log file.""" + paths = _collect_log_paths() + entries = [log_rotate.get_log_file_info(p) for p in paths] + total_bytes = sum(e.get("size_bytes", 0) or 0 for e in entries if e.get("exists")) + return render_template( + "ops_logs.html", + entries=entries, + total_bytes=total_bytes, + total_human=log_rotate._human_size(total_bytes), + binary=get_config("squid_binary", "squid"), + logrotate_available=log_rotate.is_logrotate_available(), + warn_bytes=log_rotate.SIZE_WARN_BYTES, + crit_bytes=log_rotate.SIZE_CRIT_BYTES, + ) + + +@app.route("/ops/logs/rotate", methods=["POST"]) +@login_required +def ops_logs_rotate(): + """Force ``squid -k rotate`` to rotate Squid's logs right now.""" + binary = get_config("squid_binary", "squid") + ok, msg = log_rotate.trigger_squid_rotate(binary=binary) + audit("logs_rotate", f"binary={binary} ok={ok} msg={msg[:300]}") + db.session.commit() + if ok: + flash(f"日志轮转已触发: {msg}", "ok") + else: + flash(f"日志轮转失败: {msg}", "error") + return redirect(url_for("ops_logs")) + + +@app.route("/ops/logrotate", methods=["GET"]) +@login_required +def ops_logrotate(): + """Display the active /etc/logrotate.d/squid file (if any) + install form.""" + cfg = get_all_config() + default_size = "100M" + default_count = 7 + install_size = (cfg.get("logrotate_size") or default_size).strip() or default_size + install_count = (cfg.get("logrotate_count") or str(default_count)).strip() or str(default_count) + install_compress = (cfg.get("logrotate_compress") or "1").strip() not in ("0", "false", "no", "") + + paths = _collect_log_paths() + ok, content, msg = log_rotate.read_logrotate_config() + is_installed = ok and bool(content.strip()) + preview = "" + if paths: + preview = log_rotate.generate_logrotate_config( + log_paths=paths, + compress=install_compress, + rotate_count=int(install_count), + rotate_size=install_size, + postrotate_cmd=f"{get_config('squid_binary', 'squid')} -k rotate", + ) + return render_template( + "ops_logrotate.html", + cfg=cfg, + paths=paths, + install_size=install_size, + install_count=install_count, + install_compress=install_compress, + logrotate_available=log_rotate.is_logrotate_available(), + logrotate_path=log_rotate.LOGROTATE_PATH, + is_installed=is_installed, + current_content=content if is_installed else "", + current_msg=msg, + preview=preview, + ) + + +@app.route("/ops/logrotate/install", methods=["POST"]) +@login_required +def ops_logrotate_install(): + """Write a freshly-generated /etc/logrotate.d/squid file.""" + paths = _collect_log_paths() + if not paths: + audit("logrotate_install_fail", "no log paths configured") + db.session.commit() + flash("未在「路径设置」里配置任何日志路径,无法生成配置。", "error") + return redirect(url_for("ops_logrotate")) + + rotate_size = (request.form.get("rotate_size") or "100M").strip() or "100M" + raw_count = (request.form.get("rotate_count") or "7").strip() + try: + rotate_count = max(1, min(365, int(raw_count))) + except ValueError: + rotate_count = 7 + compress = (request.form.get("compress") or "0").strip() not in ("0", "", "false", "no") + postrotate_cmd = f"{get_config('squid_binary', 'squid')} -k rotate" + + # Persist chosen settings so the user doesn't re-enter them next time. + set_config("logrotate_size", rotate_size) + set_config("logrotate_count", str(rotate_count)) + set_config("logrotate_compress", "1" if compress else "0") + + config_content = log_rotate.generate_logrotate_config( + log_paths=paths, + compress=compress, + rotate_count=rotate_count, + rotate_size=rotate_size, + postrotate_cmd=postrotate_cmd, + ) + ok, msg = log_rotate.install_logrotate(config_content) + audit( + "logrotate_install", + f"ok={ok} size={rotate_size} count={rotate_count} compress={compress} " + f"paths={len(paths)} msg={msg[:200]}", + ) + db.session.commit() + if ok: + flash(f"logrotate 配置已写入: {msg}", "ok") + else: + flash(f"写入失败: {msg}", "error") + return redirect(url_for("ops_logrotate")) + + +@app.route("/ops/logrotate/uninstall", methods=["POST"]) +@login_required +def ops_logrotate_uninstall(): + """Delete /etc/logrotate.d/squid.""" + ok, msg = log_rotate.uninstall_logrotate() + audit("logrotate_uninstall", f"ok={ok} msg={msg[:200]}") + db.session.commit() + if ok: + flash(f"已删除 logrotate 配置: {msg}", "ok") + else: + flash(f"删除失败: {msg}", "error") + return redirect(url_for("ops_logrotate")) + + +# --------------------------------------------------------------------------- +# Routes - P1-2: Alert rule engine (metrics + thresholds) +# --------------------------------------------------------------------------- + +# Lazy import keeps the existing top-of-file imports untouched and avoids a +# circular reference (alerts.py also lazy-imports `app` for audit writes). +import alerts as _alerts_mod # noqa: E402 + +# Where to persist the rule list. Lives under instance/ so it's not part of +# squid.conf backups and survives restarts. +_ALERTS_RULES_PATH = os.path.join(INSTANCE_DIR, "alert_rules.json") + + +def _alerts_path() -> str: + return _ALERTS_RULES_PATH + + +def _rule_from_form(name: str = "", metric: str = "", operator: str = "", + threshold: str = "", window_minutes: str = "", + cooldown_minutes: str = "", severity: str = "", + enabled: str = "", notify_webhook: str = "") -> "_alerts_mod.AlertRule": + """Build an AlertRule from raw form values; coerce/validate so a bad + hand-typed form doesn't corrupt the persisted rules file.""" + try: + thr = float(threshold or 0.0) + except (TypeError, ValueError): + thr = 0.0 + try: + wm = max(1, int(window_minutes or 5)) + except (TypeError, ValueError): + wm = 5 + try: + cm = max(0, int(cooldown_minutes or 30)) + except (TypeError, ValueError): + cm = 30 + sev = severity if severity in _alerts_mod.SEVERITIES else "warning" + op = operator if operator in _alerts_mod.OPERATORS else "gt" + return _alerts_mod.AlertRule( + name=name or "未命名规则", + metric=metric if metric in _alerts_mod.METRICS else "5xx_rate", + operator=op, + threshold=thr, + window_minutes=wm, + enabled=str(enabled) not in ("0", "", "false", "False"), + cooldown_minutes=cm, + severity=sev, + notify_webhook=(notify_webhook or "").strip(), + ) + + +@app.route("/alerts") +@login_required +def alerts_view(): + """List all rules + snapshot of their current values and trigger state.""" + rules = _alerts_mod.load_rules(_alerts_path()) + try: + status = _alerts_mod.evaluate_rules( + get_parsed_logs(), rules, + cache_dir=get_config("cache_dir", "/var/spool/squid"), + ) + except Exception as exc: + status = [] + flash(f"评估规则失败: {exc}", "error") + status_map = {idx: s for idx, s in enumerate(status)} + return render_template( + "alerts.html", + rules=rules, + current_status=status, + status_map=status_map, + metrics=list(_alerts_mod.METRICS.items()), + operators=list(_alerts_mod.OPERATORS.items()), + severities=_alerts_mod.SEVERITIES, + ) + + +@app.route("/alerts/add", methods=["POST"]) +@login_required +def alerts_add(): + """Append a single rule and persist.""" + rule = _rule_from_form( + name=request.form.get("name", "").strip(), + metric=request.form.get("metric", ""), + operator=request.form.get("operator", "gt"), + threshold=request.form.get("threshold", ""), + window_minutes=request.form.get("window_minutes", ""), + cooldown_minutes=request.form.get("cooldown_minutes", ""), + severity=request.form.get("severity", "warning"), + enabled=request.form.get("enabled", "1"), + notify_webhook=request.form.get("notify_webhook", ""), + ) + rules = _alerts_mod.load_rules(_alerts_path()) + rules.append(rule) + try: + _alerts_mod.save_rules(rules, _alerts_path()) + audit("alert_add", f"name={rule.name} metric={rule.metric} " + f"threshold={rule.threshold} op={rule.operator}") + db.session.commit() + flash(f"已添加告警规则: {rule.name}", "ok") + except Exception as exc: + audit("alert_add_fail", f"msg={str(exc)[:200]}") + db.session.commit() + flash(f"添加失败: {exc}", "error") + return redirect(url_for("alerts_view")) + + +@app.route("/alerts/save", methods=["POST"]) +@login_required +def alerts_save(): + """Bulk-save all rules from the in-table editor. + + Expects parallel arrays: name[], metric[], operator[], threshold[], + window_minutes[], cooldown_minutes[], severity[], enabled[], and + optional notify_webhook[]. Missing notify_webhook is treated as "". + Rows whose name is blank are skipped (operator removed that row in UI). + """ + form = request.form + names = form.getlist("name[]") + metrics = form.getlist("metric[]") + operators = form.getlist("operator[]") + thresholds = form.getlist("threshold[]") + windows = form.getlist("window_minutes[]") + cooldowns = form.getlist("cooldown_minutes[]") + severities = form.getlist("severity[]") + enableds = form.getlist("enabled[]") # only checked rows appear + webhooks = form.getlist("notify_webhook[]") + + # Maps to allow zip() even if one field is shorter. + def at(lst, i, default=""): + return lst[i] if i < len(lst) else default + + rules = [] + for i, name in enumerate(names): + n = (name or "").strip() + if not n: + continue + rules.append(_rule_from_form( + name=n, + metric=at(metrics, i, "5xx_rate"), + operator=at(operators, i, "gt"), + threshold=at(thresholds, i, "0"), + window_minutes=at(windows, i, "5"), + cooldown_minutes=at(cooldowns, i, "30"), + severity=at(severities, i, "warning"), + # If the row exists in the list, it's checked; missing = off. + enabled=("1" if n in enableds else "0"), + notify_webhook=at(webhooks, i, ""), + )) + try: + _alerts_mod.save_rules(rules, _alerts_path()) + audit("alert_save", f"count={len(rules)}") + db.session.commit() + flash(f"已保存 {len(rules)} 条告警规则", "ok") + except Exception as exc: + audit("alert_save_fail", f"msg={str(exc)[:200]}") + db.session.commit() + flash(f"保存失败: {exc}", "error") + return redirect(url_for("alerts_view")) + + +@app.route("/alerts/delete", methods=["POST"]) +@login_required +def alerts_delete(): + """Delete a rule by name. If multiple rules share a name, only the first + match is removed - the UI normally shows unique names.""" + name = (request.form.get("name") or "").strip() + if not name: + flash("缺少规则名称", "error") + return redirect(url_for("alerts_view")) + rules = _alerts_mod.load_rules(_alerts_path()) + before = len(rules) + rules = [r for r in rules if r.name != name] + if len(rules) == before: + flash(f"未找到规则: {name}", "error") + return redirect(url_for("alerts_view")) + try: + _alerts_mod.save_rules(rules, _alerts_path()) + audit("alert_delete", f"name={name}") + db.session.commit() + flash(f"已删除告警规则: {name}", "ok") + except Exception as exc: + audit("alert_delete_fail", f"msg={str(exc)[:200]}") + db.session.commit() + flash(f"删除失败: {exc}", "error") + return redirect(url_for("alerts_view")) + + +@app.route("/alerts/evaluate", methods=["POST"]) +@login_required +def alerts_evaluate(): + """Run all rules once against the current access.log snapshot. + + Returns JSON so the operator can plug this into a cron job or + integrate with an external monitor. When called from the UI we + still flash a summary and bounce back to /alerts so the user + sees what happened. + """ + rules = _alerts_mod.load_rules(_alerts_path()) + try: + status = _alerts_mod.evaluate_rules( + get_parsed_logs(), rules, + cache_dir=get_config("cache_dir", "/var/spool/squid"), + ) + except Exception as exc: + if request.headers.get("Accept", "").startswith("application/json") \ + or request.args.get("format") == "json": + return jsonify({"ok": False, "error": str(exc), + "triggered": [], "evaluated": 0}), 500 + flash(f"评估失败: {exc}", "error") + return redirect(url_for("alerts_view")) + triggered = [ + { + "name": s["rule"].name, + "metric": s["rule"].metric, + "value": s["value"], + "threshold": s["threshold"], + "operator": s["operator"], + "severity": s["severity"], + "message": s["message"], + "ts": s["ts"], + } + for s in status if s["triggered"] + ] + audit("alert_evaluate", f"rules={len(rules)} triggered={len(triggered)}") + db.session.commit() + payload = { + "ok": True, + "evaluated": len(status), + "triggered_count": len(triggered), + "triggered": triggered, + } + if request.headers.get("Accept", "").startswith("application/json") \ + or request.args.get("format") == "json": + return jsonify(payload) + if triggered: + flash(f"评估完成: {len(triggered)} 条规则触发,请查看告警列表", "warning") + else: + flash(f"评估完成: {len(rules)} 条规则均未触发", "ok") + return redirect(url_for("alerts_view")) + + +@app.route("/alerts/history") +@login_required +def alerts_history(): + """Show recent alert trigger events from the audit_log table.""" + triggered = (AuditLog.query + .filter(AuditLog.action == "alert_triggered") + .order_by(AuditLog.timestamp.desc()) + .limit(200) + .all()) + # Also pull a few support events (add / save / delete / evaluate) so the + # operator can correlate "rule was saved" → "rule then fired". + extras = (AuditLog.query + .filter(AuditLog.action.in_([ + "alert_add", "alert_save", "alert_delete", + "alert_evaluate", "alert_add_fail", "alert_save_fail", + "alert_delete_fail", + ])) + .order_by(AuditLog.timestamp.desc()) + .limit(100) + .all()) + return render_template( + "alerts_history.html", + triggered=triggered, + extras=extras, + ) + + +# --------------------------------------------------------------------------- +# P1-3: Multi-instance management +# --------------------------------------------------------------------------- +# +# Endpoints for managing SquidInstance rows. The active instance (one per +# user session) overrides AppConfig for the six _INSTANCE_KEYS in get_config +# / get_all_config above, so the existing config/* / logs / service routes +# transparently target whichever instance the user has selected. +# +# No new imports, no new dependencies. + +def _coerce_bool(val) -> bool: + if isinstance(val, bool): + return val + if val is None: + return False + s = str(val).strip().lower() + return s in ("1", "true", "yes", "on", "y", "t") + + +def _instance_form_payload() -> dict: + """Extract a SquidInstance-shaped dict from request.form. + + Missing fields fall back to model defaults so partial forms (e.g. quick + add with just a name) still produce a usable row. + """ + def _s(k, default=""): + v = request.form.get(k, None) + if v is None: + return default + return str(v).strip() + + port_raw = request.form.get("ssh_port", "") + try: + port = int(port_raw) if str(port_raw).strip() else 22 + except (TypeError, ValueError): + port = 22 + + return { + "name": _s("name", ""), + "description": _s("description", ""), + "squid_conf_path": _s("squid_conf_path", "/etc/squid/squid.conf"), + "access_log_path": _s("access_log_path", "/var/log/squid/access.log"), + "cache_log_path": _s("cache_log_path", "/var/log/squid/cache.log"), + "squid_binary": _s("squid_binary", "squid"), + "systemctl": _s("systemctl", "systemctl"), + "squid_service": _s("squid_service", "squid"), + "ssh_host": _s("ssh_host", ""), + "ssh_user": _s("ssh_user", ""), + "ssh_port": port, + "ssh_key_path": _s("ssh_key_path", ""), + "is_local": _coerce_bool(request.form.get("is_local", "1")), + "is_active": _coerce_bool(request.form.get("is_active", "")), + } + + +@app.route("/instances") +@login_required +def instances_view(): + """List all managed Squid instances.""" + instances = SquidInstance.query.order_by(SquidInstance.id.asc()).all() + active = _resolve_active_instance() + return render_template( + "instances.html", + instances=instances, + active=active, + ) + + +@app.route("/instances/add", methods=["POST"]) +@login_required +def instances_add(): + """Create a new SquidInstance from the add form.""" + data = _instance_form_payload() + if not data["name"]: + flash("实例名称不能为空", "error") + return redirect(url_for("instances_view")) + if len(data["name"]) > 80: + flash("实例名称过长 (最多 80 字符)", "error") + return redirect(url_for("instances_view")) + existing = SquidInstance.query.filter_by(name=data["name"]).first() + if existing is not None: + flash(f"实例名称已存在: {data['name']}", "error") + return redirect(url_for("instances_view")) + try: + inst = SquidInstance(**data) + if data["is_active"]: + # only one row may carry is_active=True + SquidInstance.query.update({SquidInstance.is_active: False}) + inst.is_active = True + session["active_instance_id"] = inst.id + db.session.add(inst) + audit("instance_add", f"name={inst.name}") + db.session.commit() + flash(f"已添加实例: {inst.name}", "ok") + except Exception as exc: + db.session.rollback() + audit("instance_add_fail", f"name={data['name']} msg={str(exc)[:200]}") + db.session.commit() + flash(f"添加实例失败: {exc}", "error") + return redirect(url_for("instances_view")) + + +@app.route("/instances/save", methods=["POST"]) +@login_required +def instances_save(): + """Bulk-save edits to all instances. + + Each instance submits a row with name_=, description_, + squid_conf_path_, etc. (single underscore between id and field). + We iterate the existing rows and apply the matching form fields. + Rows missing from the form are left untouched (use /instances/delete + to remove). + """ + instances = SquidInstance.query.order_by(SquidInstance.id.asc()).all() + saved = 0 + try: + # Pre-compute new active flag: at most one row may end up active. + active_changed_to = None + for inst in instances: + flag_key = f"is_active_{inst.id}" + if flag_key in request.form: + active_changed_to = inst.id + for inst in instances: + # Field naming convention used by templates/instances.html: + # name_, description_, squid_conf_path_, ... + sid = str(inst.id) + name = request.form.get(f"name_{sid}", "").strip() + if not name: + # empty name => skip this row's edits but keep it as-is + continue + if name != inst.name: + clash = SquidInstance.query.filter( + SquidInstance.name == name, + SquidInstance.id != inst.id, + ).first() + if clash is not None: + flash(f"实例名称冲突 (已被 {clash.name} 占用): {name}", "error") + continue + inst.name = name + inst.description = (request.form.get(f"description_{sid}", "") or "").strip() + inst.squid_conf_path = (request.form.get(f"squid_conf_path_{sid}", "") + or inst.squid_conf_path) + inst.access_log_path = (request.form.get(f"access_log_path_{sid}", "") + or inst.access_log_path) + inst.cache_log_path = (request.form.get(f"cache_log_path_{sid}", "") + or inst.cache_log_path) + inst.squid_binary = (request.form.get(f"squid_binary_{sid}", "") + or inst.squid_binary) + inst.systemctl = (request.form.get(f"systemctl_{sid}", "") + or inst.systemctl) + inst.squid_service = (request.form.get(f"squid_service_{sid}", "") + or inst.squid_service) + inst.ssh_host = (request.form.get(f"ssh_host_{sid}", "") or "").strip() + inst.ssh_user = (request.form.get(f"ssh_user_{sid}", "") or "").strip() + port_raw = request.form.get(f"ssh_port_{sid}", "") + try: + inst.ssh_port = int(port_raw) if str(port_raw).strip() else inst.ssh_port + except (TypeError, ValueError): + pass + inst.ssh_key_path = (request.form.get(f"ssh_key_path_{sid}", "") + or inst.ssh_key_path) + inst.is_local = _coerce_bool(request.form.get(f"is_local_{sid}", "1")) + saved += 1 + # Reconcile the is_active flag - only one row at a time + for inst in instances: + inst.is_active = (inst.id == active_changed_to) + if active_changed_to is not None: + session["active_instance_id"] = active_changed_to + audit("instance_save", f"updated={saved} active={active_changed_to}") + db.session.commit() + flash(f"已保存 {saved} 个实例", "ok") + except Exception as exc: + db.session.rollback() + audit("instance_save_fail", f"msg={str(exc)[:200]}") + db.session.commit() + flash(f"保存失败: {exc}", "error") + return redirect(url_for("instances_view")) + + +@app.route("/instances/delete", methods=["POST"]) +@login_required +def instances_delete(): + """Delete one instance by id.""" + raw_id = request.form.get("id", "") + try: + inst_id = int(raw_id) + except (TypeError, ValueError): + flash("无效的实例 ID", "error") + return redirect(url_for("instances_view")) + inst = db.session.get(SquidInstance, inst_id) + if inst is None: + flash(f"实例不存在: #{inst_id}", "error") + return redirect(url_for("instances_view")) + name = inst.name + try: + # If we're deleting the active one, clear the session pointer. + if session.get("active_instance_id") == inst.id: + session.pop("active_instance_id", None) + db.session.delete(inst) + # If a different row is still flagged is_active=True that's fine; + # otherwise leave it False and let the next request pick one (or none). + audit("instance_delete", f"name={name}") + db.session.commit() + flash(f"已删除实例: {name}", "ok") + except Exception as exc: + db.session.rollback() + audit("instance_delete_fail", f"name={name} msg={str(exc)[:200]}") + db.session.commit() + flash(f"删除失败: {exc}", "error") + return redirect(url_for("instances_view")) + + +@app.route("/instances/activate/", methods=["POST"]) +@login_required +def instances_activate(inst_id: int): + """Switch the current session's active instance.""" + inst = db.session.get(SquidInstance, inst_id) + if inst is None: + flash(f"实例不存在: #{inst_id}", "error") + return redirect(url_for("instances_view")) + try: + # Mark this row active and clear others - keeps DB state consistent + # with the session pointer. + SquidInstance.query.update({SquidInstance.is_active: False}) + inst.is_active = True + session["active_instance_id"] = inst.id + # also bust the per-request cache so the redirect sees the new value + g.pop("active_instance", None) + audit("instance_activate", f"name={inst.name}") + db.session.commit() + flash(f"已切换到实例: {inst.name}", "ok") + except Exception as exc: + db.session.rollback() + audit("instance_activate_fail", f"id={inst_id} msg={str(exc)[:200]}") + db.session.commit() + flash(f"切换失败: {exc}", "error") + # Redirect to wherever the user came from when possible, default to / + nxt = request.form.get("next") or request.args.get("next") or url_for("dashboard") + # only allow same-app paths + if not nxt.startswith("/"): + nxt = url_for("dashboard") + return redirect(nxt) + + +@app.route("/api/instances/test/") +@login_required +def instances_test(inst_id: int): + """Test an instance: check conf_path readable + log paths exist. + + For local instances this uses os.access / os.path.isfile directly. + For remote (is_local=False) instances we report the same checks as + 'not checked' - SSH-based probing would need paramiko which we don't + want to add as a hard dependency for this MVP endpoint. + """ + inst = db.session.get(SquidInstance, inst_id) + if inst is None: + return jsonify({"ok": False, "error": "instance not found"}), 404 + + checks = [] + if not inst.is_local: + checks.append({"key": "remote", "ok": None, + "detail": "远程实例需要 SSH 凭据 (本接口未探测)"}) + else: + conf = inst.squid_conf_path or "" + if not conf: + checks.append({"key": "squid_conf_path", "ok": False, + "detail": "未配置"}) + else: + ok = os.path.isfile(conf) and os.access(conf, os.R_OK) + checks.append({"key": "squid_conf_path", "ok": ok, + "detail": conf + (" (可读)" if ok else " (不可读或不存在)")}) + + for key in ("access_log_path", "cache_log_path"): + p = getattr(inst, key, "") or "" + if not p: + checks.append({"key": key, "ok": False, "detail": "未配置"}) + else: + ok = os.path.isfile(p) + checks.append({"key": key, "ok": ok, + "detail": p + (" (存在)" if ok else " (不存在)")}) + + overall = all(c["ok"] for c in checks if c["ok"] is not None) + return jsonify({ + "ok": overall, + "instance": {"id": inst.id, "name": inst.name}, + "is_local": inst.is_local, + "checks": checks, + }) + + +# --------------------------------------------------------------------------- +# P1-4: squidclient mgr: real-time performance metrics +# --------------------------------------------------------------------------- + +def _resolve_perf_connection() -> dict: + """Pick squidclient binary, host, port and optional mgr password. + + All values are user-configurable via get_config() but ship with sensible + defaults so a fresh install works out of the box. + """ + return { + "squidclient": get_config("squidclient_binary", "squidclient") or "squidclient", + "host": get_config("squid_mgr_host", "127.0.0.1") or "127.0.0.1", + "port": int(get_config("squid_mgr_port", "3128") or "3128"), + "password": get_config("squid_mgr_password", "") or "", + "timeout": int(get_config("squid_mgr_timeout", "5") or "5"), + } + + +@app.route("/performance") +@login_required +def performance_view(): + """Real-time performance dashboard, fed by squidclient mgr:*.""" + conn = _resolve_perf_connection() + summary = squid_mgr.get_performance_summary(**conn) + # Lightweight breadcrumb for the sidebar active state + g.page_title = "实时性能" + audit("performance_view", f"ok={summary.get('fetched_ok')}") + db.session.commit() + return render_template( + "performance.html", + summary=summary, + conn=conn, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration, + ) + + +@app.route("/performance/refresh", methods=["POST"]) +@login_required +def performance_refresh(): + """AJAX endpoint: re-run squidclient mgr:* and return JSON for the UI.""" + conn = _resolve_perf_connection() + summary = squid_mgr.get_performance_summary(**conn) + audit("performance_refresh", f"ok={summary.get('fetched_ok')}") + db.session.commit() + # Strip the raw_info blob from the JSON payload to keep the response + # small; the page only needs it for the
fold-out which is + # fetched lazily below via /performance/raw. + payload = {k: v for k, v in summary.items() if k != "raw_info"} + return jsonify(payload) + + +@app.route("/performance/raw/
") +@login_required +def performance_raw(section: str): + """Return the raw mgr:
output for the fold-out
block. + + Whitelisted to a small set of sections to avoid letting arbitrary callers + tunnel commands through squidclient — squidclient doesn't take arbitrary + file args so this is mostly defense-in-depth. + """ + allowed = {"info", "5min", "counters", "storedir", "io", "menu", "config"} + if section not in allowed: + return jsonify({"ok": False, "error": f"section not allowed: {section}"}), 400 + conn = _resolve_perf_connection() + ok, raw, err = squid_mgr.fetch_mgr_info(mgr=section, **conn) + audit("performance_raw", f"section={section} ok={ok}") + db.session.commit() + return jsonify({"ok": ok, "section": section, "raw": raw, "error": err}) + + +# --------------------------------------------------------------------------- +# P1-5: traffic anomaly detection +# --------------------------------------------------------------------------- + +# Local import - keeps the top-of-file import block stable and means an +# anomaly.py typo can never break module import of app.py. +try: + import anomaly as _anomaly +except Exception: # pragma: no cover - defensive only + _anomaly = None + + +def _parse_threshold_float(name: str, default: float, lo: float = 0.0) -> float: + """Read a numeric threshold from request.args with safe fallback.""" + try: + val = request.args.get(name, type=str) + if val is None or val == "": + return default + f = float(val) + return max(lo, f) + except Exception: + return default + + +def _parse_threshold_int(name: str, default: int, lo: int = 1) -> int: + try: + val = request.args.get(name, type=str) + if val is None or val == "": + return default + return max(lo, int(float(val))) + except Exception: + return default + + +def _compute_anomaly_summary( + entries, + spike_ratio: float = 3.0, + large_mb: float = 100.0, + small_kb: float = 1.0, + min_small_req: int = 100, +) -> dict: + """Build a summary honoring caller-supplied thresholds; never raises.""" + if _anomaly is None: + return { + "anomalous_clients": [], "large_requests": [], + "high_freq_small": [], "new_clients": [], + "total_anomalies": 0, "thresholds": {}, + "header_stats": {}, + } + try: + # Re-use the module-level analyzer but override thresholds + large = _anomaly.detect_large_requests( + entries, size_threshold_mb=large_mb, limit=100) + small = _anomaly.detect_high_freq_small( + entries, size_threshold_kb=small_kb, + min_requests=min_small_req, limit=50) + new_clients = _anomaly.detect_new_clients(entries, limit=50) + # For spikes we need to filter by the requested ratio; run the + # module default then post-filter to avoid touching anom.py + spikes_all = _anomaly.detect_client_anomalies(entries) + spikes = [ + a for a in spikes_all + if ( + a.get("ratio") == float("inf") + or ( + isinstance(a.get("ratio"), (int, float)) + and a["ratio"] > spike_ratio + ) + ) + ] + # re-label severities against caller threshold + for a in spikes: + r = a.get("ratio", 0) + if r == float("inf") or (isinstance(r, (int, float)) and r > spike_ratio * 3): + # treat anything > 3x the warn threshold as critical + # when warn threshold was raised by the user + a["severity"] = "warning" if isinstance(r, (int, float)) and r <= spike_ratio * 3 else "critical" + else: + a["severity"] = "warning" + return { + "anomalous_clients": spikes, + "large_requests": large, + "high_freq_small": small, + "new_clients": new_clients, + "total_anomalies": len(spikes) + len(large) + len(small) + len(new_clients), + "thresholds": { + "spike_warn_ratio": spike_ratio, + "spike_crit_ratio": max(spike_ratio * 3, 10.0), + "large_size_mb": large_mb, + "small_size_kb": small_kb, + "min_small_requests": min_small_req, + }, + "header_stats": {}, + } + except Exception: + return { + "anomalous_clients": [], "large_requests": [], + "high_freq_small": [], "new_clients": [], + "total_anomalies": 0, "thresholds": {}, + "header_stats": {}, + } + + +def _build_anomaly_summary(entries) -> dict: + """Build the full summary using request-derived thresholds.""" + spike_ratio = _parse_threshold_float("spike_ratio", 3.0, lo=1.0) + large_mb = _parse_threshold_float("large_mb", 100.0, lo=1.0) + small_kb = _parse_threshold_float("small_kb", 1.0, lo=0.1) + min_small_req = _parse_threshold_int("min_small_req", 100, lo=1) + summary = _compute_anomaly_summary( + entries, + spike_ratio=spike_ratio, + large_mb=large_mb, + small_kb=small_kb, + min_small_req=min_small_req, + ) + # header stats + try: + if entries: + n = len(entries) + cut = max(1, int(n * 0.2)) + current = entries[cut:] + summary["header_stats"] = { + "current_window_count": len(current), + "current_window_clients": len({(e.get("client") if isinstance(e, dict) else "") for e in current}), + } + except Exception: + summary.setdefault("header_stats", {}) + return summary + + +@app.route("/anomalies") +@login_required +def anomalies_view(): + """Render the anomaly dashboard page.""" + try: + entries = get_parsed_logs() + except Exception: + entries = [] + summary = _build_anomaly_summary(entries) + g.page_title = "流量异常" + try: + audit("anomalies_view", f"anomalies={summary.get('total_anomalies', 0)}") + db.session.commit() + except Exception: + pass + return render_template( + "anomalies.html", + summary=summary, + thresholds=summary.get("thresholds", {}), + entries_count=len(entries), + ) + + +@app.route("/anomalies/refresh", methods=["POST"]) +@login_required +def anomalies_refresh(): + """AJAX endpoint: force re-read of access.log and return anomaly counts.""" + try: + entries = get_parsed_logs(force=True) + except Exception: + entries = [] + summary = _build_anomaly_summary(entries) + try: + audit("anomalies_refresh", f"anomalies={summary.get('total_anomalies', 0)}") + db.session.commit() + except Exception: + pass + return jsonify({ + "ok": True, + "total_anomalies": summary.get("total_anomalies", 0), + "anomalous_clients": len(summary.get("anomalous_clients", [])), + "large_requests": len(summary.get("large_requests", [])), + "high_freq_small": len(summary.get("high_freq_small", [])), + "new_clients": len(summary.get("new_clients", [])), + }) + + +@app.route("/api/anomalies") +@login_required +def api_anomalies(): + """JSON endpoint: full anomaly summary, suitable for external integrations.""" + try: + entries = get_parsed_logs() + except Exception: + entries = [] + summary = _build_anomaly_summary(entries) + try: + audit("api_anomalies", f"anomalies={summary.get('total_anomalies', 0)}") + db.session.commit() + except Exception: + pass + return jsonify(summary) + + +# --------------------------------------------------------------------------- +# P2-2: Export endpoints (CSV / JSON) +# --------------------------------------------------------------------------- +# These routes share filter parameters with the corresponding view pages +# (logs, anomalies, audit) so the UI can hand off "the same query, but +# download the result" by passing the existing GET query-string through. +# +# All routes are GET-only, login-protected, and stream the file as an +# attachment so browsers save instead of rendering inline. +# --------------------------------------------------------------------------- + +# Local import - keeps the top of this module unchanged. +import exporters # noqa: E402 + + +def _export_logs_payload(): + """Build (entries, filters) for export, reusing ``logs_view`` logic. + + Mirrors the persistent / legacy branching in :func:`logs_view` so + exports stay consistent with what the user sees on the page. + """ + cfg = get_all_config() + use_persistent = cfg.get("log_use_persistent", "1") not in ("0", "false", "False") + + f_client = request.args.get("client", "").strip() + f_method = request.args.get("method", "").strip() + f_result = request.args.get("result_code", "").strip() + f_host = request.args.get("host", "").strip() + f_url = request.args.get("url", "").strip() + f_min_size = request.args.get("min_size", type=int) + f_max_size = request.args.get("max_size", type=int) + f_min_elapsed = request.args.get("min_elapsed", type=int) + f_max_elapsed = request.args.get("max_elapsed", type=int) + f_start = request.args.get("start_time", "").strip() + f_end = request.args.get("end_time", "").strip() + + instance_id = 0 + try: + inst = _resolve_active_instance() + if inst is not None: + instance_id = inst.id + except Exception: + instance_id = 0 + + base_filters = { + "instance_id": instance_id, + "client": f_client or None, + "method": f_method or None, + "result_code": f_result or None, + "host": f_host or None, + "url": f_url or None, + "min_size": f_min_size, + "max_size": f_max_size, + "min_elapsed": f_min_elapsed, + "max_elapsed": f_max_elapsed, + "start_time": f_start or None, + "end_time": f_end or None, + } + + if use_persistent: + import log_storage + try: + if _should_auto_sync(instance_id): + path = access_log_path() + if os.path.isfile(path): + log_storage.sync_log_to_db(path=path, instance_id=instance_id) + except Exception: + pass + rows = log_storage.query_logs({**base_filters, "limit": 100000}) + return rows, base_filters + + # legacy: parse file tail then filter in memory + entries = get_parsed_logs() + try: + filtered = log_parser.filter_entries( + entries, + client=f_client or None, method=f_method or None, + result_code=f_result or None, host=f_host or None, + url=f_url or None, + min_size=f_min_size, max_size=f_max_size, + min_elapsed=f_min_elapsed, max_elapsed=f_max_elapsed, + limit=100000, + ) + except Exception: + filtered = [] + return filtered, base_filters + + +def _audit_query(): + """Build a SQLAlchemy query for the audit log honoring ``?username`` / ``?action``.""" + q = AuditLog.query.order_by(AuditLog.timestamp.desc()) + f_user = request.args.get("username", "").strip() + f_action = request.args.get("action", "").strip() + if f_user: + q = q.filter(AuditLog.username == f_user) + if f_action: + # case-insensitive substring match - users often pass partial actions + # like "login" expecting to also catch "login_fail" + q = q.filter(AuditLog.action.ilike(f"%{f_action}%")) + return q + + +@app.route("/export/logs.csv") +@login_required +def export_logs_csv(): + """Export the current filtered log view to a CSV file.""" + try: + entries, _filters = _export_logs_payload() + except Exception: + entries = [] + csv_text = exporters.entries_to_csv(entries) + try: + audit("export_logs_csv", f"count={len(entries) if isinstance(entries, list) else 0}") + db.session.commit() + except Exception: + pass + return exporters.csv_response( + csv_text, exporters.timestamped_filename("squid_logs", "csv") + ) + + +@app.route("/export/logs.json") +@login_required +def export_logs_json(): + """Export the current filtered log view to a JSON file.""" + try: + entries, _filters = _export_logs_payload() + except Exception: + entries = [] + json_text = exporters.entries_to_json(entries) + try: + audit("export_logs_json", f"count={len(entries) if isinstance(entries, list) else 0}") + db.session.commit() + except Exception: + pass + return exporters.json_response( + json_text, exporters.timestamped_filename("squid_logs", "json") + ) + + +@app.route("/export/stats.json") +@login_required +def export_stats_json(): + """Export the current KPI summary to a JSON file.""" + try: + stats = get_stats() + except Exception: + stats = {} + # Add a top-level generated_at for downstream consumers. + try: + from datetime import datetime as _dt + if isinstance(stats, dict): + stats = dict(stats) + stats["generated_at"] = _dt.now().isoformat(timespec="seconds") + stats["active_instance"] = active_instance_name() + except Exception: + pass + json_text = exporters.stats_to_json(stats) + try: + audit("export_stats_json") + db.session.commit() + except Exception: + pass + return exporters.json_response( + json_text, exporters.timestamped_filename("squid_stats", "json") + ) + + +@app.route("/export/anomalies.csv") +@login_required +def export_anomalies_csv(): + """Export the current anomaly summary (long-form CSV) to a file.""" + try: + entries = get_parsed_logs() + except Exception: + entries = [] + try: + summary = _build_anomaly_summary(entries) + except Exception: + summary = {} + csv_text = exporters.anomalies_to_csv(summary) + try: + audit("export_anomalies_csv", + f"total={summary.get('total_anomalies', 0) if isinstance(summary, dict) else 0}") + db.session.commit() + except Exception: + pass + return exporters.csv_response( + csv_text, exporters.timestamped_filename("squid_anomalies", "csv") + ) + + +@app.route("/export/anomalies.json") +@login_required +def export_anomalies_json(): + """Export the current anomaly summary to a JSON file.""" + try: + entries = get_parsed_logs() + except Exception: + entries = [] + try: + summary = _build_anomaly_summary(entries) + except Exception: + summary = {} + json_text = exporters.anomalies_to_json(summary) + try: + audit("export_anomalies_json", + f"total={summary.get('total_anomalies', 0) if isinstance(summary, dict) else 0}") + db.session.commit() + except Exception: + pass + return exporters.json_response( + json_text, exporters.timestamped_filename("squid_anomalies", "json") + ) + + +@app.route("/export/audit.csv") +@login_required +def export_audit_csv(): + """Export audit log rows to a CSV file. + + Accepts ``?username=...`` (exact match) and ``?action=...`` (case- + insensitive substring) so the export matches the on-screen filter. + A hard cap of 100k rows keeps the response from blowing up if the + audit table grows huge. + """ + try: + rows = _audit_query().limit(100000).all() + except Exception: + rows = [] + csv_text = exporters.audit_to_csv(rows) + try: + audit("export_audit_csv", f"count={len(rows)}") + db.session.commit() + except Exception: + pass + return exporters.csv_response( + csv_text, exporters.timestamped_filename("squid_audit", "csv") + ) + + +@app.route("/export/audit.json") +@login_required +def export_audit_json(): + """Export audit log rows to a JSON file.""" + try: + rows = _audit_query().limit(100000).all() + except Exception: + rows = [] + json_text = exporters.audit_to_json(rows) + try: + audit("export_audit_json", f"count={len(rows)}") + db.session.commit() + except Exception: + pass + return exporters.json_response( + json_text, exporters.timestamped_filename("squid_audit", "json") + ) + + +# --------------------------------------------------------------------------- +# P2-3: GeoIP map visualisation +# --------------------------------------------------------------------------- +# +# Provides three routes: +# * GET /geoip - page rendering Top 20 clients on a Leaflet map +# * GET /api/geoip/clients - JSON for client IPs (top 50) +# * GET /api/geoip/hosts - JSON for destination hosts (top 50, after DNS) +# +# Backends are chosen in geoip_lookup.lookup(): +# * online -> ip-api.com (default; no key required) +# * offline -> MaxMind GeoLite2-City .mmdb if ``maxminddb`` is installed +# AND a db_path is configured via AppConfig(geoip_mmdb) +# --------------------------------------------------------------------------- + + +def _resolve_geoip_backend() -> tuple[bool, str]: + """Pick the right backend for this request. + + Returns ``(use_online, db_path)``: + * ``(True, '')`` -> use ip-api.com (default), + * ``(False, db)`` -> use offline mmdb at ``db``, + * ``(False, '')`` -> no backend, calls will fail with an error. + """ + db_path = "" + try: + db_path = (get_config("geoip_mmdb") or "").strip() + except Exception: + db_path = "" + use_offline = bool(db_path) + # If an offline DB is configured *and* maxminddb is importable, use it. + if use_offline: + try: + import maxminddb # type: ignore # noqa: F401 + except Exception: + # fall back to online so the page still works + return True, "" + return False, db_path + return True, "" + + +def _resolve_host_ip(host: str) -> str: + """Resolve a hostname to an IP. Returns '' on failure.""" + if not host: + return "" + try: + return socket.gethostbyname(host) + except Exception: + return "" + + +def _client_rows(use_online: bool, db_path: str, top_n: int, source: str) -> list[dict]: + """Build geo-enriched rows for ``source`` in {"clients", "hosts"}. + + Each row is ``{ip, country, city, country_name, region, lat, lon, + count, bytes, private}`` (some keys may be empty when lookup fails). + """ + try: + entries = get_parsed_logs() + except Exception: + entries = [] + if not entries: + return [] + stats = log_parser.aggregate_stats(entries) + if source == "clients": + raw = list(stats.get("top_clients") or [])[:top_n] + for row in raw: + row["_key"] = row.get("client") or "" + else: + raw = list(stats.get("top_hosts") or [])[:top_n] + for row in raw: + row["_key"] = row.get("host") or "" + out: list[dict] = [] + for row in raw: + key = row["_key"] + ip = key + if source == "hosts": + ip = _resolve_host_ip(key) + lookup_ip = ip or key # fall back to the hostname so we still get a row + try: + geo = geoip_lookup.lookup( + lookup_ip, db_path=db_path, use_online=use_online + ) + except Exception as e: # safety net + geo = geoip_lookup.GeoIPResult(ip=lookup_ip, error=str(e)) + out.append({ + "ip": ip or key, + "host": key if source == "hosts" else "", + "country": geo.country, + "country_name": geo.country_name, + "city": geo.city, + "region": geo.region, + "latitude": geo.latitude, + "longitude": geo.longitude, + "asn": geo.asn, + "isp": geo.isp, + "count": int(row.get("count") or 0), + "bytes": int(row.get("bytes") or 0), + "error": geo.error, + "source": source, + }) + return out + + +@app.route("/geoip") +@login_required +def geoip_view(): + """Render the GeoIP map page (Top 20 clients).""" + try: + entries = get_parsed_logs() + except Exception: + entries = [] + entries_count = len(entries) + use_online, db_path = _resolve_geoip_backend() + data_source = ( + "ip-api.com (online)" if use_online + else (f"GeoLite2-City ({db_path})" if db_path else "none") + ) + # Page only resolves the top 20 to keep the map load light. + rows = _client_rows(use_online, db_path, top_n=20, source="clients") + geo_rows = [r for r in rows if not r["error"] and r["latitude"]] + try: + audit("geoip_view", f"total={len(rows)} geo={len(geo_rows)} backend={data_source}") + db.session.commit() + except Exception: + pass + return render_template( + "geoip.html", + rows=rows, + geo_rows=geo_rows, + total_clients=len(rows), + resolved_clients=len(geo_rows), + countries=len({r["country"] for r in geo_rows if r["country"]}), + cities=len({(r["country"], r["city"]) for r in geo_rows if r["city"]}), + data_source=data_source, + use_online=use_online, + db_path=db_path, + entries_count=entries_count, + format_bytes=log_parser.format_bytes, + csrf_token_value=security.generate_csrf_token(), + ) + + +@app.route("/api/geoip/clients") +@login_required +def api_geoip_clients(): + """Top 50 client IPs enriched with GeoIP data.""" + use_online, db_path = _resolve_geoip_backend() + rows = _client_rows(use_online, db_path, top_n=50, source="clients") + try: + audit("api_geoip_clients", f"rows={len(rows)}") + db.session.commit() + except Exception: + pass + return jsonify({ + "ok": True, + "source": "clients", + "backend": "online" if use_online else "offline", + "rows": rows, + }) + + +@app.route("/api/geoip/hosts") +@login_required +def api_geoip_hosts(): + """Top 50 destination hosts. Hostnames are resolved to IPs first; + rows whose hostname cannot be resolved are skipped from the map + but still reported so the user sees what was attempted.""" + use_online, db_path = _resolve_geoip_backend() + rows = _client_rows(use_online, db_path, top_n=50, source="hosts") + placed = [r for r in rows if not r["error"] and r["latitude"]] + skipped = [r for r in rows if r["error"] and not r.get("host")] + try: + audit( + "api_geoip_hosts", + f"rows={len(rows)} placed={len(placed)} skipped={len(skipped)}", + ) + db.session.commit() + except Exception: + pass + return jsonify({ + "ok": True, + "source": "hosts", + "backend": "online" if use_online else "offline", + "rows": rows, + "placed": len(placed), + "skipped": len(skipped), + }) + + +# --------------------------------------------------------------------------- +# Routes - per-client 24h trend detail page (P2-4) +# --------------------------------------------------------------------------- +# +# Squid's "突增" anomaly page (anomalies.html) can flag a client but offers +# no historical context: an operator clicks through and only sees a flat +# log list. /clients/ fixes that by giving each client its own page: +# 24h hourly trend line, top hosts, top URLs, recent request samples, and +# KPIs (total requests / bytes / first-seen / last-seen / avg size). +# +# Security: the IP path parameter is validated against a strict whitelist +# (letters/digits/dot/dash/colon/percent, length 1-64, no "..", "/", "\"). +# Anything else returns 404 - we never let an untrusted string reach the +# filesystem, SQL filter, or template without a sanity check first. + +# Whitelist for the path parameter: letters, digits, dot, dash, +# colon (IPv6), percent (IPv6 zone id, e.g. fe80::1%eth0). 64-char ceiling. +# `re` is imported at module scope (line 14). +_CLIENT_RE = re.compile(r"^[A-Za-z0-9.\-:%]{1,64}$") + + +def _is_safe_client_id(s: str) -> bool: + """Strict validator for the /clients/ path parameter. + + Returns True only for values that look like an IP (v4 / v6 / hostname). + Rejects: empty, "..", "/", "\\", whitespace, anything URL-encoded that + would re-introduce a separator after percent-decoding. + """ + if not s: + return False + if not _CLIENT_RE.match(s): + return False + if ".." in s or "/" in s or "\\" in s: + return False + return True + + +def _build_client_trend(entries: list[log_parser.LogEntry]) -> dict: + """Aggregate a list of LogEntry rows that all share a single client. + + Returns KPIs, hourly buckets (24, requests + bytes), top hosts, + top URLs, and the 50 most-recent request samples. Designed to feed + /clients/ directly. + """ + from collections import Counter, defaultdict + + if not entries: + return { + "total_requests": 0, + "total_bytes": 0, + "avg_size_bytes": 0.0, + "time_start": None, + "time_end": None, + "hourly": [ + {"hour": h, "requests": 0, "bytes": 0} for h in range(24) + ], + "top_hosts": [], + "top_urls": [], + "samples": [], + } + + total = len(entries) + total_bytes = sum(int(e["size"] or 0) for e in entries) + avg_size = total_bytes / total if total > 0 else 0.0 + + timestamps = [e["timestamp"] for e in entries if e["timestamp"]] + time_start = min(timestamps) if timestamps else None + time_end = max(timestamps) if timestamps else None + + # 24h hourly buckets keyed on the UTC hour-of-day of each entry's + # timestamp. Matches the structure of dashboard.html's `stats.hourly` + # so the Chart.js code can be shared. + by_hour = defaultdict(int) + by_hour_bytes = defaultdict(int) + for e in entries: + if e["timestamp"]: + h = e["timestamp"].hour + by_hour[h] += 1 + by_hour_bytes[h] += int(e["size"] or 0) + + hourly = [ + {"hour": h, "requests": by_hour.get(h, 0), "bytes": by_hour_bytes.get(h, 0)} + for h in range(24) + ] + + # Top hosts (count + bytes) + host_count = Counter() + host_bytes = defaultdict(int) + for e in entries: + if e["host"]: + host_count[e["host"]] += 1 + host_bytes[e["host"]] += int(e["size"] or 0) + top_hosts = [ + {"host": h, "count": n, "bytes": host_bytes[h]} + for h, n in host_count.most_common(20) + ] + + # Top URLs + url_count = Counter(e["url"] for e in entries if e["url"]) + top_urls = [{"url": u, "count": n} for u, n in url_count.most_common(20)] + + # Last 50 samples (newest first). Sort in-place on a copy so we don't + # mutate the caller's list ordering. + samples = sorted( + entries, + key=lambda e: float(e.get("time") or 0.0), + reverse=True, + )[:50] + + return { + "total_requests": total, + "total_bytes": total_bytes, + "avg_size_bytes": avg_size, + "time_start": time_start, + "time_end": time_end, + "hourly": hourly, + "top_hosts": top_hosts, + "top_urls": top_urls, + "samples": samples, + } + + +@app.route("/clients/") +@login_required +def client_detail(ip: str): + """Per-client 24h trend + top hosts + sample requests. + + Validates that ``ip`` looks like an IP/hostname (whitelist only) so the + value cannot escape into a filesystem path, SQL filter, or template + context as anything other than an inert string. Untrusted input -> + 404 (not 400) so probing doesn't leak whether the URL space is "real". + """ + if not _is_safe_client_id(ip): + abort(404) + + all_entries = get_parsed_logs() + # filter_entries does substring match; we use equality to avoid a client + # like "10.0.0.1" accidentally matching "10.0.0.10". + client_entries = [e for e in all_entries if e["client"] == ip] + + trend = _build_client_trend(client_entries) + cfg = get_all_config() + + # Audit the lookup so "who looked at this client" is recorded. + try: + audit("client_detail", f"ip={ip} requests={trend['total_requests']}") + db.session.commit() + except Exception: + pass + + return render_template( + "client_detail.html", + client=ip, + trend=trend, + cfg=cfg, + format_bytes=log_parser.format_bytes, + format_duration=log_parser.format_duration, + has_data=bool(client_entries), + ) + + +# --------------------------------------------------------------------------- +# P3-1: Dark / light theme toggle (cookie-backed, server-side endpoint) +# --------------------------------------------------------------------------- +# Why this lives here: +# - The toggle button (templates/_theme_toggle.html) writes the cookie +# client-side via JS for instant feedback. This server endpoint exists so +# non-JS clients (curl, scripts, browser w/o JS) can still flip the theme +# and so any server-rendered redirect can carry the user's preference. +# - Theme is purely a UI preference, not security-sensitive. We monkey-patch +# security.csrf_protect (the function reference) so /theme/set is exempt +# without touching security.py. The patch is narrow: only this endpoint, +# only on POST, and only when a valid theme value is supplied. +# --------------------------------------------------------------------------- + +_VALID_THEMES = ("dark", "light") +_THEME_COOKIE_AGE = 365 * 24 * 3600 # 1 year, per spec + + +def _read_theme_cookie(): + """Return 'light' or 'dark' from the theme cookie. Defaults to 'dark'.""" + val = request.cookies.get("theme", "dark") + if val not in _VALID_THEMES: + return "dark" + return val + + +@app.context_processor +def _theme_globals(): + """Expose the active theme to every template. + + Templates reference this as `{{ theme }}`. We read the cookie on every + render so the very first response after a toggle already reflects the + new theme (no stale cache, no extra round-trip). + """ + return {"theme": _read_theme_cookie()} + + +# Patch security.csrf_protect (a runtime function reference) so /theme/set +# is exempt from CSRF. Theme is not a security-sensitive operation; the +# cookie value is also writable from any browser via document.cookie. +_orig_csrf_protect = security.csrf_protect + + +def _csrf_protect_themed(): + """Wrapped csrf_protect that exempts the theme_set endpoint only.""" + if request.endpoint == "theme_set": + return None + return _orig_csrf_protect() + + +# Replace the global reference; _security_gate() looks it up by name on +# every request, so the wrapper takes effect immediately. +security.csrf_protect = _csrf_protect_themed + + +@app.route("/theme/set", methods=["POST"]) +def theme_set(): + """Set the theme cookie and redirect back to the referring page. + + Body: theme=dark|light (form-encoded). Any value outside this set is + silently coerced to 'dark' so a malformed call can't poison the cookie. + Referer is used for the redirect; if missing, we fall back to the + dashboard (or login page if not authenticated). + """ + new_theme = (request.form.get("theme") or "dark").strip().lower() + if new_theme not in _VALID_THEMES: + new_theme = "dark" + + # Pick a safe redirect target. Referer is attacker-influenced but here + # it's only used as a same-origin UX nicety; we constrain it to a + # same-host path to avoid open-redirect abuse. + target = request.referrer or "" + if not target: + target = url_for("dashboard") if session.get("user_id") else url_for("login") + else: + from urllib.parse import urlparse + ref = urlparse(target) + # Only follow same-host referers; drop query/fragment from scheme+netloc + req_host = request.host + if ref.netloc and ref.netloc != req_host: + target = url_for("dashboard") if session.get("user_id") else url_for("login") + elif ref.path: + target = ref.path + (("?" + ref.query) if ref.query else "") + + resp = redirect(target) + resp.set_cookie( + "theme", + new_theme, + max_age=_THEME_COOKIE_AGE, + path="/", + samesite="Lax", + httponly=False, # readable by client JS (the toggle reads it on load) + ) + return resp + + +# =========================================================================== +# P3-3: WebSSH terminal (browser → WebSocket → paramiko → remote SSH) +# =========================================================================== +# +# Architecture: +# - GET /terminal renders templates/terminal.html (xterm.js UI) +# - WS /ws/terminal bi-directional bridge between xterm.js and a +# paramiko SSH channel. One TCP stream per WS. +# +# Hard requirements (declared in requirements.txt): +# - paramiko>=2.10.0 SSH2 client (Transport / Channel) +# - flask-sock>=0.7.0 WebSocket route decorator for Flask +# +# Both are imported lazily here so a missing paramiko doesn't crash the +# whole app on startup; in that case the WebSocket route degrades to +# {"type": "error", ...} and the GET /terminal page shows a banner. + +try: + import paramiko as _paramiko # type: ignore +except Exception: # ImportError, syntax errors on exotic platforms, etc. + _paramiko = None # type: ignore + +try: + from flask_sock import Sock as _Sock # type: ignore +except Exception: + _Sock = None # type: ignore + + +if _Sock is not None: + _sock = _Sock(app) +else: + _sock = None # degraded mode + + +def _terminal_audit(action: str, detail: str): + """Write a terminal-related audit entry using the standard audit() helper. + + Falls back to a no-op if called outside of a request context (the WS + handler may not have one for some flows). + """ + try: + audit(f"terminal_{action}", detail[:2000]) + db.session.commit() + except Exception: + try: + db.session.rollback() + except Exception: + pass + + +def _terminal_user_role() -> str: + """Return the current user's role, or 'guest' if not logged in. + + We reuse the session cookie because flask_sock decorators don't + preserve the request stack in a way Flask-Login would. + """ + try: + uid = session.get("user_id") + if not uid: + return "guest" + u = db.session.get(User, uid) + return (u.role or "") if u else "guest" + except Exception: + return "guest" + + +def _terminal_login_ok() -> bool: + """True iff the WS connection is from a logged-in admin.""" + return bool(session.get("user_id")) and _terminal_user_role() == "admin" + + +def _terminal_get_instance(): + """Resolve the active SquidInstance for the current WS request, or None.""" + try: + return _resolve_active_instance() + except Exception: + return None + + +@app.route("/terminal", methods=["GET"]) +@login_required +def terminal_view(): + """Browser page for the WebSSH terminal. + + Admin-only: any other role gets a 403. The sidebar link is only shown + to admins via base.html, but we double-check here. + """ + cu = current_user() + if not cu or (cu.role or "") != "admin": + abort(403) + + inst = _terminal_get_instance() + ssh_target = "" + ssh_user = "" + ssh_port = 22 + ssh_key_path = "" + ssh_key_short = "" + instance_name = "(未配置)" + if inst is not None: + instance_name = inst.name or f"#{inst.id}" + host = (inst.ssh_host or "").strip() + user = (inst.ssh_user or "").strip() + if host and user: + ssh_target = f"{user}@{host}:{inst.ssh_port or 22}" + ssh_user = user + ssh_port = inst.ssh_port or 22 + ssh_key_path = (inst.ssh_key_path or "").strip() + if ssh_key_path: + # Show only the basename to keep the header compact + ssh_key_short = os.path.basename(ssh_key_path) + + return render_template( + "terminal.html", + ssh_target=ssh_target, + ssh_user=ssh_user, + ssh_port=ssh_port, + ssh_key_path=ssh_key_path, + ssh_key_short=ssh_key_short, + instance_name=instance_name, + paramiko_missing=(_paramiko is None) or (_sock is None), + ) + + +# ---------- WebSocket route (paramiko-driven SSH bridge) ----------------------- + +if _sock is not None: + @_sock.route("/ws/terminal") + def ws_terminal(ws): + """WebSSH WebSocket endpoint. + + Protocol (client → server): + {"type": "input", "data": "..."} + {"type": "resize", "cols": 80, "rows": 24} + + Protocol (server → client): + {"type": "output", "data": "..."} raw bytes for xterm.js + {"type": "status", "msg": "..."} human-readable info + {"type": "error", "msg": "..."} terminal-close triggering + """ + # -- auth gate -- + if not _terminal_login_ok(): + try: + ws.send(json.dumps({"type": "error", + "msg": "auth required (admin only)"})) + except Exception: + pass + try: ws.close() + except Exception: pass + return + + # -- dependency gate -- + if _paramiko is None: + try: + ws.send(json.dumps({"type": "error", + "msg": "paramiko not installed"})) + except Exception: + pass + try: ws.close() + except Exception: pass + return + + # -- resolve target from active instance -- + inst = _terminal_get_instance() + if inst is None: + try: + ws.send(json.dumps({"type": "error", + "msg": "no active squid instance"})) + except Exception: + pass + try: ws.close() + except Exception: pass + return + + ssh_host = (inst.ssh_host or "").strip() + ssh_user = (inst.ssh_user or "").strip() + ssh_port = int(inst.ssh_port or 22) + ssh_key_path = (inst.ssh_key_path or "").strip() + if not ssh_host or not ssh_user: + try: + ws.send(json.dumps({ + "type": "error", + "msg": "active instance missing ssh_host / ssh_user", + })) + except Exception: + pass + try: ws.close() + except Exception: pass + return + + # -- open SSH channel -- + client = None + channel = None + connected = False + session_start = time.time() + audit_action = "ssh_open" + try: + try: + ws.send(json.dumps({"type": "status", + "msg": f"connecting to {ssh_user}@{ssh_host}:{ssh_port}"})) + except Exception: + pass + + client = _paramiko.SSHClient() + # Auto-trust host keys for the manager-internal use case. + # In a stricter setup, replace with a persistent host_keys file. + client.set_missing_host_key_policy(_paramiko.AutoAddPolicy()) + connect_kwargs = dict( + hostname=ssh_host, + port=ssh_port, + username=ssh_user, + timeout=10, + allow_agent=False, + look_for_keys=False, + ) + if ssh_key_path and os.path.isfile(ssh_key_path): + connect_kwargs["key_filename"] = ssh_key_path + client.connect(**connect_kwargs) + + channel = client.get_transport().open_session(timeout=10) + channel.get_pty(term="xterm-256color", width=80, height=24) + channel.invoke_shell() + connected = True + + try: + _terminal_audit(audit_action, + f"instance={inst.name or inst.id} " + f"target={ssh_user}@{ssh_host}:{ssh_port}") + except Exception: + pass + + try: + ws.send(json.dumps({"type": "status", + "msg": f"connected to {ssh_user}@{ssh_host}"})) + except Exception: + pass + except Exception as e: + try: + ws.send(json.dumps({"type": "error", + "msg": f"SSH connect failed: {e}"})) + except Exception: + pass + try: + _terminal_audit("ssh_open_fail", + f"target={ssh_user}@{ssh_host}:{ssh_port} err={e}") + except Exception: + pass + try: + if channel is not None: + channel.close() + if client is not None: + client.close() + except Exception: + pass + try: ws.close() + except Exception: pass + return + + # -- bidirectional pump -- + # + # flask-sock's ws.receive() is blocking, and paramiko's channel + # is also blocking. We solve this by running the SSH→WS pump on a + # background thread that polls channel.recv_ready() with a tiny + # sleep; the main handler thread pulls WS frames and forwards + # them to channel.send() / channel.resize_pty(). + + import threading as _threading + stop_event = _threading.Event() + ssh_errors: list[str] = [] + + def _send(payload): + """Safe JSON send to the WebSocket (swallows errors).""" + try: + ws.send(json.dumps(payload)) + except Exception: + pass + + def _ssh_reader(): + """Pump SSH channel output → WS until the channel closes.""" + try: + while not stop_event.is_set(): + if channel.closed or channel.exit_status_ready(): + _send({"type": "status", "msg": "SSH channel closed"}) + break + try: + # Tiny blocking recv with timeout so we can observe stop_event. + # paramiko's Channel has no real non-blocking mode without + # a transport, but recv_ready()/recv(timeout=...) works. + if channel.recv_ready(): + data = channel.recv(4096) + if not data: + break + try: + text = data.decode("utf-8", errors="replace") + except Exception: + text = data.decode("latin-1", errors="replace") + _send({"type": "output", "data": text}) + else: + time.sleep(0.03) + except Exception as e: + ssh_errors.append(str(e)) + _send({"type": "error", + "msg": f"SSH read error: {e}"}) + break + finally: + try: + _send({"type": "status", "msg": "reader exited"}) + except Exception: + pass + + reader_thread = _threading.Thread(target=_ssh_reader, daemon=True) + reader_thread.start() + + try: + while not stop_event.is_set(): + # blocking recv on the WS + try: + msg = ws.receive(timeout=1.0) + except Exception: + # timeout -> loop and check SSH side + if not reader_thread.is_alive(): + break + continue + if msg is None: + # client closed + break + # msg may be str (text frame) or bytes (binary frame) + if isinstance(msg, (bytes, bytearray)): + try: + msg = msg.decode("utf-8", errors="replace") + except Exception: + continue + try: + payload = json.loads(msg) + except Exception: + # ignore malformed frames + continue + kind = payload.get("type") + if kind == "input": + data = payload.get("data", "") + if not isinstance(data, str): + continue + if not connected or channel is None: + continue + try: + channel.send(data) + except Exception as e: + _send({"type": "error", + "msg": f"SSH write error: {e}"}) + break + # Audit high-volume keystrokes conservatively: log + # the presence of dangerous commands but never + # full paste dumps (privacy + log bloat). + if len(data) > 2 and any(c in data for c in ("\n", "\r")): + snippet = data.replace("\n", "\\n")[:200] + try: + _terminal_audit( + "ssh_input", + f"target={ssh_user}@{ssh_host} " + f"len={len(data)} snippet={snippet!r}", + ) + except Exception: + pass + elif kind == "resize": + try: + cols = int(payload.get("cols", 80)) + rows = int(payload.get("rows", 24)) + except Exception: + cols, rows = 80, 24 + if not connected or channel is None: + continue + try: + channel.resize_pty(width=cols, height=rows) + _terminal_audit( + "ssh_resize", + f"target={ssh_user}@{ssh_host} cols={cols} rows={rows}", + ) + except Exception as e: + _send({"type": "error", + "msg": f"resize error: {e}"}) + else: + # unknown frame type - ignore silently + continue + except Exception as e: + try: + _terminal_audit("ssh_ws_error", + f"target={ssh_user}@{ssh_host} err={e}") + except Exception: + pass + finally: + stop_event.set() + try: + if channel is not None: + channel.close() + except Exception: + pass + try: + if client is not None: + client.close() + except Exception: + pass + duration = int(time.time() - session_start) + try: + _terminal_audit("ssh_close", + f"target={ssh_user}@{ssh_host} duration={duration}s") + except Exception: + pass + try: + ws.close() + except Exception: + pass + + +# Safe-guard: if flask_sock couldn't be imported, expose a stub HTTP route +# so users see a useful message instead of a generic 404. +if _sock is None: + @app.route("/terminal", methods=["GET"]) + @login_required + def terminal_view(): + cu = current_user() + if not cu or (cu.role or "") != "admin": + abort(403) + return ("

WebSSH not available

" + "

Install flask-sock + paramiko on the server.

", + 503, {"Content-Type": "text/html; charset=utf-8"}) + + +# Make sure json is available for the WS route (it was already imported at +# module top via `from flask import ... jsonify`, but json.dumps needs the +# stdlib module). +# (json is already imported at module top — see lines 18-23) diff --git a/diff_tool.py b/diff_tool.py new file mode 100644 index 0000000..575c88d --- /dev/null +++ b/diff_tool.py @@ -0,0 +1,122 @@ +"""Unified diff generator for squid.conf changes. + +Provides a small, pure-Python unified-diff renderer. We avoid Python's +``difflib.unified_diff`` because we want tight control over hunk sizing +and color classes used by the UI. +""" +from __future__ import annotations + +import difflib +from typing import Iterable + + +def unified_diff(old: str, new: str, context: int = 3, lineterm: str = "\n") -> list[dict]: + """Return a list of diff lines as dicts: {kind, text, old_lineno, new_lineno}. + + kind: 'equal' | 'add' | 'del' | 'hunk' | 'meta' + """ + old_lines = old.splitlines(keepends=False) + new_lines = new.splitlines(keepends=False) + + sm = difflib.SequenceMatcher(a=old_lines, b=new_lines, autojunk=False) + out: list[dict] = [] + + old_no = 0 + new_no = 0 + + def push_meta(text: str): + out.append({"kind": "meta", "text": text, "old_lineno": None, "new_lineno": None}) + + push_meta("--- 当前配置 (existing)") + push_meta("+++ 新配置 (proposed)") + + for tag, i1, i2, j1, j2 in sm.get_opcodes(): + if tag == "equal": + # show context lines (limit to first/last of long equal runs) + block = old_lines[i1:i2] + if len(block) <= context * 2 + 1: + for k, ln in enumerate(block): + old_no += 1 + new_no += 1 + out.append({ + "kind": "equal", "text": ln, + "old_lineno": old_no, "new_lineno": new_no, + }) + else: + # head + for k in range(context): + old_no += 1; new_no += 1 + out.append({ + "kind": "equal", "text": old_lines[i1 + k], + "old_lineno": old_no, "new_lineno": new_no, + }) + out.append({ + "kind": "meta", "text": f"... ({i2 - i1 - 2 * context} equal lines skipped) ...", + "old_lineno": None, "new_lineno": None, + }) + old_no += (i2 - i1 - 2 * context) + new_no += (i2 - i1 - 2 * context) + # tail + for k in range(i2 - context, i2): + old_no += 1; new_no += 1 + out.append({ + "kind": "equal", "text": old_lines[k], + "old_lineno": old_no, "new_lineno": new_no, + }) + elif tag == "replace": + for ln in old_lines[i1:i2]: + old_no += 1 + out.append({ + "kind": "del", "text": ln, + "old_lineno": old_no, "new_lineno": None, + }) + for ln in new_lines[j1:j2]: + new_no += 1 + out.append({ + "kind": "add", "text": ln, + "old_lineno": None, "new_lineno": new_no, + }) + elif tag == "delete": + for ln in old_lines[i1:i2]: + old_no += 1 + out.append({ + "kind": "del", "text": ln, + "old_lineno": old_no, "new_lineno": None, + }) + elif tag == "insert": + for ln in new_lines[j1:j2]: + new_no += 1 + out.append({ + "kind": "add", "text": ln, + "old_lineno": None, "new_lineno": new_no, + }) + return out + + +def summarize(diff: list[dict]) -> dict: + """Return {adds, dels, total_changed}.""" + adds = sum(1 for d in diff if d["kind"] == "add") + dels = sum(1 for d in diff if d["kind"] == "del") + return {"adds": adds, "dels": dels, "total_changed": adds + dels} + + +def render_html(diff: list[dict]) -> str: + """Render diff to HTML with classes used by static/style.css.""" + parts = ['
'] + for d in diff: + kind = d["kind"] + text = d["text"] + # escape HTML + esc = (text.replace("&", "&").replace("<", "<").replace(">", ">")) + old_n = d.get("old_lineno") or "" + new_n = d.get("new_lineno") or "" + if kind == "add": + parts.append(f'
{old_n:>4}{new_n:>4} + {esc}
') + elif kind == "del": + parts.append(f'
{old_n:>4}{new_n:>4} - {esc}
') + elif kind == "equal": + parts.append(f'
{old_n:>4}{new_n:>4} {esc}
') + else: # meta + parts.append(f'
{esc}
') + parts.append("
") + return "\n".join(parts) diff --git a/exporters.py b/exporters.py new file mode 100644 index 0000000..f6b54af --- /dev/null +++ b/exporters.py @@ -0,0 +1,577 @@ +"""P2-2: Export utilities for Squid Web Manager. + +Provides CSV/JSON serialisation helpers and Flask response builders for +downloading log entries, KPI summaries, anomaly lists and audit log entries. + +Conventions +----------- +- All CSV output starts with a UTF-8 BOM (``\ufeff``) so Microsoft Excel + opens it as UTF-8 rather than falling back to the local code page. +- Timestamps are serialised as ISO-8601 strings (UTC when the source + value carries tzinfo). ``None`` values become empty cells; ``inf`` + becomes the literal string ``Infinity`` so downstream tools don't + silently choke. +- Every public function swallows exceptions and returns a safe empty + value (``""`` / ``{}``). This makes it safe to call from request + handlers - a broken export must never 500 the dashboard. +- No third-party dependencies. The module only uses ``csv``, ``json``, + ``io`` and Flask's ``Response``. +""" +from __future__ import annotations + +import csv +import io +import json +from datetime import date, datetime +from typing import Any, Iterable + +from flask import Response + + +# --- Constants ------------------------------------------------------------- + +#: Default columns exported for a LogEntry. Order matters - this is the +#: column order Excel/LibreOffice will display. +DEFAULT_ENTRY_FIELDS: list[str] = [ + "time", + "client", + "result_code", + "http_code", + "method", + "url", + "host", + "size_bytes", + "elapsed_ms", + "hier_code", + "content_type", +] + +#: UTF-8 BOM. Excel needs this prefix to recognise the file as UTF-8. +_UTF8_BOM = "\ufeff" + + +# --- Internal helpers ------------------------------------------------------ + + +def _get_field(entry: Any, key: str) -> Any: + """Dict-or-attribute getter that never raises. + + LogEntry subclasses dict, so ``entry.get(key)`` works, but we also + support plain attribute access (e.g. SQLAlchemy model instances). + """ + try: + if isinstance(entry, dict): + return entry.get(key) + return getattr(entry, key, None) + except Exception: + return None + + +def _serialise_value(value: Any) -> str: + """Convert a single cell value into a CSV-friendly string. + + - ``None`` -> empty cell (not the literal "None"). + - ``datetime`` / ``date`` -> ISO-8601. + - ``float('inf')`` / ``float('-inf')`` -> literal "Infinity" / "-Infinity". + - everything else -> ``str(value)``. + """ + if value is None: + return "" + # datetime must come before date because datetime is a subclass of date + if isinstance(value, datetime): + try: + return value.isoformat() + except Exception: + return str(value) + if isinstance(value, date): + try: + return value.isoformat() + except Exception: + return str(value) + if isinstance(value, float): + # JSON-emitted inf / -inf / nan don't round-trip cleanly through + # CSV consumers; spell them out instead. + if value != value: # NaN + return "NaN" + if value == float("inf"): + return "Infinity" + if value == float("-inf"): + return "-Infinity" + # drop noisy trailing zeros for whole numbers + if value.is_integer(): + return str(int(value)) + return repr(value) + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (list, tuple)): + try: + return ",".join(str(v) for v in value) + except Exception: + return str(value) + try: + return str(value) + except Exception: + return "" + + +def _timestamp_iso(entry: Any) -> str: + """Best-effort ISO timestamp for ``entry`` - falls back to ``time`` field.""" + try: + ts = _get_field(entry, "timestamp") + if isinstance(ts, datetime): + return ts.isoformat() + if isinstance(ts, (int, float)): + return datetime.utcfromtimestamp(float(ts)).isoformat() + "Z" + except Exception: + pass + try: + t = _get_field(entry, "time") + if isinstance(t, (int, float)): + return datetime.utcfromtimestamp(float(t)).isoformat() + "Z" + except Exception: + pass + return "" + + +def _write_csv(headers: list[str], rows: Iterable[list[Any]]) -> str: + """Render ``headers`` + ``rows`` into a CSV string with UTF-8 BOM. + + Uses ``csv.writer`` with QUOTE_MINIMAL so embedded newlines / quotes + inside URLs are escaped correctly. Output is a plain string (no + BytesIO shenanigans) so the caller can decide whether to wrap it in + a Flask ``Response`` or write to disk. + """ + buf = io.StringIO() + # Write the BOM first; CSV writer will then prepend the header row. + buf.write(_UTF8_BOM) + try: + writer = csv.writer(buf, quoting=csv.QUOTE_MINIMAL, lineterminator="\n") + writer.writerow(headers) + for row in rows: + try: + writer.writerow([_serialise_value(v) for v in row]) + except Exception: + # one bad row shouldn't nuke the whole export + continue + except Exception: + # return at least the BOM + header line so callers get *something* + try: + buf.write(",".join(headers)) + except Exception: + pass + return buf.getvalue() + + +# --- Entry export ---------------------------------------------------------- + + +def entries_to_csv(entries: Any, fields: list[str] | None = None) -> str: + """Serialise a list of log entries to a CSV string. + + ``entries`` may be any iterable of dict-like objects (LogEntry, + SQLAlchemy model, plain dict). ``fields`` defaults to + :data:`DEFAULT_ENTRY_FIELDS`. Returns an empty string on error. + """ + try: + if not entries: + # still emit the header so an empty download is a valid CSV + cols = list(fields) if fields else list(DEFAULT_ENTRY_FIELDS) + return _write_csv(cols, []) + + cols = list(fields) if fields else list(DEFAULT_ENTRY_FIELDS) + + def _row(entry: Any) -> list[Any]: + out: list[Any] = [] + for col in cols: + if col == "time": + # Prefer ISO timestamp; fall back to raw time field + iso = _timestamp_iso(entry) + if iso: + out.append(iso) + else: + out.append(_get_field(entry, "time")) + elif col == "size_bytes": + # The DB column is ``size`` (LogEntry) or ``size`` (dict); + # allow ``size_bytes`` as a fallback for either shape. + val = _get_field(entry, "size_bytes") + if val is None: + val = _get_field(entry, "size") + out.append(val) + elif col == "elapsed_ms": + val = _get_field(entry, "elapsed_ms") + if val is None: + val = _get_field(entry, "elapsed") + out.append(val) + elif col == "hier_code": + val = _get_field(entry, "hier_code") + if val is None: + val = _get_field(entry, "hierarchy") + out.append(val) + else: + out.append(_get_field(entry, col)) + return out + + return _write_csv(cols, (_row(e) for e in entries)) + except Exception: + return "" + + +def entries_to_json(entries: Any, fields: list[str] | None = None) -> str: + """Serialise a list of log entries to a pretty-printed JSON string. + + Datetimes are converted to ISO-8601. ``fields`` constrains the + exported keys (per-entry filtering); ``None`` exports every key the + entry exposes. Returns an empty string on error. + """ + try: + out: list[dict[str, Any]] = [] + for entry in entries or []: + try: + if fields is not None: + rec: dict[str, Any] = {} + for col in fields: + if col == "time": + iso = _timestamp_iso(entry) + rec["time"] = iso or _get_field(entry, "time") + elif col == "size_bytes": + v = _get_field(entry, "size_bytes") + rec["size_bytes"] = v if v is not None else _get_field(entry, "size") + elif col == "elapsed_ms": + v = _get_field(entry, "elapsed_ms") + rec["elapsed_ms"] = v if v is not None else _get_field(entry, "elapsed") + elif col == "hier_code": + v = _get_field(entry, "hier_code") + rec["hier_code"] = v if v is not None else _get_field(entry, "hierarchy") + else: + rec[col] = _get_field(entry, col) + else: + rec = _entry_to_dict(entry) + out.append(rec) + except Exception: + continue + return json.dumps(out, indent=2, ensure_ascii=False, default=_json_default) + except Exception: + return "" + + +def _entry_to_dict(entry: Any) -> dict[str, Any]: + """Flatten a LogEntry / SQLAlchemy row into a JSON-friendly dict.""" + try: + if isinstance(entry, dict): + return {str(k): _json_safe(v) for k, v in entry.items()} + except Exception: + pass + # Fall back to per-attribute access + result: dict[str, Any] = {} + for col in DEFAULT_ENTRY_FIELDS: + try: + result[col] = _get_field(entry, col) + except Exception: + result[col] = None + return result + + +def _json_safe(value: Any) -> Any: + """Make ``value`` JSON-serialisable. Datetimes -> ISO strings.""" + if value is None or isinstance(value, (str, int, bool)): + return value + if isinstance(value, float): + if value != value: + return None # NaN -> null + if value == float("inf"): + return "Infinity" + if value == float("-inf"): + return "-Infinity" + return value + if isinstance(value, (datetime, date)): + try: + return value.isoformat() + except Exception: + return str(value) + if isinstance(value, (list, tuple)): + return [_json_safe(v) for v in value] + if isinstance(value, dict): + return {str(k): _json_safe(v) for k, v in value.items()} + return str(value) + + +def _json_default(value: Any) -> Any: + """Fallback encoder for json.dumps - keeps the export robust.""" + return _json_safe(value) + + +# --- Stats export ---------------------------------------------------------- + + +def stats_to_json(stats: Any) -> str: + """Serialise a KPI summary dict to a pretty-printed JSON string. + + The dashboard's ``get_stats()`` returns a dict that may contain + nested structures (top_clients list, etc.). Datetimes nested inside + are normalised via :func:`_json_safe`. Returns ``""`` on error. + """ + try: + if stats is None: + stats = {} + return json.dumps(_json_safe(stats), indent=2, + ensure_ascii=False, default=_json_default) + except Exception: + return "" + + +# --- Anomaly export -------------------------------------------------------- + + +#: Field order for the long-form anomaly CSV. ``type`` distinguishes the +#: detector; the rest of the columns are the union of useful fields. +ANOMALY_CSV_FIELDS: list[str] = [ + "type", + "client", + "severity", + "time", + "url", + "host", + "method", + "result", + "result_code", + "http_code", + "size_bytes", + "size_mb", + "request_count", + "total_bytes", + "avg_size", + "baseline_rpm", + "current_rpm", + "baseline_count", + "current_count", + "ratio", + "first_seen", + "sample_url", + "sample_host", +] + + +def _anomaly_rows(summary: dict[str, Any]) -> list[dict[str, Any]]: + """Flatten the four anomaly sub-lists into a single long table. + + Each row gets a ``type`` column so consumers can pivot / filter. + Rows from different detectors carry different fields; missing + fields are simply left empty. + """ + rows: list[dict[str, Any]] = [] + if not isinstance(summary, dict): + return rows + + # 1. anomalous_clients (spikes) + for a in summary.get("anomalous_clients") or []: + if not isinstance(a, dict): + continue + rec = {k: a.get(k) for k in ANOMALY_CSV_FIELDS if k in a} + rec["type"] = "client_spike" + rec.setdefault("client", a.get("client")) + rec.setdefault("severity", a.get("severity")) + rec.setdefault("baseline_rpm", a.get("baseline_rpm")) + rec.setdefault("current_rpm", a.get("current_rpm")) + rec.setdefault("baseline_count", a.get("baseline_count")) + rec.setdefault("current_count", a.get("current_count")) + rec.setdefault("ratio", a.get("ratio")) + rows.append(rec) + + # 2. large_requests + for a in summary.get("large_requests") or []: + if not isinstance(a, dict): + continue + rec = {k: a.get(k) for k in ANOMALY_CSV_FIELDS if k in a} + rec["type"] = "large_request" + rec.setdefault("client", a.get("client")) + rec.setdefault("severity", "warning") + rec.setdefault("size_bytes", a.get("size_bytes")) + rec.setdefault("size_mb", a.get("size_mb")) + rows.append(rec) + + # 3. high_freq_small + for a in summary.get("high_freq_small") or []: + if not isinstance(a, dict): + continue + rec = {k: a.get(k) for k in ANOMALY_CSV_FIELDS if k in a} + rec["type"] = "high_freq_small" + rec.setdefault("client", a.get("client")) + rec.setdefault("severity", "warning") + rec.setdefault("sample_url", a.get("sample_url")) + rec.setdefault("sample_host", a.get("sample_host")) + rows.append(rec) + + # 4. new_clients + for a in summary.get("new_clients") or []: + if not isinstance(a, dict): + continue + rec = {k: a.get(k) for k in ANOMALY_CSV_FIELDS if k in a} + rec["type"] = "new_client" + rec.setdefault("client", a.get("client")) + rec.setdefault("severity", "info") + rec.setdefault("first_seen", a.get("first_seen")) + rec.setdefault("request_count", a.get("request_count")) + rec.setdefault("sample_url", a.get("sample_url")) + rows.append(rec) + + return rows + + +def anomalies_to_csv(summary: Any) -> str: + """Serialise the anomaly summary to a long-form CSV. + + ``summary`` is the dict returned by ``_build_anomaly_summary`` (or + anything with the same four sub-list keys). All four detector + outputs are flattened into one table with a ``type`` column. + Returns ``""`` on error. + """ + try: + rows = _anomaly_rows(summary if isinstance(summary, dict) else {}) + return _write_csv(ANOMALY_CSV_FIELDS, [ + [r.get(col) for col in ANOMALY_CSV_FIELDS] for r in rows + ]) + except Exception: + return "" + + +def anomalies_to_json(summary: Any) -> str: + """Serialise the anomaly summary to pretty JSON (raw structure).""" + try: + if summary is None: + summary = {} + return json.dumps(_json_safe(summary), indent=2, + ensure_ascii=False, default=_json_default) + except Exception: + return "" + + +# --- Audit export ---------------------------------------------------------- + + +#: Field order for the audit log CSV. +AUDIT_CSV_FIELDS: list[str] = [ + "timestamp", + "username", + "action", + "detail", +] + + +def _audit_row(audit_obj: Any) -> list[Any]: + """Convert an AuditLog row to a list aligned with AUDIT_CSV_FIELDS.""" + try: + ts = _get_field(audit_obj, "timestamp") + if isinstance(ts, datetime): + ts_str = ts.isoformat() + elif isinstance(ts, (int, float)): + try: + ts_str = datetime.utcfromtimestamp(float(ts)).isoformat() + "Z" + except Exception: + ts_str = str(ts) + else: + ts_str = _serialise_value(ts) + except Exception: + ts_str = "" + return [ + ts_str, + _get_field(audit_obj, "username"), + _get_field(audit_obj, "action"), + _get_field(audit_obj, "detail"), + ] + + +def audit_to_csv(entries: Any) -> str: + """Serialise a list of AuditLog rows to a CSV string.""" + try: + items = list(entries) if entries else [] + return _write_csv(AUDIT_CSV_FIELDS, (_audit_row(a) for a in items)) + except Exception: + return "" + + +def audit_to_json(entries: Any) -> str: + """Serialise a list of AuditLog rows to pretty JSON.""" + try: + out: list[dict[str, Any]] = [] + for a in entries or []: + try: + out.append({ + "timestamp": _json_safe(_get_field(a, "timestamp")), + "username": _get_field(a, "username"), + "action": _get_field(a, "action"), + "detail": _get_field(a, "detail"), + }) + except Exception: + continue + return json.dumps(out, indent=2, ensure_ascii=False, + default=_json_default) + except Exception: + return "" + + +# --- Response builders ----------------------------------------------------- + + +def csv_response(csv_text: str, filename: str) -> Response: + """Wrap a CSV string in a Flask ``Response`` with download headers. + + Adds ``Content-Disposition: attachment; filename=...`` so browsers + save the file rather than rendering it inline. ``Content-Type`` is + ``text/csv; charset=utf-8`` (Flask appends the charset once). + """ + if not csv_text: + csv_text = _UTF8_BOM # at least the BOM, so the file isn't empty + # The CSV already has the UTF-8 BOM baked in; we hand Flask a plain + # str response and let it encode via utf-8. + safe_name = _safe_filename(filename, default="export.csv") + resp = Response( + csv_text, + status=200, + ) + resp.headers["Content-Type"] = "text/csv; charset=utf-8" + resp.headers["Content-Disposition"] = f'attachment; filename="{safe_name}"' + resp.headers["Cache-Control"] = "no-store" + return resp + + +def json_response(json_text: str, filename: str) -> Response: + """Wrap a JSON string in a Flask ``Response`` with download headers.""" + if not json_text: + json_text = "{}" + safe_name = _safe_filename(filename, default="export.json") + resp = Response( + json_text, + status=200, + ) + resp.headers["Content-Type"] = "application/json; charset=utf-8" + resp.headers["Content-Disposition"] = f'attachment; filename="{safe_name}"' + resp.headers["Cache-Control"] = "no-store" + return resp + + +def _safe_filename(name: str, default: str = "export") -> str: + """Strip path separators / control chars from a download filename.""" + try: + if not name: + return default + # Drop any directory components - browsers only care about the basename. + name = name.replace("\\", "/").split("/")[-1] + # Reject anything that isn't a basic filename char. + cleaned = "".join(c for c in name if c.isprintable() and c not in '"<>|:*?\n\r\t') + return cleaned or default + except Exception: + return default + + +# --- Filename helpers ------------------------------------------------------ + + +def timestamped_filename(prefix: str, ext: str) -> str: + """Build ``prefix_YYYYMMDD_HHMMSS.ext`` (local time).""" + try: + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + except Exception: + ts = "unknown" + # ext should be a plain suffix like "csv" or "json" (no leading dot) + ext = (ext or "").lstrip(".").lower() or "bin" + return f"{prefix}_{ts}.{ext}" \ No newline at end of file diff --git a/geoip_lookup.py b/geoip_lookup.py new file mode 100644 index 0000000..52cb9c9 --- /dev/null +++ b/geoip_lookup.py @@ -0,0 +1,378 @@ +"""GeoIP lookup helpers for Squid Web Manager. + +Two backends are supported: + +1. **Online** — public free `ip-api.com` JSON endpoint (no API key, + 45 requests/min from the same source IP). This is the default + because it requires no extra files and no extra dependencies. +2. **Offline** — a local MaxMind GeoLite2-City ``.mmdb`` file. We + only use the optional ``maxminddb`` package if it's installed; + otherwise we report an error rather than failing the page render. + +Both backends write to a small in-process dict cache so the same IP +is never looked up twice within ``CACHE_TTL`` seconds (default 24 h). + +Private IPs (RFC 1918 / loopback / link-local) are short-circuited and +return a GeoIPResult with ``error="private IP"`` so they never hit the +network or DB. +""" +from __future__ import annotations + +import ipaddress +import json +import socket +import time +import urllib.error +import urllib.request +from typing import Any + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +#: In-process cache. Maps ``ip`` -> ``(GeoIPResult, ts)``. +CACHE: dict[str, tuple["GeoIPResult", float]] = {} + +#: Cache entry lifetime, in seconds. Geo data is stable for ~weeks. +CACHE_TTL = 86400 # 24h + +#: ip-api.com endpoint. Field list pulled to a minimum to stay polite. +_API_FIELDS = ( + "status,message,country,countryCode,region,regionName,city," + "lat,lon,isp,org,as,query" +) +_API_URL = "http://ip-api.com/json/{ip}?fields={fields}" + +#: Per-request timeout for the online endpoint. +DEFAULT_TIMEOUT = 5 + +#: User agent — ip-api.com prefers something identifying. +_USER_AGENT = "squid-web-manager/1.0 (geoip-lookup)" + + +# --------------------------------------------------------------------------- +# Result container +# --------------------------------------------------------------------------- + +class GeoIPResult: + """Lightweight GeoIP result. Plain attributes; safe to JSON-serialize.""" + + __slots__ = ( + "ip", + "country", + "country_name", + "city", + "region", + "latitude", + "longitude", + "asn", + "isp", + "error", + ) + + def __init__( + self, + ip: str = "", + country: str = "", + country_name: str = "", + city: str = "", + region: str = "", + latitude: float = 0.0, + longitude: float = 0.0, + asn: str = "", + isp: str = "", + error: str = "", + ) -> None: + self.ip = ip + self.country = country + self.country_name = country_name + self.city = city + self.region = region + self.latitude = latitude + self.longitude = longitude + self.asn = asn + self.isp = isp + self.error = error + + # -- helpers ----------------------------------------------------------- + def ok(self) -> bool: + return not self.error and self.latitude != 0.0 and self.longitude != 0.0 + + def to_dict(self) -> dict[str, Any]: + return { + "ip": self.ip, + "country": self.country, + "country_name": self.country_name, + "city": self.city, + "region": self.region, + "latitude": self.latitude, + "longitude": self.longitude, + "asn": self.asn, + "isp": self.isp, + "error": self.error, + } + + def __repr__(self) -> str: + if self.error: + return f"GeoIPResult({self.ip!r}, error={self.error!r})" + return ( + f"GeoIPResult({self.ip!r}, {self.country_name!r}, " + f"{self.city!r}, ({self.latitude}, {self.longitude}))" + ) + + +# --------------------------------------------------------------------------- +# Private-IP classification (RFC 1918 + loopback + link-local) +# --------------------------------------------------------------------------- + +def _is_private_ip(ip: str) -> bool: + """Return True for RFC 1918 / loopback / link-local addresses.""" + if not ip: + return True + # cheap pre-filter so we don't pay ipaddress.parse for garbage + head = ip.split(".", 1)[0] if "." in ip else "" + if head in ("127", "10"): + return True + try: + addr = ipaddress.ip_address(ip) + except ValueError: + # not an IP at all (could be a hostname) — treat as "not private" + return False + return bool( + addr.is_private + or addr.is_loopback + or addr.is_link_local + or addr.is_multicast + or addr.is_reserved + or addr.is_unspecified + ) + + +# --------------------------------------------------------------------------- +# Online backend — ip-api.com +# --------------------------------------------------------------------------- + +def lookup_online(ip: str, timeout: int = DEFAULT_TIMEOUT) -> GeoIPResult: + """Look up ``ip`` against the public ip-api.com endpoint. + + Returns a :class:`GeoIPResult` whose ``error`` attribute is set on + any failure (network error, bad JSON, private IP, rate-limit…). Never + raises. + """ + result = GeoIPResult(ip=ip) + if not ip: + result.error = "empty ip" + return result + if _is_private_ip(ip): + result.error = "private IP" + return result + + url = _API_URL.format(ip=ip, fields=_API_FIELDS) + req = urllib.request.Request( + url, + headers={ + "User-Agent": _USER_AGENT, + "Accept": "application/json", + }, + ) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + raw = resp.read() + payload = json.loads(raw.decode("utf-8", errors="replace")) + except urllib.error.HTTPError as e: + # 429 = rate limited + result.error = f"http {e.code} {e.reason}" + return result + except urllib.error.URLError as e: + result.error = f"url error: {e.reason}" + return result + except (socket.timeout, TimeoutError): + result.error = "timeout" + return result + except json.JSONDecodeError as e: + result.error = f"json decode: {e.msg}" + return result + except Exception as e: # noqa: BLE001 + result.error = f"unexpected: {type(e).__name__}: {e}" + return result + + if not isinstance(payload, dict): + result.error = "unexpected payload shape" + return result + + if payload.get("status") == "fail": + result.error = payload.get("message") or "lookup failed" + return result + + result.country = (payload.get("countryCode") or "").strip() + result.country_name = (payload.get("country") or "").strip() + result.city = (payload.get("city") or "").strip() + result.region = (payload.get("regionName") or "").strip() + try: + result.latitude = float(payload.get("lat") or 0.0) + except (TypeError, ValueError): + result.latitude = 0.0 + try: + result.longitude = float(payload.get("lon") or 0.0) + except (TypeError, ValueError): + result.longitude = 0.0 + result.asn = (payload.get("as") or "").split()[0] if payload.get("as") else "" + result.isp = (payload.get("isp") or payload.get("org") or "").strip() + return result + + +# --------------------------------------------------------------------------- +# Offline backend — MaxMind GeoLite2-City .mmdb +# --------------------------------------------------------------------------- + +def lookup_offline(ip: str, db_path: str = "") -> GeoIPResult: + """Look up ``ip`` against a local MaxMind GeoLite2-City ``.mmdb``. + + Requires the optional ``maxminddb`` package; if it's not installed, + the function returns a result with ``error="maxminddb not installed"`` + rather than raising. + """ + result = GeoIPResult(ip=ip) + if not ip: + result.error = "empty ip" + return result + if _is_private_ip(ip): + result.error = "private IP" + return result + if not db_path: + result.error = "no db_path provided" + return result + + try: + import maxminddb # type: ignore + except ImportError: + result.error = "maxminddb not installed" + return result + + try: + reader = maxminddb.open_database(db_path) + except Exception as e: # noqa: BLE001 + result.error = f"cannot open db: {e}" + return result + try: + record = reader.get(ip) + except Exception as e: # noqa: BLE001 + reader.close() + result.error = f"db error: {e}" + return result + reader.close() + + if not record: + result.error = "no record" + return result + + country = record.get("country") or {} + iso = country.get("iso_code") or "" + name = (country.get("names") or {}).get("en") or "" + city = ((record.get("city") or {}).get("names") or {}).get("en") or "" + sub = ((record.get("subdivisions") or [{}])[0]) if record.get("subdivisions") else {} + region = (sub.get("names") or {}).get("en") or "" + loc = record.get("location") or {} + try: + lat = float(loc.get("latitude") or 0.0) + lon = float(loc.get("longitude") or 0.0) + except (TypeError, ValueError): + lat = lon = 0.0 + + result.country = iso + result.country_name = name + result.city = city + result.region = region + result.latitude = lat + result.longitude = lon + return result + + +# --------------------------------------------------------------------------- +# Cached unified entry point +# --------------------------------------------------------------------------- + +def _cache_get(ip: str) -> GeoIPResult | None: + entry = CACHE.get(ip) + if not entry: + return None + result, ts = entry + if (time.time() - ts) > CACHE_TTL: + CACHE.pop(ip, None) + return None + return result + + +def _cache_put(result: GeoIPResult) -> None: + if not result.ip: + return + CACHE[result.ip] = (result, time.time()) + + +def lookup( + ip: str, + *, + db_path: str = "", + use_online: bool = True, +) -> GeoIPResult: + """Unified GeoIP entry point. + + Returns a cached :class:`GeoIPResult` if present; otherwise performs + an online or offline lookup depending on ``use_online``. + + - Private IPs short-circuit and are cached too (cheap; also lets + us avoid repeatedly network-attempting them). + - Failures are cached as well — but only briefly. Negative caching + for 60 s is plenty. + """ + cached = _cache_get(ip) + if cached is not None: + return cached + + if _is_private_ip(ip): + result = GeoIPResult(ip=ip, error="private IP") + _cache_put(result) + return result + + if use_online: + result = lookup_online(ip) + elif db_path: + result = lookup_offline(ip, db_path) + else: + result = GeoIPResult(ip=ip, error="no backend") + + _cache_put(result) + return result + + +def cache_clear() -> int: + """Clear the in-process cache. Returns the number of entries purged.""" + n = len(CACHE) + CACHE.clear() + return n + + +def cache_stats() -> dict[str, int]: + """Return a small snapshot of the cache for diagnostics pages.""" + now = time.time() + total = len(CACHE) + fresh = 0 + stale = 0 + for _ip, (_, ts) in CACHE.items(): + if (now - ts) <= CACHE_TTL: + fresh += 1 + else: + stale += 1 + return {"total": total, "fresh": fresh, "stale": stale} + + +__all__ = [ + "GeoIPResult", + "CACHE", + "CACHE_TTL", + "lookup", + "lookup_online", + "lookup_offline", + "cache_clear", + "cache_stats", +] diff --git a/log_parser.py b/log_parser.py new file mode 100644 index 0000000..5bfb8ce --- /dev/null +++ b/log_parser.py @@ -0,0 +1,442 @@ +"""Squid access.log parser and analyzer. + +Squid native access.log line format (default): + time elapsed client action/code size method URL ident hierarchy content-type + +Example: + 1783841223.438 2488 172.16.12.232 TCP_MISS_ABORTED/000 0 GET \ + http://169.254.169.254/metadata/instance/compute - HIER_DIRECT/169.254.169.254 - +""" +from __future__ import annotations + +import re +from collections import Counter, defaultdict +from datetime import datetime, timezone +from urllib.parse import urlsplit + +# --- Result code taxonomy --------------------------------------------------- +# Squid result codes split into categories for analysis. + +RESULT_CATEGORIES = { + # cache hits + "TCP_HIT": "Hit", + "TCP_MEM_HIT": "Hit", + "TCP_NEGATIVE_HIT": "Hit", + "TCP_IMS_HIT": "Hit", + "TCP_REFRESH_HIT": "Hit", + "TCP_REFRESH_FAIL_HIT": "Hit", + "TCP_REF_FAIL_HIT": "Hit", + "TCP_IMS_HIT": "Hit", + "TCP_DENIED_REPLY": "Denied", + # cache misses + "TCP_MISS": "Miss", + "TCP_MISS_ABORTED": "Aborted", + "TCP_REFRESH_MISS": "Miss", + "TCP_CLIENT_REFRESH_MISS": "Miss", + "TCP_SWAPFAIL_MISS": "Miss", + # tunnels / connect + "TCP_TUNNEL": "Tunnel", + "CONNECT": "Tunnel", + # denied + "TCP_DENIED": "Denied", + "TCP_REDIRECT": "Redirect", + "NONE": "Error", + # udp + "UDP_HIT": "Hit", + "UDP_MISS": "Miss", + "UDP_DENIED": "Denied", + "UDP_INVALID": "Error", + "UDP_MISS_NO_FETCH": "Miss", +} + +RESULT_CODE_DESCRIPTION = { + "TCP_HIT": "Cache hit - served from cache", + "TCP_MEM_HIT": "Memory cache hit", + "TCP_NEGATIVE_HIT": "Negative cached (e.g. 404)", + "TCP_IMS_HIT": "IMS hit (If-Modified-Since, 304)", + "TCP_MISS": "Cache miss - fetched from origin", + "TCP_MISS_ABORTED": "Miss - client aborted the request", + "TCP_TUNNEL": "CONNECT tunnel established (HTTPS proxy)", + "TCP_DENIED": "Request denied by ACL", + "TCP_DENIED_REPLY": "Reply denied by ACL", + "TCP_REFRESH_HIT": "Revalidation: still fresh", + "TCP_REFRESH_MISS": "Revalidation: stale, re-fetched", + "TCP_CLIENT_REFRESH_MISS": "Client forced refresh (Ctrl+F5)", + "TCP_REDIRECT": "Redirected by redirector", + "TCP_SWAPFAIL_MISS": "Swapfile miss (cache corruption?)", + "NONE": "Error before request was processed", + "UDP_HIT": "ICP cache hit", + "UDP_MISS": "ICP cache miss", + "UDP_DENIED": "ICP request denied", + "UDP_INVALID": "Invalid ICP request", +} + +# hierarchy code -> short label +HIERARCHY_LABELS = { + "HIER_DIRECT": "Direct (no peer)", + "HIER_NONE": "None", + "HIER_SIBLING_HIT": "Sibling hit", + "HIER_PARENT_HIT": "Parent hit", + "HIER_DEFAULT_PARENT": "Default parent", + "HIER_SINGLE_PARENT": "Single parent", + "HIER_FIRST_UP_PARENT": "First up parent", + "HIER_NO_DIRECT_FAIL": "No direct fail", + "HIER_SRC_RTT_FASTEST": "Fastest RTT", + "HIER_NEAREST_NEIGHBOR_HIT": "Nearest neighbor hit", + "HIER_CD_SIBLING_HIT": "CARP sibling hit", + "HIER_CD_PARENT_HIT": "CARP parent hit", + "HIER_CARPLIST": "CARP list", + "HIER_MULTICAST_SIBLING": "Multicast sibling", + "HIER_ANY_PARENT": "Any parent", + "HIER_INVALID_CODE": "Invalid", +} + +_LOG_LINE_RE = re.compile( + r"""^ + (?P