Replace inventory-counting risk with bounded workload scoring and separate confidence

This commit is contained in:
2026-10-30 12:34:39 -07:00
parent f65ade9fff
commit 57e716b710
26 changed files with 10351 additions and 87 deletions

View File

@@ -22,7 +22,7 @@ Report source is included in locally collected server metadata. Import removes t
## Analysis
Each finding has classification, supporting evidence, severity and points. Score is capped at 100: 0 informational, 1–29 low, 30–59 medium, 60–89 high, 90–100 critical. This is demonstration heuristic logic, not a validated production risk model. Known Linux application service names receive potential-dependency findings; other running services remain unknown rather than all being classified as application workloads. Incomplete checks and observation limits always require review.
Model v2 groups workload evidence, selects its strongest rule and takes the maximum workload score. It never sums inventory observations. `dependencies.py` classifies OS/client/unknown activity separately and correlates positive nonzero PIDs with listeners; `policy.py` validates the data-only weights/thresholds; `assessment.py` computes the audit breakdown and independent coverage/confidence. Full policy identity, evidence references and score-determining groups are serialized. Thresholds and limitations are documented in SCORING.md. No result grants retirement approval.
## Verification
@@ -49,3 +49,9 @@ OS ProductType 2/3 plus build mapping gates Server 2016/2019/2022/2025. Workstat
`models/inventory.py` now owns the shared v1 validator and field allowlists. Metadata fields are additive and old demo/Linux results remain valid. Live Windows results are projected to safe fields; raw CIM instances are never serialized. Imported errors retain recognized categorical codes and timestamps but discard arbitrary diagnostic/permission text. Imports cannot assert the trusted local source label. Registry software covers both read-only machine uninstall views; services omit PathName; tasks emit safe action basenames and selected triggers rather than Arguments/XML; certificates never access private-key properties. Reports include all check statuses, timestamps and limitations.
Fixtures for all four Server releases live in tests/fixtures/windows. Python tests cover contract/identity/array handling, partial permission results, invalid output, process cancellation/reaping, timeouts, output bounds, secret-field exclusion, imports and the unchanged GUI/history/report flow. `tests/powershell/Test-Collectors.ps1` uses the native PowerShell AST parser and synthetic function/registry objects to test the scripts without querying the host; it runs automatically where PowerShell is present. This test is skipped in the current Linux environment. **No real Windows inventory has yet been produced here; do not mark the Windows milestone complete.** Use the README's explicit real-host validation command and review private reports before recording compatibility.
## Model v2 UI/report behavior
Settings can load validated scoring JSON or restore defaults. The policy is snapshotted at job start; history keeps prior model/policy results. Summary displays evidence coverage and confidence independently of workload risk. Networking augments the table with inferred/reported direction and rationale without changing the raw rows. Dependencies excludes OS/unknown context; Risk Findings retains zero-point review context and the exact maximum-score formula. The HTML report writes a content-addressed full-inventory sidecar and links it rather than embedding raw inventory. Tests cover sidecar collision safety, escaping, preservation and reimport.
Windows shares now optionally query Get-SmbOpenFile read-only. They match full paths against share root plus ShareRelativePath, avoiding administrative drive access being mistaken for nested business-share use. Only aggregate counts/time/source are exported. Same-root administrative aliases are ambiguous. Native scripts/tests require Windows validation; the Python fixtures distinguish confirmed aggregate activity from generic SMB transport.

View File

@@ -34,3 +34,7 @@ The original empty repository had no existing application or project instruction
## Windows local collection milestone
Added independent read-only PowerShell modules and the local Python backend, without replacing the existing application. OS identity is accepted only for known Server build/product-type combinations. The existing QThread routes Windows auto-detect/profile choices to the backend and retains the same result, history and reporting flow. Python runs each fixed local section in a separate supervised PowerShell process, with cancellation, per-section/overall budgets, output bounds and cleanup. Per-check UTC timestamps, sanitized errors, scope and permission limitations extend schema v1 additively; old fixtures remain valid. The real-host validation tool produces full private evidence and a separate metadata-only summary for review. Remote Windows collection is not part of this milestone.
## Retirement scoring correction
Implemented model v2 using the existing analyzer/report/UI boundaries: workload correlation and OS/client/unknown classification, maximum-of-grouped-evidence scoring, independent coverage/confidence, validated configurable policy and full audit metadata. HTML now leads with actionable findings and saves complete inventory separately. Regression scenarios and the synthetic 1,565-point incident reconstruction are documented in SCORING.md. Remote collection remains outside this change.

65
docs/SCORING.md Normal file
View File

@@ -0,0 +1,65 @@
# Retirement scoring model v2
## Purpose and migration
The previous engine added points for every running service, installed role, listener, connection and administrative share. A routine Windows snapshot could accumulate thousands of raw points and saturate at Critical. Model v2 removes that inventory-counting behavior. Collection schema remains v1; assessment metadata is versioned separately as `model_version: 2.0`.
Re-analyze a previous JSON report through **Analyze → Saved inventory file**. Its raw inventory and collection time are retained; the old assessment is ignored and a v2 assessment is calculated with the selected policy. No new Windows access is needed to re-score existing evidence. Imported data remain explicitly unverified. Old HTML scores do not change; generate a new report. Scores from v1 and v2 are not directly comparable.
## Evidence classifications
- **Confirmed workload dependency:** observed activity attributed to a recognized application listener, a verified domain-controller identity, or read-only SMB open-file evidence correlated to a business share root. For TCP this confirms observed transport activity, not connection initiation, business ownership, application health, or permission to retire.
- **Potential workload dependency:** a meaningful installed workload role, running recognized application service, attributed listener, non-administrative share or nonroutine configured schedule. Configuration alone does not prove current use.
- **Standard operating system activity:** baseline services/components, administrative shares, management networking and recognized maintenance schedules. Zero workload points; complete data remain in inventory.
- **Unknown / insufficient evidence:** unmatched services, direction/ownership gaps, client activity, installed/stopped historical application hints and failed checks. Unknown observations never imply an active workload and never imply non-use.
Classification is conservative and name-based in places. Unknown applications, custom service names and interpreters such as Java cannot be reliably mapped from a port alone. SQL client libraries, stopped database/game services, browsers and Steam client traffic do not establish hosted workloads. Review unmatched services with owners; a low score can reflect rule/collection gaps.
## Bounded scoring
1. Group correlated observations by workload family (SQL, IIS, web, file sharing, etc.). Multiple instances or clients in a family do not add points.
2. Select the strongest applicable rule within each workload group. A service, its listener and 100 connections remain one workload score.
3. Final score is **the maximum workload score**, or zero if none are scored. Scores are never added, even across independent workload families. Independent workloads still appear separately for administrator review.
| Strongest evidence | Default workload score |
| --- | --- |
| Meaningful installed workload role only | 15 |
| Running recognized application service | 35 |
| Listener attributed to a recognized application | 45 |
| Application transport activity on an attributed listener | 70 |
| Configured non-administrative share | 20 |
| File-sharing transport with share access still unverified | 40 |
| Business-share-root-correlated open files | 80 |
| Enabled custom/nonroutine schedule | 20 |
| CIM ProductType=2 domain controller identity | 95 |
Thresholds: 0 informational, 1–24 low, 25–59 medium, 60–89 high, 90–100 critical. These are triage categories, not probabilities or a validated retirement-readiness percentage. A running IIS service without identifiable worker traffic is potential evidence (35), not automatically an actively consumed application. HTTP.sys `System`/PID 4 traffic alone cannot identify an IIS site; those connections remain unknown unless more precise attribution is available. A role install without a domain-controller identity is not a verified DC.
The bundled policy is `samsara/assets/scoring_policy.json`. Copy it, adjust documented integer weights/ordered thresholds and coverage weights, then use **Settings → Load scoring policy JSON**. Data-only policies accept no executable expressions. Changes apply to future assessments; history retains earlier results and policy snapshots. Every assessment includes the full policy, SHA-256 identity, selected rules, evidence references and which groups determine the final maximum. Co-leading groups can share that maximum; their scores still are not added. Raw evidence references such as `services[4]` and `network[12]` point into the separately preserved inventory arrays.
## Direction and correlation
- Listener address/port plus the same nonzero owning PID can support **Inbound (inferred)**. This does not capture the initiating TCP handshake. A known application attribution is also required to score observed application clients.
- Explicit source direction is labeled **reported**, not independently verified, and still requires listener/process correlation to upgrade to observed application-client evidence.
- Ephemeral local ports to non-ephemeral remote ports without an attributed listener can be **Outbound (inferred)**. The port-range assumption is heuristic and may differ on configured hosts.
- Loopback, IPv4-mapped loopback and same-host addresses are identified first. These and outbound/unknown directions do not become external client evidence.
- A bare 443, 1433 or game-related port never identifies an application. Steam and browser process activity never create a dedicated game-server workload.
- Listener matching uses local address compatibility and a positive PID, not just a common port. HTTP.sys, shared processes, missing PIDs, process exits, dual-stack behavior and non-TCP activity can leave attribution unknown.
File-server confirmation is stricter than TCP 445: a connection could be administrative access. `Get-SmbOpenFile` reads open-file metadata; the collector requires `Path = configured share root + ShareRelativePath` to distinguish access through an administrative drive root from a nested business share. It exports only aggregate counts and observation time. Same-root administrative aliases are explicitly ambiguous and cannot confirm business-share use. Same-root business aliases cannot be individually distinguished and remain one workload family. File contents, file names, session user names and client identities are not exported by this query. Denied access reduces confidence. Zero open files never proves inactivity. See [Microsoft's Get-SmbOpenFile metadata](https://learn.microsoft.com/en-us/powershell/module/smbshare/get-smbopenfile).
## Coverage, confidence and review
Coverage is calculated separately from weighted check statuses: complete=1, partial=0.5 by default, failed/not_collected/missing=0. Permission limitations or errors on a nominally complete check conservatively receive partial credit. Required scopes and weights are explicit in policy. Scope limitations continue to apply even with 100% coverage.
Evidence confidence is low below 60% coverage, moderate below 90% or with incomplete checks/unclassified services/unknown directions, and high otherwise. Confidence describes the collected and classified snapshot, not historical inactivity. It is not a statistical confidence interval. Checks with access errors trigger manual review; they never add workload score. Positive evidence is not erased when unrelated collection is incomplete.
Historical monitoring over a representative business cycle, infrequent workloads, ownership, service accounts, external dependencies, backups/retention, rollback and administrator approval must be verified before retirement. `review_required` is always true. The application does not declare a server safe to retire.
## Reporting and validation
HTML leads with actionable workload groups, scoring rationale, confidence and missing checks. Routine inventory is summarized in zero-point context groups rather than repeated as medium-risk dependencies. A content-addressed `.inventory-<hash>.json` sidecar preserves every collected inventory row; its SHA-256 and link are in the report. Keep it with the HTML. JSON assessment export also retains full inventory, model/policy and detailed observations. Existing different sidecars are never overwritten.
Regression fixtures cover a clean Windows Server, an inactive former game server, active SQL, active IIS, a domain controller, business file-share use and incomplete collection. An additional regression reconstructs the reported 1,565-point pattern (86 services, 19 connections, 14 features, 23 listeners, three admin shares and two failed checks): v2 gives zero workload risk and reduced confidence, while retaining all observations. This is a synthetic reproduction of the supplied counts, not a replay of an unavailable original inventory file.
These tests do not certify business dependency discovery. Re-score the real test server's saved JSON, review results with its owner, and validate any new SMB activity fields on Windows. Native PowerShell tests remain skipped in the Linux development environment.