Fix Windows task and certificate collector reliability
This commit is contained in:
123
docs/WINDOWS-COLLECTOR-RELIABILITY.md
Normal file
123
docs/WINDOWS-COLLECTOR-RELIABILITY.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Windows collector reliability validation — 2026-10-30
|
||||
|
||||
## Root causes
|
||||
|
||||
The supplied Server 2025 assessment reported tasks failed and 16 certificate
|
||||
errors, but its generic errors did not retain exception types. Read-only native
|
||||
reproduction established two defects:
|
||||
|
||||
* Tasks: one action executable made .NET Framework `Path.GetFileName` throw a
|
||||
`MethodInvocationException` wrapping `ArgumentException`. The projection was
|
||||
inside a single assignment/query catch, so that one value discarded all tasks.
|
||||
* Certificates: the Windows certificate provider returned store `PSPath` values
|
||||
such as `Microsoft.PowerShell.Security\Certificate::My`, without LocalMachine.
|
||||
Using those paths produced `ItemNotFoundException` / `ObjectNotFound` for each
|
||||
store. Store-list success alone incorrectly gave partial credit when every
|
||||
store query failed.
|
||||
|
||||
The original code already enumerated stores separately; recursion was not the
|
||||
cause. Missing modules and JSON conversion were checked as failure paths, but
|
||||
were not the reproduced causes on this host.
|
||||
|
||||
## Changes
|
||||
|
||||
Executable basenames now use string operations and the existing strict allowlist,
|
||||
avoiding filesystem path validation. Malformed values become an omission marker;
|
||||
task records remain visible. Each task is projected and JSON-checked separately.
|
||||
Principal group accounts are used when UserId is absent. Arguments, XML, inline
|
||||
scripts, event queries and credentials are never projected.
|
||||
|
||||
Certificates use explicit `Cert:\LocalMachine\<store>` paths. Each store query
|
||||
and certificate projection is isolated; empty successful stores receive credit,
|
||||
inaccessible stores produce errors, and listing names alone does not give credit
|
||||
when all listed stores fail. Only subject, issuer, thumbprint, expiration and
|
||||
store are collected. No private key or certificate export APIs are used.
|
||||
|
||||
Read helpers promote nonterminating errors to terminating errors. Per-record JSON
|
||||
checks reject unrepresentable records. Final JSON failures exit with code 3 and
|
||||
Python records `serialization_failed`, never a successful empty collection.
|
||||
Existing subprocess timeout, UTF-8, output limits and schema validation remain.
|
||||
|
||||
Optional `-Diagnostic` / Python `--diagnostic` adds exception class and category to
|
||||
existing sanitized error messages. Fixed command stages appear in error scope.
|
||||
No raw messages, task names, arguments, store names, targets, stack traces or
|
||||
error identifiers are included in diagnostics. Inventory remains sensitive and
|
||||
should be stored privately. Task/certificate UI detail pages display these error
|
||||
scopes, codes and messages; Collection checks and reports retain their contract.
|
||||
|
||||
The production Python backend deliberately continues to use trusted Windows
|
||||
PowerShell 5.1. Direct PowerShell 7 invocation also includes the inbox Windows
|
||||
module directory, without opening module lookup to user directories. Commands
|
||||
autoload from those inbox module paths; missing commands are explicit failures.
|
||||
No scoring policy, Linux collector or other inventory module was changed.
|
||||
|
||||
## Results
|
||||
|
||||
On the local Windows Server 2025 host, native 5.1 collection returned:
|
||||
|
||||
| Collector | Status | Records | Errors |
|
||||
| --- | --- | ---: | ---: |
|
||||
| Tasks | complete | 192 (46 disabled) | 0 |
|
||||
| Certificates | complete | 49 across 7 populated stores | 0 |
|
||||
|
||||
Full Python subprocess collection passed schema validation, returned the same
|
||||
record counts, and completed all nine checks. Evidence coverage rose from the
|
||||
supplied assessment's 81% to 100%. Confidence remained moderate under the existing
|
||||
classification policy; complete collection does not establish retirement safety.
|
||||
|
||||
`python -m unittest discover -s tests -v`: **64 tests, 62 passed, 2 skipped**.
|
||||
Skips: the Linux-only subprocess test on Windows and PowerShell 7 (not installed).
|
||||
UI navigation/history/export tests, Linux parsing tests, scoring regressions,
|
||||
Windows subprocess limits and schema tests passed. Native PowerShell AST and
|
||||
synthetic tests passed under 5.1. Added cases cover normal/disabled/empty tasks,
|
||||
malformed actions, per-task failures, nonterminating query denial, multiple and
|
||||
empty stores, partial/all-store denial, redacted diagnostics and serialization
|
||||
failures. A dedicated regression checks lower coverage/confidence without risk
|
||||
inflation for failed tasks and partial certificates.
|
||||
|
||||
## Exact validation commands
|
||||
|
||||
Run from the repository root in an authorized elevated Windows PowerShell console.
|
||||
These commands only read inventory; they do not change Server configuration or
|
||||
execution policy. Choose a new output path for each saved inventory.
|
||||
|
||||
```powershell
|
||||
$PSVersionTable.PSVersion
|
||||
powershell.exe -NoLogo -NoProfile -NonInteractive -File .\tests\powershell\Test-Collectors.ps1
|
||||
python -m unittest discover -s tests -v
|
||||
|
||||
foreach ($section in @('tasks', 'certificates')) {
|
||||
$raw = & powershell.exe -NoLogo -NoProfile -NonInteractive -File .\powershell\Get-ServerInventory.ps1 -Section $section -Diagnostic
|
||||
if ($LASTEXITCODE -ne 0) { throw 'Collector process failed' }
|
||||
$r = $raw | ConvertFrom-Json
|
||||
[pscustomobject]@{
|
||||
Section = $section
|
||||
Status = $r.status
|
||||
Records = @($r.data).Count
|
||||
Errors = $r.errors
|
||||
} | ConvertTo-Json -Depth 5
|
||||
if ($r.status -ne 'complete') { throw 'Review incomplete collection' }
|
||||
if (@($r.data).Count -eq 0) { throw 'No records: verify actual host contents independently' }
|
||||
}
|
||||
|
||||
python -m samsara.collectors.windows --diagnostic --output .\windows-validation-new.json
|
||||
python .\tools\validate_windows_host.py --output-dir .\windows-validation-new
|
||||
|
||||
# When PowerShell 7 is already installed:
|
||||
pwsh.exe -NoLogo -NoProfile -NonInteractive -File .\tests\powershell\Test-Collectors.ps1
|
||||
pwsh.exe -NoLogo -NoProfile -NonInteractive -File .\powershell\Get-ServerInventory.ps1 -Section tasks -Diagnostic
|
||||
pwsh.exe -NoLogo -NoProfile -NonInteractive -File .\powershell\Get-ServerInventory.ps1 -Section certificates -Diagnostic
|
||||
```
|
||||
|
||||
The nonempty assertion above is a validation expectation for this host, not a
|
||||
collector rule: legitimately empty successful queries remain complete.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
PowerShell 7 runtime execution and other Server releases were not available for
|
||||
live validation. Tests run under each runtime when installed; 5.1 syntax and
|
||||
Server 2016/2019/2022/2025 identity fixtures pass. Access restrictions may hide
|
||||
objects even when an API reports success. A terminating failure of the initial
|
||||
task enumeration cannot recover task registrations the API never returned, and
|
||||
is reported failed. Inventory is a current observation, not historical activity.
|
||||
No Windows Server configuration was modified.
|
||||
Reference in New Issue
Block a user