Resiliate® CLI & API Testing Framework
1. Introduction
Purpose: Ensure the aifs CLI and REST API workflows meet end users’ operational needs and expectations.
Scope: Covers all primary aifs subcommands (config, filesystem, storage, rectify), API endpoints, documentation examples, error messaging, and beta feedback.
2. Personas & CLI Journeys
- Security Analyst:
aifs policy apply(deprecated),aifs alerts list(missing), fetch remediation via API. - DevOps Engineer: Automate setup (
aifs config upgrade), check for upgrades (aifs config upgradeable), mount filesystem (aifs filesystem mount /dev/sdb1 /mnt/resiliate), initialize storage (aifs storage init /dev/sdb). - Data Owner: List recovery points (
aifs rectify snapshots /data/project), show details (aifs rectify show /data/project), auto-recover (aifs rectify auto /data/project), verify via API checksums. - Auditor/Compliance Officer: Bind storage (
aifs storage bind /dev/sdb /mnt/storage), export audit logs (aifs audit export --format json), API endpoint/api/v1/audit.
3. Testing Objectives
- Usability: CLI flags reflect documentation and are discoverable via
--help. - Functionality: Commands and endpoints succeed with valid inputs.
- Reliability: Consistent across Linux and macOS environments.
- Error Handling: Non-zero exit codes and clear, actionable error messages.
- Documentation Accuracy: Code snippets execute without manual edits.
4. Test Environment
- Local Sandbox: Docker Compose with mocked Ceph and ZFS services.
- Staging Cluster: Kubernetes cluster replicating production dependencies.
- Data Fixtures: Sample datasets and threat simulation files.
- User Accounts: CLI profiles and API tokens representing each persona.
5. Tooling & Frameworks
- CLI Tests: Bash scripts with
expectand Pythonpytest+subprocess. - API Tests: Python
pytest+requests. - Mock Services: Dockerized Ceph, ZFS, telemetry stubs.
- Documentation Validation: Scripts extracting and running Markdown code blocks.
6. Commands Coverage & Status
| Group | Command | Status |
|---|---|---|
| config | aifs config upgrade | ✓ Tested |
| config | aifs config upgradeable | ✓ Tested |
| config | aifs config kernel | Missing |
| config | aifs config license | Missing |
| config | aifs config build-info | Missing |
| filesystem | aifs filesystem mount /dev/sdb1 /mnt/resiliate | Missing |
| filesystem | aifs filesystem umount /mnt/resiliate | Missing |
| filesystem | aifs filesystem mkfs /dev/sdb1 | Missing |
| storage | aifs storage init /dev/sdb | Missing |
| storage | aifs storage bind /dev/sdb /mnt/storage | Missing |
| storage | aifs storage --version | Missing |
| rectify | aifs rectify snapshots /data/project | Missing |
| rectify | aifs rectify show /data/project | ✓ Tested |
| rectify | aifs rectify auto /data/project | Missing |
| rectify | aifs rectify ransomware /data/project | Missing |
| rectify | aifs rectify wiperware /data/project | Missing |
| rectify | aifs rectify scan /data/project | Missing |
| rectify | aifs rectify cleanup /data/project | Missing |
| other CLI | aifs version | Missing |
| other CLI | aifs simulate threat | Not a valid cmd |
| other CLI | aifs policy | Not in docs |
7. Directory Layout
tests/user/
├── cli/
│ ├── test_config_upgrade.py
│ ├── test_config_upgradeable.py
│ ├── test_filesystem_mount.sh
│ ├── test_storage_init.sh
│ ├── test_rectify_show.sh
│ └── ...
└── api/
├── test_audit_endpoints.py
└── ...
8. Sample CLI Test Case
#!/usr/bin/env bash
set -e
# tests/user/cli/test_rectify_show.sh
OUTPUT=$(aifs rectify show /data/project)
if [[ "$OUTPUT" != *"Parsed timestamps"* ]]; then
echo "Expected human-readable snapshots, got: $OUTPUT"
exit 1
fi
9. CI Integration
- Workflow File:
ci-user-tests-cli.ymlon GitHub Actions. - Matrix: Ubuntu and macOS runners.
- Parallelization: Splitting jobs by command group.
- Reporting: JUnit XML for CLI tests; HTML reports for API tests.
10. Maintenance & Next Steps
- Prioritize writing tests for commands highlighted in red.
- Keep fixtures and mocks updated when CLI flags or API endpoints change.
- Review coverage regularly against docs.saf.ai.