frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-08-10 20:31:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(repo): move v1/ → archive/v1/ + add archive/README.md (#430)

Browse Source 
The Rust port at v2/ has been the primary codebase since the rename
in #427. The Python implementation at v1/ is no longer the active
target; the only load-bearing path is the deterministic proof bundle
at v1/data/proof/ (per ADR-011 / ADR-028 witness verification).

Move the whole Python tree into archive/v1/ and document the policy
in archive/README.md: no new features, bug fixes only when they affect
a still-load-bearing path (currently just the proof), CI continues to
verify the proof on every push and PR.

Path references updated in 26 files via path-pattern sed (only
matches v1/<known-child> patterns, never bare v1 or API URLs like
/api/v1/). Two double-prefix typos (archive/archive/v1/) caught and
hand-fixed in verify-pipeline.yml and ADR-011.

Validated:
- Python proof verify.py imports cleanly at archive/v1/data/proof/
  (numpy/scipy still required; CI installs requirements-lock.txt
  from archive/v1/ now)
- cargo test --workspace --no-default-features → 1,539 passed,
  0 failed, 8 ignored (unaffected by Python tree relocation)
- ESP32-S3 on COM7 untouched (no firmware paths changed)

After-merge: contributors should re-run any local `python v1/...`
commands as `python archive/v1/...` (CLAUDE.md and CHANGELOG already
updated).
This commit is contained in:
rUv
2026-04-25 23:07:52 -04:00
committed by GitHub
parent 74233cfb23
commit 81cc241b9e
183 changed files with 290 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archive/v1/README.md
+54
View File
@@ -0,0 +1,54 @@
# WiFi-DensePose v1 (Python Implementation)
This directory contains the original Python implementation of WiFi-DensePose.
## Structure
```
v1/
├── src/ # Python source code
│ ├── api/ # REST API endpoints
│ ├── config/ # Configuration management
│ ├── core/ # Core processing logic
│ ├── database/ # Database models and migrations
│ ├── hardware/ # Hardware interfaces
│ ├── middleware/ # API middleware
│ ├── models/ # Neural network models
│ ├── services/ # Business logic services
│ └── tasks/ # Background tasks
├── tests/ # Test suite
├── docs/ # Documentation
├── scripts/ # Utility scripts
├── data/ # Data files
├── setup.py # Package setup
├── test_application.py # Application tests
└── test_auth_rate_limit.py # Auth/rate limit tests
```
## Requirements
- Python 3.10+
- PyTorch 2.0+
- FastAPI
- PostgreSQL/SQLite
## Installation
```bash
cd v1
pip install -e .
```
## Usage
```bash
# Start API server
python -m src.main
# Run tests
pytest tests/
```
## Note
This is the legacy Python implementation. For the new Rust implementation with improved performance, see `/v2/`.
archive/v1/__init__.py
+1
View File
@@ -0,0 +1 @@
# WiFi-DensePose v1 package
archive/v1/data/proof/expected_features.sha256
+1
View File
@@ -0,0 +1 @@
8c0680d7d285739ea9597715e84959d9c356c87ee3ad35b5f1e69a4ca41151c6
archive/v1/data/proof/generate_reference_signal.py
+324
View File
@@ -0,0 +1,324 @@
#!/usr/bin/env python3
"""
Deterministic Reference CSI Signal Generator for WiFi-DensePose Proof Bundle.
This script generates a SYNTHETIC, DETERMINISTIC CSI (Channel State Information)
reference signal for pipeline verification. It is NOT a real WiFi capture.
The signal models a 3-antenna, 56-subcarrier WiFi system with:
- Human breathing modulation at 0.3 Hz
- Walking motion modulation at 1.2 Hz
- Structured (deterministic) multipath propagation with known delays
- 10 seconds of data at 100 Hz sampling rate (1000 frames total)
Generation Formula
==================
For each frame t (t = 0..999) at time s = t / 100.0:
CSI[antenna_a, subcarrier_k] = sum over P paths of:
A_p * exp(j * (2*pi*f_k*tau_p + phi_p,a))
* (1 + alpha_breathe * sin(2*pi * 0.3 * s + psi_breathe_a))
* (1 + alpha_walk * sin(2*pi * 1.2 * s + psi_walk_a))
Where:
- f_k = center_freq + (k - 28) * subcarrier_spacing [subcarrier frequency]
- tau_p = deterministic path delay for path p
- A_p = deterministic path amplitude for path p
- phi_p,a = deterministic phase offset per path per antenna
- alpha_breathe = 0.02 (breathing modulation depth)
- alpha_walk = 0.08 (walking modulation depth)
- psi_breathe_a, psi_walk_a = deterministic per-antenna phase offsets
All parameters are computed from numpy with seed=42. No randomness is used
at generation time -- the seed is used ONLY to select fixed parameter values
once, which are then documented in the metadata file.
Output:
- sample_csi_data.json: All 1000 CSI frames with amplitude and phase arrays
- sample_csi_meta.json: Complete parameter documentation
Author: WiFi-DensePose Project (synthetic test data)
"""
import json
import os
import sys
import numpy as np
def generate_deterministic_parameters():
"""Generate all fixed parameters using seed=42.
These parameters define the multipath channel model and human motion
modulation. Once generated, they are constants -- no further randomness
is used.
Returns:
dict: All channel and motion parameters.
"""
rng = np.random.RandomState(42)
# System parameters (fixed by design, not random)
num_antennas = 3
num_subcarriers = 56
sampling_rate_hz = 100
duration_s = 10.0
center_freq_hz = 5.21e9 # WiFi 5 GHz channel 42
subcarrier_spacing_hz = 312.5e3 # Standard 802.11n/ac
# Multipath channel: 5 deterministic paths
num_paths = 5
# Path delays in nanoseconds (typical indoor)
path_delays_ns = np.array([0.0, 15.0, 42.0, 78.0, 120.0])
# Path amplitudes (linear scale, decreasing with delay)
path_amplitudes = np.array([1.0, 0.6, 0.35, 0.18, 0.08])
# Phase offsets per path per antenna (from seed=42, then fixed)
path_phase_offsets = rng.uniform(-np.pi, np.pi, size=(num_paths, num_antennas))
# Human motion modulation parameters
breathing_freq_hz = 0.3
walking_freq_hz = 1.2
breathing_depth = 0.02 # 2% amplitude modulation
walking_depth = 0.08 # 8% amplitude modulation
# Per-antenna phase offsets for motion signals (from seed=42, then fixed)
breathing_phase_offsets = rng.uniform(0, 2 * np.pi, size=num_antennas)
walking_phase_offsets = rng.uniform(0, 2 * np.pi, size=num_antennas)
return {
"num_antennas": num_antennas,
"num_subcarriers": num_subcarriers,
"sampling_rate_hz": sampling_rate_hz,
"duration_s": duration_s,
"center_freq_hz": center_freq_hz,
"subcarrier_spacing_hz": subcarrier_spacing_hz,
"num_paths": num_paths,
"path_delays_ns": path_delays_ns,
"path_amplitudes": path_amplitudes,
"path_phase_offsets": path_phase_offsets,
"breathing_freq_hz": breathing_freq_hz,
"walking_freq_hz": walking_freq_hz,
"breathing_depth": breathing_depth,
"walking_depth": walking_depth,
"breathing_phase_offsets": breathing_phase_offsets,
"walking_phase_offsets": walking_phase_offsets,
}
def generate_csi_frames(params):
"""Generate all CSI frames deterministically from the given parameters.
Args:
params: Dictionary of channel/motion parameters.
Returns:
list: List of dicts, each containing amplitude and phase arrays
for one frame, plus timestamp.
"""
num_antennas = params["num_antennas"]
num_subcarriers = params["num_subcarriers"]
sampling_rate = params["sampling_rate_hz"]
duration = params["duration_s"]
center_freq = params["center_freq_hz"]
subcarrier_spacing = params["subcarrier_spacing_hz"]
num_paths = params["num_paths"]
path_delays_ns = params["path_delays_ns"]
path_amplitudes = params["path_amplitudes"]
path_phase_offsets = params["path_phase_offsets"]
breathing_freq = params["breathing_freq_hz"]
walking_freq = params["walking_freq_hz"]
breathing_depth = params["breathing_depth"]
walking_depth = params["walking_depth"]
breathing_phase = params["breathing_phase_offsets"]
walking_phase = params["walking_phase_offsets"]
num_frames = int(duration * sampling_rate)
# Precompute subcarrier frequencies relative to center
k_indices = np.arange(num_subcarriers) - num_subcarriers // 2
subcarrier_freqs = center_freq + k_indices * subcarrier_spacing
# Convert path delays to seconds
path_delays_s = path_delays_ns * 1e-9
frames = []
for frame_idx in range(num_frames):
t = frame_idx / sampling_rate
# Build complex CSI matrix: (num_antennas, num_subcarriers)
csi_complex = np.zeros((num_antennas, num_subcarriers), dtype=complex)
for a in range(num_antennas):
# Human motion modulation for this antenna at this time
breathing_mod = 1.0 + breathing_depth * np.sin(
2.0 * np.pi * breathing_freq * t + breathing_phase[a]
)
walking_mod = 1.0 + walking_depth * np.sin(
2.0 * np.pi * walking_freq * t + walking_phase[a]
)
motion_factor = breathing_mod * walking_mod
for p in range(num_paths):
# Phase shift from path delay across subcarriers
phase_from_delay = 2.0 * np.pi * subcarrier_freqs * path_delays_s[p]
# Add per-path per-antenna offset
total_phase = phase_from_delay + path_phase_offsets[p, a]
# Accumulate path contribution
csi_complex[a, :] += (
path_amplitudes[p] * motion_factor * np.exp(1j * total_phase)
)
amplitude = np.abs(csi_complex)
phase = np.angle(csi_complex) # in [-pi, pi]
frames.append({
"frame_index": frame_idx,
"timestamp_s": round(t, 4),
"amplitude": amplitude.tolist(),
"phase": phase.tolist(),
})
return frames
def save_data(frames, params, output_dir):
"""Save CSI frames and metadata to JSON files.
Args:
frames: List of CSI frame dicts.
params: Generation parameters.
output_dir: Directory to write output files.
"""
# Save CSI data
csi_data = {
"description": (
"SYNTHETIC deterministic CSI reference signal for pipeline verification. "
"This is NOT a real WiFi capture. Generated mathematically with known "
"parameters for reproducibility testing."
),
"generator": "generate_reference_signal.py",
"generator_version": "1.0.0",
"numpy_seed": 42,
"num_frames": len(frames),
"num_antennas": params["num_antennas"],
"num_subcarriers": params["num_subcarriers"],
"sampling_rate_hz": params["sampling_rate_hz"],
"frequency_hz": params["center_freq_hz"],
"bandwidth_hz": params["subcarrier_spacing_hz"] * params["num_subcarriers"],
"frames": frames,
}
data_path = os.path.join(output_dir, "sample_csi_data.json")
with open(data_path, "w") as f:
json.dump(csi_data, f, indent=2)
print(f"Wrote {len(frames)} frames to {data_path}")
# Save metadata
meta = {
"description": (
"Metadata for the SYNTHETIC deterministic CSI reference signal. "
"Documents all generation parameters so the signal can be independently "
"reproduced and verified."
),
"is_synthetic": True,
"is_real_capture": False,
"generator_script": "generate_reference_signal.py",
"numpy_seed": 42,
"system_parameters": {
"num_antennas": params["num_antennas"],
"num_subcarriers": params["num_subcarriers"],
"sampling_rate_hz": params["sampling_rate_hz"],
"duration_s": params["duration_s"],
"center_frequency_hz": params["center_freq_hz"],
"subcarrier_spacing_hz": params["subcarrier_spacing_hz"],
"total_frames": int(params["duration_s"] * params["sampling_rate_hz"]),
},
"multipath_channel": {
"num_paths": params["num_paths"],
"path_delays_ns": params["path_delays_ns"].tolist(),
"path_amplitudes": params["path_amplitudes"].tolist(),
"path_phase_offsets_rad": params["path_phase_offsets"].tolist(),
"description": (
"5-path indoor multipath model with deterministic delays and "
"amplitudes. Path amplitudes decrease with delay (typical indoor)."
),
},
"human_motion_signals": {
"breathing": {
"frequency_hz": params["breathing_freq_hz"],
"modulation_depth": params["breathing_depth"],
"per_antenna_phase_offsets_rad": params["breathing_phase_offsets"].tolist(),
"description": (
"Sinusoidal amplitude modulation at 0.3 Hz modeling human "
"breathing (typical adult resting rate: 12-20 breaths/min = 0.2-0.33 Hz)."
),
},
"walking": {
"frequency_hz": params["walking_freq_hz"],
"modulation_depth": params["walking_depth"],
"per_antenna_phase_offsets_rad": params["walking_phase_offsets"].tolist(),
"description": (
"Sinusoidal amplitude modulation at 1.2 Hz modeling human "
"walking motion (typical stride rate: ~1.0-1.4 Hz)."
),
},
},
"generation_formula": (
"CSI[a,k,t] = sum_p { A_p * exp(j*(2*pi*f_k*tau_p + phi_{p,a})) "
"* (1 + d_breathe * sin(2*pi*0.3*t + psi_breathe_a)) "
"* (1 + d_walk * sin(2*pi*1.2*t + psi_walk_a)) }"
),
"determinism_guarantee": (
"All parameters are derived from numpy.random.RandomState(42) at "
"script initialization. The generation loop itself uses NO randomness. "
"Running this script on any platform with the same numpy version will "
"produce bit-identical output."
),
}
meta_path = os.path.join(output_dir, "sample_csi_meta.json")
with open(meta_path, "w") as f:
json.dump(meta, f, indent=2)
print(f"Wrote metadata to {meta_path}")
def main():
"""Main entry point."""
# Determine output directory
output_dir = os.path.dirname(os.path.abspath(__file__))
print("=" * 70)
print("WiFi-DensePose: Deterministic Reference CSI Signal Generator")
print("=" * 70)
print(f"Output directory: {output_dir}")
print()
# Step 1: Generate deterministic parameters
print("[1/3] Generating deterministic channel parameters (seed=42)...")
params = generate_deterministic_parameters()
print(f" - {params['num_paths']} multipath paths")
print(f" - {params['num_antennas']} antennas, {params['num_subcarriers']} subcarriers")
print(f" - Breathing: {params['breathing_freq_hz']} Hz, depth={params['breathing_depth']}")
print(f" - Walking: {params['walking_freq_hz']} Hz, depth={params['walking_depth']}")
print()
# Step 2: Generate all frames
num_frames = int(params["duration_s"] * params["sampling_rate_hz"])
print(f"[2/3] Generating {num_frames} CSI frames...")
print(f" - Duration: {params['duration_s']}s at {params['sampling_rate_hz']} Hz")
frames = generate_csi_frames(params)
print(f" - Generated {len(frames)} frames")
print()
# Step 3: Save output
print("[3/3] Saving output files...")
save_data(frames, params, output_dir)
print()
print("Done. Reference signal generated successfully.")
print("=" * 70)
if __name__ == "__main__":
main()
archive/v1/data/proof/sample_csi_data.json
+356014
View File
File diff suppressed because it is too large Load Diff
archive/v1/data/proof/sample_csi_meta.json
+85
View File
@@ -0,0 +1,85 @@
{
"description": "Metadata for the SYNTHETIC deterministic CSI reference signal. Documents all generation parameters so the signal can be independently reproduced and verified.",
"is_synthetic": true,
"is_real_capture": false,
"generator_script": "generate_reference_signal.py",
"numpy_seed": 42,
"system_parameters": {
"num_antennas": 3,
"num_subcarriers": 56,
"sampling_rate_hz": 100,
"duration_s": 10.0,
"center_frequency_hz": 5210000000.0,
"subcarrier_spacing_hz": 312500.0,
"total_frames": 1000
},
"multipath_channel": {
"num_paths": 5,
"path_delays_ns": [
0.0,
15.0,
42.0,
78.0,
120.0
],
"path_amplitudes": [
1.0,
0.6,
0.35,
0.18,
0.08
],
"path_phase_offsets_rad": [
[
-0.788287681898749,
2.8319215077704234,
1.4576609265440963
],
[
0.6198895383354297,
-2.1612986243157413,
-2.1614501754128375
],
[
-2.776642555026645,
2.3007525789727232,
0.6353243561202211
],
[
1.3073585636350948,
-3.012256461474685,
2.952530678803174
],
[
2.088798716157191,
-1.8074266732364683,
-1.9991526911557285
]
],
"description": "5-path indoor multipath model with deterministic delays and amplitudes. Path amplitudes decrease with delay (typical indoor)."
},
"human_motion_signals": {
"breathing": {
"frequency_hz": 0.3,
"modulation_depth": 0.02,
"per_antenna_phase_offsets_rad": [
1.152364521581569,
1.9116103907867292,
3.297141901079666
],
"description": "Sinusoidal amplitude modulation at 0.3 Hz modeling human breathing (typical adult resting rate: 12-20 breaths/min = 0.2-0.33 Hz)."
},
"walking": {
"frequency_hz": 1.2,
"modulation_depth": 0.08,
"per_antenna_phase_offsets_rad": [
2.713990594641554,
1.8298466547148808,
3.844385118274953
],
"description": "Sinusoidal amplitude modulation at 1.2 Hz modeling human walking motion (typical stride rate: ~1.0-1.4 Hz)."
}
},
"generation_formula": "CSI[a,k,t] = sum_p { A_p * exp(j*(2*pi*f_k*tau_p + phi_{p,a})) * (1 + d_breathe * sin(2*pi*0.3*t + psi_breathe_a)) * (1 + d_walk * sin(2*pi*1.2*t + psi_walk_a)) }",
"determinism_guarantee": "All parameters are derived from numpy.random.RandomState(42) at script initialization. The generation loop itself uses NO randomness. Running this script on any platform with the same numpy version will produce bit-identical output."
}
archive/v1/data/proof/verify.py
+533
View File
@@ -0,0 +1,533 @@
#!/usr/bin/env python3
"""
Proof-of-Reality Verification Script for WiFi-DensePose Pipeline.
TRUST KILL SWITCH: A one-command proof replay that makes "it is mocked"
a falsifiable, measurable claim that fails against evidence.
This script verifies that the signal processing pipeline produces
DETERMINISTIC, REPRODUCIBLE output from a known reference signal.
Steps:
1. Load the published reference CSI signal from sample_csi_data.json
2. Feed each frame through the ACTUAL CSI processor feature extraction
3. Collect all feature outputs into a canonical byte representation
4. Compute SHA-256 hash of the full feature output
5. Compare against the published expected hash in expected_features.sha256
6. Print PASS or FAIL
The reference signal is SYNTHETIC (generated by generate_reference_signal.py)
and is used purely for pipeline determinism verification. The point is not
that the signal is real -- the point is that the PIPELINE CODE is real.
The same code that processes this reference also processes live captures.
If someone claims "it is mocked":
1. Run: ./verify
2. If PASS: the pipeline code is the same code that produced the published hash
3. If FAIL: something changed -- investigate
Usage:
python verify.py # Run verification against stored hash
python verify.py --verbose # Show detailed feature statistics
python verify.py --audit # Scan codebase for mock/random patterns
python verify.py --generate-hash # Generate and print the expected hash
"""
import hashlib
import inspect
import json
import os
import struct
import sys
import argparse
import time
from datetime import datetime, timezone
import numpy as np
# Add the v1 directory to sys.path so we can import the actual modules
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
V1_DIR = os.path.abspath(os.path.join(SCRIPT_DIR, "..", "..")) # v1/data/proof -> v1/
if V1_DIR not in sys.path:
sys.path.insert(0, V1_DIR)
# Import the actual pipeline modules -- these are the PRODUCTION modules,
# not test doubles. The source paths are printed below for verification.
from src.hardware.csi_extractor import CSIData
from src.core.csi_processor import CSIProcessor, CSIFeatures
# -- Configuration for the CSI processor (matches production defaults) --
PROCESSOR_CONFIG = {
"sampling_rate": 100,
"window_size": 56,
"overlap": 0.5,
"noise_threshold": -60,
"human_detection_threshold": 0.8,
"smoothing_factor": 0.9,
"max_history_size": 500,
"enable_preprocessing": True,
"enable_feature_extraction": True,
"enable_human_detection": True,
}
# Number of frames to process for the feature hash.
# We process a representative subset to keep verification fast while
# still covering temporal dynamics (Doppler requires history).
VERIFICATION_FRAME_COUNT = 100 # First 100 frames = 1 second
def print_banner():
"""Print the verification banner."""
print("=" * 72)
print(" WiFi-DensePose: Trust Kill Switch -- Pipeline Proof Replay")
print("=" * 72)
print()
print(' "If the public demo is a one-command replay that produces a matching')
print(' hash from a published real capture, \'it is mocked\' becomes a')
print(' measurable claim that fails."')
print()
def print_source_provenance():
"""Print the actual source file paths used by this verification.
This lets anyone confirm that the imported modules are the production
code, not test doubles or mocks.
"""
csi_processor_file = inspect.getfile(CSIProcessor)
csi_data_file = inspect.getfile(CSIData)
csi_features_file = inspect.getfile(CSIFeatures)
print(" SOURCE PROVENANCE (verify these are production modules):")
print(f" CSIProcessor : {os.path.abspath(csi_processor_file)}")
print(f" CSIData : {os.path.abspath(csi_data_file)}")
print(f" CSIFeatures : {os.path.abspath(csi_features_file)}")
print(f" numpy : {np.__file__}")
print(f" numpy version: {np.__version__}")
try:
import scipy
print(f" scipy : {scipy.__file__}")
print(f" scipy version: {scipy.__version__}")
except ImportError:
print(" scipy : NOT AVAILABLE")
print()
def load_reference_signal(data_path):
"""Load the reference CSI signal from JSON.
Args:
data_path: Path to sample_csi_data.json.
Returns:
dict: Parsed JSON data.
Raises:
FileNotFoundError: If the data file doesn't exist.
json.JSONDecodeError: If the data is malformed.
"""
with open(data_path, "r") as f:
data = json.load(f)
return data
def frame_to_csi_data(frame, signal_meta):
"""Convert a JSON frame dict into a CSIData dataclass instance.
Args:
frame: Dict with 'amplitude', 'phase', 'timestamp_s', 'frame_index'.
signal_meta: Top-level signal metadata (num_antennas, frequency, etc).
Returns:
CSIData instance.
"""
amplitude = np.array(frame["amplitude"], dtype=np.float64)
phase = np.array(frame["phase"], dtype=np.float64)
timestamp = datetime.fromtimestamp(frame["timestamp_s"], tz=timezone.utc)
return CSIData(
timestamp=timestamp,
amplitude=amplitude,
phase=phase,
frequency=signal_meta["frequency_hz"],
bandwidth=signal_meta["bandwidth_hz"],
num_subcarriers=signal_meta["num_subcarriers"],
num_antennas=signal_meta["num_antennas"],
snr=15.0, # Fixed SNR for synthetic signal
metadata={
"source": "synthetic_reference",
"frame_index": frame["frame_index"],
},
)
def features_to_bytes(features):
"""Convert CSIFeatures to a deterministic byte representation.
We serialize each numpy array to bytes in a canonical order
using little-endian float64 representation. This ensures the
hash is platform-independent for IEEE 754 compliant systems.
Args:
features: CSIFeatures instance.
Returns:
bytes: Canonical byte representation.
"""
parts = []
# Serialize each feature array in declaration order
for array in [
features.amplitude_mean,
features.amplitude_variance,
features.phase_difference,
features.correlation_matrix,
features.doppler_shift,
features.power_spectral_density,
]:
flat = np.asarray(array, dtype=np.float64).ravel()
# Pack as little-endian double (8 bytes each)
parts.append(struct.pack(f"<{len(flat)}d", *flat))
return b"".join(parts)
def compute_pipeline_hash(data_path, verbose=False):
"""Run the full pipeline and compute the SHA-256 hash of all features.
Args:
data_path: Path to sample_csi_data.json.
verbose: If True, print detailed feature statistics.
Returns:
tuple: (hex_hash, stats_dict) where stats_dict contains metrics.
"""
# Load reference signal
signal_data = load_reference_signal(data_path)
frames = signal_data["frames"][:VERIFICATION_FRAME_COUNT]
print(f" Reference signal: {os.path.basename(data_path)}")
print(f" Signal description: {signal_data.get('description', 'N/A')}")
print(f" Generator: {signal_data.get('generator', 'N/A')} v{signal_data.get('generator_version', '?')}")
print(f" Numpy seed used: {signal_data.get('numpy_seed', 'N/A')}")
print(f" Total frames in file: {signal_data.get('num_frames', len(signal_data['frames']))}")
print(f" Frames to process: {len(frames)}")
print(f" Subcarriers: {signal_data.get('num_subcarriers', 'N/A')}")
print(f" Antennas: {signal_data.get('num_antennas', 'N/A')}")
print(f" Frequency: {signal_data.get('frequency_hz', 0) / 1e9:.3f} GHz")
print(f" Bandwidth: {signal_data.get('bandwidth_hz', 0) / 1e6:.1f} MHz")
print(f" Sampling rate: {signal_data.get('sampling_rate_hz', 'N/A')} Hz")
print()
# Create processor with production config
print(" Configuring CSIProcessor with production parameters...")
processor = CSIProcessor(PROCESSOR_CONFIG)
print(f" Window size: {processor.window_size}")
print(f" Overlap: {processor.overlap}")
print(f" Noise threshold: {processor.noise_threshold} dB")
print(f" Preprocessing: {'ENABLED' if processor.enable_preprocessing else 'DISABLED'}")
print(f" Feature extraction: {'ENABLED' if processor.enable_feature_extraction else 'DISABLED'}")
print()
# Process all frames and accumulate feature bytes
hasher = hashlib.sha256()
features_count = 0
total_feature_bytes = 0
last_features = None
doppler_nonzero_count = 0
doppler_shape = None
psd_shape = None
t_start = time.perf_counter()
for i, frame in enumerate(frames):
csi_data = frame_to_csi_data(frame, signal_data)
# Run through the actual pipeline: preprocess -> extract features
preprocessed = processor.preprocess_csi_data(csi_data)
features = processor.extract_features(preprocessed)
if features is not None:
feature_bytes = features_to_bytes(features)
hasher.update(feature_bytes)
features_count += 1
total_feature_bytes += len(feature_bytes)
last_features = features
# Track Doppler statistics
doppler_shape = features.doppler_shift.shape
doppler_nonzero_count = int(np.count_nonzero(features.doppler_shift))
psd_shape = features.power_spectral_density.shape
# Add to history for Doppler computation in subsequent frames
processor.add_to_history(csi_data)
if verbose and (i + 1) % 25 == 0:
print(f" ... processed frame {i + 1}/{len(frames)}")
t_elapsed = time.perf_counter() - t_start
print(f" Processing complete.")
print(f" Frames processed: {len(frames)}")
print(f" Feature vectors extracted: {features_count}")
print(f" Total feature bytes hashed: {total_feature_bytes:,}")
print(f" Processing time: {t_elapsed:.4f}s ({len(frames) / t_elapsed:.0f} frames/sec)")
print()
# Print feature vector details
if last_features is not None:
print(" FEATURE VECTOR DETAILS (from last frame):")
print(f" amplitude_mean : shape={last_features.amplitude_mean.shape}, "
f"min={np.min(last_features.amplitude_mean):.6f}, "
f"max={np.max(last_features.amplitude_mean):.6f}, "
f"mean={np.mean(last_features.amplitude_mean):.6f}")
print(f" amplitude_variance : shape={last_features.amplitude_variance.shape}, "
f"min={np.min(last_features.amplitude_variance):.6f}, "
f"max={np.max(last_features.amplitude_variance):.6f}")
print(f" phase_difference : shape={last_features.phase_difference.shape}, "
f"mean={np.mean(last_features.phase_difference):.6f}")
print(f" correlation_matrix : shape={last_features.correlation_matrix.shape}")
print(f" doppler_shift : shape={doppler_shape}, "
f"non-zero bins={doppler_nonzero_count}/{doppler_shape[0] if doppler_shape else 0}")
print(f" power_spectral_density: shape={psd_shape}")
print()
if verbose:
print(" DOPPLER SPECTRUM (proves real FFT, not random):")
ds = last_features.doppler_shift
print(f" First 8 bins: {ds[:8]}")
print(f" Sum: {np.sum(ds):.6f}")
print(f" Max bin index: {np.argmax(ds)}")
print(f" Spectral entropy: {-np.sum(ds[ds > 0] * np.log2(ds[ds > 0] + 1e-15)):.4f}")
print()
print(" PSD DETAILS (proves scipy.fft, not random):")
psd = last_features.power_spectral_density
print(f" First 8 bins: {psd[:8]}")
print(f" Total power: {np.sum(psd):.4f}")
print(f" Peak frequency bin: {np.argmax(psd)}")
print()
stats = {
"frames_processed": len(frames),
"features_extracted": features_count,
"total_bytes_hashed": total_feature_bytes,
"elapsed_seconds": t_elapsed,
"doppler_shape": doppler_shape,
"doppler_nonzero": doppler_nonzero_count,
"psd_shape": psd_shape,
}
return hasher.hexdigest(), stats
def audit_codebase(base_dir=None):
"""Scan the production codebase for mock/random patterns.
Looks for:
- np.random.rand / np.random.randn calls (outside testing/)
- mock/Mock imports (outside testing/)
- random.random() calls (outside testing/)
Args:
base_dir: Root directory to scan. Defaults to v1/src/.
Returns:
list of (filepath, line_number, line_text, pattern_type) tuples.
"""
if base_dir is None:
base_dir = os.path.join(V1_DIR, "src")
suspicious_patterns = [
("np.random.rand", "RANDOM_GENERATOR"),
("np.random.randn", "RANDOM_GENERATOR"),
("np.random.random", "RANDOM_GENERATOR"),
("np.random.uniform", "RANDOM_GENERATOR"),
("np.random.normal", "RANDOM_GENERATOR"),
("np.random.choice", "RANDOM_GENERATOR"),
("random.random(", "RANDOM_GENERATOR"),
("random.randint(", "RANDOM_GENERATOR"),
("from unittest.mock import", "MOCK_IMPORT"),
("from unittest import mock", "MOCK_IMPORT"),
("import mock", "MOCK_IMPORT"),
("MagicMock", "MOCK_USAGE"),
("@patch(", "MOCK_USAGE"),
("@mock.patch", "MOCK_USAGE"),
]
# Directories to exclude from the audit
excluded_dirs = {"testing", "tests", "test", "__pycache__", ".git"}
findings = []
for root, dirs, files in os.walk(base_dir):
# Skip excluded directories
dirs[:] = [d for d in dirs if d not in excluded_dirs]
for fname in files:
if not fname.endswith(".py"):
continue
fpath = os.path.join(root, fname)
try:
with open(fpath, "r", encoding="utf-8", errors="replace") as f:
for line_num, line in enumerate(f, 1):
for pattern, ptype in suspicious_patterns:
if pattern in line:
findings.append((fpath, line_num, line.rstrip(), ptype))
except (IOError, OSError):
pass
return findings
def main():
"""Main verification entry point."""
parser = argparse.ArgumentParser(
description="WiFi-DensePose Trust Kill Switch -- Pipeline Proof Replay"
)
parser.add_argument(
"--generate-hash",
action="store_true",
help="Generate and print the expected hash (do not verify)",
)
parser.add_argument(
"--verbose",
action="store_true",
help="Show detailed feature statistics and Doppler spectrum",
)
parser.add_argument(
"--audit",
action="store_true",
help="Scan production codebase for mock/random patterns",
)
args = parser.parse_args()
print_banner()
# Locate data file
data_path = os.path.join(SCRIPT_DIR, "sample_csi_data.json")
hash_path = os.path.join(SCRIPT_DIR, "expected_features.sha256")
# ---------------------------------------------------------------
# Step 0: Print source provenance
# ---------------------------------------------------------------
print("[0/4] SOURCE PROVENANCE")
print_source_provenance()
# ---------------------------------------------------------------
# Step 1: Load and describe reference signal
# ---------------------------------------------------------------
print("[1/4] LOADING REFERENCE SIGNAL")
if not os.path.exists(data_path):
print(f" FAIL: Reference data not found at {data_path}")
print(" Run generate_reference_signal.py first.")
sys.exit(1)
print(f" Path: {data_path}")
print(f" Size: {os.path.getsize(data_path):,} bytes")
print()
# ---------------------------------------------------------------
# Step 2: Process through the real pipeline
# ---------------------------------------------------------------
print("[2/4] PROCESSING THROUGH PRODUCTION PIPELINE")
print(" This runs the SAME CSIProcessor.preprocess_csi_data() and")
print(" CSIProcessor.extract_features() used in production.")
print()
computed_hash, stats = compute_pipeline_hash(data_path, verbose=args.verbose)
# ---------------------------------------------------------------
# Step 3: Hash comparison
# ---------------------------------------------------------------
print("[3/4] SHA-256 HASH COMPARISON")
print(f" Computed: {computed_hash}")
if args.generate_hash:
with open(hash_path, "w") as f:
f.write(computed_hash + "\n")
print(f" Wrote expected hash to {hash_path}")
print()
print(" HASH GENERATED -- run without --generate-hash to verify.")
print("=" * 72)
return
if not os.path.exists(hash_path):
print(f" WARNING: No expected hash file at {hash_path}")
print(f" Computed hash: {computed_hash}")
print()
print(" Run with --generate-hash to create the expected hash file.")
print()
print(" SKIP (no expected hash to compare against)")
print("=" * 72)
sys.exit(2)
with open(hash_path, "r") as f:
expected_hash = f.read().strip()
print(f" Expected: {expected_hash}")
if computed_hash == expected_hash:
match_status = "MATCH"
else:
match_status = "MISMATCH"
print(f" Status: {match_status}")
print()
# ---------------------------------------------------------------
# Step 4: Audit (if requested or always in full mode)
# ---------------------------------------------------------------
if args.audit:
print("[4/4] CODEBASE AUDIT -- scanning for mock/random patterns")
findings = audit_codebase()
if findings:
print(f" Found {len(findings)} suspicious pattern(s) in production code:")
for fpath, line_num, line, ptype in findings:
relpath = os.path.relpath(fpath, V1_DIR)
print(f" [{ptype}] {relpath}:{line_num}: {line.strip()}")
else:
print(" CLEAN -- no mock/random patterns found in production code.")
print()
else:
print("[4/4] CODEBASE AUDIT (skipped -- use --audit to enable)")
print()
# ---------------------------------------------------------------
# Final verdict
# ---------------------------------------------------------------
print("=" * 72)
if computed_hash == expected_hash:
print(" VERDICT: PASS")
print()
print(" The pipeline produced a SHA-256 hash that matches the published")
print(" expected hash. This proves:")
print(" 1. The SAME signal processing code ran on the reference signal")
print(" 2. The output is DETERMINISTIC (same input -> same output)")
print(" 3. No randomness was introduced (hash would differ)")
print(" 4. The code path includes: noise removal, Hamming windowing,")
print(" amplitude normalization, FFT-based Doppler extraction,")
print(" and power spectral density computation")
print()
print(f" Pipeline hash: {computed_hash}")
print("=" * 72)
sys.exit(0)
else:
print(" VERDICT: FAIL")
print()
print(" The pipeline output does NOT match the expected hash.")
print()
print(" Possible causes:")
print(" - Numpy/scipy version mismatch (check requirements)")
print(" - Code change in CSI processor that alters numerical output")
print(" - Platform floating-point differences (unlikely for IEEE 754)")
print()
print(" To update the expected hash after intentional changes:")
print(" python verify.py --generate-hash")
print("=" * 72)
sys.exit(1)
if __name__ == "__main__":
main()
archive/v1/data/test_wifi_densepose.db
View File
archive/v1/data/wifi_densepose_fallback.db
BIN
View File
Binary file not shown.
archive/v1/docs/api-endpoints-summary.md
+312
View File
@@ -0,0 +1,312 @@
# WiFi-DensePose API Endpoints Summary
## Overview
The WiFi-DensePose API provides RESTful endpoints and WebSocket connections for real-time human pose estimation using WiFi CSI (Channel State Information) data. The API is built with FastAPI and supports both synchronous REST operations and real-time streaming via WebSockets.
## Base URL
- **Development**: `http://localhost:8000`
- **API Prefix**: `/api/v1`
- **Documentation**: `http://localhost:8000/docs`
## Authentication
Authentication is configurable via environment variables:
- When `ENABLE_AUTHENTICATION=true`, protected endpoints require JWT tokens
- Tokens can be passed via:
- Authorization header: `Bearer <token>`
- Query parameter: `?token=<token>`
- Cookie: `access_token`
## Rate Limiting
Rate limiting is configurable and when enabled (`ENABLE_RATE_LIMITING=true`):
- Anonymous: 100 requests/hour
- Authenticated: 1000 requests/hour
- Admin: 10000 requests/hour
## Endpoints
### 1. Health & Status
#### GET `/health/health`
System health check with component status and metrics.
**Response Example:**
```json
{
"status": "healthy",
"timestamp": "2025-06-09T16:00:00Z",
"uptime_seconds": 3600.0,
"components": {
"hardware": {...},
"pose": {...},
"stream": {...}
},
"system_metrics": {
"cpu": {"percent": 24.1, "count": 2},
"memory": {"total_gb": 7.75, "available_gb": 3.73},
"disk": {"total_gb": 31.33, "free_gb": 7.09}
}
}
```
#### GET `/health/ready`
Readiness check for load balancers.
#### GET `/health/live`
Simple liveness check.
#### GET `/health/metrics` 🔒
Detailed system metrics (requires auth).
### 2. Pose Estimation
#### GET `/api/v1/pose/current`
Get current pose estimation from WiFi signals.
**Query Parameters:**
- `zone_ids`: List of zone IDs to analyze
- `confidence_threshold`: Minimum confidence (0.0-1.0)
- `max_persons`: Maximum persons to detect
- `include_keypoints`: Include keypoint data (default: true)
- `include_segmentation`: Include DensePose segmentation (default: false)
**Response Example:**
```json
{
"timestamp": "2025-06-09T16:00:00Z",
"frame_id": "frame_123456",
"persons": [
{
"person_id": "0",
"confidence": 0.95,
"bounding_box": {"x": 0.1, "y": 0.2, "width": 0.3, "height": 0.6},
"keypoints": [...],
"zone_id": "zone_1",
"activity": "standing"
}
],
"zone_summary": {"zone_1": 1, "zone_2": 0},
"processing_time_ms": 45.2
}
```
#### POST `/api/v1/pose/analyze` 🔒
Analyze pose data with custom parameters (requires auth).
#### GET `/api/v1/pose/zones/{zone_id}/occupancy`
Get occupancy for a specific zone.
#### GET `/api/v1/pose/zones/summary`
Get occupancy summary for all zones.
#### GET `/api/v1/pose/activities`
Get recently detected activities.
**Query Parameters:**
- `zone_id`: Filter by zone
- `limit`: Maximum results (1-100)
#### POST `/api/v1/pose/historical` 🔒
Query historical pose data (requires auth).
**Request Body:**
```json
{
"start_time": "2025-06-09T15:00:00Z",
"end_time": "2025-06-09T16:00:00Z",
"zone_ids": ["zone_1"],
"aggregation_interval": 300,
"include_raw_data": false
}
```
#### GET `/api/v1/pose/stats`
Get pose estimation statistics.
**Query Parameters:**
- `hours`: Hours of data to analyze (1-168)
### 3. Calibration
#### POST `/api/v1/pose/calibrate` 🔒
Start system calibration (requires auth).
#### GET `/api/v1/pose/calibration/status` 🔒
Get calibration status (requires auth).
### 4. Streaming
#### GET `/api/v1/stream/status`
Get streaming service status.
#### POST `/api/v1/stream/start` 🔒
Start streaming service (requires auth).
#### POST `/api/v1/stream/stop` 🔒
Stop streaming service (requires auth).
#### GET `/api/v1/stream/clients` 🔒
List connected WebSocket clients (requires auth).
#### DELETE `/api/v1/stream/clients/{client_id}` 🔒
Disconnect specific client (requires auth).
#### POST `/api/v1/stream/broadcast` 🔒
Broadcast message to clients (requires auth).
### 5. WebSocket Endpoints
#### WS `/api/v1/stream/pose`
Real-time pose data streaming.
**Query Parameters:**
- `zone_ids`: Comma-separated zone IDs
- `min_confidence`: Minimum confidence (0.0-1.0)
- `max_fps`: Maximum frames per second (1-60)
- `token`: Auth token (if authentication enabled)
**Message Types:**
- `connection_established`: Initial connection confirmation
- `pose_update`: Pose data updates
- `error`: Error messages
- `ping`/`pong`: Keep-alive
#### WS `/api/v1/stream/events`
Real-time event streaming.
**Query Parameters:**
- `event_types`: Comma-separated event types
- `zone_ids`: Comma-separated zone IDs
- `token`: Auth token (if authentication enabled)
### 6. API Information
#### GET `/`
Root endpoint with API information.
#### GET `/api/v1/info`
Detailed API configuration.
#### GET `/api/v1/status`
Current API and service status.
#### GET `/api/v1/metrics`
API performance metrics (if enabled).
### 7. Development Endpoints
These endpoints are only available when `ENABLE_TEST_ENDPOINTS=true`:
#### GET `/api/v1/dev/config`
Get current configuration (development only).
#### POST `/api/v1/dev/reset`
Reset services (development only).
## Error Handling
All errors follow a consistent format:
```json
{
"error": {
"code": 400,
"message": "Error description",
"type": "error_type"
}
}
```
Error types:
- `http_error`: HTTP-related errors
- `validation_error`: Request validation errors
- `authentication_error`: Authentication failures
- `rate_limit_exceeded`: Rate limit violations
- `internal_error`: Server errors
## WebSocket Protocol
### Connection Flow
1. **Connect**: `ws://host/api/v1/stream/pose?params`
2. **Receive**: Connection confirmation message
3. **Send/Receive**: Bidirectional communication
4. **Disconnect**: Clean connection closure
### Message Format
All WebSocket messages use JSON format:
```json
{
"type": "message_type",
"timestamp": "ISO-8601 timestamp",
"data": {...}
}
```
### Client Messages
- `{"type": "ping"}`: Keep-alive ping
- `{"type": "update_config", "config": {...}}`: Update stream config
- `{"type": "get_status"}`: Request status
- `{"type": "disconnect"}`: Clean disconnect
### Server Messages
- `{"type": "connection_established", ...}`: Connection confirmed
- `{"type": "pose_update", ...}`: Pose data update
- `{"type": "event", ...}`: Event notification
- `{"type": "pong"}`: Ping response
- `{"type": "error", "message": "..."}`: Error message
## CORS Configuration
CORS is enabled with configurable origins:
- Development: Allow all origins (`*`)
- Production: Restrict to specific domains
## Security Headers
The API includes security headers:
- `X-Content-Type-Options: nosniff`
- `X-Frame-Options: DENY`
- `X-XSS-Protection: 1; mode=block`
- `Referrer-Policy: strict-origin-when-cross-origin`
- `Content-Security-Policy: ...`
## Performance Considerations
1. **Batch Requests**: Use zone summaries instead of individual zone queries
2. **WebSocket Streaming**: Adjust `max_fps` to reduce bandwidth
3. **Historical Data**: Use appropriate `aggregation_interval`
4. **Caching**: Results are cached when Redis is enabled
## Testing
Use the provided test scripts:
- `scripts/test_api_endpoints.py`: Comprehensive endpoint testing
- `scripts/test_websocket_streaming.py`: WebSocket functionality testing
## Production Deployment
For production:
1. Set `ENVIRONMENT=production`
2. Enable authentication and rate limiting
3. Configure proper database (PostgreSQL)
4. Enable Redis for caching
5. Use HTTPS with valid certificates
6. Restrict CORS origins
7. Disable debug mode and test endpoints
8. Configure monitoring and logging
## API Versioning
The API uses URL versioning:
- Current version: `v1`
- Base path: `/api/v1`
Future versions will be available at `/api/v2`, etc.
archive/v1/docs/api-test-results.md
+309
View File
@@ -0,0 +1,309 @@
# WiFi-DensePose API Test Results
## Test Summary
**Date**: June 9, 2025
**Environment**: Development
**Server**: http://localhost:8000
**Total Tests**: 26
**Passed**: 18
**Failed**: 8
**Success Rate**: 69.2%
## Test Configuration
### Environment Settings
- **Authentication**: Disabled
- **Rate Limiting**: Disabled
- **Mock Hardware**: Enabled
- **Mock Pose Data**: Enabled
- **WebSockets**: Enabled
- **Real-time Processing**: Enabled
### Key Configuration Parameters
```env
ENVIRONMENT=development
DEBUG=true
ENABLE_AUTHENTICATION=false
ENABLE_RATE_LIMITING=false
MOCK_HARDWARE=true
MOCK_POSE_DATA=true
ENABLE_WEBSOCKETS=true
ENABLE_REAL_TIME_PROCESSING=true
```
## Endpoint Test Results
### 1. Health Check Endpoints ✅
#### `/health/health` - System Health Check
- **Status**: ✅ PASSED
- **Response Time**: ~1015ms
- **Response**: Complete system health including hardware, pose, and stream services
- **Notes**: Shows CPU, memory, disk, and network metrics
#### `/health/ready` - Readiness Check
- **Status**: ✅ PASSED
- **Response Time**: ~1.6ms
- **Response**: System readiness status with individual service checks
### 2. Pose Detection Endpoints 🔧
#### `/api/v1/pose/current` - Current Pose Estimation
- **Status**: ✅ PASSED
- **Response Time**: ~1.2ms
- **Response**: Current pose data with mock poses
- **Notes**: Working with mock data in development mode
#### `/api/v1/pose/zones/{zone_id}/occupancy` - Zone Occupancy
- **Status**: ✅ PASSED
- **Response Time**: ~1.2ms
- **Response**: Zone-specific occupancy data
#### `/api/v1/pose/zones/summary` - All Zones Summary
- **Status**: ✅ PASSED
- **Response Time**: ~1.2ms
- **Response**: Summary of all zones with total persons count
#### `/api/v1/pose/activities` - Recent Activities
- **Status**: ✅ PASSED
- **Response Time**: ~1.4ms
- **Response**: List of recently detected activities
#### `/api/v1/pose/stats` - Pose Statistics
- **Status**: ✅ PASSED
- **Response Time**: ~1.1ms
- **Response**: Statistical data for specified time period
### 3. Protected Endpoints (Authentication Required) 🔒
These endpoints require authentication, which is disabled in development:
#### `/api/v1/pose/analyze` - Pose Analysis
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
#### `/api/v1/pose/historical` - Historical Data
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
#### `/api/v1/pose/calibrate` - Start Calibration
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
#### `/api/v1/pose/calibration/status` - Calibration Status
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
### 4. Streaming Endpoints 📡
#### `/api/v1/stream/status` - Stream Status
- **Status**: ✅ PASSED
- **Response Time**: ~1.0ms
- **Response**: Current streaming status and connected clients
#### `/api/v1/stream/start` - Start Streaming
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
#### `/api/v1/stream/stop` - Stop Streaming
- **Status**: ❌ FAILED (401 Unauthorized)
- **Note**: Requires authentication token
### 5. WebSocket Endpoints 🌐
#### `/api/v1/stream/pose` - Pose WebSocket
- **Status**: ✅ PASSED
- **Connection Time**: ~15.1ms
- **Features**: Real-time pose data streaming
- **Parameters**: zone_ids, min_confidence, max_fps, token (optional)
#### `/api/v1/stream/events` - Events WebSocket
- **Status**: ✅ PASSED
- **Connection Time**: ~2.9ms
- **Features**: Real-time event streaming
- **Parameters**: event_types, zone_ids, token (optional)
### 6. Documentation Endpoints 📚
#### `/docs` - API Documentation
- **Status**: ✅ PASSED
- **Response Time**: ~1.0ms
- **Features**: Interactive Swagger UI documentation
#### `/openapi.json` - OpenAPI Schema
- **Status**: ✅ PASSED
- **Response Time**: ~14.6ms
- **Features**: Complete OpenAPI 3.0 specification
### 7. API Information Endpoints ℹ️
#### `/` - Root Endpoint
- **Status**: ✅ PASSED
- **Response Time**: ~0.9ms
- **Response**: API name, version, environment, and feature flags
#### `/api/v1/info` - API Information
- **Status**: ✅ PASSED
- **Response Time**: ~0.8ms
- **Response**: Detailed API configuration and limits
#### `/api/v1/status` - API Status
- **Status**: ✅ PASSED
- **Response Time**: ~1.0ms
- **Response**: Current API and service statuses
### 8. Error Handling ⚠️
#### `/nonexistent` - 404 Error
- **Status**: ✅ PASSED
- **Response Time**: ~1.4ms
- **Response**: Proper 404 error with formatted error response
## Authentication Status
Authentication is currently **DISABLED** in development mode. The following endpoints require authentication when enabled:
1. **POST** `/api/v1/pose/analyze` - Analyze pose data with custom parameters
2. **POST** `/api/v1/pose/historical` - Query historical pose data
3. **POST** `/api/v1/pose/calibrate` - Start system calibration
4. **GET** `/api/v1/pose/calibration/status` - Get calibration status
5. **POST** `/api/v1/stream/start` - Start streaming service
6. **POST** `/api/v1/stream/stop` - Stop streaming service
7. **GET** `/api/v1/stream/clients` - List connected clients
8. **DELETE** `/api/v1/stream/clients/{client_id}` - Disconnect specific client
9. **POST** `/api/v1/stream/broadcast` - Broadcast message to clients
## Rate Limiting Status
Rate limiting is currently **DISABLED** in development mode. When enabled:
- Anonymous users: 100 requests/hour
- Authenticated users: 1000 requests/hour
- Admin users: 10000 requests/hour
Path-specific limits:
- `/api/v1/pose/current`: 60 requests/minute
- `/api/v1/pose/analyze`: 10 requests/minute
- `/api/v1/pose/calibrate`: 1 request/5 minutes
- `/api/v1/stream/start`: 5 requests/minute
- `/api/v1/stream/stop`: 5 requests/minute
## Error Response Format
All error responses follow a consistent format:
```json
{
"error": {
"code": 404,
"message": "Endpoint not found",
"type": "http_error"
}
}
```
Validation errors include additional details:
```json
{
"error": {
"code": 422,
"message": "Validation error",
"type": "validation_error",
"details": [...]
}
}
```
## WebSocket Message Format
### Connection Establishment
```json
{
"type": "connection_established",
"client_id": "unique-client-id",
"timestamp": "2025-06-09T16:00:00.000Z",
"config": {
"zone_ids": ["zone_1"],
"min_confidence": 0.5,
"max_fps": 30
}
}
```
### Pose Data Stream
```json
{
"type": "pose_update",
"timestamp": "2025-06-09T16:00:00.000Z",
"frame_id": "frame-123",
"persons": [...],
"zone_summary": {...}
}
```
### Error Messages
```json
{
"type": "error",
"message": "Error description"
}
```
## Performance Metrics
- **Average Response Time**: ~2.5ms (excluding health check)
- **Health Check Time**: ~1015ms (includes system metrics collection)
- **WebSocket Connection Time**: ~9ms average
- **OpenAPI Schema Generation**: ~14.6ms
## Known Issues
1. **CSI Processing**: Initial implementation had method name mismatch (`add_data` vs `add_to_history`)
2. **Phase Sanitizer**: Required configuration parameters were missing
3. **Stream Service**: Missing `shutdown` method implementation
4. **WebSocket Paths**: Documentation showed incorrect paths (`/ws/pose` instead of `/api/v1/stream/pose`)
## Recommendations
### For Development
1. Keep authentication and rate limiting disabled for easier testing
2. Use mock data for hardware and pose estimation
3. Enable all documentation endpoints
4. Use verbose logging for debugging
### For Production
1. **Enable Authentication**: Set `ENABLE_AUTHENTICATION=true`
2. **Enable Rate Limiting**: Set `ENABLE_RATE_LIMITING=true`
3. **Disable Mock Data**: Set `MOCK_HARDWARE=false` and `MOCK_POSE_DATA=false`
4. **Secure Endpoints**: Disable documentation endpoints in production
5. **Configure CORS**: Restrict `CORS_ORIGINS` to specific domains
6. **Set Secret Key**: Use a strong, unique `SECRET_KEY`
7. **Database**: Use PostgreSQL instead of SQLite
8. **Redis**: Enable Redis for caching and rate limiting
9. **HTTPS**: Use HTTPS in production with proper certificates
10. **Monitoring**: Enable metrics and health monitoring
## Test Script Usage
To run the API tests:
```bash
python scripts/test_api_endpoints.py
```
Test results are saved to: `scripts/api_test_results_[timestamp].json`
## Conclusion
The WiFi-DensePose API is functioning correctly in development mode with:
- ✅ All public endpoints working
- ✅ WebSocket connections established successfully
- ✅ Proper error handling and response formats
- ✅ Mock data generation for testing
- ❌ Protected endpoints correctly requiring authentication (when enabled)
The system is ready for development and testing. For production deployment, follow the recommendations above to enable security features and use real hardware/model implementations.
archive/v1/docs/api/rest-endpoints.md
+992
View File
@@ -0,0 +1,992 @@
# REST API Endpoints
## Overview
The WiFi-DensePose REST API provides comprehensive access to pose estimation data, system configuration, and analytics. This document details all available endpoints, request/response formats, authentication requirements, and usage examples.
## Table of Contents
1. [API Overview](#api-overview)
2. [Authentication](#authentication)
3. [Common Response Formats](#common-response-formats)
4. [Error Handling](#error-handling)
5. [Pose Estimation Endpoints](#pose-estimation-endpoints)
6. [System Management Endpoints](#system-management-endpoints)
7. [Configuration Endpoints](#configuration-endpoints)
8. [Analytics Endpoints](#analytics-endpoints)
9. [Health and Status Endpoints](#health-and-status-endpoints)
10. [Rate Limiting](#rate-limiting)
## API Overview
### Base URL
```
Production: https://api.wifi-densepose.com/api/v1
Staging: https://staging-api.wifi-densepose.com/api/v1
Development: http://localhost:8000/api/v1
```
### API Versioning
The API uses URL path versioning. The current version is `v1`. Future versions will be available at `/api/v2`, etc.
### Content Types
- **Request Content-Type**: `application/json`
- **Response Content-Type**: `application/json`
- **File Upload**: `multipart/form-data`
### HTTP Methods
- **GET**: Retrieve data
- **POST**: Create new resources
- **PUT**: Update existing resources (full replacement)
- **PATCH**: Partial updates
- **DELETE**: Remove resources
## Authentication
### JWT Token Authentication
Most endpoints require JWT token authentication. Include the token in the Authorization header:
```http
Authorization: Bearer <jwt_token>
```
### API Key Authentication
For service-to-service communication, use API key authentication:
```http
X-API-Key: <api_key>
```
### Getting an Access Token
```http
POST /api/v1/auth/token
Content-Type: application/json
{
"username": "your_username",
"password": "your_password"
}
```
**Response:**
```json
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"token_type": "bearer",
"expires_in": 86400,
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
```
## Common Response Formats
### Success Response
```json
{
"success": true,
"data": {
// Response data
},
"timestamp": "2025-01-07T10:30:00Z",
"request_id": "req_123456789"
}
```
### Error Response
```json
{
"success": false,
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid request parameters",
"details": {
"field": "confidence_threshold",
"reason": "Value must be between 0 and 1"
}
},
"timestamp": "2025-01-07T10:30:00Z",
"request_id": "req_123456789"
}
```
### Pagination
```json
{
"success": true,
"data": [
// Array of items
],
"pagination": {
"page": 1,
"per_page": 50,
"total": 1250,
"total_pages": 25,
"has_next": true,
"has_prev": false
}
}
```
## Error Handling
### HTTP Status Codes
- **200 OK**: Request successful
- **201 Created**: Resource created successfully
- **400 Bad Request**: Invalid request parameters
- **401 Unauthorized**: Authentication required
- **403 Forbidden**: Insufficient permissions
- **404 Not Found**: Resource not found
- **422 Unprocessable Entity**: Validation error
- **429 Too Many Requests**: Rate limit exceeded
- **500 Internal Server Error**: Server error
### Error Codes
| Code | Description |
|------|-------------|
| `VALIDATION_ERROR` | Request validation failed |
| `AUTHENTICATION_ERROR` | Authentication failed |
| `AUTHORIZATION_ERROR` | Insufficient permissions |
| `RESOURCE_NOT_FOUND` | Requested resource not found |
| `RATE_LIMIT_EXCEEDED` | Too many requests |
| `SYSTEM_ERROR` | Internal system error |
| `HARDWARE_ERROR` | Hardware communication error |
| `MODEL_ERROR` | Neural network model error |
## Pose Estimation Endpoints
### Get Latest Pose Data
Retrieve the most recent pose estimation results.
```http
GET /api/v1/pose/latest
Authorization: Bearer <token>
```
**Query Parameters:**
- `environment_id` (optional): Filter by environment ID
- `min_confidence` (optional): Minimum confidence threshold (0.0-1.0)
- `include_keypoints` (optional): Include detailed keypoint data (default: true)
**Response:**
```json
{
"success": true,
"data": {
"timestamp": "2025-01-07T10:30:00.123Z",
"frame_id": 12345,
"environment_id": "room_001",
"processing_time_ms": 45.2,
"persons": [
{
"person_id": 1,
"track_id": 7,
"confidence": 0.87,
"bounding_box": {
"x": 120,
"y": 80,
"width": 180,
"height": 320
},
"keypoints": [
{
"name": "nose",
"x": 210,
"y": 95,
"confidence": 0.92,
"visible": true
},
{
"name": "left_eye",
"x": 205,
"y": 90,
"confidence": 0.89,
"visible": true
}
// ... additional keypoints
],
"dense_pose": {
"iuv_image": "base64_encoded_image_data",
"confidence_map": "base64_encoded_confidence_data"
}
}
],
"metadata": {
"model_version": "v1.2.0",
"processing_mode": "real_time",
"csi_quality": 0.85
}
}
}
```
### Get Historical Pose Data
Retrieve pose estimation data for a specific time range.
```http
GET /api/v1/pose/history
Authorization: Bearer <token>
```
**Query Parameters:**
- `start_time` (required): Start timestamp (ISO 8601)
- `end_time` (required): End timestamp (ISO 8601)
- `environment_id` (optional): Filter by environment ID
- `person_id` (optional): Filter by person ID
- `track_id` (optional): Filter by track ID
- `min_confidence` (optional): Minimum confidence threshold
- `page` (optional): Page number (default: 1)
- `per_page` (optional): Items per page (default: 50, max: 1000)
**Response:**
```json
{
"success": true,
"data": [
{
"timestamp": "2025-01-07T10:30:00.123Z",
"frame_id": 12345,
"person_id": 1,
"track_id": 7,
"confidence": 0.87,
"bounding_box": {
"x": 120,
"y": 80,
"width": 180,
"height": 320
},
"keypoints": [
// Keypoint data
]
}
// ... additional pose data
],
"pagination": {
"page": 1,
"per_page": 50,
"total": 1250,
"total_pages": 25,
"has_next": true,
"has_prev": false
}
}
```
### Get Person Tracking Data
Retrieve tracking information for a specific person or track.
```http
GET /api/v1/pose/tracking/{track_id}
Authorization: Bearer <token>
```
**Path Parameters:**
- `track_id` (required): Track identifier
**Query Parameters:**
- `start_time` (optional): Start timestamp
- `end_time` (optional): End timestamp
- `include_trajectory` (optional): Include movement trajectory (default: false)
**Response:**
```json
{
"success": true,
"data": {
"track_id": 7,
"person_id": 1,
"first_seen": "2025-01-07T10:25:00Z",
"last_seen": "2025-01-07T10:35:00Z",
"duration_seconds": 600,
"total_frames": 18000,
"average_confidence": 0.84,
"status": "active",
"trajectory": [
{
"timestamp": "2025-01-07T10:25:00Z",
"center_x": 210,
"center_y": 240,
"confidence": 0.87
}
// ... trajectory points
],
"statistics": {
"movement_distance": 15.7,
"average_speed": 0.026,
"time_stationary": 420,
"time_moving": 180
}
}
}
```
### Submit CSI Data for Processing
Submit raw CSI data for pose estimation processing.
```http
POST /api/v1/pose/process
Authorization: Bearer <token>
Content-Type: application/json
{
"csi_data": {
"timestamp": "2025-01-07T10:30:00.123Z",
"antenna_data": [
[
{"real": 1.23, "imag": -0.45},
{"real": 0.87, "imag": 1.12}
// ... subcarrier data
]
// ... antenna data
],
"metadata": {
"router_id": "router_001",
"sampling_rate": 30,
"signal_strength": -45
}
},
"processing_options": {
"confidence_threshold": 0.5,
"max_persons": 10,
"enable_tracking": true,
"return_dense_pose": false
}
}
```
**Response:**
```json
{
"success": true,
"data": {
"processing_id": "proc_123456",
"status": "completed",
"processing_time_ms": 67.3,
"poses": [
// Pose estimation results
]
}
}
```
## System Management Endpoints
### Start System
Start the pose estimation system with specified configuration.
```http
POST /api/v1/system/start
Authorization: Bearer <token>
Content-Type: application/json
{
"configuration": {
"domain": "healthcare",
"environment_id": "room_001",
"detection_settings": {
"confidence_threshold": 0.7,
"max_persons": 5,
"enable_tracking": true
},
"hardware_settings": {
"csi_sampling_rate": 30,
"buffer_size": 1000
}
}
}
```
**Response:**
```json
{
"success": true,
"data": {
"status": "starting",
"session_id": "session_123456",
"estimated_startup_time": 15,
"configuration_applied": {
// Applied configuration
}
}
}
```
### Stop System
Stop the pose estimation system.
```http
POST /api/v1/system/stop
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"status": "stopping",
"session_id": "session_123456",
"shutdown_initiated": "2025-01-07T10:30:00Z"
}
}
```
### Get System Status
Get current system status and performance metrics.
```http
GET /api/v1/system/status
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"status": "running",
"session_id": "session_123456",
"uptime_seconds": 3600,
"started_at": "2025-01-07T09:30:00Z",
"performance": {
"frames_processed": 108000,
"average_fps": 29.8,
"average_latency_ms": 45.2,
"cpu_usage": 65.4,
"memory_usage": 78.2,
"gpu_usage": 82.1
},
"components": {
"csi_processor": {
"status": "healthy",
"last_heartbeat": "2025-01-07T10:29:55Z"
},
"neural_network": {
"status": "healthy",
"model_loaded": true,
"inference_queue_size": 3
},
"tracker": {
"status": "healthy",
"active_tracks": 2
},
"database": {
"status": "healthy",
"connection_pool": "8/20"
}
}
}
}
```
### Restart System
Restart the pose estimation system.
```http
POST /api/v1/system/restart
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"status": "restarting",
"previous_session_id": "session_123456",
"new_session_id": "session_789012",
"estimated_restart_time": 30
}
}
```
## Configuration Endpoints
### Get Current Configuration
Retrieve the current system configuration.
```http
GET /api/v1/config
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"domain": "healthcare",
"environment_id": "room_001",
"detection": {
"confidence_threshold": 0.7,
"max_persons": 5,
"enable_tracking": true,
"tracking_max_age": 30,
"tracking_min_hits": 3
},
"neural_network": {
"model_version": "v1.2.0",
"batch_size": 32,
"enable_gpu": true,
"inference_timeout": 1000
},
"hardware": {
"csi_sampling_rate": 30,
"buffer_size": 1000,
"antenna_count": 3,
"subcarrier_count": 56
},
"analytics": {
"enable_fall_detection": true,
"enable_activity_recognition": true,
"alert_thresholds": {
"fall_confidence": 0.8,
"inactivity_timeout": 300
}
},
"privacy": {
"data_retention_days": 30,
"anonymize_data": true,
"enable_encryption": true
}
}
}
```
### Update Configuration
Update system configuration (requires system restart for some changes).
```http
PUT /api/v1/config
Authorization: Bearer <token>
Content-Type: application/json
{
"detection": {
"confidence_threshold": 0.8,
"max_persons": 3
},
"analytics": {
"enable_fall_detection": true,
"alert_thresholds": {
"fall_confidence": 0.9
}
}
}
```
**Response:**
```json
{
"success": true,
"data": {
"updated_fields": [
"detection.confidence_threshold",
"detection.max_persons",
"analytics.alert_thresholds.fall_confidence"
],
"requires_restart": false,
"applied_at": "2025-01-07T10:30:00Z",
"configuration": {
// Updated configuration
}
}
}
```
### Get Configuration Schema
Get the configuration schema with validation rules and descriptions.
```http
GET /api/v1/config/schema
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"schema": {
"type": "object",
"properties": {
"detection": {
"type": "object",
"properties": {
"confidence_threshold": {
"type": "number",
"minimum": 0.0,
"maximum": 1.0,
"description": "Minimum confidence for pose detection"
}
}
}
}
},
"defaults": {
// Default configuration values
}
}
}
```
## Analytics Endpoints
### Get Analytics Summary
Get analytics summary for a specified time period.
```http
GET /api/v1/analytics/summary
Authorization: Bearer <token>
```
**Query Parameters:**
- `start_time` (required): Start timestamp
- `end_time` (required): End timestamp
- `environment_id` (optional): Filter by environment
- `granularity` (optional): Data granularity (hour, day, week)
**Response:**
```json
{
"success": true,
"data": {
"time_period": {
"start": "2025-01-07T00:00:00Z",
"end": "2025-01-07T23:59:59Z",
"duration_hours": 24
},
"detection_stats": {
"total_detections": 15420,
"unique_persons": 47,
"average_confidence": 0.84,
"peak_occupancy": 8,
"peak_occupancy_time": "2025-01-07T14:30:00Z"
},
"activity_stats": {
"total_movement_events": 1250,
"fall_detections": 2,
"alert_count": 5,
"average_activity_level": 0.67
},
"system_stats": {
"uptime_percentage": 99.8,
"average_processing_time": 45.2,
"frames_processed": 2592000,
"error_count": 12
},
"hourly_breakdown": [
{
"hour": "2025-01-07T00:00:00Z",
"detections": 420,
"unique_persons": 2,
"average_confidence": 0.82
}
// ... hourly data
]
}
}
```
### Get Activity Events
Retrieve detected activity events (falls, alerts, etc.).
```http
GET /api/v1/analytics/events
Authorization: Bearer <token>
```
**Query Parameters:**
- `start_time` (optional): Start timestamp
- `end_time` (optional): End timestamp
- `event_type` (optional): Filter by event type (fall, alert, activity)
- `severity` (optional): Filter by severity (low, medium, high)
- `environment_id` (optional): Filter by environment
**Response:**
```json
{
"success": true,
"data": [
{
"event_id": "event_123456",
"type": "fall_detection",
"severity": "high",
"timestamp": "2025-01-07T14:25:30Z",
"environment_id": "room_001",
"person_id": 3,
"track_id": 15,
"confidence": 0.92,
"location": {
"x": 210,
"y": 180
},
"metadata": {
"fall_duration": 2.3,
"impact_severity": 0.85,
"recovery_detected": false
},
"actions_taken": [
"alert_sent",
"notification_dispatched"
]
}
// ... additional events
]
}
```
### Get Occupancy Data
Get occupancy statistics and trends.
```http
GET /api/v1/analytics/occupancy
Authorization: Bearer <token>
```
**Query Parameters:**
- `start_time` (required): Start timestamp
- `end_time` (required): End timestamp
- `environment_id` (optional): Filter by environment
- `interval` (optional): Data interval (5min, 15min, 1hour)
**Response:**
```json
{
"success": true,
"data": {
"summary": {
"average_occupancy": 3.2,
"peak_occupancy": 8,
"peak_time": "2025-01-07T14:30:00Z",
"total_person_hours": 76.8
},
"time_series": [
{
"timestamp": "2025-01-07T00:00:00Z",
"occupancy": 2,
"confidence": 0.89
},
{
"timestamp": "2025-01-07T00:15:00Z",
"occupancy": 1,
"confidence": 0.92
}
// ... time series data
],
"distribution": {
"0_persons": 15.2,
"1_person": 42.8,
"2_persons": 28.5,
"3_persons": 10.1,
"4_plus_persons": 3.4
}
}
}
```
## Health and Status Endpoints
### Health Check
Basic health check endpoint for load balancers and monitoring.
```http
GET /api/v1/health
```
**Response:**
```json
{
"status": "healthy",
"timestamp": "2025-01-07T10:30:00Z",
"version": "1.2.0",
"uptime": 3600
}
```
### Detailed Health Check
Comprehensive health check with component status.
```http
GET /api/v1/health/detailed
Authorization: Bearer <token>
```
**Response:**
```json
{
"success": true,
"data": {
"overall_status": "healthy",
"timestamp": "2025-01-07T10:30:00Z",
"version": "1.2.0",
"uptime": 3600,
"components": {
"api": {
"status": "healthy",
"response_time_ms": 12.3,
"requests_per_second": 45.2
},
"database": {
"status": "healthy",
"connection_pool": "8/20",
"query_time_ms": 5.7
},
"redis": {
"status": "healthy",
"memory_usage": "45%",
"connected_clients": 12
},
"neural_network": {
"status": "healthy",
"model_loaded": true,
"gpu_memory_usage": "78%",
"inference_queue": 2
},
"csi_processor": {
"status": "healthy",
"data_rate": 30.1,
"buffer_usage": "23%"
}
},
"metrics": {
"cpu_usage": 65.4,
"memory_usage": 78.2,
"disk_usage": 45.8,
"network_io": {
"bytes_in": 1024000,
"bytes_out": 2048000
}
}
}
}
```
### System Metrics
Get detailed system performance metrics.
```http
GET /api/v1/metrics
Authorization: Bearer <token>
```
**Query Parameters:**
- `start_time` (optional): Start timestamp for historical metrics
- `end_time` (optional): End timestamp for historical metrics
- `metric_type` (optional): Filter by metric type
**Response:**
```json
{
"success": true,
"data": {
"current": {
"timestamp": "2025-01-07T10:30:00Z",
"performance": {
"frames_per_second": 29.8,
"average_latency_ms": 45.2,
"processing_queue_size": 3,
"error_rate": 0.001
},
"resources": {
"cpu_usage": 65.4,
"memory_usage": 78.2,
"gpu_usage": 82.1,
"disk_io": {
"read_mb_per_sec": 12.5,
"write_mb_per_sec": 8.3
}
},
"business": {
"active_persons": 3,
"detections_per_minute": 89.5,
"tracking_accuracy": 0.94
}
},
"historical": [
{
"timestamp": "2025-01-07T10:25:00Z",
"frames_per_second": 30.1,
"average_latency_ms": 43.8,
"cpu_usage": 62.1
}
// ... historical data points
]
}
}
```
## Rate Limiting
### Rate Limit Headers
All API responses include rate limiting headers:
```http
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1704686400
X-RateLimit-Window: 3600
```
### Rate Limits by Endpoint Category
| Category | Limit | Window |
|----------|-------|--------|
| Authentication | 10 requests | 1 minute |
| Pose Data (GET) | 1000 requests | 1 hour |
| Pose Processing (POST) | 100 requests | 1 hour |
| Configuration | 50 requests | 1 hour |
| Analytics | 500 requests | 1 hour |
| Health Checks | 10000 requests | 1 hour |
### Rate Limit Exceeded Response
```json
{
"success": false,
"error": {
"code": "RATE_LIMIT_EXCEEDED",
"message": "Rate limit exceeded. Try again in 45 seconds.",
"details": {
"limit": 1000,
"window": 3600,
"reset_at": "2025-01-07T11:00:00Z"
}
}
}
```
---
This REST API documentation provides comprehensive coverage of all available endpoints. For real-time data streaming, see the [WebSocket API documentation](websocket-api.md). For authentication details, see the [Authentication documentation](authentication.md).
For code examples in multiple languages, see the [API Examples documentation](examples.md).
archive/v1/docs/api/websocket-api.md
+998
View File
@@ -0,0 +1,998 @@
# WebSocket API Documentation
## Overview
The WiFi-DensePose WebSocket API provides real-time streaming of pose estimation data, system events, and analytics. This enables applications to receive live updates without polling REST endpoints, making it ideal for real-time monitoring dashboards and interactive applications.
## Table of Contents
1. [Connection Setup](#connection-setup)
2. [Authentication](#authentication)
3. [Message Format](#message-format)
4. [Event Types](#event-types)
5. [Subscription Management](#subscription-management)
6. [Real-time Pose Streaming](#real-time-pose-streaming)
7. [System Events](#system-events)
8. [Analytics Streaming](#analytics-streaming)
9. [Error Handling](#error-handling)
10. [Connection Management](#connection-management)
11. [Rate Limiting](#rate-limiting)
12. [Code Examples](#code-examples)
## Connection Setup
### WebSocket Endpoint
```
Production: wss://api.wifi-densepose.com/ws/v1
Staging: wss://staging-api.wifi-densepose.com/ws/v1
Development: ws://localhost:8000/ws/v1
```
### Connection URL Parameters
```
wss://api.wifi-densepose.com/ws/v1?token=<jwt_token>&client_id=<client_id>
```
**Parameters:**
- `token` (required): JWT authentication token
- `client_id` (optional): Unique client identifier for connection tracking
- `compression` (optional): Enable compression (gzip, deflate)
### Connection Headers
```http
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: wifi-densepose-v1
Authorization: Bearer <jwt_token>
```
## Authentication
### JWT Token Authentication
Include the JWT token in the connection URL or as a header:
```javascript
// URL parameter method
const ws = new WebSocket('wss://api.wifi-densepose.com/ws/v1?token=your_jwt_token');
// Header method (if supported by client)
const ws = new WebSocket('wss://api.wifi-densepose.com/ws/v1', [], {
headers: {
'Authorization': 'Bearer your_jwt_token'
}
});
```
### Token Refresh
When a token expires, the server will send a `token_expired` event. Clients should refresh their token and reconnect:
```json
{
"type": "token_expired",
"timestamp": "2025-01-07T10:30:00Z",
"message": "JWT token has expired. Please refresh and reconnect."
}
```
## Message Format
### Standard Message Structure
All WebSocket messages follow this JSON structure:
```json
{
"type": "message_type",
"timestamp": "2025-01-07T10:30:00.123Z",
"data": {
// Message-specific data
},
"metadata": {
"client_id": "client_123",
"sequence": 12345,
"compression": "gzip"
}
}
```
### Message Types
| Type | Direction | Description |
|------|-----------|-------------|
| `subscribe` | Client → Server | Subscribe to event streams |
| `unsubscribe` | Client → Server | Unsubscribe from event streams |
| `pose_data` | Server → Client | Real-time pose estimation data |
| `system_event` | Server → Client | System status and events |
| `analytics_update` | Server → Client | Analytics and metrics updates |
| `error` | Server → Client | Error notifications |
| `heartbeat` | Bidirectional | Connection keep-alive |
| `ack` | Server → Client | Acknowledgment of client messages |
## Event Types
### Pose Data Events
#### Real-time Pose Detection
```json
{
"type": "pose_data",
"timestamp": "2025-01-07T10:30:00.123Z",
"data": {
"frame_id": 12345,
"environment_id": "room_001",
"processing_time_ms": 45.2,
"persons": [
{
"person_id": 1,
"track_id": 7,
"confidence": 0.87,
"bounding_box": {
"x": 120,
"y": 80,
"width": 180,
"height": 320
},
"keypoints": [
{
"name": "nose",
"x": 210,
"y": 95,
"confidence": 0.92,
"visible": true
}
// ... additional keypoints
],
"activity": {
"type": "walking",
"confidence": 0.78,
"velocity": {
"x": 0.5,
"y": 0.2
}
}
}
],
"metadata": {
"model_version": "v1.2.0",
"csi_quality": 0.85,
"frame_rate": 29.8
}
}
}
```
#### Person Tracking Updates
```json
{
"type": "tracking_update",
"timestamp": "2025-01-07T10:30:00.123Z",
"data": {
"track_id": 7,
"person_id": 1,
"event": "track_started",
"position": {
"x": 210,
"y": 240
},
"confidence": 0.87,
"metadata": {
"first_detection": "2025-01-07T10:29:45Z",
"track_quality": 0.92
}
}
}
```
### System Events
#### System Status Changes
```json
{
"type": "system_event",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"event": "system_started",
"status": "running",
"session_id": "session_123456",
"configuration": {
"domain": "healthcare",
"environment_id": "room_001"
},
"components": {
"neural_network": "healthy",
"csi_processor": "healthy",
"tracker": "healthy"
}
}
}
```
#### Hardware Events
```json
{
"type": "hardware_event",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"event": "router_disconnected",
"router_id": "router_001",
"severity": "warning",
"message": "Router connection lost. Attempting reconnection...",
"metadata": {
"last_seen": "2025-01-07T10:29:30Z",
"reconnect_attempts": 1
}
}
}
```
### Analytics Events
#### Activity Detection
```json
{
"type": "activity_event",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"event_type": "fall_detected",
"severity": "high",
"person_id": 3,
"track_id": 15,
"confidence": 0.92,
"location": {
"x": 210,
"y": 180
},
"details": {
"fall_duration": 2.3,
"impact_severity": 0.85,
"recovery_detected": false
},
"actions": [
"alert_triggered",
"notification_sent"
]
}
}
```
#### Occupancy Updates
```json
{
"type": "occupancy_update",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"environment_id": "room_001",
"current_occupancy": 3,
"previous_occupancy": 2,
"change_type": "person_entered",
"confidence": 0.89,
"persons": [
{
"person_id": 1,
"track_id": 7,
"status": "active"
},
{
"person_id": 2,
"track_id": 8,
"status": "active"
},
{
"person_id": 4,
"track_id": 12,
"status": "new"
}
]
}
}
```
## Subscription Management
### Subscribe to Events
Send a subscription message to start receiving specific event types:
```json
{
"type": "subscribe",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"subscriptions": [
{
"event_type": "pose_data",
"filters": {
"environment_id": "room_001",
"min_confidence": 0.7,
"include_keypoints": true,
"include_dense_pose": false
},
"throttle": {
"max_fps": 10,
"buffer_size": 5
}
},
{
"event_type": "system_event",
"filters": {
"severity": ["warning", "error", "critical"]
}
},
{
"event_type": "activity_event",
"filters": {
"event_types": ["fall_detected", "alert_triggered"]
}
}
]
}
}
```
### Subscription Acknowledgment
Server responds with subscription confirmation:
```json
{
"type": "ack",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"message_type": "subscribe",
"status": "success",
"active_subscriptions": [
{
"subscription_id": "sub_123",
"event_type": "pose_data",
"status": "active"
},
{
"subscription_id": "sub_124",
"event_type": "system_event",
"status": "active"
}
]
}
}
```
### Unsubscribe from Events
```json
{
"type": "unsubscribe",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"subscription_ids": ["sub_123", "sub_124"]
}
}
```
### Update Subscription Filters
```json
{
"type": "update_subscription",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"subscription_id": "sub_123",
"filters": {
"min_confidence": 0.8,
"max_fps": 15
}
}
}
```
## Real-time Pose Streaming
### High-Frequency Pose Data
For applications requiring high-frequency updates:
```json
{
"type": "subscribe",
"data": {
"subscriptions": [
{
"event_type": "pose_data",
"filters": {
"environment_id": "room_001",
"min_confidence": 0.5,
"include_keypoints": true,
"include_dense_pose": true,
"include_velocity": true
},
"throttle": {
"max_fps": 30,
"buffer_size": 1,
"compression": "gzip"
},
"quality": "high"
}
]
}
}
```
### Pose Data with Trajectory
```json
{
"type": "pose_data_trajectory",
"timestamp": "2025-01-07T10:30:00.123Z",
"data": {
"track_id": 7,
"person_id": 1,
"trajectory": [
{
"timestamp": "2025-01-07T10:29:58.123Z",
"position": {"x": 200, "y": 230},
"confidence": 0.89
},
{
"timestamp": "2025-01-07T10:29:59.123Z",
"position": {"x": 205, "y": 235},
"confidence": 0.91
},
{
"timestamp": "2025-01-07T10:30:00.123Z",
"position": {"x": 210, "y": 240},
"confidence": 0.87
}
],
"prediction": {
"next_position": {"x": 215, "y": 245},
"confidence": 0.73,
"time_horizon": 1.0
}
}
}
```
## System Events
### Performance Monitoring
```json
{
"type": "performance_update",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"metrics": {
"frames_per_second": 29.8,
"average_latency_ms": 45.2,
"processing_queue_size": 3,
"cpu_usage": 65.4,
"memory_usage": 78.2,
"gpu_usage": 82.1
},
"alerts": [
{
"type": "high_latency",
"severity": "warning",
"value": 67.3,
"threshold": 50.0
}
]
}
}
```
### Configuration Changes
```json
{
"type": "config_update",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"changed_fields": [
"detection.confidence_threshold",
"analytics.enable_fall_detection"
],
"new_values": {
"detection.confidence_threshold": 0.8,
"analytics.enable_fall_detection": true
},
"applied_by": "admin_user",
"requires_restart": false
}
}
```
## Analytics Streaming
### Real-time Analytics
```json
{
"type": "analytics_stream",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"window": "1_minute",
"metrics": {
"occupancy": {
"current": 3,
"average": 2.7,
"peak": 5
},
"activity": {
"movement_events": 15,
"stationary_time": 45.2,
"activity_level": 0.67
},
"detection": {
"total_detections": 1800,
"average_confidence": 0.84,
"tracking_accuracy": 0.92
}
},
"trends": {
"occupancy_trend": "increasing",
"activity_trend": "stable",
"confidence_trend": "improving"
}
}
}
```
## Error Handling
### Connection Errors
```json
{
"type": "error",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"error_code": "CONNECTION_ERROR",
"message": "WebSocket connection lost",
"details": {
"reason": "network_timeout",
"retry_after": 5,
"max_retries": 3
}
}
}
```
### Subscription Errors
```json
{
"type": "error",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"error_code": "SUBSCRIPTION_ERROR",
"message": "Invalid subscription filter",
"details": {
"subscription_id": "sub_123",
"field": "min_confidence",
"reason": "Value must be between 0 and 1"
}
}
}
```
### Rate Limit Errors
```json
{
"type": "error",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"error_code": "RATE_LIMIT_EXCEEDED",
"message": "Message rate limit exceeded",
"details": {
"current_rate": 150,
"limit": 100,
"window": "1_minute",
"retry_after": 30
}
}
}
```
## Connection Management
### Heartbeat
Both client and server should send periodic heartbeat messages:
```json
{
"type": "heartbeat",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"client_id": "client_123",
"uptime": 3600,
"last_message": "2025-01-07T10:29:55Z"
}
}
```
### Connection Status
```json
{
"type": "connection_status",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"status": "connected",
"client_id": "client_123",
"session_id": "session_789",
"connected_since": "2025-01-07T09:30:00Z",
"active_subscriptions": 3,
"message_count": 1250
}
}
```
### Graceful Disconnect
```json
{
"type": "disconnect",
"timestamp": "2025-01-07T10:30:00Z",
"data": {
"reason": "client_requested",
"message": "Graceful disconnect initiated by client"
}
}
```
## Rate Limiting
### Message Rate Limits
| Message Type | Limit | Window |
|--------------|-------|--------|
| Subscribe/Unsubscribe | 10 messages | 1 minute |
| Heartbeat | 1 message | 30 seconds |
| General Commands | 60 messages | 1 minute |
### Data Rate Limits
| Subscription Type | Max Rate | Buffer Size |
|-------------------|----------|-------------|
| Pose Data (Low Quality) | 10 FPS | 5 frames |
| Pose Data (High Quality) | 30 FPS | 1 frame |
| System Events | 100 events/min | 10 events |
| Analytics | 60 updates/min | 5 updates |
## Code Examples
### JavaScript Client
```javascript
class WiFiDensePoseWebSocket {
constructor(token, options = {}) {
this.token = token;
this.options = {
url: 'wss://api.wifi-densepose.com/ws/v1',
reconnectInterval: 5000,
maxReconnectAttempts: 5,
...options
};
this.ws = null;
this.reconnectAttempts = 0;
this.subscriptions = new Map();
}
connect() {
const url = `${this.options.url}?token=${this.token}`;
this.ws = new WebSocket(url);
this.ws.onopen = () => {
console.log('Connected to WiFi-DensePose WebSocket');
this.reconnectAttempts = 0;
this.startHeartbeat();
};
this.ws.onmessage = (event) => {
const message = JSON.parse(event.data);
this.handleMessage(message);
};
this.ws.onclose = (event) => {
console.log('WebSocket connection closed:', event.code);
this.stopHeartbeat();
this.attemptReconnect();
};
this.ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
}
subscribeToPoseData(environmentId, options = {}) {
const subscription = {
event_type: 'pose_data',
filters: {
environment_id: environmentId,
min_confidence: options.minConfidence || 0.7,
include_keypoints: options.includeKeypoints !== false,
include_dense_pose: options.includeDensePose || false
},
throttle: {
max_fps: options.maxFps || 10,
buffer_size: options.bufferSize || 5
}
};
this.send({
type: 'subscribe',
timestamp: new Date().toISOString(),
data: {
subscriptions: [subscription]
}
});
}
subscribeToSystemEvents() {
this.send({
type: 'subscribe',
timestamp: new Date().toISOString(),
data: {
subscriptions: [{
event_type: 'system_event',
filters: {
severity: ['warning', 'error', 'critical']
}
}]
}
});
}
handleMessage(message) {
switch (message.type) {
case 'pose_data':
this.onPoseData(message.data);
break;
case 'system_event':
this.onSystemEvent(message.data);
break;
case 'activity_event':
this.onActivityEvent(message.data);
break;
case 'error':
this.onError(message.data);
break;
case 'ack':
this.onAcknowledgment(message.data);
break;
}
}
onPoseData(data) {
// Handle pose data
console.log('Received pose data:', data);
}
onSystemEvent(data) {
// Handle system events
console.log('System event:', data);
}
onActivityEvent(data) {
// Handle activity events
console.log('Activity event:', data);
}
onError(data) {
console.error('WebSocket error:', data);
}
send(message) {
if (this.ws && this.ws.readyState === WebSocket.OPEN) {
this.ws.send(JSON.stringify(message));
}
}
startHeartbeat() {
this.heartbeatInterval = setInterval(() => {
this.send({
type: 'heartbeat',
timestamp: new Date().toISOString(),
data: {
client_id: this.options.clientId,
uptime: Date.now() - this.connectTime
}
});
}, 30000);
}
stopHeartbeat() {
if (this.heartbeatInterval) {
clearInterval(this.heartbeatInterval);
}
}
attemptReconnect() {
if (this.reconnectAttempts < this.options.maxReconnectAttempts) {
this.reconnectAttempts++;
console.log(`Attempting to reconnect (${this.reconnectAttempts}/${this.options.maxReconnectAttempts})`);
setTimeout(() => {
this.connect();
}, this.options.reconnectInterval);
}
}
disconnect() {
this.stopHeartbeat();
if (this.ws) {
this.ws.close();
}
}
}
// Usage example
const client = new WiFiDensePoseWebSocket('your_jwt_token', {
clientId: 'dashboard_client_001'
});
client.onPoseData = (data) => {
// Update UI with pose data
updatePoseVisualization(data);
};
client.onActivityEvent = (data) => {
if (data.event_type === 'fall_detected') {
showFallAlert(data);
}
};
client.connect();
client.subscribeToPoseData('room_001', {
minConfidence: 0.8,
maxFps: 15,
includeKeypoints: true
});
```
### Python Client
```python
import asyncio
import websockets
import json
from datetime import datetime
class WiFiDensePoseWebSocket:
def __init__(self, token, url='wss://api.wifi-densepose.com/ws/v1'):
self.token = token
self.url = f"{url}?token={token}"
self.websocket = None
self.subscriptions = {}
async def connect(self):
"""Connect to the WebSocket server."""
try:
self.websocket = await websockets.connect(self.url)
print("Connected to WiFi-DensePose WebSocket")
# Start heartbeat task
asyncio.create_task(self.heartbeat())
# Listen for messages
await self.listen()
except Exception as e:
print(f"Connection error: {e}")
async def listen(self):
"""Listen for incoming messages."""
try:
async for message in self.websocket:
data = json.loads(message)
await self.handle_message(data)
except websockets.exceptions.ConnectionClosed:
print("WebSocket connection closed")
except Exception as e:
print(f"Error listening for messages: {e}")
async def handle_message(self, message):
"""Handle incoming messages."""
message_type = message.get('type')
data = message.get('data', {})
if message_type == 'pose_data':
await self.on_pose_data(data)
elif message_type == 'system_event':
await self.on_system_event(data)
elif message_type == 'activity_event':
await self.on_activity_event(data)
elif message_type == 'error':
await self.on_error(data)
async def subscribe_to_pose_data(self, environment_id, **options):
"""Subscribe to pose data stream."""
subscription = {
'event_type': 'pose_data',
'filters': {
'environment_id': environment_id,
'min_confidence': options.get('min_confidence', 0.7),
'include_keypoints': options.get('include_keypoints', True),
'include_dense_pose': options.get('include_dense_pose', False)
},
'throttle': {
'max_fps': options.get('max_fps', 10),
'buffer_size': options.get('buffer_size', 5)
}
}
await self.send({
'type': 'subscribe',
'timestamp': datetime.utcnow().isoformat() + 'Z',
'data': {
'subscriptions': [subscription]
}
})
async def send(self, message):
"""Send a message to the server."""
if self.websocket:
await self.websocket.send(json.dumps(message))
async def heartbeat(self):
"""Send periodic heartbeat messages."""
while True:
try:
await self.send({
'type': 'heartbeat',
'timestamp': datetime.utcnow().isoformat() + 'Z',
'data': {
'client_id': 'python_client'
}
})
await asyncio.sleep(30)
except Exception as e:
print(f"Heartbeat error: {e}")
break
async def on_pose_data(self, data):
"""Handle pose data."""
print(f"Received pose data: {len(data.get('persons', []))} persons detected")
async def on_system_event(self, data):
"""Handle system events."""
print(f"System event: {data.get('event')} - {data.get('message', '')}")
async def on_activity_event(self, data):
"""Handle activity events."""
if data.get('event_type') == 'fall_detected':
print(f"FALL DETECTED: Person {data.get('person_id')} at {data.get('location')}")
async def on_error(self, data):
"""Handle errors."""
print(f"WebSocket error: {data.get('message')}")
# Usage example
async def main():
client = WiFiDensePoseWebSocket('your_jwt_token')
# Connect and subscribe
await client.connect()
await client.subscribe_to_pose_data('room_001', min_confidence=0.8)
if __name__ == "__main__":
asyncio.run(main())
```
---
This WebSocket API documentation provides comprehensive coverage of real-time communication capabilities. For authentication details, see the [Authentication documentation](authentication.md). For REST API endpoints, see the [REST Endpoints documentation](rest-endpoints.md).
archive/v1/docs/api_reference.md
+931
View File
@@ -0,0 +1,931 @@
# WiFi-DensePose API Reference
## Table of Contents
1. [Overview](#overview)
2. [Authentication](#authentication)
3. [Base URL and Versioning](#base-url-and-versioning)
4. [Request/Response Format](#requestresponse-format)
5. [Error Handling](#error-handling)
6. [Rate Limiting](#rate-limiting)
7. [Pose Estimation API](#pose-estimation-api)
8. [System Management API](#system-management-api)
9. [Health Check API](#health-check-api)
10. [WebSocket API](#websocket-api)
11. [Data Models](#data-models)
12. [SDK Examples](#sdk-examples)
## Overview
The WiFi-DensePose API provides comprehensive access to WiFi-based human pose estimation capabilities. The API follows REST principles and supports both synchronous HTTP requests and real-time WebSocket connections.
### Key Features
- **RESTful Design**: Standard HTTP methods and status codes
- **Real-time Streaming**: WebSocket support for live pose data
- **Authentication**: JWT-based authentication with role-based access
- **Rate Limiting**: Configurable rate limits to prevent abuse
- **Comprehensive Documentation**: OpenAPI/Swagger documentation
- **Error Handling**: Detailed error responses with actionable messages
### API Capabilities
- Real-time pose estimation from WiFi CSI data
- Historical pose data retrieval and analysis
- System health monitoring and diagnostics
- Multi-zone occupancy tracking
- Activity recognition and analytics
- System configuration and calibration
## Authentication
### JWT Authentication
The API uses JSON Web Tokens (JWT) for authentication. Include the token in the `Authorization` header:
```http
Authorization: Bearer <your-jwt-token>
```
### Obtaining a Token
```bash
# Login to get JWT token
curl -X POST http://localhost:8000/api/v1/auth/login \
-H "Content-Type: application/json" \
-d '{
"username": "your-username",
"password": "your-password"
}'
```
**Response:**
```json
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"token_type": "bearer",
"expires_in": 86400
}
```
### Token Refresh
```bash
# Refresh expired token
curl -X POST http://localhost:8000/api/v1/auth/refresh \
-H "Authorization: Bearer <your-refresh-token>"
```
### Public Endpoints
Some endpoints are publicly accessible without authentication:
- `GET /api/v1/health/*` - Health check endpoints
- `GET /api/v1/version` - Version information
- `GET /docs` - API documentation
## Base URL and Versioning
### Base URL
```
http://localhost:8000/api/v1
```
### API Versioning
The API uses URL path versioning. Current version is `v1`.
### Content Types
- **Request**: `application/json`
- **Response**: `application/json`
- **WebSocket**: `application/json` messages
## Request/Response Format
### Standard Response Format
```json
{
"data": {},
"timestamp": "2025-01-07T10:00:00Z",
"status": "success"
}
```
### Error Response Format
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid request parameters",
"details": {
"field": "confidence_threshold",
"issue": "Value must be between 0.0 and 1.0"
}
},
"timestamp": "2025-01-07T10:00:00Z",
"status": "error"
}
```
## Error Handling
### HTTP Status Codes
| Code | Description |
|------|-------------|
| 200 | Success |
| 201 | Created |
| 400 | Bad Request |
| 401 | Unauthorized |
| 403 | Forbidden |
| 404 | Not Found |
| 409 | Conflict |
| 422 | Validation Error |
| 429 | Rate Limited |
| 500 | Internal Server Error |
| 503 | Service Unavailable |
### Error Codes
| Code | Description |
|------|-------------|
| `VALIDATION_ERROR` | Request validation failed |
| `AUTHENTICATION_ERROR` | Authentication failed |
| `AUTHORIZATION_ERROR` | Insufficient permissions |
| `RESOURCE_NOT_FOUND` | Requested resource not found |
| `RATE_LIMIT_EXCEEDED` | Rate limit exceeded |
| `HARDWARE_ERROR` | Hardware communication error |
| `PROCESSING_ERROR` | Pose processing error |
| `CALIBRATION_ERROR` | System calibration error |
## Rate Limiting
### Default Limits
- **Authenticated users**: 1000 requests per hour
- **Anonymous users**: 100 requests per hour
- **WebSocket connections**: 10 concurrent per user
### Rate Limit Headers
```http
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1641556800
```
### Rate Limit Response
```json
{
"error": {
"code": "RATE_LIMIT_EXCEEDED",
"message": "Rate limit exceeded. Try again in 60 seconds."
}
}
```
## Pose Estimation API
### Get Current Pose Estimation
Get real-time pose estimation from WiFi signals.
```http
GET /api/v1/pose/current
```
**Query Parameters:**
- `zone_ids` (array, optional): Specific zones to analyze
- `confidence_threshold` (float, optional): Minimum confidence (0.0-1.0)
- `max_persons` (integer, optional): Maximum persons to detect (1-50)
- `include_keypoints` (boolean, optional): Include keypoint data (default: true)
- `include_segmentation` (boolean, optional): Include segmentation masks (default: false)
**Example Request:**
```bash
curl "http://localhost:8000/api/v1/pose/current?confidence_threshold=0.7&max_persons=5" \
-H "Authorization: Bearer <token>"
```
**Response:**
```json
{
"timestamp": "2025-01-07T10:00:00Z",
"frame_id": "frame_12345",
"persons": [
{
"person_id": "person_001",
"confidence": 0.85,
"bounding_box": {
"x": 100,
"y": 150,
"width": 80,
"height": 180
},
"keypoints": [
{
"name": "nose",
"x": 140,
"y": 160,
"confidence": 0.9
}
],
"zone_id": "zone_001",
"activity": "standing",
"timestamp": "2025-01-07T10:00:00Z"
}
],
"zone_summary": {
"zone_001": 1,
"zone_002": 0
},
"processing_time_ms": 45.2
}
```
### Analyze Pose Data
Trigger pose analysis with custom parameters.
```http
POST /api/v1/pose/analyze
```
**Request Body:**
```json
{
"zone_ids": ["zone_001", "zone_002"],
"confidence_threshold": 0.8,
"max_persons": 10,
"include_keypoints": true,
"include_segmentation": false
}
```
**Response:** Same format as current pose estimation.
### Get Zone Occupancy
Get current occupancy for a specific zone.
```http
GET /api/v1/pose/zones/{zone_id}/occupancy
```
**Path Parameters:**
- `zone_id` (string): Zone identifier
**Example Request:**
```bash
curl "http://localhost:8000/api/v1/pose/zones/zone_001/occupancy" \
-H "Authorization: Bearer <token>"
```
**Response:**
```json
{
"zone_id": "zone_001",
"current_occupancy": 3,
"max_occupancy": 10,
"persons": [
{
"person_id": "person_001",
"confidence": 0.85,
"activity": "standing"
}
],
"timestamp": "2025-01-07T10:00:00Z"
}
```
### Get Zones Summary
Get occupancy summary for all zones.
```http
GET /api/v1/pose/zones/summary
```
**Response:**
```json
{
"timestamp": "2025-01-07T10:00:00Z",
"total_persons": 5,
"zones": {
"zone_001": {
"occupancy": 3,
"max_occupancy": 10,
"status": "normal"
},
"zone_002": {
"occupancy": 2,
"max_occupancy": 8,
"status": "normal"
}
},
"active_zones": 2
}
```
### Get Historical Data
Retrieve historical pose estimation data.
```http
POST /api/v1/pose/historical
```
**Request Body:**
```json
{
"start_time": "2025-01-07T00:00:00Z",
"end_time": "2025-01-07T23:59:59Z",
"zone_ids": ["zone_001"],
"aggregation_interval": 300,
"include_raw_data": false
}
```
**Response:**
```json
{
"query": {
"start_time": "2025-01-07T00:00:00Z",
"end_time": "2025-01-07T23:59:59Z",
"zone_ids": ["zone_001"],
"aggregation_interval": 300
},
"data": [
{
"timestamp": "2025-01-07T00:00:00Z",
"average_occupancy": 2.5,
"max_occupancy": 5,
"total_detections": 150
}
],
"total_records": 288
}
```
### Get Detected Activities
Get recently detected activities.
```http
GET /api/v1/pose/activities
```
**Query Parameters:**
- `zone_id` (string, optional): Filter by zone
- `limit` (integer, optional): Maximum activities (1-100, default: 10)
**Response:**
```json
{
"activities": [
{
"activity": "walking",
"person_id": "person_001",
"zone_id": "zone_001",
"confidence": 0.9,
"timestamp": "2025-01-07T10:00:00Z",
"duration_seconds": 15.5
}
],
"total_count": 1,
"zone_id": "zone_001"
}
```
### Calibrate System
Start system calibration process.
```http
POST /api/v1/pose/calibrate
```
**Response:**
```json
{
"calibration_id": "cal_12345",
"status": "started",
"estimated_duration_minutes": 5,
"message": "Calibration process started"
}
```
### Get Calibration Status
Check calibration progress.
```http
GET /api/v1/pose/calibration/status
```
**Response:**
```json
{
"is_calibrating": true,
"calibration_id": "cal_12345",
"progress_percent": 60,
"current_step": "phase_sanitization",
"estimated_remaining_minutes": 2,
"last_calibration": "2025-01-06T15:30:00Z"
}
```
### Get Pose Statistics
Get pose estimation statistics.
```http
GET /api/v1/pose/stats
```
**Query Parameters:**
- `hours` (integer, optional): Hours of data to analyze (1-168, default: 24)
**Response:**
```json
{
"period": {
"start_time": "2025-01-06T10:00:00Z",
"end_time": "2025-01-07T10:00:00Z",
"hours": 24
},
"statistics": {
"total_detections": 1500,
"average_confidence": 0.82,
"unique_persons": 25,
"average_processing_time_ms": 47.3,
"zones": {
"zone_001": {
"detections": 800,
"average_occupancy": 3.2
}
}
}
}
```
## System Management API
### System Status
Get current system status.
```http
GET /api/v1/system/status
```
**Response:**
```json
{
"status": "running",
"uptime_seconds": 86400,
"services": {
"hardware": "healthy",
"pose_estimation": "healthy",
"streaming": "healthy"
},
"configuration": {
"domain": "healthcare",
"max_persons": 10,
"confidence_threshold": 0.7
},
"timestamp": "2025-01-07T10:00:00Z"
}
```
### Start System
Start the pose estimation system.
```http
POST /api/v1/system/start
```
**Request Body:**
```json
{
"configuration": {
"domain": "healthcare",
"environment_id": "room_001",
"calibration_required": true
}
}
```
### Stop System
Stop the pose estimation system.
```http
POST /api/v1/system/stop
```
### Restart System
Restart the system with new configuration.
```http
POST /api/v1/system/restart
```
### Get Configuration
Get current system configuration.
```http
GET /api/v1/config
```
### Update Configuration
Update system configuration.
```http
PUT /api/v1/config
```
**Request Body:**
```json
{
"detection": {
"confidence_threshold": 0.8,
"max_persons": 8
},
"analytics": {
"enable_fall_detection": true
}
}
```
## Health Check API
### Comprehensive Health Check
Get detailed system health information.
```http
GET /api/v1/health
```
**Response:**
```json
{
"status": "healthy",
"timestamp": "2025-01-07T10:00:00Z",
"uptime_seconds": 86400,
"components": {
"hardware": {
"name": "Hardware Service",
"status": "healthy",
"message": "All routers connected",
"last_check": "2025-01-07T10:00:00Z",
"uptime_seconds": 86400,
"metrics": {
"connected_routers": 3,
"csi_data_rate": 30.5
}
},
"pose": {
"name": "Pose Service",
"status": "healthy",
"message": "Processing normally",
"last_check": "2025-01-07T10:00:00Z",
"metrics": {
"processing_rate": 29.8,
"average_latency_ms": 45.2
}
}
},
"system_metrics": {
"cpu": {
"percent": 65.2,
"count": 8
},
"memory": {
"total_gb": 16.0,
"available_gb": 8.5,
"percent": 46.9
},
"disk": {
"total_gb": 500.0,
"free_gb": 350.0,
"percent": 30.0
}
}
}
```
### Readiness Check
Check if system is ready to serve requests.
```http
GET /api/v1/ready
```
**Response:**
```json
{
"ready": true,
"timestamp": "2025-01-07T10:00:00Z",
"checks": {
"hardware_ready": true,
"pose_ready": true,
"stream_ready": true,
"memory_available": true,
"disk_space_available": true
},
"message": "System is ready"
}
```
### Liveness Check
Simple liveness check for load balancers.
```http
GET /api/v1/live
```
**Response:**
```json
{
"status": "alive",
"timestamp": "2025-01-07T10:00:00Z"
}
```
### System Metrics
Get detailed system metrics.
```http
GET /api/v1/metrics
```
### Version Information
Get application version information.
```http
GET /api/v1/version
```
**Response:**
```json
{
"name": "WiFi-DensePose API",
"version": "1.0.0",
"environment": "production",
"debug": false,
"timestamp": "2025-01-07T10:00:00Z"
}
```
## WebSocket API
### Connection
Connect to WebSocket endpoint:
```javascript
const ws = new WebSocket('ws://localhost:8000/ws/pose/stream');
```
### Authentication
Send authentication message after connection:
```javascript
ws.send(JSON.stringify({
type: 'auth',
token: 'your-jwt-token'
}));
```
### Subscribe to Pose Updates
```javascript
ws.send(JSON.stringify({
type: 'subscribe',
channel: 'pose_updates',
filters: {
zone_ids: ['zone_001'],
min_confidence: 0.7
}
}));
```
### Pose Data Message
```json
{
"type": "pose_data",
"channel": "pose_updates",
"data": {
"timestamp": "2025-01-07T10:00:00Z",
"frame_id": "frame_12345",
"persons": [
{
"person_id": "person_001",
"confidence": 0.85,
"bounding_box": {
"x": 100,
"y": 150,
"width": 80,
"height": 180
},
"zone_id": "zone_001"
}
]
}
}
```
### System Events
Subscribe to system events:
```javascript
ws.send(JSON.stringify({
type: 'subscribe',
channel: 'system_events'
}));
```
### Event Message
```json
{
"type": "system_event",
"channel": "system_events",
"data": {
"event_type": "fall_detected",
"person_id": "person_001",
"zone_id": "zone_001",
"confidence": 0.95,
"timestamp": "2025-01-07T10:00:00Z"
}
}
```
## Data Models
### PersonPose
```json
{
"person_id": "string",
"confidence": 0.85,
"bounding_box": {
"x": 100,
"y": 150,
"width": 80,
"height": 180
},
"keypoints": [
{
"name": "nose",
"x": 140,
"y": 160,
"confidence": 0.9,
"visible": true
}
],
"segmentation": {
"mask": "base64-encoded-mask",
"body_parts": ["torso", "left_arm", "right_arm"]
},
"zone_id": "zone_001",
"activity": "standing",
"timestamp": "2025-01-07T10:00:00Z"
}
```
### Keypoint Names
Standard keypoint names following COCO format:
- `nose`, `left_eye`, `right_eye`, `left_ear`, `right_ear`
- `left_shoulder`, `right_shoulder`, `left_elbow`, `right_elbow`
- `left_wrist`, `right_wrist`, `left_hip`, `right_hip`
- `left_knee`, `right_knee`, `left_ankle`, `right_ankle`
### Activity Types
Supported activity classifications:
- `standing`, `sitting`, `walking`, `running`, `lying_down`
- `falling`, `jumping`, `bending`, `reaching`, `waving`
### Zone Configuration
```json
{
"zone_id": "zone_001",
"name": "Living Room",
"coordinates": {
"x": 0,
"y": 0,
"width": 500,
"height": 300
},
"max_occupancy": 10,
"alerts_enabled": true,
"privacy_level": "high"
}
```
## SDK Examples
### Python SDK
```python
from wifi_densepose import WiFiDensePoseClient
# Initialize client
client = WiFiDensePoseClient(
base_url="http://localhost:8000",
api_key="your-api-key"
)
# Get current poses
poses = client.get_current_poses(
confidence_threshold=0.7,
max_persons=5
)
# Get historical data
history = client.get_historical_data(
start_time="2025-01-07T00:00:00Z",
end_time="2025-01-07T23:59:59Z",
zone_ids=["zone_001"]
)
# Subscribe to real-time updates
def pose_callback(poses):
print(f"Received {len(poses)} poses")
client.subscribe_to_poses(callback=pose_callback)
```
### JavaScript SDK
```javascript
import { WiFiDensePoseClient } from 'wifi-densepose-js';
// Initialize client
const client = new WiFiDensePoseClient({
baseUrl: 'http://localhost:8000',
apiKey: 'your-api-key'
});
// Get current poses
const poses = await client.getCurrentPoses({
confidenceThreshold: 0.7,
maxPersons: 5
});
// Subscribe to WebSocket updates
client.subscribeToPoses({
onData: (poses) => {
console.log(`Received ${poses.length} poses`);
},
onError: (error) => {
console.error('WebSocket error:', error);
}
});
```
### cURL Examples
```bash
# Get current poses
curl -X GET "http://localhost:8000/api/v1/pose/current?confidence_threshold=0.7" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json"
# Start system
curl -X POST "http://localhost:8000/api/v1/system/start" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"configuration": {
"domain": "healthcare",
"environment_id": "room_001"
}
}'
# Get zone occupancy
curl -X GET "http://localhost:8000/api/v1/pose/zones/zone_001/occupancy" \
-H "Authorization: Bearer <token>"
```
---
For more information, see:
- [User Guide](user_guide.md)
- [Deployment Guide](deployment.md)
- [Troubleshooting Guide](troubleshooting.md)
- [Interactive API Documentation](http://localhost:8000/docs)
archive/v1/docs/deployment.md
+1103
View File
File diff suppressed because it is too large Load Diff
archive/v1/docs/deployment/README.md
+484
View File
@@ -0,0 +1,484 @@
# WiFi-DensePose DevOps & Deployment Guide
This guide provides comprehensive instructions for deploying and managing the WiFi-DensePose application infrastructure using modern DevOps practices.
## 🏗️ Architecture Overview
The WiFi-DensePose deployment architecture includes:
- **Container Orchestration**: Kubernetes with auto-scaling capabilities
- **Infrastructure as Code**: Terraform for AWS resource provisioning
- **CI/CD Pipelines**: GitHub Actions and GitLab CI support
- **Monitoring**: Prometheus, Grafana, and comprehensive alerting
- **Logging**: Centralized log aggregation with Fluentd and Elasticsearch
- **Security**: Automated security scanning and compliance checks
## 📋 Prerequisites
### Required Tools
Ensure the following tools are installed on your system:
```bash
# AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# Terraform
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform
# Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
### AWS Configuration
Configure AWS credentials with appropriate permissions:
```bash
aws configure
# Enter your AWS Access Key ID, Secret Access Key, and default region
```
Required AWS permissions:
- EC2 (VPC, Subnets, Security Groups, Load Balancers)
- EKS (Cluster management)
- ECR (Container registry)
- IAM (Roles and policies)
- S3 (State storage and log backup)
- CloudWatch (Monitoring and logging)
## 🚀 Quick Start
### 1. Clone and Setup
```bash
git clone <repository-url>
cd wifi-densepose
```
### 2. Configure Environment
```bash
# Set environment variables
export ENVIRONMENT=production
export AWS_REGION=us-west-2
export PROJECT_NAME=wifi-densepose
```
### 3. Deploy Everything
```bash
# Deploy complete infrastructure and application
./deploy.sh all
```
### 4. Verify Deployment
```bash
# Check application status
kubectl get pods -n wifi-densepose
# Access Grafana dashboard
kubectl port-forward svc/grafana 3000:80 -n monitoring
# Open http://localhost:3000 (admin/admin)
# Access application
kubectl get ingress -n wifi-densepose
```
## 📁 Directory Structure
```
├── deploy.sh # Main deployment script
├── Dockerfile # Application container image
├── docker-compose.yml # Local development setup
├── docker-compose.prod.yml # Production deployment
├── .dockerignore # Docker build context optimization
├── .github/workflows/ # GitHub Actions CI/CD
│ ├── ci.yml # Continuous Integration
│ ├── cd.yml # Continuous Deployment
│ └── security-scan.yml # Security scanning
├── .gitlab-ci.yml # GitLab CI configuration
├── k8s/ # Kubernetes manifests
│ ├── namespace.yaml # Namespace definition
│ ├── deployment.yaml # Application deployment
│ ├── service.yaml # Service configuration
│ ├── ingress.yaml # Ingress rules
│ ├── configmap.yaml # Configuration management
│ ├── secrets.yaml # Secret management template
│ └── hpa.yaml # Horizontal Pod Autoscaler
├── terraform/ # Infrastructure as Code
│ ├── main.tf # Main infrastructure definition
│ ├── variables.tf # Configuration variables
│ └── outputs.tf # Output values
├── ansible/ # Server configuration
│ └── playbook.yml # Ansible playbook
├── monitoring/ # Monitoring configuration
│ ├── prometheus-config.yml # Prometheus configuration
│ ├── grafana-dashboard.json # Grafana dashboard
│ └── alerting-rules.yml # Alert rules
└── logging/ # Logging configuration
└── fluentd-config.yml # Fluentd configuration
```
## 🔧 Deployment Options
### Individual Component Deployment
```bash
# Deploy only infrastructure
./deploy.sh infrastructure
# Deploy only Kubernetes resources
./deploy.sh kubernetes
# Deploy only monitoring stack
./deploy.sh monitoring
# Build and push Docker images
./deploy.sh images
# Run health checks
./deploy.sh health
# Setup CI/CD
./deploy.sh cicd
```
### Environment-Specific Deployment
```bash
# Development environment
ENVIRONMENT=development ./deploy.sh all
# Staging environment
ENVIRONMENT=staging ./deploy.sh all
# Production environment
ENVIRONMENT=production ./deploy.sh all
```
## 🐳 Docker Configuration
### Local Development
```bash
# Start local development environment
docker-compose up -d
# View logs
docker-compose logs -f
# Stop environment
docker-compose down
```
### Production Build
```bash
# Build production image
docker build -f Dockerfile -t wifi-densepose:latest .
# Multi-stage build for optimization
docker build --target production -t wifi-densepose:prod .
```
## ☸️ Kubernetes Management
### Common Operations
```bash
# View application logs
kubectl logs -f deployment/wifi-densepose -n wifi-densepose
# Scale application
kubectl scale deployment wifi-densepose --replicas=5 -n wifi-densepose
# Update application
kubectl set image deployment/wifi-densepose wifi-densepose=new-image:tag -n wifi-densepose
# Rollback deployment
kubectl rollout undo deployment/wifi-densepose -n wifi-densepose
# View resource usage
kubectl top pods -n wifi-densepose
kubectl top nodes
```
### Configuration Management
```bash
# Update ConfigMap
kubectl create configmap wifi-densepose-config \
--from-file=config/ \
--dry-run=client -o yaml | kubectl apply -f -
# Update Secrets
kubectl create secret generic wifi-densepose-secrets \
--from-literal=database-password=secret \
--dry-run=client -o yaml | kubectl apply -f -
```
## 📊 Monitoring & Observability
### Prometheus Metrics
Access Prometheus at: `http://localhost:9090` (via port-forward)
Key metrics to monitor:
- `http_requests_total` - HTTP request count
- `http_request_duration_seconds` - Request latency
- `wifi_densepose_data_processed_total` - Data processing metrics
- `wifi_densepose_model_inference_duration_seconds` - ML model performance
### Grafana Dashboards
Access Grafana at: `http://localhost:3000` (admin/admin)
Pre-configured dashboards:
- Application Overview
- Infrastructure Metrics
- Database Performance
- Kubernetes Cluster Status
- Security Alerts
### Log Analysis
```bash
# View application logs
kubectl logs -f -l app=wifi-densepose -n wifi-densepose
# Search logs in Elasticsearch
curl -X GET "elasticsearch:9200/wifi-densepose-*/_search" \
-H 'Content-Type: application/json' \
-d '{"query": {"match": {"level": "error"}}}'
```
## 🔒 Security Best Practices
### Implemented Security Measures
1. **Container Security**
- Non-root user execution
- Minimal base images
- Regular vulnerability scanning
- Resource limits and quotas
2. **Kubernetes Security**
- Network policies
- Pod security policies
- RBAC configuration
- Secret management
3. **Infrastructure Security**
- VPC with private subnets
- Security groups with minimal access
- IAM roles with least privilege
- Encrypted storage and transit
4. **CI/CD Security**
- Automated security scanning
- Dependency vulnerability checks
- Container image scanning
- Secret scanning
### Security Scanning
```bash
# Run security scan
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
aquasec/trivy image wifi-densepose:latest
# Kubernetes security scan
kubectl run --rm -i --tty kube-bench --image=aquasec/kube-bench:latest \
--restart=Never -- --version 1.20
```
## 🔄 CI/CD Pipelines
### GitHub Actions
Workflows are triggered on:
- **CI Pipeline** (`ci.yml`): Pull requests and pushes to main
- **CD Pipeline** (`cd.yml`): Tags and main branch pushes
- **Security Scan** (`security-scan.yml`): Daily scheduled runs
### GitLab CI
Configure GitLab CI variables:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `KUBE_CONFIG`
- `ECR_REPOSITORY`
## 🏗️ Infrastructure as Code
### Terraform Configuration
```bash
# Initialize Terraform
cd terraform
terraform init
# Plan deployment
terraform plan -var="environment=production"
# Apply changes
terraform apply
# Destroy infrastructure
terraform destroy
```
### Ansible Configuration
```bash
# Run Ansible playbook
ansible-playbook -i inventory ansible/playbook.yml
```
## 🚨 Troubleshooting
### Common Issues
1. **Pod Startup Issues**
```bash
kubectl describe pod <pod-name> -n wifi-densepose
kubectl logs <pod-name> -n wifi-densepose
```
2. **Service Discovery Issues**
```bash
kubectl get endpoints -n wifi-densepose
kubectl get services -n wifi-densepose
```
3. **Ingress Issues**
```bash
kubectl describe ingress wifi-densepose-ingress -n wifi-densepose
kubectl get events -n wifi-densepose
```
4. **Resource Issues**
```bash
kubectl top pods -n wifi-densepose
kubectl describe nodes
```
### Health Checks
```bash
# Application health
curl http://<ingress-url>/health
# Database connectivity
kubectl exec -it <pod-name> -n wifi-densepose -- pg_isready
# Redis connectivity
kubectl exec -it <pod-name> -n wifi-densepose -- redis-cli ping
```
## 📈 Scaling & Performance
### Horizontal Pod Autoscaler
```bash
# View HPA status
kubectl get hpa -n wifi-densepose
# Update HPA configuration
kubectl patch hpa wifi-densepose-hpa -n wifi-densepose -p '{"spec":{"maxReplicas":10}}'
```
### Cluster Autoscaler
```bash
# View cluster autoscaler logs
kubectl logs -f deployment/cluster-autoscaler -n kube-system
```
### Performance Tuning
1. **Resource Requests/Limits**
- CPU: Request 100m, Limit 500m
- Memory: Request 256Mi, Limit 512Mi
2. **Database Optimization**
- Connection pooling
- Query optimization
- Index management
3. **Caching Strategy**
- Redis for session storage
- Application-level caching
- CDN for static assets
## 🔄 Backup & Recovery
### Database Backup
```bash
# Create database backup
kubectl exec -it postgres-pod -n wifi-densepose -- \
pg_dump -U postgres wifi_densepose > backup.sql
# Restore database
kubectl exec -i postgres-pod -n wifi-densepose -- \
psql -U postgres wifi_densepose < backup.sql
```
### Configuration Backup
```bash
# Backup Kubernetes resources
kubectl get all -n wifi-densepose -o yaml > k8s-backup.yaml
# Backup ConfigMaps and Secrets
kubectl get configmaps,secrets -n wifi-densepose -o yaml > config-backup.yaml
```
## 📞 Support & Maintenance
### Regular Maintenance Tasks
1. **Weekly**
- Review monitoring alerts
- Check resource utilization
- Update dependencies
2. **Monthly**
- Security patch updates
- Performance optimization
- Backup verification
3. **Quarterly**
- Disaster recovery testing
- Security audit
- Capacity planning
### Contact Information
- **DevOps Team**: devops@wifi-densepose.com
- **On-Call**: +1-555-0123
- **Documentation**: https://docs.wifi-densepose.com
- **Status Page**: https://status.wifi-densepose.com
## 📚 Additional Resources
- [Kubernetes Documentation](https://kubernetes.io/docs/)
- [Terraform AWS Provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)
- [Prometheus Monitoring](https://prometheus.io/docs/)
- [Grafana Dashboards](https://grafana.com/docs/)
- [AWS EKS Best Practices](https://aws.github.io/aws-eks-best-practices/)
archive/v1/docs/developer/architecture-overview.md
+848
View File
@@ -0,0 +1,848 @@
# Architecture Overview
## Overview
The WiFi-DensePose system is a distributed, microservices-based architecture that transforms WiFi Channel State Information (CSI) into real-time human pose estimation. This document provides a comprehensive overview of the system architecture, component interactions, and design principles.
## Table of Contents
1. [System Architecture](#system-architecture)
2. [Core Components](#core-components)
3. [Data Flow](#data-flow)
4. [Processing Pipeline](#processing-pipeline)
5. [API Architecture](#api-architecture)
6. [Storage Architecture](#storage-architecture)
7. [Security Architecture](#security-architecture)
8. [Deployment Architecture](#deployment-architecture)
9. [Scalability and Performance](#scalability-and-performance)
10. [Design Principles](#design-principles)
## System Architecture
### High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ WiFi-DensePose System │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Client Apps │ │ Web Dashboard │ │ Mobile Apps │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ API Gateway │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ REST API │ │ WebSocket API │ │ MQTT Broker │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Processing Layer │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Pose Estimation │ │ Tracking │ │ Analytics │ │
│ │ Service │ │ Service │ │ Service │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Data Layer │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ CSI Processor │ │ Data Pipeline │ │ Model Manager │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Hardware Layer │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ WiFi Routers │ │ Processing Unit │ │ GPU Cluster │ │
│ │ (CSI Data) │ │ (CPU/Memory) │ │ (Neural Net) │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
### Component Interaction Diagram
```
┌─────────────┐ CSI Data ┌─────────────┐ Features ┌─────────────┐
│ Router │ ──────────────▶│ CSI │ ──────────────▶│ Feature │
│ Network │ │ Processor │ │ Extractor │
└─────────────┘ └─────────────┘ └─────────────┘
│ │
▼ ▼
┌─────────────┐ Poses ┌─────────────┐ Inference ┌─────────────┐
│ Client │ ◀──────────────│ Pose │ ◀──────────────│ Neural │
│ Applications│ │ Tracker │ │ Network │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
▼ ▼ ▼
┌─────────────┐ Events ┌─────────────┐ Models ┌─────────────┐
│ Alert │ ◀──────────────│ Analytics │ ◀──────────────│ Model │
│ System │ │ Engine │ │ Manager │
└─────────────┘ └─────────────┘ └─────────────┘
```
## Core Components
### 1. CSI Data Processor
**Purpose**: Receives and processes raw Channel State Information from WiFi routers.
**Key Features**:
- Real-time CSI data ingestion from multiple routers
- Signal preprocessing and noise reduction
- Phase sanitization and amplitude normalization
- Multi-antenna data fusion
**Implementation**: [`src/hardware/csi_processor.py`](../../src/hardware/csi_processor.py)
```python
class CSIProcessor:
"""Processes raw CSI data from WiFi routers."""
def __init__(self, config: CSIConfig):
self.routers = self._initialize_routers(config.routers)
self.buffer = CircularBuffer(config.buffer_size)
self.preprocessor = CSIPreprocessor()
async def process_stream(self) -> AsyncGenerator[CSIData, None]:
"""Process continuous CSI data stream."""
async for raw_data in self._receive_csi_data():
processed_data = self.preprocessor.process(raw_data)
yield processed_data
```
### 2. Neural Network Service
**Purpose**: Performs pose estimation using deep learning models.
**Key Features**:
- DensePose model inference
- Batch processing optimization
- GPU acceleration support
- Model versioning and hot-swapping
**Implementation**: [`src/neural_network/inference.py`](../../src/neural_network/inference.py)
```python
class PoseEstimationService:
"""Neural network service for pose estimation."""
def __init__(self, model_config: ModelConfig):
self.model = self._load_model(model_config.model_path)
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
self.batch_processor = BatchProcessor(model_config.batch_size)
async def estimate_poses(self, csi_features: CSIFeatures) -> List[PoseEstimation]:
"""Estimate human poses from CSI features."""
with torch.no_grad():
predictions = self.model(csi_features.to(self.device))
return self._postprocess_predictions(predictions)
```
### 3. Tracking Service
**Purpose**: Maintains temporal consistency and person identity across frames.
**Key Features**:
- Multi-object tracking with Kalman filters
- Person re-identification
- Track lifecycle management
- Trajectory smoothing
**Implementation**: [`src/tracking/tracker.py`](../../src/tracking/tracker.py)
```python
class PersonTracker:
"""Tracks multiple persons across time."""
def __init__(self, tracking_config: TrackingConfig):
self.tracks = {}
self.track_id_counter = 0
self.kalman_filter = KalmanFilter()
self.reid_model = ReIDModel()
def update(self, detections: List[PoseDetection]) -> List[TrackedPose]:
"""Update tracks with new detections."""
matched_tracks, unmatched_detections = self._associate_detections(detections)
self._update_matched_tracks(matched_tracks)
self._create_new_tracks(unmatched_detections)
return self._get_active_tracks()
```
### 4. API Gateway
**Purpose**: Provides unified access to system functionality through REST and WebSocket APIs.
**Key Features**:
- Authentication and authorization
- Rate limiting and throttling
- Request routing and load balancing
- API versioning
**Implementation**: [`src/api/main.py`](../../src/api/main.py)
```python
from fastapi import FastAPI, Depends
from fastapi.middleware.cors import CORSMiddleware
app = FastAPI(
title="WiFi-DensePose API",
version="1.0.0",
description="Privacy-preserving human pose estimation using WiFi signals"
)
# Middleware
app.add_middleware(CORSMiddleware, **get_cors_config())
app.add_middleware(RateLimitMiddleware)
app.add_middleware(AuthenticationMiddleware)
# Routers
app.include_router(pose_router, prefix="/api/v1/pose")
app.include_router(system_router, prefix="/api/v1/system")
app.include_router(analytics_router, prefix="/api/v1/analytics")
```
### 5. Analytics Engine
**Purpose**: Processes pose data to generate insights and trigger alerts.
**Key Features**:
- Real-time event detection (falls, intrusions)
- Statistical analysis and reporting
- Domain-specific analytics (healthcare, retail, security)
- Machine learning-based pattern recognition
**Implementation**: [`src/analytics/engine.py`](../../src/analytics/engine.py)
```python
class AnalyticsEngine:
"""Processes pose data for insights and alerts."""
def __init__(self, domain_config: DomainConfig):
self.domain = domain_config.domain
self.event_detectors = self._load_event_detectors(domain_config)
self.alert_manager = AlertManager(domain_config.alerts)
async def process_poses(self, poses: List[TrackedPose]) -> AnalyticsResult:
"""Process poses and generate analytics."""
events = []
for detector in self.event_detectors:
detected_events = await detector.detect(poses)
events.extend(detected_events)
await self.alert_manager.process_events(events)
return AnalyticsResult(events=events, metrics=self._calculate_metrics(poses))
```
## Data Flow
### Real-Time Processing Pipeline
```
1. CSI Data Acquisition
┌─────────────┐
│ Router 1 │ ──┐
└─────────────┘ │
┌─────────────┐ │ ┌─────────────┐
│ Router 2 │ ──┼───▶│ CSI Buffer │
└─────────────┘ │ └─────────────┘
┌─────────────┐ │ │
│ Router N │ ──┘ ▼
└─────────────┘ ┌─────────────┐
│ Preprocessor│
└─────────────┘
│
2. Feature Extraction ▼
┌─────────────┐ ┌─────────────┐
│ Phase │ ◀─────│ Feature │
│ Sanitizer │ │ Extractor │
└─────────────┘ └─────────────┘
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Amplitude │ │ Frequency │
│ Processor │ │ Analyzer │
└─────────────┘ └─────────────┘
│ │
└──────┬──────────────┘
▼
3. Neural Network Inference
┌─────────────┐
│ DensePose │
│ Model │
└─────────────┘
│
▼
┌─────────────┐
│ Pose │
│ Decoder │
└─────────────┘
│
4. Tracking and Analytics ▼
┌─────────────┐ ┌─────────────┐
│ Person │ ◀─────│ Raw Pose │
│ Tracker │ │ Detections │
└─────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ Analytics │
│ Engine │
└─────────────┘
│
5. Output and Storage ▼
┌─────────────┐ ┌─────────────┐
│ WebSocket │ ◀─────│ Tracked │
│ Streams │ │ Poses │
└─────────────┘ └─────────────┘
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Client │ │ Database │
│ Applications│ │ Storage │
└─────────────┘ └─────────────┘
```
### Data Models
#### CSI Data Structure
```python
@dataclass
class CSIData:
"""Channel State Information data structure."""
timestamp: datetime
router_id: str
antenna_pairs: List[AntennaPair]
subcarriers: List[SubcarrierData]
metadata: CSIMetadata
@dataclass
class SubcarrierData:
"""Individual subcarrier information."""
frequency: float
amplitude: complex
phase: float
snr: float
```
#### Pose Data Structure
```python
@dataclass
class PoseEstimation:
"""Human pose estimation result."""
person_id: Optional[int]
confidence: float
bounding_box: BoundingBox
keypoints: List[Keypoint]
dense_pose: Optional[DensePoseResult]
timestamp: datetime
@dataclass
class TrackedPose:
"""Tracked pose with temporal information."""
track_id: int
pose: PoseEstimation
velocity: Vector2D
track_age: int
track_confidence: float
```
## Processing Pipeline
### 1. CSI Preprocessing
```python
class CSIPreprocessor:
"""Preprocesses raw CSI data for neural network input."""
def __init__(self, config: PreprocessingConfig):
self.phase_sanitizer = PhaseSanitizer()
self.amplitude_normalizer = AmplitudeNormalizer()
self.noise_filter = NoiseFilter(config.filter_params)
def process(self, raw_csi: RawCSIData) -> ProcessedCSIData:
"""Process raw CSI data."""
# Phase unwrapping and sanitization
sanitized_phase = self.phase_sanitizer.sanitize(raw_csi.phase)
# Amplitude normalization
normalized_amplitude = self.amplitude_normalizer.normalize(raw_csi.amplitude)
# Noise filtering
filtered_data = self.noise_filter.filter(sanitized_phase, normalized_amplitude)
return ProcessedCSIData(
phase=filtered_data.phase,
amplitude=filtered_data.amplitude,
timestamp=raw_csi.timestamp,
metadata=raw_csi.metadata
)
```
### 2. Feature Extraction
```python
class FeatureExtractor:
"""Extracts features from processed CSI data."""
def __init__(self, config: FeatureConfig):
self.window_size = config.window_size
self.feature_types = config.feature_types
self.pca_reducer = PCAReducer(config.pca_components)
def extract_features(self, csi_data: ProcessedCSIData) -> CSIFeatures:
"""Extract features for neural network input."""
features = {}
if 'amplitude' in self.feature_types:
features['amplitude'] = self._extract_amplitude_features(csi_data)
if 'phase' in self.feature_types:
features['phase'] = self._extract_phase_features(csi_data)
if 'doppler' in self.feature_types:
features['doppler'] = self._extract_doppler_features(csi_data)
# Dimensionality reduction
reduced_features = self.pca_reducer.transform(features)
return CSIFeatures(
features=reduced_features,
timestamp=csi_data.timestamp,
feature_types=self.feature_types
)
```
### 3. Neural Network Architecture
```python
class DensePoseNet(nn.Module):
"""DensePose neural network for WiFi-based pose estimation."""
def __init__(self, config: ModelConfig):
super().__init__()
self.backbone = self._build_backbone(config.backbone)
self.feature_pyramid = FeaturePyramidNetwork(config.fpn)
self.pose_head = PoseEstimationHead(config.pose_head)
self.dense_pose_head = DensePoseHead(config.dense_pose_head)
def forward(self, csi_features: torch.Tensor) -> Dict[str, torch.Tensor]:
"""Forward pass through the network."""
# Feature extraction
backbone_features = self.backbone(csi_features)
pyramid_features = self.feature_pyramid(backbone_features)
# Pose estimation
pose_predictions = self.pose_head(pyramid_features)
dense_pose_predictions = self.dense_pose_head(pyramid_features)
return {
'poses': pose_predictions,
'dense_poses': dense_pose_predictions
}
```
## API Architecture
### REST API Design
The REST API follows RESTful principles with clear resource hierarchies:
```
/api/v1/
├── auth/
│ ├── token # POST: Get authentication token
│ └── verify # POST: Verify token validity
├── system/
│ ├── status # GET: System health status
│ ├── start # POST: Start pose estimation
│ ├── stop # POST: Stop pose estimation
│ └── diagnostics # GET: System diagnostics
├── pose/
│ ├── latest # GET: Latest pose data
│ ├── history # GET: Historical pose data
│ └── query # POST: Complex pose queries
├── config/
│ └── [resource] # GET/PUT: Configuration management
└── analytics/
├── healthcare # GET: Healthcare analytics
├── retail # GET: Retail analytics
└── security # GET: Security analytics
```
### WebSocket API Design
```python
class WebSocketManager:
"""Manages WebSocket connections and subscriptions."""
def __init__(self):
self.connections: Dict[str, WebSocket] = {}
self.subscriptions: Dict[str, Set[str]] = {}
async def handle_connection(self, websocket: WebSocket, client_id: str):
"""Handle new WebSocket connection."""
await websocket.accept()
self.connections[client_id] = websocket
try:
async for message in websocket.iter_text():
await self._handle_message(client_id, json.loads(message))
except WebSocketDisconnect:
self._cleanup_connection(client_id)
async def broadcast_pose_update(self, pose_data: TrackedPose):
"""Broadcast pose updates to subscribed clients."""
message = {
'type': 'pose_update',
'data': pose_data.to_dict(),
'timestamp': datetime.utcnow().isoformat()
}
for client_id in self.subscriptions.get('pose_updates', set()):
if client_id in self.connections:
await self.connections[client_id].send_text(json.dumps(message))
```
## Storage Architecture
### Database Design
#### Time-Series Data (PostgreSQL + TimescaleDB)
```sql
-- Pose data table with time-series optimization
CREATE TABLE pose_data (
id BIGSERIAL PRIMARY KEY,
timestamp TIMESTAMPTZ NOT NULL,
frame_id BIGINT NOT NULL,
person_id INTEGER,
track_id INTEGER,
confidence REAL NOT NULL,
bounding_box JSONB NOT NULL,
keypoints JSONB NOT NULL,
dense_pose JSONB,
metadata JSONB,
environment_id VARCHAR(50) NOT NULL
);
-- Convert to hypertable for time-series optimization
SELECT create_hypertable('pose_data', 'timestamp');
-- Create indexes for common queries
CREATE INDEX idx_pose_data_timestamp ON pose_data (timestamp DESC);
CREATE INDEX idx_pose_data_person_id ON pose_data (person_id, timestamp DESC);
CREATE INDEX idx_pose_data_environment ON pose_data (environment_id, timestamp DESC);
```
#### Configuration Storage (PostgreSQL)
```sql
-- System configuration
CREATE TABLE system_config (
id SERIAL PRIMARY KEY,
domain VARCHAR(50) NOT NULL,
environment_id VARCHAR(50) NOT NULL,
config_data JSONB NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(domain, environment_id)
);
-- Model metadata
CREATE TABLE model_metadata (
id SERIAL PRIMARY KEY,
model_name VARCHAR(100) NOT NULL,
model_version VARCHAR(20) NOT NULL,
model_path TEXT NOT NULL,
config JSONB NOT NULL,
performance_metrics JSONB,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(model_name, model_version)
);
```
### Caching Strategy (Redis)
```python
class CacheManager:
"""Manages Redis caching for frequently accessed data."""
def __init__(self, redis_client: Redis):
self.redis = redis_client
self.default_ttl = 300 # 5 minutes
async def cache_pose_data(self, pose_data: TrackedPose, ttl: int = None):
"""Cache pose data with automatic expiration."""
key = f"pose:latest:{pose_data.track_id}"
value = json.dumps(pose_data.to_dict(), default=str)
await self.redis.setex(key, ttl or self.default_ttl, value)
async def get_cached_poses(self, track_ids: List[int]) -> List[TrackedPose]:
"""Retrieve cached pose data for multiple tracks."""
keys = [f"pose:latest:{track_id}" for track_id in track_ids]
cached_data = await self.redis.mget(keys)
poses = []
for data in cached_data:
if data:
pose_dict = json.loads(data)
poses.append(TrackedPose.from_dict(pose_dict))
return poses
```
## Security Architecture
### Authentication and Authorization
```python
class SecurityManager:
"""Handles authentication and authorization."""
def __init__(self, config: SecurityConfig):
self.jwt_secret = config.jwt_secret
self.jwt_algorithm = config.jwt_algorithm
self.token_expiry = config.token_expiry
def create_access_token(self, user_data: dict) -> str:
"""Create JWT access token."""
payload = {
'sub': user_data['username'],
'exp': datetime.utcnow() + timedelta(hours=self.token_expiry),
'iat': datetime.utcnow(),
'permissions': user_data.get('permissions', [])
}
return jwt.encode(payload, self.jwt_secret, algorithm=self.jwt_algorithm)
def verify_token(self, token: str) -> dict:
"""Verify and decode JWT token."""
try:
payload = jwt.decode(token, self.jwt_secret, algorithms=[self.jwt_algorithm])
return payload
except jwt.ExpiredSignatureError:
raise HTTPException(status_code=401, detail="Token expired")
except jwt.InvalidTokenError:
raise HTTPException(status_code=401, detail="Invalid token")
```
### Data Privacy
```python
class PrivacyManager:
"""Manages data privacy and anonymization."""
def __init__(self, config: PrivacyConfig):
self.anonymization_enabled = config.anonymization_enabled
self.data_retention_days = config.data_retention_days
self.encryption_key = config.encryption_key
def anonymize_pose_data(self, pose_data: TrackedPose) -> TrackedPose:
"""Anonymize pose data for privacy protection."""
if not self.anonymization_enabled:
return pose_data
# Remove or hash identifying information
anonymized_data = pose_data.copy()
anonymized_data.track_id = self._hash_track_id(pose_data.track_id)
# Apply differential privacy to keypoints
anonymized_data.pose.keypoints = self._add_noise_to_keypoints(
pose_data.pose.keypoints
)
return anonymized_data
```
## Deployment Architecture
### Container Architecture
```yaml
# docker-compose.yml
version: '3.8'
services:
wifi-densepose-api:
build: .
ports:
- "8000:8000"
environment:
- DATABASE_URL=postgresql://user:pass@postgres:5432/wifi_densepose
- REDIS_URL=redis://redis:6379/0
depends_on:
- postgres
- redis
- neural-network
volumes:
- ./data:/app/data
- ./models:/app/models
neural-network:
build: ./neural_network
runtime: nvidia
environment:
- CUDA_VISIBLE_DEVICES=0
volumes:
- ./models:/app/models
postgres:
image: timescale/timescaledb:latest-pg14
environment:
- POSTGRES_DB=wifi_densepose
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
volumes:
- postgres_data:/var/lib/postgresql/data
redis:
image: redis:7-alpine
volumes:
- redis_data:/data
volumes:
postgres_data:
redis_data:
```
### Kubernetes Deployment
```yaml
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: wifi-densepose-api
spec:
replicas: 3
selector:
matchLabels:
app: wifi-densepose-api
template:
metadata:
labels:
app: wifi-densepose-api
spec:
containers:
- name: api
image: wifi-densepose:latest
ports:
- containerPort: 8000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: database-secret
key: url
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "2000m"
```
## Scalability and Performance
### Horizontal Scaling
```python
class LoadBalancer:
"""Distributes processing load across multiple instances."""
def __init__(self, config: LoadBalancerConfig):
self.processing_nodes = config.processing_nodes
self.load_balancing_strategy = config.strategy
self.health_checker = HealthChecker()
async def distribute_csi_data(self, csi_data: CSIData) -> str:
"""Distribute CSI data to available processing nodes."""
available_nodes = await self.health_checker.get_healthy_nodes()
if self.load_balancing_strategy == 'round_robin':
node = self._round_robin_selection(available_nodes)
elif self.load_balancing_strategy == 'least_loaded':
node = await self._least_loaded_selection(available_nodes)
else:
node = random.choice(available_nodes)
await self._send_to_node(node, csi_data)
return node.id
```
### Performance Optimization
```python
class PerformanceOptimizer:
"""Optimizes system performance based on runtime metrics."""
def __init__(self, config: OptimizationConfig):
self.metrics_collector = MetricsCollector()
self.auto_scaling_enabled = config.auto_scaling_enabled
self.optimization_interval = config.optimization_interval
async def optimize_processing_pipeline(self):
"""Optimize processing pipeline based on current metrics."""
metrics = await self.metrics_collector.get_current_metrics()
# Adjust batch size based on GPU utilization
if metrics.gpu_utilization < 0.7:
await self._increase_batch_size()
elif metrics.gpu_utilization > 0.9:
await self._decrease_batch_size()
# Scale processing nodes based on queue length
if metrics.processing_queue_length > 100:
await self._scale_up_processing_nodes()
elif metrics.processing_queue_length < 10:
await self._scale_down_processing_nodes()
```
## Design Principles
### 1. Modularity and Separation of Concerns
- Each component has a single, well-defined responsibility
- Clear interfaces between components
- Pluggable architecture for easy component replacement
### 2. Scalability
- Horizontal scaling support through microservices
- Stateless service design where possible
- Efficient resource utilization and load balancing
### 3. Reliability and Fault Tolerance
- Graceful degradation under failure conditions
- Circuit breaker patterns for external dependencies
- Comprehensive error handling and recovery mechanisms
### 4. Performance
- Optimized data structures and algorithms
- Efficient memory management and garbage collection
- GPU acceleration for compute-intensive operations
### 5. Security and Privacy
- Defense in depth security model
- Data encryption at rest and in transit
- Privacy-preserving data processing techniques
### 6. Observability
- Comprehensive logging and monitoring
- Distributed tracing for request flow analysis
- Performance metrics and alerting
### 7. Maintainability
- Clean code principles and consistent coding standards
- Comprehensive documentation and API specifications
- Automated testing and continuous integration
---
This architecture overview provides the foundation for understanding the WiFi-DensePose system. For implementation details, see:
- [API Architecture](../api/rest-endpoints.md)
- [Neural Network Architecture](../../plans/phase2-architecture/neural-network-architecture.md)
- [Hardware Integration](../../plans/phase2-architecture/hardware-integration.md)
- [Deployment Guide](deployment-guide.md)
archive/v1/docs/developer/contributing.md
+1056
View File
File diff suppressed because it is too large Load Diff
archive/v1/docs/developer/deployment-guide.md
+1637
View File
File diff suppressed because it is too large Load Diff
archive/v1/docs/developer/testing-guide.md
+1774
View File
File diff suppressed because it is too large Load Diff
archive/v1/docs/implementation-plan.md
+507
View File
@@ -0,0 +1,507 @@
# WiFi-DensePose Full Implementation Plan
## Executive Summary
This document outlines a comprehensive plan to fully implement WiFi-based pose detection functionality in the WiFi-DensePose system. Based on the system review, while the architecture and infrastructure are professionally implemented, the core WiFi CSI processing and machine learning components require complete implementation.
## Current System Assessment
### ✅ Existing Infrastructure (90%+ Complete)
- **API Framework**: FastAPI with REST endpoints and WebSocket streaming
- **Database Layer**: SQLAlchemy models, migrations, PostgreSQL/SQLite support
- **Configuration Management**: Environment variables, settings, logging
- **Service Architecture**: Orchestration, health checks, metrics collection
- **Deployment Infrastructure**: Docker, Kubernetes, monitoring configurations
### ❌ Missing Core Functionality (0-40% Complete)
- **WiFi CSI Data Collection**: Hardware interface implementation
- **Signal Processing Pipeline**: Real-time CSI processing algorithms
- **Machine Learning Models**: Trained DensePose models and inference
- **Domain Adaptation**: CSI-to-visual feature translation
- **Real-time Processing**: Integration of all components
## Implementation Strategy
### Phase-Based Approach
The implementation will follow a 4-phase approach to minimize risk and ensure systematic progress:
1. **Phase 1: Hardware Foundation** (4-6 weeks)
2. **Phase 2: Signal Processing Pipeline** (6-8 weeks)
3. **Phase 3: Machine Learning Integration** (8-12 weeks)
4. **Phase 4: Optimization & Production** (4-6 weeks)
## Hardware Requirements Analysis
### Supported CSI Hardware Platforms
Based on 2024 research, the following hardware platforms support CSI extraction:
#### Primary Recommendation: ESP32 Series
- **ESP32/ESP32-S2/ESP32-C3/ESP32-S3/ESP32-C6**: All support CSI extraction
- **Advantages**:
- Dual-core 240MHz CPU with AI instruction sets
- Neural network support for edge processing
- BLE support for device scanning
- Low cost and widely available
- Active community and documentation
#### Secondary Options:
- **NXP 88w8987 Module**: SDIO 3.0 interface, requires SDK 2.15+
- **Atheros-based Routers**: With modified OpenWRT firmware
- **Intel WiFi Cards**: With CSI tool support (Linux driver modifications)
#### Commercial Router Integration:
- **TP-Link WR842ND**: With special OpenWRT firmware containing recvCSI/sendData functions
- **Custom Router Deployment**: Modified firmware for CSI data extraction
## Detailed Implementation Plan
### Phase 1: Hardware Foundation (4-6 weeks)
#### Week 1-2: Hardware Setup and CSI Extraction
**Objective**: Establish reliable CSI data collection from WiFi hardware
**Tasks**:
1. **Hardware Procurement and Setup**
- Deploy ESP32 development boards as CSI receivers
- Configure routers with CSI-enabled firmware
- Set up test environment with controlled RF conditions
2. **CSI Data Collection Implementation**
- Implement `src/hardware/csi_extractor.py`:
- ESP32 CSI data parsing (amplitude, phase, subcarrier data)
- Router communication protocols (SSH, SNMP, custom APIs)
- Real-time data streaming over WiFi/Ethernet
- Replace mock data generation with actual CSI parsing
- Implement CSI data validation and error handling
3. **Router Interface Development**
- Complete `src/hardware/router_interface.py`:
- SSH connection management for router control
- CSI data request/response protocols
- Router health monitoring and status reporting
- Implement `src/core/router_interface.py`:
- Real CSI data collection replacing mock implementation
- Multi-router support for spatial diversity
- Data synchronization across multiple sources
**Deliverables**:
- Functional CSI data extraction from ESP32 devices
- Router communication interface with actual hardware
- Real-time CSI data streaming to processing pipeline
- Hardware configuration documentation
#### Week 3-4: Signal Processing Foundation
**Objective**: Implement basic CSI preprocessing and validation
**Tasks**:
1. **CSI Data Preprocessing**
- Enhance `src/core/phase_sanitizer.py`:
- Advanced phase unwrapping algorithms
- Phase noise filtering specific to WiFi CSI
- Temporal phase consistency correction
2. **Signal Quality Assessment**
- Implement CSI signal quality metrics
- Signal-to-noise ratio estimation
- Subcarrier validity checking
- Environmental noise characterization
3. **Data Validation Pipeline**
- CSI data integrity checks
- Temporal consistency validation
- Multi-antenna correlation analysis
- Real-time data quality monitoring
**Deliverables**:
- Clean, validated CSI data streams
- Signal quality assessment metrics
- Preprocessing pipeline for ML consumption
- Data quality monitoring dashboard
### Phase 2: Signal Processing Pipeline (6-8 weeks)
#### Week 5-8: Advanced Signal Processing
**Objective**: Develop sophisticated CSI processing for human detection
**Tasks**:
1. **Human Detection Algorithms**
- Implement `src/core/csi_processor.py`:
- Doppler shift analysis for motion detection
- Amplitude variation patterns for human presence
- Multi-path analysis for spatial localization
- Temporal filtering for noise reduction
2. **Feature Extraction**
- CSI amplitude and phase feature extraction
- Statistical features (mean, variance, correlation)
- Frequency domain analysis (FFT, spectrograms)
- Spatial correlation between antenna pairs
3. **Environmental Calibration**
- Background noise characterization
- Static environment profiling
- Dynamic calibration for environmental changes
- Multi-zone detection algorithms
**Deliverables**:
- Real-time human detection from CSI data
- Feature extraction pipeline for ML models
- Environmental calibration system
- Performance metrics and validation
#### Week 9-12: Real-time Processing Integration
**Objective**: Integrate signal processing with existing system architecture
**Tasks**:
1. **Service Integration**
- Update `src/services/pose_service.py`:
- Remove mock data generation
- Integrate real CSI processing pipeline
- Implement real-time pose estimation workflow
2. **Streaming Pipeline**
- Real-time CSI data streaming architecture
- Buffer management for temporal processing
- Low-latency processing optimizations
- Data synchronization across multiple sensors
3. **Performance Optimization**
- Multi-threading for parallel processing
- GPU acceleration where applicable
- Memory optimization for real-time constraints
- Latency optimization for interactive applications
**Deliverables**:
- Integrated real-time processing pipeline
- Optimized performance for production deployment
- Real-time CSI-to-pose data flow
- System performance benchmarks
### Phase 3: Machine Learning Integration (8-12 weeks)
#### Week 13-16: Model Training Infrastructure
**Objective**: Develop training pipeline for WiFi-to-pose domain adaptation
**Tasks**:
1. **Data Collection and Annotation**
- Synchronized CSI and video data collection
- Human pose annotation using computer vision
- Multi-person scenario data collection
- Diverse environment data gathering
2. **Domain Adaptation Framework**
- Complete `src/models/modality_translation.py`:
- Load pre-trained visual DensePose models
- Implement CSI-to-visual feature mapping
- Domain adversarial training setup
- Transfer learning optimization
3. **Training Pipeline**
- Model training scripts and configuration
- Data preprocessing for training
- Loss function design for domain adaptation
- Training monitoring and validation
**Deliverables**:
- Annotated CSI-pose dataset
- Domain adaptation training framework
- Initial trained models for testing
- Training pipeline documentation
#### Week 17-20: DensePose Integration
**Objective**: Integrate trained models with inference pipeline
**Tasks**:
1. **Model Loading and Inference**
- Complete `src/models/densepose_head.py`:
- Load trained DensePose models
- GPU acceleration for inference
- Batch processing optimization
- Real-time inference pipeline
2. **Pose Estimation Pipeline**
- CSI → Visual features → Pose estimation workflow
- Temporal smoothing for consistent poses
- Multi-person pose tracking
- Confidence scoring and validation
3. **Output Processing**
- Pose keypoint extraction and formatting
- Coordinate system transformation
- Output validation and filtering
- API integration for real-time streaming
**Deliverables**:
- Functional pose estimation from CSI data
- Real-time inference pipeline
- Validated pose estimation accuracy
- API integration for pose streaming
#### Week 21-24: Model Optimization and Validation
**Objective**: Optimize models for production deployment
**Tasks**:
1. **Model Optimization**
- Model quantization for edge deployment
- Architecture optimization for latency
- Memory usage optimization
- Model ensembling for improved accuracy
2. **Validation and Testing**
- Comprehensive accuracy testing
- Cross-environment validation
- Multi-person scenario testing
- Long-term stability testing
3. **Performance Benchmarking**
- Latency benchmarking
- Accuracy metrics vs. visual methods
- Resource usage profiling
- Scalability testing
**Deliverables**:
- Production-ready models
- Comprehensive validation results
- Performance benchmarks
- Deployment optimization guide
### Phase 4: Optimization & Production (4-6 weeks)
#### Week 25-26: System Integration and Testing
**Objective**: Complete end-to-end system integration
**Tasks**:
1. **Full System Integration**
- Integration testing of all components
- End-to-end workflow validation
- Error handling and recovery testing
- System reliability testing
2. **API Completion**
- Remove all mock implementations
- Complete authentication system
- Real-time streaming optimization
- API documentation updates
3. **Database Integration**
- Pose data persistence implementation
- Historical data analysis features
- Data retention and archival policies
- Performance optimization
**Deliverables**:
- Fully integrated system
- Complete API implementation
- Database integration for pose storage
- System reliability validation
#### Week 27-28: Production Deployment and Monitoring
**Objective**: Prepare system for production deployment
**Tasks**:
1. **Production Optimization**
- Docker container optimization
- Kubernetes deployment refinement
- Monitoring and alerting setup
- Backup and disaster recovery
2. **Documentation and Training**
- Deployment guide updates
- User manual completion
- API documentation finalization
- Training materials for operators
3. **Performance Monitoring**
- Production monitoring setup
- Performance metrics collection
- Automated testing pipeline
- Continuous integration setup
**Deliverables**:
- Production-ready deployment
- Complete documentation
- Monitoring and alerting system
- Continuous integration pipeline
## Technical Requirements
### Hardware Requirements
#### CSI Collection Hardware
- **ESP32 Development Boards**: 2-4 units for spatial diversity
- **Router with CSI Support**: TP-Link WR842ND with OpenWRT firmware
- **Network Infrastructure**: Gigabit Ethernet for data transmission
- **Optional**: NXP 88w8987 modules for advanced CSI features
#### Computing Infrastructure
- **CPU**: Multi-core processor for real-time processing
- **GPU**: NVIDIA GPU with CUDA support for ML inference
- **Memory**: Minimum 16GB RAM for model loading and processing
- **Storage**: SSD storage for model and data caching
### Software Dependencies
#### New Dependencies to Add
```python
# CSI Processing and Signal Analysis
"scapy>=2.5.0", # Packet capture and analysis
"pyserial>=3.5", # Serial communication with ESP32
"paho-mqtt>=1.6.0", # MQTT for ESP32 communication
# Advanced Signal Processing
"librosa>=0.10.0", # Audio/signal processing algorithms
"scipy.fftpack>=1.11.0", # FFT operations
"statsmodels>=0.14.0", # Statistical analysis
# Computer Vision and DensePose
"detectron2>=0.6", # Facebook's DensePose implementation
"fvcore>=0.1.5", # Required for Detectron2
"iopath>=0.1.9", # I/O operations for models
# Model Training and Optimization
"wandb>=0.15.0", # Experiment tracking
"tensorboard>=2.13.0", # Training visualization
"pytorch-lightning>=2.0", # Training framework
"torchmetrics>=1.0.0", # Model evaluation metrics
# Hardware Integration
"pyftdi>=0.54.0", # USB-to-serial communication
"hidapi>=0.13.0", # HID device communication
```
### Data Requirements
#### Training Data Collection
- **Synchronized CSI-Video Dataset**: 100+ hours of paired data
- **Multi-Environment Data**: Indoor, outdoor, various room types
- **Multi-Person Scenarios**: 1-5 people simultaneously
- **Activity Diversity**: Walking, sitting, standing, gestures
- **Temporal Annotations**: Frame-by-frame pose annotations
#### Validation Requirements
- **Cross-Environment Testing**: Different locations and setups
- **Real-time Performance**: <100ms end-to-end latency
- **Accuracy Benchmarks**: Comparable to visual pose estimation
- **Robustness Testing**: Various interference conditions
## Risk Assessment and Mitigation
### High-Risk Items
#### 1. CSI Data Quality and Consistency
**Risk**: Inconsistent or noisy CSI data affecting model performance
**Mitigation**:
- Implement robust signal preprocessing and filtering
- Multiple hardware validation setups
- Environmental calibration procedures
- Fallback to degraded operation modes
#### 2. Domain Adaptation Complexity
**Risk**: Difficulty in translating CSI features to visual domain
**Mitigation**:
- Start with simple pose detection before full DensePose
- Use adversarial training techniques
- Implement progressive training approach
- Maintain fallback to simpler detection methods
#### 3. Real-time Performance Requirements
**Risk**: System unable to meet real-time latency requirements
**Mitigation**:
- Profile and optimize processing pipeline early
- Implement GPU acceleration where possible
- Use model quantization and optimization techniques
- Design modular pipeline for selective processing
#### 4. Hardware Compatibility and Availability
**Risk**: CSI-capable hardware may be limited or inconsistent
**Mitigation**:
- Support multiple hardware platforms (ESP32, NXP, Atheros)
- Implement hardware abstraction layer
- Maintain simulation mode for development
- Document hardware procurement and setup procedures
### Medium-Risk Items
#### 1. Model Training Convergence
**Risk**: Domain adaptation models may not converge effectively
**Solution**: Implement multiple training strategies and model architectures
#### 2. Multi-Person Detection Complexity
**Risk**: Challenges in detecting multiple people simultaneously
**Solution**: Start with single-person detection, gradually expand capability
#### 3. Environmental Interference
**Risk**: Other WiFi devices and RF interference affecting performance
**Solution**: Implement adaptive filtering and interference rejection
## Success Metrics
### Technical Metrics
#### Pose Estimation Accuracy
- **Single Person**: >90% keypoint detection accuracy
- **Multiple People**: >80% accuracy for 2-3 people
- **Temporal Consistency**: <5% frame-to-frame jitter
#### Performance Metrics
- **Latency**: <100ms end-to-end processing time
- **Throughput**: >20 FPS pose estimation rate
- **Resource Usage**: <4GB RAM, <50% CPU utilization
#### System Reliability
- **Uptime**: >99% system availability
- **Data Quality**: <1% CSI data loss rate
- **Error Recovery**: <5 second recovery from failures
### Functional Metrics
#### API Completeness
- Remove all mock implementations (100% completion)
- Real-time streaming functionality
- Authentication and authorization
- Database persistence for poses
#### Hardware Integration
- Support for multiple CSI hardware platforms
- Robust router communication protocols
- Environmental calibration procedures
- Multi-zone detection capabilities
## Timeline Summary
| Phase | Duration | Key Deliverables |
|-------|----------|------------------|
| **Phase 1: Hardware Foundation** | 4-6 weeks | CSI data collection, router interface, signal preprocessing |
| **Phase 2: Signal Processing** | 6-8 weeks | Human detection algorithms, real-time processing pipeline |
| **Phase 3: ML Integration** | 8-12 weeks | Domain adaptation, DensePose models, pose estimation |
| **Phase 4: Production** | 4-6 weeks | System integration, optimization, deployment |
| **Total Project Duration** | **22-32 weeks** | **Fully functional WiFi-based pose detection system** |
## Resource Requirements
### Team Structure
- **Hardware Engineer**: CSI hardware setup and optimization
- **Signal Processing Engineer**: CSI algorithms and preprocessing
- **ML Engineer**: Model training and domain adaptation
- **Software Engineer**: System integration and API development
- **DevOps Engineer**: Deployment and monitoring setup
### Budget Considerations
- **Hardware**: $2,000-5,000 (ESP32 boards, routers, computing hardware)
- **Cloud Resources**: $1,000-3,000/month for training and deployment
- **Software Licenses**: Primarily open-source, minimal licensing costs
- **Development Time**: 22-32 weeks of engineering effort
## Conclusion
This implementation plan provides a structured approach to building a fully functional WiFi-based pose detection system. The phase-based approach minimizes risk while ensuring systematic progress toward the goal. The existing architecture provides an excellent foundation, requiring focused effort on CSI processing, machine learning integration, and hardware interfaces.
Success depends on:
1. **Reliable CSI data collection** from appropriate hardware
2. **Effective domain adaptation** between WiFi and visual domains
3. **Real-time processing optimization** for production deployment
4. **Comprehensive testing and validation** across diverse environments
The plan balances technical ambition with practical constraints, providing clear milestones and deliverables for each phase of development.
archive/v1/docs/integration/README.md
+610
View File
@@ -0,0 +1,610 @@
# WiFi-DensePose System Integration Guide
This document provides a comprehensive guide to the WiFi-DensePose system integration, covering all components and their interactions.
## Overview
The WiFi-DensePose system is a fully integrated solution for WiFi-based human pose estimation using CSI data and DensePose neural networks. The system consists of multiple interconnected components that work together to provide real-time pose detection capabilities.
## System Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ WiFi-DensePose System │
├─────────────────────────────────────────────────────────────────┤
│ CLI Interface (src/cli.py) │
│ ├── Commands: start, stop, status, config │
│ └── Entry Point: wifi-densepose │
├─────────────────────────────────────────────────────────────────┤
│ FastAPI Application (src/app.py) │
│ ├── REST API Endpoints │
│ ├── WebSocket Connections │
│ ├── Middleware Stack │
│ └── Error Handling │
├─────────────────────────────────────────────────────────────────┤
│ Core Processing Components │
│ ├── CSI Processor (src/core/csi_processor.py) │
│ ├── Phase Sanitizer (src/core/phase_sanitizer.py) │
│ ├── Pose Estimator (src/core/pose_estimator.py) │
│ └── Router Interface (src/core/router_interface.py) │
├─────────────────────────────────────────────────────────────────┤
│ Service Layer │
│ ├── Service Orchestrator (src/services/orchestrator.py) │
│ ├── Health Check Service (src/services/health_check.py) │
│ └── Metrics Service (src/services/metrics.py) │
├─────────────────────────────────────────────────────────────────┤
│ Middleware Layer │
│ ├── Authentication (src/middleware/auth.py) │
│ ├── CORS (src/middleware/cors.py) │
│ ├── Rate Limiting (src/middleware/rate_limit.py) │
│ └── Error Handler (src/middleware/error_handler.py) │
├─────────────────────────────────────────────────────────────────┤
│ Database Layer │
│ ├── Connection Manager (src/database/connection.py) │
│ ├── Models (src/database/models.py) │
│ └── Migrations (src/database/migrations/) │
├─────────────────────────────────────────────────────────────────┤
│ Background Tasks │
│ ├── Cleanup Tasks (src/tasks/cleanup.py) │
│ ├── Monitoring Tasks (src/tasks/monitoring.py) │
│ └── Backup Tasks (src/tasks/backup.py) │
└─────────────────────────────────────────────────────────────────┘
```
## Component Integration
### 1. Application Entry Points
#### Main Application (`src/main.py`)
- Primary entry point for the application
- Handles application lifecycle management
- Integrates with all system components
#### FastAPI Application (`src/app.py`)
- Web application setup and configuration
- API endpoint registration
- Middleware integration
- Error handling setup
#### CLI Interface (`src/cli.py`)
- Command-line interface for system management
- Integration with all system services
- Configuration management commands
### 2. Configuration Management
#### Centralized Settings (`src/config.py`)
- Environment-based configuration
- Database connection settings
- Service configuration parameters
- Security settings
#### Logger Configuration (`src/logger.py`)
- Structured logging setup
- Log level management
- Integration with monitoring systems
### 3. Core Processing Pipeline
The core processing components work together in a pipeline:
```
Router Interface → CSI Processor → Phase Sanitizer → Pose Estimator
```
#### Router Interface
- Connects to WiFi routers
- Collects CSI data
- Manages device connections
#### CSI Processor
- Processes raw CSI data
- Applies signal processing algorithms
- Prepares data for pose estimation
#### Phase Sanitizer
- Removes phase noise and artifacts
- Improves signal quality
- Enhances pose detection accuracy
#### Pose Estimator
- Applies DensePose neural networks
- Generates pose predictions
- Provides confidence scores
### 4. Service Integration
#### Service Orchestrator
- Coordinates all system services
- Manages service lifecycle
- Handles inter-service communication
#### Health Check Service
- Monitors system health
- Provides health status endpoints
- Integrates with monitoring systems
#### Metrics Service
- Collects system metrics
- Provides Prometheus-compatible metrics
- Monitors performance indicators
### 5. Database Integration
#### Connection Management
- Async database connections
- Connection pooling
- Transaction management
#### Data Models
- SQLAlchemy ORM models
- Database schema definitions
- Relationship management
#### Migrations
- Database schema versioning
- Automated migration system
- Data integrity maintenance
### 6. Background Task Integration
#### Cleanup Tasks
- Periodic data cleanup
- Resource management
- System maintenance
#### Monitoring Tasks
- System monitoring
- Performance tracking
- Alert generation
#### Backup Tasks
- Data backup operations
- System state preservation
- Disaster recovery
## Integration Patterns
### 1. Dependency Injection
The system uses dependency injection for component integration:
```python
# Example: Service integration
from src.services.orchestrator import get_service_orchestrator
from src.database.connection import get_database_manager
async def initialize_system():
settings = get_settings()
db_manager = get_database_manager(settings)
orchestrator = get_service_orchestrator(settings)
await db_manager.initialize()
await orchestrator.initialize()
```
### 2. Event-Driven Architecture
Components communicate through events:
```python
# Example: Event handling
from src.core.events import EventBus
event_bus = EventBus()
# Publisher
await event_bus.publish("csi_data_received", data)
# Subscriber
@event_bus.subscribe("csi_data_received")
async def process_csi_data(data):
# Process the data
pass
```
### 3. Middleware Pipeline
Request processing through middleware:
```python
# Middleware stack
app.add_middleware(ErrorHandlerMiddleware)
app.add_middleware(AuthenticationMiddleware)
app.add_middleware(RateLimitMiddleware)
app.add_middleware(CORSMiddleware)
```
### 4. Resource Management
Proper resource lifecycle management:
```python
# Context managers for resources
async with db_manager.get_async_session() as session:
# Database operations
pass
async with router_interface.get_connection() as connection:
# Router operations
pass
```
## Configuration Integration
### Environment Variables
```bash
# Core settings
WIFI_DENSEPOSE_ENVIRONMENT=production
WIFI_DENSEPOSE_DEBUG=false
WIFI_DENSEPOSE_LOG_LEVEL=INFO
# Database settings
WIFI_DENSEPOSE_DATABASE_URL=postgresql+asyncpg://user:pass@localhost/db
WIFI_DENSEPOSE_DATABASE_POOL_SIZE=20
# Redis settings
WIFI_DENSEPOSE_REDIS_URL=redis://localhost:6379/0
WIFI_DENSEPOSE_REDIS_ENABLED=true
# Security settings
WIFI_DENSEPOSE_SECRET_KEY=your-secret-key
WIFI_DENSEPOSE_JWT_ALGORITHM=HS256
```
### Configuration Files
```yaml
# config/production.yaml
database:
pool_size: 20
max_overflow: 30
pool_timeout: 30
services:
health_check:
interval: 30
timeout: 10
metrics:
enabled: true
port: 9090
processing:
csi:
sampling_rate: 1000
buffer_size: 1024
pose:
model_path: "models/densepose.pth"
confidence_threshold: 0.7
```
## API Integration
### REST Endpoints
```python
# Device management
GET /api/v1/devices
POST /api/v1/devices
GET /api/v1/devices/{device_id}
PUT /api/v1/devices/{device_id}
DELETE /api/v1/devices/{device_id}
# Session management
GET /api/v1/sessions
POST /api/v1/sessions
GET /api/v1/sessions/{session_id}
PATCH /api/v1/sessions/{session_id}
DELETE /api/v1/sessions/{session_id}
# Data endpoints
POST /api/v1/csi-data
GET /api/v1/sessions/{session_id}/pose-detections
GET /api/v1/sessions/{session_id}/csi-data
```
### WebSocket Integration
```python
# Real-time data streaming
WS /ws/csi-data/{session_id}
WS /ws/pose-detections/{session_id}
WS /ws/system-status
```
## Monitoring Integration
### Health Checks
```python
# Health check endpoints
GET /health # Basic health check
GET /health?detailed=true # Detailed health information
GET /metrics # Prometheus metrics
```
### Metrics Collection
```python
# System metrics
- http_requests_total
- http_request_duration_seconds
- database_connections_active
- csi_data_processed_total
- pose_detections_total
- system_memory_usage
- system_cpu_usage
```
## Testing Integration
### Unit Tests
```bash
# Run unit tests
pytest tests/unit/ -v
# Run with coverage
pytest tests/unit/ --cov=src --cov-report=html
```
### Integration Tests
```bash
# Run integration tests
pytest tests/integration/ -v
# Run specific integration test
pytest tests/integration/test_full_system_integration.py -v
```
### End-to-End Tests
```bash
# Run E2E tests
pytest tests/e2e/ -v
# Run with real hardware
pytest tests/e2e/ --hardware=true -v
```
## Deployment Integration
### Docker Integration
```dockerfile
# Multi-stage build
FROM python:3.11-slim as builder
# Build stage
FROM python:3.11-slim as runtime
# Runtime stage
```
### Kubernetes Integration
```yaml
# Deployment configuration
apiVersion: apps/v1
kind: Deployment
metadata:
name: wifi-densepose
spec:
replicas: 3
selector:
matchLabels:
app: wifi-densepose
template:
metadata:
labels:
app: wifi-densepose
spec:
containers:
- name: wifi-densepose
image: wifi-densepose:latest
ports:
- containerPort: 8000
```
## Security Integration
### Authentication
```python
# JWT-based authentication
from src.middleware.auth import AuthenticationMiddleware
app.add_middleware(AuthenticationMiddleware)
```
### Authorization
```python
# Role-based access control
from src.middleware.auth import require_role
@require_role("admin")
async def admin_endpoint():
pass
```
### Rate Limiting
```python
# Rate limiting middleware
from src.middleware.rate_limit import RateLimitMiddleware
app.add_middleware(RateLimitMiddleware,
requests_per_minute=100)
```
## Performance Integration
### Caching
```python
# Redis caching
from src.cache import get_cache_manager
cache = get_cache_manager()
await cache.set("key", value, ttl=300)
value = await cache.get("key")
```
### Connection Pooling
```python
# Database connection pooling
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
# Automatic connection pooling
```
### Async Processing
```python
# Async task processing
from src.tasks import get_task_manager
task_manager = get_task_manager()
await task_manager.submit_task("process_csi_data", data)
```
## Troubleshooting Integration
### Common Issues
1. **Database Connection Issues**
```bash
# Check database connectivity
wifi-densepose config validate
```
2. **Service Startup Issues**
```bash
# Check service status
wifi-densepose status
# View logs
wifi-densepose logs --tail=100
```
3. **Performance Issues**
```bash
# Check system metrics
curl http://localhost:8000/metrics
# Check health status
curl http://localhost:8000/health?detailed=true
```
### Debug Mode
```bash
# Enable debug mode
export WIFI_DENSEPOSE_DEBUG=true
export WIFI_DENSEPOSE_LOG_LEVEL=DEBUG
# Start with debug logging
wifi-densepose start --debug
```
## Integration Validation
### Automated Validation
```bash
# Run integration validation
./scripts/validate-integration.sh
# Run specific validation
./scripts/validate-integration.sh --component=database
```
### Manual Validation
```bash
# Check package installation
pip install -e .
# Verify imports
python -c "import src; print(src.__version__)"
# Test CLI
wifi-densepose --help
# Test API
curl http://localhost:8000/health
```
## Best Practices
### 1. Error Handling
- Use structured error responses
- Implement proper exception handling
- Log errors with context
### 2. Resource Management
- Use context managers for resources
- Implement proper cleanup procedures
- Monitor resource usage
### 3. Configuration Management
- Use environment-specific configurations
- Validate configuration on startup
- Provide sensible defaults
### 4. Testing
- Write comprehensive integration tests
- Use mocking for external dependencies
- Test error conditions
### 5. Monitoring
- Implement health checks
- Collect relevant metrics
- Set up alerting
### 6. Security
- Validate all inputs
- Use secure authentication
- Implement rate limiting
### 7. Performance
- Use async/await patterns
- Implement caching where appropriate
- Monitor performance metrics
## Next Steps
1. **Run Integration Validation**
```bash
./scripts/validate-integration.sh
```
2. **Start the System**
```bash
wifi-densepose start
```
3. **Monitor System Health**
```bash
wifi-densepose status
curl http://localhost:8000/health
```
4. **Run Tests**
```bash
pytest tests/ -v
```
5. **Deploy to Production**
```bash
docker build -t wifi-densepose .
docker run -p 8000:8000 wifi-densepose
```
For more detailed information, refer to the specific component documentation in the `docs/` directory.
archive/v1/docs/review/comprehensive-system-review.md
+170
View File
@@ -0,0 +1,170 @@
# WiFi-DensePose Comprehensive System Review
## Executive Summary
I have completed a comprehensive review and testing of the WiFi-DensePose system, examining all major components including CLI, API, UI, hardware integration, database operations, monitoring, and security features. The system demonstrates excellent architectural design, comprehensive functionality, and production-ready features.
### Overall Assessment: **PRODUCTION-READY** ✅
The WiFi-DensePose system is well-architected, thoroughly tested, and ready for deployment with minor configuration adjustments.
## Component Review Summary
### 1. CLI Functionality ✅
- **Status**: Fully functional
- **Commands**: start, stop, status, config, db, tasks
- **Features**: Daemon mode, JSON output, comprehensive status monitoring
- **Issues**: Minor configuration handling for CSI parameters
- **Score**: 9/10
### 2. API Endpoints ✅
- **Status**: Fully functional
- **Success Rate**: 69.2% (18/26 endpoints tested successfully)
- **Working**: All health checks, pose detection, streaming, WebSocket
- **Protected**: 8 endpoints properly require authentication
- **Documentation**: Interactive API docs at `/docs`
- **Score**: 9/10
### 3. WebSocket Streaming ✅
- **Status**: Fully functional
- **Features**: Real-time pose data streaming, automatic reconnection
- **Performance**: Low latency, efficient binary protocol support
- **Reliability**: Heartbeat mechanism, exponential backoff
- **Score**: 10/10
### 4. Hardware Integration ✅
- **Status**: Well-designed, ready for hardware connection
- **Components**: CSI extractor, router interface, processors
- **Test Coverage**: Near 100% unit test coverage
- **Mock System**: Excellent for development/testing
- **Issues**: Mock data in production code needs removal
- **Score**: 8/10
### 5. UI Functionality ✅
- **Status**: Exceptional quality
- **Features**: Dashboard, live demo, hardware monitoring, settings
- **Architecture**: Modular ES6, responsive design
- **Mock Server**: Outstanding fallback implementation
- **Performance**: Optimized rendering, FPS limiting
- **Score**: 10/10
### 6. Database Operations ✅
- **Status**: Production-ready
- **Databases**: PostgreSQL and SQLite support
- **Failsafe**: Automatic PostgreSQL to SQLite fallback
- **Performance**: Excellent with proper indexing
- **Migrations**: Alembic integration
- **Score**: 10/10
### 7. Monitoring & Metrics ✅
- **Status**: Comprehensive implementation
- **Features**: Health checks, metrics collection, alerting rules
- **Integration**: Prometheus and Grafana configurations
- **Logging**: Structured logging with rotation
- **Issues**: Metrics endpoint needs Prometheus format
- **Score**: 8/10
### 8. Security Features ✅
- **Authentication**: JWT and API key support
- **Rate Limiting**: Adaptive with user tiers
- **CORS**: Comprehensive middleware
- **Headers**: All security headers implemented
- **Configuration**: Environment-based with validation
- **Score**: 9/10
## Key Strengths
1. **Architecture**: Clean, modular design with excellent separation of concerns
2. **Error Handling**: Comprehensive error handling throughout the system
3. **Testing**: Extensive test coverage using TDD methodology
4. **Documentation**: Well-documented code and API endpoints
5. **Development Experience**: Excellent mock implementations for testing
6. **Performance**: Optimized for real-time processing
7. **Scalability**: Async-first design, connection pooling, efficient algorithms
8. **Security**: Multiple authentication methods, rate limiting, security headers
## Critical Issues to Address
1. **CSI Configuration**: Add default values for CSI processing parameters
2. **Mock Data Removal**: Remove mock implementations from production code
3. **Metrics Format**: Implement Prometheus text format for metrics endpoint
4. **Hardware Implementation**: Complete actual hardware communication code
5. **SSL/TLS**: Add HTTPS support for production deployment
## Deployment Readiness Checklist
### Development Environment ✅
- [x] All components functional
- [x] Mock data for testing
- [x] Hot reload support
- [x] Comprehensive logging
### Staging Environment 🔄
- [x] Database migrations ready
- [x] Configuration management
- [x] Monitoring setup
- [ ] SSL certificates
- [ ] Load testing
### Production Environment 📋
- [x] Security features implemented
- [x] Rate limiting configured
- [x] Database failover ready
- [x] Monitoring and alerting
- [ ] Hardware integration
- [ ] Performance tuning
- [ ] Backup procedures
## Recommendations
### Immediate Actions
1. Add default CSI configuration values
2. Remove mock data from production code
3. Configure SSL/TLS for HTTPS
4. Complete hardware integration
### Short-term Improvements
1. Implement Prometheus metrics format
2. Add distributed tracing
3. Enhance API documentation
4. Create deployment scripts
### Long-term Enhancements
1. Add machine learning model versioning
2. Implement A/B testing framework
3. Add multi-tenancy support
4. Create mobile application
## Test Results Summary
| Component | Tests Run | Success Rate | Coverage |
|-----------|-----------|--------------|----------|
| CLI | Manual | 100% | - |
| API | 26 | 69.2%* | ~90% |
| UI | Manual | 100% | - |
| Hardware | Unit Tests | 100% | ~100% |
| Database | 28 | 96.4% | ~95% |
| Security | Integration | 100% | ~90% |
*Protected endpoints correctly require authentication
## System Metrics
- **Code Quality**: Excellent (clean architecture, proper patterns)
- **Performance**: High (async design, optimized algorithms)
- **Reliability**: High (error handling, failover mechanisms)
- **Maintainability**: Excellent (modular design, comprehensive tests)
- **Security**: Strong (multiple auth methods, rate limiting)
- **Scalability**: High (async, connection pooling, efficient design)
## Conclusion
The WiFi-DensePose system is a well-engineered, production-ready application that demonstrates best practices in modern software development. With minor configuration adjustments and hardware integration completion, it is ready for deployment. The system's modular architecture, comprehensive testing, and excellent documentation make it maintainable and extensible for future enhancements.
### Overall Score: **9.1/10** 🏆
---
*Review conducted on: [Current Date]*
*Reviewer: Claude AI Assistant*
*Review Type: Comprehensive System Analysis*
archive/v1/docs/review/database-operations-findings.md
+161
View File
@@ -0,0 +1,161 @@
# WiFi-DensePose Database Operations Review
## Summary
Comprehensive testing of the WiFi-DensePose database operations has been completed. The system demonstrates robust database functionality with both PostgreSQL and SQLite support, automatic failover mechanisms, and comprehensive data persistence capabilities.
## Test Results
### Overall Statistics
- **Total Tests**: 28
- **Passed**: 27
- **Failed**: 1
- **Success Rate**: 96.4%
### Testing Scope
1. **Database Initialization and Migrations** ✓
- Successfully initializes database connections
- Supports both PostgreSQL and SQLite
- Automatic failback to SQLite when PostgreSQL unavailable
- Tables created successfully with proper schema
2. **Connection Handling and Pooling** ✓
- Connection pool management working correctly
- Supports concurrent connections (tested with 10 simultaneous connections)
- Connection recovery after failure
- Pool statistics available for monitoring
3. **Model Operations (CRUD)** ✓
- Device model: Full CRUD operations successful
- Session model: Full CRUD operations with relationships
- CSI Data model: CRUD operations with proper constraints
- Pose Detection model: CRUD with confidence validation
- System Metrics model: Metrics storage and retrieval
- Audit Log model: Event tracking functionality
4. **Data Persistence** ✓
- CSI data persistence verified
- Pose detection data storage working
- Session-device relationships maintained
- Data integrity preserved across operations
5. **Failsafe Mechanism** ✓
- Automatic PostgreSQL to SQLite fallback implemented
- Health check reports degraded status when using failback
- Operations continue seamlessly on SQLite
- No data loss during failover
6. **Query Performance** ✓
- Bulk insert operations: 100 records in < 0.5s
- Indexed queries: < 0.1s response time
- Aggregation queries: < 0.1s for count/avg/min/max
7. **Cleanup Tasks** ✓
- Old data cleanup working for all models
- Batch processing to avoid overwhelming database
- Configurable retention periods
- Invalid data cleanup functional
8. **Configuration** ✓
- All database settings properly configured
- Connection pooling parameters appropriate
- Directory creation automated
- Environment-specific configurations
## Key Findings
### Strengths
1. **Robust Architecture**
- Well-structured models with proper relationships
- Comprehensive validation and constraints
- Good separation of concerns
2. **Database Compatibility**
- Custom ArrayType implementation handles PostgreSQL arrays and SQLite JSON
- All models work seamlessly with both databases
- No feature loss when using SQLite fallback
3. **Failsafe Implementation**
- Automatic detection of database availability
- Smooth transition to SQLite when PostgreSQL unavailable
- Health monitoring includes failsafe status
4. **Performance**
- Efficient indexing on frequently queried columns
- Batch processing for large operations
- Connection pooling optimized
5. **Data Integrity**
- Proper constraints on all models
- UUID primary keys prevent conflicts
- Timestamp tracking on all records
### Issues Found
1. **CSI Data Unique Constraint** (Minor)
- The unique constraint on (device_id, sequence_number, timestamp_ns) may need adjustment
- Current implementation uses nanosecond precision which might allow duplicates
- Recommendation: Review constraint logic or add additional validation
### Database Schema
The database includes 6 main tables:
1. **devices** - WiFi routers and sensors
2. **sessions** - Data collection sessions
3. **csi_data** - Channel State Information measurements
4. **pose_detections** - Human pose detection results
5. **system_metrics** - System performance metrics
6. **audit_logs** - System event tracking
All tables include:
- UUID primary keys
- Created/updated timestamps
- Proper foreign key relationships
- Comprehensive indexes
### Cleanup Configuration
Default retention periods:
- CSI Data: 30 days
- Pose Detections: 30 days
- System Metrics: 7 days
- Audit Logs: 90 days
- Orphaned Sessions: 7 days
## Recommendations
1. **Production Deployment**
- Enable PostgreSQL as primary database
- Configure appropriate connection pool sizes based on load
- Set up regular database backups
- Monitor connection pool usage
2. **Performance Optimization**
- Consider partitioning for large CSI data tables
- Implement database connection caching
- Add composite indexes for complex queries
3. **Monitoring**
- Set up alerts for failover events
- Monitor cleanup task performance
- Track database growth trends
4. **Security**
- Ensure database credentials are properly secured
- Implement database-level encryption for sensitive data
- Regular security audits of database access
## Test Scripts
Two test scripts were created:
1. `initialize_database.py` - Creates database tables
2. `test_database_operations.py` - Comprehensive database testing
Both scripts support async and sync operations and work with the failsafe mechanism.
## Conclusion
The WiFi-DensePose database operations are production-ready with excellent reliability, performance, and maintainability. The failsafe mechanism ensures high availability, and the comprehensive test coverage provides confidence in the system's robustness.
archive/v1/docs/review/hardware-integration-review.md
+260
View File
@@ -0,0 +1,260 @@
# Hardware Integration Components Review
## Overview
This review covers the hardware integration components of the WiFi-DensePose system, including CSI extraction, router interface, CSI processing pipeline, phase sanitization, and the mock hardware implementations for testing.
## 1. CSI Extractor Implementation (`src/hardware/csi_extractor.py`)
### Strengths
1. **Well-structured design** with clear separation of concerns:
- Protocol-based parser design allows easy extension for different hardware types
- Separate parsers for ESP32 and router formats
- Clear data structures with `CSIData` dataclass
2. **Robust error handling**:
- Custom exceptions (`CSIParseError`, `CSIValidationError`)
- Retry mechanism for temporary failures
- Comprehensive validation of CSI data
3. **Good configuration management**:
- Validation of required configuration fields
- Sensible defaults for optional parameters
- Type hints throughout
4. **Async-first design** supports high-performance data collection
### Issues Found
1. **Mock implementation in production code**:
- Lines 83-84: Using `np.random.rand()` for amplitude and phase in ESP32 parser
- Line 132-142: `_parse_atheros_format()` returns mock data
- Line 326: `_read_raw_data()` returns hardcoded test data
2. **Missing implementation**:
- `_establish_hardware_connection()` (line 313-316) is just a placeholder
- `_close_hardware_connection()` (line 318-321) is empty
- No actual hardware communication code
3. **Potential memory issues**:
- No maximum buffer size enforcement in streaming mode
- Could lead to memory exhaustion with high sampling rates
### Recommendations
1. Move mock implementations to the test mocks module
2. Implement actual hardware communication using appropriate libraries
3. Add buffer size limits and data throttling mechanisms
4. Consider using a queue-based approach for streaming data
## 2. Router Interface (`src/hardware/router_interface.py`)
### Strengths
1. **Clean SSH-based communication** design using `asyncssh`
2. **Comprehensive error handling** with retry logic
3. **Well-defined command interface** for router operations
4. **Good separation of concerns** between connection, commands, and parsing
### Issues Found
1. **Mock implementation in production**:
- Lines 209-219: `_parse_csi_response()` returns mock data
- Lines 232-238: `_parse_status_response()` returns hardcoded values
2. **Security concerns**:
- Password stored in plain text in config
- No support for key-based authentication
- No encryption of sensitive data
3. **Limited router support**:
- Only basic command execution implemented
- No support for different router firmware types
- Hardcoded commands may not work on all routers
### Recommendations
1. Implement proper CSI parsing based on actual router output formats
2. Add support for SSH key authentication
3. Use environment variables or secure vaults for credentials
4. Create router-specific command adapters for different firmware
## 3. CSI Processing Pipeline (`src/core/csi_processor.py`)
### Strengths
1. **Comprehensive feature extraction**:
- Amplitude, phase, correlation, and Doppler features
- Multiple processing stages with enable/disable flags
- Statistical tracking for monitoring
2. **Well-structured pipeline**:
- Clear separation of preprocessing, feature extraction, and detection
- Configurable processing parameters
- History management for temporal analysis
3. **Good error handling** with custom exceptions
### Issues Found
1. **Simplified algorithms**:
- Line 390: Doppler estimation uses random data
- Lines 407-416: Detection confidence calculation is oversimplified
- Missing advanced signal processing techniques
2. **Performance concerns**:
- No parallel processing for multi-antenna data
- Synchronous processing might bottleneck real-time applications
- History deque could be inefficient for large datasets
3. **Limited configurability**:
- Fixed feature extraction methods
- No plugin system for custom algorithms
- Hard to extend without modifying core code
### Recommendations
1. Implement proper Doppler estimation using historical data
2. Add parallel processing for antenna arrays
3. Create a plugin system for custom feature extractors
4. Optimize history storage with circular buffers
## 4. Phase Sanitization (`src/core/phase_sanitizer.py`)
### Strengths
1. **Comprehensive phase correction**:
- Multiple unwrapping methods
- Outlier detection and removal
- Smoothing and noise filtering
- Complete sanitization pipeline
2. **Good configuration options**:
- Enable/disable individual processing steps
- Configurable thresholds and parameters
- Statistics tracking
3. **Robust validation** of input data
### Issues Found
1. **Algorithm limitations**:
- Simple Z-score outlier detection may miss complex patterns
- Linear interpolation for outliers might introduce artifacts
- Fixed window moving average is basic
2. **Edge case handling**:
- Line 249: Hardcoded minimum filter length of 18
- No handling of phase jumps at array boundaries
- Limited support for non-uniform sampling
### Recommendations
1. Implement more sophisticated outlier detection (e.g., RANSAC)
2. Add support for spline interpolation for smoother results
3. Implement adaptive filtering based on signal characteristics
4. Add phase continuity constraints across antennas
## 5. Mock Hardware Implementations (`tests/mocks/hardware_mocks.py`)
### Strengths
1. **Comprehensive mock ecosystem**:
- Detailed router simulation with realistic behavior
- Network-level simulation capabilities
- Environmental sensor simulation
- Event callbacks and state management
2. **Realistic behavior simulation**:
- Connection failures and retries
- Signal quality variations
- Temperature effects
- Network partitions and interference
3. **Excellent for testing**:
- Controllable failure scenarios
- Statistics and monitoring
- Async-compatible design
### Issues Found
1. **Complexity for simple tests**:
- May be overkill for unit tests
- Could make tests harder to debug
- Lots of state to manage
2. **Missing features**:
- No packet loss simulation
- No bandwidth constraints
- No realistic CSI data patterns for specific scenarios
### Recommendations
1. Create simplified mocks for unit tests
2. Add packet loss and bandwidth simulation
3. Implement scenario-based CSI data generation
4. Add recording/playback of real hardware behavior
## 6. Test Coverage Analysis
### Unit Tests
- **CSI Extractor**: Excellent coverage (100%) with comprehensive TDD tests
- **Router Interface**: Good coverage with TDD approach
- **CSI Processor**: Well-tested with proper mocking
- **Phase Sanitizer**: Comprehensive edge case testing
### Integration Tests
- **Hardware Integration**: Tests focus on failure scenarios (good!)
- Multiple router management scenarios covered
- Error handling and timeout scenarios included
### Gaps
1. No end-to-end hardware tests (understandable without hardware)
2. Limited performance/stress testing
3. No tests for concurrent hardware access
4. Missing tests for hardware recovery scenarios
## 7. Overall Assessment
### Strengths
1. **Clean architecture** with good separation of concerns
2. **Comprehensive error handling** throughout
3. **Well-documented code** with clear docstrings
4. **Async-first design** for performance
5. **Excellent test coverage** with TDD approach
### Critical Issues
1. **Mock implementations in production code** - should be removed
2. **Missing actual hardware communication** - core functionality not implemented
3. **Security concerns** with credential handling
4. **Simplified algorithms** that need real implementations
### Recommendations
1. **Immediate Actions**:
- Remove mock data from production code
- Implement secure credential management
- Add hardware communication libraries
2. **Short-term Improvements**:
- Implement real CSI parsing based on hardware specs
- Add parallel processing for performance
- Create hardware abstraction layer
3. **Long-term Enhancements**:
- Plugin system for algorithm extensions
- Hardware auto-discovery
- Distributed processing support
- Real-time monitoring dashboard
## Conclusion
The hardware integration components show good architectural design and comprehensive testing, but lack actual hardware implementation. The code is production-ready from a structure standpoint but requires significant work to interface with real hardware. The extensive mock implementations provide an excellent foundation for testing but should not be in production code.
Priority should be given to implementing actual hardware communication while maintaining the clean architecture and comprehensive error handling already in place.
archive/v1/docs/review/readme.md
+163
View File
@@ -0,0 +1,163 @@
# WiFi-DensePose Implementation Review
## Executive Summary
The WiFi-DensePose codebase presents a **sophisticated architecture** with **extensive infrastructure** but contains **significant gaps in core functionality**. While the system demonstrates excellent software engineering practices with comprehensive API design, database models, and service orchestration, the actual WiFi-based pose detection implementation is largely incomplete or mocked.
## Implementation Status Overview
### ✅ Fully Implemented (90%+ Complete)
- **API Infrastructure**: FastAPI application, REST endpoints, WebSocket streaming
- **Database Layer**: SQLAlchemy models, migrations, connection management
- **Configuration Management**: Settings, environment variables, logging
- **Service Architecture**: Orchestration, health checks, metrics
### ⚠️ Partially Implemented (50-80% Complete)
- **WebSocket Streaming**: Infrastructure complete, missing real data integration
- **Authentication**: Framework present, missing token validation
- **Middleware**: CORS, rate limiting, error handling implemented
### ❌ Incomplete/Mocked (0-40% Complete)
- **Hardware Interface**: Router communication, CSI data collection
- **Machine Learning Models**: DensePose integration, inference pipeline
- **Pose Service**: Mock data generation instead of real estimation
- **Signal Processing**: Basic structure, missing real-time algorithms
## Critical Implementation Gaps
### 1. Hardware Interface Layer (30% Complete)
**File: `src/core/router_interface.py`**
- **Lines 197-202**: Real CSI data collection not implemented
- Returns `None` with warning message instead of actual data
**File: `src/hardware/router_interface.py`**
- **Lines 94-116**: SSH connection and command execution are placeholders
- Missing router communication protocols and CSI data parsing
**File: `src/hardware/csi_extractor.py`**
- **Lines 152-189**: CSI parsing generates synthetic test data
- **Lines 164-170**: Creates random amplitude/phase data instead of parsing real CSI
### 2. Machine Learning Models (40% Complete)
**File: `src/models/densepose_head.py`**
- **Lines 88-117**: Architecture defined but not integrated with inference
- Missing model loading and WiFi-to-visual domain adaptation
**File: `src/models/modality_translation.py`**
- **Lines 166-229**: Network architecture complete but no trained weights
- Missing CSI-to-visual feature mapping validation
### 3. Pose Service Core Logic (50% Complete)
**File: `src/services/pose_service.py`**
- **Lines 174-177**: Generates mock pose data instead of real estimation
- **Lines 217-240**: Simplified mock pose output parsing
- **Lines 242-263**: Mock generation replacing neural network inference
## Detailed Findings by Component
### Hardware Integration Issues
1. **Router Communication**
- No actual SSH/SNMP implementation for router control
- Missing vendor-specific CSI extraction protocols
- No real WiFi monitoring mode setup
2. **CSI Data Collection**
- No integration with actual WiFi hardware drivers
- Missing real-time CSI stream processing
- No antenna diversity handling
### Machine Learning Issues
1. **Model Integration**
- DensePose models not loaded or initialized
- No GPU acceleration implementation
- Missing model inference pipeline
2. **Training Infrastructure**
- No training scripts or data preprocessing
- Missing domain adaptation between WiFi and visual data
- No model evaluation metrics
### Data Flow Issues
1. **Real-time Processing**
- Mock data flows throughout the system
- No actual CSI → Pose estimation pipeline
- Missing temporal consistency in pose tracking
2. **Database Integration**
- Models defined but no actual data persistence for poses
- Missing historical pose data analysis
## Implementation Priority Matrix
### Critical Priority (Blocking Core Functionality)
1. **Real CSI Data Collection** - Implement router interface
2. **Pose Estimation Models** - Load and integrate trained DensePose models
3. **CSI Processing Pipeline** - Real-time signal processing for human detection
4. **Model Training Infrastructure** - WiFi-to-pose domain adaptation
### High Priority (Essential Features)
1. **Authentication System** - JWT token validation implementation
2. **Real-time Streaming** - Integration with actual pose data
3. **Hardware Monitoring** - Actual router health and status checking
4. **Performance Optimization** - GPU acceleration, batching
### Medium Priority (Enhancement Features)
1. **Advanced Analytics** - Historical data analysis and reporting
2. **Multi-zone Support** - Coordinate multiple router deployments
3. **Alert System** - Real-time pose-based notifications
4. **Model Management** - Version control and A/B testing
## Code Quality Assessment
### Strengths
- **Professional Architecture**: Well-structured modular design
- **Comprehensive API**: FastAPI with proper documentation
- **Robust Database Design**: SQLAlchemy models with relationships
- **Deployment Ready**: Docker, Kubernetes, monitoring configurations
- **Testing Framework**: Unit and integration test structure
### Areas for Improvement
- **Core Functionality**: Missing actual WiFi-based pose detection
- **Hardware Integration**: No real router communication
- **Model Training**: No training or model loading implementation
- **Documentation**: API docs present, missing implementation guides
## Mock/Fake Implementation Summary
| Component | File | Lines | Description |
|-----------|------|-------|-------------|
| CSI Data Collection | `core/router_interface.py` | 197-202 | Returns None instead of real CSI data |
| CSI Parsing | `hardware/csi_extractor.py` | 164-170 | Generates synthetic CSI data |
| Pose Estimation | `services/pose_service.py` | 174-177 | Mock pose data generation |
| Router Commands | `hardware/router_interface.py` | 94-116 | Placeholder SSH execution |
| Authentication | `api/middleware/auth.py` | Various | Returns mock users in dev mode |
## Recommendations
### Immediate Actions Required
1. **Implement real CSI data collection** from WiFi routers
2. **Integrate trained DensePose models** for inference
3. **Complete hardware interface layer** with actual router communication
4. **Remove mock data generation** and implement real pose estimation
### Development Roadmap
1. **Phase 1**: Hardware integration and CSI data collection
2. **Phase 2**: Model training and inference pipeline
3. **Phase 3**: Real-time processing optimization
4. **Phase 4**: Advanced features and analytics
## Conclusion
The WiFi-DensePose project represents a **framework/prototype** rather than a functional WiFi-based pose detection system. While the architecture is excellent and deployment-ready, the core functionality requiring WiFi signal processing and pose estimation is largely unimplemented.
**Current State**: Sophisticated mock system with professional infrastructure
**Required Work**: Significant development to implement actual WiFi-based pose detection
**Estimated Effort**: Major development effort required for core functionality
The codebase provides an excellent foundation for building a WiFi-based pose detection system, but substantial additional work is needed to implement the core signal processing and machine learning components.
archive/v1/docs/security-features.md
+420
View File
@@ -0,0 +1,420 @@
# WiFi-DensePose Security Features Documentation
## Overview
This document details the authentication and rate limiting features implemented in the WiFi-DensePose API, including configuration options, usage examples, and security best practices.
## Table of Contents
1. [Authentication](#authentication)
2. [Rate Limiting](#rate-limiting)
3. [CORS Configuration](#cors-configuration)
4. [Security Headers](#security-headers)
5. [Configuration](#configuration)
6. [Testing](#testing)
7. [Best Practices](#best-practices)
## Authentication
### JWT Authentication
The API uses JWT (JSON Web Token) based authentication for securing endpoints.
#### Features
- **Token-based authentication**: Stateless authentication using JWT tokens
- **Role-based access control**: Support for different user roles (admin, user)
- **Token expiration**: Configurable token lifetime
- **Refresh token support**: Ability to refresh expired tokens
- **Multiple authentication sources**: Support for headers, query params, and cookies
#### Implementation Details
```python
# Location: src/api/middleware/auth.py
class AuthMiddleware(BaseHTTPMiddleware):
"""JWT Authentication middleware."""
```
**Public Endpoints** (No authentication required):
- `/` - Root endpoint
- `/health`, `/ready`, `/live` - Health check endpoints
- `/docs`, `/redoc`, `/openapi.json` - API documentation
- `/api/v1/pose/current` - Current pose data
- `/api/v1/pose/zones/*` - Zone information
- `/api/v1/pose/activities` - Activity data
- `/api/v1/pose/stats` - Statistics
- `/api/v1/stream/status` - Stream status
**Protected Endpoints** (Authentication required):
- `/api/v1/pose/analyze` - Pose analysis
- `/api/v1/pose/calibrate` - System calibration
- `/api/v1/pose/historical` - Historical data
- `/api/v1/stream/start` - Start streaming
- `/api/v1/stream/stop` - Stop streaming
- `/api/v1/stream/clients` - Client management
- `/api/v1/stream/broadcast` - Broadcasting
#### Usage Examples
**1. Obtaining a Token:**
```bash
# Login endpoint (if implemented)
curl -X POST http://localhost:8000/auth/login \
-H "Content-Type: application/json" \
-d '{"username": "user", "password": "password"}'
```
**2. Using Bearer Token:**
```bash
# Authorization header
curl -X POST http://localhost:8000/api/v1/pose/analyze \
-H "Authorization: Bearer <your-jwt-token>" \
-H "Content-Type: application/json" \
-d '{"data": "..."}'
```
**3. WebSocket Authentication:**
```javascript
// Query parameter for WebSocket
const ws = new WebSocket('ws://localhost:8000/ws/pose?token=<your-jwt-token>');
```
### API Key Authentication
Alternative authentication method for service-to-service communication.
```python
# Location: src/api/middleware/auth.py
class APIKeyAuth:
"""Alternative API key authentication for service-to-service communication."""
```
**Features:**
- Simple key-based authentication
- Service identification
- Key management (add/revoke)
**Usage:**
```bash
# API Key in header
curl -X GET http://localhost:8000/api/v1/pose/current \
-H "X-API-Key: your-api-key-here"
```
### Token Blacklist
Support for token revocation and logout functionality.
```python
class TokenBlacklist:
"""Simple in-memory token blacklist for logout functionality."""
```
## Rate Limiting
### Overview
The API implements sophisticated rate limiting using a sliding window algorithm with support for different user tiers.
#### Features
- **Sliding window algorithm**: Accurate request counting
- **Token bucket algorithm**: Alternative rate limiting method
- **User-based limits**: Different limits for anonymous/authenticated/admin users
- **Path-specific limits**: Custom limits for specific endpoints
- **Adaptive rate limiting**: Adjust limits based on system load
- **Temporary blocking**: Block clients after excessive violations
#### Implementation Details
```python
# Location: src/api/middleware/rate_limit.py
class RateLimitMiddleware(BaseHTTPMiddleware):
"""Rate limiting middleware with sliding window algorithm."""
```
**Default Rate Limits:**
- Anonymous users: 100 requests/hour (configurable)
- Authenticated users: 1000 requests/hour (configurable)
- Admin users: 10000 requests/hour
**Path-Specific Limits:**
- `/api/v1/pose/current`: 60 requests/minute
- `/api/v1/pose/analyze`: 10 requests/minute
- `/api/v1/pose/calibrate`: 1 request/5 minutes
- `/api/v1/stream/start`: 5 requests/minute
- `/api/v1/stream/stop`: 5 requests/minute
#### Response Headers
Rate limit information is included in response headers:
```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Window: 3600
X-RateLimit-Reset: 1641234567
```
When rate limit is exceeded:
```
HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: Exceeded
X-RateLimit-Remaining: 0
```
### Adaptive Rate Limiting
The system can adjust rate limits based on system load:
```python
class AdaptiveRateLimit:
"""Adaptive rate limiting based on system load."""
```
**Load-based adjustments:**
- High load (>80%): Reduce limits by 50%
- Medium load (>60%): Reduce limits by 30%
- Low load (<30%): Increase limits by 20%
## CORS Configuration
### Overview
Cross-Origin Resource Sharing (CORS) configuration for browser-based clients.
#### Features
- **Configurable origins**: Whitelist specific origins
- **Wildcard support**: Allow all origins in development
- **Preflight handling**: Proper OPTIONS request handling
- **Credential support**: Allow cookies and auth headers
- **Custom headers**: Expose rate limit and other headers
#### Configuration
```python
# Development configuration
cors_config = {
"allow_origins": ["*"],
"allow_credentials": True,
"allow_methods": ["*"],
"allow_headers": ["*"]
}
# Production configuration
cors_config = {
"allow_origins": ["https://app.example.com", "https://admin.example.com"],
"allow_credentials": True,
"allow_methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
"allow_headers": ["Authorization", "Content-Type"]
}
```
## Security Headers
The API includes various security headers for enhanced protection:
```python
class SecurityHeaders:
"""Security headers for API responses."""
```
**Headers included:**
- `X-Content-Type-Options: nosniff` - Prevent MIME sniffing
- `X-Frame-Options: DENY` - Prevent clickjacking
- `X-XSS-Protection: 1; mode=block` - Enable XSS protection
- `Referrer-Policy: strict-origin-when-cross-origin` - Control referrer
- `Content-Security-Policy` - Control resource loading
## Configuration
### Environment Variables
```bash
# Authentication
ENABLE_AUTHENTICATION=true
SECRET_KEY=your-secret-key-here
JWT_ALGORITHM=HS256
JWT_EXPIRE_HOURS=24
# Rate Limiting
ENABLE_RATE_LIMITING=true
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_AUTHENTICATED_REQUESTS=1000
RATE_LIMIT_WINDOW=3600
# CORS
CORS_ENABLED=true
CORS_ORIGINS=["https://app.example.com"]
CORS_ALLOW_CREDENTIALS=true
# Security
ALLOWED_HOSTS=["api.example.com", "localhost"]
```
### Settings Class
```python
# src/config/settings.py
class Settings(BaseSettings):
# Authentication settings
enable_authentication: bool = Field(default=True)
secret_key: str = Field(...)
jwt_algorithm: str = Field(default="HS256")
jwt_expire_hours: int = Field(default=24)
# Rate limiting settings
enable_rate_limiting: bool = Field(default=True)
rate_limit_requests: int = Field(default=100)
rate_limit_authenticated_requests: int = Field(default=1000)
rate_limit_window: int = Field(default=3600)
# CORS settings
cors_enabled: bool = Field(default=True)
cors_origins: List[str] = Field(default=["*"])
cors_allow_credentials: bool = Field(default=True)
```
## Testing
### Test Script
A comprehensive test script is provided to verify security features:
```bash
# Run the test script
python test_auth_rate_limit.py
```
The test script covers:
- Public endpoint access
- Protected endpoint authentication
- JWT token validation
- Rate limiting behavior
- CORS headers
- Security headers
- Feature flag verification
### Manual Testing
**1. Test Authentication:**
```bash
# Without token (should fail)
curl -X POST http://localhost:8000/api/v1/pose/analyze
# With token (should succeed)
curl -X POST http://localhost:8000/api/v1/pose/analyze \
-H "Authorization: Bearer <token>"
```
**2. Test Rate Limiting:**
```bash
# Send multiple requests quickly
for i in {1..150}; do
curl -s -o /dev/null -w "%{http_code}\n" \
http://localhost:8000/api/v1/pose/current
done
```
**3. Test CORS:**
```bash
# Preflight request
curl -X OPTIONS http://localhost:8000/api/v1/pose/current \
-H "Origin: https://example.com" \
-H "Access-Control-Request-Method: GET" \
-H "Access-Control-Request-Headers: Authorization"
```
## Best Practices
### Security Recommendations
1. **Production Configuration:**
- Always use strong secret keys
- Disable debug mode
- Restrict CORS origins
- Use HTTPS only
- Enable all security headers
2. **Token Management:**
- Implement token refresh mechanism
- Use short-lived tokens
- Implement logout/blacklist functionality
- Store tokens securely on client
3. **Rate Limiting:**
- Set appropriate limits for your use case
- Monitor and adjust based on usage
- Implement different tiers for users
- Use Redis for distributed systems
4. **API Keys:**
- Use for service-to-service communication
- Rotate keys regularly
- Monitor key usage
- Implement key scoping
### Monitoring
1. **Authentication Events:**
- Log failed authentication attempts
- Monitor suspicious patterns
- Alert on repeated failures
2. **Rate Limit Violations:**
- Track clients hitting limits
- Identify potential abuse
- Adjust limits as needed
3. **Security Headers:**
- Verify headers in responses
- Test with security tools
- Regular security audits
### Troubleshooting
**Common Issues:**
1. **401 Unauthorized:**
- Check token format
- Verify token expiration
- Ensure correct secret key
2. **429 Too Many Requests:**
- Check rate limit configuration
- Verify client identification
- Look for Retry-After header
3. **CORS Errors:**
- Verify allowed origins
- Check preflight responses
- Ensure credentials setting matches
## Disabling Security Features
For development or testing, security features can be disabled:
```bash
# Disable authentication
ENABLE_AUTHENTICATION=false
# Disable rate limiting
ENABLE_RATE_LIMITING=false
# Allow all CORS origins
CORS_ORIGINS=["*"]
```
**Warning:** Never disable security features in production!
## Future Enhancements
1. **OAuth2/OpenID Connect Support**
2. **API Key Scoping and Permissions**
3. **IP-based Rate Limiting**
4. **Geographic Restrictions**
5. **Request Signing**
6. **Mutual TLS Authentication**
archive/v1/docs/troubleshooting.md
+1058
View File
File diff suppressed because it is too large Load Diff
archive/v1/docs/user-guide/api-reference.md
+989
View File
@@ -0,0 +1,989 @@
# API Reference
## Overview
The WiFi-DensePose API provides comprehensive access to pose estimation data, system control, and configuration management through RESTful endpoints and real-time WebSocket connections.
## Table of Contents
1. [Authentication](#authentication)
2. [Base URL and Versioning](#base-url-and-versioning)
3. [Pose Data Endpoints](#pose-data-endpoints)
4. [System Control Endpoints](#system-control-endpoints)
5. [Configuration Endpoints](#configuration-endpoints)
6. [Analytics Endpoints](#analytics-endpoints)
7. [WebSocket API](#websocket-api)
8. [Error Handling](#error-handling)
9. [Rate Limiting](#rate-limiting)
10. [Code Examples](#code-examples)
## Authentication
### Bearer Token Authentication
All API endpoints require authentication using JWT Bearer tokens:
```http
Authorization: Bearer <your-jwt-token>
```
### Obtaining a Token
```bash
# Get authentication token
curl -X POST http://localhost:8000/api/v1/auth/token \
-H "Content-Type: application/json" \
-d '{
"username": "your-username",
"password": "your-password"
}'
```
**Response:**
```json
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"token_type": "bearer",
"expires_in": 86400
}
```
### API Key Authentication
For service-to-service communication:
```http
X-API-Key: <your-api-key>
```
## Base URL and Versioning
- **Base URL**: `http://localhost:8000/api/v1`
- **Current Version**: v1
- **Content-Type**: `application/json`
## Pose Data Endpoints
### Get Latest Pose Data
Retrieve the most recent pose estimation results.
**Endpoint:** `GET /pose/latest`
**Headers:**
```http
Authorization: Bearer <token>
```
**Response:**
```json
{
"timestamp": "2025-01-07T04:46:32.123Z",
"frame_id": 12345,
"processing_time_ms": 45,
"persons": [
{
"id": 1,
"confidence": 0.87,
"bounding_box": {
"x": 120,
"y": 80,
"width": 200,
"height": 400
},
"keypoints": [
{
"name": "nose",
"x": 220,
"y": 100,
"confidence": 0.95,
"visible": true
},
{
"name": "left_shoulder",
"x": 200,
"y": 150,
"confidence": 0.89,
"visible": true
}
],
"dense_pose": {
"body_parts": [
{
"part_id": 1,
"part_name": "torso",
"uv_coordinates": [[0.5, 0.3], [0.6, 0.4]],
"confidence": 0.89
}
]
},
"tracking_info": {
"track_id": "track_001",
"track_age": 150,
"velocity": {"x": 0.1, "y": 0.05}
}
}
],
"metadata": {
"environment_id": "room_001",
"router_count": 3,
"signal_quality": 0.82,
"processing_pipeline": "standard"
}
}
```
**Status Codes:**
- `200 OK`: Success
- `404 Not Found`: No pose data available
- `401 Unauthorized`: Authentication required
- `503 Service Unavailable`: System not initialized
### Get Historical Pose Data
Retrieve historical pose data with filtering options.
**Endpoint:** `GET /pose/history`
**Query Parameters:**
- `start_time` (optional): ISO 8601 timestamp for range start
- `end_time` (optional): ISO 8601 timestamp for range end
- `limit` (optional): Maximum number of records (default: 100, max: 1000)
- `person_id` (optional): Filter by specific person ID
- `confidence_threshold` (optional): Minimum confidence score (0.0-1.0)
**Example:**
```bash
curl "http://localhost:8000/api/v1/pose/history?start_time=2025-01-07T00:00:00Z&limit=50&confidence_threshold=0.7" \
-H "Authorization: Bearer <token>"
```
**Response:**
```json
{
"poses": [
{
"timestamp": "2025-01-07T04:46:32.123Z",
"persons": [...],
"metadata": {...}
}
],
"pagination": {
"total_count": 1500,
"returned_count": 50,
"has_more": true,
"next_cursor": "eyJpZCI6MTIzNDV9"
}
}
```
### Query Pose Data
Execute complex queries on pose data with aggregation support.
**Endpoint:** `POST /pose/query`
**Request Body:**
```json
{
"query": {
"time_range": {
"start": "2025-01-07T00:00:00Z",
"end": "2025-01-07T23:59:59Z"
},
"filters": {
"person_count": {"min": 1, "max": 5},
"confidence": {"min": 0.7},
"activity": ["walking", "standing"]
},
"aggregation": {
"type": "hourly_summary",
"metrics": ["person_count", "avg_confidence"]
}
}
}
```
**Response:**
```json
{
"results": [
{
"timestamp": "2025-01-07T10:00:00Z",
"person_count": 3,
"avg_confidence": 0.85,
"activities": {
"walking": 0.6,
"standing": 0.4
}
}
],
"query_metadata": {
"execution_time_ms": 150,
"total_records_scanned": 10000,
"cache_hit": false
}
}
```
## System Control Endpoints
### Get System Status
Get comprehensive system health and status information.
**Endpoint:** `GET /system/status`
**Response:**
```json
{
"status": "running",
"uptime_seconds": 86400,
"version": "1.0.0",
"components": {
"csi_receiver": {
"status": "active",
"data_rate_hz": 25.3,
"packet_loss_rate": 0.02,
"last_packet_time": "2025-01-07T04:46:32Z"
},
"neural_network": {
"status": "active",
"model_loaded": true,
"inference_time_ms": 45,
"gpu_utilization": 0.65
},
"tracking": {
"status": "active",
"active_tracks": 2,
"track_quality": 0.89
}
},
"hardware": {
"cpu_usage": 0.45,
"memory_usage": 0.62,
"gpu_memory_usage": 0.78,
"disk_usage": 0.23
},
"network": {
"connected_routers": 3,
"signal_strength": -45,
"interference_level": 0.15
}
}
```
### Start System
Start the pose estimation system with configuration options.
**Endpoint:** `POST /system/start`
**Request Body:**
```json
{
"configuration": {
"domain": "healthcare",
"environment_id": "room_001",
"calibration_required": true
}
}
```
**Response:**
```json
{
"status": "starting",
"estimated_ready_time": "2025-01-07T04:47:00Z",
"initialization_steps": [
{
"step": "hardware_initialization",
"status": "in_progress",
"progress": 0.3
},
{
"step": "model_loading",
"status": "pending",
"progress": 0.0
}
]
}
```
### Stop System
Gracefully stop the pose estimation system.
**Endpoint:** `POST /system/stop`
**Request Body:**
```json
{
"force": false,
"save_state": true
}
```
**Response:**
```json
{
"status": "stopping",
"estimated_stop_time": "2025-01-07T04:47:30Z",
"shutdown_steps": [
{
"step": "data_pipeline_stop",
"status": "completed",
"progress": 1.0
},
{
"step": "model_unloading",
"status": "in_progress",
"progress": 0.7
}
]
}
```
## Configuration Endpoints
### Get Configuration
Retrieve current system configuration.
**Endpoint:** `GET /config`
**Response:**
```json
{
"domain": "healthcare",
"environment": {
"id": "room_001",
"name": "Patient Room 1",
"calibration_timestamp": "2025-01-07T04:00:00Z"
},
"detection": {
"confidence_threshold": 0.7,
"max_persons": 5,
"tracking_enabled": true
},
"alerts": {
"fall_detection": {
"enabled": true,
"sensitivity": 0.8,
"notification_delay_seconds": 5
},
"inactivity_detection": {
"enabled": true,
"threshold_minutes": 30
}
},
"streaming": {
"restream_enabled": false,
"websocket_enabled": true,
"mqtt_enabled": true
}
}
```
### Update Configuration
Update system configuration with partial updates supported.
**Endpoint:** `PUT /config`
**Request Body:**
```json
{
"detection": {
"confidence_threshold": 0.75,
"max_persons": 3
},
"alerts": {
"fall_detection": {
"sensitivity": 0.9
}
}
}
```
**Response:**
```json
{
"status": "updated",
"changes_applied": [
"detection.confidence_threshold",
"detection.max_persons",
"alerts.fall_detection.sensitivity"
],
"restart_required": false,
"validation_warnings": []
}
```
## Analytics Endpoints
### Healthcare Analytics
Get healthcare-specific analytics and insights.
**Endpoint:** `GET /analytics/healthcare`
**Query Parameters:**
- `period`: Time period (hour, day, week, month)
- `metrics`: Comma-separated list of metrics
**Example:**
```bash
curl "http://localhost:8000/api/v1/analytics/healthcare?period=day&metrics=fall_events,activity_summary" \
-H "Authorization: Bearer <token>"
```
**Response:**
```json
{
"period": "day",
"date": "2025-01-07",
"metrics": {
"fall_events": {
"count": 2,
"events": [
{
"timestamp": "2025-01-07T14:30:15Z",
"person_id": 1,
"severity": "moderate",
"response_time_seconds": 45,
"location": {"x": 150, "y": 200}
}
]
},
"activity_summary": {
"walking_minutes": 120,
"sitting_minutes": 480,
"lying_minutes": 360,
"standing_minutes": 180
},
"mobility_score": 0.75,
"sleep_quality": {
"total_sleep_hours": 7.5,
"sleep_efficiency": 0.89,
"restlessness_events": 3
}
}
}
```
### Retail Analytics
Get retail-specific analytics and customer insights.
**Endpoint:** `GET /analytics/retail`
**Response:**
```json
{
"period": "day",
"date": "2025-01-07",
"metrics": {
"traffic": {
"total_visitors": 245,
"unique_visitors": 198,
"peak_hour": "14:00",
"peak_count": 15,
"average_dwell_time_minutes": 12.5
},
"zones": [
{
"zone_id": "entrance",
"zone_name": "Store Entrance",
"visitor_count": 245,
"avg_dwell_time_minutes": 2.1,
"conversion_rate": 0.85
},
{
"zone_id": "electronics",
"zone_name": "Electronics Section",
"visitor_count": 89,
"avg_dwell_time_minutes": 8.7,
"conversion_rate": 0.34
}
],
"conversion_funnel": {
"entrance": 245,
"product_interaction": 156,
"checkout_area": 89,
"purchase": 67
},
"heat_map": {
"high_traffic_areas": [
{"zone": "entrance", "intensity": 0.95},
{"zone": "checkout", "intensity": 0.78}
]
}
}
}
```
### Security Analytics
Get security-specific analytics and threat assessments.
**Endpoint:** `GET /analytics/security`
**Response:**
```json
{
"period": "day",
"date": "2025-01-07",
"metrics": {
"intrusion_events": {
"count": 1,
"events": [
{
"timestamp": "2025-01-07T02:15:30Z",
"zone": "restricted_area",
"person_count": 1,
"threat_level": "medium",
"response_time_seconds": 120
}
]
},
"perimeter_monitoring": {
"total_detections": 45,
"authorized_entries": 42,
"unauthorized_attempts": 3,
"false_positives": 0
},
"crowd_analysis": {
"max_occupancy": 12,
"average_occupancy": 3.2,
"crowd_formation_events": 0
}
}
}
```
## WebSocket API
### Connection
Connect to the WebSocket endpoint for real-time data streaming.
**Endpoint:** `ws://localhost:8000/ws/pose`
**Authentication:** Include token as query parameter or in headers:
```javascript
const ws = new WebSocket('ws://localhost:8000/ws/pose?token=<your-jwt-token>');
```
### Connection Establishment
**Server Message:**
```json
{
"type": "connection_established",
"client_id": "client_12345",
"server_time": "2025-01-07T04:46:32Z",
"supported_protocols": ["pose_v1", "alerts_v1"]
}
```
### Subscription Management
**Subscribe to Pose Updates:**
```json
{
"type": "subscribe",
"channel": "pose_updates",
"filters": {
"min_confidence": 0.7,
"person_ids": [1, 2, 3],
"include_keypoints": true,
"include_dense_pose": false
}
}
```
**Subscription Confirmation:**
```json
{
"type": "subscription_confirmed",
"channel": "pose_updates",
"subscription_id": "sub_67890",
"filters_applied": {
"min_confidence": 0.7,
"person_ids": [1, 2, 3]
}
}
```
### Real-Time Data Streaming
**Pose Update Message:**
```json
{
"type": "pose_update",
"subscription_id": "sub_67890",
"timestamp": "2025-01-07T04:46:32.123Z",
"data": {
"frame_id": 12345,
"persons": [...],
"metadata": {...}
}
}
```
**System Status Update:**
```json
{
"type": "system_status",
"timestamp": "2025-01-07T04:46:32Z",
"status": {
"processing_fps": 25.3,
"active_persons": 2,
"system_health": "good",
"gpu_utilization": 0.65
}
}
```
### Alert Streaming
**Subscribe to Alerts:**
```json
{
"type": "subscribe",
"channel": "alerts",
"filters": {
"alert_types": ["fall_detection", "intrusion"],
"severity": ["high", "critical"]
}
}
```
**Alert Message:**
```json
{
"type": "alert",
"alert_id": "alert_12345",
"timestamp": "2025-01-07T04:46:32Z",
"alert_type": "fall_detection",
"severity": "high",
"data": {
"person_id": 1,
"location": {"x": 220, "y": 180},
"confidence": 0.92,
"video_clip_url": "/clips/fall_12345.mp4"
},
"actions_required": ["medical_response", "notification"]
}
```
## Error Handling
### Standard Error Response Format
```json
{
"error": {
"code": "POSE_DATA_NOT_FOUND",
"message": "No pose data available for the specified time range",
"details": {
"requested_range": {
"start": "2025-01-07T00:00:00Z",
"end": "2025-01-07T01:00:00Z"
},
"available_range": {
"start": "2025-01-07T02:00:00Z",
"end": "2025-01-07T04:46:32Z"
}
},
"timestamp": "2025-01-07T04:46:32Z",
"request_id": "req_12345"
}
}
```
### HTTP Status Codes
#### Success Codes
- `200 OK`: Request successful
- `201 Created`: Resource created successfully
- `202 Accepted`: Request accepted for processing
- `204 No Content`: Request successful, no content returned
#### Client Error Codes
- `400 Bad Request`: Invalid request format or parameters
- `401 Unauthorized`: Authentication required or invalid
- `403 Forbidden`: Insufficient permissions
- `404 Not Found`: Resource not found
- `409 Conflict`: Resource conflict (e.g., system already running)
- `422 Unprocessable Entity`: Validation errors
- `429 Too Many Requests`: Rate limit exceeded
#### Server Error Codes
- `500 Internal Server Error`: Unexpected server error
- `502 Bad Gateway`: Upstream service error
- `503 Service Unavailable`: System not ready or overloaded
- `504 Gateway Timeout`: Request timeout
### Validation Error Response
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Request validation failed",
"details": {
"field_errors": [
{
"field": "confidence_threshold",
"message": "Value must be between 0.0 and 1.0",
"received_value": 1.5
},
{
"field": "max_persons",
"message": "Value must be a positive integer",
"received_value": -1
}
]
},
"timestamp": "2025-01-07T04:46:32Z",
"request_id": "req_12346"
}
}
```
## Rate Limiting
### Rate Limit Headers
All responses include rate limiting information:
```http
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1704686400
X-RateLimit-Window: 3600
```
### Rate Limits by Endpoint Type
- **REST API**: 1000 requests per hour per API key
- **WebSocket**: 100 connections per IP address
- **Streaming**: 10 concurrent streams per account
- **Webhook**: 10,000 events per hour per endpoint
### Rate Limit Exceeded Response
```json
{
"error": {
"code": "RATE_LIMIT_EXCEEDED",
"message": "Rate limit exceeded. Try again later.",
"details": {
"limit": 1000,
"window_seconds": 3600,
"reset_time": "2025-01-07T05:46:32Z"
},
"timestamp": "2025-01-07T04:46:32Z",
"request_id": "req_12347"
}
}
```
## Code Examples
### Python Example
```python
import requests
import json
from datetime import datetime, timedelta
class WiFiDensePoseClient:
def __init__(self, base_url, token):
self.base_url = base_url
self.headers = {
'Authorization': f'Bearer {token}',
'Content-Type': 'application/json'
}
def get_latest_pose(self):
"""Get the latest pose data."""
response = requests.get(
f'{self.base_url}/pose/latest',
headers=self.headers
)
response.raise_for_status()
return response.json()
def get_historical_poses(self, start_time=None, end_time=None, limit=100):
"""Get historical pose data."""
params = {'limit': limit}
if start_time:
params['start_time'] = start_time.isoformat()
if end_time:
params['end_time'] = end_time.isoformat()
response = requests.get(
f'{self.base_url}/pose/history',
headers=self.headers,
params=params
)
response.raise_for_status()
return response.json()
def start_system(self, domain='general', environment_id='default'):
"""Start the pose estimation system."""
data = {
'configuration': {
'domain': domain,
'environment_id': environment_id,
'calibration_required': True
}
}
response = requests.post(
f'{self.base_url}/system/start',
headers=self.headers,
json=data
)
response.raise_for_status()
return response.json()
# Usage example
client = WiFiDensePoseClient('http://localhost:8000/api/v1', 'your-token')
# Get latest pose data
latest = client.get_latest_pose()
print(f"Found {len(latest['persons'])} persons")
# Get historical data for the last hour
end_time = datetime.now()
start_time = end_time - timedelta(hours=1)
history = client.get_historical_poses(start_time, end_time)
print(f"Retrieved {len(history['poses'])} historical records")
```
### JavaScript Example
```javascript
class WiFiDensePoseClient {
constructor(baseUrl, token) {
this.baseUrl = baseUrl;
this.headers = {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json'
};
}
async getLatestPose() {
const response = await fetch(`${this.baseUrl}/pose/latest`, {
headers: this.headers
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return await response.json();
}
async updateConfiguration(config) {
const response = await fetch(`${this.baseUrl}/config`, {
method: 'PUT',
headers: this.headers,
body: JSON.stringify(config)
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return await response.json();
}
connectWebSocket() {
const ws = new WebSocket(`ws://localhost:8000/ws/pose?token=${this.token}`);
ws.onopen = () => {
console.log('WebSocket connected');
// Subscribe to pose updates
ws.send(JSON.stringify({
type: 'subscribe',
channel: 'pose_updates',
filters: {
min_confidence: 0.7
}
}));
};
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
console.log('Received:', data);
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
return ws;
}
}
// Usage example
const client = new WiFiDensePoseClient('http://localhost:8000/api/v1', 'your-token');
// Get latest pose data
client.getLatestPose()
.then(data => console.log('Latest pose:', data))
.catch(error => console.error('Error:', error));
// Connect to WebSocket for real-time updates
const ws = client.connectWebSocket();
```
### cURL Examples
```bash
# Get authentication token
curl -X POST http://localhost:8000/api/v1/auth/token \
-H "Content-Type: application/json" \
-d '{"username": "admin", "password": "password"}'
# Get latest pose data
curl http://localhost:8000/api/v1/pose/latest \
-H "Authorization: Bearer <token>"
# Start system
curl -X POST http://localhost:8000/api/v1/system/start \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"configuration": {
"domain": "healthcare",
"environment_id": "room_001"
}
}'
# Update configuration
curl -X PUT http://localhost:8000/api/v1/config \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"detection": {
"confidence_threshold": 0.8
}
}'
# Get healthcare analytics
curl "http://localhost:8000/api/v1/analytics/healthcare?period=day" \
-H "Authorization: Bearer <token>"
```
---
For more detailed information, see:
- [Getting Started Guide](getting-started.md)
- [Configuration Guide](configuration.md)
- [WebSocket API Documentation](../api/websocket-api.md)
- [Authentication Guide](../api/authentication.md)
archive/v1/docs/user-guide/configuration.md
+722
View File
@@ -0,0 +1,722 @@
# Configuration Guide
## Overview
This guide covers comprehensive configuration options for the WiFi-DensePose system, including domain-specific settings, hardware configuration, performance tuning, and security settings.
## Table of Contents
1. [Configuration Files](#configuration-files)
2. [Environment Variables](#environment-variables)
3. [Domain-Specific Configuration](#domain-specific-configuration)
4. [Hardware Configuration](#hardware-configuration)
5. [Performance Tuning](#performance-tuning)
6. [Security Configuration](#security-configuration)
7. [Integration Settings](#integration-settings)
8. [Monitoring and Logging](#monitoring-and-logging)
9. [Advanced Configuration](#advanced-configuration)
## Configuration Files
### Primary Configuration File
The system uses environment variables and configuration files for settings management:
```bash
# Main configuration file
.env
# Domain-specific configurations
config/domains/healthcare.yaml
config/domains/retail.yaml
config/domains/security.yaml
# Hardware configurations
config/hardware/routers.yaml
config/hardware/processing.yaml
```
### Configuration Hierarchy
Configuration is loaded in the following order (later values override earlier ones):
1. Default values in [`src/config/settings.py`](../../src/config/settings.py)
2. Environment-specific configuration files
3. `.env` file
4. Environment variables
5. Command-line arguments
## Environment Variables
### Application Settings
```bash
# Basic application settings
APP_NAME="WiFi-DensePose API"
VERSION="1.0.0"
ENVIRONMENT="development" # development, staging, production
DEBUG=false
# Server configuration
HOST="0.0.0.0"
PORT=8000
RELOAD=false
WORKERS=1
```
### Security Settings
```bash
# JWT Configuration
SECRET_KEY="your-super-secret-key-change-in-production"
JWT_ALGORITHM="HS256"
JWT_EXPIRE_HOURS=24
# CORS and Host Settings
ALLOWED_HOSTS="localhost,127.0.0.1,your-domain.com"
CORS_ORIGINS="http://localhost:3000,https://your-frontend.com"
# Rate Limiting
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_AUTHENTICATED_REQUESTS=1000
RATE_LIMIT_WINDOW=3600 # seconds
```
### Database Configuration
```bash
# Database Settings
DATABASE_URL="postgresql://user:password@localhost:5432/wifi_densepose"
DATABASE_POOL_SIZE=10
DATABASE_MAX_OVERFLOW=20
# Redis Configuration
REDIS_URL="redis://localhost:6379/0"
REDIS_PASSWORD=""
REDIS_DB=0
```
### Hardware Settings
```bash
# WiFi Interface
WIFI_INTERFACE="wlan0"
CSI_BUFFER_SIZE=1000
HARDWARE_POLLING_INTERVAL=0.1
# Development/Testing
MOCK_HARDWARE=false
MOCK_POSE_DATA=false
```
### Pose Estimation Settings
```bash
# Model Configuration
POSE_MODEL_PATH="./models/densepose_model.pth"
POSE_CONFIDENCE_THRESHOLD=0.5
POSE_PROCESSING_BATCH_SIZE=32
POSE_MAX_PERSONS=10
# Streaming Settings
STREAM_FPS=30
STREAM_BUFFER_SIZE=100
WEBSOCKET_PING_INTERVAL=60
WEBSOCKET_TIMEOUT=300
```
### Storage Settings
```bash
# Storage Paths
DATA_STORAGE_PATH="./data"
MODEL_STORAGE_PATH="./models"
TEMP_STORAGE_PATH="./temp"
MAX_STORAGE_SIZE_GB=100
```
### Feature Flags
```bash
# Feature Toggles
ENABLE_AUTHENTICATION=true
ENABLE_RATE_LIMITING=true
ENABLE_WEBSOCKETS=true
ENABLE_HISTORICAL_DATA=true
ENABLE_REAL_TIME_PROCESSING=true
ENABLE_TEST_ENDPOINTS=false
```
## Domain-Specific Configuration
### Healthcare Domain
Healthcare deployments require enhanced privacy and accuracy settings:
```yaml
# config/domains/healthcare.yaml
domain: healthcare
description: "Healthcare monitoring and patient safety"
detection:
confidence_threshold: 0.8
max_persons: 3
tracking_enabled: true
privacy_mode: true
alerts:
fall_detection:
enabled: true
sensitivity: 0.9
notification_delay_seconds: 5
emergency_contacts:
- "nurse-station@hospital.com"
- "+1-555-0123"
inactivity_detection:
enabled: true
threshold_minutes: 30
alert_levels: ["warning", "critical"]
vital_signs_monitoring:
enabled: true
heart_rate_estimation: true
breathing_pattern_analysis: true
privacy:
data_retention_days: 30
anonymization_enabled: true
audit_logging: true
hipaa_compliance: true
notifications:
webhook_urls:
- "https://hospital-system.com/api/alerts"
mqtt_topics:
- "hospital/room/{room_id}/alerts"
email_alerts: true
```
### Retail Domain
Retail deployments focus on customer analytics and traffic patterns:
```yaml
# config/domains/retail.yaml
domain: retail
description: "Retail analytics and customer insights"
detection:
confidence_threshold: 0.7
max_persons: 15
tracking_enabled: true
zone_detection: true
analytics:
traffic_counting:
enabled: true
entrance_zones: ["entrance", "exit"]
dwell_time_tracking: true
heat_mapping:
enabled: true
zone_definitions:
- name: "entrance"
coordinates: [[0, 0], [100, 50]]
- name: "electronics"
coordinates: [[100, 0], [200, 100]]
- name: "checkout"
coordinates: [[200, 0], [300, 50]]
conversion_tracking:
enabled: true
interaction_threshold_seconds: 10
purchase_correlation: true
privacy:
data_retention_days: 90
anonymization_enabled: true
gdpr_compliance: true
reporting:
daily_reports: true
weekly_summaries: true
real_time_dashboard: true
```
### Security Domain
Security deployments prioritize intrusion detection and perimeter monitoring:
```yaml
# config/domains/security.yaml
domain: security
description: "Security monitoring and intrusion detection"
detection:
confidence_threshold: 0.9
max_persons: 10
tracking_enabled: true
motion_sensitivity: 0.95
security:
intrusion_detection:
enabled: true
restricted_zones:
- name: "secure_area"
coordinates: [[50, 50], [150, 150]]
alert_immediately: true
- name: "perimeter"
coordinates: [[0, 0], [300, 300]]
alert_delay_seconds: 10
unauthorized_access:
enabled: true
authorized_persons: [] # Empty for general detection
time_restrictions:
- days: ["monday", "tuesday", "wednesday", "thursday", "friday"]
hours: ["09:00", "17:00"]
threat_assessment:
enabled: true
aggressive_behavior_detection: true
crowd_formation_detection: true
alerts:
immediate_notification: true
escalation_levels:
- level: 1
delay_seconds: 0
contacts: ["security@company.com"]
- level: 2
delay_seconds: 30
contacts: ["security@company.com", "manager@company.com"]
- level: 3
delay_seconds: 60
contacts: ["security@company.com", "manager@company.com", "emergency@company.com"]
integration:
security_system_api: "https://security-system.com/api"
camera_system_integration: true
access_control_integration: true
```
## Hardware Configuration
### Router Configuration
```yaml
# config/hardware/routers.yaml
routers:
- id: "router_001"
type: "atheros"
model: "TP-Link Archer C7"
ip_address: "192.168.1.1"
mac_address: "aa:bb:cc:dd:ee:01"
location:
room: "living_room"
coordinates: [0, 0, 2.5] # x, y, z in meters
csi_config:
sampling_rate: 30 # Hz
antenna_count: 3
subcarrier_count: 56
data_port: 5500
- id: "router_002"
type: "atheros"
model: "Netgear Nighthawk"
ip_address: "192.168.1.2"
mac_address: "aa:bb:cc:dd:ee:02"
location:
room: "living_room"
coordinates: [5, 0, 2.5]
csi_config:
sampling_rate: 30
antenna_count: 3
subcarrier_count: 56
data_port: 5501
network:
csi_data_interface: "eth0"
buffer_size: 1000
timeout_seconds: 5
retry_attempts: 3
```
### Processing Hardware Configuration
```yaml
# config/hardware/processing.yaml
processing:
cpu:
cores: 8
threads_per_core: 2
optimization: "performance" # performance, balanced, power_save
memory:
total_gb: 16
allocation:
csi_processing: 4
neural_network: 8
api_services: 2
system_overhead: 2
gpu:
enabled: true
device_id: 0
memory_gb: 8
cuda_version: "11.8"
optimization:
batch_size: 32
mixed_precision: true
tensor_cores: true
storage:
data_drive:
path: "/data"
type: "ssd"
size_gb: 500
model_drive:
path: "/models"
type: "ssd"
size_gb: 100
temp_drive:
path: "/tmp"
type: "ram"
size_gb: 8
```
## Performance Tuning
### Processing Pipeline Optimization
```bash
# Neural Network Settings
POSE_PROCESSING_BATCH_SIZE=32 # Adjust based on GPU memory
POSE_CONFIDENCE_THRESHOLD=0.7 # Higher = fewer false positives
POSE_MAX_PERSONS=5 # Limit for performance
# Streaming Optimization
STREAM_FPS=30 # Reduce for lower bandwidth
STREAM_BUFFER_SIZE=100 # Increase for smoother streaming
WEBSOCKET_PING_INTERVAL=60 # Connection keep-alive
# Database Optimization
DATABASE_POOL_SIZE=20 # Increase for high concurrency
DATABASE_MAX_OVERFLOW=40 # Additional connections when needed
# Caching Settings
REDIS_URL="redis://localhost:6379/0"
CACHE_TTL_SECONDS=300 # Cache expiration time
```
### Resource Allocation
```yaml
# docker-compose.override.yml
version: '3.8'
services:
wifi-densepose-api:
deploy:
resources:
limits:
cpus: '4.0'
memory: 8G
reservations:
cpus: '2.0'
memory: 4G
environment:
- WORKERS=4
- POSE_PROCESSING_BATCH_SIZE=64
neural-network:
deploy:
resources:
limits:
cpus: '2.0'
memory: 6G
reservations:
cpus: '1.0'
memory: 4G
runtime: nvidia
environment:
- CUDA_VISIBLE_DEVICES=0
```
### Performance Monitoring
```bash
# Enable performance monitoring
PERFORMANCE_MONITORING=true
METRICS_ENABLED=true
HEALTH_CHECK_INTERVAL=30
# Logging for performance analysis
LOG_LEVEL="INFO"
LOG_PERFORMANCE_METRICS=true
LOG_SLOW_QUERIES=true
SLOW_QUERY_THRESHOLD_MS=1000
```
## Security Configuration
### Authentication and Authorization
```bash
# JWT Configuration
SECRET_KEY="$(openssl rand -base64 32)" # Generate secure key
JWT_ALGORITHM="HS256"
JWT_EXPIRE_HOURS=8 # Shorter expiration for production
# API Key Configuration
API_KEY_LENGTH=32
API_KEY_EXPIRY_DAYS=90
API_KEY_ROTATION_ENABLED=true
```
### Network Security
```bash
# HTTPS Configuration
ENABLE_HTTPS=true
SSL_CERT_PATH="/etc/ssl/certs/wifi-densepose.crt"
SSL_KEY_PATH="/etc/ssl/private/wifi-densepose.key"
# Firewall Settings
ALLOWED_IPS="192.168.1.0/24,10.0.0.0/8"
BLOCKED_IPS=""
RATE_LIMIT_ENABLED=true
```
### Data Protection
```bash
# Encryption Settings
DATABASE_ENCRYPTION=true
DATA_AT_REST_ENCRYPTION=true
BACKUP_ENCRYPTION=true
# Privacy Settings
ANONYMIZATION_ENABLED=true
DATA_RETENTION_DAYS=30
AUDIT_LOGGING=true
GDPR_COMPLIANCE=true
```
## Integration Settings
### MQTT Configuration
```bash
# MQTT Broker Settings
MQTT_BROKER_HOST="localhost"
MQTT_BROKER_PORT=1883
MQTT_USERNAME="wifi_densepose"
MQTT_PASSWORD="secure_password"
MQTT_TLS_ENABLED=true
# Topic Configuration
MQTT_TOPIC_PREFIX="wifi-densepose"
MQTT_QOS_LEVEL=1
MQTT_RETAIN_MESSAGES=false
```
### Webhook Configuration
```bash
# Webhook Settings
WEBHOOK_TIMEOUT_SECONDS=30
WEBHOOK_RETRY_ATTEMPTS=3
WEBHOOK_RETRY_DELAY_SECONDS=5
# Security
WEBHOOK_SIGNATURE_ENABLED=true
WEBHOOK_SECRET_KEY="webhook_secret_key"
```
### External API Integration
```bash
# Restream Integration
RESTREAM_API_KEY="your_restream_api_key"
RESTREAM_ENABLED=false
RESTREAM_PLATFORMS="youtube,twitch"
# Third-party APIs
EXTERNAL_API_TIMEOUT=30
EXTERNAL_API_RETRY_ATTEMPTS=3
```
## Monitoring and Logging
### Logging Configuration
```bash
# Log Levels
LOG_LEVEL="INFO" # DEBUG, INFO, WARNING, ERROR, CRITICAL
LOG_FORMAT="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
# Log Files
LOG_FILE="/var/log/wifi-densepose/app.log"
LOG_MAX_SIZE=10485760 # 10MB
LOG_BACKUP_COUNT=5
# Structured Logging
LOG_JSON_FORMAT=true
LOG_CORRELATION_ID=true
```
### Metrics and Monitoring
```bash
# Prometheus Metrics
METRICS_ENABLED=true
METRICS_PORT=9090
METRICS_PATH="/metrics"
# Health Checks
HEALTH_CHECK_INTERVAL=30
HEALTH_CHECK_TIMEOUT=10
DEEP_HEALTH_CHECKS=true
# Performance Monitoring
PERFORMANCE_MONITORING=true
SLOW_QUERY_LOGGING=true
RESOURCE_MONITORING=true
```
## Advanced Configuration
### Custom Model Configuration
```yaml
# config/models/custom_model.yaml
model:
name: "custom_densepose_v2"
path: "./models/custom_densepose_v2.pth"
type: "pytorch"
preprocessing:
input_size: [256, 256]
normalization:
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
inference:
batch_size: 32
device: "cuda:0"
precision: "fp16" # fp32, fp16, int8
postprocessing:
confidence_threshold: 0.7
nms_threshold: 0.5
max_detections: 10
```
### Environment-Specific Overrides
```bash
# config/environments/production.env
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=WARNING
WORKERS=8
POSE_PROCESSING_BATCH_SIZE=64
ENABLE_TEST_ENDPOINTS=false
MOCK_HARDWARE=false
```
```bash
# config/environments/development.env
ENVIRONMENT=development
DEBUG=true
LOG_LEVEL=DEBUG
WORKERS=1
RELOAD=true
MOCK_HARDWARE=true
ENABLE_TEST_ENDPOINTS=true
```
### Configuration Validation
The system automatically validates configuration on startup:
```bash
# Run configuration validation
python -m src.config.validate
# Check specific configuration
python -c "
from src.config.settings import get_settings, validate_settings
settings = get_settings()
issues = validate_settings(settings)
if issues:
print('Configuration issues:')
for issue in issues:
print(f' - {issue}')
else:
print('Configuration is valid')
"
```
### Dynamic Configuration Updates
Some settings can be updated without restarting the system:
```bash
# Update detection settings
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{
"detection": {
"confidence_threshold": 0.8,
"max_persons": 3
}
}'
# Update alert settings
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{
"alerts": {
"fall_detection": {
"sensitivity": 0.9
}
}
}'
```
## Configuration Best Practices
### Security Best Practices
1. **Use Strong Secret Keys**: Generate cryptographically secure keys
2. **Restrict CORS Origins**: Don't use wildcards in production
3. **Enable Rate Limiting**: Protect against abuse
4. **Use HTTPS**: Encrypt all communications
5. **Regular Key Rotation**: Rotate API keys and JWT secrets
### Performance Best Practices
1. **Right-size Resources**: Allocate appropriate CPU/memory
2. **Use GPU Acceleration**: Enable CUDA for neural network processing
3. **Optimize Batch Sizes**: Balance throughput and latency
4. **Configure Caching**: Use Redis for frequently accessed data
5. **Monitor Resource Usage**: Set up alerts for resource exhaustion
### Operational Best Practices
1. **Environment Separation**: Use different configs for dev/staging/prod
2. **Configuration Validation**: Validate settings before deployment
3. **Backup Configurations**: Version control all configuration files
4. **Document Changes**: Maintain change logs for configuration updates
5. **Test Configuration**: Validate configuration in staging environment
---
For more specific configuration examples, see:
- [Hardware Setup Guide](../hardware/router-setup.md)
- [API Reference](api-reference.md)
- [Deployment Guide](../developer/deployment-guide.md)
archive/v1/docs/user-guide/getting-started.md
+501
View File
@@ -0,0 +1,501 @@
# Getting Started with WiFi-DensePose
## Overview
WiFi-DensePose is a revolutionary privacy-preserving human pose estimation system that transforms commodity WiFi infrastructure into a powerful human sensing platform. This guide will help you install, configure, and start using the system.
## Table of Contents
1. [System Requirements](#system-requirements)
2. [Installation](#installation)
3. [Quick Start](#quick-start)
4. [Basic Configuration](#basic-configuration)
5. [First Pose Detection](#first-pose-detection)
6. [Troubleshooting](#troubleshooting)
7. [Next Steps](#next-steps)
## System Requirements
### Hardware Requirements
#### WiFi Router Requirements
- **Compatible Hardware**: Atheros-based routers (TP-Link Archer series, Netgear Nighthawk), Intel 5300 NIC-based systems, or ASUS RT-AC68U series
- **Antenna Configuration**: Minimum 3×3 MIMO antenna configuration
- **Frequency Bands**: 2.4GHz and 5GHz support
- **Firmware**: OpenWRT firmware compatibility with CSI extraction patches
#### Processing Hardware
- **CPU**: Multi-core processor (4+ cores recommended)
- **RAM**: 8GB minimum, 16GB recommended
- **Storage**: 50GB available space
- **Network**: Gigabit Ethernet for CSI data streams
- **GPU** (Optional): NVIDIA GPU with CUDA capability and 4GB+ memory for real-time processing
### Software Requirements
#### Operating System
- **Primary**: Linux (Ubuntu 20.04+, CentOS 8+)
- **Secondary**: Windows 10/11 with WSL2
- **Container**: Docker support for deployment
#### Runtime Dependencies
- Python 3.8+
- PyTorch (GPU-accelerated recommended)
- OpenCV
- FFmpeg
- FastAPI
## Installation
### Method 1: Docker Installation (Recommended)
#### Prerequisites
```bash
# Install Docker and Docker Compose
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
#### Download and Setup
```bash
# Clone the repository
git clone https://github.com/your-org/wifi-densepose.git
cd wifi-densepose
# Copy environment configuration
cp .env.example .env
# Edit configuration (see Configuration section)
nano .env
# Start the system
docker-compose up -d
```
### Method 2: Native Installation
#### Install System Dependencies
```bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y python3.9 python3.9-pip python3.9-venv
sudo apt install -y build-essential cmake
sudo apt install -y libopencv-dev ffmpeg
# CentOS/RHEL
sudo yum update
sudo yum install -y python39 python39-pip
sudo yum groupinstall -y "Development Tools"
sudo yum install -y opencv-devel ffmpeg
```
#### Install Python Dependencies
```bash
# Create virtual environment
python3.9 -m venv venv
source venv/bin/activate
# Install requirements
pip install -r requirements.txt
# Install PyTorch with CUDA support (if GPU available)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```
#### Install WiFi-DensePose
```bash
# Install in development mode
pip install -e .
# Or install from PyPI (when available)
pip install wifi-densepose
```
## Quick Start
### 1. Environment Configuration
Create and configure your environment file:
```bash
# Copy the example configuration
cp .env.example .env
```
Edit the `.env` file with your settings:
```bash
# Application settings
APP_NAME="WiFi-DensePose API"
VERSION="1.0.0"
ENVIRONMENT="development"
DEBUG=true
# Server settings
HOST="0.0.0.0"
PORT=8000
# Security settings (CHANGE IN PRODUCTION!)
SECRET_KEY="your-secret-key-here"
JWT_EXPIRE_HOURS=24
# Hardware settings
WIFI_INTERFACE="wlan0"
CSI_BUFFER_SIZE=1000
MOCK_HARDWARE=true # Set to false when using real hardware
# Pose estimation settings
POSE_CONFIDENCE_THRESHOLD=0.5
POSE_MAX_PERSONS=5
# Storage settings
DATA_STORAGE_PATH="./data"
MODEL_STORAGE_PATH="./models"
```
### 2. Start the System
#### Using Docker
```bash
# Start all services
docker-compose up -d
# Check service status
docker-compose ps
# View logs
docker-compose logs -f
```
#### Using Native Installation
```bash
# Activate virtual environment
source venv/bin/activate
# Start the API server
python -m src.api.main
# Or use uvicorn directly
uvicorn src.api.main:app --host 0.0.0.0 --port 8000 --reload
```
### 3. Verify Installation
Check that the system is running:
```bash
# Check API health
curl http://localhost:8000/health
# Expected response:
# {"status": "healthy", "timestamp": "2025-01-07T10:00:00Z"}
```
Access the web interface:
- **API Documentation**: http://localhost:8000/docs
- **Alternative Docs**: http://localhost:8000/redoc
- **Health Check**: http://localhost:8000/health
## Basic Configuration
### Domain Configuration
WiFi-DensePose supports different domain-specific configurations:
#### Healthcare Domain
```bash
# Set healthcare-specific settings
export DOMAIN="healthcare"
export POSE_CONFIDENCE_THRESHOLD=0.8
export ENABLE_FALL_DETECTION=true
export ALERT_SENSITIVITY=0.9
```
#### Retail Domain
```bash
# Set retail-specific settings
export DOMAIN="retail"
export POSE_CONFIDENCE_THRESHOLD=0.7
export ENABLE_TRAFFIC_ANALYTICS=true
export ZONE_TRACKING=true
```
#### Security Domain
```bash
# Set security-specific settings
export DOMAIN="security"
export POSE_CONFIDENCE_THRESHOLD=0.9
export ENABLE_INTRUSION_DETECTION=true
export ALERT_IMMEDIATE=true
```
### Router Configuration
#### Configure WiFi Routers for CSI Extraction
1. **Flash OpenWRT Firmware**:
```bash
# Download OpenWRT firmware for your router model
wget https://downloads.openwrt.org/releases/22.03.0/targets/...
# Flash firmware (router-specific process)
# Follow your router's flashing instructions
```
2. **Install CSI Extraction Patches**:
```bash
# SSH into router
ssh root@192.168.1.1
# Install CSI tools
opkg update
opkg install csi-tools
# Configure CSI extraction
echo "csi_enable=1" >> /etc/config/wireless
echo "csi_rate=30" >> /etc/config/wireless
```
3. **Configure Network Settings**:
```bash
# Set router to monitor mode
iwconfig wlan0 mode monitor
# Start CSI data streaming
csi_tool -i wlan0 -d 192.168.1.100 -p 5500
```
### Database Configuration
#### SQLite (Development)
```bash
# Default SQLite database (no additional configuration needed)
DATABASE_URL="sqlite:///./data/wifi_densepose.db"
```
#### PostgreSQL (Production)
```bash
# Install PostgreSQL with TimescaleDB extension
sudo apt install postgresql-14 postgresql-14-timescaledb
# Configure database
DATABASE_URL="postgresql://user:password@localhost:5432/wifi_densepose"
DATABASE_POOL_SIZE=10
DATABASE_MAX_OVERFLOW=20
```
#### Redis (Caching)
```bash
# Install Redis
sudo apt install redis-server
# Configure Redis
REDIS_URL="redis://localhost:6379/0"
REDIS_PASSWORD="" # Set password for production
```
## First Pose Detection
### 1. Start the System
```bash
# Using Docker
docker-compose up -d
# Using native installation
python -m src.api.main
```
### 2. Initialize Hardware
```bash
# Check system status
curl http://localhost:8000/api/v1/system/status
# Start pose estimation system
curl -X POST http://localhost:8000/api/v1/system/start \
-H "Content-Type: application/json" \
-d '{
"configuration": {
"domain": "general",
"environment_id": "room_001",
"calibration_required": true
}
}'
```
### 3. Get Pose Data
#### REST API
```bash
# Get latest pose data
curl http://localhost:8000/api/v1/pose/latest
# Get historical data
curl "http://localhost:8000/api/v1/pose/history?limit=10"
```
#### WebSocket Streaming
```javascript
// Connect to WebSocket
const ws = new WebSocket('ws://localhost:8000/ws/pose');
// Subscribe to pose updates
ws.onopen = function() {
ws.send(JSON.stringify({
type: 'subscribe',
channel: 'pose_updates',
filters: {
min_confidence: 0.7
}
}));
};
// Handle pose data
ws.onmessage = function(event) {
const data = JSON.parse(event.data);
console.log('Pose data:', data);
};
```
### 4. View Results
Access the web dashboard:
- **Main Dashboard**: http://localhost:8000/dashboard
- **Real-time View**: http://localhost:8000/dashboard/live
- **Analytics**: http://localhost:8000/dashboard/analytics
## Troubleshooting
### Common Issues
#### 1. System Won't Start
```bash
# Check logs
docker-compose logs
# Common solutions:
# - Verify port 8000 is available
# - Check environment variables
# - Ensure sufficient disk space
```
#### 2. No Pose Data
```bash
# Check hardware status
curl http://localhost:8000/api/v1/system/status
# Verify router connectivity
ping 192.168.1.1
# Check CSI data reception
netstat -an | grep 5500
```
#### 3. Poor Detection Accuracy
```bash
# Adjust confidence threshold
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{"detection": {"confidence_threshold": 0.6}}'
# Recalibrate environment
curl -X POST http://localhost:8000/api/v1/system/calibrate
```
#### 4. High CPU/Memory Usage
```bash
# Check resource usage
docker stats
# Optimize settings
export POSE_PROCESSING_BATCH_SIZE=16
export STREAM_FPS=15
```
### Getting Help
#### Log Analysis
```bash
# View application logs
docker-compose logs wifi-densepose-api
# View system logs
journalctl -u wifi-densepose
# Enable debug logging
export LOG_LEVEL="DEBUG"
```
#### Health Checks
```bash
# Comprehensive system check
curl http://localhost:8000/api/v1/system/status
# Component-specific checks
curl http://localhost:8000/api/v1/hardware/status
curl http://localhost:8000/api/v1/processing/status
```
#### Support Resources
- **Documentation**: [docs/](../README.md)
- **API Reference**: [api-reference.md](api-reference.md)
- **Troubleshooting Guide**: [troubleshooting.md](troubleshooting.md)
- **GitHub Issues**: https://github.com/your-org/wifi-densepose/issues
## Next Steps
### 1. Configure for Your Domain
- Review [configuration.md](configuration.md) for domain-specific settings
- Set up alerts and notifications
- Configure external integrations
### 2. Integrate with Your Applications
- Review [API Reference](api-reference.md)
- Set up webhooks for events
- Configure MQTT for IoT integration
### 3. Deploy to Production
- Review [deployment guide](../developer/deployment-guide.md)
- Set up monitoring and alerting
- Configure backup and recovery
### 4. Optimize Performance
- Tune processing parameters
- Set up GPU acceleration
- Configure load balancing
## Security Considerations
### Development Environment
- Use strong secret keys
- Enable authentication
- Restrict network access
### Production Environment
- Use HTTPS/TLS encryption
- Configure firewall rules
- Set up audit logging
- Regular security updates
## Performance Tips
### Hardware Optimization
- Use SSD storage for better I/O performance
- Ensure adequate cooling for continuous operation
- Use dedicated network interface for CSI data
### Software Optimization
- Enable GPU acceleration when available
- Tune batch sizes for your hardware
- Configure appropriate worker processes
- Use Redis for caching frequently accessed data
---
**Congratulations!** You now have WiFi-DensePose up and running. Continue with the [Configuration Guide](configuration.md) to customize the system for your specific needs.
archive/v1/docs/user-guide/troubleshooting.md
+948
View File
@@ -0,0 +1,948 @@
# Troubleshooting Guide
## Overview
This guide provides solutions to common issues encountered when using the WiFi-DensePose system, including installation problems, hardware connectivity issues, performance optimization, and error resolution.
## Table of Contents
1. [Quick Diagnostics](#quick-diagnostics)
2. [Installation Issues](#installation-issues)
3. [Hardware Problems](#hardware-problems)
4. [Performance Issues](#performance-issues)
5. [API and Connectivity Issues](#api-and-connectivity-issues)
6. [Data Quality Issues](#data-quality-issues)
7. [System Errors](#system-errors)
8. [Domain-Specific Issues](#domain-specific-issues)
9. [Advanced Troubleshooting](#advanced-troubleshooting)
10. [Getting Support](#getting-support)
## Quick Diagnostics
### System Health Check
Run a comprehensive system health check to identify issues:
```bash
# Check system status
curl http://localhost:8000/api/v1/system/status
# Run built-in diagnostics
curl http://localhost:8000/api/v1/system/diagnostics
# Check component health
curl http://localhost:8000/api/v1/health
```
### Log Analysis
Check system logs for error patterns:
```bash
# View recent logs
docker-compose logs --tail=100 wifi-densepose-api
# Search for errors
docker-compose logs | grep -i error
# Check specific component logs
docker-compose logs neural-network
docker-compose logs csi-processor
```
### Resource Monitoring
Monitor system resources:
```bash
# Check Docker container resources
docker stats
# Check system resources
htop
nvidia-smi # For GPU monitoring
# Check disk space
df -h
```
## Installation Issues
### Docker Installation Problems
#### Issue: Docker Compose Fails to Start
**Symptoms:**
- Services fail to start
- Port conflicts
- Permission errors
**Solutions:**
1. **Check Port Availability:**
```bash
# Check if port 8000 is in use
netstat -tulpn | grep :8000
lsof -i :8000
# Kill process using the port
sudo kill -9 <PID>
```
2. **Fix Permission Issues:**
```bash
# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker
# Fix file permissions
sudo chown -R $USER:$USER .
```
3. **Update Docker Compose:**
```bash
# Update Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
#### Issue: Out of Disk Space
**Symptoms:**
- Build failures
- Container crashes
- Database errors
**Solutions:**
1. **Clean Docker Resources:**
```bash
# Remove unused containers, networks, images
docker system prune -a
# Remove unused volumes
docker volume prune
# Check disk usage
docker system df
```
2. **Configure Storage Location:**
```bash
# Edit docker-compose.yml to use external storage
volumes:
- /external/storage/data:/app/data
- /external/storage/models:/app/models
```
### Native Installation Problems
#### Issue: Python Dependencies Fail to Install
**Symptoms:**
- pip install errors
- Compilation failures
- Missing system libraries
**Solutions:**
1. **Install System Dependencies:**
```bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y build-essential cmake python3-dev
sudo apt install -y libopencv-dev libffi-dev libssl-dev
# CentOS/RHEL
sudo yum groupinstall -y "Development Tools"
sudo yum install -y python3-devel opencv-devel
```
2. **Use Virtual Environment:**
```bash
# Create clean virtual environment
python3 -m venv venv_clean
source venv_clean/bin/activate
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
```
3. **Install PyTorch Separately:**
```bash
# Install PyTorch with specific CUDA version
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
# Or CPU-only version
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
```
#### Issue: CUDA/GPU Setup Problems
**Symptoms:**
- GPU not detected
- CUDA version mismatch
- Out of GPU memory
**Solutions:**
1. **Verify CUDA Installation:**
```bash
# Check CUDA version
nvcc --version
nvidia-smi
# Check PyTorch CUDA support
python -c "import torch; print(torch.cuda.is_available())"
```
2. **Install Correct CUDA Version:**
```bash
# Install CUDA 11.8 (example)
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run
```
3. **Configure GPU Memory:**
```bash
# Set GPU memory limit
export CUDA_VISIBLE_DEVICES=0
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
```
## Hardware Problems
### Router Connectivity Issues
#### Issue: Cannot Connect to Router
**Symptoms:**
- No CSI data received
- Connection timeouts
- Authentication failures
**Solutions:**
1. **Verify Network Connectivity:**
```bash
# Ping router
ping 192.168.1.1
# Check SSH access
ssh root@192.168.1.1
# Test CSI port
telnet 192.168.1.1 5500
```
2. **Check Router Configuration:**
```bash
# SSH into router and check CSI tools
ssh root@192.168.1.1
csi_tool --status
# Restart CSI service
/etc/init.d/csi restart
```
3. **Verify Firewall Settings:**
```bash
# Check iptables rules
iptables -L
# Allow CSI port
iptables -A INPUT -p tcp --dport 5500 -j ACCEPT
```
#### Issue: Poor CSI Data Quality
**Symptoms:**
- High packet loss
- Inconsistent data rates
- Signal interference
**Solutions:**
1. **Optimize Router Placement:**
```bash
# Check signal strength
iwconfig wlan0
# Analyze interference
iwlist wlan0 scan | grep -E "(ESSID|Frequency|Quality)"
```
2. **Adjust CSI Parameters:**
```bash
# Reduce sampling rate
echo "csi_rate=20" >> /etc/config/wireless
# Change channel
echo "channel=6" >> /etc/config/wireless
uci commit wireless
wifi reload
```
3. **Monitor Data Quality:**
```bash
# Check CSI data statistics
curl http://localhost:8000/api/v1/hardware/csi/stats
# View real-time quality metrics
curl http://localhost:8000/api/v1/hardware/status
```
### Hardware Resource Issues
#### Issue: High CPU Usage
**Symptoms:**
- System slowdown
- Processing delays
- High temperature
**Solutions:**
1. **Optimize Processing Settings:**
```bash
# Reduce batch size
export POSE_PROCESSING_BATCH_SIZE=16
# Lower frame rate
export STREAM_FPS=15
# Disable unnecessary features
export ENABLE_HISTORICAL_DATA=false
```
2. **Scale Resources:**
```bash
# Increase worker processes
export WORKERS=4
# Use process affinity
taskset -c 0-3 python -m src.api.main
```
#### Issue: GPU Memory Errors
**Symptoms:**
- CUDA out of memory errors
- Model loading failures
- Inference crashes
**Solutions:**
1. **Optimize GPU Usage:**
```bash
# Reduce batch size
export POSE_PROCESSING_BATCH_SIZE=8
# Enable mixed precision
export ENABLE_MIXED_PRECISION=true
# Clear GPU cache
python -c "import torch; torch.cuda.empty_cache()"
```
2. **Monitor GPU Memory:**
```bash
# Watch GPU memory usage
watch -n 1 nvidia-smi
# Check memory allocation
python -c "
import torch
print(f'Allocated: {torch.cuda.memory_allocated()/1024**3:.2f} GB')
print(f'Cached: {torch.cuda.memory_reserved()/1024**3:.2f} GB')
"
```
## Performance Issues
### Slow Pose Detection
#### Issue: Low Processing Frame Rate
**Symptoms:**
- FPS below expected rate
- High latency
- Delayed responses
**Solutions:**
1. **Optimize Neural Network:**
```bash
# Use TensorRT optimization
export ENABLE_TENSORRT=true
# Enable model quantization
export MODEL_QUANTIZATION=int8
# Use smaller model variant
export POSE_MODEL_PATH="./models/densepose_mobile.pth"
```
2. **Tune Processing Pipeline:**
```bash
# Increase batch size (if GPU memory allows)
export POSE_PROCESSING_BATCH_SIZE=64
# Reduce input resolution
export INPUT_RESOLUTION=256
# Skip frames for real-time processing
export FRAME_SKIP_RATIO=2
```
3. **Parallel Processing:**
```bash
# Enable multi-threading
export OMP_NUM_THREADS=4
export MKL_NUM_THREADS=4
# Use multiple GPU devices
export CUDA_VISIBLE_DEVICES=0,1
```
### Memory Issues
#### Issue: High Memory Usage
**Symptoms:**
- System running out of RAM
- Swap usage increasing
- OOM killer activated
**Solutions:**
1. **Optimize Memory Usage:**
```bash
# Reduce buffer sizes
export CSI_BUFFER_SIZE=500
export STREAM_BUFFER_SIZE=50
# Limit historical data retention
export DATA_RETENTION_HOURS=24
# Enable memory mapping for large files
export USE_MEMORY_MAPPING=true
```
2. **Configure Swap:**
```bash
# Add swap space
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
```
## API and Connectivity Issues
### Authentication Problems
#### Issue: JWT Token Errors
**Symptoms:**
- 401 Unauthorized responses
- Token expired errors
- Invalid signature errors
**Solutions:**
1. **Verify Token Configuration:**
```bash
# Check secret key
echo $SECRET_KEY
# Verify token expiration
curl -X POST http://localhost:8000/api/v1/auth/verify \
-H "Authorization: Bearer <token>"
```
2. **Regenerate Tokens:**
```bash
# Get new token
curl -X POST http://localhost:8000/api/v1/auth/token \
-H "Content-Type: application/json" \
-d '{"username": "admin", "password": "password"}'
```
3. **Check System Time:**
```bash
# Ensure system time is correct
timedatectl status
sudo ntpdate -s time.nist.gov
```
### WebSocket Connection Issues
#### Issue: WebSocket Disconnections
**Symptoms:**
- Frequent disconnections
- Connection timeouts
- No real-time data
**Solutions:**
1. **Adjust WebSocket Settings:**
```bash
# Increase timeout values
export WEBSOCKET_TIMEOUT=600
export WEBSOCKET_PING_INTERVAL=30
# Enable keep-alive
export WEBSOCKET_KEEPALIVE=true
```
2. **Check Network Configuration:**
```bash
# Test WebSocket connection
wscat -c ws://localhost:8000/ws/pose
# Check proxy settings
curl -I http://localhost:8000/ws/pose
```
### Rate Limiting Issues
#### Issue: Rate Limit Exceeded
**Symptoms:**
- 429 Too Many Requests errors
- API calls being rejected
- Slow response times
**Solutions:**
1. **Adjust Rate Limits:**
```bash
# Increase rate limits
export RATE_LIMIT_REQUESTS=1000
export RATE_LIMIT_WINDOW=3600
# Disable rate limiting for development
export ENABLE_RATE_LIMITING=false
```
2. **Implement Request Batching:**
```python
# Batch multiple requests
def batch_requests(requests, batch_size=10):
for i in range(0, len(requests), batch_size):
batch = requests[i:i+batch_size]
# Process batch
time.sleep(1) # Rate limiting delay
```
## Data Quality Issues
### Poor Detection Accuracy
#### Issue: Low Confidence Scores
**Symptoms:**
- Many false positives
- Missing detections
- Inconsistent tracking
**Solutions:**
1. **Adjust Detection Thresholds:**
```bash
# Increase confidence threshold
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{"detection": {"confidence_threshold": 0.8}}'
```
2. **Improve Environment Setup:**
```bash
# Recalibrate system
curl -X POST http://localhost:8000/api/v1/system/calibrate
# Check for interference
curl http://localhost:8000/api/v1/hardware/interference
```
3. **Optimize Model Parameters:**
```bash
# Use domain-specific model
export POSE_MODEL_PATH="./models/healthcare_optimized.pth"
# Enable post-processing filters
export ENABLE_TEMPORAL_SMOOTHING=true
export ENABLE_OUTLIER_FILTERING=true
```
### Tracking Issues
#### Issue: Person ID Switching
**Symptoms:**
- IDs change frequently
- Lost tracks
- Duplicate persons
**Solutions:**
1. **Tune Tracking Parameters:**
```bash
# Adjust tracking thresholds
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{
"tracking": {
"max_age": 30,
"min_hits": 3,
"iou_threshold": 0.3
}
}'
```
2. **Improve Detection Consistency:**
```bash
# Enable temporal smoothing
export ENABLE_TEMPORAL_SMOOTHING=true
# Use appearance features
export USE_APPEARANCE_FEATURES=true
```
## System Errors
### Database Issues
#### Issue: Database Connection Errors
**Symptoms:**
- Connection refused errors
- Timeout errors
- Data not persisting
**Solutions:**
1. **Check Database Status:**
```bash
# PostgreSQL
sudo systemctl status postgresql
sudo -u postgres psql -c "SELECT version();"
# SQLite
ls -la ./data/wifi_densepose.db
sqlite3 ./data/wifi_densepose.db ".tables"
```
2. **Fix Connection Issues:**
```bash
# Reset database connection
export DATABASE_URL="postgresql://user:password@localhost:5432/wifi_densepose"
# Restart database service
sudo systemctl restart postgresql
```
3. **Database Migration:**
```bash
# Run database migrations
python -m src.database.migrate
# Reset database (WARNING: Data loss)
python -m src.database.reset --confirm
```
### Service Crashes
#### Issue: API Service Crashes
**Symptoms:**
- Service stops unexpectedly
- No response from API
- Error 502/503 responses
**Solutions:**
1. **Check Service Logs:**
```bash
# View crash logs
journalctl -u wifi-densepose -f
# Check for segmentation faults
dmesg | grep -i "segfault"
```
2. **Restart Services:**
```bash
# Restart with Docker
docker-compose restart wifi-densepose-api
# Restart native service
sudo systemctl restart wifi-densepose
```
3. **Debug Memory Issues:**
```bash
# Run with memory debugging
valgrind --tool=memcheck python -m src.api.main
# Check for memory leaks
python -m tracemalloc
```
## Domain-Specific Issues
### Healthcare Domain Issues
#### Issue: Fall Detection False Alarms
**Symptoms:**
- Too many fall alerts
- Normal activities triggering alerts
- Delayed detection
**Solutions:**
1. **Adjust Sensitivity:**
```bash
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{
"alerts": {
"fall_detection": {
"sensitivity": 0.7,
"notification_delay_seconds": 10
}
}
}'
```
2. **Improve Training Data:**
```bash
# Collect domain-specific training data
python -m src.training.collect_healthcare_data
# Retrain model with healthcare data
python -m src.training.train_healthcare_model
```
### Retail Domain Issues
#### Issue: Inaccurate Traffic Counting
**Symptoms:**
- Wrong visitor counts
- Missing entries/exits
- Double counting
**Solutions:**
1. **Calibrate Zone Detection:**
```bash
# Define entrance/exit zones
curl -X PUT http://localhost:8000/api/v1/config \
-H "Content-Type: application/json" \
-d '{
"zones": {
"entrance": {
"coordinates": [[0, 0], [100, 50]],
"type": "entrance"
}
}
}'
```
2. **Optimize Tracking:**
```bash
# Enable zone-based tracking
export ENABLE_ZONE_TRACKING=true
# Adjust dwell time thresholds
export MIN_DWELL_TIME_SECONDS=5
```
## Advanced Troubleshooting
### Performance Profiling
#### CPU Profiling
```bash
# Profile Python code
python -m cProfile -o profile.stats -m src.api.main
# Analyze profile
python -c "
import pstats
p = pstats.Stats('profile.stats')
p.sort_stats('cumulative').print_stats(20)
"
```
#### GPU Profiling
```bash
# Profile CUDA kernels
nvprof python -m src.neural_network.inference
# Use PyTorch profiler
python -c "
import torch
with torch.profiler.profile() as prof:
# Your code here
pass
print(prof.key_averages().table())
"
```
### Network Debugging
#### Packet Capture
```bash
# Capture CSI packets
sudo tcpdump -i eth0 port 5500 -w csi_capture.pcap
# Analyze with Wireshark
wireshark csi_capture.pcap
```
#### Network Latency Testing
```bash
# Test network latency
ping -c 100 192.168.1.1 | tail -1
# Test bandwidth
iperf3 -c 192.168.1.1 -t 60
```
### System Monitoring
#### Real-time Monitoring
```bash
# Monitor system resources
htop
iotop
nethogs
# Monitor GPU
nvidia-smi -l 1
# Monitor Docker containers
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"
```
#### Log Aggregation
```bash
# Centralized logging with ELK stack
docker run -d --name elasticsearch elasticsearch:7.17.0
docker run -d --name kibana kibana:7.17.0
# Configure log shipping
echo 'LOGGING_DRIVER=syslog' >> .env
echo 'SYSLOG_ADDRESS=tcp://localhost:514' >> .env
```
## Getting Support
### Collecting Diagnostic Information
Before contacting support, collect the following information:
```bash
# System information
uname -a
cat /etc/os-release
docker --version
python --version
# Application logs
docker-compose logs --tail=1000 > logs.txt
# Configuration
cat .env > config.txt
curl http://localhost:8000/api/v1/system/status > status.json
# Hardware information
lscpu
free -h
nvidia-smi > gpu_info.txt
```
### Support Channels
1. **Documentation**: Check the comprehensive documentation first
2. **GitHub Issues**: Report bugs and feature requests
3. **Community Forum**: Ask questions and share solutions
4. **Enterprise Support**: For commercial deployments
### Creating Effective Bug Reports
Include the following information:
1. **Environment Details**:
- Operating system and version
- Hardware specifications
- Docker/Python versions
2. **Steps to Reproduce**:
- Exact commands or API calls
- Configuration settings
- Input data characteristics
3. **Expected vs Actual Behavior**:
- What you expected to happen
- What actually happened
- Error messages and logs
4. **Additional Context**:
- Screenshots or videos
- Configuration files
- System logs
### Emergency Procedures
For critical production issues:
1. **Immediate Actions**:
```bash
# Stop the system safely
curl -X POST http://localhost:8000/api/v1/system/stop
# Backup current data
cp -r ./data ./data_backup_$(date +%Y%m%d_%H%M%S)
# Restart with minimal configuration
export MOCK_HARDWARE=true
docker-compose up -d
```
2. **Rollback Procedures**:
```bash
# Rollback to previous version
git checkout <previous-tag>
docker-compose down
docker-compose up -d
# Restore data backup
rm -rf ./data
cp -r ./data_backup_<timestamp> ./data
```
3. **Contact Information**:
- Emergency support: support@wifi-densepose.com
- Phone: +1-555-SUPPORT
- Slack: #wifi-densepose-emergency
---
**Remember**: Most issues can be resolved by checking logs, verifying configuration, and ensuring proper hardware setup. When in doubt, start with the basic diagnostics and work your way through the troubleshooting steps systematically.
For additional help, see:
- [Configuration Guide](configuration.md)
- [API Reference](api-reference.md)
- [Hardware Setup Guide](../hardware/router-setup.md)
- [Deployment Guide](../developer/deployment-guide.md)
archive/v1/docs/user_guide.md
+770
View File
@@ -0,0 +1,770 @@
# WiFi-DensePose User Guide
## Table of Contents
1. [Overview](#overview)
2. [Installation](#installation)
3. [Quick Start](#quick-start)
4. [Configuration](#configuration)
5. [Basic Usage](#basic-usage)
6. [Advanced Features](#advanced-features)
7. [Examples](#examples)
8. [Best Practices](#best-practices)
## Overview
WiFi-DensePose is a revolutionary privacy-preserving human pose estimation system that leverages Channel State Information (CSI) data from standard WiFi infrastructure. Unlike traditional camera-based systems, WiFi-DensePose provides real-time pose detection while maintaining complete privacy.
### Key Features
- **Privacy-First Design**: No cameras or visual data required
- **Real-Time Processing**: Sub-50ms latency with 30 FPS pose estimation
- **Multi-Person Tracking**: Simultaneous tracking of up to 10 individuals
- **Domain-Specific Optimization**: Tailored for healthcare, fitness, retail, and security
- **Enterprise-Ready**: Production-grade API with authentication and monitoring
- **Hardware Agnostic**: Works with standard WiFi routers and access points
### System Architecture
```
WiFi Routers → CSI Data → Signal Processing → Neural Network → Pose Estimation
↓ ↓ ↓ ↓ ↓
Hardware Data Collection Phase Cleaning DensePose Person Tracking
Interface & Buffering & Filtering Model & Analytics
```
## Installation
### Prerequisites
- **Python**: 3.9 or higher
- **Operating System**: Linux (Ubuntu 18.04+), macOS (10.15+), Windows 10+
- **Memory**: Minimum 4GB RAM, Recommended 8GB+
- **Storage**: 2GB free space for models and data
- **Network**: WiFi interface with CSI capability
### Method 1: Install from PyPI (Recommended)
```bash
# Install the latest stable version
pip install wifi-densepose
# Install with optional dependencies
pip install wifi-densepose[gpu,monitoring,deployment]
# Verify installation
wifi-densepose --version
```
### Method 2: Install from Source
```bash
# Clone the repository
git clone https://github.com/ruvnet/wifi-densepose.git
cd wifi-densepose
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Install in development mode
pip install -e .
```
### Method 3: Docker Installation
```bash
# Pull the latest image
docker pull ruvnet/wifi-densepose:latest
# Run with default configuration
docker run -p 8000:8000 ruvnet/wifi-densepose:latest
# Run with custom configuration
docker run -p 8000:8000 -v $(pwd)/config:/app/config ruvnet/wifi-densepose:latest
```
### Verify Installation
```bash
# Check system information
python -c "import wifi_densepose; wifi_densepose.print_system_info()"
# Test API server
wifi-densepose start --test-mode
# Check health endpoint
curl http://localhost:8000/api/v1/health
```
## Quick Start
### 1. Basic Setup
```bash
# Create configuration file
wifi-densepose init
# Edit configuration (optional)
nano .env
# Start the system
wifi-densepose start
```
### 2. Python API Usage
```python
from wifi_densepose import WiFiDensePose
# Initialize with default configuration
system = WiFiDensePose()
# Start pose estimation
system.start()
# Get latest pose data
poses = system.get_latest_poses()
print(f"Detected {len(poses)} persons")
# Stop the system
system.stop()
```
### 3. REST API Usage
```bash
# Start the API server
wifi-densepose start --api
# Get latest poses
curl http://localhost:8000/api/v1/pose/latest
# Get system status
curl http://localhost:8000/api/v1/system/status
```
### 4. WebSocket Streaming
```python
import asyncio
import websockets
import json
async def stream_poses():
uri = "ws://localhost:8000/ws/pose/stream"
async with websockets.connect(uri) as websocket:
while True:
data = await websocket.recv()
poses = json.loads(data)
print(f"Received: {len(poses['persons'])} persons")
asyncio.run(stream_poses())
```
## Configuration
### Environment Variables
Create a `.env` file in your project directory:
```bash
# Application Settings
APP_NAME=WiFi-DensePose API
VERSION=1.0.0
ENVIRONMENT=production
DEBUG=false
# Server Settings
HOST=0.0.0.0
PORT=8000
WORKERS=4
# Security Settings
SECRET_KEY=your-secure-secret-key-here
JWT_ALGORITHM=HS256
JWT_EXPIRE_HOURS=24
# Hardware Settings
WIFI_INTERFACE=wlan0
CSI_BUFFER_SIZE=1000
HARDWARE_POLLING_INTERVAL=0.1
# Pose Estimation Settings
POSE_CONFIDENCE_THRESHOLD=0.7
POSE_PROCESSING_BATCH_SIZE=32
POSE_MAX_PERSONS=10
# Feature Flags
ENABLE_AUTHENTICATION=true
ENABLE_RATE_LIMITING=true
ENABLE_WEBSOCKETS=true
ENABLE_REAL_TIME_PROCESSING=true
```
### Domain-Specific Configuration
#### Healthcare Configuration
```python
from wifi_densepose.config import Settings
config = Settings(
domain="healthcare",
detection={
"confidence_threshold": 0.8,
"max_persons": 5,
"enable_tracking": True
},
analytics={
"enable_fall_detection": True,
"enable_activity_recognition": True,
"alert_thresholds": {
"fall_confidence": 0.9,
"inactivity_timeout": 300
}
},
privacy={
"data_retention_days": 30,
"anonymize_data": True,
"enable_encryption": True
}
)
```
#### Fitness Configuration
```python
config = Settings(
domain="fitness",
detection={
"confidence_threshold": 0.6,
"max_persons": 20,
"enable_tracking": True
},
analytics={
"enable_activity_recognition": True,
"enable_form_analysis": True,
"metrics": ["rep_count", "form_score", "intensity"]
}
)
```
#### Retail Configuration
```python
config = Settings(
domain="retail",
detection={
"confidence_threshold": 0.7,
"max_persons": 50,
"enable_tracking": True
},
analytics={
"enable_traffic_analytics": True,
"enable_zone_tracking": True,
"heatmap_generation": True
}
)
```
## Basic Usage
### Starting the System
#### Command Line Interface
```bash
# Start with default configuration
wifi-densepose start
# Start with custom configuration
wifi-densepose start --config /path/to/config.yaml
# Start in development mode
wifi-densepose start --dev --reload
# Start with specific domain
wifi-densepose start --domain healthcare
# Start API server only
wifi-densepose start --api-only
```
#### Python API
```python
from wifi_densepose import WiFiDensePose
from wifi_densepose.config import Settings
# Initialize with custom settings
settings = Settings(
pose_confidence_threshold=0.8,
max_persons=5,
enable_gpu=True
)
system = WiFiDensePose(settings=settings)
# Start the system
system.start()
# Check if system is running
if system.is_running():
print("System is active")
# Get system status
status = system.get_status()
print(f"Status: {status}")
```
### Getting Pose Data
#### Latest Poses
```python
# Get the most recent pose data
poses = system.get_latest_poses()
for person in poses:
print(f"Person {person.id}:")
print(f" Confidence: {person.confidence}")
print(f" Keypoints: {len(person.keypoints)}")
print(f" Bounding box: {person.bbox}")
```
#### Historical Data
```python
from datetime import datetime, timedelta
# Get poses from the last hour
end_time = datetime.now()
start_time = end_time - timedelta(hours=1)
history = system.get_pose_history(
start_time=start_time,
end_time=end_time,
min_confidence=0.7
)
print(f"Found {len(history)} pose records")
```
#### Real-Time Streaming
```python
def pose_callback(poses):
"""Callback function for real-time pose updates"""
print(f"Received {len(poses)} poses at {datetime.now()}")
for person in poses:
if person.confidence > 0.8:
print(f"High-confidence detection: Person {person.id}")
# Subscribe to real-time updates
system.subscribe_to_poses(callback=pose_callback)
# Unsubscribe when done
system.unsubscribe_from_poses()
```
### System Control
#### Starting and Stopping
```python
# Start the pose estimation system
system.start()
# Pause processing (keeps connections alive)
system.pause()
# Resume processing
system.resume()
# Stop the system
system.stop()
# Restart with new configuration
system.restart(new_settings)
```
#### Configuration Updates
```python
# Update configuration at runtime
new_config = {
"detection": {
"confidence_threshold": 0.8,
"max_persons": 8
}
}
system.update_config(new_config)
# Get current configuration
current_config = system.get_config()
print(current_config)
```
## Advanced Features
### Multi-Environment Support
```python
# Configure multiple environments
environments = {
"room_001": {
"calibration_file": "/path/to/room_001_cal.json",
"router_ips": ["192.168.1.1", "192.168.1.2"]
},
"room_002": {
"calibration_file": "/path/to/room_002_cal.json",
"router_ips": ["192.168.2.1", "192.168.2.2"]
}
}
# Switch between environments
system.set_environment("room_001")
poses_room1 = system.get_latest_poses()
system.set_environment("room_002")
poses_room2 = system.get_latest_poses()
```
### Custom Analytics
```python
from wifi_densepose.analytics import AnalyticsEngine
# Initialize analytics engine
analytics = AnalyticsEngine(system)
# Enable fall detection
analytics.enable_fall_detection(
sensitivity=0.9,
callback=lambda event: print(f"Fall detected: {event}")
)
# Enable activity recognition
analytics.enable_activity_recognition(
activities=["sitting", "standing", "walking", "running"],
callback=lambda activity: print(f"Activity: {activity}")
)
# Custom analytics function
def custom_analytics(poses):
"""Custom analytics function"""
person_count = len(poses)
avg_confidence = sum(p.confidence for p in poses) / person_count if person_count > 0 else 0
return {
"person_count": person_count,
"average_confidence": avg_confidence,
"timestamp": datetime.now().isoformat()
}
analytics.add_custom_function(custom_analytics)
```
### Hardware Integration
```python
from wifi_densepose.hardware import RouterManager
# Configure router connections
router_manager = RouterManager()
# Add routers
router_manager.add_router(
ip="192.168.1.1",
username="admin",
password="password",
router_type="asus_ac68u"
)
# Check router status
status = router_manager.get_router_status("192.168.1.1")
print(f"Router status: {status}")
# Configure CSI extraction
router_manager.configure_csi_extraction(
router_ip="192.168.1.1",
extraction_rate=30,
target_ip="192.168.1.100",
target_port=5500
)
```
## Examples
### Example 1: Healthcare Monitoring
```python
from wifi_densepose import WiFiDensePose
from wifi_densepose.analytics import FallDetector
import logging
# Configure for healthcare
system = WiFiDensePose(domain="healthcare")
# Set up fall detection
fall_detector = FallDetector(
sensitivity=0.95,
alert_callback=lambda event: send_alert(event)
)
def send_alert(fall_event):
"""Send alert to healthcare staff"""
logging.critical(f"FALL DETECTED: {fall_event}")
# Send notification to staff
# notify_healthcare_staff(fall_event)
# Start monitoring
system.start()
system.add_analytics_module(fall_detector)
print("Healthcare monitoring active...")
```
### Example 2: Fitness Tracking
```python
from wifi_densepose import WiFiDensePose
from wifi_densepose.analytics import ActivityTracker
# Configure for fitness
system = WiFiDensePose(domain="fitness")
# Set up activity tracking
activity_tracker = ActivityTracker(
activities=["squats", "pushups", "jumping_jacks"],
rep_counting=True
)
def workout_callback(activity_data):
"""Handle workout data"""
print(f"Exercise: {activity_data['exercise']}")
print(f"Reps: {activity_data['rep_count']}")
print(f"Form score: {activity_data['form_score']}")
activity_tracker.set_callback(workout_callback)
# Start fitness tracking
system.start()
system.add_analytics_module(activity_tracker)
print("Fitness tracking active...")
```
### Example 3: Retail Analytics
```python
from wifi_densepose import WiFiDensePose
from wifi_densepose.analytics import TrafficAnalyzer
# Configure for retail
system = WiFiDensePose(domain="retail")
# Set up traffic analysis
traffic_analyzer = TrafficAnalyzer(
zones={
"entrance": {"x": 0, "y": 0, "width": 100, "height": 50},
"checkout": {"x": 200, "y": 150, "width": 100, "height": 50},
"electronics": {"x": 50, "y": 100, "width": 150, "height": 100}
}
)
def traffic_callback(traffic_data):
"""Handle traffic analytics"""
print(f"Zone occupancy: {traffic_data['zone_occupancy']}")
print(f"Traffic flow: {traffic_data['flow_patterns']}")
print(f"Dwell times: {traffic_data['dwell_times']}")
traffic_analyzer.set_callback(traffic_callback)
# Start retail analytics
system.start()
system.add_analytics_module(traffic_analyzer)
print("Retail analytics active...")
```
### Example 4: Security Monitoring
```python
from wifi_densepose import WiFiDensePose
from wifi_densepose.analytics import IntrusionDetector
# Configure for security
system = WiFiDensePose(domain="security")
# Set up intrusion detection
intrusion_detector = IntrusionDetector(
restricted_zones=[
{"x": 100, "y": 100, "width": 50, "height": 50, "name": "server_room"},
{"x": 200, "y": 50, "width": 75, "height": 75, "name": "executive_office"}
],
alert_threshold=0.9
)
def security_alert(intrusion_event):
"""Handle security alerts"""
logging.warning(f"INTRUSION DETECTED: {intrusion_event}")
# Trigger security response
# activate_security_protocol(intrusion_event)
intrusion_detector.set_alert_callback(security_alert)
# Start security monitoring
system.start()
system.add_analytics_module(intrusion_detector)
print("Security monitoring active...")
```
## Best Practices
### Performance Optimization
1. **Hardware Configuration**
```python
# Enable GPU acceleration when available
settings = Settings(
enable_gpu=True,
batch_size=64,
mixed_precision=True
)
```
2. **Memory Management**
```python
# Configure appropriate buffer sizes
settings = Settings(
csi_buffer_size=1000,
pose_history_limit=10000,
cleanup_interval=3600 # 1 hour
)
```
3. **Network Optimization**
```python
# Optimize network settings
settings = Settings(
hardware_polling_interval=0.05, # 50ms
network_timeout=5.0,
max_concurrent_connections=100
)
```
### Security Best Practices
1. **Authentication**
```python
# Enable authentication in production
settings = Settings(
enable_authentication=True,
jwt_secret_key="your-secure-secret-key",
jwt_expire_hours=24
)
```
2. **Rate Limiting**
```python
# Configure rate limiting
settings = Settings(
enable_rate_limiting=True,
rate_limit_requests=100,
rate_limit_window=60 # per minute
)
```
3. **Data Privacy**
```python
# Enable privacy features
settings = Settings(
anonymize_data=True,
data_retention_days=30,
enable_encryption=True
)
```
### Monitoring and Logging
1. **Structured Logging**
```python
import logging
from wifi_densepose.logger import setup_logging
# Configure structured logging
setup_logging(
level=logging.INFO,
format="json",
output_file="/var/log/wifi-densepose.log"
)
```
2. **Metrics Collection**
```python
from wifi_densepose.monitoring import MetricsCollector
# Enable metrics collection
metrics = MetricsCollector()
metrics.enable_prometheus_export(port=9090)
```
3. **Health Monitoring**
```python
# Set up health checks
system.enable_health_monitoring(
check_interval=30, # seconds
alert_on_failure=True
)
```
### Error Handling
1. **Graceful Degradation**
```python
try:
system.start()
except HardwareNotAvailableError:
# Fall back to mock mode
system.start(mock_mode=True)
logging.warning("Running in mock mode - no hardware detected")
```
2. **Retry Logic**
```python
from wifi_densepose.utils import retry_on_failure
@retry_on_failure(max_attempts=3, delay=5.0)
def connect_to_router():
return router_manager.connect("192.168.1.1")
```
3. **Circuit Breaker Pattern**
```python
from wifi_densepose.resilience import CircuitBreaker
# Protect against failing services
circuit_breaker = CircuitBreaker(
failure_threshold=5,
recovery_timeout=60
)
@circuit_breaker
def process_csi_data(data):
return csi_processor.process(data)
```
---
For more detailed information, see:
- [API Reference Guide](api_reference.md)
- [Deployment Guide](deployment.md)
- [Troubleshooting Guide](troubleshooting.md)
archive/v1/requirements-lock.txt
+13
View File
@@ -0,0 +1,13 @@
# WiFi-DensePose Pipeline Verification - Pinned Dependencies
# These versions are locked to ensure deterministic pipeline output.
# The proof bundle (v1/data/proof/) depends on exact numerical behavior
# from these libraries. Changing versions may alter floating-point results
# and require regenerating the expected hash.
#
# To update: change versions, run `python v1/data/proof/verify.py --generate-hash`,
# then commit the new expected_features.sha256.
numpy==1.26.4
scipy==1.14.1
pydantic==2.10.4
pydantic-settings==2.7.1
archive/v1/scripts/api_test_results_20250607_122720.json
+1967
View File
File diff suppressed because it is too large Load Diff
archive/v1/scripts/api_test_results_20250607_122856.json
+1991
View File
File diff suppressed because it is too large Load Diff
archive/v1/scripts/api_test_results_20250607_123111.json
+2961
View File
File diff suppressed because it is too large Load Diff
archive/v1/scripts/api_test_results_20250609_161617.json
+2269
View File
File diff suppressed because it is too large Load Diff
archive/v1/scripts/api_test_results_20250609_162928.json
+2289
View File
File diff suppressed because it is too large Load Diff
archive/v1/scripts/test_api_endpoints.py
Executable
+376
View File
@@ -0,0 +1,376 @@
#!/usr/bin/env python3
"""
API Endpoint Testing Script
Tests all WiFi-DensePose API endpoints and provides debugging information.
"""
import asyncio
import json
import sys
import time
import traceback
from datetime import datetime, timedelta
from typing import Dict, List, Any, Optional
import aiohttp
import websockets
from colorama import Fore, Style, init
# Initialize colorama for colored output
init(autoreset=True)
class APITester:
"""Comprehensive API endpoint tester."""
def __init__(self, base_url: str = "http://localhost:8000"):
self.base_url = base_url
self.session = None
self.results = {
"total_tests": 0,
"passed": 0,
"failed": 0,
"errors": [],
"test_details": []
}
async def __aenter__(self):
"""Async context manager entry."""
self.session = aiohttp.ClientSession()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Async context manager exit."""
if self.session:
await self.session.close()
def log_success(self, message: str):
"""Log success message."""
print(f"{Fore.GREEN}✓ {message}{Style.RESET_ALL}")
def log_error(self, message: str):
"""Log error message."""
print(f"{Fore.RED}✗ {message}{Style.RESET_ALL}")
def log_info(self, message: str):
"""Log info message."""
print(f"{Fore.BLUE}ℹ {message}{Style.RESET_ALL}")
def log_warning(self, message: str):
"""Log warning message."""
print(f"{Fore.YELLOW}⚠ {message}{Style.RESET_ALL}")
async def test_endpoint(
self,
method: str,
endpoint: str,
expected_status: int = 200,
data: Optional[Dict] = None,
params: Optional[Dict] = None,
headers: Optional[Dict] = None,
description: str = ""
) -> Dict[str, Any]:
"""Test a single API endpoint."""
self.results["total_tests"] += 1
test_name = f"{method.upper()} {endpoint}"
try:
url = f"{self.base_url}{endpoint}"
# Prepare request
kwargs = {}
if data:
kwargs["json"] = data
if params:
kwargs["params"] = params
if headers:
kwargs["headers"] = headers
# Make request
start_time = time.time()
async with self.session.request(method, url, **kwargs) as response:
response_time = (time.time() - start_time) * 1000
response_text = await response.text()
# Try to parse JSON response
try:
response_data = json.loads(response_text) if response_text else {}
except json.JSONDecodeError:
response_data = {"raw_response": response_text}
# Check status code
status_ok = response.status == expected_status
test_result = {
"test_name": test_name,
"description": description,
"url": url,
"method": method.upper(),
"expected_status": expected_status,
"actual_status": response.status,
"response_time_ms": round(response_time, 2),
"response_data": response_data,
"success": status_ok,
"timestamp": datetime.now().isoformat()
}
if status_ok:
self.results["passed"] += 1
self.log_success(f"{test_name} - {response.status} ({response_time:.1f}ms)")
if description:
print(f" {description}")
else:
self.results["failed"] += 1
self.log_error(f"{test_name} - Expected {expected_status}, got {response.status}")
if description:
print(f" {description}")
print(f" Response: {response_text[:200]}...")
self.results["test_details"].append(test_result)
return test_result
except Exception as e:
self.results["failed"] += 1
error_msg = f"{test_name} - Exception: {str(e)}"
self.log_error(error_msg)
test_result = {
"test_name": test_name,
"description": description,
"url": f"{self.base_url}{endpoint}",
"method": method.upper(),
"expected_status": expected_status,
"actual_status": None,
"response_time_ms": None,
"response_data": None,
"success": False,
"error": str(e),
"traceback": traceback.format_exc(),
"timestamp": datetime.now().isoformat()
}
self.results["errors"].append(error_msg)
self.results["test_details"].append(test_result)
return test_result
async def test_websocket_endpoint(self, endpoint: str, description: str = "") -> Dict[str, Any]:
"""Test WebSocket endpoint."""
self.results["total_tests"] += 1
test_name = f"WebSocket {endpoint}"
try:
ws_url = f"ws://localhost:8000{endpoint}"
start_time = time.time()
async with websockets.connect(ws_url) as websocket:
# Send a test message
test_message = {"type": "subscribe", "zone_ids": ["zone_1"]}
await websocket.send(json.dumps(test_message))
# Wait for response
response = await asyncio.wait_for(websocket.recv(), timeout=3)
response_time = (time.time() - start_time) * 1000
try:
response_data = json.loads(response)
except json.JSONDecodeError:
response_data = {"raw_response": response}
test_result = {
"test_name": test_name,
"description": description,
"url": ws_url,
"method": "WebSocket",
"response_time_ms": round(response_time, 2),
"response_data": response_data,
"success": True,
"timestamp": datetime.now().isoformat()
}
self.results["passed"] += 1
self.log_success(f"{test_name} - Connected ({response_time:.1f}ms)")
if description:
print(f" {description}")
self.results["test_details"].append(test_result)
return test_result
except Exception as e:
self.results["failed"] += 1
error_msg = f"{test_name} - Exception: {str(e)}"
self.log_error(error_msg)
test_result = {
"test_name": test_name,
"description": description,
"url": f"ws://localhost:8000{endpoint}",
"method": "WebSocket",
"response_time_ms": None,
"response_data": None,
"success": False,
"error": str(e),
"traceback": traceback.format_exc(),
"timestamp": datetime.now().isoformat()
}
self.results["errors"].append(error_msg)
self.results["test_details"].append(test_result)
return test_result
async def run_all_tests(self):
"""Run all API endpoint tests."""
print(f"{Fore.CYAN}{'='*60}")
print(f"{Fore.CYAN}WiFi-DensePose API Endpoint Testing")
print(f"{Fore.CYAN}{'='*60}{Style.RESET_ALL}")
print()
# Test Health Endpoints
print(f"{Fore.MAGENTA}Testing Health Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/health/health", description="System health check")
await self.test_endpoint("GET", "/health/ready", description="Readiness check")
print()
# Test Pose Estimation Endpoints
print(f"{Fore.MAGENTA}Testing Pose Estimation Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/api/v1/pose/current", description="Current pose estimation")
await self.test_endpoint("GET", "/api/v1/pose/current",
params={"zone_ids": ["zone_1"], "confidence_threshold": 0.7},
description="Current pose estimation with parameters")
await self.test_endpoint("POST", "/api/v1/pose/analyze", description="Pose analysis (requires auth)")
await self.test_endpoint("GET", "/api/v1/pose/zones/zone_1/occupancy", description="Zone occupancy")
await self.test_endpoint("GET", "/api/v1/pose/zones/summary", description="All zones summary")
print()
# Test Historical Data Endpoints
print(f"{Fore.MAGENTA}Testing Historical Data Endpoints:{Style.RESET_ALL}")
end_time = datetime.now()
start_time = end_time - timedelta(hours=1)
historical_data = {
"start_time": start_time.isoformat(),
"end_time": end_time.isoformat(),
"zone_ids": ["zone_1"],
"aggregation_interval": 300
}
await self.test_endpoint("POST", "/api/v1/pose/historical",
data=historical_data,
description="Historical pose data (requires auth)")
await self.test_endpoint("GET", "/api/v1/pose/activities", description="Recent activities")
await self.test_endpoint("GET", "/api/v1/pose/activities",
params={"zone_id": "zone_1", "limit": 5},
description="Activities for specific zone")
print()
# Test Calibration Endpoints
print(f"{Fore.MAGENTA}Testing Calibration Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/api/v1/pose/calibration/status", description="Calibration status (requires auth)")
await self.test_endpoint("POST", "/api/v1/pose/calibrate", description="Start calibration (requires auth)")
print()
# Test Statistics Endpoints
print(f"{Fore.MAGENTA}Testing Statistics Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/api/v1/pose/stats", description="Pose statistics")
await self.test_endpoint("GET", "/api/v1/pose/stats",
params={"hours": 12}, description="Pose statistics (12 hours)")
print()
# Test Stream Endpoints
print(f"{Fore.MAGENTA}Testing Stream Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/api/v1/stream/status", description="Stream status")
await self.test_endpoint("POST", "/api/v1/stream/start", description="Start streaming (requires auth)")
await self.test_endpoint("POST", "/api/v1/stream/stop", description="Stop streaming (requires auth)")
print()
# Test WebSocket Endpoints
print(f"{Fore.MAGENTA}Testing WebSocket Endpoints:{Style.RESET_ALL}")
await self.test_websocket_endpoint("/api/v1/stream/pose", description="Pose WebSocket")
await self.test_websocket_endpoint("/api/v1/stream/events", description="Events WebSocket")
print()
# Test Documentation Endpoints
print(f"{Fore.MAGENTA}Testing Documentation Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/docs", description="API documentation")
await self.test_endpoint("GET", "/openapi.json", description="OpenAPI schema")
print()
# Test API Info Endpoints
print(f"{Fore.MAGENTA}Testing API Info Endpoints:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/", description="Root endpoint")
await self.test_endpoint("GET", "/api/v1/info", description="API information")
await self.test_endpoint("GET", "/api/v1/status", description="API status")
print()
# Test Error Cases
print(f"{Fore.MAGENTA}Testing Error Cases:{Style.RESET_ALL}")
await self.test_endpoint("GET", "/nonexistent", expected_status=404,
description="Non-existent endpoint")
await self.test_endpoint("POST", "/api/v1/pose/analyze",
data={"invalid": "data"}, expected_status=401,
description="Unauthorized request (no auth)")
print()
def print_summary(self):
"""Print test summary."""
print(f"{Fore.CYAN}{'='*60}")
print(f"{Fore.CYAN}Test Summary")
print(f"{Fore.CYAN}{'='*60}{Style.RESET_ALL}")
total = self.results["total_tests"]
passed = self.results["passed"]
failed = self.results["failed"]
success_rate = (passed / total * 100) if total > 0 else 0
print(f"Total Tests: {total}")
print(f"{Fore.GREEN}Passed: {passed}{Style.RESET_ALL}")
print(f"{Fore.RED}Failed: {failed}{Style.RESET_ALL}")
print(f"Success Rate: {success_rate:.1f}%")
print()
if self.results["errors"]:
print(f"{Fore.RED}Errors:{Style.RESET_ALL}")
for error in self.results["errors"]:
print(f" - {error}")
print()
# Save detailed results to file
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
results_file = f"scripts/api_test_results_{timestamp}.json"
try:
with open(results_file, 'w') as f:
json.dump(self.results, f, indent=2, default=str)
print(f"Detailed results saved to: {results_file}")
except Exception as e:
self.log_warning(f"Could not save results file: {e}")
return failed == 0
async def main():
"""Main test function."""
try:
async with APITester() as tester:
await tester.run_all_tests()
success = tester.print_summary()
# Exit with appropriate code
sys.exit(0 if success else 1)
except KeyboardInterrupt:
print(f"\n{Fore.YELLOW}Tests interrupted by user{Style.RESET_ALL}")
sys.exit(1)
except Exception as e:
print(f"\n{Fore.RED}Fatal error: {e}{Style.RESET_ALL}")
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
# Check if required packages are available
try:
import aiohttp
import websockets
import colorama
except ImportError as e:
print(f"Missing required package: {e}")
print("Install with: pip install aiohttp websockets colorama")
sys.exit(1)
# Run tests
asyncio.run(main())
archive/v1/scripts/test_monitoring.py
Executable
+366
View File
@@ -0,0 +1,366 @@
#!/usr/bin/env python3
"""
Test script for WiFi-DensePose monitoring functionality
"""
import asyncio
import aiohttp
import json
import sys
from datetime import datetime
from typing import Dict, Any, List
import time
class MonitoringTester:
"""Test monitoring endpoints and metrics collection."""
def __init__(self, base_url: str = "http://localhost:8000"):
self.base_url = base_url
self.session = None
self.results = []
async def setup(self):
"""Setup test session."""
self.session = aiohttp.ClientSession()
async def teardown(self):
"""Cleanup test session."""
if self.session:
await self.session.close()
async def test_health_endpoint(self):
"""Test the /health endpoint."""
print("\n[TEST] Health Endpoint")
try:
async with self.session.get(f"{self.base_url}/health") as response:
status = response.status
data = await response.json()
print(f"Status: {status}")
print(f"Response: {json.dumps(data, indent=2)}")
self.results.append({
"test": "health_endpoint",
"status": "passed" if status == 200 else "failed",
"response_code": status,
"data": data
})
# Verify structure
assert "status" in data
assert "timestamp" in data
assert "components" in data
assert "system_metrics" in data
print("✅ Health endpoint test passed")
except Exception as e:
print(f"❌ Health endpoint test failed: {e}")
self.results.append({
"test": "health_endpoint",
"status": "failed",
"error": str(e)
})
async def test_ready_endpoint(self):
"""Test the /ready endpoint."""
print("\n[TEST] Readiness Endpoint")
try:
async with self.session.get(f"{self.base_url}/ready") as response:
status = response.status
data = await response.json()
print(f"Status: {status}")
print(f"Response: {json.dumps(data, indent=2)}")
self.results.append({
"test": "ready_endpoint",
"status": "passed" if status == 200 else "failed",
"response_code": status,
"data": data
})
# Verify structure
assert "ready" in data
assert "timestamp" in data
assert "checks" in data
assert "message" in data
print("✅ Readiness endpoint test passed")
except Exception as e:
print(f"❌ Readiness endpoint test failed: {e}")
self.results.append({
"test": "ready_endpoint",
"status": "failed",
"error": str(e)
})
async def test_liveness_endpoint(self):
"""Test the /live endpoint."""
print("\n[TEST] Liveness Endpoint")
try:
async with self.session.get(f"{self.base_url}/live") as response:
status = response.status
data = await response.json()
print(f"Status: {status}")
print(f"Response: {json.dumps(data, indent=2)}")
self.results.append({
"test": "liveness_endpoint",
"status": "passed" if status == 200 else "failed",
"response_code": status,
"data": data
})
# Verify structure
assert "status" in data
assert "timestamp" in data
print("✅ Liveness endpoint test passed")
except Exception as e:
print(f"❌ Liveness endpoint test failed: {e}")
self.results.append({
"test": "liveness_endpoint",
"status": "failed",
"error": str(e)
})
async def test_metrics_endpoint(self):
"""Test the /metrics endpoint."""
print("\n[TEST] Metrics Endpoint")
try:
async with self.session.get(f"{self.base_url}/metrics") as response:
status = response.status
data = await response.json()
print(f"Status: {status}")
print(f"Response: {json.dumps(data, indent=2)}")
self.results.append({
"test": "metrics_endpoint",
"status": "passed" if status == 200 else "failed",
"response_code": status,
"data": data
})
# Verify structure
assert "timestamp" in data
assert "metrics" in data
# Check for system metrics
metrics = data.get("metrics", {})
assert "cpu" in metrics
assert "memory" in metrics
assert "disk" in metrics
assert "network" in metrics
print("✅ Metrics endpoint test passed")
except Exception as e:
print(f"❌ Metrics endpoint test failed: {e}")
self.results.append({
"test": "metrics_endpoint",
"status": "failed",
"error": str(e)
})
async def test_version_endpoint(self):
"""Test the /version endpoint."""
print("\n[TEST] Version Endpoint")
try:
async with self.session.get(f"{self.base_url}/version") as response:
status = response.status
data = await response.json()
print(f"Status: {status}")
print(f"Response: {json.dumps(data, indent=2)}")
self.results.append({
"test": "version_endpoint",
"status": "passed" if status == 200 else "failed",
"response_code": status,
"data": data
})
# Verify structure
assert "name" in data
assert "version" in data
assert "environment" in data
assert "timestamp" in data
print("✅ Version endpoint test passed")
except Exception as e:
print(f"❌ Version endpoint test failed: {e}")
self.results.append({
"test": "version_endpoint",
"status": "failed",
"error": str(e)
})
async def test_metrics_collection(self):
"""Test metrics collection over time."""
print("\n[TEST] Metrics Collection Over Time")
try:
# Collect metrics 3 times with 2-second intervals
metrics_snapshots = []
for i in range(3):
async with self.session.get(f"{self.base_url}/metrics") as response:
data = await response.json()
metrics_snapshots.append({
"timestamp": time.time(),
"metrics": data.get("metrics", {})
})
if i < 2:
await asyncio.sleep(2)
# Verify metrics are changing
cpu_values = [
snapshot["metrics"].get("cpu", {}).get("percent", 0)
for snapshot in metrics_snapshots
]
print(f"CPU usage over time: {cpu_values}")
# Check if at least some metrics are non-zero
all_zeros = all(v == 0 for v in cpu_values)
assert not all_zeros, "All CPU metrics are zero"
self.results.append({
"test": "metrics_collection",
"status": "passed",
"snapshots": len(metrics_snapshots),
"cpu_values": cpu_values
})
print("✅ Metrics collection test passed")
except Exception as e:
print(f"❌ Metrics collection test failed: {e}")
self.results.append({
"test": "metrics_collection",
"status": "failed",
"error": str(e)
})
async def test_system_load(self):
"""Test system under load to verify monitoring."""
print("\n[TEST] System Load Monitoring")
try:
# Generate some load by making multiple concurrent requests
print("Generating load with 20 concurrent requests...")
tasks = []
for i in range(20):
tasks.append(self.session.get(f"{self.base_url}/health"))
start_time = time.time()
responses = await asyncio.gather(*tasks, return_exceptions=True)
duration = time.time() - start_time
success_count = sum(
1 for r in responses
if not isinstance(r, Exception) and r.status == 200
)
print(f"Completed {len(responses)} requests in {duration:.2f}s")
print(f"Success rate: {success_count}/{len(responses)}")
# Check metrics after load
async with self.session.get(f"{self.base_url}/metrics") as response:
data = await response.json()
metrics = data.get("metrics", {})
print(f"CPU after load: {metrics.get('cpu', {}).get('percent', 0)}%")
print(f"Memory usage: {metrics.get('memory', {}).get('percent', 0)}%")
self.results.append({
"test": "system_load",
"status": "passed",
"requests": len(responses),
"success_rate": f"{success_count}/{len(responses)}",
"duration": duration
})
print("✅ System load monitoring test passed")
except Exception as e:
print(f"❌ System load monitoring test failed: {e}")
self.results.append({
"test": "system_load",
"status": "failed",
"error": str(e)
})
async def run_all_tests(self):
"""Run all monitoring tests."""
print("=== WiFi-DensePose Monitoring Tests ===")
print(f"Base URL: {self.base_url}")
print(f"Started at: {datetime.now().isoformat()}")
await self.setup()
try:
# Run all tests
await self.test_health_endpoint()
await self.test_ready_endpoint()
await self.test_liveness_endpoint()
await self.test_metrics_endpoint()
await self.test_version_endpoint()
await self.test_metrics_collection()
await self.test_system_load()
finally:
await self.teardown()
# Print summary
print("\n=== Test Summary ===")
passed = sum(1 for r in self.results if r["status"] == "passed")
failed = sum(1 for r in self.results if r["status"] == "failed")
print(f"Total tests: {len(self.results)}")
print(f"Passed: {passed}")
print(f"Failed: {failed}")
if failed > 0:
print("\nFailed tests:")
for result in self.results:
if result["status"] == "failed":
print(f" - {result['test']}: {result.get('error', 'Unknown error')}")
# Save results
with open("monitoring_test_results.json", "w") as f:
json.dump({
"timestamp": datetime.now().isoformat(),
"base_url": self.base_url,
"summary": {
"total": len(self.results),
"passed": passed,
"failed": failed
},
"results": self.results
}, f, indent=2)
print("\nResults saved to monitoring_test_results.json")
return failed == 0
async def main():
"""Main entry point."""
base_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8000"
tester = MonitoringTester(base_url)
success = await tester.run_all_tests()
sys.exit(0 if success else 1)
if __name__ == "__main__":
asyncio.run(main())
archive/v1/scripts/test_websocket_streaming.py
Executable
+157
View File
@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""
WebSocket Streaming Test Script
Tests real-time pose data streaming via WebSocket
"""
import asyncio
import json
import websockets
from datetime import datetime
async def test_pose_streaming():
"""Test pose data streaming via WebSocket."""
uri = "ws://localhost:8000/api/v1/stream/pose?zone_ids=zone_1,zone_2&min_confidence=0.3&max_fps=10"
print(f"[{datetime.now()}] Connecting to WebSocket...")
try:
async with websockets.connect(uri) as websocket:
print(f"[{datetime.now()}] Connected successfully!")
# Wait for connection confirmation
response = await websocket.recv()
data = json.loads(response)
print(f"[{datetime.now()}] Connection confirmed:")
print(json.dumps(data, indent=2))
# Send a ping message
ping_msg = {"type": "ping"}
await websocket.send(json.dumps(ping_msg))
print(f"[{datetime.now()}] Sent ping message")
# Listen for messages for 10 seconds
print(f"[{datetime.now()}] Listening for pose updates...")
start_time = asyncio.get_event_loop().time()
message_count = 0
while asyncio.get_event_loop().time() - start_time < 10:
try:
# Wait for message with timeout
message = await asyncio.wait_for(websocket.recv(), timeout=1.0)
data = json.loads(message)
message_count += 1
msg_type = data.get("type", "unknown")
if msg_type == "pose_update":
print(f"[{datetime.now()}] Pose update received:")
print(f" - Frame ID: {data.get('frame_id')}")
print(f" - Persons detected: {len(data.get('persons', []))}")
print(f" - Zone summary: {data.get('zone_summary', {})}")
elif msg_type == "pong":
print(f"[{datetime.now()}] Pong received")
else:
print(f"[{datetime.now()}] Message type '{msg_type}' received")
except asyncio.TimeoutError:
# No message received in timeout period
continue
except Exception as e:
print(f"[{datetime.now()}] Error receiving message: {e}")
print(f"\n[{datetime.now()}] Test completed!")
print(f"Total messages received: {message_count}")
# Send disconnect message
disconnect_msg = {"type": "disconnect"}
await websocket.send(json.dumps(disconnect_msg))
except Exception as e:
print(f"[{datetime.now()}] WebSocket error: {e}")
async def test_event_streaming():
"""Test event streaming via WebSocket."""
uri = "ws://localhost:8000/api/v1/stream/events?event_types=motion,presence&zone_ids=zone_1"
print(f"\n[{datetime.now()}] Testing event streaming...")
print(f"[{datetime.now()}] Connecting to WebSocket...")
try:
async with websockets.connect(uri) as websocket:
print(f"[{datetime.now()}] Connected successfully!")
# Wait for connection confirmation
response = await websocket.recv()
data = json.loads(response)
print(f"[{datetime.now()}] Connection confirmed:")
print(json.dumps(data, indent=2))
# Get status
status_msg = {"type": "get_status"}
await websocket.send(json.dumps(status_msg))
print(f"[{datetime.now()}] Requested status")
# Listen for a few messages
for i in range(5):
try:
message = await asyncio.wait_for(websocket.recv(), timeout=2.0)
data = json.loads(message)
print(f"[{datetime.now()}] Event received: {data.get('type')}")
except asyncio.TimeoutError:
print(f"[{datetime.now()}] No event received (timeout)")
except Exception as e:
print(f"[{datetime.now()}] WebSocket error: {e}")
async def test_websocket_errors():
"""Test WebSocket error handling."""
print(f"\n[{datetime.now()}] Testing error handling...")
# Test invalid endpoint
try:
uri = "ws://localhost:8000/api/v1/stream/invalid"
async with websockets.connect(uri) as websocket:
print("Connected to invalid endpoint (unexpected)")
except Exception as e:
print(f"[{datetime.now()}] Expected error for invalid endpoint: {type(e).__name__}")
# Test sending invalid JSON
try:
uri = "ws://localhost:8000/api/v1/stream/pose"
async with websockets.connect(uri) as websocket:
await websocket.send("invalid json {")
response = await websocket.recv()
data = json.loads(response)
if data.get("type") == "error":
print(f"[{datetime.now()}] Received expected error for invalid JSON")
except Exception as e:
print(f"[{datetime.now()}] Error testing invalid JSON: {e}")
async def main():
"""Run all WebSocket tests."""
print("=" * 60)
print("WiFi-DensePose WebSocket Streaming Tests")
print("=" * 60)
# Test pose streaming
await test_pose_streaming()
# Test event streaming
await test_event_streaming()
# Test error handling
await test_websocket_errors()
print("\n" + "=" * 60)
print("All tests completed!")
print("=" * 60)
if __name__ == "__main__":
asyncio.run(main())
archive/v1/scripts/validate-deployment.sh
Executable
+398
View File
@@ -0,0 +1,398 @@
#!/bin/bash
# WiFi-DensePose Deployment Validation Script
# This script validates that all deployment components are functioning correctly
set -euo pipefail
# Configuration
NAMESPACE="wifi-densepose"
MONITORING_NAMESPACE="monitoring"
TIMEOUT=300
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Logging functions
log_info() {
echo -e "${BLUE}[INFO]${NC} $1"
}
log_success() {
echo -e "${GREEN}[SUCCESS]${NC} $1"
}
log_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Check if kubectl is available and configured
check_kubectl() {
log_info "Checking kubectl configuration..."
if ! command -v kubectl &> /dev/null; then
log_error "kubectl is not installed or not in PATH"
return 1
fi
if ! kubectl cluster-info &> /dev/null; then
log_error "kubectl is not configured or cluster is not accessible"
return 1
fi
log_success "kubectl is configured and cluster is accessible"
return 0
}
# Validate namespace exists
validate_namespace() {
local ns=$1
log_info "Validating namespace: $ns"
if kubectl get namespace "$ns" &> /dev/null; then
log_success "Namespace $ns exists"
return 0
else
log_error "Namespace $ns does not exist"
return 1
fi
}
# Validate deployments are ready
validate_deployments() {
log_info "Validating deployments in namespace: $NAMESPACE"
local deployments
deployments=$(kubectl get deployments -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')
if [ -z "$deployments" ]; then
log_warning "No deployments found in namespace $NAMESPACE"
return 1
fi
local failed=0
for deployment in $deployments; do
log_info "Checking deployment: $deployment"
if kubectl wait --for=condition=available --timeout="${TIMEOUT}s" "deployment/$deployment" -n "$NAMESPACE" &> /dev/null; then
local ready_replicas
ready_replicas=$(kubectl get deployment "$deployment" -n "$NAMESPACE" -o jsonpath='{.status.readyReplicas}')
local desired_replicas
desired_replicas=$(kubectl get deployment "$deployment" -n "$NAMESPACE" -o jsonpath='{.spec.replicas}')
if [ "$ready_replicas" = "$desired_replicas" ]; then
log_success "Deployment $deployment is ready ($ready_replicas/$desired_replicas replicas)"
else
log_warning "Deployment $deployment has $ready_replicas/$desired_replicas replicas ready"
failed=1
fi
else
log_error "Deployment $deployment is not ready within ${TIMEOUT}s"
failed=1
fi
done
return $failed
}
# Validate services are accessible
validate_services() {
log_info "Validating services in namespace: $NAMESPACE"
local services
services=$(kubectl get services -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')
if [ -z "$services" ]; then
log_warning "No services found in namespace $NAMESPACE"
return 1
fi
local failed=0
for service in $services; do
log_info "Checking service: $service"
local endpoints
endpoints=$(kubectl get endpoints "$service" -n "$NAMESPACE" -o jsonpath='{.subsets[*].addresses[*].ip}')
if [ -n "$endpoints" ]; then
log_success "Service $service has endpoints: $endpoints"
else
log_error "Service $service has no endpoints"
failed=1
fi
done
return $failed
}
# Validate ingress configuration
validate_ingress() {
log_info "Validating ingress configuration in namespace: $NAMESPACE"
local ingresses
ingresses=$(kubectl get ingress -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')
if [ -z "$ingresses" ]; then
log_warning "No ingress resources found in namespace $NAMESPACE"
return 0
fi
local failed=0
for ingress in $ingresses; do
log_info "Checking ingress: $ingress"
local hosts
hosts=$(kubectl get ingress "$ingress" -n "$NAMESPACE" -o jsonpath='{.spec.rules[*].host}')
if [ -n "$hosts" ]; then
log_success "Ingress $ingress configured for hosts: $hosts"
# Check if ingress has an IP/hostname assigned
local address
address=$(kubectl get ingress "$ingress" -n "$NAMESPACE" -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')
if [ -n "$address" ]; then
log_success "Ingress $ingress has address: $address"
else
log_warning "Ingress $ingress does not have an assigned address yet"
fi
else
log_error "Ingress $ingress has no configured hosts"
failed=1
fi
done
return $failed
}
# Validate ConfigMaps and Secrets
validate_config() {
log_info "Validating ConfigMaps and Secrets in namespace: $NAMESPACE"
# Check ConfigMaps
local configmaps
configmaps=$(kubectl get configmaps -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')
if [ -n "$configmaps" ]; then
log_success "ConfigMaps found: $configmaps"
else
log_warning "No ConfigMaps found in namespace $NAMESPACE"
fi
# Check Secrets
local secrets
secrets=$(kubectl get secrets -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep -v "default-token" | tr '\n' ' ')
if [ -n "$secrets" ]; then
log_success "Secrets found: $secrets"
else
log_warning "No custom secrets found in namespace $NAMESPACE"
fi
return 0
}
# Validate HPA configuration
validate_hpa() {
log_info "Validating Horizontal Pod Autoscaler in namespace: $NAMESPACE"
local hpas
hpas=$(kubectl get hpa -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')
if [ -z "$hpas" ]; then
log_warning "No HPA resources found in namespace $NAMESPACE"
return 0
fi
local failed=0
for hpa in $hpas; do
log_info "Checking HPA: $hpa"
local current_replicas
current_replicas=$(kubectl get hpa "$hpa" -n "$NAMESPACE" -o jsonpath='{.status.currentReplicas}')
local desired_replicas
desired_replicas=$(kubectl get hpa "$hpa" -n "$NAMESPACE" -o jsonpath='{.status.desiredReplicas}')
if [ -n "$current_replicas" ] && [ -n "$desired_replicas" ]; then
log_success "HPA $hpa: current=$current_replicas, desired=$desired_replicas"
else
log_warning "HPA $hpa metrics not available yet"
fi
done
return $failed
}
# Test application health endpoints
test_health_endpoints() {
log_info "Testing application health endpoints..."
# Get application pods
local pods
pods=$(kubectl get pods -n "$NAMESPACE" -l app=wifi-densepose -o jsonpath='{.items[*].metadata.name}')
if [ -z "$pods" ]; then
log_error "No application pods found"
return 1
fi
local failed=0
for pod in $pods; do
log_info "Testing health endpoint for pod: $pod"
# Port forward and test health endpoint
kubectl port-forward "pod/$pod" 8080:8080 -n "$NAMESPACE" &
local pf_pid=$!
sleep 2
if curl -f http://localhost:8080/health &> /dev/null; then
log_success "Health endpoint for pod $pod is responding"
else
log_error "Health endpoint for pod $pod is not responding"
failed=1
fi
kill $pf_pid 2>/dev/null || true
sleep 1
done
return $failed
}
# Validate monitoring stack
validate_monitoring() {
log_info "Validating monitoring stack in namespace: $MONITORING_NAMESPACE"
if ! validate_namespace "$MONITORING_NAMESPACE"; then
log_warning "Monitoring namespace not found, skipping monitoring validation"
return 0
fi
# Check Prometheus
if kubectl get deployment prometheus-server -n "$MONITORING_NAMESPACE" &> /dev/null; then
if kubectl wait --for=condition=available --timeout=60s deployment/prometheus-server -n "$MONITORING_NAMESPACE" &> /dev/null; then
log_success "Prometheus is running"
else
log_error "Prometheus is not ready"
fi
else
log_warning "Prometheus deployment not found"
fi
# Check Grafana
if kubectl get deployment grafana -n "$MONITORING_NAMESPACE" &> /dev/null; then
if kubectl wait --for=condition=available --timeout=60s deployment/grafana -n "$MONITORING_NAMESPACE" &> /dev/null; then
log_success "Grafana is running"
else
log_error "Grafana is not ready"
fi
else
log_warning "Grafana deployment not found"
fi
return 0
}
# Validate logging stack
validate_logging() {
log_info "Validating logging stack..."
# Check Fluentd DaemonSet
if kubectl get daemonset fluentd -n kube-system &> /dev/null; then
local desired
desired=$(kubectl get daemonset fluentd -n kube-system -o jsonpath='{.status.desiredNumberScheduled}')
local ready
ready=$(kubectl get daemonset fluentd -n kube-system -o jsonpath='{.status.numberReady}')
if [ "$desired" = "$ready" ]; then
log_success "Fluentd DaemonSet is ready ($ready/$desired nodes)"
else
log_warning "Fluentd DaemonSet has $ready/$desired pods ready"
fi
else
log_warning "Fluentd DaemonSet not found"
fi
return 0
}
# Check resource usage
check_resource_usage() {
log_info "Checking resource usage..."
# Check node resource usage
log_info "Node resource usage:"
kubectl top nodes 2>/dev/null || log_warning "Metrics server not available for node metrics"
# Check pod resource usage
log_info "Pod resource usage in namespace $NAMESPACE:"
kubectl top pods -n "$NAMESPACE" 2>/dev/null || log_warning "Metrics server not available for pod metrics"
return 0
}
# Generate validation report
generate_report() {
local total_checks=$1
local failed_checks=$2
local passed_checks=$((total_checks - failed_checks))
echo ""
log_info "=== Deployment Validation Report ==="
echo "Total checks: $total_checks"
echo "Passed: $passed_checks"
echo "Failed: $failed_checks"
if [ $failed_checks -eq 0 ]; then
log_success "All validation checks passed! 🎉"
return 0
else
log_error "Some validation checks failed. Please review the output above."
return 1
fi
}
# Main validation function
main() {
log_info "Starting WiFi-DensePose deployment validation..."
local total_checks=0
local failed_checks=0
# Run validation checks
checks=(
"check_kubectl"
"validate_namespace $NAMESPACE"
"validate_deployments"
"validate_services"
"validate_ingress"
"validate_config"
"validate_hpa"
"test_health_endpoints"
"validate_monitoring"
"validate_logging"
"check_resource_usage"
)
for check in "${checks[@]}"; do
total_checks=$((total_checks + 1))
echo ""
if ! eval "$check"; then
failed_checks=$((failed_checks + 1))
fi
done
# Generate final report
generate_report $total_checks $failed_checks
}
# Run main function
main "$@"
archive/v1/scripts/validate-integration.sh
Executable
+458
View File
@@ -0,0 +1,458 @@
#!/bin/bash
# WiFi-DensePose Integration Validation Script
# This script validates the complete system integration
set -e # Exit on any error
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
VENV_PATH="${PROJECT_ROOT}/.venv"
TEST_DB_PATH="${PROJECT_ROOT}/test_integration.db"
LOG_FILE="${PROJECT_ROOT}/integration_validation.log"
# Functions
log() {
echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
}
success() {
echo -e "${GREEN}✅ $1${NC}" | tee -a "$LOG_FILE"
}
warning() {
echo -e "${YELLOW}⚠️ $1${NC}" | tee -a "$LOG_FILE"
}
error() {
echo -e "${RED}❌ $1${NC}" | tee -a "$LOG_FILE"
}
cleanup() {
log "Cleaning up test resources..."
# Stop any running servers
pkill -f "wifi-densepose" || true
pkill -f "uvicorn.*src.app" || true
# Remove test database
[ -f "$TEST_DB_PATH" ] && rm -f "$TEST_DB_PATH"
# Remove test logs
find "$PROJECT_ROOT" -name "*.log" -path "*/test*" -delete 2>/dev/null || true
success "Cleanup completed"
}
check_prerequisites() {
log "Checking prerequisites..."
# Check Python version
if ! python3 --version | grep -E "Python 3\.(9|10|11|12)" > /dev/null; then
error "Python 3.9+ is required"
exit 1
fi
success "Python version check passed"
# Check if virtual environment exists
if [ ! -d "$VENV_PATH" ]; then
warning "Virtual environment not found, creating one..."
python3 -m venv "$VENV_PATH"
fi
success "Virtual environment check passed"
# Activate virtual environment
source "$VENV_PATH/bin/activate"
# Check if requirements are installed
if ! pip list | grep -q "fastapi"; then
warning "Dependencies not installed, installing..."
pip install -e ".[dev]"
fi
success "Dependencies check passed"
}
validate_package_structure() {
log "Validating package structure..."
# Check main application files
required_files=(
"src/__init__.py"
"src/main.py"
"src/app.py"
"src/config.py"
"src/logger.py"
"src/cli.py"
"pyproject.toml"
"setup.py"
"MANIFEST.in"
)
for file in "${required_files[@]}"; do
if [ ! -f "$PROJECT_ROOT/$file" ]; then
error "Required file missing: $file"
exit 1
fi
done
success "Package structure validation passed"
# Check directory structure
required_dirs=(
"src/config"
"src/core"
"src/api"
"src/services"
"src/middleware"
"src/database"
"src/tasks"
"src/commands"
"tests/unit"
"tests/integration"
)
for dir in "${required_dirs[@]}"; do
if [ ! -d "$PROJECT_ROOT/$dir" ]; then
error "Required directory missing: $dir"
exit 1
fi
done
success "Directory structure validation passed"
}
validate_imports() {
log "Validating Python imports..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Test main package import
if ! python -c "import src; print(f'Package version: {src.__version__}')"; then
error "Failed to import main package"
exit 1
fi
success "Main package import passed"
# Test core components
core_modules=(
"src.app"
"src.config.settings"
"src.logger"
"src.cli"
"src.core.csi_processor"
"src.core.phase_sanitizer"
"src.core.pose_estimator"
"src.core.router_interface"
"src.services.orchestrator"
"src.database.connection"
"src.database.models"
)
for module in "${core_modules[@]}"; do
if ! python -c "import $module" 2>/dev/null; then
error "Failed to import module: $module"
exit 1
fi
done
success "Core modules import passed"
}
validate_configuration() {
log "Validating configuration..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Test configuration loading
if ! python -c "
from src.config.settings import get_settings
settings = get_settings()
print(f'Environment: {settings.environment}')
print(f'Debug: {settings.debug}')
print(f'API Version: {settings.api_version}')
"; then
error "Configuration validation failed"
exit 1
fi
success "Configuration validation passed"
}
validate_database() {
log "Validating database integration..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Test database connection and models
if ! python -c "
import asyncio
from src.config.settings import get_settings
from src.database.connection import get_database_manager
async def test_db():
settings = get_settings()
settings.database_url = 'sqlite+aiosqlite:///test_integration.db'
db_manager = get_database_manager(settings)
await db_manager.initialize()
await db_manager.test_connection()
# Test connection stats
stats = await db_manager.get_connection_stats()
print(f'Database connected: {stats[\"database\"][\"connected\"]}')
await db_manager.close_all_connections()
print('Database validation passed')
asyncio.run(test_db())
"; then
error "Database validation failed"
exit 1
fi
success "Database validation passed"
}
validate_api_endpoints() {
log "Validating API endpoints..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Start server in background
export WIFI_DENSEPOSE_ENVIRONMENT=test
export WIFI_DENSEPOSE_DATABASE_URL="sqlite+aiosqlite:///test_integration.db"
python -m uvicorn src.app:app --host 127.0.0.1 --port 8888 --log-level error &
SERVER_PID=$!
# Wait for server to start
sleep 5
# Test endpoints
endpoints=(
"http://127.0.0.1:8888/health"
"http://127.0.0.1:8888/metrics"
"http://127.0.0.1:8888/api/v1/devices"
"http://127.0.0.1:8888/api/v1/sessions"
)
for endpoint in "${endpoints[@]}"; do
if ! curl -s -f "$endpoint" > /dev/null; then
error "API endpoint failed: $endpoint"
kill $SERVER_PID 2>/dev/null || true
exit 1
fi
done
# Stop server
kill $SERVER_PID 2>/dev/null || true
wait $SERVER_PID 2>/dev/null || true
success "API endpoints validation passed"
}
validate_cli() {
log "Validating CLI interface..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Test CLI commands
if ! python -m src.cli --help > /dev/null; then
error "CLI help command failed"
exit 1
fi
success "CLI help command passed"
# Test version command
if ! python -m src.cli version > /dev/null; then
error "CLI version command failed"
exit 1
fi
success "CLI version command passed"
# Test config validation
export WIFI_DENSEPOSE_ENVIRONMENT=test
export WIFI_DENSEPOSE_DATABASE_URL="sqlite+aiosqlite:///test_integration.db"
if ! python -m src.cli config validate > /dev/null; then
error "CLI config validation failed"
exit 1
fi
success "CLI config validation passed"
}
validate_background_tasks() {
log "Validating background tasks..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Test task managers
if ! python -c "
import asyncio
from src.config.settings import get_settings
from src.tasks.cleanup import get_cleanup_manager
from src.tasks.monitoring import get_monitoring_manager
from src.tasks.backup import get_backup_manager
async def test_tasks():
settings = get_settings()
settings.database_url = 'sqlite+aiosqlite:///test_integration.db'
# Test cleanup manager
cleanup_manager = get_cleanup_manager(settings)
cleanup_stats = cleanup_manager.get_stats()
print(f'Cleanup manager initialized: {\"manager\" in cleanup_stats}')
# Test monitoring manager
monitoring_manager = get_monitoring_manager(settings)
monitoring_stats = monitoring_manager.get_stats()
print(f'Monitoring manager initialized: {\"manager\" in monitoring_stats}')
# Test backup manager
backup_manager = get_backup_manager(settings)
backup_stats = backup_manager.get_stats()
print(f'Backup manager initialized: {\"manager\" in backup_stats}')
print('Background tasks validation passed')
asyncio.run(test_tasks())
"; then
error "Background tasks validation failed"
exit 1
fi
success "Background tasks validation passed"
}
run_integration_tests() {
log "Running integration tests..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Set test environment
export WIFI_DENSEPOSE_ENVIRONMENT=test
export WIFI_DENSEPOSE_DATABASE_URL="sqlite+aiosqlite:///test_integration.db"
# Run integration tests
if ! python -m pytest tests/integration/ -v --tb=short; then
error "Integration tests failed"
exit 1
fi
success "Integration tests passed"
}
validate_package_build() {
log "Validating package build..."
cd "$PROJECT_ROOT"
source "$VENV_PATH/bin/activate"
# Install build tools
pip install build twine
# Build package
if ! python -m build; then
error "Package build failed"
exit 1
fi
success "Package build passed"
# Check package
if ! python -m twine check dist/*; then
error "Package check failed"
exit 1
fi
success "Package check passed"
# Clean up build artifacts
rm -rf build/ dist/ *.egg-info/
}
generate_report() {
log "Generating integration report..."
cat > "$PROJECT_ROOT/integration_report.md" << EOF
# WiFi-DensePose Integration Validation Report
**Date:** $(date)
**Status:** ✅ PASSED
## Validation Results
### Prerequisites
- ✅ Python version check
- ✅ Virtual environment setup
- ✅ Dependencies installation
### Package Structure
- ✅ Required files present
- ✅ Directory structure valid
- ✅ Python imports working
### Core Components
- ✅ Configuration management
- ✅ Database integration
- ✅ API endpoints
- ✅ CLI interface
- ✅ Background tasks
### Testing
- ✅ Integration tests passed
- ✅ Package build successful
## System Information
**Python Version:** $(python --version)
**Package Version:** $(python -c "import src; print(src.__version__)")
**Environment:** $(python -c "from src.config.settings import get_settings; print(get_settings().environment)")
## Next Steps
The WiFi-DensePose system has been successfully integrated and validated.
You can now:
1. Start the server: \`wifi-densepose start\`
2. Check status: \`wifi-densepose status\`
3. View configuration: \`wifi-densepose config show\`
4. Run tests: \`pytest tests/\`
For more information, see the documentation in the \`docs/\` directory.
EOF
success "Integration report generated: integration_report.md"
}
main() {
log "Starting WiFi-DensePose integration validation..."
# Trap cleanup on exit
trap cleanup EXIT
# Run validation steps
check_prerequisites
validate_package_structure
validate_imports
validate_configuration
validate_database
validate_api_endpoints
validate_cli
validate_background_tasks
run_integration_tests
validate_package_build
generate_report
success "🎉 All integration validations passed!"
log "Integration validation completed successfully"
}
# Run main function
main "$@"
archive/v1/setup.py
+218
View File
@@ -0,0 +1,218 @@
"""
Setup script for WiFi-DensePose API
This file is maintained for backward compatibility.
The main configuration is in pyproject.toml.
"""
from setuptools import setup, find_packages
import os
import sys
from pathlib import Path
# Ensure we're in the right directory
if __name__ == "__main__":
here = Path(__file__).parent.absolute()
os.chdir(here)
# Read version from src/__init__.py
def get_version():
"""Get version from src/__init__.py"""
version_file = here / "src" / "__init__.py"
if version_file.exists():
with open(version_file, 'r') as f:
for line in f:
if line.startswith('__version__'):
return line.split('=')[1].strip().strip('"').strip("'")
return "1.0.0"
# Read long description from README
def get_long_description():
"""Get long description from README.md"""
readme_file = here / "README.md"
if readme_file.exists():
with open(readme_file, 'r', encoding='utf-8') as f:
return f.read()
return "WiFi-based human pose estimation using CSI data and DensePose neural networks"
# Read requirements from requirements.txt if it exists
def get_requirements():
"""Get requirements from requirements.txt or use defaults"""
requirements_file = here / "requirements.txt"
if requirements_file.exists():
with open(requirements_file, 'r') as f:
return [line.strip() for line in f if line.strip() and not line.startswith('#')]
# Default requirements (should match pyproject.toml)
return [
"fastapi>=0.104.0",
"uvicorn[standard]>=0.24.0",
"pydantic>=2.5.0",
"pydantic-settings>=2.1.0",
"sqlalchemy>=2.0.0",
"alembic>=1.13.0",
"asyncpg>=0.29.0",
"psycopg2-binary>=2.9.0",
"redis>=5.0.0",
"aioredis>=2.0.0",
"torch>=2.1.0",
"torchvision>=0.16.0",
"numpy>=1.24.0",
"opencv-python>=4.8.0",
"pillow>=10.0.0",
"scikit-learn>=1.3.0",
"scipy>=1.11.0",
"matplotlib>=3.7.0",
"pandas>=2.1.0",
"scapy>=2.5.0",
"pyserial>=3.5",
"paramiko>=3.3.0",
"click>=8.1.0",
"rich>=13.6.0",
"typer>=0.9.0",
"python-multipart>=0.0.6",
"python-jose[cryptography]>=3.3.0",
"passlib[bcrypt]>=1.7.4",
"python-dotenv>=1.0.0",
"pyyaml>=6.0",
"toml>=0.10.2",
"prometheus-client>=0.19.0",
"structlog>=23.2.0",
"psutil>=5.9.0",
"httpx>=0.25.0",
"aiofiles>=23.2.0",
"marshmallow>=3.20.0",
"jsonschema>=4.19.0",
"celery>=5.3.0",
"kombu>=5.3.0",
]
# Development requirements
def get_dev_requirements():
"""Get development requirements"""
return [
"pytest>=7.4.0",
"pytest-asyncio>=0.21.0",
"pytest-cov>=4.1.0",
"pytest-mock>=3.12.0",
"pytest-xdist>=3.3.0",
"black>=23.9.0",
"isort>=5.12.0",
"flake8>=6.1.0",
"mypy>=1.6.0",
"pre-commit>=3.5.0",
"bandit>=1.7.0",
"safety>=2.3.0",
]
# Check Python version
if sys.version_info < (3, 9):
sys.exit("Python 3.9 or higher is required")
# Setup configuration
setup(
name="wifi-densepose",
version=get_version(),
description="WiFi-based human pose estimation using CSI data and DensePose neural networks",
long_description=get_long_description(),
long_description_content_type="text/markdown",
# Author information
author="rUv",
author_email="ruv@ruv.net",
maintainer="rUv",
maintainer_email="ruv@ruv.net",
# URLs
url="https://github.com/ruvnet/wifi-densepose",
project_urls={
"Documentation": "https://github.com/ruvnet/wifi-densepose#readme",
"Source": "https://github.com/ruvnet/wifi-densepose",
"Tracker": "https://github.com/ruvnet/wifi-densepose/issues",
},
# Package configuration
packages=find_packages(include=["src", "src.*"]),
package_dir={"": "."},
# Include package data
package_data={
"src": [
"*.yaml", "*.yml", "*.json", "*.toml", "*.cfg", "*.ini"
],
"src.models": ["*.pth", "*.onnx", "*.pt"],
"src.config": ["*.yaml", "*.yml", "*.json"],
},
include_package_data=True,
# Requirements
python_requires=">=3.9",
install_requires=get_requirements(),
extras_require={
"dev": get_dev_requirements(),
"docs": [
"sphinx>=7.2.0",
"sphinx-rtd-theme>=1.3.0",
"sphinx-autodoc-typehints>=1.25.0",
"myst-parser>=2.0.0",
],
"gpu": [
"torch>=2.1.0",
"torchvision>=0.16.0",
"nvidia-ml-py>=12.535.0",
],
"monitoring": [
"grafana-api>=1.0.3",
"influxdb-client>=1.38.0",
"elasticsearch>=8.10.0",
],
"deployment": [
"gunicorn>=21.2.0",
"docker>=6.1.0",
"kubernetes>=28.1.0",
],
},
# Entry points
entry_points={
"console_scripts": [
"wifi-densepose=src.cli:cli",
"wdp=src.cli:cli",
],
"wifi_densepose.plugins": [
# Plugin entry points for extensibility
],
},
# Classification
classifiers=[
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Image Processing",
"Topic :: System :: Networking",
"Topic :: Software Development :: Libraries :: Python Modules",
],
# Keywords
keywords=[
"wifi", "csi", "pose-estimation", "densepose", "neural-networks",
"computer-vision", "machine-learning", "iot", "wireless-sensing"
],
# License
license="MIT",
# Zip safe
zip_safe=False,
# Platform
platforms=["any"],
)
archive/v1/src/__init__.py
+266
View File
@@ -0,0 +1,266 @@
"""
WiFi-DensePose API Package
==========================
A comprehensive system for WiFi-based human pose estimation using CSI data
and DensePose neural networks.
This package provides:
- Real-time CSI data collection from WiFi routers
- Advanced signal processing and phase sanitization
- DensePose neural network integration for pose estimation
- RESTful API for data access and control
- Background task management for data processing
- Comprehensive monitoring and logging
Example usage:
>>> from src.app import app
>>> from src.config.settings import get_settings
>>>
>>> settings = get_settings()
>>> # Run with: uvicorn src.app:app --host 0.0.0.0 --port 8000
For CLI usage:
$ wifi-densepose start --host 0.0.0.0 --port 8000
$ wifi-densepose status
$ wifi-densepose stop
Author: WiFi-DensePose Team
License: MIT
"""
__version__ = "1.1.0"
__author__ = "WiFi-DensePose Team"
__email__ = "team@wifi-densepose.com"
__license__ = "MIT"
__copyright__ = "Copyright 2024 WiFi-DensePose Team"
# Package metadata
__title__ = "wifi-densepose"
__description__ = "WiFi-based human pose estimation using CSI data and DensePose neural networks"
__url__ = "https://github.com/wifi-densepose/wifi-densepose"
__download_url__ = "https://github.com/wifi-densepose/wifi-densepose/archive/main.zip"
# Version info tuple
__version_info__ = tuple(int(x) for x in __version__.split('.'))
# Import key components for easy access
try:
from src.app import app
from src.config.settings import get_settings, Settings
from src.logger import setup_logging, get_logger
# Core components
from src.core.csi_processor import CSIProcessor
from src.core.phase_sanitizer import PhaseSanitizer
from src.core.pose_estimator import PoseEstimator
from src.core.router_interface import RouterInterface
# Services
from src.services.orchestrator import ServiceOrchestrator
from src.services.health_check import HealthCheckService
from src.services.metrics import MetricsService
# Database
from src.database.connection import get_database_manager
from src.database.models import (
Device, Session, CSIData, PoseDetection,
SystemMetric, AuditLog
)
__all__ = [
# Core app
'app',
'get_settings',
'Settings',
'setup_logging',
'get_logger',
# Core processing
'CSIProcessor',
'PhaseSanitizer',
'PoseEstimator',
'RouterInterface',
# Services
'ServiceOrchestrator',
'HealthCheckService',
'MetricsService',
# Database
'get_database_manager',
'Device',
'Session',
'CSIData',
'PoseDetection',
'SystemMetric',
'AuditLog',
# Metadata
'__version__',
'__version_info__',
'__author__',
'__email__',
'__license__',
'__copyright__',
]
except ImportError as e:
# Handle import errors gracefully during package installation
import warnings
warnings.warn(
f"Some components could not be imported: {e}. "
"This is normal during package installation.",
ImportWarning
)
__all__ = [
'__version__',
'__version_info__',
'__author__',
'__email__',
'__license__',
'__copyright__',
]
def get_version():
"""Get the package version."""
return __version__
def get_version_info():
"""Get the package version as a tuple."""
return __version_info__
def get_package_info():
"""Get comprehensive package information."""
return {
'name': __title__,
'version': __version__,
'version_info': __version_info__,
'description': __description__,
'author': __author__,
'author_email': __email__,
'license': __license__,
'copyright': __copyright__,
'url': __url__,
'download_url': __download_url__,
}
def check_dependencies():
"""Check if all required dependencies are available."""
missing_deps = []
optional_deps = []
# Core dependencies
required_modules = [
('fastapi', 'FastAPI'),
('uvicorn', 'Uvicorn'),
('pydantic', 'Pydantic'),
('sqlalchemy', 'SQLAlchemy'),
('numpy', 'NumPy'),
('torch', 'PyTorch'),
('cv2', 'OpenCV'),
('scipy', 'SciPy'),
('pandas', 'Pandas'),
('redis', 'Redis'),
('psutil', 'psutil'),
('click', 'Click'),
]
for module_name, display_name in required_modules:
try:
__import__(module_name)
except ImportError:
missing_deps.append(display_name)
# Optional dependencies
optional_modules = [
('scapy', 'Scapy (for network packet capture)'),
('paramiko', 'Paramiko (for SSH connections)'),
('serial', 'PySerial (for serial communication)'),
('matplotlib', 'Matplotlib (for plotting)'),
('prometheus_client', 'Prometheus Client (for metrics)'),
]
for module_name, display_name in optional_modules:
try:
__import__(module_name)
except ImportError:
optional_deps.append(display_name)
return {
'missing_required': missing_deps,
'missing_optional': optional_deps,
'all_required_available': len(missing_deps) == 0,
}
def print_system_info():
"""Print system and package information."""
import sys
import platform
info = get_package_info()
deps = check_dependencies()
print(f"WiFi-DensePose v{info['version']}")
print(f"Python {sys.version}")
print(f"Platform: {platform.platform()}")
print(f"Architecture: {platform.architecture()[0]}")
print()
if deps['all_required_available']:
print("✅ All required dependencies are available")
else:
print("❌ Missing required dependencies:")
for dep in deps['missing_required']:
print(f" - {dep}")
if deps['missing_optional']:
print("\n⚠️ Missing optional dependencies:")
for dep in deps['missing_optional']:
print(f" - {dep}")
print(f"\nFor more information, visit: {info['url']}")
# Package-level configuration
import logging
# Set up basic logging configuration
logging.getLogger(__name__).addHandler(logging.NullHandler())
# Suppress some noisy third-party loggers
logging.getLogger('urllib3').setLevel(logging.WARNING)
logging.getLogger('requests').setLevel(logging.WARNING)
logging.getLogger('asyncio').setLevel(logging.WARNING)
# Package initialization message
if __name__ != '__main__':
logger = logging.getLogger(__name__)
logger.debug(f"WiFi-DensePose package v{__version__} initialized")
# Compatibility aliases for backward compatibility
try:
WifiDensePose = app # Legacy alias
except NameError:
WifiDensePose = None # Will be None if app import failed
try:
get_config = get_settings # Legacy alias
except NameError:
get_config = None # Will be None if get_settings import failed
def main():
"""Main entry point for the package when run as a module."""
print_system_info()
if __name__ == '__main__':
main()
archive/v1/src/api/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""
WiFi-DensePose FastAPI application package
"""
# API package - routers and dependencies are imported by app.py
__all__ = []
archive/v1/src/api/dependencies.py
+467
View File
@@ -0,0 +1,467 @@
"""
Dependency injection for WiFi-DensePose API
"""
import logging
from typing import Optional, Dict, Any
from functools import lru_cache
from fastapi import Depends, HTTPException, status, Request
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from src.config.settings import get_settings
from src.config.domains import get_domain_config
from src.services.pose_service import PoseService
from src.services.stream_service import StreamService
from src.services.hardware_service import HardwareService
logger = logging.getLogger(__name__)
# Security scheme for JWT authentication
security = HTTPBearer(auto_error=False)
# Service dependencies
@lru_cache()
def get_pose_service() -> PoseService:
"""Get pose service instance."""
settings = get_settings()
domain_config = get_domain_config()
return PoseService(
settings=settings,
domain_config=domain_config
)
@lru_cache()
def get_stream_service() -> StreamService:
"""Get stream service instance."""
settings = get_settings()
domain_config = get_domain_config()
return StreamService(
settings=settings,
domain_config=domain_config
)
@lru_cache()
def get_hardware_service() -> HardwareService:
"""Get hardware service instance."""
settings = get_settings()
domain_config = get_domain_config()
return HardwareService(
settings=settings,
domain_config=domain_config
)
# Authentication dependencies
async def get_current_user(
request: Request,
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security)
) -> Optional[Dict[str, Any]]:
"""Get current authenticated user."""
settings = get_settings()
# Skip authentication if disabled
if not settings.enable_authentication:
return None
# Check if user is already set by middleware
if hasattr(request.state, 'user') and request.state.user:
return request.state.user
# No credentials provided
if not credentials:
return None
# Validate the JWT token
# JWT validation must be configured via settings (e.g. JWT_SECRET, JWT_ALGORITHM)
if settings.is_development:
logger.warning(
"Authentication credentials provided in development mode but JWT "
"validation is not configured. Set up JWT authentication via "
"environment variables (JWT_SECRET, JWT_ALGORITHM) or disable "
"authentication. Rejecting request."
)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=(
"JWT authentication is not configured. In development mode, either "
"disable authentication (enable_authentication=False) or configure "
"JWT validation. Returning mock users is not permitted in any environment."
),
headers={"WWW-Authenticate": "Bearer"},
)
# In production, implement proper JWT validation
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=(
"JWT authentication is not configured. Configure JWT_SECRET and "
"JWT_ALGORITHM environment variables, or integrate an external "
"identity provider. See docs/authentication.md for setup instructions."
),
headers={"WWW-Authenticate": "Bearer"},
)
async def get_current_active_user(
current_user: Optional[Dict[str, Any]] = Depends(get_current_user)
) -> Dict[str, Any]:
"""Get current active user (required authentication)."""
if not current_user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Authentication required",
headers={"WWW-Authenticate": "Bearer"},
)
# Check if user is active
if not current_user.get("is_active", True):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Inactive user"
)
return current_user
async def get_admin_user(
current_user: Dict[str, Any] = Depends(get_current_active_user)
) -> Dict[str, Any]:
"""Get current admin user (admin privileges required)."""
if not current_user.get("is_admin", False):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Admin privileges required"
)
return current_user
# Permission dependencies
def require_permission(permission: str):
"""Dependency factory for permission checking."""
async def check_permission(
current_user: Dict[str, Any] = Depends(get_current_active_user)
) -> Dict[str, Any]:
"""Check if user has required permission."""
user_permissions = current_user.get("permissions", [])
# Admin users have all permissions
if current_user.get("is_admin", False):
return current_user
# Check specific permission
if permission not in user_permissions:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Permission '{permission}' required"
)
return current_user
return check_permission
# Zone access dependencies
async def validate_zone_access(
zone_id: str,
current_user: Optional[Dict[str, Any]] = Depends(get_current_user)
) -> str:
"""Validate user access to a specific zone."""
domain_config = get_domain_config()
# Check if zone exists
zone = domain_config.get_zone(zone_id)
if not zone:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Zone '{zone_id}' not found"
)
# Check if zone is enabled
if not zone.enabled:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Zone '{zone_id}' is disabled"
)
# If authentication is enabled, check user access
if current_user:
# Admin users have access to all zones
if current_user.get("is_admin", False):
return zone_id
# Check user's zone permissions
user_zones = current_user.get("zones", [])
if user_zones and zone_id not in user_zones:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Access denied to zone '{zone_id}'"
)
return zone_id
# Router access dependencies
async def validate_router_access(
router_id: str,
current_user: Optional[Dict[str, Any]] = Depends(get_current_user)
) -> str:
"""Validate user access to a specific router."""
domain_config = get_domain_config()
# Check if router exists
router = domain_config.get_router(router_id)
if not router:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Router '{router_id}' not found"
)
# Check if router is enabled
if not router.enabled:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Router '{router_id}' is disabled"
)
# If authentication is enabled, check user access
if current_user:
# Admin users have access to all routers
if current_user.get("is_admin", False):
return router_id
# Check user's router permissions
user_routers = current_user.get("routers", [])
if user_routers and router_id not in user_routers:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Access denied to router '{router_id}'"
)
return router_id
# Service health dependencies
async def check_service_health(
request: Request,
service_name: str
) -> bool:
"""Check if a service is healthy."""
try:
if service_name == "pose":
service = getattr(request.app.state, 'pose_service', None)
elif service_name == "stream":
service = getattr(request.app.state, 'stream_service', None)
elif service_name == "hardware":
service = getattr(request.app.state, 'hardware_service', None)
else:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Unknown service: {service_name}"
)
if not service:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=f"Service '{service_name}' not available"
)
# Check service health
status_info = await service.get_status()
if status_info.get("status") != "healthy":
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=f"Service '{service_name}' is unhealthy: {status_info.get('error', 'Unknown error')}"
)
return True
except HTTPException:
raise
except Exception as e:
logger.error(f"Error checking service health for {service_name}: {e}")
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=f"Service '{service_name}' health check failed"
)
# Rate limiting dependencies
async def check_rate_limit(
request: Request,
current_user: Optional[Dict[str, Any]] = Depends(get_current_user)
) -> bool:
"""Check rate limiting status."""
settings = get_settings()
# Skip if rate limiting is disabled
if not settings.enable_rate_limiting:
return True
# Rate limiting is handled by middleware
# This dependency can be used for additional checks
return True
# Configuration dependencies
def get_zone_config(zone_id: str = Depends(validate_zone_access)):
"""Get zone configuration."""
domain_config = get_domain_config()
return domain_config.get_zone(zone_id)
def get_router_config(router_id: str = Depends(validate_router_access)):
"""Get router configuration."""
domain_config = get_domain_config()
return domain_config.get_router(router_id)
# Pagination dependencies
class PaginationParams:
"""Pagination parameters."""
def __init__(
self,
page: int = 1,
size: int = 20,
max_size: int = 100
):
if page < 1:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Page must be >= 1"
)
if size < 1:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Size must be >= 1"
)
if size > max_size:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Size must be <= {max_size}"
)
self.page = page
self.size = size
self.offset = (page - 1) * size
self.limit = size
def get_pagination_params(
page: int = 1,
size: int = 20
) -> PaginationParams:
"""Get pagination parameters."""
return PaginationParams(page=page, size=size)
# Query filter dependencies
class QueryFilters:
"""Common query filters."""
def __init__(
self,
start_time: Optional[str] = None,
end_time: Optional[str] = None,
min_confidence: Optional[float] = None,
activity: Optional[str] = None
):
self.start_time = start_time
self.end_time = end_time
self.min_confidence = min_confidence
self.activity = activity
# Validate confidence
if min_confidence is not None:
if not 0.0 <= min_confidence <= 1.0:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="min_confidence must be between 0.0 and 1.0"
)
def get_query_filters(
start_time: Optional[str] = None,
end_time: Optional[str] = None,
min_confidence: Optional[float] = None,
activity: Optional[str] = None
) -> QueryFilters:
"""Get query filters."""
return QueryFilters(
start_time=start_time,
end_time=end_time,
min_confidence=min_confidence,
activity=activity
)
# WebSocket dependencies
async def get_websocket_user(
websocket_token: Optional[str] = None
) -> Optional[Dict[str, Any]]:
"""Get user from WebSocket token."""
settings = get_settings()
# Skip authentication if disabled
if not settings.enable_authentication:
return None
# Validate the WebSocket token
if not websocket_token:
return None
if settings.is_development:
logger.warning(
"WebSocket token provided in development mode but token validation "
"is not configured. Rejecting. Disable authentication or configure "
"JWT validation to allow WebSocket connections."
)
return None
# WebSocket token validation requires a configured JWT secret and issuer.
# Until JWT settings are provided via environment variables
# (JWT_SECRET_KEY, JWT_ALGORITHM), tokens are rejected to prevent
# unauthorised access. Configure authentication settings and implement
# token verification here using the same logic as get_current_user().
logger.warning("WebSocket token validation requires JWT configuration. Rejecting token.")
return None
async def get_current_user_ws(
websocket_token: Optional[str] = None
) -> Optional[Dict[str, Any]]:
"""Get current user for WebSocket connections."""
return await get_websocket_user(websocket_token)
# Authentication requirement dependencies
async def require_auth(
current_user: Dict[str, Any] = Depends(get_current_active_user)
) -> Dict[str, Any]:
"""Require authentication for endpoint access."""
return current_user
# Development dependencies
async def development_only():
"""Dependency that only allows access in development."""
settings = get_settings()
if not settings.is_development:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Endpoint not available in production"
)
return True
archive/v1/src/api/main.py
+436
View File
@@ -0,0 +1,436 @@
"""
FastAPI application for WiFi-DensePose API
"""
import asyncio
import logging
import logging.config
from contextlib import asynccontextmanager
from typing import Dict, Any
from fastapi import FastAPI, Request, Response
from fastapi.middleware.cors import CORSMiddleware
from fastapi.middleware.trustedhost import TrustedHostMiddleware
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from src.config.settings import get_settings
from src.config.domains import get_domain_config
from src.api.routers import pose, stream, health, auth
from src.api.middleware.auth import AuthMiddleware
from src.api.middleware.rate_limit import RateLimitMiddleware
from src.api.dependencies import get_pose_service, get_stream_service, get_hardware_service
from src.api.websocket.connection_manager import connection_manager
from src.api.websocket.pose_stream import PoseStreamHandler
# Configure logging
settings = get_settings()
logging.config.dictConfig(settings.get_logging_config())
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan manager."""
logger.info("Starting WiFi-DensePose API...")
try:
# Initialize services
await initialize_services(app)
# Start background tasks
await start_background_tasks(app)
logger.info("WiFi-DensePose API started successfully")
yield
except Exception as e:
logger.error(f"Failed to start application: {e}")
raise
finally:
# Cleanup on shutdown
logger.info("Shutting down WiFi-DensePose API...")
await cleanup_services(app)
logger.info("WiFi-DensePose API shutdown complete")
async def initialize_services(app: FastAPI):
"""Initialize application services."""
try:
# Initialize hardware service
hardware_service = get_hardware_service()
await hardware_service.initialize()
# Initialize pose service
pose_service = get_pose_service()
await pose_service.initialize()
# Initialize stream service
stream_service = get_stream_service()
await stream_service.initialize()
# Initialize pose stream handler
pose_stream_handler = PoseStreamHandler(
connection_manager=connection_manager,
pose_service=pose_service,
stream_service=stream_service
)
# Store in app state for access in routes
app.state.hardware_service = hardware_service
app.state.pose_service = pose_service
app.state.stream_service = stream_service
app.state.pose_stream_handler = pose_stream_handler
logger.info("Services initialized successfully")
except Exception as e:
logger.error(f"Failed to initialize services: {e}")
raise
async def start_background_tasks(app: FastAPI):
"""Start background tasks."""
try:
# Start pose service
pose_service = app.state.pose_service
await pose_service.start()
logger.info("Pose service started")
# Start pose streaming if enabled
if settings.enable_real_time_processing:
pose_stream_handler = app.state.pose_stream_handler
await pose_stream_handler.start_streaming()
logger.info("Background tasks started")
except Exception as e:
logger.error(f"Failed to start background tasks: {e}")
raise
async def cleanup_services(app: FastAPI):
"""Cleanup services on shutdown."""
try:
# Stop pose streaming
if hasattr(app.state, 'pose_stream_handler'):
await app.state.pose_stream_handler.shutdown()
# Shutdown connection manager
await connection_manager.shutdown()
# Cleanup services
if hasattr(app.state, 'stream_service'):
await app.state.stream_service.shutdown()
if hasattr(app.state, 'pose_service'):
await app.state.pose_service.stop()
if hasattr(app.state, 'hardware_service'):
await app.state.hardware_service.shutdown()
logger.info("Services cleaned up successfully")
except Exception as e:
logger.error(f"Error during cleanup: {e}")
# Create FastAPI application
app = FastAPI(
title=settings.app_name,
version=settings.version,
description="WiFi-based human pose estimation and activity recognition API",
docs_url=settings.docs_url if not settings.is_production else None,
redoc_url=settings.redoc_url if not settings.is_production else None,
openapi_url=settings.openapi_url if not settings.is_production else None,
lifespan=lifespan
)
# Add middleware
if settings.enable_rate_limiting:
app.add_middleware(RateLimitMiddleware)
if settings.enable_authentication:
app.add_middleware(AuthMiddleware)
# Add CORS middleware
cors_config = settings.get_cors_config()
app.add_middleware(
CORSMiddleware,
**cors_config
)
# Add trusted host middleware for production
if settings.is_production:
app.add_middleware(
TrustedHostMiddleware,
allowed_hosts=settings.allowed_hosts
)
# Exception handlers
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
"""Handle HTTP exceptions."""
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": exc.status_code,
"message": exc.detail,
"type": "http_error"
}
}
)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
"""Handle request validation errors."""
return JSONResponse(
status_code=422,
content={
"error": {
"code": 422,
"message": "Validation error",
"type": "validation_error",
"details": exc.errors()
}
}
)
@app.exception_handler(Exception)
async def general_exception_handler(request: Request, exc: Exception):
"""Handle general exceptions."""
logger.error(f"Unhandled exception: {exc}", exc_info=True)
return JSONResponse(
status_code=500,
content={
"error": {
"code": 500,
"message": "Internal server error",
"type": "internal_error"
}
}
)
# Middleware for request logging
@app.middleware("http")
async def log_requests(request: Request, call_next):
"""Log all requests."""
start_time = asyncio.get_event_loop().time()
# Process request
response = await call_next(request)
# Calculate processing time
process_time = asyncio.get_event_loop().time() - start_time
# Log request
logger.info(
f"{request.method} {request.url.path} - "
f"Status: {response.status_code} - "
f"Time: {process_time:.3f}s"
)
# Add processing time header
response.headers["X-Process-Time"] = str(process_time)
return response
# Include routers
app.include_router(
health.router,
prefix="/health",
tags=["Health"]
)
app.include_router(
pose.router,
prefix=f"{settings.api_prefix}/pose",
tags=["Pose Estimation"]
)
app.include_router(
stream.router,
prefix=f"{settings.api_prefix}/stream",
tags=["Streaming"]
)
app.include_router(
auth.router,
prefix=f"{settings.api_prefix}",
tags=["Authentication"]
)
# Root endpoint
@app.get("/")
async def root():
"""Root endpoint with API information."""
return {
"name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"docs_url": settings.docs_url,
"api_prefix": settings.api_prefix,
"features": {
"authentication": settings.enable_authentication,
"rate_limiting": settings.enable_rate_limiting,
"websockets": settings.enable_websockets,
"real_time_processing": settings.enable_real_time_processing
}
}
# API information endpoint
@app.get(f"{settings.api_prefix}/info")
async def api_info():
"""Get detailed API information."""
domain_config = get_domain_config()
return {
"api": {
"name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"prefix": settings.api_prefix
},
"configuration": {
"zones": len(domain_config.zones),
"routers": len(domain_config.routers),
"pose_models": len(domain_config.pose_models)
},
"features": {
"authentication": settings.enable_authentication,
"rate_limiting": settings.enable_rate_limiting,
"websockets": settings.enable_websockets,
"real_time_processing": settings.enable_real_time_processing,
"historical_data": settings.enable_historical_data
},
"limits": {
"rate_limit_requests": settings.rate_limit_requests,
"rate_limit_window": settings.rate_limit_window,
"max_websocket_connections": domain_config.streaming.max_connections
}
}
# Status endpoint
@app.get(f"{settings.api_prefix}/status")
async def api_status(request: Request):
"""Get current API status."""
try:
# Get services from app state
hardware_service = getattr(request.app.state, 'hardware_service', None)
pose_service = getattr(request.app.state, 'pose_service', None)
stream_service = getattr(request.app.state, 'stream_service', None)
pose_stream_handler = getattr(request.app.state, 'pose_stream_handler', None)
# Get service statuses
status = {
"api": {
"status": "healthy",
"uptime": "unknown",
"version": settings.version
},
"services": {
"hardware": await hardware_service.get_status() if hardware_service else {"status": "unavailable"},
"pose": await pose_service.get_status() if pose_service else {"status": "unavailable"},
"stream": await stream_service.get_status() if stream_service else {"status": "unavailable"}
},
"streaming": pose_stream_handler.get_stream_status() if pose_stream_handler else {"is_streaming": False},
"connections": await connection_manager.get_connection_stats()
}
return status
except Exception as e:
logger.error(f"Error getting API status: {e}")
return {
"api": {
"status": "error",
"error": str(e)
}
}
# Metrics endpoint (if enabled)
if settings.metrics_enabled:
@app.get(f"{settings.api_prefix}/metrics")
async def api_metrics(request: Request):
"""Get API metrics."""
try:
# Get services from app state
pose_stream_handler = getattr(request.app.state, 'pose_stream_handler', None)
metrics = {
"connections": await connection_manager.get_metrics(),
"streaming": await pose_stream_handler.get_performance_metrics() if pose_stream_handler else {}
}
return metrics
except Exception as e:
logger.error(f"Error getting metrics: {e}")
return {"error": str(e)}
# Development endpoints (only in development)
if settings.is_development and settings.enable_test_endpoints:
@app.get(f"{settings.api_prefix}/dev/config")
async def dev_config():
"""Get current configuration (development only).
Returns a sanitized view -- secret keys and passwords are redacted.
"""
_sensitive = {"secret", "password", "token", "key", "credential", "auth"}
raw = settings.dict()
sanitized = {
k: "***REDACTED***" if any(s in k.lower() for s in _sensitive) else v
for k, v in raw.items()
}
domain_config = get_domain_config()
return {
"settings": sanitized,
"domain_config": domain_config.to_dict()
}
@app.post(f"{settings.api_prefix}/dev/reset")
async def dev_reset(request: Request):
"""Reset services (development only)."""
try:
# Reset services
hardware_service = getattr(request.app.state, 'hardware_service', None)
pose_service = getattr(request.app.state, 'pose_service', None)
if hardware_service:
await hardware_service.reset()
if pose_service:
await pose_service.reset()
return {"message": "Services reset successfully"}
except Exception as e:
logger.error(f"Error resetting services: {e}")
return {"error": str(e)}
if __name__ == "__main__":
import uvicorn
uvicorn.run(
"src.api.main:app",
host=settings.host,
port=settings.port,
reload=settings.reload,
workers=settings.workers if not settings.reload else 1,
log_level=settings.log_level.lower()
)
archive/v1/src/api/middleware/__init__.py
+8
View File
@@ -0,0 +1,8 @@
"""
FastAPI middleware package
"""
from .auth import AuthMiddleware
from .rate_limit import RateLimitMiddleware
__all__ = ["AuthMiddleware", "RateLimitMiddleware"]
archive/v1/src/api/middleware/auth.py
+307
View File
@@ -0,0 +1,307 @@
"""
JWT Authentication middleware for WiFi-DensePose API
"""
import logging
from typing import Optional, Dict, Any
from datetime import datetime
from fastapi import Request, Response
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware
from jose import JWTError, jwt
from src.config.settings import get_settings
logger = logging.getLogger(__name__)
class AuthMiddleware(BaseHTTPMiddleware):
"""JWT Authentication middleware."""
def __init__(self, app):
super().__init__(app)
self.settings = get_settings()
# Paths that don't require authentication
self.public_paths = {
"/",
"/docs",
"/redoc",
"/openapi.json",
"/health",
"/ready",
"/live",
"/version",
"/metrics"
}
# Paths that require authentication
self.protected_paths = {
"/api/v1/pose/analyze",
"/api/v1/pose/calibrate",
"/api/v1/pose/historical",
"/api/v1/stream/start",
"/api/v1/stream/stop",
"/api/v1/stream/clients",
"/api/v1/stream/broadcast"
}
async def dispatch(self, request: Request, call_next):
"""Process request through authentication middleware."""
# Skip authentication for public paths
if self._is_public_path(request.url.path):
return await call_next(request)
# Extract and validate token
token = self._extract_token(request)
if token:
try:
# Verify token and add user info to request state
user_data = await self._verify_token(token)
request.state.user = user_data
request.state.authenticated = True
logger.debug(f"Authenticated user: {user_data.get('id')}")
except Exception as e:
logger.warning(f"Token validation failed: {e}")
# For protected paths, return 401
if self._is_protected_path(request.url.path):
return JSONResponse(
status_code=401,
content={
"error": {
"code": 401,
"message": "Invalid or expired token",
"type": "authentication_error"
}
}
)
# For other paths, continue without authentication
request.state.user = None
request.state.authenticated = False
else:
# No token provided
if self._is_protected_path(request.url.path):
return JSONResponse(
status_code=401,
content={
"error": {
"code": 401,
"message": "Authentication required",
"type": "authentication_error"
}
},
headers={"WWW-Authenticate": "Bearer"}
)
request.state.user = None
request.state.authenticated = False
# Continue with request processing
response = await call_next(request)
# Add authentication headers to response
if hasattr(request.state, 'user') and request.state.user:
response.headers["X-User-ID"] = request.state.user.get("id", "")
response.headers["X-Authenticated"] = "true"
else:
response.headers["X-Authenticated"] = "false"
return response
def _is_public_path(self, path: str) -> bool:
"""Check if path is public (doesn't require authentication)."""
# Exact match
if path in self.public_paths:
return True
# Pattern matching for public paths
public_patterns = [
"/health",
"/metrics",
"/api/v1/pose/current", # Allow anonymous access to current pose data
"/api/v1/pose/zones/", # Allow anonymous access to zone data
"/api/v1/pose/activities", # Allow anonymous access to activities
"/api/v1/pose/stats", # Allow anonymous access to stats
"/api/v1/stream/status" # Allow anonymous access to stream status
]
for pattern in public_patterns:
if path.startswith(pattern):
return True
return False
def _is_protected_path(self, path: str) -> bool:
"""Check if path requires authentication."""
# Exact match
if path in self.protected_paths:
return True
# Pattern matching for protected paths
protected_patterns = [
"/api/v1/pose/analyze",
"/api/v1/pose/calibrate",
"/api/v1/pose/historical",
"/api/v1/stream/start",
"/api/v1/stream/stop",
"/api/v1/stream/clients",
"/api/v1/stream/broadcast"
]
for pattern in protected_patterns:
if path.startswith(pattern):
return True
return False
def _extract_token(self, request: Request) -> Optional[str]:
"""Extract JWT token from request."""
# Check Authorization header
auth_header = request.headers.get("authorization")
if auth_header and auth_header.startswith("Bearer "):
return auth_header.split(" ")[1]
# Check query parameter (for WebSocket connections)
token = request.query_params.get("token")
if token:
return token
# Check cookie
token = request.cookies.get("access_token")
if token:
return token
return None
async def _verify_token(self, token: str) -> Dict[str, Any]:
"""Verify JWT token and return user data."""
try:
# Decode JWT token
payload = jwt.decode(
token,
self.settings.secret_key,
algorithms=[self.settings.jwt_algorithm]
)
# Check token blacklist (logout invalidation)
if token_blacklist.is_blacklisted(token):
raise ValueError("Token has been revoked")
# Extract user information
user_id = payload.get("sub")
if not user_id:
raise ValueError("Token missing user ID")
# Check token expiration
exp = payload.get("exp")
if exp and datetime.utcnow() > datetime.fromtimestamp(exp):
raise ValueError("Token expired")
# Build user object
user_data = {
"id": user_id,
"username": payload.get("username"),
"email": payload.get("email"),
"is_admin": payload.get("is_admin", False),
"permissions": payload.get("permissions", []),
"accessible_zones": payload.get("accessible_zones", []),
"token_issued_at": payload.get("iat"),
"token_expires_at": payload.get("exp"),
"session_id": payload.get("session_id")
}
return user_data
except JWTError as e:
raise ValueError(f"JWT validation failed: {e}")
except Exception as e:
raise ValueError(f"Token verification error: {e}")
# TODO: Wire up authentication event logging in dispatch() for
# security monitoring (login failures, token expiry, etc.).
class TokenBlacklist:
"""Simple in-memory token blacklist for logout functionality."""
def __init__(self):
self._blacklisted_tokens = set()
self._cleanup_interval = 3600 # 1 hour
self._last_cleanup = datetime.utcnow()
def add_token(self, token: str):
"""Add token to blacklist."""
self._blacklisted_tokens.add(token)
self._cleanup_if_needed()
def is_blacklisted(self, token: str) -> bool:
"""Check if token is blacklisted."""
self._cleanup_if_needed()
return token in self._blacklisted_tokens
def _cleanup_if_needed(self):
"""Clean up expired tokens from blacklist."""
now = datetime.utcnow()
if (now - self._last_cleanup).total_seconds() > self._cleanup_interval:
# In a real implementation, you would check token expiration
# For now, we'll just clear old tokens periodically
self._blacklisted_tokens.clear()
self._last_cleanup = now
# Global token blacklist instance
token_blacklist = TokenBlacklist()
class SecurityHeaders:
"""Security headers for API responses."""
@staticmethod
def add_security_headers(response: Response) -> Response:
"""Add security headers to response."""
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["X-Frame-Options"] = "DENY"
response.headers["X-XSS-Protection"] = "1; mode=block"
response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin"
response.headers["Content-Security-Policy"] = (
"default-src 'self'; "
"script-src 'self' 'unsafe-inline'; "
"style-src 'self' 'unsafe-inline'; "
"img-src 'self' data:; "
"connect-src 'self' ws: wss:;"
)
return response
class APIKeyAuth:
"""Alternative API key authentication for service-to-service communication."""
def __init__(self, api_keys: Dict[str, Dict[str, Any]] = None):
self.api_keys = api_keys or {}
def verify_api_key(self, api_key: str) -> Optional[Dict[str, Any]]:
"""Verify API key and return associated service info."""
if api_key in self.api_keys:
return self.api_keys[api_key]
return None
def add_api_key(self, api_key: str, service_info: Dict[str, Any]):
"""Add new API key."""
self.api_keys[api_key] = service_info
def revoke_api_key(self, api_key: str):
"""Revoke API key."""
if api_key in self.api_keys:
del self.api_keys[api_key]
# Global API key auth instance
api_key_auth = APIKeyAuth()
archive/v1/src/api/middleware/rate_limit.py
+325
View File
@@ -0,0 +1,325 @@
"""
Rate limiting middleware for WiFi-DensePose API
"""
import logging
import time
from typing import Dict, Optional, Tuple
from datetime import datetime, timedelta
from collections import defaultdict, deque
from fastapi import Request, Response
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware
from src.config.settings import get_settings
logger = logging.getLogger(__name__)
class RateLimitMiddleware(BaseHTTPMiddleware):
"""Rate limiting middleware with sliding window algorithm."""
def __init__(self, app):
super().__init__(app)
self.settings = get_settings()
# Rate limit storage (in production, use Redis)
self.request_counts = defaultdict(lambda: deque())
self.blocked_clients = {}
# Rate limit configurations
self.rate_limits = {
"anonymous": {
"requests": self.settings.rate_limit_requests,
"window": self.settings.rate_limit_window,
"burst": 10 # Allow burst of 10 requests
},
"authenticated": {
"requests": self.settings.rate_limit_authenticated_requests,
"window": self.settings.rate_limit_window,
"burst": 50
},
"admin": {
"requests": 10000, # Very high limit for admins
"window": self.settings.rate_limit_window,
"burst": 100
}
}
# Path-specific rate limits
self.path_limits = {
"/api/v1/pose/current": {"requests": 60, "window": 60}, # 1 per second
"/api/v1/pose/analyze": {"requests": 10, "window": 60}, # 10 per minute
"/api/v1/pose/calibrate": {"requests": 1, "window": 300}, # 1 per 5 minutes
"/api/v1/stream/start": {"requests": 5, "window": 60}, # 5 per minute
"/api/v1/stream/stop": {"requests": 5, "window": 60}, # 5 per minute
}
# Exempt paths from rate limiting
self.exempt_paths = {
"/health",
"/ready",
"/live",
"/version",
"/metrics"
}
async def dispatch(self, request: Request, call_next):
"""Process request through rate limiting middleware."""
# Skip rate limiting for exempt paths
if self._is_exempt_path(request.url.path):
return await call_next(request)
# Get client identifier
client_id = self._get_client_id(request)
# Check if client is temporarily blocked
if self._is_client_blocked(client_id):
return self._create_rate_limit_response(
"Client temporarily blocked due to excessive requests"
)
# Get user type for rate limiting
user_type = self._get_user_type(request)
# Check rate limits
rate_limit_result = self._check_rate_limits(
client_id,
request.url.path,
user_type
)
if not rate_limit_result["allowed"]:
# Log rate limit violation
self._log_rate_limit_violation(request, client_id, rate_limit_result)
# Check if client should be temporarily blocked
if rate_limit_result.get("violations", 0) > 5:
self._block_client(client_id, duration=300) # 5 minutes
return self._create_rate_limit_response(
rate_limit_result["message"],
retry_after=rate_limit_result.get("retry_after", 60)
)
# Record the request
self._record_request(client_id, request.url.path)
# Process request
response = await call_next(request)
# Add rate limit headers
self._add_rate_limit_headers(response, client_id, user_type)
return response
def _is_exempt_path(self, path: str) -> bool:
"""Check if path is exempt from rate limiting."""
return path in self.exempt_paths
def _get_client_id(self, request: Request) -> str:
"""Get unique client identifier for rate limiting."""
# Try to get user ID from request state (set by auth middleware)
if hasattr(request.state, 'user') and request.state.user:
return f"user:{request.state.user['id']}"
# Fall back to IP address
client_ip = request.client.host if request.client else "unknown"
# Include user agent for better identification
user_agent = request.headers.get("user-agent", "")
user_agent_hash = str(hash(user_agent))[:8]
return f"ip:{client_ip}:{user_agent_hash}"
def _get_user_type(self, request: Request) -> str:
"""Determine user type for rate limiting."""
if hasattr(request.state, 'user') and request.state.user:
if request.state.user.get("is_admin", False):
return "admin"
return "authenticated"
return "anonymous"
def _check_rate_limits(self, client_id: str, path: str, user_type: str) -> Dict:
"""Check if request is within rate limits."""
now = time.time()
# Get applicable rate limits
general_limit = self.rate_limits[user_type]
path_limit = self.path_limits.get(path)
# Check general rate limit
general_result = self._check_limit(
client_id,
"general",
general_limit["requests"],
general_limit["window"],
now
)
if not general_result["allowed"]:
return general_result
# Check path-specific rate limit if exists
if path_limit:
path_result = self._check_limit(
client_id,
f"path:{path}",
path_limit["requests"],
path_limit["window"],
now
)
if not path_result["allowed"]:
return path_result
return {"allowed": True}
def _check_limit(self, client_id: str, limit_type: str, max_requests: int, window: int, now: float) -> Dict:
"""Check specific rate limit using sliding window."""
key = f"{client_id}:{limit_type}"
requests = self.request_counts[key]
# Remove old requests outside the window
cutoff = now - window
while requests and requests[0] <= cutoff:
requests.popleft()
# Check if limit exceeded
if len(requests) >= max_requests:
# Calculate retry after time
oldest_request = requests[0] if requests else now
retry_after = int(oldest_request + window - now) + 1
return {
"allowed": False,
"message": f"Rate limit exceeded: {max_requests} requests per {window} seconds",
"retry_after": retry_after,
"current_count": len(requests),
"limit": max_requests,
"window": window
}
return {
"allowed": True,
"current_count": len(requests),
"limit": max_requests,
"window": window
}
def _record_request(self, client_id: str, path: str):
"""Record a request for rate limiting."""
now = time.time()
# Record general request
general_key = f"{client_id}:general"
self.request_counts[general_key].append(now)
# Record path-specific request if path has specific limits
if path in self.path_limits:
path_key = f"{client_id}:path:{path}"
self.request_counts[path_key].append(now)
def _is_client_blocked(self, client_id: str) -> bool:
"""Check if client is temporarily blocked."""
if client_id in self.blocked_clients:
block_until = self.blocked_clients[client_id]
if time.time() < block_until:
return True
else:
# Block expired, remove it
del self.blocked_clients[client_id]
return False
def _block_client(self, client_id: str, duration: int):
"""Temporarily block a client."""
self.blocked_clients[client_id] = time.time() + duration
logger.warning(f"Client {client_id} blocked for {duration} seconds due to rate limit violations")
def _create_rate_limit_response(self, message: str, retry_after: int = 60) -> JSONResponse:
"""Create rate limit exceeded response."""
return JSONResponse(
status_code=429,
content={
"error": {
"code": 429,
"message": message,
"type": "rate_limit_exceeded"
}
},
headers={
"Retry-After": str(retry_after),
"X-RateLimit-Limit": "Exceeded",
"X-RateLimit-Remaining": "0"
}
)
def _add_rate_limit_headers(self, response: Response, client_id: str, user_type: str):
"""Add rate limit headers to response."""
try:
general_limit = self.rate_limits[user_type]
general_key = f"{client_id}:general"
current_requests = len(self.request_counts[general_key])
remaining = max(0, general_limit["requests"] - current_requests)
response.headers["X-RateLimit-Limit"] = str(general_limit["requests"])
response.headers["X-RateLimit-Remaining"] = str(remaining)
response.headers["X-RateLimit-Window"] = str(general_limit["window"])
# Add reset time
if self.request_counts[general_key]:
oldest_request = self.request_counts[general_key][0]
reset_time = int(oldest_request + general_limit["window"])
response.headers["X-RateLimit-Reset"] = str(reset_time)
except Exception as e:
logger.error(f"Error adding rate limit headers: {e}")
def _log_rate_limit_violation(self, request: Request, client_id: str, result: Dict):
"""Log rate limit violations for monitoring."""
client_ip = request.client.host if request.client else "unknown"
user_agent = request.headers.get("user-agent", "unknown")
log_data = {
"event_type": "rate_limit_violation",
"timestamp": datetime.utcnow().isoformat(),
"client_id": client_id,
"client_ip": client_ip,
"user_agent": user_agent,
"path": request.url.path,
"method": request.method,
"current_count": result.get("current_count"),
"limit": result.get("limit"),
"window": result.get("window")
}
logger.warning(f"Rate limit violation: {log_data}")
def cleanup_old_data(self):
"""Clean up old rate limiting data (call periodically)."""
now = time.time()
cutoff = now - 3600 # Keep data for 1 hour
# Clean up request counts
for key in list(self.request_counts.keys()):
requests = self.request_counts[key]
while requests and requests[0] <= cutoff:
requests.popleft()
# Remove empty deques
if not requests:
del self.request_counts[key]
# Clean up expired blocks
expired_blocks = [
client_id for client_id, block_until in self.blocked_clients.items()
if now >= block_until
]
for client_id in expired_blocks:
del self.blocked_clients[client_id]
archive/v1/src/api/routers/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""
API routers package
"""
from . import pose, stream, health, auth
__all__ = ["pose", "stream", "health", "auth"]
archive/v1/src/api/routers/auth.py
+32
View File
@@ -0,0 +1,32 @@
"""
Authentication router for WiFi-DensePose API.
Provides logout (token blacklisting) endpoint.
"""
import logging
from typing import Optional
from fastapi import APIRouter, Request, HTTPException, status
from src.api.middleware.auth import token_blacklist
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@router.post("/logout")
async def logout(request: Request):
"""Logout by blacklisting the current Bearer token."""
auth_header = request.headers.get("authorization")
if not auth_header or not auth_header.startswith("Bearer "):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing or invalid Authorization header",
)
token = auth_header.split(" ", 1)[1]
token_blacklist.add_token(token)
logger.info("Token blacklisted via /auth/logout")
return {"success": True, "message": "Token revoked"}
archive/v1/src/api/routers/health.py
+421
View File
@@ -0,0 +1,421 @@
"""
Health check API endpoints
"""
import logging
import psutil
from typing import Dict, Any, Optional
from datetime import datetime, timedelta
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, Field
from src.api.dependencies import get_current_user
from src.config.settings import get_settings
logger = logging.getLogger(__name__)
router = APIRouter()
# Recorded at module import time — proxy for application startup time
_APP_START_TIME = datetime.now()
# Response models
class ComponentHealth(BaseModel):
"""Health status for a system component."""
name: str = Field(..., description="Component name")
status: str = Field(..., description="Health status (healthy, degraded, unhealthy)")
message: Optional[str] = Field(default=None, description="Status message")
last_check: datetime = Field(..., description="Last health check timestamp")
uptime_seconds: Optional[float] = Field(default=None, description="Component uptime")
metrics: Optional[Dict[str, Any]] = Field(default=None, description="Component metrics")
class SystemHealth(BaseModel):
"""Overall system health status."""
status: str = Field(..., description="Overall system status")
timestamp: datetime = Field(..., description="Health check timestamp")
uptime_seconds: float = Field(..., description="System uptime")
components: Dict[str, ComponentHealth] = Field(..., description="Component health status")
system_metrics: Dict[str, Any] = Field(..., description="System-level metrics")
class ReadinessCheck(BaseModel):
"""System readiness check result."""
ready: bool = Field(..., description="Whether system is ready to serve requests")
timestamp: datetime = Field(..., description="Readiness check timestamp")
checks: Dict[str, bool] = Field(..., description="Individual readiness checks")
message: str = Field(..., description="Readiness status message")
# Health check endpoints
@router.get("/health", response_model=SystemHealth)
async def health_check(request: Request):
"""Comprehensive system health check."""
try:
# Get services from app state
hardware_service = getattr(request.app.state, 'hardware_service', None)
pose_service = getattr(request.app.state, 'pose_service', None)
stream_service = getattr(request.app.state, 'stream_service', None)
timestamp = datetime.utcnow()
components = {}
overall_status = "healthy"
# Check hardware service
if hardware_service:
try:
hw_health = await hardware_service.health_check()
components["hardware"] = ComponentHealth(
name="Hardware Service",
status=hw_health["status"],
message=hw_health.get("message"),
last_check=timestamp,
uptime_seconds=hw_health.get("uptime_seconds"),
metrics=hw_health.get("metrics")
)
if hw_health["status"] != "healthy":
overall_status = "degraded" if overall_status == "healthy" else "unhealthy"
except Exception as e:
logger.error(f"Hardware service health check failed: {e}")
components["hardware"] = ComponentHealth(
name="Hardware Service",
status="unhealthy",
message=f"Health check failed: {str(e)}",
last_check=timestamp
)
overall_status = "unhealthy"
else:
components["hardware"] = ComponentHealth(
name="Hardware Service",
status="unavailable",
message="Service not initialized",
last_check=timestamp
)
overall_status = "degraded"
# Check pose service
if pose_service:
try:
pose_health = await pose_service.health_check()
components["pose"] = ComponentHealth(
name="Pose Service",
status=pose_health["status"],
message=pose_health.get("message"),
last_check=timestamp,
uptime_seconds=pose_health.get("uptime_seconds"),
metrics=pose_health.get("metrics")
)
if pose_health["status"] != "healthy":
overall_status = "degraded" if overall_status == "healthy" else "unhealthy"
except Exception as e:
logger.error(f"Pose service health check failed: {e}")
components["pose"] = ComponentHealth(
name="Pose Service",
status="unhealthy",
message=f"Health check failed: {str(e)}",
last_check=timestamp
)
overall_status = "unhealthy"
else:
components["pose"] = ComponentHealth(
name="Pose Service",
status="unavailable",
message="Service not initialized",
last_check=timestamp
)
overall_status = "degraded"
# Check stream service
if stream_service:
try:
stream_health = await stream_service.health_check()
components["stream"] = ComponentHealth(
name="Stream Service",
status=stream_health["status"],
message=stream_health.get("message"),
last_check=timestamp,
uptime_seconds=stream_health.get("uptime_seconds"),
metrics=stream_health.get("metrics")
)
if stream_health["status"] != "healthy":
overall_status = "degraded" if overall_status == "healthy" else "unhealthy"
except Exception as e:
logger.error(f"Stream service health check failed: {e}")
components["stream"] = ComponentHealth(
name="Stream Service",
status="unhealthy",
message=f"Health check failed: {str(e)}",
last_check=timestamp
)
overall_status = "unhealthy"
else:
components["stream"] = ComponentHealth(
name="Stream Service",
status="unavailable",
message="Service not initialized",
last_check=timestamp
)
overall_status = "degraded"
# Get system metrics
system_metrics = get_system_metrics()
uptime_seconds = (datetime.now() - _APP_START_TIME).total_seconds()
return SystemHealth(
status=overall_status,
timestamp=timestamp,
uptime_seconds=uptime_seconds,
components=components,
system_metrics=system_metrics
)
except Exception as e:
logger.error(f"Health check failed: {e}")
raise HTTPException(
status_code=500,
detail=f"Health check failed: {str(e)}"
)
@router.get("/ready", response_model=ReadinessCheck)
async def readiness_check(request: Request):
"""Check if system is ready to serve requests."""
try:
timestamp = datetime.utcnow()
checks = {}
# Check if services are available in app state
if hasattr(request.app.state, 'pose_service') and request.app.state.pose_service:
try:
checks["pose_ready"] = await request.app.state.pose_service.is_ready()
except Exception as e:
logger.warning(f"Pose service readiness check failed: {e}")
checks["pose_ready"] = False
else:
checks["pose_ready"] = False
if hasattr(request.app.state, 'stream_service') and request.app.state.stream_service:
try:
checks["stream_ready"] = await request.app.state.stream_service.is_ready()
except Exception as e:
logger.warning(f"Stream service readiness check failed: {e}")
checks["stream_ready"] = False
else:
checks["stream_ready"] = False
# Hardware service check (basic availability)
checks["hardware_ready"] = True # Basic readiness - API is responding
# Check system resources
checks["memory_available"] = check_memory_availability()
checks["disk_space_available"] = check_disk_space()
# Application is ready if at least the basic services are available
# For now, we'll consider it ready if the API is responding
ready = True # Basic readiness
message = "System is ready" if ready else "System is not ready"
if not ready:
failed_checks = [name for name, status in checks.items() if not status]
message += f". Failed checks: {', '.join(failed_checks)}"
return ReadinessCheck(
ready=ready,
timestamp=timestamp,
checks=checks,
message=message
)
except Exception as e:
logger.error(f"Readiness check failed: {e}")
return ReadinessCheck(
ready=False,
timestamp=datetime.utcnow(),
checks={},
message=f"Readiness check failed: {str(e)}"
)
@router.get("/live")
async def liveness_check():
"""Simple liveness check for load balancers."""
return {
"status": "alive",
"timestamp": datetime.utcnow().isoformat()
}
@router.get("/metrics")
async def get_health_metrics(
request: Request,
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get detailed system metrics."""
try:
metrics = get_system_metrics()
# Add additional metrics if authenticated
if current_user:
metrics.update(get_detailed_metrics())
return {
"timestamp": datetime.utcnow().isoformat(),
"metrics": metrics
}
except Exception as e:
logger.error(f"Error getting system metrics: {e}")
raise HTTPException(
status_code=500,
detail=f"Failed to get system metrics: {str(e)}"
)
@router.get("/version")
async def get_version_info():
"""Get application version information."""
settings = get_settings()
return {
"name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"debug": settings.debug,
"timestamp": datetime.utcnow().isoformat()
}
def get_system_metrics() -> Dict[str, Any]:
"""Get basic system metrics."""
try:
# CPU metrics
cpu_percent = psutil.cpu_percent(interval=1)
cpu_count = psutil.cpu_count()
# Memory metrics
memory = psutil.virtual_memory()
memory_metrics = {
"total_gb": round(memory.total / (1024**3), 2),
"available_gb": round(memory.available / (1024**3), 2),
"used_gb": round(memory.used / (1024**3), 2),
"percent": memory.percent
}
# Disk metrics
disk = psutil.disk_usage('/')
disk_metrics = {
"total_gb": round(disk.total / (1024**3), 2),
"free_gb": round(disk.free / (1024**3), 2),
"used_gb": round(disk.used / (1024**3), 2),
"percent": round((disk.used / disk.total) * 100, 2)
}
# Network metrics (basic)
network = psutil.net_io_counters()
network_metrics = {
"bytes_sent": network.bytes_sent,
"bytes_recv": network.bytes_recv,
"packets_sent": network.packets_sent,
"packets_recv": network.packets_recv
}
return {
"cpu": {
"percent": cpu_percent,
"count": cpu_count
},
"memory": memory_metrics,
"disk": disk_metrics,
"network": network_metrics
}
except Exception as e:
logger.error(f"Error getting system metrics: {e}")
return {}
def get_detailed_metrics() -> Dict[str, Any]:
"""Get detailed system metrics (requires authentication)."""
try:
# Process metrics
process = psutil.Process()
process_metrics = {
"pid": process.pid,
"cpu_percent": process.cpu_percent(),
"memory_mb": round(process.memory_info().rss / (1024**2), 2),
"num_threads": process.num_threads(),
"create_time": datetime.fromtimestamp(process.create_time()).isoformat()
}
# Load average (Unix-like systems)
load_avg = None
try:
load_avg = psutil.getloadavg()
except AttributeError:
# Windows doesn't have load average
pass
# Temperature sensors (if available)
temperatures = {}
try:
temps = psutil.sensors_temperatures()
for name, entries in temps.items():
temperatures[name] = [
{"label": entry.label, "current": entry.current}
for entry in entries
]
except AttributeError:
# Not available on all systems
pass
detailed = {
"process": process_metrics
}
if load_avg:
detailed["load_average"] = {
"1min": load_avg[0],
"5min": load_avg[1],
"15min": load_avg[2]
}
if temperatures:
detailed["temperatures"] = temperatures
return detailed
except Exception as e:
logger.error(f"Error getting detailed metrics: {e}")
return {}
def check_memory_availability() -> bool:
"""Check if sufficient memory is available."""
try:
memory = psutil.virtual_memory()
# Consider system ready if less than 90% memory is used
return memory.percent < 90.0
except Exception:
return False
def check_disk_space() -> bool:
"""Check if sufficient disk space is available."""
try:
disk = psutil.disk_usage('/')
# Consider system ready if more than 1GB free space
free_gb = disk.free / (1024**3)
return free_gb > 1.0
except Exception:
return False
archive/v1/src/api/routers/pose.py
+420
View File
@@ -0,0 +1,420 @@
"""
Pose estimation API endpoints
"""
import logging
from typing import List, Optional, Dict, Any
from datetime import datetime, timedelta
from fastapi import APIRouter, Depends, HTTPException, Query, BackgroundTasks
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from src.api.dependencies import (
get_pose_service,
get_hardware_service,
get_current_user,
require_auth
)
from src.services.pose_service import PoseService
from src.services.hardware_service import HardwareService
from src.config.settings import get_settings
logger = logging.getLogger(__name__)
router = APIRouter()
# Request/Response models
class PoseEstimationRequest(BaseModel):
"""Request model for pose estimation."""
zone_ids: Optional[List[str]] = Field(
default=None,
description="Specific zones to analyze (all zones if not specified)"
)
confidence_threshold: Optional[float] = Field(
default=None,
ge=0.0,
le=1.0,
description="Minimum confidence threshold for detections"
)
max_persons: Optional[int] = Field(
default=None,
ge=1,
le=50,
description="Maximum number of persons to detect"
)
include_keypoints: bool = Field(
default=True,
description="Include detailed keypoint data"
)
include_segmentation: bool = Field(
default=False,
description="Include DensePose segmentation masks"
)
class PersonPose(BaseModel):
"""Person pose data model."""
person_id: str = Field(..., description="Unique person identifier")
confidence: float = Field(..., description="Detection confidence score")
bounding_box: Dict[str, float] = Field(..., description="Person bounding box")
keypoints: Optional[List[Dict[str, Any]]] = Field(
default=None,
description="Body keypoints with coordinates and confidence"
)
segmentation: Optional[Dict[str, Any]] = Field(
default=None,
description="DensePose segmentation data"
)
zone_id: Optional[str] = Field(
default=None,
description="Zone where person is detected"
)
activity: Optional[str] = Field(
default=None,
description="Detected activity"
)
timestamp: datetime = Field(..., description="Detection timestamp")
class PoseEstimationResponse(BaseModel):
"""Response model for pose estimation."""
timestamp: datetime = Field(..., description="Analysis timestamp")
frame_id: str = Field(..., description="Unique frame identifier")
persons: List[PersonPose] = Field(..., description="Detected persons")
zone_summary: Dict[str, int] = Field(..., description="Person count per zone")
processing_time_ms: float = Field(..., description="Processing time in milliseconds")
metadata: Dict[str, Any] = Field(default_factory=dict, description="Additional metadata")
class HistoricalDataRequest(BaseModel):
"""Request model for historical pose data."""
start_time: datetime = Field(..., description="Start time for data query")
end_time: datetime = Field(..., description="End time for data query")
zone_ids: Optional[List[str]] = Field(
default=None,
description="Filter by specific zones"
)
aggregation_interval: Optional[int] = Field(
default=300,
ge=60,
le=3600,
description="Aggregation interval in seconds"
)
include_raw_data: bool = Field(
default=False,
description="Include raw detection data"
)
# Endpoints
@router.get("/current", response_model=PoseEstimationResponse)
async def get_current_pose_estimation(
request: PoseEstimationRequest = Depends(),
pose_service: PoseService = Depends(get_pose_service),
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get current pose estimation from WiFi signals."""
try:
logger.info(f"Processing pose estimation request from user: {current_user.get('id') if current_user else 'anonymous'}")
# Get current pose estimation
result = await pose_service.estimate_poses(
zone_ids=request.zone_ids,
confidence_threshold=request.confidence_threshold,
max_persons=request.max_persons,
include_keypoints=request.include_keypoints,
include_segmentation=request.include_segmentation
)
return PoseEstimationResponse(**result)
except Exception as e:
logger.error(f"Error in pose estimation: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/analyze", response_model=PoseEstimationResponse)
async def analyze_pose_data(
request: PoseEstimationRequest,
background_tasks: BackgroundTasks,
pose_service: PoseService = Depends(get_pose_service),
current_user: Dict = Depends(require_auth)
):
"""Trigger pose analysis with custom parameters."""
try:
logger.info(f"Custom pose analysis requested by user: {current_user['id']}")
# Trigger analysis
result = await pose_service.analyze_with_params(
zone_ids=request.zone_ids,
confidence_threshold=request.confidence_threshold,
max_persons=request.max_persons,
include_keypoints=request.include_keypoints,
include_segmentation=request.include_segmentation
)
# Schedule background processing if needed
if request.include_segmentation:
background_tasks.add_task(
pose_service.process_segmentation_data,
result["frame_id"]
)
return PoseEstimationResponse(**result)
except Exception as e:
logger.error(f"Error in pose analysis: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/zones/{zone_id}/occupancy")
async def get_zone_occupancy(
zone_id: str,
pose_service: PoseService = Depends(get_pose_service),
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get current occupancy for a specific zone."""
try:
occupancy = await pose_service.get_zone_occupancy(zone_id)
if occupancy is None:
raise HTTPException(
status_code=404,
detail=f"Zone '{zone_id}' not found"
)
return {
"zone_id": zone_id,
"current_occupancy": occupancy["count"],
"max_occupancy": occupancy.get("max_occupancy"),
"persons": occupancy["persons"],
"timestamp": occupancy["timestamp"]
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting zone occupancy: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/zones/summary")
async def get_zones_summary(
pose_service: PoseService = Depends(get_pose_service),
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get occupancy summary for all zones."""
try:
summary = await pose_service.get_zones_summary()
return {
"timestamp": datetime.utcnow(),
"total_persons": summary["total_persons"],
"zones": summary["zones"],
"active_zones": summary["active_zones"]
}
except Exception as e:
logger.error(f"Error getting zones summary: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/historical")
async def get_historical_data(
request: HistoricalDataRequest,
pose_service: PoseService = Depends(get_pose_service),
current_user: Dict = Depends(require_auth)
):
"""Get historical pose estimation data."""
try:
# Validate time range
if request.end_time <= request.start_time:
raise HTTPException(
status_code=400,
detail="End time must be after start time"
)
# Limit query range to prevent excessive data
max_range = timedelta(days=7)
if request.end_time - request.start_time > max_range:
raise HTTPException(
status_code=400,
detail="Query range cannot exceed 7 days"
)
data = await pose_service.get_historical_data(
start_time=request.start_time,
end_time=request.end_time,
zone_ids=request.zone_ids,
aggregation_interval=request.aggregation_interval,
include_raw_data=request.include_raw_data
)
return {
"query": {
"start_time": request.start_time,
"end_time": request.end_time,
"zone_ids": request.zone_ids,
"aggregation_interval": request.aggregation_interval
},
"data": data["aggregated_data"],
"raw_data": data.get("raw_data") if request.include_raw_data else None,
"total_records": data["total_records"]
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting historical data: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/activities")
async def get_detected_activities(
zone_id: Optional[str] = Query(None, description="Filter by zone ID"),
limit: int = Query(10, ge=1, le=100, description="Maximum number of activities"),
pose_service: PoseService = Depends(get_pose_service),
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get recently detected activities."""
try:
activities = await pose_service.get_recent_activities(
zone_id=zone_id,
limit=limit
)
return {
"activities": activities,
"total_count": len(activities),
"zone_id": zone_id
}
except Exception as e:
logger.error(f"Error getting activities: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/calibrate")
async def calibrate_pose_system(
background_tasks: BackgroundTasks,
pose_service: PoseService = Depends(get_pose_service),
hardware_service: HardwareService = Depends(get_hardware_service),
current_user: Dict = Depends(require_auth)
):
"""Calibrate the pose estimation system."""
try:
logger.info(f"Pose system calibration initiated by user: {current_user['id']}")
# Check if calibration is already in progress
if await pose_service.is_calibrating():
raise HTTPException(
status_code=409,
detail="Calibration already in progress"
)
# Start calibration process
calibration_id = await pose_service.start_calibration()
# Schedule background calibration task
background_tasks.add_task(
pose_service.run_calibration,
calibration_id
)
return {
"calibration_id": calibration_id,
"status": "started",
"estimated_duration_minutes": 5,
"message": "Calibration process started"
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error starting calibration: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/calibration/status")
async def get_calibration_status(
pose_service: PoseService = Depends(get_pose_service),
current_user: Dict = Depends(require_auth)
):
"""Get current calibration status."""
try:
status = await pose_service.get_calibration_status()
return {
"is_calibrating": status["is_calibrating"],
"calibration_id": status.get("calibration_id"),
"progress_percent": status.get("progress_percent", 0),
"current_step": status.get("current_step"),
"estimated_remaining_minutes": status.get("estimated_remaining_minutes"),
"last_calibration": status.get("last_calibration")
}
except Exception as e:
logger.error(f"Error getting calibration status: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/stats")
async def get_pose_statistics(
hours: int = Query(24, ge=1, le=168, description="Hours of data to analyze"),
pose_service: PoseService = Depends(get_pose_service),
current_user: Optional[Dict] = Depends(get_current_user)
):
"""Get pose estimation statistics."""
try:
end_time = datetime.utcnow()
start_time = end_time - timedelta(hours=hours)
stats = await pose_service.get_statistics(
start_time=start_time,
end_time=end_time
)
return {
"period": {
"start_time": start_time,
"end_time": end_time,
"hours": hours
},
"statistics": stats
}
except Exception as e:
logger.error(f"Error getting statistics: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
archive/v1/src/api/routers/stream.py
+523
View File
@@ -0,0 +1,523 @@
"""
WebSocket streaming API endpoints
"""
import asyncio
import json
import logging
from typing import Dict, List, Optional, Any
from datetime import datetime
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Depends, HTTPException, Query
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from src.api.dependencies import (
get_stream_service,
get_pose_service,
get_current_user_ws,
require_auth
)
from src.api.websocket.connection_manager import connection_manager
from src.services.stream_service import StreamService
from src.services.pose_service import PoseService
logger = logging.getLogger(__name__)
router = APIRouter()
# Request/Response models
class StreamSubscriptionRequest(BaseModel):
"""Request model for stream subscription."""
zone_ids: Optional[List[str]] = Field(
default=None,
description="Zones to subscribe to (all zones if not specified)"
)
stream_types: List[str] = Field(
default=["pose_data"],
description="Types of data to stream"
)
min_confidence: float = Field(
default=0.5,
ge=0.0,
le=1.0,
description="Minimum confidence threshold for streaming"
)
max_fps: int = Field(
default=30,
ge=1,
le=60,
description="Maximum frames per second"
)
include_metadata: bool = Field(
default=True,
description="Include metadata in stream"
)
class StreamStatus(BaseModel):
"""Stream status model."""
is_active: bool = Field(..., description="Whether streaming is active")
connected_clients: int = Field(..., description="Number of connected clients")
streams: List[Dict[str, Any]] = Field(..., description="Active streams")
uptime_seconds: float = Field(..., description="Stream uptime in seconds")
# WebSocket endpoints
@router.websocket("/pose")
async def websocket_pose_stream(
websocket: WebSocket,
zone_ids: Optional[str] = Query(None, description="Comma-separated zone IDs"),
min_confidence: float = Query(0.5, ge=0.0, le=1.0),
max_fps: int = Query(30, ge=1, le=60),
):
"""WebSocket endpoint for real-time pose data streaming."""
client_id = None
try:
# Accept WebSocket connection
await websocket.accept()
# First-message authentication (CWE-598 fix: no JWT in URL)
from src.config.settings import get_settings
settings = get_settings()
if settings.enable_authentication:
try:
raw = await asyncio.wait_for(websocket.receive_text(), timeout=10.0)
auth_msg = json.loads(raw)
if auth_msg.get("type") != "auth" or not auth_msg.get("token"):
await websocket.send_json({
"type": "error",
"message": "First message must be {\"type\": \"auth\", \"token\": \"<jwt>\"}"
})
await websocket.close(code=1008)
return
# Verify the token
from src.middleware.auth import get_auth_middleware
auth_middleware = get_auth_middleware(settings)
try:
auth_middleware.token_manager.verify_token(auth_msg["token"])
except Exception:
await websocket.send_json({
"type": "error",
"message": "Invalid or expired authentication token"
})
await websocket.close(code=1008)
return
except asyncio.TimeoutError:
await websocket.send_json({
"type": "error",
"message": "Authentication timeout: no auth message received within 10 seconds"
})
await websocket.close(code=1008)
return
except (json.JSONDecodeError, Exception) as e:
await websocket.send_json({
"type": "error",
"message": "Invalid authentication message format"
})
await websocket.close(code=1008)
return
# Parse zone IDs
zone_list = None
if zone_ids:
zone_list = [zone.strip() for zone in zone_ids.split(",") if zone.strip()]
# Register client with connection manager
client_id = await connection_manager.connect(
websocket=websocket,
stream_type="pose",
zone_ids=zone_list,
min_confidence=min_confidence,
max_fps=max_fps
)
logger.info(f"WebSocket client {client_id} connected for pose streaming")
# Send initial connection confirmation
await websocket.send_json({
"type": "connection_established",
"client_id": client_id,
"timestamp": datetime.utcnow().isoformat(),
"config": {
"zone_ids": zone_list,
"min_confidence": min_confidence,
"max_fps": max_fps
}
})
# Keep connection alive and handle incoming messages
while True:
try:
# Wait for client messages (ping, config updates, etc.)
message = await websocket.receive_text()
data = json.loads(message)
await handle_websocket_message(client_id, data, websocket)
except WebSocketDisconnect:
break
except json.JSONDecodeError:
await websocket.send_json({
"type": "error",
"message": "Invalid JSON format"
})
except Exception as e:
logger.error(f"Error handling WebSocket message: {e}")
await websocket.send_json({
"type": "error",
"message": "Internal server error"
})
except WebSocketDisconnect:
logger.info(f"WebSocket client {client_id} disconnected")
except Exception as e:
logger.error(f"WebSocket error: {e}")
finally:
if client_id:
await connection_manager.disconnect(client_id)
@router.websocket("/events")
async def websocket_events_stream(
websocket: WebSocket,
event_types: Optional[str] = Query(None, description="Comma-separated event types"),
zone_ids: Optional[str] = Query(None, description="Comma-separated zone IDs"),
):
"""WebSocket endpoint for real-time event streaming."""
client_id = None
try:
await websocket.accept()
# First-message authentication (CWE-598 fix: no JWT in URL)
from src.config.settings import get_settings
settings = get_settings()
if settings.enable_authentication:
try:
raw = await asyncio.wait_for(websocket.receive_text(), timeout=10.0)
auth_msg = json.loads(raw)
if auth_msg.get("type") != "auth" or not auth_msg.get("token"):
await websocket.send_json({
"type": "error",
"message": "First message must be {\"type\": \"auth\", \"token\": \"<jwt>\"}"
})
await websocket.close(code=1008)
return
from src.middleware.auth import get_auth_middleware
auth_middleware = get_auth_middleware(settings)
try:
auth_middleware.token_manager.verify_token(auth_msg["token"])
except Exception:
await websocket.send_json({
"type": "error",
"message": "Invalid or expired authentication token"
})
await websocket.close(code=1008)
return
except asyncio.TimeoutError:
await websocket.send_json({
"type": "error",
"message": "Authentication timeout: no auth message received within 10 seconds"
})
await websocket.close(code=1008)
return
except (json.JSONDecodeError, Exception) as e:
await websocket.send_json({
"type": "error",
"message": "Invalid authentication message format"
})
await websocket.close(code=1008)
return
# Parse parameters
event_list = None
if event_types:
event_list = [event.strip() for event in event_types.split(",") if event.strip()]
zone_list = None
if zone_ids:
zone_list = [zone.strip() for zone in zone_ids.split(",") if zone.strip()]
# Register client
client_id = await connection_manager.connect(
websocket=websocket,
stream_type="events",
zone_ids=zone_list,
event_types=event_list
)
logger.info(f"WebSocket client {client_id} connected for event streaming")
# Send confirmation
await websocket.send_json({
"type": "connection_established",
"client_id": client_id,
"timestamp": datetime.utcnow().isoformat(),
"config": {
"event_types": event_list,
"zone_ids": zone_list
}
})
# Handle messages
while True:
try:
message = await websocket.receive_text()
data = json.loads(message)
await handle_websocket_message(client_id, data, websocket)
except WebSocketDisconnect:
break
except Exception as e:
logger.error(f"Error in events WebSocket: {e}")
except WebSocketDisconnect:
logger.info(f"Events WebSocket client {client_id} disconnected")
except Exception as e:
logger.error(f"Events WebSocket error: {e}")
finally:
if client_id:
await connection_manager.disconnect(client_id)
async def handle_websocket_message(client_id: str, data: Dict[str, Any], websocket: WebSocket):
"""Handle incoming WebSocket messages."""
message_type = data.get("type")
if message_type == "ping":
await websocket.send_json({
"type": "pong",
"timestamp": datetime.utcnow().isoformat()
})
elif message_type == "update_config":
# Update client configuration
config = data.get("config", {})
await connection_manager.update_client_config(client_id, config)
await websocket.send_json({
"type": "config_updated",
"timestamp": datetime.utcnow().isoformat(),
"config": config
})
elif message_type == "get_status":
# Send current status
status = await connection_manager.get_client_status(client_id)
await websocket.send_json({
"type": "status",
"timestamp": datetime.utcnow().isoformat(),
"status": status
})
else:
await websocket.send_json({
"type": "error",
"message": f"Unknown message type: {message_type}"
})
# HTTP endpoints for stream management
@router.get("/status", response_model=StreamStatus)
async def get_stream_status(
stream_service: StreamService = Depends(get_stream_service)
):
"""Get current streaming status."""
try:
status = await stream_service.get_status()
connections = await connection_manager.get_connection_stats()
# Calculate uptime (simplified for now)
uptime_seconds = 0.0
if status.get("running", False):
uptime_seconds = 3600.0 # Default 1 hour for demo
return StreamStatus(
is_active=status.get("running", False),
connected_clients=connections.get("total_clients", status["connections"]["active"]),
streams=[{
"type": "pose_stream",
"active": status.get("running", False),
"buffer_size": status["buffers"]["pose_buffer_size"]
}],
uptime_seconds=uptime_seconds
)
except Exception as e:
logger.error(f"Error getting stream status: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/start")
async def start_streaming(
stream_service: StreamService = Depends(get_stream_service),
current_user: Dict = Depends(require_auth)
):
"""Start the streaming service."""
try:
logger.info(f"Starting streaming service by user: {current_user['id']}")
if await stream_service.is_active():
return JSONResponse(
status_code=200,
content={"message": "Streaming service is already active"}
)
await stream_service.start()
return {
"message": "Streaming service started successfully",
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error starting streaming: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/stop")
async def stop_streaming(
stream_service: StreamService = Depends(get_stream_service),
current_user: Dict = Depends(require_auth)
):
"""Stop the streaming service."""
try:
logger.info(f"Stopping streaming service by user: {current_user['id']}")
await stream_service.stop()
await connection_manager.disconnect_all()
return {
"message": "Streaming service stopped successfully",
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error stopping streaming: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/clients")
async def get_connected_clients(
current_user: Dict = Depends(require_auth)
):
"""Get list of connected WebSocket clients."""
try:
clients = await connection_manager.get_connected_clients()
return {
"total_clients": len(clients),
"clients": clients,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error getting connected clients: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.delete("/clients/{client_id}")
async def disconnect_client(
client_id: str,
current_user: Dict = Depends(require_auth)
):
"""Disconnect a specific WebSocket client."""
try:
logger.info(f"Disconnecting client {client_id} by user: {current_user['id']}")
success = await connection_manager.disconnect(client_id)
if not success:
raise HTTPException(
status_code=404,
detail=f"Client {client_id} not found"
)
return {
"message": f"Client {client_id} disconnected successfully",
"timestamp": datetime.utcnow().isoformat()
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error disconnecting client: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.post("/broadcast")
async def broadcast_message(
message: Dict[str, Any],
stream_type: Optional[str] = Query(None, description="Target stream type"),
zone_ids: Optional[List[str]] = Query(None, description="Target zone IDs"),
current_user: Dict = Depends(require_auth)
):
"""Broadcast a message to connected WebSocket clients."""
try:
logger.info(f"Broadcasting message by user: {current_user['id']}")
# Add metadata to message
broadcast_data = {
**message,
"broadcast_timestamp": datetime.utcnow().isoformat(),
"sender": current_user["id"]
}
# Broadcast to matching clients
sent_count = await connection_manager.broadcast(
data=broadcast_data,
stream_type=stream_type,
zone_ids=zone_ids
)
return {
"message": "Broadcast sent successfully",
"recipients": sent_count,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error broadcasting message: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
@router.get("/metrics")
async def get_streaming_metrics():
"""Get streaming performance metrics."""
try:
metrics = await connection_manager.get_metrics()
return {
"metrics": metrics,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error getting streaming metrics: {e}")
raise HTTPException(
status_code=500,
detail="An internal error occurred. Please try again later."
)
archive/v1/src/api/websocket/__init__.py
+8
View File
@@ -0,0 +1,8 @@
"""
WebSocket handlers package
"""
from .connection_manager import ConnectionManager
from .pose_stream import PoseStreamHandler
__all__ = ["ConnectionManager", "PoseStreamHandler"]
archive/v1/src/api/websocket/connection_manager.py
+461
View File
@@ -0,0 +1,461 @@
"""
WebSocket connection manager for WiFi-DensePose API
"""
import asyncio
import json
import logging
import uuid
from typing import Dict, List, Optional, Any, Set
from datetime import datetime, timedelta
from collections import defaultdict
from fastapi import WebSocket, WebSocketDisconnect
logger = logging.getLogger(__name__)
class WebSocketConnection:
"""Represents a WebSocket connection with metadata."""
def __init__(
self,
websocket: WebSocket,
client_id: str,
stream_type: str,
zone_ids: Optional[List[str]] = None,
**config
):
self.websocket = websocket
self.client_id = client_id
self.stream_type = stream_type
self.zone_ids = zone_ids or []
self.config = config
self.connected_at = datetime.utcnow()
self.last_ping = datetime.utcnow()
self.message_count = 0
self.is_active = True
async def send_json(self, data: Dict[str, Any]):
"""Send JSON data to client."""
try:
await self.websocket.send_json(data)
self.message_count += 1
except Exception as e:
logger.error(f"Error sending to client {self.client_id}: {e}")
self.is_active = False
raise
async def send_text(self, message: str):
"""Send text message to client."""
try:
await self.websocket.send_text(message)
self.message_count += 1
except Exception as e:
logger.error(f"Error sending text to client {self.client_id}: {e}")
self.is_active = False
raise
def update_config(self, config: Dict[str, Any]):
"""Update connection configuration."""
self.config.update(config)
# Update zone IDs if provided
if "zone_ids" in config:
self.zone_ids = config["zone_ids"] or []
def matches_filter(
self,
stream_type: Optional[str] = None,
zone_ids: Optional[List[str]] = None,
**filters
) -> bool:
"""Check if connection matches given filters."""
# Check stream type
if stream_type and self.stream_type != stream_type:
return False
# Check zone IDs
if zone_ids:
if not self.zone_ids: # Connection listens to all zones
return True
# Check if any requested zone is in connection's zones
if not any(zone in self.zone_ids for zone in zone_ids):
return False
# Check additional filters
for key, value in filters.items():
if key in self.config and self.config[key] != value:
return False
return True
def get_info(self) -> Dict[str, Any]:
"""Get connection information."""
return {
"client_id": self.client_id,
"stream_type": self.stream_type,
"zone_ids": self.zone_ids,
"config": self.config,
"connected_at": self.connected_at.isoformat(),
"last_ping": self.last_ping.isoformat(),
"message_count": self.message_count,
"is_active": self.is_active,
"uptime_seconds": (datetime.utcnow() - self.connected_at).total_seconds()
}
class ConnectionManager:
"""Manages WebSocket connections for real-time streaming."""
def __init__(self):
self.connections: Dict[str, WebSocketConnection] = {}
self.connections_by_type: Dict[str, Set[str]] = defaultdict(set)
self.connections_by_zone: Dict[str, Set[str]] = defaultdict(set)
self.metrics = {
"total_connections": 0,
"active_connections": 0,
"messages_sent": 0,
"errors": 0,
"start_time": datetime.utcnow()
}
self._cleanup_task = None
self._started = False
async def connect(
self,
websocket: WebSocket,
stream_type: str,
zone_ids: Optional[List[str]] = None,
**config
) -> str:
"""Register a new WebSocket connection."""
client_id = str(uuid.uuid4())
try:
# Create connection object
connection = WebSocketConnection(
websocket=websocket,
client_id=client_id,
stream_type=stream_type,
zone_ids=zone_ids,
**config
)
# Store connection
self.connections[client_id] = connection
self.connections_by_type[stream_type].add(client_id)
# Index by zones
if zone_ids:
for zone_id in zone_ids:
self.connections_by_zone[zone_id].add(client_id)
# Update metrics
self.metrics["total_connections"] += 1
self.metrics["active_connections"] = len(self.connections)
logger.info(f"WebSocket client {client_id} connected for {stream_type}")
return client_id
except Exception as e:
logger.error(f"Error connecting WebSocket client: {e}")
raise
async def disconnect(self, client_id: str) -> bool:
"""Disconnect a WebSocket client."""
if client_id not in self.connections:
return False
try:
connection = self.connections[client_id]
# Remove from indexes
self.connections_by_type[connection.stream_type].discard(client_id)
for zone_id in connection.zone_ids:
self.connections_by_zone[zone_id].discard(client_id)
# Close WebSocket if still active
if connection.is_active:
try:
await connection.websocket.close()
except Exception:
pass # Connection might already be closed
# Remove connection
del self.connections[client_id]
# Update metrics
self.metrics["active_connections"] = len(self.connections)
logger.info(f"WebSocket client {client_id} disconnected")
return True
except Exception as e:
logger.error(f"Error disconnecting client {client_id}: {e}")
return False
async def disconnect_all(self):
"""Disconnect all WebSocket clients."""
client_ids = list(self.connections.keys())
for client_id in client_ids:
await self.disconnect(client_id)
logger.info("All WebSocket clients disconnected")
async def send_to_client(self, client_id: str, data: Dict[str, Any]) -> bool:
"""Send data to a specific client."""
if client_id not in self.connections:
return False
connection = self.connections[client_id]
try:
await connection.send_json(data)
self.metrics["messages_sent"] += 1
return True
except Exception as e:
logger.error(f"Error sending to client {client_id}: {e}")
self.metrics["errors"] += 1
# Mark connection as inactive and schedule for cleanup
connection.is_active = False
return False
async def broadcast(
self,
data: Dict[str, Any],
stream_type: Optional[str] = None,
zone_ids: Optional[List[str]] = None,
**filters
) -> int:
"""Broadcast data to matching clients."""
sent_count = 0
failed_clients = []
# Get matching connections
matching_clients = self._get_matching_clients(
stream_type=stream_type,
zone_ids=zone_ids,
**filters
)
# Send to all matching clients
for client_id in matching_clients:
try:
success = await self.send_to_client(client_id, data)
if success:
sent_count += 1
else:
failed_clients.append(client_id)
except Exception as e:
logger.error(f"Error broadcasting to client {client_id}: {e}")
failed_clients.append(client_id)
# Clean up failed connections
for client_id in failed_clients:
await self.disconnect(client_id)
return sent_count
async def update_client_config(self, client_id: str, config: Dict[str, Any]) -> bool:
"""Update client configuration."""
if client_id not in self.connections:
return False
connection = self.connections[client_id]
old_zones = set(connection.zone_ids)
# Update configuration
connection.update_config(config)
# Update zone indexes if zones changed
new_zones = set(connection.zone_ids)
# Remove from old zones
for zone_id in old_zones - new_zones:
self.connections_by_zone[zone_id].discard(client_id)
# Add to new zones
for zone_id in new_zones - old_zones:
self.connections_by_zone[zone_id].add(client_id)
return True
async def get_client_status(self, client_id: str) -> Optional[Dict[str, Any]]:
"""Get status of a specific client."""
if client_id not in self.connections:
return None
return self.connections[client_id].get_info()
async def get_connected_clients(self) -> List[Dict[str, Any]]:
"""Get list of all connected clients."""
return [conn.get_info() for conn in self.connections.values()]
async def get_connection_stats(self) -> Dict[str, Any]:
"""Get connection statistics."""
stats = {
"total_clients": len(self.connections),
"clients_by_type": {
stream_type: len(clients)
for stream_type, clients in self.connections_by_type.items()
},
"clients_by_zone": {
zone_id: len(clients)
for zone_id, clients in self.connections_by_zone.items()
if clients # Only include zones with active clients
},
"active_clients": sum(1 for conn in self.connections.values() if conn.is_active),
"inactive_clients": sum(1 for conn in self.connections.values() if not conn.is_active)
}
return stats
async def get_metrics(self) -> Dict[str, Any]:
"""Get detailed metrics."""
uptime = (datetime.utcnow() - self.metrics["start_time"]).total_seconds()
return {
**self.metrics,
"active_connections": len(self.connections),
"uptime_seconds": uptime,
"messages_per_second": self.metrics["messages_sent"] / max(uptime, 1),
"error_rate": self.metrics["errors"] / max(self.metrics["messages_sent"], 1)
}
def _get_matching_clients(
self,
stream_type: Optional[str] = None,
zone_ids: Optional[List[str]] = None,
**filters
) -> List[str]:
"""Get client IDs that match the given filters."""
candidates = set(self.connections.keys())
# Filter by stream type
if stream_type:
type_clients = self.connections_by_type.get(stream_type, set())
candidates &= type_clients
# Filter by zones
if zone_ids:
zone_clients = set()
for zone_id in zone_ids:
zone_clients.update(self.connections_by_zone.get(zone_id, set()))
# Also include clients listening to all zones (empty zone list)
all_zone_clients = {
client_id for client_id, conn in self.connections.items()
if not conn.zone_ids
}
zone_clients.update(all_zone_clients)
candidates &= zone_clients
# Apply additional filters
matching_clients = []
for client_id in candidates:
connection = self.connections[client_id]
if connection.is_active and connection.matches_filter(**filters):
matching_clients.append(client_id)
return matching_clients
async def ping_clients(self):
"""Send ping to all connected clients."""
ping_data = {
"type": "ping",
"timestamp": datetime.utcnow().isoformat()
}
failed_clients = []
for client_id, connection in self.connections.items():
try:
await connection.send_json(ping_data)
connection.last_ping = datetime.utcnow()
except Exception as e:
logger.warning(f"Ping failed for client {client_id}: {e}")
failed_clients.append(client_id)
# Clean up failed connections
for client_id in failed_clients:
await self.disconnect(client_id)
async def cleanup_inactive_connections(self):
"""Clean up inactive or stale connections."""
now = datetime.utcnow()
stale_threshold = timedelta(minutes=5) # 5 minutes without ping
stale_clients = []
for client_id, connection in self.connections.items():
# Check if connection is inactive
if not connection.is_active:
stale_clients.append(client_id)
continue
# Check if connection is stale (no ping response)
if now - connection.last_ping > stale_threshold:
logger.warning(f"Client {client_id} appears stale, disconnecting")
stale_clients.append(client_id)
# Clean up stale connections
for client_id in stale_clients:
await self.disconnect(client_id)
if stale_clients:
logger.info(f"Cleaned up {len(stale_clients)} stale connections")
async def start(self):
"""Start the connection manager."""
if not self._started:
self._start_cleanup_task()
self._started = True
logger.info("Connection manager started")
def _start_cleanup_task(self):
"""Start background cleanup task."""
async def cleanup_loop():
while True:
try:
await asyncio.sleep(60) # Run every minute
await self.cleanup_inactive_connections()
# Send periodic ping every 2 minutes
if datetime.utcnow().minute % 2 == 0:
await self.ping_clients()
except Exception as e:
logger.error(f"Error in cleanup task: {e}")
try:
self._cleanup_task = asyncio.create_task(cleanup_loop())
except RuntimeError:
# No event loop running, will start later
logger.debug("No event loop running, cleanup task will start later")
async def shutdown(self):
"""Shutdown connection manager."""
# Cancel cleanup task
if self._cleanup_task:
self._cleanup_task.cancel()
try:
await self._cleanup_task
except asyncio.CancelledError:
pass
# Disconnect all clients
await self.disconnect_all()
logger.info("Connection manager shutdown complete")
# Global connection manager instance
connection_manager = ConnectionManager()
archive/v1/src/api/websocket/pose_stream.py
+384
View File
@@ -0,0 +1,384 @@
"""
Pose streaming WebSocket handler
"""
import asyncio
import json
import logging
from typing import Dict, List, Optional, Any
from datetime import datetime
from fastapi import WebSocket
from pydantic import BaseModel, Field
from src.api.websocket.connection_manager import ConnectionManager
from src.services.pose_service import PoseService
from src.services.stream_service import StreamService
logger = logging.getLogger(__name__)
class PoseStreamData(BaseModel):
"""Pose stream data model."""
timestamp: datetime = Field(..., description="Data timestamp")
zone_id: str = Field(..., description="Zone identifier")
pose_data: Dict[str, Any] = Field(..., description="Pose estimation data")
confidence: float = Field(..., ge=0.0, le=1.0, description="Confidence score")
activity: Optional[str] = Field(default=None, description="Detected activity")
metadata: Optional[Dict[str, Any]] = Field(default=None, description="Additional metadata")
class PoseStreamHandler:
"""Handles pose data streaming to WebSocket clients."""
def __init__(
self,
connection_manager: ConnectionManager,
pose_service: PoseService,
stream_service: StreamService
):
self.connection_manager = connection_manager
self.pose_service = pose_service
self.stream_service = stream_service
self.is_streaming = False
self.stream_task = None
self.subscribers = {}
self.stream_config = {
"fps": 30,
"min_confidence": 0.5,
"include_metadata": True,
"buffer_size": 100
}
async def start_streaming(self):
"""Start pose data streaming."""
if self.is_streaming:
logger.warning("Pose streaming already active")
return
self.is_streaming = True
self.stream_task = asyncio.create_task(self._stream_loop())
logger.info("Pose streaming started")
async def stop_streaming(self):
"""Stop pose data streaming."""
if not self.is_streaming:
return
self.is_streaming = False
if self.stream_task:
self.stream_task.cancel()
try:
await self.stream_task
except asyncio.CancelledError:
pass
logger.info("Pose streaming stopped")
async def _stream_loop(self):
"""Main streaming loop."""
try:
logger.info("🚀 Starting pose streaming loop")
while self.is_streaming:
try:
# Get current pose data from all zones
logger.debug("📡 Getting current pose data...")
pose_data = await self.pose_service.get_current_pose_data()
logger.debug(f"📊 Received pose data: {pose_data}")
if pose_data:
logger.debug("📤 Broadcasting pose data...")
await self._process_and_broadcast_pose_data(pose_data)
else:
logger.debug("⚠️ No pose data received")
# Control streaming rate
await asyncio.sleep(1.0 / self.stream_config["fps"])
except Exception as e:
logger.error(f"Error in pose streaming loop: {e}")
await asyncio.sleep(1.0) # Brief pause on error
except asyncio.CancelledError:
logger.info("Pose streaming loop cancelled")
except Exception as e:
logger.error(f"Fatal error in pose streaming loop: {e}")
finally:
logger.info("🛑 Pose streaming loop stopped")
self.is_streaming = False
async def _process_and_broadcast_pose_data(self, raw_pose_data: Dict[str, Any]):
"""Process and broadcast pose data to subscribers."""
try:
# Process data for each zone
for zone_id, zone_data in raw_pose_data.items():
if not zone_data:
continue
# Create structured pose data
pose_stream_data = PoseStreamData(
timestamp=datetime.utcnow(),
zone_id=zone_id,
pose_data=zone_data.get("pose", {}),
confidence=zone_data.get("confidence", 0.0),
activity=zone_data.get("activity"),
metadata=zone_data.get("metadata") if self.stream_config["include_metadata"] else None
)
# Filter by minimum confidence
if pose_stream_data.confidence < self.stream_config["min_confidence"]:
continue
# Broadcast to subscribers
await self._broadcast_pose_data(pose_stream_data)
except Exception as e:
logger.error(f"Error processing pose data: {e}")
async def _broadcast_pose_data(self, pose_data: PoseStreamData):
"""Broadcast pose data to matching WebSocket clients."""
try:
logger.debug(f"📡 Preparing to broadcast pose data for zone {pose_data.zone_id}")
# Prepare broadcast data
broadcast_data = {
"type": "pose_data",
"timestamp": pose_data.timestamp.isoformat(),
"zone_id": pose_data.zone_id,
"data": {
"pose": pose_data.pose_data,
"confidence": pose_data.confidence,
"activity": pose_data.activity
}
}
# Add metadata if enabled
if pose_data.metadata and self.stream_config["include_metadata"]:
broadcast_data["metadata"] = pose_data.metadata
logger.debug(f"📤 Broadcasting data: {broadcast_data}")
# Broadcast to pose stream subscribers
sent_count = await self.connection_manager.broadcast(
data=broadcast_data,
stream_type="pose",
zone_ids=[pose_data.zone_id]
)
logger.info(f"✅ Broadcasted pose data for zone {pose_data.zone_id} to {sent_count} clients")
except Exception as e:
logger.error(f"Error broadcasting pose data: {e}")
async def handle_client_subscription(
self,
client_id: str,
subscription_config: Dict[str, Any]
):
"""Handle client subscription configuration."""
try:
# Store client subscription config
self.subscribers[client_id] = {
"zone_ids": subscription_config.get("zone_ids", []),
"min_confidence": subscription_config.get("min_confidence", 0.5),
"max_fps": subscription_config.get("max_fps", 30),
"include_metadata": subscription_config.get("include_metadata", True),
"stream_types": subscription_config.get("stream_types", ["pose_data"]),
"subscribed_at": datetime.utcnow()
}
logger.info(f"Updated subscription for client {client_id}")
# Send confirmation
confirmation = {
"type": "subscription_updated",
"client_id": client_id,
"config": self.subscribers[client_id],
"timestamp": datetime.utcnow().isoformat()
}
await self.connection_manager.send_to_client(client_id, confirmation)
except Exception as e:
logger.error(f"Error handling client subscription: {e}")
async def handle_client_disconnect(self, client_id: str):
"""Handle client disconnection."""
if client_id in self.subscribers:
del self.subscribers[client_id]
logger.info(f"Removed subscription for disconnected client {client_id}")
async def send_historical_data(
self,
client_id: str,
zone_id: str,
start_time: datetime,
end_time: datetime,
limit: int = 100
):
"""Send historical pose data to client."""
try:
# Get historical data from pose service
historical_data = await self.pose_service.get_historical_data(
zone_id=zone_id,
start_time=start_time,
end_time=end_time,
limit=limit
)
# Send data in chunks to avoid overwhelming the client
chunk_size = 10
for i in range(0, len(historical_data), chunk_size):
chunk = historical_data[i:i + chunk_size]
message = {
"type": "historical_data",
"zone_id": zone_id,
"chunk_index": i // chunk_size,
"total_chunks": (len(historical_data) + chunk_size - 1) // chunk_size,
"data": chunk,
"timestamp": datetime.utcnow().isoformat()
}
await self.connection_manager.send_to_client(client_id, message)
# Small delay between chunks
await asyncio.sleep(0.1)
# Send completion message
completion_message = {
"type": "historical_data_complete",
"zone_id": zone_id,
"total_records": len(historical_data),
"timestamp": datetime.utcnow().isoformat()
}
await self.connection_manager.send_to_client(client_id, completion_message)
except Exception as e:
logger.error(f"Error sending historical data: {e}")
# Send error message to client
error_message = {
"type": "error",
"message": f"Failed to retrieve historical data: {str(e)}",
"timestamp": datetime.utcnow().isoformat()
}
await self.connection_manager.send_to_client(client_id, error_message)
async def send_zone_statistics(self, client_id: str, zone_id: str):
"""Send zone statistics to client."""
try:
# Get zone statistics
stats = await self.pose_service.get_zone_statistics(zone_id)
message = {
"type": "zone_statistics",
"zone_id": zone_id,
"statistics": stats,
"timestamp": datetime.utcnow().isoformat()
}
await self.connection_manager.send_to_client(client_id, message)
except Exception as e:
logger.error(f"Error sending zone statistics: {e}")
async def broadcast_system_event(self, event_type: str, event_data: Dict[str, Any]):
"""Broadcast system events to all connected clients."""
try:
message = {
"type": "system_event",
"event_type": event_type,
"data": event_data,
"timestamp": datetime.utcnow().isoformat()
}
# Broadcast to all pose stream clients
sent_count = await self.connection_manager.broadcast(
data=message,
stream_type="pose"
)
logger.info(f"Broadcasted system event '{event_type}' to {sent_count} clients")
except Exception as e:
logger.error(f"Error broadcasting system event: {e}")
async def update_stream_config(self, config: Dict[str, Any]):
"""Update streaming configuration."""
try:
# Validate and update configuration
if "fps" in config:
fps = max(1, min(60, config["fps"]))
self.stream_config["fps"] = fps
if "min_confidence" in config:
confidence = max(0.0, min(1.0, config["min_confidence"]))
self.stream_config["min_confidence"] = confidence
if "include_metadata" in config:
self.stream_config["include_metadata"] = bool(config["include_metadata"])
if "buffer_size" in config:
buffer_size = max(10, min(1000, config["buffer_size"]))
self.stream_config["buffer_size"] = buffer_size
logger.info(f"Updated stream configuration: {self.stream_config}")
# Broadcast configuration update to clients
await self.broadcast_system_event("stream_config_updated", {
"new_config": self.stream_config
})
except Exception as e:
logger.error(f"Error updating stream configuration: {e}")
def get_stream_status(self) -> Dict[str, Any]:
"""Get current streaming status."""
return {
"is_streaming": self.is_streaming,
"config": self.stream_config,
"subscriber_count": len(self.subscribers),
"subscribers": {
client_id: {
"zone_ids": sub["zone_ids"],
"min_confidence": sub["min_confidence"],
"subscribed_at": sub["subscribed_at"].isoformat()
}
for client_id, sub in self.subscribers.items()
}
}
async def get_performance_metrics(self) -> Dict[str, Any]:
"""Get streaming performance metrics."""
try:
# Get connection manager metrics
conn_metrics = await self.connection_manager.get_metrics()
# Get pose service metrics
pose_metrics = await self.pose_service.get_performance_metrics()
return {
"streaming": {
"is_active": self.is_streaming,
"fps": self.stream_config["fps"],
"subscriber_count": len(self.subscribers)
},
"connections": conn_metrics,
"pose_service": pose_metrics,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error getting performance metrics: {e}")
return {}
async def shutdown(self):
"""Shutdown pose stream handler."""
await self.stop_streaming()
self.subscribers.clear()
logger.info("Pose stream handler shutdown complete")
archive/v1/src/app.py
+337
View File
@@ -0,0 +1,337 @@
"""
FastAPI application factory and configuration
"""
import logging
from contextlib import asynccontextmanager
from typing import Optional
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.middleware.trustedhost import TrustedHostMiddleware
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from src.config.settings import Settings
from src.services.orchestrator import ServiceOrchestrator
from src.middleware.auth import AuthenticationMiddleware
from src.middleware.rate_limit import RateLimitMiddleware
from src.middleware.error_handler import ErrorHandlingMiddleware
from src.api.routers import pose, stream, health
from src.api.websocket.connection_manager import connection_manager
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan manager."""
logger.info("Starting WiFi-DensePose API...")
try:
# Get orchestrator from app state
orchestrator: ServiceOrchestrator = app.state.orchestrator
# Start connection manager
await connection_manager.start()
# Start all services
await orchestrator.start()
logger.info("WiFi-DensePose API started successfully")
yield
except Exception as e:
logger.error(f"Failed to start application: {e}")
raise
finally:
# Cleanup on shutdown
logger.info("Shutting down WiFi-DensePose API...")
# Shutdown connection manager
await connection_manager.shutdown()
if hasattr(app.state, 'orchestrator'):
await app.state.orchestrator.shutdown()
logger.info("WiFi-DensePose API shutdown complete")
def create_app(settings: Settings, orchestrator: ServiceOrchestrator) -> FastAPI:
"""Create and configure FastAPI application."""
# Create FastAPI application
app = FastAPI(
title=settings.app_name,
version=settings.version,
description="WiFi-based human pose estimation and activity recognition API",
docs_url=settings.docs_url if not settings.is_production else None,
redoc_url=settings.redoc_url if not settings.is_production else None,
openapi_url=settings.openapi_url if not settings.is_production else None,
lifespan=lifespan
)
# Store orchestrator in app state
app.state.orchestrator = orchestrator
app.state.settings = settings
# Add middleware in reverse order (last added = first executed)
setup_middleware(app, settings)
# Add exception handlers
setup_exception_handlers(app)
# Include routers
setup_routers(app, settings)
# Add root endpoints
setup_root_endpoints(app, settings)
return app
def setup_middleware(app: FastAPI, settings: Settings):
"""Setup application middleware."""
# Rate limiting middleware
if settings.enable_rate_limiting:
app.add_middleware(RateLimitMiddleware, settings=settings)
# Authentication middleware
if settings.enable_authentication:
app.add_middleware(AuthenticationMiddleware, settings=settings)
# CORS middleware
if settings.cors_enabled:
app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_origins,
allow_credentials=settings.cors_allow_credentials,
allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS", "PATCH"],
allow_headers=["*"],
)
# Trusted host middleware for production
if settings.is_production:
app.add_middleware(
TrustedHostMiddleware,
allowed_hosts=settings.allowed_hosts
)
def setup_exception_handlers(app: FastAPI):
"""Setup global exception handlers."""
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
"""Handle HTTP exceptions."""
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": exc.status_code,
"message": exc.detail,
"type": "http_error",
"path": str(request.url.path)
}
}
)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
"""Handle request validation errors."""
return JSONResponse(
status_code=422,
content={
"error": {
"code": 422,
"message": "Validation error",
"type": "validation_error",
"path": str(request.url.path),
"details": exc.errors()
}
}
)
@app.exception_handler(Exception)
async def general_exception_handler(request: Request, exc: Exception):
"""Handle general exceptions."""
logger.error(f"Unhandled exception on {request.url.path}: {exc}", exc_info=True)
return JSONResponse(
status_code=500,
content={
"error": {
"code": 500,
"message": "Internal server error",
"type": "internal_error",
"path": str(request.url.path)
}
}
)
def setup_routers(app: FastAPI, settings: Settings):
"""Setup API routers."""
# Health check router (no prefix)
app.include_router(
health.router,
prefix="/health",
tags=["Health"]
)
# API routers with prefix
app.include_router(
pose.router,
prefix=f"{settings.api_prefix}/pose",
tags=["Pose Estimation"]
)
app.include_router(
stream.router,
prefix=f"{settings.api_prefix}/stream",
tags=["Streaming"]
)
def setup_root_endpoints(app: FastAPI, settings: Settings):
"""Setup root application endpoints."""
@app.get("/")
async def root():
"""Root endpoint with API information."""
return {
"name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"docs_url": settings.docs_url,
"api_prefix": settings.api_prefix,
"features": {
"authentication": settings.enable_authentication,
"rate_limiting": settings.enable_rate_limiting,
"websockets": settings.enable_websockets,
"real_time_processing": settings.enable_real_time_processing
}
}
@app.get(f"{settings.api_prefix}/info")
async def api_info(request: Request):
"""Get detailed API information."""
orchestrator: ServiceOrchestrator = request.app.state.orchestrator
return {
"api": {
"name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"prefix": settings.api_prefix
},
"services": await orchestrator.get_service_info(),
"features": {
"authentication": settings.enable_authentication,
"rate_limiting": settings.enable_rate_limiting,
"websockets": settings.enable_websockets,
"real_time_processing": settings.enable_real_time_processing,
"historical_data": settings.enable_historical_data
},
"limits": {
"rate_limit_requests": settings.rate_limit_requests,
"rate_limit_window": settings.rate_limit_window
}
}
@app.get(f"{settings.api_prefix}/status")
async def api_status(request: Request):
"""Get current API status."""
try:
orchestrator: ServiceOrchestrator = request.app.state.orchestrator
status = {
"api": {
"status": "healthy",
"version": settings.version,
"environment": settings.environment
},
"services": await orchestrator.get_service_status(),
"connections": await connection_manager.get_connection_stats()
}
return status
except Exception as e:
logger.error(f"Error getting API status: {e}")
return {
"api": {
"status": "error",
"error": str(e)
}
}
# Metrics endpoint (if enabled)
if settings.metrics_enabled:
@app.get(f"{settings.api_prefix}/metrics")
async def api_metrics(request: Request):
"""Get API metrics."""
try:
orchestrator: ServiceOrchestrator = request.app.state.orchestrator
metrics = {
"connections": await connection_manager.get_metrics(),
"services": await orchestrator.get_service_metrics()
}
return metrics
except Exception as e:
logger.error(f"Error getting metrics: {e}")
return {"error": str(e)}
# Development endpoints (only in development)
if settings.is_development and settings.enable_test_endpoints:
@app.get(f"{settings.api_prefix}/dev/config")
async def dev_config():
"""Get current configuration (development only).
Returns a sanitized view of settings. Secret keys,
passwords, and raw environment variables are never exposed.
"""
# Build a sanitized copy -- redact any key that looks secret
_sensitive = {"secret", "password", "token", "key", "credential", "auth"}
raw = settings.dict()
sanitized = {
k: "***REDACTED***" if any(s in k.lower() for s in _sensitive) else v
for k, v in raw.items()
}
return {
"settings": sanitized,
"environment": settings.environment,
}
@app.post(f"{settings.api_prefix}/dev/reset")
async def dev_reset(request: Request):
"""Reset services (development only)."""
try:
orchestrator: ServiceOrchestrator = request.app.state.orchestrator
await orchestrator.reset_services()
return {"message": "Services reset successfully"}
except Exception as e:
logger.error(f"Error resetting services: {e}")
return {"error": str(e)}
# Create default app instance for uvicorn
def get_app() -> FastAPI:
"""Get the default application instance."""
from src.config.settings import get_settings
from src.services.orchestrator import ServiceOrchestrator
settings = get_settings()
orchestrator = ServiceOrchestrator(settings)
return create_app(settings, orchestrator)
# Default app instance for uvicorn
app = get_app()
archive/v1/src/cli.py
+620
View File
@@ -0,0 +1,620 @@
"""
Command-line interface for WiFi-DensePose API
"""
import asyncio
import click
import sys
from typing import Optional
from src.config.settings import get_settings, load_settings_from_file
from src.logger import setup_logging, get_logger
from src.commands.start import start_command
from src.commands.stop import stop_command
from src.commands.status import status_command
# Get default settings and setup logging for CLI
settings = get_settings()
setup_logging(settings)
logger = get_logger(__name__)
def get_settings_with_config(config_file: Optional[str] = None):
"""Get settings with optional config file."""
if config_file:
return load_settings_from_file(config_file)
else:
return get_settings()
@click.group()
@click.option(
'--config',
'-c',
type=click.Path(exists=True),
help='Path to configuration file'
)
@click.option(
'--verbose',
'-v',
is_flag=True,
help='Enable verbose logging'
)
@click.option(
'--debug',
is_flag=True,
help='Enable debug mode'
)
@click.pass_context
def cli(ctx, config: Optional[str], verbose: bool, debug: bool):
"""WiFi-DensePose API Command Line Interface."""
# Ensure context object exists
ctx.ensure_object(dict)
# Store CLI options in context
ctx.obj['config_file'] = config
ctx.obj['verbose'] = verbose
ctx.obj['debug'] = debug
# Setup logging level
if debug:
import logging
logging.getLogger().setLevel(logging.DEBUG)
logger.debug("Debug mode enabled")
elif verbose:
import logging
logging.getLogger().setLevel(logging.INFO)
logger.info("Verbose mode enabled")
@cli.command()
@click.option(
'--host',
default='0.0.0.0',
help='Host to bind to (default: 0.0.0.0)'
)
@click.option(
'--port',
default=8000,
type=int,
help='Port to bind to (default: 8000)'
)
@click.option(
'--workers',
default=1,
type=int,
help='Number of worker processes (default: 1)'
)
@click.option(
'--reload',
is_flag=True,
help='Enable auto-reload for development'
)
@click.option(
'--daemon',
'-d',
is_flag=True,
help='Run as daemon (background process)'
)
@click.pass_context
def start(ctx, host: str, port: int, workers: int, reload: bool, daemon: bool):
"""Start the WiFi-DensePose API server."""
try:
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Override settings with CLI options
if ctx.obj.get('debug'):
settings.debug = True
# Run start command
asyncio.run(start_command(
settings=settings,
host=host,
port=port,
workers=workers,
reload=reload,
daemon=daemon
))
except KeyboardInterrupt:
logger.info("Received interrupt signal, shutting down...")
sys.exit(0)
except Exception as e:
logger.error(f"Failed to start server: {e}")
sys.exit(1)
@cli.command()
@click.option(
'--force',
'-f',
is_flag=True,
help='Force stop without graceful shutdown'
)
@click.option(
'--timeout',
default=30,
type=int,
help='Timeout for graceful shutdown (default: 30 seconds)'
)
@click.pass_context
def stop(ctx, force: bool, timeout: int):
"""Stop the WiFi-DensePose API server."""
try:
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Run stop command
asyncio.run(stop_command(
settings=settings,
force=force,
timeout=timeout
))
except Exception as e:
logger.error(f"Failed to stop server: {e}")
sys.exit(1)
@cli.command()
@click.option(
'--format',
type=click.Choice(['text', 'json']),
default='text',
help='Output format (default: text)'
)
@click.option(
'--detailed',
is_flag=True,
help='Show detailed status information'
)
@click.pass_context
def status(ctx, format: str, detailed: bool):
"""Show the status of the WiFi-DensePose API server."""
try:
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Run status command
asyncio.run(status_command(
settings=settings,
output_format=format,
detailed=detailed
))
except Exception as e:
logger.error(f"Failed to get status: {e}")
sys.exit(1)
@cli.group()
def db():
"""Database management commands."""
pass
@db.command()
@click.option(
'--url',
help='Database URL (overrides config)'
)
@click.pass_context
def init(ctx, url: Optional[str]):
"""Initialize the database schema."""
try:
from src.database.connection import get_database_manager
from alembic.config import Config
from alembic import command
import os
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
if url:
settings.database_url = url
# Initialize database
db_manager = get_database_manager(settings)
async def init_db():
await db_manager.initialize()
logger.info("Database initialized successfully")
asyncio.run(init_db())
# Run migrations if alembic.ini exists
alembic_ini_path = "alembic.ini"
if os.path.exists(alembic_ini_path):
try:
alembic_cfg = Config(alembic_ini_path)
# Set the database URL in the config
alembic_cfg.set_main_option("sqlalchemy.url", settings.get_database_url())
command.upgrade(alembic_cfg, "head")
logger.info("Database migrations applied successfully")
except Exception as migration_error:
logger.warning(f"Migration failed, but database is initialized: {migration_error}")
else:
logger.info("No alembic.ini found, skipping migrations")
except Exception as e:
logger.error(f"Failed to initialize database: {e}")
sys.exit(1)
@db.command()
@click.option(
'--revision',
default='head',
help='Target revision (default: head)'
)
@click.pass_context
def migrate(ctx, revision: str):
"""Run database migrations."""
try:
from alembic.config import Config
from alembic import command
# Run migrations
alembic_cfg = Config("alembic.ini")
command.upgrade(alembic_cfg, revision)
logger.info(f"Database migrated to revision: {revision}")
except Exception as e:
logger.error(f"Failed to run migrations: {e}")
sys.exit(1)
@db.command()
@click.option(
'--steps',
default=1,
type=int,
help='Number of steps to rollback (default: 1)'
)
@click.pass_context
def rollback(ctx, steps: int):
"""Rollback database migrations."""
try:
from alembic.config import Config
from alembic import command
# Rollback migrations
alembic_cfg = Config("alembic.ini")
command.downgrade(alembic_cfg, f"-{steps}")
logger.info(f"Database rolled back {steps} step(s)")
except Exception as e:
logger.error(f"Failed to rollback database: {e}")
sys.exit(1)
@cli.group()
def tasks():
"""Background task management commands."""
pass
@tasks.command()
@click.option(
'--task',
type=click.Choice(['cleanup', 'monitoring', 'backup']),
help='Specific task to run'
)
@click.pass_context
def run(ctx, task: Optional[str]):
"""Run background tasks."""
try:
from src.tasks.cleanup import get_cleanup_manager
from src.tasks.monitoring import get_monitoring_manager
from src.tasks.backup import get_backup_manager
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
async def run_tasks():
if task == 'cleanup' or task is None:
cleanup_manager = get_cleanup_manager(settings)
result = await cleanup_manager.run_all_tasks()
logger.info(f"Cleanup result: {result}")
if task == 'monitoring' or task is None:
monitoring_manager = get_monitoring_manager(settings)
result = await monitoring_manager.run_all_tasks()
logger.info(f"Monitoring result: {result}")
if task == 'backup' or task is None:
backup_manager = get_backup_manager(settings)
result = await backup_manager.run_all_tasks()
logger.info(f"Backup result: {result}")
asyncio.run(run_tasks())
except Exception as e:
logger.error(f"Failed to run tasks: {e}")
sys.exit(1)
@tasks.command()
@click.pass_context
def status(ctx):
"""Show background task status."""
try:
from src.tasks.cleanup import get_cleanup_manager
from src.tasks.monitoring import get_monitoring_manager
from src.tasks.backup import get_backup_manager
import json
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Get task managers
cleanup_manager = get_cleanup_manager(settings)
monitoring_manager = get_monitoring_manager(settings)
backup_manager = get_backup_manager(settings)
# Collect status
status_data = {
"cleanup": cleanup_manager.get_stats(),
"monitoring": monitoring_manager.get_stats(),
"backup": backup_manager.get_stats(),
}
# Print status
click.echo(json.dumps(status_data, indent=2))
except Exception as e:
logger.error(f"Failed to get task status: {e}")
sys.exit(1)
@cli.group()
def config():
"""Configuration management commands."""
pass
@config.command()
@click.pass_context
def show(ctx):
"""Show current configuration."""
try:
import json
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Convert settings to dict (excluding sensitive data)
config_dict = {
"app_name": settings.app_name,
"version": settings.version,
"environment": settings.environment,
"debug": settings.debug,
"host": settings.host,
"port": settings.port,
"api_prefix": settings.api_prefix,
"docs_url": settings.docs_url,
"redoc_url": settings.redoc_url,
"log_level": settings.log_level,
"log_file": settings.log_file,
"data_storage_path": settings.data_storage_path,
"model_storage_path": settings.model_storage_path,
"temp_storage_path": settings.temp_storage_path,
"wifi_interface": settings.wifi_interface,
"csi_buffer_size": settings.csi_buffer_size,
"pose_confidence_threshold": settings.pose_confidence_threshold,
"stream_fps": settings.stream_fps,
"websocket_ping_interval": settings.websocket_ping_interval,
"features": {
"authentication": settings.enable_authentication,
"rate_limiting": settings.enable_rate_limiting,
"websockets": settings.enable_websockets,
"historical_data": settings.enable_historical_data,
"real_time_processing": settings.enable_real_time_processing,
"cors": settings.cors_enabled,
}
}
click.echo(json.dumps(config_dict, indent=2))
except Exception as e:
logger.error(f"Failed to show configuration: {e}")
sys.exit(1)
@config.command()
@click.pass_context
def validate(ctx):
"""Validate configuration."""
try:
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
# Validate database connection
from src.database.connection import get_database_manager
async def validate_config():
db_manager = get_database_manager(settings)
try:
await db_manager.test_connection()
click.echo("✓ Database connection: OK")
except Exception as e:
click.echo(f"✗ Database connection: FAILED - {e}")
return False
# Validate Redis connection (if configured)
redis_url = settings.get_redis_url()
if redis_url:
try:
import redis.asyncio as redis
redis_client = redis.from_url(redis_url)
await redis_client.ping()
click.echo("✓ Redis connection: OK")
await redis_client.close()
except Exception as e:
click.echo(f"✗ Redis connection: FAILED - {e}")
return False
else:
click.echo("- Redis connection: NOT CONFIGURED")
# Validate directories
from pathlib import Path
directories = [
("Data storage", settings.data_storage_path),
("Model storage", settings.model_storage_path),
("Temp storage", settings.temp_storage_path),
]
for name, directory in directories:
path = Path(directory)
if path.exists() and path.is_dir():
click.echo(f"✓ {name}: OK")
else:
try:
path.mkdir(parents=True, exist_ok=True)
click.echo(f"✓ {name}: CREATED - {directory}")
except Exception as e:
click.echo(f"✗ {name}: FAILED TO CREATE - {directory} ({e})")
return False
click.echo("\n✓ Configuration validation passed")
return True
result = asyncio.run(validate_config())
if not result:
sys.exit(1)
except Exception as e:
logger.error(f"Failed to validate configuration: {e}")
sys.exit(1)
@config.command()
@click.option(
'--format',
type=click.Choice(['text', 'json']),
default='text',
help='Output format (default: text)'
)
@click.pass_context
def failsafe(ctx, format: str):
"""Show failsafe status and configuration."""
try:
import json
from src.database.connection import get_database_manager
# Get settings
settings = get_settings_with_config(ctx.obj.get('config_file'))
async def check_failsafe_status():
db_manager = get_database_manager(settings)
# Initialize database to check current state
try:
await db_manager.initialize()
except Exception as e:
logger.warning(f"Database initialization failed: {e}")
# Collect failsafe status
failsafe_status = {
"database": {
"failsafe_enabled": settings.enable_database_failsafe,
"using_sqlite_fallback": db_manager.is_using_sqlite_fallback(),
"sqlite_fallback_path": settings.sqlite_fallback_path,
"primary_database_url": settings.get_database_url() if not db_manager.is_using_sqlite_fallback() else None,
},
"redis": {
"failsafe_enabled": settings.enable_redis_failsafe,
"redis_enabled": settings.redis_enabled,
"redis_required": settings.redis_required,
"redis_available": db_manager.is_redis_available(),
"redis_url": settings.get_redis_url() if settings.redis_enabled else None,
},
"overall_status": "healthy"
}
# Determine overall status
if failsafe_status["database"]["using_sqlite_fallback"] or not failsafe_status["redis"]["redis_available"]:
failsafe_status["overall_status"] = "degraded"
# Output results
if format == 'json':
click.echo(json.dumps(failsafe_status, indent=2))
else:
click.echo("=== Failsafe Status ===\n")
# Database status
click.echo("Database:")
if failsafe_status["database"]["using_sqlite_fallback"]:
click.echo(" ⚠️ Using SQLite fallback database")
click.echo(f" Path: {failsafe_status['database']['sqlite_fallback_path']}")
else:
click.echo(" ✓ Using primary database (PostgreSQL)")
click.echo(f" Failsafe enabled: {'Yes' if failsafe_status['database']['failsafe_enabled'] else 'No'}")
# Redis status
click.echo("\nRedis:")
if not failsafe_status["redis"]["redis_enabled"]:
click.echo(" - Redis disabled")
elif not failsafe_status["redis"]["redis_available"]:
click.echo(" ⚠️ Redis unavailable (failsafe active)")
else:
click.echo(" ✓ Redis available")
click.echo(f" Failsafe enabled: {'Yes' if failsafe_status['redis']['failsafe_enabled'] else 'No'}")
click.echo(f" Required: {'Yes' if failsafe_status['redis']['redis_required'] else 'No'}")
# Overall status
status_icon = "✓" if failsafe_status["overall_status"] == "healthy" else "⚠️"
click.echo(f"\nOverall Status: {status_icon} {failsafe_status['overall_status'].upper()}")
if failsafe_status["overall_status"] == "degraded":
click.echo("\nNote: System is running in degraded mode using failsafe configurations.")
asyncio.run(check_failsafe_status())
except Exception as e:
logger.error(f"Failed to check failsafe status: {e}")
sys.exit(1)
@cli.command()
def version():
"""Show version information."""
try:
from src.config.settings import get_settings
settings = get_settings()
click.echo(f"WiFi-DensePose API v{settings.version}")
click.echo(f"Environment: {settings.environment}")
click.echo(f"Python: {sys.version}")
except Exception as e:
logger.error(f"Failed to get version: {e}")
sys.exit(1)
def create_cli(orchestrator=None):
"""Create CLI interface for the application."""
return cli
if __name__ == '__main__':
cli()
archive/v1/src/commands/start.py
+359
View File
@@ -0,0 +1,359 @@
"""
Start command implementation for WiFi-DensePose API
"""
import asyncio
import os
import signal
import sys
import uvicorn
from pathlib import Path
from typing import Optional
from src.config.settings import Settings
from src.logger import get_logger
logger = get_logger(__name__)
async def start_command(
settings: Settings,
host: str = "0.0.0.0",
port: int = 8000,
workers: int = 1,
reload: bool = False,
daemon: bool = False
) -> None:
"""Start the WiFi-DensePose API server."""
logger.info(f"Starting WiFi-DensePose API server...")
logger.info(f"Environment: {settings.environment}")
logger.info(f"Debug mode: {settings.debug}")
logger.info(f"Host: {host}")
logger.info(f"Port: {port}")
logger.info(f"Workers: {workers}")
# Validate settings
await _validate_startup_requirements(settings)
# Setup signal handlers
_setup_signal_handlers()
# Create PID file if running as daemon
pid_file = None
if daemon:
pid_file = _create_pid_file(settings)
try:
# Initialize database
await _initialize_database(settings)
# Start background tasks
background_tasks = await _start_background_tasks(settings)
# Configure uvicorn
uvicorn_config = {
"app": "src.app:app",
"host": host,
"port": port,
"reload": reload,
"workers": workers if not reload else 1, # Reload doesn't work with multiple workers
"log_level": "debug" if settings.debug else "info",
"access_log": True,
"use_colors": not daemon,
}
if daemon:
# Run as daemon
await _run_as_daemon(uvicorn_config, pid_file)
else:
# Run in foreground
await _run_server(uvicorn_config)
except KeyboardInterrupt:
logger.info("Received interrupt signal, shutting down...")
except Exception as e:
logger.error(f"Server startup failed: {e}")
raise
finally:
# Cleanup
if pid_file and pid_file.exists():
pid_file.unlink()
# Stop background tasks
if 'background_tasks' in locals():
await _stop_background_tasks(background_tasks)
async def _validate_startup_requirements(settings: Settings) -> None:
"""Validate that all startup requirements are met."""
logger.info("Validating startup requirements...")
# Check database connection
try:
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
await db_manager.test_connection()
logger.info("✓ Database connection validated")
except Exception as e:
logger.error(f"✗ Database connection failed: {e}")
raise
# Check Redis connection (if enabled)
if settings.redis_enabled:
try:
redis_stats = await db_manager.get_connection_stats()
if "redis" in redis_stats and not redis_stats["redis"].get("error"):
logger.info("✓ Redis connection validated")
else:
logger.warning("⚠ Redis connection failed, continuing without Redis")
except Exception as e:
logger.warning(f"⚠ Redis connection failed: {e}, continuing without Redis")
# Check required directories
directories = [
("Log directory", settings.log_directory),
("Backup directory", settings.backup_directory),
]
for name, directory in directories:
path = Path(directory)
path.mkdir(parents=True, exist_ok=True)
logger.info(f"✓ {name} ready: {directory}")
logger.info("All startup requirements validated")
async def _initialize_database(settings: Settings) -> None:
"""Initialize database connection and run migrations if needed."""
logger.info("Initializing database...")
try:
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
await db_manager.initialize()
logger.info("Database initialized successfully")
except Exception as e:
logger.error(f"Database initialization failed: {e}")
raise
async def _start_background_tasks(settings: Settings) -> dict:
"""Start background tasks."""
logger.info("Starting background tasks...")
tasks = {}
try:
# Start cleanup task
if settings.cleanup_interval_seconds > 0:
from src.tasks.cleanup import run_periodic_cleanup
cleanup_task = asyncio.create_task(run_periodic_cleanup(settings))
tasks['cleanup'] = cleanup_task
logger.info("✓ Cleanup task started")
# Start monitoring task
if settings.monitoring_interval_seconds > 0:
from src.tasks.monitoring import run_periodic_monitoring
monitoring_task = asyncio.create_task(run_periodic_monitoring(settings))
tasks['monitoring'] = monitoring_task
logger.info("✓ Monitoring task started")
# Start backup task
if settings.backup_interval_seconds > 0:
from src.tasks.backup import run_periodic_backup
backup_task = asyncio.create_task(run_periodic_backup(settings))
tasks['backup'] = backup_task
logger.info("✓ Backup task started")
logger.info(f"Started {len(tasks)} background tasks")
return tasks
except Exception as e:
logger.error(f"Failed to start background tasks: {e}")
# Cancel any started tasks
for task in tasks.values():
task.cancel()
raise
async def _stop_background_tasks(tasks: dict) -> None:
"""Stop background tasks gracefully."""
logger.info("Stopping background tasks...")
# Cancel all tasks
for name, task in tasks.items():
if not task.done():
logger.info(f"Stopping {name} task...")
task.cancel()
# Wait for tasks to complete
if tasks:
await asyncio.gather(*tasks.values(), return_exceptions=True)
logger.info("Background tasks stopped")
def _setup_signal_handlers() -> None:
"""Setup signal handlers for graceful shutdown."""
def signal_handler(signum, frame):
logger.info(f"Received signal {signum}, initiating graceful shutdown...")
# The actual shutdown will be handled by the main loop
sys.exit(0)
# Setup signal handlers
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
if hasattr(signal, 'SIGHUP'):
signal.signal(signal.SIGHUP, signal_handler)
def _create_pid_file(settings: Settings) -> Path:
"""Create PID file for daemon mode."""
pid_file = Path(settings.log_directory) / "wifi-densepose-api.pid"
# Check if PID file already exists
if pid_file.exists():
try:
with open(pid_file, 'r') as f:
old_pid = int(f.read().strip())
# Check if process is still running
try:
os.kill(old_pid, 0) # Signal 0 just checks if process exists
logger.error(f"Server already running with PID {old_pid}")
sys.exit(1)
except OSError:
# Process doesn't exist, remove stale PID file
pid_file.unlink()
logger.info("Removed stale PID file")
except (ValueError, IOError):
# Invalid PID file, remove it
pid_file.unlink()
logger.info("Removed invalid PID file")
# Write current PID
with open(pid_file, 'w') as f:
f.write(str(os.getpid()))
logger.info(f"Created PID file: {pid_file}")
return pid_file
async def _run_server(config: dict) -> None:
"""Run the server in foreground mode."""
logger.info("Starting server in foreground mode...")
# Create uvicorn server
server = uvicorn.Server(uvicorn.Config(**config))
# Run server
await server.serve()
async def _run_as_daemon(config: dict, pid_file: Path) -> None:
"""Run the server as a daemon."""
logger.info("Starting server in daemon mode...")
# Fork process
try:
pid = os.fork()
if pid > 0:
# Parent process
logger.info(f"Server started as daemon with PID {pid}")
sys.exit(0)
except OSError as e:
logger.error(f"Fork failed: {e}")
sys.exit(1)
# Child process continues
# Decouple from parent environment
os.chdir("/")
os.setsid()
os.umask(0)
# Second fork
try:
pid = os.fork()
if pid > 0:
# Exit second parent
sys.exit(0)
except OSError as e:
logger.error(f"Second fork failed: {e}")
sys.exit(1)
# Update PID file with daemon PID
with open(pid_file, 'w') as f:
f.write(str(os.getpid()))
# Redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
# Redirect stdin, stdout, stderr to /dev/null
with open('/dev/null', 'r') as f:
os.dup2(f.fileno(), sys.stdin.fileno())
with open('/dev/null', 'w') as f:
os.dup2(f.fileno(), sys.stdout.fileno())
os.dup2(f.fileno(), sys.stderr.fileno())
# Create uvicorn server
server = uvicorn.Server(uvicorn.Config(**config))
# Run server
await server.serve()
def get_server_status(settings: Settings) -> dict:
"""Get current server status."""
pid_file = Path(settings.log_directory) / "wifi-densepose-api.pid"
status = {
"running": False,
"pid": None,
"pid_file": str(pid_file),
"pid_file_exists": pid_file.exists(),
}
if pid_file.exists():
try:
with open(pid_file, 'r') as f:
pid = int(f.read().strip())
status["pid"] = pid
# Check if process is running
try:
os.kill(pid, 0) # Signal 0 just checks if process exists
status["running"] = True
except OSError:
# Process doesn't exist
status["running"] = False
except (ValueError, IOError):
# Invalid PID file
status["running"] = False
return status
archive/v1/src/commands/status.py
+511
View File
@@ -0,0 +1,511 @@
"""
Status command implementation for WiFi-DensePose API
"""
import asyncio
import json
import psutil
import time
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, Any, Optional
from src.config.settings import Settings
from src.logger import get_logger
logger = get_logger(__name__)
async def status_command(
settings: Settings,
output_format: str = "text",
detailed: bool = False
) -> None:
"""Show the status of the WiFi-DensePose API server."""
logger.debug("Gathering server status information...")
try:
# Collect status information
status_data = await _collect_status_data(settings, detailed)
# Output status
if output_format == "json":
print(json.dumps(status_data, indent=2, default=str))
else:
_print_text_status(status_data, detailed)
except Exception as e:
logger.error(f"Failed to get status: {e}")
raise
async def _collect_status_data(settings: Settings, detailed: bool) -> Dict[str, Any]:
"""Collect comprehensive status data."""
status_data = {
"timestamp": datetime.utcnow().isoformat(),
"server": await _get_server_status(settings),
"system": _get_system_status(),
"configuration": _get_configuration_status(settings),
}
if detailed:
status_data.update({
"database": await _get_database_status(settings),
"background_tasks": await _get_background_tasks_status(settings),
"resources": _get_resource_usage(),
"health": await _get_health_status(settings),
})
return status_data
async def _get_server_status(settings: Settings) -> Dict[str, Any]:
"""Get server process status."""
from src.commands.stop import get_server_status
status = get_server_status(settings)
server_info = {
"running": status["running"],
"pid": status["pid"],
"pid_file": status["pid_file"],
"pid_file_exists": status["pid_file_exists"],
}
if status["running"] and status["pid"]:
try:
# Get process information
process = psutil.Process(status["pid"])
server_info.update({
"start_time": datetime.fromtimestamp(process.create_time()).isoformat(),
"uptime_seconds": time.time() - process.create_time(),
"memory_usage_mb": process.memory_info().rss / (1024 * 1024),
"cpu_percent": process.cpu_percent(),
"status": process.status(),
"num_threads": process.num_threads(),
"connections": len(process.connections()) if hasattr(process, 'connections') else None,
})
except (psutil.NoSuchProcess, psutil.AccessDenied) as e:
server_info["error"] = f"Cannot access process info: {e}"
return server_info
def _get_system_status() -> Dict[str, Any]:
"""Get system status information."""
uname_info = psutil.os.uname()
return {
"hostname": uname_info.nodename,
"platform": uname_info.sysname,
"architecture": uname_info.machine,
"python_version": f"{psutil.sys.version_info.major}.{psutil.sys.version_info.minor}.{psutil.sys.version_info.micro}",
"boot_time": datetime.fromtimestamp(psutil.boot_time()).isoformat(),
"uptime_seconds": time.time() - psutil.boot_time(),
}
def _get_configuration_status(settings: Settings) -> Dict[str, Any]:
"""Get configuration status."""
return {
"environment": settings.environment,
"debug": settings.debug,
"version": settings.version,
"host": settings.host,
"port": settings.port,
"database_configured": bool(settings.database_url or (settings.db_host and settings.db_name)),
"redis_enabled": settings.redis_enabled,
"monitoring_enabled": settings.monitoring_interval_seconds > 0,
"cleanup_enabled": settings.cleanup_interval_seconds > 0,
"backup_enabled": settings.backup_interval_seconds > 0,
}
async def _get_database_status(settings: Settings) -> Dict[str, Any]:
"""Get database status."""
db_status = {
"connected": False,
"connection_pool": None,
"tables": {},
"error": None,
}
try:
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
# Test connection
await db_manager.test_connection()
db_status["connected"] = True
# Get connection stats
connection_stats = await db_manager.get_connection_stats()
db_status["connection_pool"] = connection_stats
# Get table counts
async with db_manager.get_async_session() as session:
import sqlalchemy as sa
from sqlalchemy import text, func, select
from src.database.models import Device, Session, CSIData, PoseDetection, SystemMetric, AuditLog
tables = {
"devices": Device,
"sessions": Session,
"csi_data": CSIData,
"pose_detections": PoseDetection,
"system_metrics": SystemMetric,
"audit_logs": AuditLog,
}
# Whitelist of allowed table names to prevent SQL injection
allowed_table_names = set(tables.keys())
for table_name, model in tables.items():
try:
# Validate table_name against whitelist to prevent SQL injection
if table_name not in allowed_table_names:
db_status["tables"][table_name] = {"error": "Invalid table name"}
continue
# Use SQLAlchemy ORM model for safe query instead of raw SQL
result = await session.execute(
select(func.count()).select_from(model)
)
count = result.scalar()
db_status["tables"][table_name] = {"count": count}
except Exception as e:
db_status["tables"][table_name] = {"error": str(e)}
except Exception as e:
db_status["error"] = str(e)
return db_status
async def _get_background_tasks_status(settings: Settings) -> Dict[str, Any]:
"""Get background tasks status."""
tasks_status = {}
try:
# Cleanup tasks
from src.tasks.cleanup import get_cleanup_manager
cleanup_manager = get_cleanup_manager(settings)
tasks_status["cleanup"] = cleanup_manager.get_stats()
except Exception as e:
tasks_status["cleanup"] = {"error": str(e)}
try:
# Monitoring tasks
from src.tasks.monitoring import get_monitoring_manager
monitoring_manager = get_monitoring_manager(settings)
tasks_status["monitoring"] = monitoring_manager.get_stats()
except Exception as e:
tasks_status["monitoring"] = {"error": str(e)}
try:
# Backup tasks
from src.tasks.backup import get_backup_manager
backup_manager = get_backup_manager(settings)
tasks_status["backup"] = backup_manager.get_stats()
except Exception as e:
tasks_status["backup"] = {"error": str(e)}
return tasks_status
def _get_resource_usage() -> Dict[str, Any]:
"""Get system resource usage."""
# CPU usage
cpu_percent = psutil.cpu_percent(interval=1)
cpu_count = psutil.cpu_count()
# Memory usage
memory = psutil.virtual_memory()
swap = psutil.swap_memory()
# Disk usage
disk = psutil.disk_usage('/')
# Network I/O
network = psutil.net_io_counters()
return {
"cpu": {
"usage_percent": cpu_percent,
"count": cpu_count,
},
"memory": {
"total_mb": memory.total / (1024 * 1024),
"used_mb": memory.used / (1024 * 1024),
"available_mb": memory.available / (1024 * 1024),
"usage_percent": memory.percent,
},
"swap": {
"total_mb": swap.total / (1024 * 1024),
"used_mb": swap.used / (1024 * 1024),
"usage_percent": swap.percent,
},
"disk": {
"total_gb": disk.total / (1024 * 1024 * 1024),
"used_gb": disk.used / (1024 * 1024 * 1024),
"free_gb": disk.free / (1024 * 1024 * 1024),
"usage_percent": (disk.used / disk.total) * 100,
},
"network": {
"bytes_sent": network.bytes_sent,
"bytes_recv": network.bytes_recv,
"packets_sent": network.packets_sent,
"packets_recv": network.packets_recv,
} if network else None,
}
async def _get_health_status(settings: Settings) -> Dict[str, Any]:
"""Get overall health status."""
health = {
"status": "healthy",
"checks": {},
"issues": [],
}
# Check database health
try:
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
await db_manager.test_connection()
health["checks"]["database"] = "healthy"
except Exception as e:
health["checks"]["database"] = "unhealthy"
health["issues"].append(f"Database connection failed: {e}")
health["status"] = "unhealthy"
# Check disk space
disk = psutil.disk_usage('/')
disk_usage_percent = (disk.used / disk.total) * 100
if disk_usage_percent > 90:
health["checks"]["disk_space"] = "critical"
health["issues"].append(f"Disk usage critical: {disk_usage_percent:.1f}%")
health["status"] = "critical"
elif disk_usage_percent > 80:
health["checks"]["disk_space"] = "warning"
health["issues"].append(f"Disk usage high: {disk_usage_percent:.1f}%")
if health["status"] == "healthy":
health["status"] = "warning"
else:
health["checks"]["disk_space"] = "healthy"
# Check memory usage
memory = psutil.virtual_memory()
if memory.percent > 90:
health["checks"]["memory"] = "critical"
health["issues"].append(f"Memory usage critical: {memory.percent:.1f}%")
health["status"] = "critical"
elif memory.percent > 80:
health["checks"]["memory"] = "warning"
health["issues"].append(f"Memory usage high: {memory.percent:.1f}%")
if health["status"] == "healthy":
health["status"] = "warning"
else:
health["checks"]["memory"] = "healthy"
# Check log directory
log_dir = Path(settings.log_directory)
if log_dir.exists() and log_dir.is_dir():
health["checks"]["log_directory"] = "healthy"
else:
health["checks"]["log_directory"] = "unhealthy"
health["issues"].append(f"Log directory not accessible: {log_dir}")
health["status"] = "unhealthy"
# Check backup directory
backup_dir = Path(settings.backup_directory)
if backup_dir.exists() and backup_dir.is_dir():
health["checks"]["backup_directory"] = "healthy"
else:
health["checks"]["backup_directory"] = "unhealthy"
health["issues"].append(f"Backup directory not accessible: {backup_dir}")
health["status"] = "unhealthy"
return health
def _print_text_status(status_data: Dict[str, Any], detailed: bool) -> None:
"""Print status in human-readable text format."""
print("=" * 60)
print("WiFi-DensePose API Server Status")
print("=" * 60)
print(f"Timestamp: {status_data['timestamp']}")
print()
# Server status
server = status_data["server"]
print("🖥️ Server Status:")
if server["running"]:
print(f" ✅ Running (PID: {server['pid']})")
if "start_time" in server:
uptime = timedelta(seconds=int(server["uptime_seconds"]))
print(f" ⏱️ Uptime: {uptime}")
print(f" 💾 Memory: {server['memory_usage_mb']:.1f} MB")
print(f" 🔧 CPU: {server['cpu_percent']:.1f}%")
print(f" 🧵 Threads: {server['num_threads']}")
else:
print(" ❌ Not running")
if server["pid_file_exists"]:
print(" ⚠️ Stale PID file exists")
print()
# System status
system = status_data["system"]
print("🖥️ System:")
print(f" Hostname: {system['hostname']}")
print(f" Platform: {system['platform']} ({system['architecture']})")
print(f" Python: {system['python_version']}")
uptime = timedelta(seconds=int(system["uptime_seconds"]))
print(f" Uptime: {uptime}")
print()
# Configuration
config = status_data["configuration"]
print("⚙️ Configuration:")
print(f" Environment: {config['environment']}")
print(f" Debug: {config['debug']}")
print(f" API Version: {config['version']}")
print(f" Listen: {config['host']}:{config['port']}")
print(f" Database: {'✅' if config['database_configured'] else '❌'}")
print(f" Redis: {'✅' if config['redis_enabled'] else '❌'}")
print(f" Monitoring: {'✅' if config['monitoring_enabled'] else '❌'}")
print(f" Cleanup: {'✅' if config['cleanup_enabled'] else '❌'}")
print(f" Backup: {'✅' if config['backup_enabled'] else '❌'}")
print()
if detailed:
# Database status
if "database" in status_data:
db = status_data["database"]
print("🗄️ Database:")
if db["connected"]:
print(" ✅ Connected")
if "tables" in db:
print(" 📊 Table counts:")
for table, info in db["tables"].items():
if "count" in info:
print(f" {table}: {info['count']:,}")
else:
print(f" {table}: Error - {info.get('error', 'Unknown')}")
else:
print(f" ❌ Not connected: {db.get('error', 'Unknown error')}")
print()
# Background tasks
if "background_tasks" in status_data:
tasks = status_data["background_tasks"]
print("🔄 Background Tasks:")
for task_name, task_info in tasks.items():
if "error" in task_info:
print(f" ❌ {task_name}: {task_info['error']}")
else:
manager_info = task_info.get("manager", {})
print(f" 📋 {task_name}:")
print(f" Running: {manager_info.get('running', 'Unknown')}")
print(f" Last run: {manager_info.get('last_run', 'Never')}")
print(f" Run count: {manager_info.get('run_count', 0)}")
print()
# Resource usage
if "resources" in status_data:
resources = status_data["resources"]
print("📊 Resource Usage:")
cpu = resources["cpu"]
print(f" 🔧 CPU: {cpu['usage_percent']:.1f}% ({cpu['count']} cores)")
memory = resources["memory"]
print(f" 💾 Memory: {memory['usage_percent']:.1f}% "
f"({memory['used_mb']:.0f}/{memory['total_mb']:.0f} MB)")
disk = resources["disk"]
print(f" 💿 Disk: {disk['usage_percent']:.1f}% "
f"({disk['used_gb']:.1f}/{disk['total_gb']:.1f} GB)")
print()
# Health status
if "health" in status_data:
health = status_data["health"]
print("🏥 Health Status:")
status_emoji = {
"healthy": "✅",
"warning": "⚠️",
"critical": "❌",
"unhealthy": "❌"
}
print(f" Overall: {status_emoji.get(health['status'], '❓')} {health['status'].upper()}")
if health["issues"]:
print(" Issues:")
for issue in health["issues"]:
print(f" • {issue}")
print(" Checks:")
for check, status in health["checks"].items():
emoji = status_emoji.get(status, "❓")
print(f" {emoji} {check}: {status}")
print()
print("=" * 60)
def get_quick_status(settings: Settings) -> str:
"""Get a quick one-line status."""
from src.commands.stop import get_server_status
status = get_server_status(settings)
if status["running"]:
return f"✅ Running (PID: {status['pid']})"
elif status["pid_file_exists"]:
return "⚠️ Not running (stale PID file)"
else:
return "❌ Not running"
async def check_health(settings: Settings) -> bool:
"""Quick health check - returns True if healthy."""
try:
status_data = await _collect_status_data(settings, detailed=True)
# Check if server is running
if not status_data["server"]["running"]:
return False
# Check health status
if "health" in status_data:
health_status = status_data["health"]["status"]
return health_status in ["healthy", "warning"]
return True
except Exception:
return False
archive/v1/src/commands/stop.py
+294
View File
@@ -0,0 +1,294 @@
"""
Stop command implementation for WiFi-DensePose API
"""
import asyncio
import os
import signal
import time
from pathlib import Path
from typing import Optional
from src.config.settings import Settings
from src.logger import get_logger
logger = get_logger(__name__)
async def stop_command(
settings: Settings,
force: bool = False,
timeout: int = 30
) -> None:
"""Stop the WiFi-DensePose API server."""
logger.info("Stopping WiFi-DensePose API server...")
# Get server status
status = get_server_status(settings)
if not status["running"]:
if status["pid_file_exists"]:
logger.info("Server is not running, but PID file exists. Cleaning up...")
_cleanup_pid_file(settings)
else:
logger.info("Server is not running")
return
pid = status["pid"]
logger.info(f"Found running server with PID {pid}")
try:
if force:
await _force_stop_server(pid, settings)
else:
await _graceful_stop_server(pid, timeout, settings)
except Exception as e:
logger.error(f"Failed to stop server: {e}")
raise
async def _graceful_stop_server(pid: int, timeout: int, settings: Settings) -> None:
"""Stop server gracefully with timeout."""
logger.info(f"Attempting graceful shutdown (timeout: {timeout}s)...")
try:
# Send SIGTERM for graceful shutdown
os.kill(pid, signal.SIGTERM)
logger.info("Sent SIGTERM signal")
# Wait for process to terminate
start_time = time.time()
while time.time() - start_time < timeout:
try:
# Check if process is still running
os.kill(pid, 0)
await asyncio.sleep(1)
except OSError:
# Process has terminated
logger.info("Server stopped gracefully")
_cleanup_pid_file(settings)
return
# Timeout reached, force kill
logger.warning(f"Graceful shutdown timeout ({timeout}s) reached, forcing stop...")
await _force_stop_server(pid, settings)
except OSError as e:
if e.errno == 3: # No such process
logger.info("Process already terminated")
_cleanup_pid_file(settings)
else:
logger.error(f"Failed to send signal to process {pid}: {e}")
raise
async def _force_stop_server(pid: int, settings: Settings) -> None:
"""Force stop server immediately."""
logger.info("Force stopping server...")
try:
# Send SIGKILL for immediate termination
os.kill(pid, signal.SIGKILL)
logger.info("Sent SIGKILL signal")
# Wait a moment for process to die
await asyncio.sleep(2)
# Verify process is dead
try:
os.kill(pid, 0)
logger.error(f"Process {pid} still running after SIGKILL")
except OSError:
logger.info("Server force stopped")
except OSError as e:
if e.errno == 3: # No such process
logger.info("Process already terminated")
else:
logger.error(f"Failed to force kill process {pid}: {e}")
raise
finally:
_cleanup_pid_file(settings)
def _cleanup_pid_file(settings: Settings) -> None:
"""Clean up PID file."""
pid_file = Path(settings.log_directory) / "wifi-densepose-api.pid"
if pid_file.exists():
try:
pid_file.unlink()
logger.info("Cleaned up PID file")
except Exception as e:
logger.warning(f"Failed to remove PID file: {e}")
def get_server_status(settings: Settings) -> dict:
"""Get current server status."""
pid_file = Path(settings.log_directory) / "wifi-densepose-api.pid"
status = {
"running": False,
"pid": None,
"pid_file": str(pid_file),
"pid_file_exists": pid_file.exists(),
}
if pid_file.exists():
try:
with open(pid_file, 'r') as f:
pid = int(f.read().strip())
status["pid"] = pid
# Check if process is running
try:
os.kill(pid, 0) # Signal 0 just checks if process exists
status["running"] = True
except OSError:
# Process doesn't exist
status["running"] = False
except (ValueError, IOError):
# Invalid PID file
status["running"] = False
return status
async def stop_all_background_tasks(settings: Settings) -> None:
"""Stop all background tasks if they're running."""
logger.info("Stopping background tasks...")
try:
# This would typically involve connecting to a task queue or
# sending signals to background processes
# For now, we'll just log the action
logger.info("Background tasks stop signal sent")
except Exception as e:
logger.error(f"Failed to stop background tasks: {e}")
async def cleanup_resources(settings: Settings) -> None:
"""Clean up system resources."""
logger.info("Cleaning up resources...")
try:
# Close database connections
from src.database.connection import get_database_manager
db_manager = get_database_manager(settings)
await db_manager.close_all_connections()
logger.info("Database connections closed")
except Exception as e:
logger.warning(f"Failed to close database connections: {e}")
try:
# Clean up temporary files
temp_files = [
Path(settings.log_directory) / "temp",
Path(settings.backup_directory) / "temp",
]
for temp_path in temp_files:
if temp_path.exists() and temp_path.is_dir():
import shutil
shutil.rmtree(temp_path)
logger.info(f"Cleaned up temporary directory: {temp_path}")
except Exception as e:
logger.warning(f"Failed to clean up temporary files: {e}")
logger.info("Resource cleanup completed")
def is_server_running(settings: Settings) -> bool:
"""Check if server is currently running."""
status = get_server_status(settings)
return status["running"]
def get_server_pid(settings: Settings) -> Optional[int]:
"""Get server PID if running."""
status = get_server_status(settings)
return status["pid"] if status["running"] else None
async def wait_for_server_stop(settings: Settings, timeout: int = 30) -> bool:
"""Wait for server to stop with timeout."""
start_time = time.time()
while time.time() - start_time < timeout:
if not is_server_running(settings):
return True
await asyncio.sleep(1)
return False
def send_reload_signal(settings: Settings) -> bool:
"""Send reload signal to running server."""
status = get_server_status(settings)
if not status["running"]:
logger.error("Server is not running")
return False
try:
# Send SIGHUP for reload
os.kill(status["pid"], signal.SIGHUP)
logger.info("Sent reload signal to server")
return True
except OSError as e:
logger.error(f"Failed to send reload signal: {e}")
return False
async def restart_server(settings: Settings, timeout: int = 30) -> None:
"""Restart the server (stop then start)."""
logger.info("Restarting server...")
# Stop server if running
if is_server_running(settings):
await stop_command(settings, timeout=timeout)
# Wait for server to stop
if not await wait_for_server_stop(settings, timeout):
logger.error("Server did not stop within timeout, forcing restart")
await stop_command(settings, force=True)
# Start server
from src.commands.start import start_command
await start_command(settings)
def get_stop_status_summary(settings: Settings) -> dict:
"""Get a summary of stop operation status."""
status = get_server_status(settings)
return {
"server_running": status["running"],
"pid": status["pid"],
"pid_file_exists": status["pid_file_exists"],
"can_stop": status["running"],
"cleanup_needed": status["pid_file_exists"] and not status["running"],
}
archive/v1/src/config.py
+310
View File
@@ -0,0 +1,310 @@
"""
Centralized configuration management for WiFi-DensePose API
"""
import os
import logging
from pathlib import Path
from typing import Dict, Any, Optional, List
from functools import lru_cache
from src.config.settings import Settings, get_settings
from src.config.domains import DomainConfig, get_domain_config
logger = logging.getLogger(__name__)
class ConfigManager:
"""Centralized configuration manager."""
def __init__(self):
self._settings: Optional[Settings] = None
self._domain_config: Optional[DomainConfig] = None
self._environment_overrides: Dict[str, Any] = {}
@property
def settings(self) -> Settings:
"""Get application settings."""
if self._settings is None:
self._settings = get_settings()
return self._settings
@property
def domain_config(self) -> DomainConfig:
"""Get domain configuration."""
if self._domain_config is None:
self._domain_config = get_domain_config()
return self._domain_config
def reload_settings(self) -> Settings:
"""Reload settings from environment."""
self._settings = None
return self.settings
def reload_domain_config(self) -> DomainConfig:
"""Reload domain configuration."""
self._domain_config = None
return self.domain_config
def set_environment_override(self, key: str, value: Any):
"""Set environment variable override."""
self._environment_overrides[key] = value
os.environ[key] = str(value)
def get_environment_override(self, key: str, default: Any = None) -> Any:
"""Get environment variable override."""
return self._environment_overrides.get(key, os.environ.get(key, default))
def clear_environment_overrides(self):
"""Clear all environment overrides."""
for key in self._environment_overrides:
if key in os.environ:
del os.environ[key]
self._environment_overrides.clear()
def get_database_config(self) -> Dict[str, Any]:
"""Get database configuration."""
settings = self.settings
config = {
"url": settings.get_database_url(),
"pool_size": settings.database_pool_size,
"max_overflow": settings.database_max_overflow,
"echo": settings.is_development and settings.debug,
"pool_pre_ping": True,
"pool_recycle": 3600, # 1 hour
}
return config
def get_redis_config(self) -> Optional[Dict[str, Any]]:
"""Get Redis configuration."""
settings = self.settings
redis_url = settings.get_redis_url()
if not redis_url:
return None
config = {
"url": redis_url,
"password": settings.redis_password,
"db": settings.redis_db,
"decode_responses": True,
"socket_connect_timeout": 5,
"socket_timeout": 5,
"retry_on_timeout": True,
"health_check_interval": 30,
}
return config
def get_logging_config(self) -> Dict[str, Any]:
"""Get logging configuration."""
return self.settings.get_logging_config()
def get_cors_config(self) -> Dict[str, Any]:
"""Get CORS configuration."""
return self.settings.get_cors_config()
def get_security_config(self) -> Dict[str, Any]:
"""Get security configuration."""
settings = self.settings
config = {
"secret_key": settings.secret_key,
"jwt_algorithm": settings.jwt_algorithm,
"jwt_expire_hours": settings.jwt_expire_hours,
"allowed_hosts": settings.allowed_hosts,
"enable_authentication": settings.enable_authentication,
}
return config
def get_hardware_config(self) -> Dict[str, Any]:
"""Get hardware configuration."""
settings = self.settings
domain_config = self.domain_config
config = {
"wifi_interface": settings.wifi_interface,
"csi_buffer_size": settings.csi_buffer_size,
"polling_interval": settings.hardware_polling_interval,
"mock_hardware": settings.mock_hardware,
"routers": [router.dict() for router in domain_config.routers],
}
return config
def get_pose_config(self) -> Dict[str, Any]:
"""Get pose estimation configuration."""
settings = self.settings
domain_config = self.domain_config
config = {
"model_path": settings.pose_model_path,
"confidence_threshold": settings.pose_confidence_threshold,
"batch_size": settings.pose_processing_batch_size,
"max_persons": settings.pose_max_persons,
"mock_pose_data": settings.mock_pose_data,
"models": [model.dict() for model in domain_config.pose_models],
}
return config
def get_streaming_config(self) -> Dict[str, Any]:
"""Get streaming configuration."""
settings = self.settings
domain_config = self.domain_config
config = {
"fps": settings.stream_fps,
"buffer_size": settings.stream_buffer_size,
"websocket_ping_interval": settings.websocket_ping_interval,
"websocket_timeout": settings.websocket_timeout,
"enable_websockets": settings.enable_websockets,
"enable_real_time_processing": settings.enable_real_time_processing,
"max_connections": domain_config.streaming.max_connections,
"compression": domain_config.streaming.compression,
}
return config
def get_storage_config(self) -> Dict[str, Any]:
"""Get storage configuration."""
settings = self.settings
config = {
"data_path": Path(settings.data_storage_path),
"model_path": Path(settings.model_storage_path),
"temp_path": Path(settings.temp_storage_path),
"max_size_gb": settings.max_storage_size_gb,
"enable_historical_data": settings.enable_historical_data,
}
# Ensure directories exist
for path in [config["data_path"], config["model_path"], config["temp_path"]]:
path.mkdir(parents=True, exist_ok=True)
return config
def get_monitoring_config(self) -> Dict[str, Any]:
"""Get monitoring configuration."""
settings = self.settings
config = {
"metrics_enabled": settings.metrics_enabled,
"health_check_interval": settings.health_check_interval,
"performance_monitoring": settings.performance_monitoring,
"log_level": settings.log_level,
"log_file": settings.log_file,
}
return config
def get_rate_limiting_config(self) -> Dict[str, Any]:
"""Get rate limiting configuration."""
settings = self.settings
config = {
"enabled": settings.enable_rate_limiting,
"requests": settings.rate_limit_requests,
"authenticated_requests": settings.rate_limit_authenticated_requests,
"window": settings.rate_limit_window,
}
return config
def validate_configuration(self) -> List[str]:
"""Validate complete configuration and return issues."""
issues = []
try:
# Validate settings
from src.config.settings import validate_settings
settings_issues = validate_settings(self.settings)
issues.extend(settings_issues)
# Validate database configuration
try:
db_config = self.get_database_config()
if not db_config["url"]:
issues.append("Database URL is not configured")
except Exception as e:
issues.append(f"Database configuration error: {e}")
# Validate storage paths
try:
storage_config = self.get_storage_config()
for name, path in storage_config.items():
if name.endswith("_path") and not path.exists():
issues.append(f"Storage path does not exist: {path}")
except Exception as e:
issues.append(f"Storage configuration error: {e}")
# Validate hardware configuration
try:
hw_config = self.get_hardware_config()
if not hw_config["routers"]:
issues.append("No routers configured")
except Exception as e:
issues.append(f"Hardware configuration error: {e}")
# Validate pose configuration
try:
pose_config = self.get_pose_config()
if not pose_config["models"]:
issues.append("No pose models configured")
except Exception as e:
issues.append(f"Pose configuration error: {e}")
except Exception as e:
issues.append(f"Configuration validation error: {e}")
return issues
def get_full_config(self) -> Dict[str, Any]:
"""Get complete configuration dictionary."""
return {
"settings": self.settings.dict(),
"domain_config": self.domain_config.to_dict(),
"database": self.get_database_config(),
"redis": self.get_redis_config(),
"security": self.get_security_config(),
"hardware": self.get_hardware_config(),
"pose": self.get_pose_config(),
"streaming": self.get_streaming_config(),
"storage": self.get_storage_config(),
"monitoring": self.get_monitoring_config(),
"rate_limiting": self.get_rate_limiting_config(),
}
# Global configuration manager instance
@lru_cache()
def get_config_manager() -> ConfigManager:
"""Get cached configuration manager instance."""
return ConfigManager()
# Convenience functions
def get_app_settings() -> Settings:
"""Get application settings."""
return get_config_manager().settings
def get_app_domain_config() -> DomainConfig:
"""Get domain configuration."""
return get_config_manager().domain_config
def validate_app_configuration() -> List[str]:
"""Validate application configuration."""
return get_config_manager().validate_configuration()
def reload_configuration():
"""Reload all configuration."""
config_manager = get_config_manager()
config_manager.reload_settings()
config_manager.reload_domain_config()
logger.info("Configuration reloaded")
archive/v1/src/config/__init__.py
+8
View File
@@ -0,0 +1,8 @@
"""
Configuration management package
"""
from .settings import get_settings, Settings
from .domains import DomainConfig, get_domain_config
__all__ = ["get_settings", "Settings", "DomainConfig", "get_domain_config"]
archive/v1/src/config/domains.py
+481
View File
@@ -0,0 +1,481 @@
"""
Domain-specific configuration for WiFi-DensePose
"""
from typing import Dict, List, Optional, Any
from dataclasses import dataclass, field
from enum import Enum
from functools import lru_cache
from pydantic import BaseModel, Field, validator
class ZoneType(str, Enum):
"""Zone types for pose detection."""
ROOM = "room"
HALLWAY = "hallway"
ENTRANCE = "entrance"
OUTDOOR = "outdoor"
OFFICE = "office"
MEETING_ROOM = "meeting_room"
KITCHEN = "kitchen"
BATHROOM = "bathroom"
BEDROOM = "bedroom"
LIVING_ROOM = "living_room"
class ActivityType(str, Enum):
"""Activity types for pose classification."""
STANDING = "standing"
SITTING = "sitting"
WALKING = "walking"
LYING = "lying"
RUNNING = "running"
JUMPING = "jumping"
FALLING = "falling"
UNKNOWN = "unknown"
class HardwareType(str, Enum):
"""Hardware types for WiFi devices."""
ROUTER = "router"
ACCESS_POINT = "access_point"
REPEATER = "repeater"
MESH_NODE = "mesh_node"
CUSTOM = "custom"
@dataclass
class ZoneConfig:
"""Configuration for a detection zone."""
zone_id: str
name: str
zone_type: ZoneType
description: Optional[str] = None
# Physical boundaries (in meters)
x_min: float = 0.0
x_max: float = 10.0
y_min: float = 0.0
y_max: float = 10.0
z_min: float = 0.0
z_max: float = 3.0
# Detection settings
enabled: bool = True
confidence_threshold: float = 0.5
max_persons: int = 5
activity_detection: bool = True
# Hardware assignments
primary_router: Optional[str] = None
secondary_routers: List[str] = field(default_factory=list)
# Processing settings
processing_interval: float = 0.1 # seconds
data_retention_hours: int = 24
# Alert settings
enable_alerts: bool = False
alert_threshold: float = 0.8
alert_activities: List[ActivityType] = field(default_factory=list)
@dataclass
class RouterConfig:
"""Configuration for a WiFi router/device."""
router_id: str
name: str
hardware_type: HardwareType
# Network settings
ip_address: str
mac_address: str
interface: str = "wlan0"
channel: int = 6
frequency: float = 2.4 # GHz
# CSI settings
csi_enabled: bool = True
csi_rate: int = 100 # Hz
csi_subcarriers: int = 56
antenna_count: int = 3
# Position (in meters)
x_position: float = 0.0
y_position: float = 0.0
z_position: float = 2.5 # typical ceiling mount
# Calibration
calibrated: bool = False
calibration_data: Optional[Dict[str, Any]] = None
# Status
enabled: bool = True
last_seen: Optional[str] = None
# Performance settings
max_connections: int = 50
power_level: int = 20 # dBm
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"router_id": self.router_id,
"name": self.name,
"hardware_type": self.hardware_type.value,
"ip_address": self.ip_address,
"mac_address": self.mac_address,
"interface": self.interface,
"channel": self.channel,
"frequency": self.frequency,
"csi_enabled": self.csi_enabled,
"csi_rate": self.csi_rate,
"csi_subcarriers": self.csi_subcarriers,
"antenna_count": self.antenna_count,
"position": {
"x": self.x_position,
"y": self.y_position,
"z": self.z_position
},
"calibrated": self.calibrated,
"calibration_data": self.calibration_data,
"enabled": self.enabled,
"last_seen": self.last_seen,
"max_connections": self.max_connections,
"power_level": self.power_level
}
class PoseModelConfig(BaseModel):
"""Configuration for pose estimation models."""
model_name: str = Field(..., description="Model name")
model_path: str = Field(..., description="Path to model file")
model_type: str = Field(default="densepose", description="Model type")
# Input settings
input_width: int = Field(default=256, description="Input image width")
input_height: int = Field(default=256, description="Input image height")
input_channels: int = Field(default=3, description="Input channels")
# Processing settings
batch_size: int = Field(default=1, description="Batch size for inference")
confidence_threshold: float = Field(default=0.5, description="Confidence threshold")
nms_threshold: float = Field(default=0.4, description="NMS threshold")
# Output settings
max_detections: int = Field(default=10, description="Maximum detections per frame")
keypoint_count: int = Field(default=17, description="Number of keypoints")
# Performance settings
use_gpu: bool = Field(default=True, description="Use GPU acceleration")
gpu_memory_fraction: float = Field(default=0.5, description="GPU memory fraction")
num_threads: int = Field(default=4, description="Number of CPU threads")
@validator("confidence_threshold", "nms_threshold", "gpu_memory_fraction")
def validate_thresholds(cls, v):
"""Validate threshold values."""
if not 0.0 <= v <= 1.0:
raise ValueError("Threshold must be between 0.0 and 1.0")
return v
class StreamingConfig(BaseModel):
"""Configuration for real-time streaming."""
# Stream settings
fps: int = Field(default=30, description="Frames per second")
resolution: str = Field(default="720p", description="Stream resolution")
quality: str = Field(default="medium", description="Stream quality")
# Buffer settings
buffer_size: int = Field(default=100, description="Buffer size")
max_latency_ms: int = Field(default=100, description="Maximum latency in milliseconds")
# Compression settings
compression_enabled: bool = Field(default=True, description="Enable compression")
compression_level: int = Field(default=5, description="Compression level (1-9)")
# WebSocket settings
ping_interval: int = Field(default=60, description="Ping interval in seconds")
timeout: int = Field(default=300, description="Connection timeout in seconds")
max_connections: int = Field(default=100, description="Maximum concurrent connections")
# Data filtering
min_confidence: float = Field(default=0.5, description="Minimum confidence for streaming")
include_metadata: bool = Field(default=True, description="Include metadata in stream")
@validator("fps")
def validate_fps(cls, v):
"""Validate FPS value."""
if not 1 <= v <= 60:
raise ValueError("FPS must be between 1 and 60")
return v
@validator("compression_level")
def validate_compression_level(cls, v):
"""Validate compression level."""
if not 1 <= v <= 9:
raise ValueError("Compression level must be between 1 and 9")
return v
class AlertConfig(BaseModel):
"""Configuration for alerts and notifications."""
# Alert types
enable_pose_alerts: bool = Field(default=False, description="Enable pose-based alerts")
enable_activity_alerts: bool = Field(default=False, description="Enable activity-based alerts")
enable_zone_alerts: bool = Field(default=False, description="Enable zone-based alerts")
enable_system_alerts: bool = Field(default=True, description="Enable system alerts")
# Thresholds
confidence_threshold: float = Field(default=0.8, description="Alert confidence threshold")
duration_threshold: int = Field(default=5, description="Alert duration threshold in seconds")
# Activities that trigger alerts
alert_activities: List[ActivityType] = Field(
default=[ActivityType.FALLING],
description="Activities that trigger alerts"
)
# Notification settings
email_enabled: bool = Field(default=False, description="Enable email notifications")
webhook_enabled: bool = Field(default=False, description="Enable webhook notifications")
sms_enabled: bool = Field(default=False, description="Enable SMS notifications")
# Rate limiting
max_alerts_per_hour: int = Field(default=10, description="Maximum alerts per hour")
cooldown_minutes: int = Field(default=5, description="Cooldown between similar alerts")
class DomainConfig:
"""Main domain configuration container."""
def __init__(self):
self.zones: Dict[str, ZoneConfig] = {}
self.routers: Dict[str, RouterConfig] = {}
self.pose_models: Dict[str, PoseModelConfig] = {}
self.streaming = StreamingConfig()
self.alerts = AlertConfig()
# Load default configurations
self._load_defaults()
def _load_defaults(self):
"""Load default configurations."""
# Default pose model
self.pose_models["default"] = PoseModelConfig(
model_name="densepose_rcnn_R_50_FPN_s1x",
model_path="./models/densepose_rcnn_R_50_FPN_s1x.pkl",
model_type="densepose"
)
# Example zone
self.zones["living_room"] = ZoneConfig(
zone_id="living_room",
name="Living Room",
zone_type=ZoneType.LIVING_ROOM,
description="Main living area",
x_max=5.0,
y_max=4.0,
z_max=3.0
)
# Example router
self.routers["main_router"] = RouterConfig(
router_id="main_router",
name="Main Router",
hardware_type=HardwareType.ROUTER,
ip_address="192.168.1.1",
mac_address="00:11:22:33:44:55",
x_position=2.5,
y_position=2.0,
z_position=2.5
)
def add_zone(self, zone: ZoneConfig):
"""Add a zone configuration."""
self.zones[zone.zone_id] = zone
def add_router(self, router: RouterConfig):
"""Add a router configuration."""
self.routers[router.router_id] = router
def add_pose_model(self, model: PoseModelConfig):
"""Add a pose model configuration."""
self.pose_models[model.model_name] = model
def get_zone(self, zone_id: str) -> Optional[ZoneConfig]:
"""Get zone configuration by ID."""
return self.zones.get(zone_id)
def get_router(self, router_id: str) -> Optional[RouterConfig]:
"""Get router configuration by ID."""
return self.routers.get(router_id)
def get_pose_model(self, model_name: str) -> Optional[PoseModelConfig]:
"""Get pose model configuration by name."""
return self.pose_models.get(model_name)
def get_zones_for_router(self, router_id: str) -> List[ZoneConfig]:
"""Get zones that use a specific router."""
zones = []
for zone in self.zones.values():
if (zone.primary_router == router_id or
router_id in zone.secondary_routers):
zones.append(zone)
return zones
def get_routers_for_zone(self, zone_id: str) -> List[RouterConfig]:
"""Get routers assigned to a specific zone."""
zone = self.get_zone(zone_id)
if not zone:
return []
routers = []
# Add primary router
if zone.primary_router and zone.primary_router in self.routers:
routers.append(self.routers[zone.primary_router])
# Add secondary routers
for router_id in zone.secondary_routers:
if router_id in self.routers:
routers.append(self.routers[router_id])
return routers
def get_all_routers(self) -> List[RouterConfig]:
"""Get all router configurations."""
return list(self.routers.values())
def validate_configuration(self) -> List[str]:
"""Validate the entire configuration."""
issues = []
# Validate zones
for zone_id, zone in self.zones.items():
if zone.primary_router and zone.primary_router not in self.routers:
issues.append(f"Zone {zone_id} references unknown primary router: {zone.primary_router}")
for router_id in zone.secondary_routers:
if router_id not in self.routers:
issues.append(f"Zone {zone_id} references unknown secondary router: {router_id}")
# Validate routers
for router_id, router in self.routers.items():
if not router.ip_address:
issues.append(f"Router {router_id} missing IP address")
if not router.mac_address:
issues.append(f"Router {router_id} missing MAC address")
# Validate pose models
for model_name, model in self.pose_models.items():
import os
if not os.path.exists(model.model_path):
issues.append(f"Pose model {model_name} file not found: {model.model_path}")
return issues
def to_dict(self) -> Dict[str, Any]:
"""Convert configuration to dictionary."""
return {
"zones": {
zone_id: {
"zone_id": zone.zone_id,
"name": zone.name,
"zone_type": zone.zone_type.value,
"description": zone.description,
"boundaries": {
"x_min": zone.x_min,
"x_max": zone.x_max,
"y_min": zone.y_min,
"y_max": zone.y_max,
"z_min": zone.z_min,
"z_max": zone.z_max
},
"settings": {
"enabled": zone.enabled,
"confidence_threshold": zone.confidence_threshold,
"max_persons": zone.max_persons,
"activity_detection": zone.activity_detection
},
"hardware": {
"primary_router": zone.primary_router,
"secondary_routers": zone.secondary_routers
}
}
for zone_id, zone in self.zones.items()
},
"routers": {
router_id: router.to_dict()
for router_id, router in self.routers.items()
},
"pose_models": {
model_name: model.dict()
for model_name, model in self.pose_models.items()
},
"streaming": self.streaming.dict(),
"alerts": self.alerts.dict()
}
@lru_cache()
def get_domain_config() -> DomainConfig:
"""Get cached domain configuration instance."""
return DomainConfig()
def load_domain_config_from_file(file_path: str) -> DomainConfig:
"""Load domain configuration from file."""
import json
config = DomainConfig()
try:
with open(file_path, 'r') as f:
data = json.load(f)
# Load zones
for zone_data in data.get("zones", []):
zone = ZoneConfig(**zone_data)
config.add_zone(zone)
# Load routers
for router_data in data.get("routers", []):
router = RouterConfig(**router_data)
config.add_router(router)
# Load pose models
for model_data in data.get("pose_models", []):
model = PoseModelConfig(**model_data)
config.add_pose_model(model)
# Load streaming config
if "streaming" in data:
config.streaming = StreamingConfig(**data["streaming"])
# Load alerts config
if "alerts" in data:
config.alerts = AlertConfig(**data["alerts"])
except Exception as e:
raise ValueError(f"Failed to load domain configuration: {e}")
return config
def save_domain_config_to_file(config: DomainConfig, file_path: str):
"""Save domain configuration to file."""
import json
try:
with open(file_path, 'w') as f:
json.dump(config.to_dict(), f, indent=2)
except Exception as e:
raise ValueError(f"Failed to save domain configuration: {e}")
archive/v1/src/config/settings.py
+437
View File
@@ -0,0 +1,437 @@
"""
Pydantic settings for WiFi-DensePose API
"""
import os
from typing import List, Optional, Dict, Any
from functools import lru_cache
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
"""Application settings with environment variable support."""
# Application settings
app_name: str = Field(default="WiFi-DensePose API", description="Application name")
version: str = Field(default="1.0.0", description="Application version")
environment: str = Field(default="development", description="Environment (development, staging, production)")
debug: bool = Field(default=False, description="Debug mode")
# Server settings
host: str = Field(default="0.0.0.0", description="Server host")
port: int = Field(default=8000, description="Server port")
reload: bool = Field(default=False, description="Auto-reload on code changes")
workers: int = Field(default=1, description="Number of worker processes")
# Security settings
secret_key: str = Field(..., description="Secret key for JWT tokens")
jwt_algorithm: str = Field(default="HS256", description="JWT algorithm")
jwt_expire_hours: int = Field(default=24, description="JWT token expiration in hours")
allowed_hosts: List[str] = Field(default=["*"], description="Allowed hosts")
cors_origins: List[str] = Field(default=["*"], description="CORS allowed origins")
# Rate limiting settings
rate_limit_requests: int = Field(default=100, description="Rate limit requests per window")
rate_limit_authenticated_requests: int = Field(default=1000, description="Rate limit for authenticated users")
rate_limit_window: int = Field(default=3600, description="Rate limit window in seconds")
# Database settings
database_url: Optional[str] = Field(default=None, description="Database connection URL")
database_pool_size: int = Field(default=10, description="Database connection pool size")
database_max_overflow: int = Field(default=20, description="Database max overflow connections")
# Database connection pool settings (alternative naming for compatibility)
db_pool_size: int = Field(default=10, description="Database connection pool size")
db_max_overflow: int = Field(default=20, description="Database max overflow connections")
db_pool_timeout: int = Field(default=30, description="Database pool timeout in seconds")
db_pool_recycle: int = Field(default=3600, description="Database pool recycle time in seconds")
# Database connection settings
db_host: Optional[str] = Field(default=None, description="Database host")
db_port: int = Field(default=5432, description="Database port")
db_name: Optional[str] = Field(default=None, description="Database name")
db_user: Optional[str] = Field(default=None, description="Database user")
db_password: Optional[str] = Field(default=None, description="Database password")
db_echo: bool = Field(default=False, description="Enable database query logging")
# Redis settings (for caching and rate limiting)
redis_url: Optional[str] = Field(default=None, description="Redis connection URL")
redis_password: Optional[str] = Field(default=None, description="Redis password")
redis_db: int = Field(default=0, description="Redis database number")
redis_enabled: bool = Field(default=True, description="Enable Redis")
redis_host: str = Field(default="localhost", description="Redis host")
redis_port: int = Field(default=6379, description="Redis port")
redis_required: bool = Field(default=False, description="Require Redis connection (fail if unavailable)")
redis_max_connections: int = Field(default=10, description="Maximum Redis connections")
redis_socket_timeout: int = Field(default=5, description="Redis socket timeout in seconds")
redis_connect_timeout: int = Field(default=5, description="Redis connection timeout in seconds")
# Failsafe settings
enable_database_failsafe: bool = Field(default=True, description="Enable automatic SQLite failsafe when PostgreSQL unavailable")
enable_redis_failsafe: bool = Field(default=True, description="Enable automatic Redis failsafe (disable when unavailable)")
sqlite_fallback_path: str = Field(default="./data/wifi_densepose_fallback.db", description="SQLite fallback database path")
# Hardware settings
wifi_interface: str = Field(default="wlan0", description="WiFi interface name")
csi_buffer_size: int = Field(default=1000, description="CSI data buffer size")
hardware_polling_interval: float = Field(default=0.1, description="Hardware polling interval in seconds")
router_ssh_username: str = Field(default="admin", description="Default SSH username for router connections")
router_ssh_password: str = Field(default="", description="Default SSH password for router connections (set via ROUTER_SSH_PASSWORD env var)")
# CSI Processing settings
csi_sampling_rate: int = Field(default=1000, description="CSI sampling rate")
csi_window_size: int = Field(default=512, description="CSI window size")
csi_overlap: float = Field(default=0.5, description="CSI window overlap")
csi_noise_threshold: float = Field(default=0.1, description="CSI noise threshold")
csi_human_detection_threshold: float = Field(default=0.8, description="CSI human detection threshold")
csi_smoothing_factor: float = Field(default=0.9, description="CSI smoothing factor")
csi_max_history_size: int = Field(default=500, description="CSI max history size")
# Pose estimation settings
pose_model_path: Optional[str] = Field(default=None, description="Path to pose estimation model")
pose_confidence_threshold: float = Field(default=0.5, description="Minimum confidence threshold")
pose_processing_batch_size: int = Field(default=32, description="Batch size for pose processing")
pose_max_persons: int = Field(default=10, description="Maximum persons to detect per frame")
# Streaming settings
stream_fps: int = Field(default=30, description="Streaming frames per second")
stream_buffer_size: int = Field(default=100, description="Stream buffer size")
websocket_ping_interval: int = Field(default=60, description="WebSocket ping interval in seconds")
websocket_timeout: int = Field(default=300, description="WebSocket timeout in seconds")
# Logging settings
log_level: str = Field(default="INFO", description="Logging level")
log_format: str = Field(
default="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
description="Log format"
)
log_file: Optional[str] = Field(default=None, description="Log file path")
log_directory: str = Field(default="./logs", description="Log directory path")
log_max_size: int = Field(default=10485760, description="Max log file size in bytes (10MB)")
log_backup_count: int = Field(default=5, description="Number of log backup files")
# Monitoring settings
metrics_enabled: bool = Field(default=True, description="Enable metrics collection")
health_check_interval: int = Field(default=30, description="Health check interval in seconds")
performance_monitoring: bool = Field(default=True, description="Enable performance monitoring")
monitoring_interval_seconds: int = Field(default=60, description="Monitoring task interval in seconds")
cleanup_interval_seconds: int = Field(default=3600, description="Cleanup task interval in seconds")
backup_interval_seconds: int = Field(default=86400, description="Backup task interval in seconds")
# Storage settings
data_storage_path: str = Field(default="./data", description="Data storage directory")
model_storage_path: str = Field(default="./models", description="Model storage directory")
temp_storage_path: str = Field(default="./temp", description="Temporary storage directory")
backup_directory: str = Field(default="./backups", description="Backup storage directory")
max_storage_size_gb: int = Field(default=100, description="Maximum storage size in GB")
# API settings
api_prefix: str = Field(default="/api/v1", description="API prefix")
docs_url: str = Field(default="/docs", description="API documentation URL")
redoc_url: str = Field(default="/redoc", description="ReDoc documentation URL")
openapi_url: str = Field(default="/openapi.json", description="OpenAPI schema URL")
# Feature flags
enable_authentication: bool = Field(default=True, description="Enable authentication")
enable_rate_limiting: bool = Field(default=True, description="Enable rate limiting")
enable_websockets: bool = Field(default=True, description="Enable WebSocket support")
enable_historical_data: bool = Field(default=True, description="Enable historical data storage")
enable_real_time_processing: bool = Field(default=True, description="Enable real-time processing")
cors_enabled: bool = Field(default=True, description="Enable CORS middleware")
cors_allow_credentials: bool = Field(default=True, description="Allow credentials in CORS")
# Development settings
mock_hardware: bool = Field(default=False, description="Use mock hardware for development")
mock_pose_data: bool = Field(default=False, description="Use mock pose data for development")
enable_test_endpoints: bool = Field(default=False, description="Enable test endpoints")
# Cleanup settings
csi_data_retention_days: int = Field(default=30, description="CSI data retention in days")
pose_detection_retention_days: int = Field(default=30, description="Pose detection retention in days")
metrics_retention_days: int = Field(default=7, description="Metrics retention in days")
audit_log_retention_days: int = Field(default=90, description="Audit log retention in days")
orphaned_session_threshold_days: int = Field(default=7, description="Orphaned session threshold in days")
cleanup_batch_size: int = Field(default=1000, description="Cleanup batch size")
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
case_sensitive=False
)
@field_validator("environment")
@classmethod
def validate_environment(cls, v):
"""Validate environment setting."""
allowed_environments = ["development", "staging", "production"]
if v not in allowed_environments:
raise ValueError(f"Environment must be one of: {allowed_environments}")
return v
@field_validator("log_level")
@classmethod
def validate_log_level(cls, v):
"""Validate log level setting."""
allowed_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
if v.upper() not in allowed_levels:
raise ValueError(f"Log level must be one of: {allowed_levels}")
return v.upper()
@field_validator("pose_confidence_threshold")
@classmethod
def validate_confidence_threshold(cls, v):
"""Validate confidence threshold."""
if not 0.0 <= v <= 1.0:
raise ValueError("Confidence threshold must be between 0.0 and 1.0")
return v
@field_validator("stream_fps")
@classmethod
def validate_stream_fps(cls, v):
"""Validate streaming FPS."""
if not 1 <= v <= 60:
raise ValueError("Stream FPS must be between 1 and 60")
return v
@field_validator("port")
@classmethod
def validate_port(cls, v):
"""Validate port number."""
if not 1 <= v <= 65535:
raise ValueError("Port must be between 1 and 65535")
return v
@field_validator("workers")
@classmethod
def validate_workers(cls, v):
"""Validate worker count."""
if v < 1:
raise ValueError("Workers must be at least 1")
return v
@field_validator("db_port")
@classmethod
def validate_db_port(cls, v):
"""Validate database port."""
if not 1 <= v <= 65535:
raise ValueError("Database port must be between 1 and 65535")
return v
@field_validator("redis_port")
@classmethod
def validate_redis_port(cls, v):
"""Validate Redis port."""
if not 1 <= v <= 65535:
raise ValueError("Redis port must be between 1 and 65535")
return v
@field_validator("db_pool_size")
@classmethod
def validate_db_pool_size(cls, v):
"""Validate database pool size."""
if v < 1:
raise ValueError("Database pool size must be at least 1")
return v
@field_validator("monitoring_interval_seconds", "cleanup_interval_seconds", "backup_interval_seconds")
@classmethod
def validate_interval_seconds(cls, v):
"""Validate interval settings."""
if v < 0:
raise ValueError("Interval seconds must be non-negative")
return v
@property
def is_development(self) -> bool:
"""Check if running in development environment."""
return self.environment == "development"
@property
def is_production(self) -> bool:
"""Check if running in production environment."""
return self.environment == "production"
@property
def is_testing(self) -> bool:
"""Check if running in testing environment."""
return self.environment == "testing"
def get_database_url(self) -> str:
"""Get database URL with fallback."""
if self.database_url:
return self.database_url
# Build URL from individual components if available
if self.db_host and self.db_name and self.db_user:
password_part = f":{self.db_password}" if self.db_password else ""
return f"postgresql://{self.db_user}{password_part}@{self.db_host}:{self.db_port}/{self.db_name}"
# Default SQLite database for development
if self.is_development:
return f"sqlite:///{self.data_storage_path}/wifi_densepose.db"
# SQLite failsafe for production if enabled
if self.enable_database_failsafe:
return f"sqlite:///{self.sqlite_fallback_path}"
raise ValueError("Database URL must be configured for non-development environments")
def get_sqlite_fallback_url(self) -> str:
"""Get SQLite fallback database URL."""
return f"sqlite:///{self.sqlite_fallback_path}"
def get_redis_url(self) -> Optional[str]:
"""Get Redis URL with fallback."""
if not self.redis_enabled:
return None
if self.redis_url:
return self.redis_url
# Build URL from individual components
password_part = f":{self.redis_password}@" if self.redis_password else ""
return f"redis://{password_part}{self.redis_host}:{self.redis_port}/{self.redis_db}"
def get_cors_config(self) -> Dict[str, Any]:
"""Get CORS configuration."""
if self.is_development:
return {
"allow_origins": ["*"],
"allow_credentials": True,
"allow_methods": ["*"],
"allow_headers": ["*"],
}
return {
"allow_origins": self.cors_origins,
"allow_credentials": True,
"allow_methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
"allow_headers": ["Authorization", "Content-Type"],
}
def get_logging_config(self) -> Dict[str, Any]:
"""Get logging configuration."""
config = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"default": {
"format": self.log_format,
},
"detailed": {
"format": "%(asctime)s - %(name)s - %(levelname)s - %(module)s:%(lineno)d - %(message)s",
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": self.log_level,
"formatter": "default",
"stream": "ext://sys.stdout",
},
},
"loggers": {
"": {
"level": self.log_level,
"handlers": ["console"],
},
"uvicorn": {
"level": "INFO",
"handlers": ["console"],
"propagate": False,
},
"fastapi": {
"level": "INFO",
"handlers": ["console"],
"propagate": False,
},
},
}
# Add file handler if log file is specified
if self.log_file:
config["handlers"]["file"] = {
"class": "logging.handlers.RotatingFileHandler",
"level": self.log_level,
"formatter": "detailed",
"filename": self.log_file,
"maxBytes": self.log_max_size,
"backupCount": self.log_backup_count,
}
# Add file handler to all loggers
for logger_config in config["loggers"].values():
logger_config["handlers"].append("file")
return config
def create_directories(self):
"""Create necessary directories."""
directories = [
self.data_storage_path,
self.model_storage_path,
self.temp_storage_path,
self.log_directory,
self.backup_directory,
]
for directory in directories:
os.makedirs(directory, exist_ok=True)
@lru_cache()
def get_settings() -> Settings:
"""Get cached settings instance."""
settings = Settings()
settings.create_directories()
return settings
def get_test_settings() -> Settings:
"""Get settings for testing."""
return Settings(
environment="testing",
debug=True,
secret_key="test-secret-key",
database_url="sqlite:///:memory:",
mock_hardware=True,
mock_pose_data=True,
enable_test_endpoints=True,
log_level="DEBUG"
)
def load_settings_from_file(file_path: str) -> Settings:
"""Load settings from a specific file."""
return Settings(_env_file=file_path)
def validate_settings(settings: Settings) -> List[str]:
"""Validate settings and return list of issues."""
issues = []
# Check required settings for production
if settings.is_production:
if not settings.secret_key or settings.secret_key == "change-me":
issues.append("Secret key must be set for production")
if not settings.database_url and not (settings.db_host and settings.db_name and settings.db_user):
issues.append("Database URL or database connection parameters must be set for production")
if settings.debug:
issues.append("Debug mode should be disabled in production")
if "*" in settings.allowed_hosts:
issues.append("Allowed hosts should be restricted in production")
if "*" in settings.cors_origins:
issues.append("CORS origins should be restricted in production")
# Check storage paths exist
try:
settings.create_directories()
except Exception as e:
issues.append(f"Cannot create storage directories: {e}")
return issues
archive/v1/src/core/__init__.py
+13
View File
@@ -0,0 +1,13 @@
"""
Core package for WiFi-DensePose API
"""
from .csi_processor import CSIProcessor
from .phase_sanitizer import PhaseSanitizer
from .router_interface import RouterInterface
__all__ = [
'CSIProcessor',
'PhaseSanitizer',
'RouterInterface'
]
archive/v1/src/core/csi_processor.py
+470
View File
@@ -0,0 +1,470 @@
"""CSI data processor for WiFi-DensePose system using TDD approach."""
import asyncio
import itertools
import logging
import numpy as np
from datetime import datetime, timezone
from typing import Dict, Any, Optional, List
from dataclasses import dataclass
from collections import deque
import scipy.signal
import scipy.fft
try:
from ..hardware.csi_extractor import CSIData
except ImportError:
# Handle import for testing
from src.hardware.csi_extractor import CSIData
class CSIProcessingError(Exception):
"""Exception raised for CSI processing errors."""
pass
@dataclass
class CSIFeatures:
"""Data structure for extracted CSI features."""
amplitude_mean: np.ndarray
amplitude_variance: np.ndarray
phase_difference: np.ndarray
correlation_matrix: np.ndarray
doppler_shift: np.ndarray
power_spectral_density: np.ndarray
timestamp: datetime
metadata: Dict[str, Any]
@dataclass
class HumanDetectionResult:
"""Data structure for human detection results."""
human_detected: bool
confidence: float
motion_score: float
timestamp: datetime
features: CSIFeatures
metadata: Dict[str, Any]
class CSIProcessor:
"""Processes CSI data for human detection and pose estimation."""
def __init__(self, config: Dict[str, Any], logger: Optional[logging.Logger] = None):
"""Initialize CSI processor.
Args:
config: Configuration dictionary
logger: Optional logger instance
Raises:
ValueError: If configuration is invalid
"""
self._validate_config(config)
self.config = config
self.logger = logger or logging.getLogger(__name__)
# Processing parameters
self.sampling_rate = config['sampling_rate']
self.window_size = config['window_size']
self.overlap = config['overlap']
self.noise_threshold = config['noise_threshold']
self.human_detection_threshold = config.get('human_detection_threshold', 0.8)
self.smoothing_factor = config.get('smoothing_factor', 0.9)
self.max_history_size = config.get('max_history_size', 500)
# Feature extraction flags
self.enable_preprocessing = config.get('enable_preprocessing', True)
self.enable_feature_extraction = config.get('enable_feature_extraction', True)
self.enable_human_detection = config.get('enable_human_detection', True)
# Processing state
self.csi_history = deque(maxlen=self.max_history_size)
self.previous_detection_confidence = 0.0
# Doppler cache: pre-computed mean phase per frame for O(1) append
self._phase_cache = deque(maxlen=self.max_history_size)
self._doppler_window = min(config.get('doppler_window', 64), self.max_history_size)
# Statistics tracking
self._total_processed = 0
self._processing_errors = 0
self._human_detections = 0
def _validate_config(self, config: Dict[str, Any]) -> None:
"""Validate configuration parameters.
Args:
config: Configuration to validate
Raises:
ValueError: If configuration is invalid
"""
required_fields = ['sampling_rate', 'window_size', 'overlap', 'noise_threshold']
missing_fields = [field for field in required_fields if field not in config]
if missing_fields:
raise ValueError(f"Missing required configuration: {missing_fields}")
if config['sampling_rate'] <= 0:
raise ValueError("sampling_rate must be positive")
if config['window_size'] <= 0:
raise ValueError("window_size must be positive")
if not 0 <= config['overlap'] < 1:
raise ValueError("overlap must be between 0 and 1")
def preprocess_csi_data(self, csi_data: CSIData) -> CSIData:
"""Preprocess CSI data for feature extraction.
Args:
csi_data: Raw CSI data
Returns:
Preprocessed CSI data
Raises:
CSIProcessingError: If preprocessing fails
"""
if not self.enable_preprocessing:
return csi_data
try:
# Remove noise from the signal
cleaned_data = self._remove_noise(csi_data)
# Apply windowing function
windowed_data = self._apply_windowing(cleaned_data)
# Normalize amplitude values
normalized_data = self._normalize_amplitude(windowed_data)
return normalized_data
except Exception as e:
raise CSIProcessingError(f"Failed to preprocess CSI data: {e}")
def extract_features(self, csi_data: CSIData) -> Optional[CSIFeatures]:
"""Extract features from CSI data.
Args:
csi_data: Preprocessed CSI data
Returns:
Extracted features or None if disabled
Raises:
CSIProcessingError: If feature extraction fails
"""
if not self.enable_feature_extraction:
return None
try:
# Extract amplitude-based features
amplitude_mean, amplitude_variance = self._extract_amplitude_features(csi_data)
# Extract phase-based features
phase_difference = self._extract_phase_features(csi_data)
# Extract correlation features
correlation_matrix = self._extract_correlation_features(csi_data)
# Extract Doppler and frequency features
doppler_shift, power_spectral_density = self._extract_doppler_features(csi_data)
return CSIFeatures(
amplitude_mean=amplitude_mean,
amplitude_variance=amplitude_variance,
phase_difference=phase_difference,
correlation_matrix=correlation_matrix,
doppler_shift=doppler_shift,
power_spectral_density=power_spectral_density,
timestamp=datetime.now(timezone.utc),
metadata={'processing_params': self.config}
)
except Exception as e:
raise CSIProcessingError(f"Failed to extract features: {e}")
def detect_human_presence(self, features: CSIFeatures) -> Optional[HumanDetectionResult]:
"""Detect human presence from CSI features.
Args:
features: Extracted CSI features
Returns:
Detection result or None if disabled
Raises:
CSIProcessingError: If detection fails
"""
if not self.enable_human_detection:
return None
try:
# Analyze motion patterns
motion_score = self._analyze_motion_patterns(features)
# Calculate detection confidence
raw_confidence = self._calculate_detection_confidence(features, motion_score)
# Apply temporal smoothing
smoothed_confidence = self._apply_temporal_smoothing(raw_confidence)
# Determine if human is detected
human_detected = smoothed_confidence >= self.human_detection_threshold
if human_detected:
self._human_detections += 1
return HumanDetectionResult(
human_detected=human_detected,
confidence=smoothed_confidence,
motion_score=motion_score,
timestamp=datetime.now(timezone.utc),
features=features,
metadata={'threshold': self.human_detection_threshold}
)
except Exception as e:
raise CSIProcessingError(f"Failed to detect human presence: {e}")
async def process_csi_data(self, csi_data: CSIData) -> HumanDetectionResult:
"""Process CSI data through the complete pipeline.
Args:
csi_data: Raw CSI data
Returns:
Human detection result
Raises:
CSIProcessingError: If processing fails
"""
try:
self._total_processed += 1
# Preprocess the data
preprocessed_data = self.preprocess_csi_data(csi_data)
# Extract features
features = self.extract_features(preprocessed_data)
# Detect human presence
detection_result = self.detect_human_presence(features)
# Add to history
self.add_to_history(csi_data)
return detection_result
except Exception as e:
self._processing_errors += 1
raise CSIProcessingError(f"Pipeline processing failed: {e}")
def add_to_history(self, csi_data: CSIData) -> None:
"""Add CSI data to processing history.
Args:
csi_data: CSI data to add to history
"""
self.csi_history.append(csi_data)
# Cache mean phase for fast Doppler extraction
if csi_data.phase.ndim == 2:
self._phase_cache.append(np.mean(csi_data.phase, axis=0))
else:
self._phase_cache.append(csi_data.phase.flatten())
def clear_history(self) -> None:
"""Clear the CSI data history."""
self.csi_history.clear()
self._phase_cache.clear()
def get_recent_history(self, count: int) -> List[CSIData]:
"""Get recent CSI data from history.
Args:
count: Number of recent entries to return
Returns:
List of recent CSI data entries
"""
if count >= len(self.csi_history):
return list(self.csi_history)
else:
start = len(self.csi_history) - count
return list(itertools.islice(self.csi_history, start, len(self.csi_history)))
def get_processing_statistics(self) -> Dict[str, Any]:
"""Get processing statistics.
Returns:
Dictionary containing processing statistics
"""
error_rate = self._processing_errors / self._total_processed if self._total_processed > 0 else 0
detection_rate = self._human_detections / self._total_processed if self._total_processed > 0 else 0
return {
'total_processed': self._total_processed,
'processing_errors': self._processing_errors,
'human_detections': self._human_detections,
'error_rate': error_rate,
'detection_rate': detection_rate,
'history_size': len(self.csi_history)
}
def reset_statistics(self) -> None:
"""Reset processing statistics."""
self._total_processed = 0
self._processing_errors = 0
self._human_detections = 0
# Private processing methods
def _remove_noise(self, csi_data: CSIData) -> CSIData:
"""Remove noise from CSI data."""
# Apply noise filtering based on threshold
amplitude_db = 20 * np.log10(np.abs(csi_data.amplitude) + 1e-12)
noise_mask = amplitude_db > self.noise_threshold
filtered_amplitude = csi_data.amplitude.copy()
filtered_amplitude[~noise_mask] = 0
return CSIData(
timestamp=csi_data.timestamp,
amplitude=filtered_amplitude,
phase=csi_data.phase,
frequency=csi_data.frequency,
bandwidth=csi_data.bandwidth,
num_subcarriers=csi_data.num_subcarriers,
num_antennas=csi_data.num_antennas,
snr=csi_data.snr,
metadata={**csi_data.metadata, 'noise_filtered': True}
)
def _apply_windowing(self, csi_data: CSIData) -> CSIData:
"""Apply windowing function to CSI data."""
# Apply Hamming window to reduce spectral leakage
window = scipy.signal.windows.hamming(csi_data.num_subcarriers)
windowed_amplitude = csi_data.amplitude * window[np.newaxis, :]
return CSIData(
timestamp=csi_data.timestamp,
amplitude=windowed_amplitude,
phase=csi_data.phase,
frequency=csi_data.frequency,
bandwidth=csi_data.bandwidth,
num_subcarriers=csi_data.num_subcarriers,
num_antennas=csi_data.num_antennas,
snr=csi_data.snr,
metadata={**csi_data.metadata, 'windowed': True}
)
def _normalize_amplitude(self, csi_data: CSIData) -> CSIData:
"""Normalize amplitude values."""
# Normalize to unit variance
normalized_amplitude = csi_data.amplitude / (np.std(csi_data.amplitude) + 1e-12)
return CSIData(
timestamp=csi_data.timestamp,
amplitude=normalized_amplitude,
phase=csi_data.phase,
frequency=csi_data.frequency,
bandwidth=csi_data.bandwidth,
num_subcarriers=csi_data.num_subcarriers,
num_antennas=csi_data.num_antennas,
snr=csi_data.snr,
metadata={**csi_data.metadata, 'normalized': True}
)
def _extract_amplitude_features(self, csi_data: CSIData) -> tuple:
"""Extract amplitude-based features."""
amplitude_mean = np.mean(csi_data.amplitude, axis=0)
amplitude_variance = np.var(csi_data.amplitude, axis=0)
return amplitude_mean, amplitude_variance
def _extract_phase_features(self, csi_data: CSIData) -> np.ndarray:
"""Extract phase-based features."""
# Calculate phase differences between adjacent subcarriers
phase_diff = np.diff(csi_data.phase, axis=1)
return np.mean(phase_diff, axis=0)
def _extract_correlation_features(self, csi_data: CSIData) -> np.ndarray:
"""Extract correlation features between antennas."""
# Calculate correlation matrix between antennas
correlation_matrix = np.corrcoef(csi_data.amplitude)
return correlation_matrix
def _extract_doppler_features(self, csi_data: CSIData) -> tuple:
"""Extract Doppler and frequency domain features from temporal CSI history.
Uses cached mean-phase values for O(1) access instead of recomputing
from raw CSI frames. Only uses the last `doppler_window` frames
(default 64) for bounded computation time.
Returns:
tuple: (doppler_shift, power_spectral_density) as numpy arrays
"""
n_doppler_bins = 64
if len(self._phase_cache) >= 2:
# Use cached mean-phase values (pre-computed in add_to_history)
# Only take the last doppler_window frames for bounded cost
window = min(len(self._phase_cache), self._doppler_window)
start = len(self._phase_cache) - window
cache_list = list(itertools.islice(self._phase_cache, start, len(self._phase_cache)))
phase_matrix = np.array(cache_list)
# Temporal phase differences between consecutive frames
phase_diffs = np.diff(phase_matrix, axis=0)
# Average across subcarriers for each time step
mean_phase_diff = np.mean(phase_diffs, axis=1)
# FFT for Doppler spectrum
doppler_spectrum = np.abs(scipy.fft.fft(mean_phase_diff, n=n_doppler_bins)) ** 2
# Normalize
max_val = np.max(doppler_spectrum)
if max_val > 0:
doppler_spectrum = doppler_spectrum / max_val
doppler_shift = doppler_spectrum
else:
doppler_shift = np.zeros(n_doppler_bins)
# Power spectral density of the current frame
psd = np.abs(scipy.fft.fft(csi_data.amplitude.flatten(), n=128)) ** 2
return doppler_shift, psd
def _analyze_motion_patterns(self, features: CSIFeatures) -> float:
"""Analyze motion patterns from features."""
# Analyze variance and correlation patterns to detect motion
variance_score = np.mean(features.amplitude_variance)
correlation_score = np.mean(np.abs(features.correlation_matrix - np.eye(features.correlation_matrix.shape[0])))
# Combine scores (simplified approach)
motion_score = 0.6 * variance_score + 0.4 * correlation_score
return np.clip(motion_score, 0.0, 1.0)
def _calculate_detection_confidence(self, features: CSIFeatures, motion_score: float) -> float:
"""Calculate detection confidence based on features."""
# Combine multiple feature indicators
amplitude_indicator = np.mean(features.amplitude_mean) > 0.1
phase_indicator = np.std(features.phase_difference) > 0.05
motion_indicator = motion_score > 0.3
# Weight the indicators
confidence = (0.4 * amplitude_indicator + 0.3 * phase_indicator + 0.3 * motion_indicator)
return np.clip(confidence, 0.0, 1.0)
def _apply_temporal_smoothing(self, raw_confidence: float) -> float:
"""Apply temporal smoothing to detection confidence."""
# Exponential moving average
smoothed_confidence = (self.smoothing_factor * self.previous_detection_confidence +
(1 - self.smoothing_factor) * raw_confidence)
self.previous_detection_confidence = smoothed_confidence
return smoothed_confidence
archive/v1/src/core/phase_sanitizer.py
+347
View File
@@ -0,0 +1,347 @@
"""Phase sanitization module for WiFi-DensePose system using TDD approach."""
import numpy as np
import logging
from typing import Dict, Any, Optional, Tuple
from datetime import datetime, timezone
from scipy import signal
class PhaseSanitizationError(Exception):
"""Exception raised for phase sanitization errors."""
pass
class PhaseSanitizer:
"""Sanitizes phase data from CSI signals for reliable processing."""
def __init__(self, config: Dict[str, Any], logger: Optional[logging.Logger] = None):
"""Initialize phase sanitizer.
Args:
config: Configuration dictionary
logger: Optional logger instance
Raises:
ValueError: If configuration is invalid
"""
self._validate_config(config)
self.config = config
self.logger = logger or logging.getLogger(__name__)
# Processing parameters
self.unwrapping_method = config['unwrapping_method']
self.outlier_threshold = config['outlier_threshold']
self.smoothing_window = config['smoothing_window']
# Optional parameters with defaults
self.enable_outlier_removal = config.get('enable_outlier_removal', True)
self.enable_smoothing = config.get('enable_smoothing', True)
self.enable_noise_filtering = config.get('enable_noise_filtering', False)
self.noise_threshold = config.get('noise_threshold', 0.05)
self.phase_range = config.get('phase_range', (-np.pi, np.pi))
# Statistics tracking
self._total_processed = 0
self._outliers_removed = 0
self._sanitization_errors = 0
def _validate_config(self, config: Dict[str, Any]) -> None:
"""Validate configuration parameters.
Args:
config: Configuration to validate
Raises:
ValueError: If configuration is invalid
"""
required_fields = ['unwrapping_method', 'outlier_threshold', 'smoothing_window']
missing_fields = [field for field in required_fields if field not in config]
if missing_fields:
raise ValueError(f"Missing required configuration: {missing_fields}")
# Validate unwrapping method
valid_methods = ['numpy', 'scipy', 'custom']
if config['unwrapping_method'] not in valid_methods:
raise ValueError(f"Invalid unwrapping method: {config['unwrapping_method']}. Must be one of {valid_methods}")
# Validate thresholds
if config['outlier_threshold'] <= 0:
raise ValueError("outlier_threshold must be positive")
if config['smoothing_window'] <= 0:
raise ValueError("smoothing_window must be positive")
def unwrap_phase(self, phase_data: np.ndarray) -> np.ndarray:
"""Unwrap phase data to remove discontinuities.
Args:
phase_data: Wrapped phase data (2D array)
Returns:
Unwrapped phase data
Raises:
PhaseSanitizationError: If unwrapping fails
"""
try:
if self.unwrapping_method == 'numpy':
return self._unwrap_numpy(phase_data)
elif self.unwrapping_method == 'scipy':
return self._unwrap_scipy(phase_data)
elif self.unwrapping_method == 'custom':
return self._unwrap_custom(phase_data)
else:
raise ValueError(f"Unknown unwrapping method: {self.unwrapping_method}")
except Exception as e:
raise PhaseSanitizationError(f"Failed to unwrap phase: {e}")
def _unwrap_numpy(self, phase_data: np.ndarray) -> np.ndarray:
"""Unwrap phase using numpy's unwrap function."""
if phase_data.size == 0:
raise ValueError("Cannot unwrap empty phase data")
return np.unwrap(phase_data, axis=1)
def _unwrap_scipy(self, phase_data: np.ndarray) -> np.ndarray:
"""Unwrap phase using scipy's unwrap function."""
if phase_data.size == 0:
raise ValueError("Cannot unwrap empty phase data")
return np.unwrap(phase_data, axis=1)
def _unwrap_custom(self, phase_data: np.ndarray) -> np.ndarray:
"""Unwrap phase using custom algorithm."""
if phase_data.size == 0:
raise ValueError("Cannot unwrap empty phase data")
# Simple custom unwrapping algorithm
unwrapped = phase_data.copy()
for i in range(phase_data.shape[0]):
unwrapped[i, :] = np.unwrap(phase_data[i, :])
return unwrapped
def remove_outliers(self, phase_data: np.ndarray) -> np.ndarray:
"""Remove outliers from phase data.
Args:
phase_data: Phase data (2D array)
Returns:
Phase data with outliers removed
Raises:
PhaseSanitizationError: If outlier removal fails
"""
if not self.enable_outlier_removal:
return phase_data
try:
# Detect outliers
outlier_mask = self._detect_outliers(phase_data)
# Interpolate outliers
clean_data = self._interpolate_outliers(phase_data, outlier_mask)
return clean_data
except Exception as e:
raise PhaseSanitizationError(f"Failed to remove outliers: {e}")
def _detect_outliers(self, phase_data: np.ndarray) -> np.ndarray:
"""Detect outliers using statistical methods."""
# Use Z-score method to detect outliers
z_scores = np.abs((phase_data - np.mean(phase_data, axis=1, keepdims=True)) /
(np.std(phase_data, axis=1, keepdims=True) + 1e-8))
outlier_mask = z_scores > self.outlier_threshold
# Update statistics
self._outliers_removed += np.sum(outlier_mask)
return outlier_mask
def _interpolate_outliers(self, phase_data: np.ndarray, outlier_mask: np.ndarray) -> np.ndarray:
"""Interpolate outlier values."""
clean_data = phase_data.copy()
for i in range(phase_data.shape[0]):
outliers = outlier_mask[i, :]
if np.any(outliers):
# Linear interpolation for outliers
valid_indices = np.where(~outliers)[0]
outlier_indices = np.where(outliers)[0]
if len(valid_indices) > 1:
clean_data[i, outlier_indices] = np.interp(
outlier_indices, valid_indices, phase_data[i, valid_indices]
)
return clean_data
def smooth_phase(self, phase_data: np.ndarray) -> np.ndarray:
"""Smooth phase data to reduce noise.
Args:
phase_data: Phase data (2D array)
Returns:
Smoothed phase data
Raises:
PhaseSanitizationError: If smoothing fails
"""
if not self.enable_smoothing:
return phase_data
try:
smoothed_data = self._apply_moving_average(phase_data, self.smoothing_window)
return smoothed_data
except Exception as e:
raise PhaseSanitizationError(f"Failed to smooth phase: {e}")
def _apply_moving_average(self, phase_data: np.ndarray, window_size: int) -> np.ndarray:
"""Apply moving average smoothing."""
smoothed_data = phase_data.copy()
# Ensure window size is odd
if window_size % 2 == 0:
window_size += 1
half_window = window_size // 2
for i in range(phase_data.shape[0]):
for j in range(half_window, phase_data.shape[1] - half_window):
start_idx = j - half_window
end_idx = j + half_window + 1
smoothed_data[i, j] = np.mean(phase_data[i, start_idx:end_idx])
return smoothed_data
def filter_noise(self, phase_data: np.ndarray) -> np.ndarray:
"""Filter noise from phase data.
Args:
phase_data: Phase data (2D array)
Returns:
Filtered phase data
Raises:
PhaseSanitizationError: If noise filtering fails
"""
if not self.enable_noise_filtering:
return phase_data
try:
filtered_data = self._apply_low_pass_filter(phase_data, self.noise_threshold)
return filtered_data
except Exception as e:
raise PhaseSanitizationError(f"Failed to filter noise: {e}")
def _apply_low_pass_filter(self, phase_data: np.ndarray, threshold: float) -> np.ndarray:
"""Apply low-pass filter to remove high-frequency noise."""
filtered_data = phase_data.copy()
# Check if data is large enough for filtering
min_filter_length = 18 # Minimum length required for filtfilt with order 4
if phase_data.shape[1] < min_filter_length:
# Skip filtering for small arrays
return filtered_data
# Apply Butterworth low-pass filter
nyquist = 0.5
cutoff = threshold * nyquist
# Design filter
b, a = signal.butter(4, cutoff, btype='low')
# Apply filter to each antenna
for i in range(phase_data.shape[0]):
filtered_data[i, :] = signal.filtfilt(b, a, phase_data[i, :])
return filtered_data
def sanitize_phase(self, phase_data: np.ndarray) -> np.ndarray:
"""Sanitize phase data through complete pipeline.
Args:
phase_data: Raw phase data (2D array)
Returns:
Sanitized phase data
Raises:
PhaseSanitizationError: If sanitization fails
"""
try:
self._total_processed += 1
# Validate input data
self.validate_phase_data(phase_data)
# Apply complete sanitization pipeline
sanitized_data = self.unwrap_phase(phase_data)
sanitized_data = self.remove_outliers(sanitized_data)
sanitized_data = self.smooth_phase(sanitized_data)
sanitized_data = self.filter_noise(sanitized_data)
return sanitized_data
except PhaseSanitizationError:
self._sanitization_errors += 1
raise
except Exception as e:
self._sanitization_errors += 1
raise PhaseSanitizationError(f"Sanitization pipeline failed: {e}")
def validate_phase_data(self, phase_data: np.ndarray) -> bool:
"""Validate phase data format and values.
Args:
phase_data: Phase data to validate
Returns:
True if valid
Raises:
PhaseSanitizationError: If validation fails
"""
# Check if data is 2D
if phase_data.ndim != 2:
raise PhaseSanitizationError("Phase data must be 2D array")
# Check if data is not empty
if phase_data.size == 0:
raise PhaseSanitizationError("Phase data cannot be empty")
# Check if values are within valid range
min_val, max_val = self.phase_range
if np.any(phase_data < min_val) or np.any(phase_data > max_val):
raise PhaseSanitizationError(f"Phase values outside valid range [{min_val}, {max_val}]")
return True
def get_sanitization_statistics(self) -> Dict[str, Any]:
"""Get sanitization statistics.
Returns:
Dictionary containing sanitization statistics
"""
outlier_rate = self._outliers_removed / self._total_processed if self._total_processed > 0 else 0
error_rate = self._sanitization_errors / self._total_processed if self._total_processed > 0 else 0
return {
'total_processed': self._total_processed,
'outliers_removed': self._outliers_removed,
'sanitization_errors': self._sanitization_errors,
'outlier_rate': outlier_rate,
'error_rate': error_rate
}
def reset_statistics(self) -> None:
"""Reset sanitization statistics."""
self._total_processed = 0
self._outliers_removed = 0
self._sanitization_errors = 0
archive/v1/src/core/router_interface.py
+294
View File
@@ -0,0 +1,294 @@
"""
Router interface for WiFi CSI data collection
"""
import logging
import asyncio
import time
from typing import Dict, List, Optional, Any
from datetime import datetime
import numpy as np
logger = logging.getLogger(__name__)
class RouterInterface:
"""Interface for connecting to WiFi routers and collecting CSI data."""
def __init__(
self,
router_id: str,
host: str,
port: int = 22,
username: str = "admin",
password: str = "",
interface: str = "wlan0",
mock_mode: bool = False
):
"""Initialize router interface.
Args:
router_id: Unique identifier for the router
host: Router IP address or hostname
port: SSH port for connection
username: SSH username
password: SSH password
interface: WiFi interface name
mock_mode: Whether to use mock data instead of real connection
"""
self.router_id = router_id
self.host = host
self.port = port
self.username = username
self.password = password
self.interface = interface
self.mock_mode = mock_mode
self.logger = logging.getLogger(f"{__name__}.{router_id}")
# Connection state
self.is_connected = False
self.connection = None
self.last_error = None
# Data collection state
self.last_data_time = None
self.error_count = 0
self.sample_count = 0
# Mock data generation (delegated to testing module)
self._mock_csi_generator = None
if mock_mode:
self._initialize_mock_generator()
def _initialize_mock_generator(self):
"""Initialize mock data generator from the testing module."""
from src.testing.mock_csi_generator import MockCSIGenerator
self._mock_csi_generator = MockCSIGenerator()
self._mock_csi_generator.show_banner()
async def connect(self):
"""Connect to the router."""
if self.mock_mode:
self.is_connected = True
self.logger.info(f"Mock connection established to router {self.router_id}")
return
try:
self.logger.info(f"Connecting to router {self.router_id} at {self.host}:{self.port}")
# In a real implementation, this would establish SSH connection
# For now, we'll simulate the connection
await asyncio.sleep(0.1) # Simulate connection delay
self.is_connected = True
self.error_count = 0
self.logger.info(f"Connected to router {self.router_id}")
except Exception as e:
self.last_error = str(e)
self.error_count += 1
self.logger.error(f"Failed to connect to router {self.router_id}: {e}")
raise
async def disconnect(self):
"""Disconnect from the router."""
try:
if self.connection:
# Close SSH connection
self.connection = None
self.is_connected = False
self.logger.info(f"Disconnected from router {self.router_id}")
except Exception as e:
self.logger.error(f"Error disconnecting from router {self.router_id}: {e}")
async def reconnect(self):
"""Reconnect to the router."""
await self.disconnect()
await asyncio.sleep(1) # Wait before reconnecting
await self.connect()
async def get_csi_data(self) -> Optional[np.ndarray]:
"""Get CSI data from the router.
Returns:
CSI data as numpy array, or None if no data available
"""
if not self.is_connected:
raise RuntimeError(f"Router {self.router_id} is not connected")
try:
if self.mock_mode:
csi_data = self._generate_mock_csi_data()
else:
csi_data = await self._collect_real_csi_data()
if csi_data is not None:
self.last_data_time = datetime.now()
self.sample_count += 1
self.error_count = 0
return csi_data
except Exception as e:
self.last_error = str(e)
self.error_count += 1
self.logger.error(f"Error getting CSI data from router {self.router_id}: {e}")
return None
def _generate_mock_csi_data(self) -> np.ndarray:
"""Generate mock CSI data for testing.
Delegates to the MockCSIGenerator in the testing module.
This method is only callable when mock_mode is True.
"""
if self._mock_csi_generator is None:
self._initialize_mock_generator()
return self._mock_csi_generator.generate()
async def _collect_real_csi_data(self) -> Optional[np.ndarray]:
"""Collect real CSI data from the router.
Raises:
RuntimeError: Always in the current state, because real CSI
data collection requires hardware setup that has not been
configured. This method must never silently return random
or placeholder data.
"""
raise RuntimeError(
f"Real CSI data collection from router '{self.router_id}' requires "
"hardware setup that is not configured. You must: "
"(1) install CSI-capable firmware (e.g., Atheros CSI Tool, Nexmon CSI) on the router, "
"(2) configure the SSH connection to the router, and "
"(3) implement the CSI extraction command for your specific firmware. "
"For development/testing, use mock_mode=True. "
"See docs/hardware-setup.md for complete setup instructions."
)
async def check_health(self) -> bool:
"""Check if the router connection is healthy.
Returns:
True if healthy, False otherwise
"""
if not self.is_connected:
return False
try:
# In mock mode, always healthy
if self.mock_mode:
return True
# For real connections, we could ping the router or check SSH connection
# For now, consider healthy if error count is low
return self.error_count < 5
except Exception as e:
self.logger.error(f"Error checking health of router {self.router_id}: {e}")
return False
async def get_status(self) -> Dict[str, Any]:
"""Get router status information.
Returns:
Dictionary containing router status
"""
return {
"router_id": self.router_id,
"connected": self.is_connected,
"mock_mode": self.mock_mode,
"last_data_time": self.last_data_time.isoformat() if self.last_data_time else None,
"error_count": self.error_count,
"sample_count": self.sample_count,
"last_error": self.last_error,
"configuration": {
"host": self.host,
"port": self.port,
"username": self.username,
"interface": self.interface
}
}
async def get_router_info(self) -> Dict[str, Any]:
"""Get router hardware information.
Returns:
Dictionary containing router information
"""
if self.mock_mode:
if self._mock_csi_generator is None:
self._initialize_mock_generator()
return self._mock_csi_generator.get_router_info()
# For real routers, this would query the actual hardware
return {
"model": "Unknown",
"firmware": "Unknown",
"wifi_standard": "Unknown",
"antennas": 1,
"supported_bands": ["Unknown"],
"csi_capabilities": {
"max_subcarriers": 64,
"max_antennas": 1,
"sampling_rate": 100
}
}
async def configure_csi_collection(self, config: Dict[str, Any]) -> bool:
"""Configure CSI data collection parameters.
Args:
config: Configuration dictionary
Returns:
True if configuration successful, False otherwise
"""
try:
if self.mock_mode:
if self._mock_csi_generator is None:
self._initialize_mock_generator()
self._mock_csi_generator.configure(config)
self.logger.info(f"Mock CSI collection configured for router {self.router_id}")
return True
# For real routers, this would send configuration commands
self.logger.warning("Real CSI configuration not implemented")
return False
except Exception as e:
self.logger.error(f"Error configuring CSI collection for router {self.router_id}: {e}")
return False
def get_metrics(self) -> Dict[str, Any]:
"""Get router interface metrics.
Returns:
Dictionary containing metrics
"""
uptime = 0
if self.last_data_time:
uptime = (datetime.now() - self.last_data_time).total_seconds()
success_rate = 0
if self.sample_count > 0:
success_rate = (self.sample_count - self.error_count) / self.sample_count
return {
"router_id": self.router_id,
"sample_count": self.sample_count,
"error_count": self.error_count,
"success_rate": success_rate,
"uptime_seconds": uptime,
"is_connected": self.is_connected,
"mock_mode": self.mock_mode
}
def reset_stats(self):
"""Reset statistics counters."""
self.error_count = 0
self.sample_count = 0
self.last_error = None
self.logger.info(f"Statistics reset for router {self.router_id}")
archive/v1/src/database/connection.py
+640
View File
@@ -0,0 +1,640 @@
"""
Database connection management for WiFi-DensePose API
"""
import asyncio
import logging
from typing import Optional, Dict, Any, AsyncGenerator
from contextlib import asynccontextmanager
from datetime import datetime
from sqlalchemy import create_engine, event, pool, text
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
from sqlalchemy.orm import sessionmaker, Session
from sqlalchemy.pool import QueuePool, NullPool
from sqlalchemy.exc import SQLAlchemyError, DisconnectionError
import redis.asyncio as redis
from redis.exceptions import ConnectionError as RedisConnectionError
from src.config.settings import Settings
from src.logger import get_logger
logger = get_logger(__name__)
class DatabaseConnectionError(Exception):
"""Database connection error."""
pass
class DatabaseManager:
"""Database connection manager."""
def __init__(self, settings: Settings):
self.settings = settings
self._async_engine = None
self._sync_engine = None
self._async_session_factory = None
self._sync_session_factory = None
self._redis_client = None
self._initialized = False
self._connection_pool_size = settings.db_pool_size
self._max_overflow = settings.db_max_overflow
self._pool_timeout = settings.db_pool_timeout
self._pool_recycle = settings.db_pool_recycle
async def initialize(self):
"""Initialize database connections."""
if self._initialized:
return
logger.info("Initializing database connections")
try:
# Initialize PostgreSQL connections
await self._initialize_postgresql()
# Initialize Redis connection
await self._initialize_redis()
self._initialized = True
logger.info("Database connections initialized successfully")
except Exception as e:
logger.error(f"Failed to initialize database connections: {e}")
raise DatabaseConnectionError(f"Database initialization failed: {e}")
async def _initialize_postgresql(self):
"""Initialize PostgreSQL connections with SQLite failsafe."""
postgresql_failed = False
try:
# Try PostgreSQL first
await self._initialize_postgresql_primary()
logger.info("PostgreSQL connections initialized")
return
except Exception as e:
postgresql_failed = True
logger.error(f"PostgreSQL initialization failed: {e}")
if not self.settings.enable_database_failsafe:
raise DatabaseConnectionError(f"PostgreSQL connection failed and failsafe disabled: {e}")
logger.warning("Falling back to SQLite database")
# Fallback to SQLite if PostgreSQL failed and failsafe is enabled
if postgresql_failed and self.settings.enable_database_failsafe:
await self._initialize_sqlite_fallback()
logger.info("SQLite fallback database initialized")
async def _initialize_postgresql_primary(self):
"""Initialize primary PostgreSQL connections."""
# Build database URL
if self.settings.database_url and "postgresql" in self.settings.database_url:
db_url = self.settings.database_url
async_db_url = self.settings.database_url.replace("postgresql://", "postgresql+asyncpg://")
elif self.settings.db_host and self.settings.db_name and self.settings.db_user:
db_url = (
f"postgresql://{self.settings.db_user}:{self.settings.db_password}"
f"@{self.settings.db_host}:{self.settings.db_port}/{self.settings.db_name}"
)
async_db_url = (
f"postgresql+asyncpg://{self.settings.db_user}:{self.settings.db_password}"
f"@{self.settings.db_host}:{self.settings.db_port}/{self.settings.db_name}"
)
else:
raise ValueError("PostgreSQL connection parameters not configured")
# Create async engine (don't specify poolclass for async engines)
self._async_engine = create_async_engine(
async_db_url,
pool_size=self._connection_pool_size,
max_overflow=self._max_overflow,
pool_timeout=self._pool_timeout,
pool_recycle=self._pool_recycle,
pool_pre_ping=True,
echo=self.settings.db_echo,
future=True,
)
# Create sync engine for migrations and admin tasks
self._sync_engine = create_engine(
db_url,
poolclass=QueuePool,
pool_size=max(2, self._connection_pool_size // 2),
max_overflow=self._max_overflow // 2,
pool_timeout=self._pool_timeout,
pool_recycle=self._pool_recycle,
pool_pre_ping=True,
echo=self.settings.db_echo,
future=True,
)
# Create session factories
self._async_session_factory = async_sessionmaker(
self._async_engine,
class_=AsyncSession,
expire_on_commit=False,
)
self._sync_session_factory = sessionmaker(
self._sync_engine,
expire_on_commit=False,
)
# Add connection event listeners
self._setup_connection_events()
# Test connections
await self._test_postgresql_connection()
async def _initialize_sqlite_fallback(self):
"""Initialize SQLite fallback database."""
import os
# Ensure directory exists
sqlite_path = self.settings.sqlite_fallback_path
os.makedirs(os.path.dirname(sqlite_path), exist_ok=True)
# Build SQLite URLs
db_url = f"sqlite:///{sqlite_path}"
async_db_url = f"sqlite+aiosqlite:///{sqlite_path}"
# Create async engine for SQLite
self._async_engine = create_async_engine(
async_db_url,
echo=self.settings.db_echo,
future=True,
)
# Create sync engine for SQLite
self._sync_engine = create_engine(
db_url,
poolclass=NullPool, # SQLite doesn't need connection pooling
echo=self.settings.db_echo,
future=True,
)
# Create session factories
self._async_session_factory = async_sessionmaker(
self._async_engine,
class_=AsyncSession,
expire_on_commit=False,
)
self._sync_session_factory = sessionmaker(
self._sync_engine,
expire_on_commit=False,
)
# Add connection event listeners
self._setup_connection_events()
# Test SQLite connection
await self._test_sqlite_connection()
async def _test_sqlite_connection(self):
"""Test SQLite connection."""
try:
async with self._async_engine.begin() as conn:
result = await conn.execute(text("SELECT 1"))
result.fetchone() # Don't await this - fetchone() is not async
logger.debug("SQLite connection test successful")
except Exception as e:
logger.error(f"SQLite connection test failed: {e}")
raise DatabaseConnectionError(f"SQLite connection test failed: {e}")
async def _initialize_redis(self):
"""Initialize Redis connection with failsafe."""
if not self.settings.redis_enabled:
logger.info("Redis disabled, skipping initialization")
return
try:
# Build Redis URL
if self.settings.redis_url:
redis_url = self.settings.redis_url
else:
redis_url = (
f"redis://{self.settings.redis_host}:{self.settings.redis_port}"
f"/{self.settings.redis_db}"
)
# Create Redis client
self._redis_client = redis.from_url(
redis_url,
password=self.settings.redis_password,
encoding="utf-8",
decode_responses=True,
max_connections=self.settings.redis_max_connections,
retry_on_timeout=True,
socket_timeout=self.settings.redis_socket_timeout,
socket_connect_timeout=self.settings.redis_connect_timeout,
)
# Test Redis connection
await self._test_redis_connection()
logger.info("Redis connection initialized")
except Exception as e:
logger.error(f"Failed to initialize Redis: {e}")
if self.settings.redis_required:
raise DatabaseConnectionError(f"Redis connection failed and is required: {e}")
elif self.settings.enable_redis_failsafe:
logger.warning("Redis initialization failed, continuing without Redis (failsafe enabled)")
self._redis_client = None
else:
logger.warning("Redis initialization failed but not required, continuing without Redis")
self._redis_client = None
def _setup_connection_events(self):
"""Setup database connection event listeners."""
@event.listens_for(self._sync_engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
"""Set database-specific settings on connection."""
if "sqlite" in str(self._sync_engine.url):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
@event.listens_for(self._sync_engine, "checkout")
def receive_checkout(dbapi_connection, connection_record, connection_proxy):
"""Log connection checkout."""
logger.debug("Database connection checked out")
@event.listens_for(self._sync_engine, "checkin")
def receive_checkin(dbapi_connection, connection_record):
"""Log connection checkin."""
logger.debug("Database connection checked in")
@event.listens_for(self._sync_engine, "invalidate")
def receive_invalidate(dbapi_connection, connection_record, exception):
"""Handle connection invalidation."""
logger.warning(f"Database connection invalidated: {exception}")
async def _test_postgresql_connection(self):
"""Test PostgreSQL connection."""
try:
async with self._async_engine.begin() as conn:
result = await conn.execute(text("SELECT 1"))
result.fetchone() # Don't await this - fetchone() is not async
logger.debug("PostgreSQL connection test successful")
except Exception as e:
logger.error(f"PostgreSQL connection test failed: {e}")
raise DatabaseConnectionError(f"PostgreSQL connection test failed: {e}")
async def _test_redis_connection(self):
"""Test Redis connection."""
if not self._redis_client:
return
try:
await self._redis_client.ping()
logger.debug("Redis connection test successful")
except Exception as e:
logger.error(f"Redis connection test failed: {e}")
if self.settings.redis_required:
raise DatabaseConnectionError(f"Redis connection test failed: {e}")
@asynccontextmanager
async def get_async_session(self) -> AsyncGenerator[AsyncSession, None]:
"""Get async database session."""
if not self._initialized:
await self.initialize()
if not self._async_session_factory:
raise DatabaseConnectionError("Async session factory not initialized")
session = self._async_session_factory()
try:
yield session
await session.commit()
except Exception as e:
await session.rollback()
logger.error(f"Database session error: {e}")
raise
finally:
await session.close()
@asynccontextmanager
async def get_sync_session(self) -> Session:
"""Get sync database session."""
if not self._initialized:
await self.initialize()
if not self._sync_session_factory:
raise DatabaseConnectionError("Sync session factory not initialized")
session = self._sync_session_factory()
try:
yield session
session.commit()
except Exception as e:
session.rollback()
logger.error(f"Database session error: {e}")
raise
finally:
session.close()
async def get_redis_client(self) -> Optional[redis.Redis]:
"""Get Redis client."""
if not self._initialized:
await self.initialize()
return self._redis_client
async def health_check(self) -> Dict[str, Any]:
"""Perform database health check."""
health_status = {
"database": {"status": "unknown", "details": {}},
"redis": {"status": "unknown", "details": {}},
"overall": "unknown"
}
# Check Database (PostgreSQL or SQLite)
try:
start_time = datetime.utcnow()
async with self.get_async_session() as session:
result = await session.execute(text("SELECT 1"))
result.fetchone() # Don't await this - fetchone() is not async
response_time = (datetime.utcnow() - start_time).total_seconds()
# Determine database type and status
is_sqlite = self.is_using_sqlite_fallback()
db_type = "sqlite_fallback" if is_sqlite else "postgresql"
details = {
"type": db_type,
"response_time_ms": round(response_time * 1000, 2),
}
# Add pool info for PostgreSQL
if not is_sqlite and hasattr(self._async_engine, 'pool'):
details.update({
"pool_size": self._async_engine.pool.size(),
"checked_out": self._async_engine.pool.checkedout(),
"overflow": self._async_engine.pool.overflow(),
})
# Add failsafe info
if is_sqlite:
details["failsafe_active"] = True
details["fallback_path"] = self.settings.sqlite_fallback_path
health_status["database"] = {
"status": "healthy",
"details": details
}
except Exception as e:
health_status["database"] = {
"status": "unhealthy",
"details": {"error": str(e)}
}
# Check Redis
if self._redis_client:
try:
start_time = datetime.utcnow()
await self._redis_client.ping()
response_time = (datetime.utcnow() - start_time).total_seconds()
info = await self._redis_client.info()
health_status["redis"] = {
"status": "healthy",
"details": {
"response_time_ms": round(response_time * 1000, 2),
"connected_clients": info.get("connected_clients", 0),
"used_memory": info.get("used_memory_human", "unknown"),
"uptime": info.get("uptime_in_seconds", 0),
}
}
except Exception as e:
health_status["redis"] = {
"status": "unhealthy",
"details": {"error": str(e)}
}
else:
health_status["redis"] = {
"status": "disabled",
"details": {"message": "Redis not enabled"}
}
# Determine overall status
database_healthy = health_status["database"]["status"] == "healthy"
redis_healthy = (
health_status["redis"]["status"] in ["healthy", "disabled"] or
not self.settings.redis_required
)
# Check if using failsafe modes
using_sqlite_fallback = self.is_using_sqlite_fallback()
redis_unavailable = not self.is_redis_available() and self.settings.redis_enabled
if database_healthy and redis_healthy:
if using_sqlite_fallback or redis_unavailable:
health_status["overall"] = "degraded" # Working but using failsafe
else:
health_status["overall"] = "healthy"
elif database_healthy:
health_status["overall"] = "degraded"
else:
health_status["overall"] = "unhealthy"
return health_status
async def get_connection_stats(self) -> Dict[str, Any]:
"""Get database connection statistics."""
stats = {
"postgresql": {},
"redis": {}
}
# PostgreSQL stats
if self._async_engine:
pool = self._async_engine.pool
stats["postgresql"] = {
"pool_size": pool.size(),
"checked_out": pool.checkedout(),
"overflow": pool.overflow(),
"checked_in": pool.checkedin(),
"total_connections": pool.size() + pool.overflow(),
"available_connections": pool.size() - pool.checkedout(),
}
# Redis stats
if self._redis_client:
try:
info = await self._redis_client.info()
stats["redis"] = {
"connected_clients": info.get("connected_clients", 0),
"blocked_clients": info.get("blocked_clients", 0),
"total_connections_received": info.get("total_connections_received", 0),
"rejected_connections": info.get("rejected_connections", 0),
}
except Exception as e:
stats["redis"] = {"error": str(e)}
return stats
async def close_connections(self):
"""Close all database connections."""
logger.info("Closing database connections")
# Close PostgreSQL connections
if self._async_engine:
await self._async_engine.dispose()
logger.debug("Async PostgreSQL engine disposed")
if self._sync_engine:
self._sync_engine.dispose()
logger.debug("Sync PostgreSQL engine disposed")
# Close Redis connection
if self._redis_client:
await self._redis_client.close()
logger.debug("Redis connection closed")
self._initialized = False
logger.info("Database connections closed")
def is_using_sqlite_fallback(self) -> bool:
"""Check if currently using SQLite fallback database."""
if not self._async_engine:
return False
return "sqlite" in str(self._async_engine.url)
def is_redis_available(self) -> bool:
"""Check if Redis is available."""
return self._redis_client is not None
async def test_connection(self) -> bool:
"""Test database connection for CLI validation."""
try:
if not self._initialized:
await self.initialize()
# Test database connection (PostgreSQL or SQLite)
async with self.get_async_session() as session:
result = await session.execute(text("SELECT 1"))
result.fetchone() # Don't await this - fetchone() is not async
# Test Redis connection if enabled
if self._redis_client:
await self._redis_client.ping()
return True
except Exception as e:
logger.error(f"Database connection test failed: {e}")
return False
async def reset_connections(self):
"""Reset all database connections."""
logger.info("Resetting database connections")
await self.close_connections()
await self.initialize()
logger.info("Database connections reset")
# Global database manager instance
_db_manager: Optional[DatabaseManager] = None
def get_database_manager(settings: Settings) -> DatabaseManager:
"""Get database manager instance."""
global _db_manager
if _db_manager is None:
_db_manager = DatabaseManager(settings)
return _db_manager
async def get_async_session(settings: Settings) -> AsyncGenerator[AsyncSession, None]:
"""Dependency to get async database session."""
db_manager = get_database_manager(settings)
async with db_manager.get_async_session() as session:
yield session
async def get_redis_client(settings: Settings) -> Optional[redis.Redis]:
"""Dependency to get Redis client."""
db_manager = get_database_manager(settings)
return await db_manager.get_redis_client()
class DatabaseHealthCheck:
"""Database health check utility."""
def __init__(self, db_manager: DatabaseManager):
self.db_manager = db_manager
async def check_postgresql(self) -> Dict[str, Any]:
"""Check PostgreSQL health."""
try:
start_time = datetime.utcnow()
async with self.db_manager.get_async_session() as session:
result = await session.execute(text("SELECT version()"))
version = result.fetchone()[0] # Don't await this - fetchone() is not async
response_time = (datetime.utcnow() - start_time).total_seconds()
return {
"status": "healthy",
"version": version,
"response_time_ms": round(response_time * 1000, 2),
}
except Exception as e:
return {
"status": "unhealthy",
"error": str(e),
}
async def check_redis(self) -> Dict[str, Any]:
"""Check Redis health."""
redis_client = await self.db_manager.get_redis_client()
if not redis_client:
return {
"status": "disabled",
"message": "Redis not configured"
}
try:
start_time = datetime.utcnow()
pong = await redis_client.ping()
response_time = (datetime.utcnow() - start_time).total_seconds()
info = await redis_client.info("server")
return {
"status": "healthy",
"ping": pong,
"version": info.get("redis_version", "unknown"),
"response_time_ms": round(response_time * 1000, 2),
}
except Exception as e:
return {
"status": "unhealthy",
"error": str(e),
}
async def full_health_check(self) -> Dict[str, Any]:
"""Perform full database health check."""
postgresql_health = await self.check_postgresql()
redis_health = await self.check_redis()
overall_status = "healthy"
if postgresql_health["status"] != "healthy":
overall_status = "unhealthy"
elif redis_health["status"] == "unhealthy":
overall_status = "degraded"
return {
"overall_status": overall_status,
"postgresql": postgresql_health,
"redis": redis_health,
"timestamp": datetime.utcnow().isoformat(),
}
archive/v1/src/database/migrations/001_initial.py
+413
View File
@@ -0,0 +1,413 @@
"""
Initial database migration for WiFi-DensePose API
Revision ID: 001_initial
Revises:
Create Date: 2025-01-07 07:58:00.000000
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers
revision = '001_initial'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
"""Create initial database schema."""
# Create devices table
op.create_table(
'devices',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('device_type', sa.String(length=50), nullable=False),
sa.Column('mac_address', sa.String(length=17), nullable=False),
sa.Column('ip_address', sa.String(length=45), nullable=True),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('firmware_version', sa.String(length=50), nullable=True),
sa.Column('hardware_version', sa.String(length=50), nullable=True),
sa.Column('location_name', sa.String(length=255), nullable=True),
sa.Column('room_id', sa.String(length=100), nullable=True),
sa.Column('coordinates_x', sa.Float(), nullable=True),
sa.Column('coordinates_y', sa.Float(), nullable=True),
sa.Column('coordinates_z', sa.Float(), nullable=True),
sa.Column('config', sa.JSON(), nullable=True),
sa.Column('capabilities', postgresql.ARRAY(sa.String()), nullable=True),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('tags', postgresql.ARRAY(sa.String()), nullable=True),
sa.CheckConstraint("status IN ('active', 'inactive', 'maintenance', 'error')", name='check_device_status'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('mac_address')
)
# Create indexes for devices table
op.create_index('idx_device_mac_address', 'devices', ['mac_address'])
op.create_index('idx_device_status', 'devices', ['status'])
op.create_index('idx_device_type', 'devices', ['device_type'])
# Create sessions table
op.create_table(
'sessions',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('started_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('ended_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('duration_seconds', sa.Integer(), nullable=True),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('config', sa.JSON(), nullable=True),
sa.Column('device_id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('tags', postgresql.ARRAY(sa.String()), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.Column('total_frames', sa.Integer(), nullable=False),
sa.Column('processed_frames', sa.Integer(), nullable=False),
sa.Column('error_count', sa.Integer(), nullable=False),
sa.CheckConstraint("status IN ('active', 'completed', 'failed', 'cancelled')", name='check_session_status'),
sa.CheckConstraint('total_frames >= 0', name='check_total_frames_positive'),
sa.CheckConstraint('processed_frames >= 0', name='check_processed_frames_positive'),
sa.CheckConstraint('error_count >= 0', name='check_error_count_positive'),
sa.ForeignKeyConstraint(['device_id'], ['devices.id'], ),
sa.PrimaryKeyConstraint('id')
)
# Create indexes for sessions table
op.create_index('idx_session_device_id', 'sessions', ['device_id'])
op.create_index('idx_session_status', 'sessions', ['status'])
op.create_index('idx_session_started_at', 'sessions', ['started_at'])
# Create csi_data table
op.create_table(
'csi_data',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('sequence_number', sa.Integer(), nullable=False),
sa.Column('timestamp_ns', sa.BigInteger(), nullable=False),
sa.Column('device_id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('session_id', postgresql.UUID(as_uuid=True), nullable=True),
sa.Column('amplitude', postgresql.ARRAY(sa.Float()), nullable=False),
sa.Column('phase', postgresql.ARRAY(sa.Float()), nullable=False),
sa.Column('frequency', sa.Float(), nullable=False),
sa.Column('bandwidth', sa.Float(), nullable=False),
sa.Column('rssi', sa.Float(), nullable=True),
sa.Column('snr', sa.Float(), nullable=True),
sa.Column('noise_floor', sa.Float(), nullable=True),
sa.Column('tx_antenna', sa.Integer(), nullable=True),
sa.Column('rx_antenna', sa.Integer(), nullable=True),
sa.Column('num_subcarriers', sa.Integer(), nullable=False),
sa.Column('processing_status', sa.String(length=20), nullable=False),
sa.Column('processed_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('quality_score', sa.Float(), nullable=True),
sa.Column('is_valid', sa.Boolean(), nullable=False),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.CheckConstraint('frequency > 0', name='check_frequency_positive'),
sa.CheckConstraint('bandwidth > 0', name='check_bandwidth_positive'),
sa.CheckConstraint('num_subcarriers > 0', name='check_subcarriers_positive'),
sa.CheckConstraint("processing_status IN ('pending', 'processing', 'completed', 'failed')", name='check_processing_status'),
sa.ForeignKeyConstraint(['device_id'], ['devices.id'], ),
sa.ForeignKeyConstraint(['session_id'], ['sessions.id'], ),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('device_id', 'sequence_number', 'timestamp_ns', name='uq_csi_device_seq_time')
)
# Create indexes for csi_data table
op.create_index('idx_csi_device_id', 'csi_data', ['device_id'])
op.create_index('idx_csi_session_id', 'csi_data', ['session_id'])
op.create_index('idx_csi_timestamp', 'csi_data', ['timestamp_ns'])
op.create_index('idx_csi_sequence', 'csi_data', ['sequence_number'])
op.create_index('idx_csi_processing_status', 'csi_data', ['processing_status'])
# Create pose_detections table
op.create_table(
'pose_detections',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('frame_number', sa.Integer(), nullable=False),
sa.Column('timestamp_ns', sa.BigInteger(), nullable=False),
sa.Column('session_id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('person_count', sa.Integer(), nullable=False),
sa.Column('keypoints', sa.JSON(), nullable=True),
sa.Column('bounding_boxes', sa.JSON(), nullable=True),
sa.Column('detection_confidence', sa.Float(), nullable=True),
sa.Column('pose_confidence', sa.Float(), nullable=True),
sa.Column('overall_confidence', sa.Float(), nullable=True),
sa.Column('processing_time_ms', sa.Float(), nullable=True),
sa.Column('model_version', sa.String(length=50), nullable=True),
sa.Column('algorithm', sa.String(length=100), nullable=True),
sa.Column('image_quality', sa.Float(), nullable=True),
sa.Column('pose_quality', sa.Float(), nullable=True),
sa.Column('is_valid', sa.Boolean(), nullable=False),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.CheckConstraint('person_count >= 0', name='check_person_count_positive'),
sa.CheckConstraint('detection_confidence >= 0 AND detection_confidence <= 1', name='check_detection_confidence_range'),
sa.CheckConstraint('pose_confidence >= 0 AND pose_confidence <= 1', name='check_pose_confidence_range'),
sa.CheckConstraint('overall_confidence >= 0 AND overall_confidence <= 1', name='check_overall_confidence_range'),
sa.ForeignKeyConstraint(['session_id'], ['sessions.id'], ),
sa.PrimaryKeyConstraint('id')
)
# Create indexes for pose_detections table
op.create_index('idx_pose_session_id', 'pose_detections', ['session_id'])
op.create_index('idx_pose_timestamp', 'pose_detections', ['timestamp_ns'])
op.create_index('idx_pose_frame', 'pose_detections', ['frame_number'])
op.create_index('idx_pose_person_count', 'pose_detections', ['person_count'])
# Create system_metrics table
op.create_table(
'system_metrics',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('metric_name', sa.String(length=255), nullable=False),
sa.Column('metric_type', sa.String(length=50), nullable=False),
sa.Column('value', sa.Float(), nullable=False),
sa.Column('unit', sa.String(length=50), nullable=True),
sa.Column('labels', sa.JSON(), nullable=True),
sa.Column('tags', postgresql.ARRAY(sa.String()), nullable=True),
sa.Column('source', sa.String(length=255), nullable=True),
sa.Column('component', sa.String(length=100), nullable=True),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.PrimaryKeyConstraint('id')
)
# Create indexes for system_metrics table
op.create_index('idx_metric_name', 'system_metrics', ['metric_name'])
op.create_index('idx_metric_type', 'system_metrics', ['metric_type'])
op.create_index('idx_metric_created_at', 'system_metrics', ['created_at'])
op.create_index('idx_metric_source', 'system_metrics', ['source'])
op.create_index('idx_metric_component', 'system_metrics', ['component'])
# Create audit_logs table
op.create_table(
'audit_logs',
sa.Column('id', postgresql.UUID(as_uuid=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('event_type', sa.String(length=100), nullable=False),
sa.Column('event_name', sa.String(length=255), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('user_id', sa.String(length=255), nullable=True),
sa.Column('session_id', sa.String(length=255), nullable=True),
sa.Column('ip_address', sa.String(length=45), nullable=True),
sa.Column('user_agent', sa.Text(), nullable=True),
sa.Column('resource_type', sa.String(length=100), nullable=True),
sa.Column('resource_id', sa.String(length=255), nullable=True),
sa.Column('before_state', sa.JSON(), nullable=True),
sa.Column('after_state', sa.JSON(), nullable=True),
sa.Column('changes', sa.JSON(), nullable=True),
sa.Column('success', sa.Boolean(), nullable=False),
sa.Column('error_message', sa.Text(), nullable=True),
sa.Column('metadata', sa.JSON(), nullable=True),
sa.Column('tags', postgresql.ARRAY(sa.String()), nullable=True),
sa.PrimaryKeyConstraint('id')
)
# Create indexes for audit_logs table
op.create_index('idx_audit_event_type', 'audit_logs', ['event_type'])
op.create_index('idx_audit_user_id', 'audit_logs', ['user_id'])
op.create_index('idx_audit_resource', 'audit_logs', ['resource_type', 'resource_id'])
op.create_index('idx_audit_created_at', 'audit_logs', ['created_at'])
op.create_index('idx_audit_success', 'audit_logs', ['success'])
# Create triggers for updated_at columns
op.execute("""
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = now();
RETURN NEW;
END;
$$ language 'plpgsql';
""")
# Add triggers to all tables with updated_at column
tables_with_updated_at = [
'devices', 'sessions', 'csi_data', 'pose_detections',
'system_metrics', 'audit_logs'
]
# Whitelist validation to prevent SQL injection
allowed_tables = set(tables_with_updated_at)
for table in tables_with_updated_at:
# Validate table name against whitelist
if table not in allowed_tables:
continue
# Use parameterized query with SQLAlchemy's text() and bindparam
# Note: For table names in DDL, we validate against whitelist
# SQLAlchemy's op.execute with text() is safe when table names are whitelisted
op.execute(
sa.text(f"""
CREATE TRIGGER update_{table}_updated_at
BEFORE UPDATE ON {table}
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
""")
)
# Insert initial data
_insert_initial_data()
def downgrade():
"""Drop all tables and functions."""
# Drop triggers first
tables_with_updated_at = [
'devices', 'sessions', 'csi_data', 'pose_detections',
'system_metrics', 'audit_logs'
]
# Whitelist validation to prevent SQL injection
allowed_tables = set(tables_with_updated_at)
for table in tables_with_updated_at:
# Validate table name against whitelist
if table not in allowed_tables:
continue
# Use parameterized query with SQLAlchemy's text()
op.execute(
sa.text(f"DROP TRIGGER IF EXISTS update_{table}_updated_at ON {table};")
)
# Drop function
op.execute("DROP FUNCTION IF EXISTS update_updated_at_column();")
# Drop tables in reverse order (respecting foreign key constraints)
op.drop_table('audit_logs')
op.drop_table('system_metrics')
op.drop_table('pose_detections')
op.drop_table('csi_data')
op.drop_table('sessions')
op.drop_table('devices')
def _insert_initial_data():
"""Insert initial data into tables."""
# Insert sample device
op.execute("""
INSERT INTO devices (
id, name, device_type, mac_address, ip_address, status,
firmware_version, hardware_version, location_name, room_id,
coordinates_x, coordinates_y, coordinates_z,
config, capabilities, description, tags
) VALUES (
gen_random_uuid(),
'Demo Router',
'router',
'00:11:22:33:44:55',
'192.168.1.1',
'active',
'1.0.0',
'v1.0',
'Living Room',
'room_001',
0.0,
0.0,
2.5,
'{"channel": 6, "power": 20, "bandwidth": 80}',
ARRAY['wifi6', 'csi', 'beamforming'],
'Demo WiFi router for testing',
ARRAY['demo', 'testing']
);
""")
# Insert sample session
op.execute("""
INSERT INTO sessions (
id, name, description, started_at, status, config,
device_id, tags, metadata, total_frames, processed_frames, error_count
) VALUES (
gen_random_uuid(),
'Demo Session',
'Initial demo session for testing',
now(),
'active',
'{"duration": 3600, "sampling_rate": 100}',
(SELECT id FROM devices WHERE name = 'Demo Router' LIMIT 1),
ARRAY['demo', 'initial'],
'{"purpose": "testing", "environment": "lab"}',
0,
0,
0
);
""")
# Insert initial system metrics
metrics_data = [
('system_startup', 'counter', 1.0, 'count', 'system', 'application'),
('database_connections', 'gauge', 0.0, 'count', 'database', 'postgresql'),
('api_requests_total', 'counter', 0.0, 'count', 'api', 'http'),
('memory_usage', 'gauge', 0.0, 'bytes', 'system', 'memory'),
('cpu_usage', 'gauge', 0.0, 'percent', 'system', 'cpu'),
]
for metric_name, metric_type, value, unit, source, component in metrics_data:
# Use parameterized query to prevent SQL injection
# Escape single quotes in string values
safe_metric_name = metric_name.replace("'", "''")
safe_metric_type = metric_type.replace("'", "''")
safe_unit = unit.replace("'", "''") if unit else ''
safe_source = source.replace("'", "''") if source else ''
safe_component = component.replace("'", "''") if component else ''
safe_description = f'Initial {safe_metric_name} metric'.replace("'", "''")
# Use SQLAlchemy's text() with proper escaping
op.execute(
sa.text(f"""
INSERT INTO system_metrics (
id, metric_name, metric_type, value, unit, source, component,
description, metadata
) VALUES (
gen_random_uuid(),
:metric_name,
:metric_type,
:value,
:unit,
:source,
:component,
:description,
:metadata
)
""").bindparams(
metric_name=safe_metric_name,
metric_type=safe_metric_type,
value=value,
unit=safe_unit,
source=safe_source,
component=safe_component,
description=safe_description,
metadata='{"initial": true, "version": "1.0.0"}'
)
)
# Insert initial audit log
op.execute("""
INSERT INTO audit_logs (
id, event_type, event_name, description, user_id, success,
resource_type, metadata
) VALUES (
gen_random_uuid(),
'system',
'database_migration',
'Initial database schema created',
'system',
true,
'database',
'{"migration": "001_initial", "version": "1.0.0"}'
);
""")
archive/v1/src/database/migrations/env.py
+109
View File
@@ -0,0 +1,109 @@
"""Alembic environment configuration for WiFi-DensePose API."""
import asyncio
import os
import sys
from logging.config import fileConfig
from pathlib import Path
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import async_engine_from_config
from alembic import context
# Add the project root to the Python path
project_root = Path(__file__).parent.parent.parent.parent
sys.path.insert(0, str(project_root))
# Import the models and settings
from src.database.models import Base
from src.config.settings import get_settings
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
target_metadata = Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def get_database_url():
"""Get the database URL from settings."""
try:
settings = get_settings()
return settings.get_database_url()
except Exception:
# Fallback to SQLite if settings can't be loaded
return "sqlite:///./data/wifi_densepose_fallback.db"
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = get_database_url()
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
"""Run migrations with a database connection."""
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
"""Run migrations in async mode."""
configuration = config.get_section(config.config_ini_section)
configuration["sqlalchemy.url"] = get_database_url()
connectable = async_engine_from_config(
configuration,
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode."""
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
archive/v1/src/database/migrations/script.py.mako
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
"""Upgrade database schema."""
${upgrades if upgrades else "pass"}
def downgrade() -> None:
"""Downgrade database schema."""
${downgrades if downgrades else "pass"}
archive/v1/src/database/model_types.py
+60
View File
@@ -0,0 +1,60 @@
"""
Database type compatibility helpers for WiFi-DensePose API
"""
from typing import Type, Any
from sqlalchemy import String, Text, JSON
from sqlalchemy.dialects.postgresql import ARRAY as PostgreSQL_ARRAY
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql import sqltypes
class ArrayType(sqltypes.TypeDecorator):
"""Array type that works with both PostgreSQL and SQLite."""
impl = Text
cache_ok = True
def __init__(self, item_type: Type = String):
super().__init__()
self.item_type = item_type
def load_dialect_impl(self, dialect):
"""Load dialect-specific implementation."""
if dialect.name == 'postgresql':
return dialect.type_descriptor(PostgreSQL_ARRAY(self.item_type))
else:
# For SQLite and others, use JSON
return dialect.type_descriptor(JSON)
def process_bind_param(self, value, dialect):
"""Process value before saving to database."""
if value is None:
return value
if dialect.name == 'postgresql':
return value
else:
# For SQLite, convert to JSON
return value if isinstance(value, (list, type(None))) else list(value)
def process_result_value(self, value, dialect):
"""Process value after loading from database."""
if value is None:
return value
if dialect.name == 'postgresql':
return value
else:
# For SQLite, value is already a list from JSON
return value if isinstance(value, list) else []
def get_array_type(item_type: Type = String) -> Type:
"""Get appropriate array type based on database."""
return ArrayType(item_type)
# Convenience types
StringArray = ArrayType(String)
FloatArray = ArrayType(sqltypes.Float)
archive/v1/src/database/models.py
+498
View File
@@ -0,0 +1,498 @@
"""
SQLAlchemy models for WiFi-DensePose API
"""
import uuid
from datetime import datetime
from typing import Optional, Dict, Any, List
from enum import Enum
from sqlalchemy import (
Column, String, Integer, Float, Boolean, DateTime, Text, JSON,
ForeignKey, Index, UniqueConstraint, CheckConstraint
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, validates
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.sql import func
# Import custom array type for compatibility
from src.database.model_types import StringArray, FloatArray
Base = declarative_base()
class TimestampMixin:
"""Mixin for timestamp fields."""
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now(), nullable=False)
class UUIDMixin:
"""Mixin for UUID primary key."""
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, nullable=False)
class DeviceStatus(str, Enum):
"""Device status enumeration."""
ACTIVE = "active"
INACTIVE = "inactive"
MAINTENANCE = "maintenance"
ERROR = "error"
class SessionStatus(str, Enum):
"""Session status enumeration."""
ACTIVE = "active"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
class ProcessingStatus(str, Enum):
"""Processing status enumeration."""
PENDING = "pending"
PROCESSING = "processing"
COMPLETED = "completed"
FAILED = "failed"
class Device(Base, UUIDMixin, TimestampMixin):
"""Device model for WiFi routers and sensors."""
__tablename__ = "devices"
# Basic device information
name = Column(String(255), nullable=False)
device_type = Column(String(50), nullable=False) # router, sensor, etc.
mac_address = Column(String(17), unique=True, nullable=False)
ip_address = Column(String(45), nullable=True) # IPv4 or IPv6
# Device status and configuration
status = Column(String(20), default=DeviceStatus.INACTIVE, nullable=False)
firmware_version = Column(String(50), nullable=True)
hardware_version = Column(String(50), nullable=True)
# Location information
location_name = Column(String(255), nullable=True)
room_id = Column(String(100), nullable=True)
coordinates_x = Column(Float, nullable=True)
coordinates_y = Column(Float, nullable=True)
coordinates_z = Column(Float, nullable=True)
# Configuration
config = Column(JSON, nullable=True)
capabilities = Column(StringArray, nullable=True)
# Metadata
description = Column(Text, nullable=True)
tags = Column(StringArray, nullable=True)
# Relationships
sessions = relationship("Session", back_populates="device", cascade="all, delete-orphan")
csi_data = relationship("CSIData", back_populates="device", cascade="all, delete-orphan")
# Constraints and indexes
__table_args__ = (
Index("idx_device_mac_address", "mac_address"),
Index("idx_device_status", "status"),
Index("idx_device_type", "device_type"),
CheckConstraint("status IN ('active', 'inactive', 'maintenance', 'error')", name="check_device_status"),
)
@validates('mac_address')
def validate_mac_address(self, key, address):
"""Validate MAC address format."""
if address and len(address) == 17:
# Basic MAC address format validation
parts = address.split(':')
if len(parts) == 6 and all(len(part) == 2 for part in parts):
return address.lower()
raise ValueError("Invalid MAC address format")
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"name": self.name,
"device_type": self.device_type,
"mac_address": self.mac_address,
"ip_address": self.ip_address,
"status": self.status,
"firmware_version": self.firmware_version,
"hardware_version": self.hardware_version,
"location_name": self.location_name,
"room_id": self.room_id,
"coordinates": {
"x": self.coordinates_x,
"y": self.coordinates_y,
"z": self.coordinates_z,
} if any([self.coordinates_x, self.coordinates_y, self.coordinates_z]) else None,
"config": self.config,
"capabilities": self.capabilities,
"description": self.description,
"tags": self.tags,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
class Session(Base, UUIDMixin, TimestampMixin):
"""Session model for tracking data collection sessions."""
__tablename__ = "sessions"
# Session identification
name = Column(String(255), nullable=False)
description = Column(Text, nullable=True)
# Session timing
started_at = Column(DateTime(timezone=True), nullable=True)
ended_at = Column(DateTime(timezone=True), nullable=True)
duration_seconds = Column(Integer, nullable=True)
# Session status and configuration
status = Column(String(20), default=SessionStatus.ACTIVE, nullable=False)
config = Column(JSON, nullable=True)
# Device relationship
device_id = Column(UUID(as_uuid=True), ForeignKey("devices.id"), nullable=False)
device = relationship("Device", back_populates="sessions")
# Data relationships
csi_data = relationship("CSIData", back_populates="session", cascade="all, delete-orphan")
pose_detections = relationship("PoseDetection", back_populates="session", cascade="all, delete-orphan")
# Metadata
tags = Column(StringArray, nullable=True)
meta_data = Column(JSON, nullable=True)
# Statistics
total_frames = Column(Integer, default=0, nullable=False)
processed_frames = Column(Integer, default=0, nullable=False)
error_count = Column(Integer, default=0, nullable=False)
# Constraints and indexes
__table_args__ = (
Index("idx_session_device_id", "device_id"),
Index("idx_session_status", "status"),
Index("idx_session_started_at", "started_at"),
CheckConstraint("status IN ('active', 'completed', 'failed', 'cancelled')", name="check_session_status"),
CheckConstraint("total_frames >= 0", name="check_total_frames_positive"),
CheckConstraint("processed_frames >= 0", name="check_processed_frames_positive"),
CheckConstraint("error_count >= 0", name="check_error_count_positive"),
)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"name": self.name,
"description": self.description,
"started_at": self.started_at.isoformat() if self.started_at else None,
"ended_at": self.ended_at.isoformat() if self.ended_at else None,
"duration_seconds": self.duration_seconds,
"status": self.status,
"config": self.config,
"device_id": str(self.device_id),
"tags": self.tags,
"metadata": self.meta_data,
"total_frames": self.total_frames,
"processed_frames": self.processed_frames,
"error_count": self.error_count,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
class CSIData(Base, UUIDMixin, TimestampMixin):
"""CSI (Channel State Information) data model."""
__tablename__ = "csi_data"
# Data identification
sequence_number = Column(Integer, nullable=False)
timestamp_ns = Column(Integer, nullable=False) # Nanosecond timestamp
# Device and session relationships
device_id = Column(UUID(as_uuid=True), ForeignKey("devices.id"), nullable=False)
session_id = Column(UUID(as_uuid=True), ForeignKey("sessions.id"), nullable=True)
device = relationship("Device", back_populates="csi_data")
session = relationship("Session", back_populates="csi_data")
# CSI data
amplitude = Column(FloatArray, nullable=False)
phase = Column(FloatArray, nullable=False)
frequency = Column(Float, nullable=False) # MHz
bandwidth = Column(Float, nullable=False) # MHz
# Signal characteristics
rssi = Column(Float, nullable=True) # dBm
snr = Column(Float, nullable=True) # dB
noise_floor = Column(Float, nullable=True) # dBm
# Antenna information
tx_antenna = Column(Integer, nullable=True)
rx_antenna = Column(Integer, nullable=True)
num_subcarriers = Column(Integer, nullable=False)
# Processing status
processing_status = Column(String(20), default=ProcessingStatus.PENDING, nullable=False)
processed_at = Column(DateTime(timezone=True), nullable=True)
# Quality metrics
quality_score = Column(Float, nullable=True)
is_valid = Column(Boolean, default=True, nullable=False)
# Metadata
meta_data = Column(JSON, nullable=True)
# Constraints and indexes
__table_args__ = (
Index("idx_csi_device_id", "device_id"),
Index("idx_csi_session_id", "session_id"),
Index("idx_csi_timestamp", "timestamp_ns"),
Index("idx_csi_sequence", "sequence_number"),
Index("idx_csi_processing_status", "processing_status"),
UniqueConstraint("device_id", "sequence_number", "timestamp_ns", name="uq_csi_device_seq_time"),
CheckConstraint("frequency > 0", name="check_frequency_positive"),
CheckConstraint("bandwidth > 0", name="check_bandwidth_positive"),
CheckConstraint("num_subcarriers > 0", name="check_subcarriers_positive"),
CheckConstraint("processing_status IN ('pending', 'processing', 'completed', 'failed')", name="check_processing_status"),
)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"sequence_number": self.sequence_number,
"timestamp_ns": self.timestamp_ns,
"device_id": str(self.device_id),
"session_id": str(self.session_id) if self.session_id else None,
"amplitude": self.amplitude,
"phase": self.phase,
"frequency": self.frequency,
"bandwidth": self.bandwidth,
"rssi": self.rssi,
"snr": self.snr,
"noise_floor": self.noise_floor,
"tx_antenna": self.tx_antenna,
"rx_antenna": self.rx_antenna,
"num_subcarriers": self.num_subcarriers,
"processing_status": self.processing_status,
"processed_at": self.processed_at.isoformat() if self.processed_at else None,
"quality_score": self.quality_score,
"is_valid": self.is_valid,
"metadata": self.meta_data,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
class PoseDetection(Base, UUIDMixin, TimestampMixin):
"""Pose detection results model."""
__tablename__ = "pose_detections"
# Detection identification
frame_number = Column(Integer, nullable=False)
timestamp_ns = Column(Integer, nullable=False)
# Session relationship
session_id = Column(UUID(as_uuid=True), ForeignKey("sessions.id"), nullable=False)
session = relationship("Session", back_populates="pose_detections")
# Detection results
person_count = Column(Integer, default=0, nullable=False)
keypoints = Column(JSON, nullable=True) # Array of person keypoints
bounding_boxes = Column(JSON, nullable=True) # Array of bounding boxes
# Confidence scores
detection_confidence = Column(Float, nullable=True)
pose_confidence = Column(Float, nullable=True)
overall_confidence = Column(Float, nullable=True)
# Processing information
processing_time_ms = Column(Float, nullable=True)
model_version = Column(String(50), nullable=True)
algorithm = Column(String(100), nullable=True)
# Quality metrics
image_quality = Column(Float, nullable=True)
pose_quality = Column(Float, nullable=True)
is_valid = Column(Boolean, default=True, nullable=False)
# Metadata
meta_data = Column(JSON, nullable=True)
# Constraints and indexes
__table_args__ = (
Index("idx_pose_session_id", "session_id"),
Index("idx_pose_timestamp", "timestamp_ns"),
Index("idx_pose_frame", "frame_number"),
Index("idx_pose_person_count", "person_count"),
CheckConstraint("person_count >= 0", name="check_person_count_positive"),
CheckConstraint("detection_confidence >= 0 AND detection_confidence <= 1", name="check_detection_confidence_range"),
CheckConstraint("pose_confidence >= 0 AND pose_confidence <= 1", name="check_pose_confidence_range"),
CheckConstraint("overall_confidence >= 0 AND overall_confidence <= 1", name="check_overall_confidence_range"),
)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"frame_number": self.frame_number,
"timestamp_ns": self.timestamp_ns,
"session_id": str(self.session_id),
"person_count": self.person_count,
"keypoints": self.keypoints,
"bounding_boxes": self.bounding_boxes,
"detection_confidence": self.detection_confidence,
"pose_confidence": self.pose_confidence,
"overall_confidence": self.overall_confidence,
"processing_time_ms": self.processing_time_ms,
"model_version": self.model_version,
"algorithm": self.algorithm,
"image_quality": self.image_quality,
"pose_quality": self.pose_quality,
"is_valid": self.is_valid,
"metadata": self.meta_data,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
class SystemMetric(Base, UUIDMixin, TimestampMixin):
"""System metrics model for monitoring."""
__tablename__ = "system_metrics"
# Metric identification
metric_name = Column(String(255), nullable=False)
metric_type = Column(String(50), nullable=False) # counter, gauge, histogram
# Metric value
value = Column(Float, nullable=False)
unit = Column(String(50), nullable=True)
# Labels and tags
labels = Column(JSON, nullable=True)
tags = Column(StringArray, nullable=True)
# Source information
source = Column(String(255), nullable=True)
component = Column(String(100), nullable=True)
# Metadata
description = Column(Text, nullable=True)
meta_data = Column(JSON, nullable=True)
# Constraints and indexes
__table_args__ = (
Index("idx_metric_name", "metric_name"),
Index("idx_metric_type", "metric_type"),
Index("idx_metric_created_at", "created_at"),
Index("idx_metric_source", "source"),
Index("idx_metric_component", "component"),
)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"metric_name": self.metric_name,
"metric_type": self.metric_type,
"value": self.value,
"unit": self.unit,
"labels": self.labels,
"tags": self.tags,
"source": self.source,
"component": self.component,
"description": self.description,
"metadata": self.meta_data,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
class AuditLog(Base, UUIDMixin, TimestampMixin):
"""Audit log model for tracking system events."""
__tablename__ = "audit_logs"
# Event information
event_type = Column(String(100), nullable=False)
event_name = Column(String(255), nullable=False)
description = Column(Text, nullable=True)
# User and session information
user_id = Column(String(255), nullable=True)
session_id = Column(String(255), nullable=True)
ip_address = Column(String(45), nullable=True)
user_agent = Column(Text, nullable=True)
# Resource information
resource_type = Column(String(100), nullable=True)
resource_id = Column(String(255), nullable=True)
# Event details
before_state = Column(JSON, nullable=True)
after_state = Column(JSON, nullable=True)
changes = Column(JSON, nullable=True)
# Result information
success = Column(Boolean, nullable=False)
error_message = Column(Text, nullable=True)
# Metadata
meta_data = Column(JSON, nullable=True)
tags = Column(StringArray, nullable=True)
# Constraints and indexes
__table_args__ = (
Index("idx_audit_event_type", "event_type"),
Index("idx_audit_user_id", "user_id"),
Index("idx_audit_resource", "resource_type", "resource_id"),
Index("idx_audit_created_at", "created_at"),
Index("idx_audit_success", "success"),
)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"id": str(self.id),
"event_type": self.event_type,
"event_name": self.event_name,
"description": self.description,
"user_id": self.user_id,
"session_id": self.session_id,
"ip_address": self.ip_address,
"user_agent": self.user_agent,
"resource_type": self.resource_type,
"resource_id": self.resource_id,
"before_state": self.before_state,
"after_state": self.after_state,
"changes": self.changes,
"success": self.success,
"error_message": self.error_message,
"metadata": self.meta_data,
"tags": self.tags,
"created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None,
}
# Model registry for easy access
MODEL_REGISTRY = {
"Device": Device,
"Session": Session,
"CSIData": CSIData,
"PoseDetection": PoseDetection,
"SystemMetric": SystemMetric,
"AuditLog": AuditLog,
}
def get_model_by_name(name: str):
"""Get model class by name."""
return MODEL_REGISTRY.get(name)
def get_all_models() -> List:
"""Get all model classes."""
return list(MODEL_REGISTRY.values())
archive/v1/src/hardware/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Hardware abstraction layer for WiFi-DensePose system."""
archive/v1/src/hardware/csi_extractor.py
+516
View File
@@ -0,0 +1,516 @@
"""CSI data extraction from WiFi hardware using Test-Driven Development approach."""
import asyncio
import struct
import numpy as np
from datetime import datetime, timezone
from typing import Dict, Any, Optional, Callable, Protocol
from dataclasses import dataclass
import logging
class CSIParseError(Exception):
"""Exception raised for CSI parsing errors."""
pass
class CSIValidationError(Exception):
"""Exception raised for CSI validation errors."""
pass
class CSIExtractionError(Exception):
"""Exception raised when CSI data extraction fails.
This error is raised instead of silently returning random/placeholder data.
Callers should handle this to inform users that real hardware data is required.
"""
pass
@dataclass
class CSIData:
"""Data structure for CSI measurements."""
timestamp: datetime
amplitude: np.ndarray
phase: np.ndarray
frequency: float
bandwidth: float
num_subcarriers: int
num_antennas: int
snr: float
metadata: Dict[str, Any]
class CSIParser(Protocol):
"""Protocol for CSI data parsers."""
def parse(self, raw_data: bytes) -> CSIData:
"""Parse raw CSI data into structured format."""
...
class ESP32CSIParser:
"""Parser for ESP32 CSI data format."""
def parse(self, raw_data: bytes) -> CSIData:
"""Parse ESP32 CSI data format.
Args:
raw_data: Raw bytes from ESP32
Returns:
Parsed CSI data
Raises:
CSIParseError: If data format is invalid
"""
if not raw_data:
raise CSIParseError("Empty data received")
try:
data_str = raw_data.decode('utf-8')
if not data_str.startswith('CSI_DATA:'):
raise CSIParseError("Invalid ESP32 CSI data format")
# Parse ESP32 format: CSI_DATA:timestamp,antennas,subcarriers,freq,bw,snr,[amp],[phase]
parts = data_str[9:].split(',') # Remove 'CSI_DATA:' prefix
timestamp_ms = int(parts[0])
num_antennas = int(parts[1])
num_subcarriers = int(parts[2])
frequency_mhz = float(parts[3])
bandwidth_mhz = float(parts[4])
snr = float(parts[5])
# Convert to proper units
frequency = frequency_mhz * 1e6 # MHz to Hz
bandwidth = bandwidth_mhz * 1e6 # MHz to Hz
# Parse amplitude and phase arrays from the remaining CSV fields.
# Expected format after the header fields: comma-separated float values
# representing interleaved amplitude and phase per antenna per subcarrier.
data_values = parts[6:]
expected_values = num_antennas * num_subcarriers * 2 # amplitude + phase
if len(data_values) < expected_values:
raise CSIExtractionError(
f"ESP32 CSI data incomplete: expected {expected_values} values "
f"(amplitude + phase for {num_antennas} antennas x {num_subcarriers} subcarriers), "
f"but received {len(data_values)} values. "
"Ensure the ESP32 firmware is configured to output full CSI matrix data. "
"See docs/hardware-setup.md for ESP32 CSI configuration."
)
try:
float_values = [float(v) for v in data_values[:expected_values]]
except ValueError as ve:
raise CSIExtractionError(
f"ESP32 CSI data contains non-numeric values: {ve}. "
"Raw CSI fields must be numeric float values."
)
all_values = np.array(float_values)
amplitude = all_values[:num_antennas * num_subcarriers].reshape(num_antennas, num_subcarriers)
phase = all_values[num_antennas * num_subcarriers:].reshape(num_antennas, num_subcarriers)
return CSIData(
timestamp=datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc),
amplitude=amplitude,
phase=phase,
frequency=frequency,
bandwidth=bandwidth,
num_subcarriers=num_subcarriers,
num_antennas=num_antennas,
snr=snr,
metadata={'source': 'esp32', 'raw_length': len(raw_data)}
)
except (ValueError, IndexError) as e:
raise CSIParseError(f"Failed to parse ESP32 data: {e}")
class ESP32BinaryParser:
"""Parser for ADR-018 binary CSI frames from ESP32 nodes.
Binary frame format:
Offset Size Field
0 4 Magic: 0xC5110001 (LE)
4 1 Node ID
5 1 Number of antennas
6 2 Number of subcarriers (LE u16)
8 4 Frequency MHz (LE u32)
12 4 Sequence number (LE u32)
16 1 RSSI (i8)
17 1 Noise floor (i8)
18 2 Reserved
20 N*2 I/Q pairs (n_antennas * n_subcarriers * 2 bytes, signed i8)
"""
MAGIC = 0xC5110001
HEADER_SIZE = 20
HEADER_FMT = '<IBBHIIBB2x' # magic, node_id, n_ant, n_sc, freq, seq, rssi, noise
def parse(self, raw_data: bytes) -> CSIData:
"""Parse an ADR-018 binary frame into CSIData.
Args:
raw_data: Raw binary frame bytes.
Returns:
Parsed CSI data with amplitude/phase arrays shaped (n_antennas, n_subcarriers).
Raises:
CSIParseError: If frame is too short, has invalid magic, or malformed I/Q data.
"""
if len(raw_data) < self.HEADER_SIZE:
raise CSIParseError(
f"Frame too short: need {self.HEADER_SIZE} bytes, got {len(raw_data)}"
)
magic, node_id, n_antennas, n_subcarriers, freq_mhz, sequence, rssi_u8, noise_u8 = \
struct.unpack_from(self.HEADER_FMT, raw_data, 0)
if magic != self.MAGIC:
raise CSIParseError(
f"Invalid magic: expected 0x{self.MAGIC:08X}, got 0x{magic:08X}"
)
# Convert unsigned bytes to signed i8
rssi = rssi_u8 if rssi_u8 < 128 else rssi_u8 - 256
noise_floor = noise_u8 if noise_u8 < 128 else noise_u8 - 256
iq_count = n_antennas * n_subcarriers
iq_bytes = iq_count * 2
expected_len = self.HEADER_SIZE + iq_bytes
if len(raw_data) < expected_len:
raise CSIParseError(
f"Frame too short for I/Q data: need {expected_len} bytes, got {len(raw_data)}"
)
# Parse I/Q pairs as signed bytes
iq_raw = struct.unpack_from(f'<{iq_count * 2}b', raw_data, self.HEADER_SIZE)
i_vals = np.array(iq_raw[0::2], dtype=np.float64).reshape(n_antennas, n_subcarriers)
q_vals = np.array(iq_raw[1::2], dtype=np.float64).reshape(n_antennas, n_subcarriers)
amplitude = np.sqrt(i_vals ** 2 + q_vals ** 2)
phase = np.arctan2(q_vals, i_vals)
snr = float(rssi - noise_floor)
frequency = float(freq_mhz) * 1e6
bandwidth = 20e6 # default; could infer from n_subcarriers
if n_subcarriers <= 56:
bandwidth = 20e6
elif n_subcarriers <= 114:
bandwidth = 40e6
elif n_subcarriers <= 242:
bandwidth = 80e6
else:
bandwidth = 160e6
return CSIData(
timestamp=datetime.now(tz=timezone.utc),
amplitude=amplitude,
phase=phase,
frequency=frequency,
bandwidth=bandwidth,
num_subcarriers=n_subcarriers,
num_antennas=n_antennas,
snr=snr,
metadata={
'source': 'esp32_binary',
'node_id': node_id,
'sequence': sequence,
'rssi_dbm': rssi,
'noise_floor_dbm': noise_floor,
'channel_freq_mhz': freq_mhz,
}
)
class RouterCSIParser:
"""Parser for router CSI data format."""
def parse(self, raw_data: bytes) -> CSIData:
"""Parse router CSI data format.
Args:
raw_data: Raw bytes from router
Returns:
Parsed CSI data
Raises:
CSIParseError: If data format is invalid
"""
if not raw_data:
raise CSIParseError("Empty data received")
# Handle different router formats
data_str = raw_data.decode('utf-8')
if data_str.startswith('ATHEROS_CSI:'):
return self._parse_atheros_format(raw_data)
else:
raise CSIParseError("Unknown router CSI format")
def _parse_atheros_format(self, raw_data: bytes) -> CSIData:
"""Parse Atheros CSI format.
Raises:
CSIExtractionError: Always, because Atheros CSI parsing requires
the Atheros CSI Tool binary format parser which has not been
implemented yet. Use the ESP32 parser or contribute an
Atheros implementation.
"""
raise CSIExtractionError(
"Atheros CSI format parsing is not yet implemented. "
"The Atheros CSI Tool outputs a binary format that requires a dedicated parser. "
"To collect real CSI data from Atheros-based routers, you must implement "
"the binary format parser following the Atheros CSI Tool specification. "
"See docs/hardware-setup.md for supported hardware and data formats."
)
class CSIExtractor:
"""Main CSI data extractor supporting multiple hardware types."""
def __init__(self, config: Dict[str, Any], logger: Optional[logging.Logger] = None):
"""Initialize CSI extractor.
Args:
config: Configuration dictionary
logger: Optional logger instance
Raises:
ValueError: If configuration is invalid
"""
self._validate_config(config)
self.config = config
self.logger = logger or logging.getLogger(__name__)
self.hardware_type = config['hardware_type']
self.sampling_rate = config['sampling_rate']
self.buffer_size = config['buffer_size']
self.timeout = config['timeout']
self.validation_enabled = config.get('validation_enabled', True)
self.retry_attempts = config.get('retry_attempts', 3)
# State management
self.is_connected = False
self.is_streaming = False
# Create appropriate parser
if self.hardware_type == 'esp32':
if config.get('parser_format') == 'binary':
self.parser = ESP32BinaryParser()
else:
self.parser = ESP32CSIParser()
elif self.hardware_type == 'router':
self.parser = RouterCSIParser()
else:
raise ValueError(f"Unsupported hardware type: {self.hardware_type}")
def _validate_config(self, config: Dict[str, Any]) -> None:
"""Validate configuration parameters.
Args:
config: Configuration to validate
Raises:
ValueError: If configuration is invalid
"""
required_fields = ['hardware_type', 'sampling_rate', 'buffer_size', 'timeout']
missing_fields = [field for field in required_fields if field not in config]
if missing_fields:
raise ValueError(f"Missing required configuration: {missing_fields}")
if config['sampling_rate'] <= 0:
raise ValueError("sampling_rate must be positive")
if config['buffer_size'] <= 0:
raise ValueError("buffer_size must be positive")
if config['timeout'] <= 0:
raise ValueError("timeout must be positive")
async def connect(self) -> bool:
"""Establish connection to CSI hardware.
Returns:
True if connection successful, False otherwise
"""
try:
success = await self._establish_hardware_connection()
self.is_connected = success
return success
except Exception as e:
self.logger.error(f"Failed to connect to hardware: {e}")
self.is_connected = False
return False
async def disconnect(self) -> None:
"""Disconnect from CSI hardware."""
if self.is_connected:
await self._close_hardware_connection()
self.is_connected = False
async def extract_csi(self) -> CSIData:
"""Extract CSI data from hardware.
Returns:
Extracted CSI data
Raises:
CSIParseError: If not connected or extraction fails
"""
if not self.is_connected:
raise CSIParseError("Not connected to hardware")
# Retry mechanism for temporary failures
for attempt in range(self.retry_attempts):
try:
raw_data = await self._read_raw_data()
csi_data = self.parser.parse(raw_data)
if self.validation_enabled:
self.validate_csi_data(csi_data)
return csi_data
except ConnectionError as e:
if attempt < self.retry_attempts - 1:
self.logger.warning(f"Extraction attempt {attempt + 1} failed, retrying: {e}")
await asyncio.sleep(0.1) # Brief delay before retry
else:
raise CSIParseError(f"Extraction failed after {self.retry_attempts} attempts: {e}")
def validate_csi_data(self, csi_data: CSIData) -> bool:
"""Validate CSI data structure and values.
Args:
csi_data: CSI data to validate
Returns:
True if valid
Raises:
CSIValidationError: If data is invalid
"""
if csi_data.amplitude.size == 0:
raise CSIValidationError("Empty amplitude data")
if csi_data.phase.size == 0:
raise CSIValidationError("Empty phase data")
if csi_data.frequency <= 0:
raise CSIValidationError("Invalid frequency")
if csi_data.bandwidth <= 0:
raise CSIValidationError("Invalid bandwidth")
if csi_data.num_subcarriers <= 0:
raise CSIValidationError("Invalid number of subcarriers")
if csi_data.num_antennas <= 0:
raise CSIValidationError("Invalid number of antennas")
if csi_data.snr < -50 or csi_data.snr > 50: # Reasonable SNR range
raise CSIValidationError("Invalid SNR value")
return True
async def start_streaming(self, callback: Callable[[CSIData], None]) -> None:
"""Start streaming CSI data.
Args:
callback: Function to call with each CSI sample
"""
self.is_streaming = True
try:
while self.is_streaming:
csi_data = await self.extract_csi()
callback(csi_data)
await asyncio.sleep(1.0 / self.sampling_rate)
except Exception as e:
self.logger.error(f"Streaming error: {e}")
finally:
self.is_streaming = False
def stop_streaming(self) -> None:
"""Stop streaming CSI data."""
self.is_streaming = False
async def _establish_hardware_connection(self) -> bool:
"""Establish connection to hardware (to be implemented by subclasses)."""
# Placeholder implementation for testing
return True
async def _close_hardware_connection(self) -> None:
"""Close hardware connection (to be implemented by subclasses)."""
# Placeholder implementation for testing
pass
async def _read_raw_data(self) -> bytes:
"""Read raw data from hardware.
When parser_format='binary', reads from the configured UDP socket.
Otherwise returns placeholder text data for legacy compatibility.
Raises:
CSIExtractionError: If UDP read times out or fails.
"""
if self.config.get('parser_format') == 'binary':
return await self._read_udp_data()
# Placeholder implementation for legacy text-mode testing
return b"CSI_DATA:1234567890,3,56,2400,20,15.5,[1.0,2.0,3.0],[0.5,1.5,2.5]"
async def _read_udp_data(self) -> bytes:
"""Read a single UDP packet from the aggregator.
Raises:
CSIExtractionError: If read times out or connection fails.
"""
host = self.config.get('aggregator_host', '0.0.0.0')
port = self.config.get('aggregator_port', 5005)
loop = asyncio.get_event_loop()
# Create UDP endpoint if not already cached
if not hasattr(self, '_udp_transport'):
self._udp_future: asyncio.Future = loop.create_future()
class _UdpProtocol(asyncio.DatagramProtocol):
def __init__(self, future):
self._future = future
def datagram_received(self, data, addr):
if not self._future.done():
self._future.set_result(data)
def error_received(self, exc):
if not self._future.done():
self._future.set_exception(exc)
transport, protocol = await loop.create_datagram_endpoint(
lambda: _UdpProtocol(self._udp_future),
local_addr=(host, port),
)
self._udp_transport = transport
self._udp_protocol = protocol
try:
data = await asyncio.wait_for(self._udp_future, timeout=self.timeout)
# Reset future for next read
self._udp_future = loop.create_future()
self._udp_protocol._future = self._udp_future
return data
except asyncio.TimeoutError:
raise CSIExtractionError(
f"UDP read timed out after {self.timeout}s. "
f"Ensure the aggregator is running and sending to {host}:{port}."
)
archive/v1/src/hardware/router_interface.py
+241
View File
@@ -0,0 +1,241 @@
"""Router interface for WiFi-DensePose system using TDD approach."""
import asyncio
import logging
from typing import Dict, Any, Optional
import asyncssh
from datetime import datetime, timezone
import numpy as np
try:
from .csi_extractor import CSIData
except ImportError:
# Handle import for testing
from src.hardware.csi_extractor import CSIData
class RouterConnectionError(Exception):
"""Exception raised for router connection errors."""
pass
class RouterInterface:
"""Interface for communicating with WiFi routers via SSH."""
def __init__(self, config: Dict[str, Any], logger: Optional[logging.Logger] = None):
"""Initialize router interface.
Args:
config: Configuration dictionary with connection parameters
logger: Optional logger instance
Raises:
ValueError: If configuration is invalid
"""
self._validate_config(config)
self.config = config
self.logger = logger or logging.getLogger(__name__)
# Connection parameters
self.host = config['host']
self.port = config['port']
self.username = config['username']
self.password = config['password']
self.command_timeout = config.get('command_timeout', 30)
self.connection_timeout = config.get('connection_timeout', 10)
self.max_retries = config.get('max_retries', 3)
self.retry_delay = config.get('retry_delay', 1.0)
# Connection state
self.is_connected = False
self.ssh_client = None
def _validate_config(self, config: Dict[str, Any]) -> None:
"""Validate configuration parameters.
Args:
config: Configuration to validate
Raises:
ValueError: If configuration is invalid
"""
required_fields = ['host', 'port', 'username', 'password']
missing_fields = [field for field in required_fields if field not in config]
if missing_fields:
raise ValueError(f"Missing required configuration: {missing_fields}")
if not isinstance(config['port'], int) or config['port'] <= 0:
raise ValueError("Port must be a positive integer")
async def connect(self) -> bool:
"""Establish SSH connection to router.
Returns:
True if connection successful, False otherwise
"""
try:
self.ssh_client = await asyncssh.connect(
self.host,
port=self.port,
username=self.username,
password=self.password,
connect_timeout=self.connection_timeout
)
self.is_connected = True
self.logger.info(f"Connected to router at {self.host}:{self.port}")
return True
except Exception as e:
self.logger.error(f"Failed to connect to router: {e}")
self.is_connected = False
self.ssh_client = None
return False
async def disconnect(self) -> None:
"""Disconnect from router."""
if self.is_connected and self.ssh_client:
self.ssh_client.close()
self.is_connected = False
self.ssh_client = None
self.logger.info("Disconnected from router")
async def execute_command(self, command: str) -> str:
"""Execute command on router via SSH.
Args:
command: Command to execute
Returns:
Command output
Raises:
RouterConnectionError: If not connected or command fails
"""
if not self.is_connected:
raise RouterConnectionError("Not connected to router")
# Retry mechanism for temporary failures
for attempt in range(self.max_retries):
try:
result = await self.ssh_client.run(command, timeout=self.command_timeout)
if result.returncode != 0:
raise RouterConnectionError(f"Command failed: {result.stderr}")
return result.stdout
except ConnectionError as e:
if attempt < self.max_retries - 1:
self.logger.warning(f"Command attempt {attempt + 1} failed, retrying: {e}")
await asyncio.sleep(self.retry_delay)
else:
raise RouterConnectionError(f"Command execution failed after {self.max_retries} retries: {e}")
except Exception as e:
raise RouterConnectionError(f"Command execution error: {e}")
async def get_csi_data(self) -> CSIData:
"""Retrieve CSI data from router.
Returns:
CSI data structure
Raises:
RouterConnectionError: If data retrieval fails
"""
try:
response = await self.execute_command("iwlist scan | grep CSI")
return self._parse_csi_response(response)
except Exception as e:
raise RouterConnectionError(f"Failed to retrieve CSI data: {e}")
async def get_router_status(self) -> Dict[str, Any]:
"""Get router system status.
Returns:
Dictionary containing router status information
Raises:
RouterConnectionError: If status retrieval fails
"""
try:
response = await self.execute_command("cat /proc/stat && free && iwconfig")
return self._parse_status_response(response)
except Exception as e:
raise RouterConnectionError(f"Failed to retrieve router status: {e}")
async def configure_csi_monitoring(self, config: Dict[str, Any]) -> bool:
"""Configure CSI monitoring on router.
Args:
config: CSI monitoring configuration
Returns:
True if configuration successful, False otherwise
"""
try:
channel = config.get('channel', 6)
# Validate channel is an integer in a safe range to prevent command injection
if not isinstance(channel, int) or not (1 <= channel <= 196):
raise ValueError(f"Invalid WiFi channel: {channel}. Must be an integer between 1 and 196.")
command = f"iwconfig wlan0 channel {channel} && echo 'CSI monitoring configured'"
await self.execute_command(command)
return True
except Exception as e:
self.logger.error(f"Failed to configure CSI monitoring: {e}")
return False
async def health_check(self) -> bool:
"""Perform health check on router.
Returns:
True if router is healthy, False otherwise
"""
try:
response = await self.execute_command("echo 'ping' && echo 'pong'")
return "pong" in response
except Exception as e:
self.logger.error(f"Health check failed: {e}")
return False
def _parse_csi_response(self, response: str) -> CSIData:
"""Parse CSI response data.
Args:
response: Raw response from router
Returns:
Parsed CSI data
Raises:
RouterConnectionError: Always in current state, because real CSI
parsing from router command output requires hardware-specific
format knowledge that must be implemented per router model.
"""
raise RouterConnectionError(
"Real CSI data parsing from router responses is not yet implemented. "
"Collecting CSI data from a router requires: "
"(1) a router with CSI-capable firmware (e.g., Atheros CSI Tool, Nexmon), "
"(2) proper hardware setup and configuration, and "
"(3) a parser for the specific binary/text format produced by the firmware. "
"See docs/hardware-setup.md for instructions on configuring your router for CSI collection."
)
def _parse_status_response(self, response: str) -> Dict[str, Any]:
"""Parse router status response.
Args:
response: Raw response from router
Returns:
Parsed status information
"""
# Mock implementation for testing
# In real implementation, this would parse actual system status
return {
'cpu_usage': 25.5,
'memory_usage': 60.2,
'wifi_status': 'active',
'uptime': '5 days, 3 hours',
'raw_response': response
}
archive/v1/src/logger.py
+330
View File
@@ -0,0 +1,330 @@
"""
Logging configuration for WiFi-DensePose API
"""
import logging
import logging.config
import logging.handlers
import sys
import os
from pathlib import Path
from typing import Dict, Any, Optional
from datetime import datetime
from src.config.settings import Settings
class ColoredFormatter(logging.Formatter):
"""Colored log formatter for console output."""
# ANSI color codes
COLORS = {
'DEBUG': '\033[36m', # Cyan
'INFO': '\033[32m', # Green
'WARNING': '\033[33m', # Yellow
'ERROR': '\033[31m', # Red
'CRITICAL': '\033[35m', # Magenta
'RESET': '\033[0m' # Reset
}
def format(self, record):
"""Format log record with colors."""
if hasattr(record, 'levelname'):
color = self.COLORS.get(record.levelname, self.COLORS['RESET'])
record.levelname = f"{color}{record.levelname}{self.COLORS['RESET']}"
return super().format(record)
class StructuredFormatter(logging.Formatter):
"""Structured JSON formatter for log files."""
def format(self, record):
"""Format log record as structured JSON."""
import json
log_entry = {
'timestamp': datetime.utcnow().isoformat(),
'level': record.levelname,
'logger': record.name,
'message': record.getMessage(),
'module': record.module,
'function': record.funcName,
'line': record.lineno,
}
# Add exception info if present
if record.exc_info:
log_entry['exception'] = self.formatException(record.exc_info)
# Add extra fields
for key, value in record.__dict__.items():
if key not in ['name', 'msg', 'args', 'levelname', 'levelno', 'pathname',
'filename', 'module', 'lineno', 'funcName', 'created',
'msecs', 'relativeCreated', 'thread', 'threadName',
'processName', 'process', 'getMessage', 'exc_info',
'exc_text', 'stack_info']:
log_entry[key] = value
return json.dumps(log_entry)
class RequestContextFilter(logging.Filter):
"""Filter to add request context to log records."""
def filter(self, record):
"""Add request context to log record."""
# Try to get request context from contextvars or thread local
try:
import contextvars
request_id = contextvars.ContextVar('request_id', default=None).get()
user_id = contextvars.ContextVar('user_id', default=None).get()
if request_id:
record.request_id = request_id
if user_id:
record.user_id = user_id
except (ImportError, LookupError):
pass
return True
def setup_logging(settings: Settings) -> None:
"""Setup application logging configuration."""
# Create log directory if file logging is enabled
if settings.log_file:
log_path = Path(settings.log_file)
log_path.parent.mkdir(parents=True, exist_ok=True)
# Build logging configuration
config = build_logging_config(settings)
# Apply configuration
logging.config.dictConfig(config)
# Set up root logger
root_logger = logging.getLogger()
root_logger.setLevel(settings.log_level)
# Add request context filter to all handlers
request_filter = RequestContextFilter()
for handler in root_logger.handlers:
handler.addFilter(request_filter)
# Log startup message
logger = logging.getLogger(__name__)
logger.info(f"Logging configured - Level: {settings.log_level}, File: {settings.log_file}")
def build_logging_config(settings: Settings) -> Dict[str, Any]:
"""Build logging configuration dictionary."""
config = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'console': {
'()': ColoredFormatter,
'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
'file': {
'format': '%(asctime)s - %(name)s - %(levelname)s - %(module)s:%(lineno)d - %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
'structured': {
'()': StructuredFormatter
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': settings.log_level,
'formatter': 'console',
'stream': 'ext://sys.stdout'
}
},
'loggers': {
'': { # Root logger
'level': settings.log_level,
'handlers': ['console'],
'propagate': False
},
'src': { # Application logger
'level': settings.log_level,
'handlers': ['console'],
'propagate': False
},
'uvicorn': {
'level': 'INFO',
'handlers': ['console'],
'propagate': False
},
'uvicorn.access': {
'level': 'INFO',
'handlers': ['console'],
'propagate': False
},
'fastapi': {
'level': 'INFO',
'handlers': ['console'],
'propagate': False
},
'sqlalchemy': {
'level': 'WARNING',
'handlers': ['console'],
'propagate': False
},
'sqlalchemy.engine': {
'level': 'INFO' if settings.debug else 'WARNING',
'handlers': ['console'],
'propagate': False
}
}
}
# Add file handler if log file is specified
if settings.log_file:
config['handlers']['file'] = {
'class': 'logging.handlers.RotatingFileHandler',
'level': settings.log_level,
'formatter': 'file',
'filename': settings.log_file,
'maxBytes': settings.log_max_size,
'backupCount': settings.log_backup_count,
'encoding': 'utf-8'
}
# Add structured log handler for JSON logs
structured_log_file = str(Path(settings.log_file).with_suffix('.json'))
config['handlers']['structured'] = {
'class': 'logging.handlers.RotatingFileHandler',
'level': settings.log_level,
'formatter': 'structured',
'filename': structured_log_file,
'maxBytes': settings.log_max_size,
'backupCount': settings.log_backup_count,
'encoding': 'utf-8'
}
# Add file handlers to all loggers
for logger_config in config['loggers'].values():
logger_config['handlers'].extend(['file', 'structured'])
return config
def get_logger(name: str) -> logging.Logger:
"""Get a logger with the specified name."""
return logging.getLogger(name)
def configure_third_party_loggers(settings: Settings) -> None:
"""Configure third-party library loggers."""
# Suppress noisy loggers in production
if settings.is_production:
logging.getLogger('urllib3').setLevel(logging.WARNING)
logging.getLogger('requests').setLevel(logging.WARNING)
logging.getLogger('asyncio').setLevel(logging.WARNING)
logging.getLogger('multipart').setLevel(logging.WARNING)
# Configure SQLAlchemy logging
if settings.debug and settings.is_development:
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
logging.getLogger('sqlalchemy.pool').setLevel(logging.DEBUG)
else:
logging.getLogger('sqlalchemy').setLevel(logging.WARNING)
# Configure Redis logging
logging.getLogger('redis').setLevel(logging.WARNING)
# Configure WebSocket logging
logging.getLogger('websockets').setLevel(logging.INFO)
class LoggerMixin:
"""Mixin class to add logging capabilities to any class."""
@property
def logger(self) -> logging.Logger:
"""Get logger for this class."""
return logging.getLogger(f"{self.__class__.__module__}.{self.__class__.__name__}")
def log_function_call(func):
"""Decorator to log function calls."""
import functools
@functools.wraps(func)
def wrapper(*args, **kwargs):
logger = logging.getLogger(func.__module__)
logger.debug(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
try:
result = func(*args, **kwargs)
logger.debug(f"{func.__name__} completed successfully")
return result
except Exception as e:
logger.error(f"{func.__name__} failed with error: {e}")
raise
return wrapper
def log_async_function_call(func):
"""Decorator to log async function calls."""
import functools
@functools.wraps(func)
async def wrapper(*args, **kwargs):
logger = logging.getLogger(func.__module__)
logger.debug(f"Calling async {func.__name__} with args={args}, kwargs={kwargs}")
try:
result = await func(*args, **kwargs)
logger.debug(f"Async {func.__name__} completed successfully")
return result
except Exception as e:
logger.error(f"Async {func.__name__} failed with error: {e}")
raise
return wrapper
def setup_request_logging():
"""Setup request-specific logging context."""
import contextvars
import uuid
# Create context variables for request tracking
request_id_var = contextvars.ContextVar('request_id')
user_id_var = contextvars.ContextVar('user_id')
def set_request_context(request_id: Optional[str] = None, user_id: Optional[str] = None):
"""Set request context for logging."""
if request_id is None:
request_id = str(uuid.uuid4())
request_id_var.set(request_id)
if user_id:
user_id_var.set(user_id)
def get_request_context():
"""Get current request context."""
try:
return {
'request_id': request_id_var.get(),
'user_id': user_id_var.get(None)
}
except LookupError:
return {}
return set_request_context, get_request_context
# Initialize request logging context
set_request_context, get_request_context = setup_request_logging()
archive/v1/src/main.py
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""
Main application entry point for WiFi-DensePose API
"""
import sys
import os
import asyncio
import logging
import signal
from pathlib import Path
from typing import Optional
# Add src to Python path
sys.path.insert(0, str(Path(__file__).parent))
from src.config.settings import get_settings, validate_settings
from src.logger import setup_logging
from src.app import create_app
from src.services.orchestrator import ServiceOrchestrator
from src.cli import create_cli
def setup_signal_handlers(orchestrator: ServiceOrchestrator):
"""Setup signal handlers for graceful shutdown."""
def signal_handler(signum, frame):
logging.info(f"Received signal {signum}, initiating graceful shutdown...")
asyncio.create_task(orchestrator.shutdown())
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
async def main():
"""Main application entry point."""
try:
# Load settings
settings = get_settings()
# Setup logging
setup_logging(settings)
logger = logging.getLogger(__name__)
logger.info(f"Starting {settings.app_name} v{settings.version}")
logger.info(f"Environment: {settings.environment}")
# Validate settings
issues = validate_settings(settings)
if issues:
logger.error("Configuration issues found:")
for issue in issues:
logger.error(f" - {issue}")
if settings.is_production:
sys.exit(1)
else:
logger.warning("Continuing with configuration issues in development mode")
# Create service orchestrator
orchestrator = ServiceOrchestrator(settings)
# Setup signal handlers
setup_signal_handlers(orchestrator)
# Initialize services
await orchestrator.initialize()
# Create FastAPI app
app = create_app(settings, orchestrator)
# Start the application
if len(sys.argv) > 1:
# CLI mode
cli = create_cli(orchestrator)
await cli.run(sys.argv[1:])
else:
# Server mode
import uvicorn
logger.info(f"Starting server on {settings.host}:{settings.port}")
config = uvicorn.Config(
app,
host=settings.host,
port=settings.port,
reload=settings.reload and settings.is_development,
workers=settings.workers if not settings.reload else 1,
log_level=settings.log_level.lower(),
access_log=True,
use_colors=True
)
server = uvicorn.Server(config)
await server.serve()
except KeyboardInterrupt:
logger.info("Received keyboard interrupt, shutting down...")
except Exception as e:
logger.error(f"Application failed to start: {e}", exc_info=True)
sys.exit(1)
finally:
# Cleanup
if 'orchestrator' in locals():
await orchestrator.shutdown()
logger.info("Application shutdown complete")
def run():
"""Entry point for package installation."""
try:
asyncio.run(main())
except KeyboardInterrupt:
pass
if __name__ == "__main__":
run()
archive/v1/src/middleware/auth.py
+455
View File
@@ -0,0 +1,455 @@
"""
Authentication middleware for WiFi-DensePose API
"""
import logging
import time
from typing import Optional, Dict, Any, Callable
from datetime import datetime, timedelta
from fastapi import Request, Response, HTTPException, status
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import JWTError, jwt
from passlib.context import CryptContext
from src.config.settings import Settings
from src.logger import set_request_context
logger = logging.getLogger(__name__)
# Password hashing
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
# JWT token handler
security = HTTPBearer(auto_error=False)
class AuthenticationError(Exception):
"""Authentication error."""
pass
class AuthorizationError(Exception):
"""Authorization error."""
pass
class TokenManager:
"""JWT token management."""
def __init__(self, settings: Settings):
self.settings = settings
self.secret_key = settings.secret_key
self.algorithm = settings.jwt_algorithm
self.expire_hours = settings.jwt_expire_hours
def create_access_token(self, data: Dict[str, Any]) -> str:
"""Create JWT access token."""
to_encode = data.copy()
expire = datetime.utcnow() + timedelta(hours=self.expire_hours)
to_encode.update({"exp": expire, "iat": datetime.utcnow()})
encoded_jwt = jwt.encode(to_encode, self.secret_key, algorithm=self.algorithm)
return encoded_jwt
def verify_token(self, token: str) -> Dict[str, Any]:
"""Verify and decode JWT token."""
try:
payload = jwt.decode(token, self.secret_key, algorithms=[self.algorithm])
# Check token blacklist (logout invalidation)
from src.api.middleware.auth import token_blacklist
if token_blacklist.is_blacklisted(token):
raise AuthenticationError("Token has been revoked")
return payload
except JWTError as e:
logger.warning(f"JWT verification failed: {e}")
raise AuthenticationError("Invalid token")
def decode_token_claims(self, token: str) -> Optional[Dict[str, Any]]:
"""Decode and verify token, returning its claims.
Unlike the previous implementation, this method always verifies
the token signature. Use verify_token() for full validation
including expiry checks; this helper is provided only for
inspecting claims from an already-verified token.
"""
try:
return jwt.decode(token, self.secret_key, algorithms=[self.algorithm])
except JWTError:
return None
class UserManager:
"""User management for authentication."""
def __init__(self):
# In a real application, this would connect to a database.
# No default users are created -- users must be provisioned
# through the create_user() method or an external identity provider.
self._users: Dict[str, Dict[str, Any]] = {}
@staticmethod
def hash_password(password: str) -> str:
"""Hash a password."""
return pwd_context.hash(password)
@staticmethod
def verify_password(plain_password: str, hashed_password: str) -> bool:
"""Verify a password against its hash."""
return pwd_context.verify(plain_password, hashed_password)
def get_user(self, username: str) -> Optional[Dict[str, Any]]:
"""Get user by username."""
return self._users.get(username)
def authenticate_user(self, username: str, password: str) -> Optional[Dict[str, Any]]:
"""Authenticate user with username and password."""
user = self.get_user(username)
if not user:
return None
if not self.verify_password(password, user["hashed_password"]):
return None
if not user.get("is_active", False):
return None
return user
def create_user(self, username: str, email: str, password: str, roles: list = None) -> Dict[str, Any]:
"""Create a new user."""
if username in self._users:
raise ValueError("User already exists")
user = {
"username": username,
"email": email,
"hashed_password": self.hash_password(password),
"roles": roles or ["user"],
"is_active": True,
"created_at": datetime.utcnow(),
}
self._users[username] = user
return user
def update_user(self, username: str, updates: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Update user information."""
user = self._users.get(username)
if not user:
return None
# Don't allow updating certain fields
protected_fields = {"username", "created_at", "hashed_password"}
updates = {k: v for k, v in updates.items() if k not in protected_fields}
user.update(updates)
return user
def deactivate_user(self, username: str) -> bool:
"""Deactivate a user."""
user = self._users.get(username)
if user:
user["is_active"] = False
return True
return False
class AuthenticationMiddleware:
"""Authentication middleware for FastAPI."""
def __init__(self, settings: Settings):
self.settings = settings
self.token_manager = TokenManager(settings)
self.user_manager = UserManager()
self.enabled = settings.enable_authentication
async def __call__(self, request: Request, call_next: Callable) -> Response:
"""Process request through authentication middleware."""
start_time = time.time()
try:
# Skip authentication for certain paths
if self._should_skip_auth(request):
response = await call_next(request)
return response
# Skip if authentication is disabled
if not self.enabled:
response = await call_next(request)
return response
# Extract and verify token
user_info = await self._authenticate_request(request)
# Set user context
if user_info:
request.state.user = user_info
set_request_context(user_id=user_info.get("username"))
# Process request
response = await call_next(request)
# Add authentication headers
self._add_auth_headers(response, user_info)
return response
except AuthenticationError as e:
logger.warning(f"Authentication failed: {e}")
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=str(e),
headers={"WWW-Authenticate": "Bearer"},
)
except AuthorizationError as e:
logger.warning(f"Authorization failed: {e}")
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=str(e),
)
except Exception as e:
logger.error(f"Authentication middleware error: {e}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Authentication service error",
)
finally:
# Log request processing time
processing_time = time.time() - start_time
logger.debug(f"Auth middleware processing time: {processing_time:.3f}s")
def _should_skip_auth(self, request: Request) -> bool:
"""Check if authentication should be skipped for this request."""
path = request.url.path
# Skip authentication for these paths
skip_paths = [
"/health",
"/metrics",
"/docs",
"/redoc",
"/openapi.json",
"/auth/login",
"/auth/register",
"/static",
]
return any(path.startswith(skip_path) for skip_path in skip_paths)
async def _authenticate_request(self, request: Request) -> Optional[Dict[str, Any]]:
"""Authenticate the request and return user info."""
# Try to get token from Authorization header
authorization = request.headers.get("Authorization")
if not authorization:
if self._requires_auth(request):
raise AuthenticationError("Missing authorization header")
return None
# Extract token
try:
scheme, token = authorization.split()
if scheme.lower() != "bearer":
raise AuthenticationError("Invalid authentication scheme")
except ValueError:
raise AuthenticationError("Invalid authorization header format")
# Verify token
try:
payload = self.token_manager.verify_token(token)
username = payload.get("sub")
if not username:
raise AuthenticationError("Invalid token payload")
# Get user info
user = self.user_manager.get_user(username)
if not user:
raise AuthenticationError("User not found")
if not user.get("is_active", False):
raise AuthenticationError("User account is disabled")
# Return user info without sensitive data
return {
"username": user["username"],
"email": user["email"],
"roles": user["roles"],
"is_active": user["is_active"],
}
except AuthenticationError:
raise
except Exception as e:
logger.error(f"Token verification error: {e}")
raise AuthenticationError("Token verification failed")
def _requires_auth(self, request: Request) -> bool:
"""Check if the request requires authentication."""
# All API endpoints require authentication by default
path = request.url.path
return path.startswith("/api/") or path.startswith("/ws/")
def _add_auth_headers(self, response: Response, user_info: Optional[Dict[str, Any]]):
"""Add authentication-related headers to response."""
if user_info:
response.headers["X-User"] = user_info["username"]
response.headers["X-User-Roles"] = ",".join(user_info["roles"])
async def login(self, username: str, password: str) -> Dict[str, Any]:
"""Authenticate user and return token."""
user = self.user_manager.authenticate_user(username, password)
if not user:
raise AuthenticationError("Invalid username or password")
# Create token
token_data = {
"sub": user["username"],
"email": user["email"],
"roles": user["roles"],
}
access_token = self.token_manager.create_access_token(token_data)
return {
"access_token": access_token,
"token_type": "bearer",
"expires_in": self.settings.jwt_expire_hours * 3600,
"user": {
"username": user["username"],
"email": user["email"],
"roles": user["roles"],
}
}
async def register(self, username: str, email: str, password: str) -> Dict[str, Any]:
"""Register a new user."""
try:
user = self.user_manager.create_user(username, email, password)
# Create token for new user
token_data = {
"sub": user["username"],
"email": user["email"],
"roles": user["roles"],
}
access_token = self.token_manager.create_access_token(token_data)
return {
"access_token": access_token,
"token_type": "bearer",
"expires_in": self.settings.jwt_expire_hours * 3600,
"user": {
"username": user["username"],
"email": user["email"],
"roles": user["roles"],
}
}
except ValueError as e:
raise AuthenticationError(str(e))
async def refresh_token(self, token: str) -> Dict[str, Any]:
"""Refresh an access token."""
try:
payload = self.token_manager.verify_token(token)
username = payload.get("sub")
user = self.user_manager.get_user(username)
if not user or not user.get("is_active", False):
raise AuthenticationError("User not found or inactive")
# Create new token
token_data = {
"sub": user["username"],
"email": user["email"],
"roles": user["roles"],
}
new_token = self.token_manager.create_access_token(token_data)
return {
"access_token": new_token,
"token_type": "bearer",
"expires_in": self.settings.jwt_expire_hours * 3600,
}
except Exception as e:
raise AuthenticationError("Token refresh failed")
def check_permission(self, user_info: Dict[str, Any], required_role: str) -> bool:
"""Check if user has required role/permission."""
user_roles = user_info.get("roles", [])
# Admin role has all permissions
if "admin" in user_roles:
return True
# Check specific role
return required_role in user_roles
def require_role(self, required_role: str):
"""Decorator to require specific role."""
def decorator(func):
import functools
@functools.wraps(func)
async def wrapper(request: Request, *args, **kwargs):
user_info = getattr(request.state, "user", None)
if not user_info:
raise AuthorizationError("Authentication required")
if not self.check_permission(user_info, required_role):
raise AuthorizationError(f"Role '{required_role}' required")
return await func(request, *args, **kwargs)
return wrapper
return decorator
# Global authentication middleware instance
_auth_middleware: Optional[AuthenticationMiddleware] = None
def get_auth_middleware(settings: Settings) -> AuthenticationMiddleware:
"""Get authentication middleware instance."""
global _auth_middleware
if _auth_middleware is None:
_auth_middleware = AuthenticationMiddleware(settings)
return _auth_middleware
def get_current_user(request: Request) -> Optional[Dict[str, Any]]:
"""Get current authenticated user from request."""
return getattr(request.state, "user", None)
def require_authentication(request: Request) -> Dict[str, Any]:
"""Require authentication and return user info."""
user = get_current_user(request)
if not user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Authentication required",
headers={"WWW-Authenticate": "Bearer"},
)
return user
def require_role(role: str):
"""Dependency to require specific role."""
def dependency(request: Request) -> Dict[str, Any]:
user = require_authentication(request)
auth_middleware = get_auth_middleware(request.app.state.settings)
if not auth_middleware.check_permission(user, role):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"Role '{role}' required",
)
return user
return dependency
archive/v1/src/middleware/cors.py
+375
View File
@@ -0,0 +1,375 @@
"""
CORS middleware for WiFi-DensePose API
"""
import logging
from typing import List, Optional, Union, Callable
from urllib.parse import urlparse
from fastapi import Request, Response
from fastapi.middleware.cors import CORSMiddleware as FastAPICORSMiddleware
from starlette.types import ASGIApp
from src.config.settings import Settings
logger = logging.getLogger(__name__)
class CORSMiddleware:
"""Enhanced CORS middleware with additional security features."""
def __init__(
self,
app: ASGIApp,
settings: Settings,
allow_origins: Optional[List[str]] = None,
allow_methods: Optional[List[str]] = None,
allow_headers: Optional[List[str]] = None,
allow_credentials: bool = False,
expose_headers: Optional[List[str]] = None,
max_age: int = 600,
):
self.app = app
self.settings = settings
self.allow_origins = allow_origins or settings.cors_origins
self.allow_methods = allow_methods or ["GET", "POST", "PUT", "DELETE", "OPTIONS", "PATCH"]
self.allow_headers = allow_headers or [
"Accept",
"Accept-Language",
"Content-Language",
"Content-Type",
"Authorization",
"X-Requested-With",
"X-Request-ID",
"X-User-Agent",
]
self.allow_credentials = allow_credentials or settings.cors_allow_credentials
self.expose_headers = expose_headers or [
"X-Request-ID",
"X-Response-Time",
"X-Rate-Limit-Remaining",
"X-Rate-Limit-Reset",
]
self.max_age = max_age
# Security settings
self.strict_origin_check = settings.is_production
self.log_cors_violations = True
async def __call__(self, scope, receive, send):
"""ASGI middleware implementation."""
if scope["type"] != "http":
await self.app(scope, receive, send)
return
request = Request(scope, receive)
# Check if this is a CORS preflight request
if request.method == "OPTIONS" and "access-control-request-method" in request.headers:
response = await self._handle_preflight(request)
await response(scope, receive, send)
return
# Handle actual request
async def send_wrapper(message):
if message["type"] == "http.response.start":
# Add CORS headers to response
headers = dict(message.get("headers", []))
cors_headers = self._get_cors_headers(request)
for key, value in cors_headers.items():
headers[key.encode()] = value.encode()
message["headers"] = list(headers.items())
await send(message)
await self.app(scope, receive, send_wrapper)
async def _handle_preflight(self, request: Request) -> Response:
"""Handle CORS preflight request."""
origin = request.headers.get("origin")
requested_method = request.headers.get("access-control-request-method")
requested_headers = request.headers.get("access-control-request-headers", "")
# Validate origin
if not self._is_origin_allowed(origin):
if self.log_cors_violations:
logger.warning(f"CORS preflight rejected for origin: {origin}")
return Response(
status_code=403,
content="CORS preflight request rejected",
headers={"Content-Type": "text/plain"}
)
# Validate method
if requested_method not in self.allow_methods:
if self.log_cors_violations:
logger.warning(f"CORS preflight rejected for method: {requested_method}")
return Response(
status_code=405,
content="Method not allowed",
headers={"Content-Type": "text/plain"}
)
# Validate headers
if requested_headers:
requested_header_list = [h.strip().lower() for h in requested_headers.split(",")]
allowed_headers_lower = [h.lower() for h in self.allow_headers]
for header in requested_header_list:
if header not in allowed_headers_lower:
if self.log_cors_violations:
logger.warning(f"CORS preflight rejected for header: {header}")
return Response(
status_code=400,
content="Header not allowed",
headers={"Content-Type": "text/plain"}
)
# Build preflight response headers
headers = {
"Access-Control-Allow-Origin": origin,
"Access-Control-Allow-Methods": ", ".join(self.allow_methods),
"Access-Control-Allow-Headers": ", ".join(self.allow_headers),
"Access-Control-Max-Age": str(self.max_age),
}
if self.allow_credentials:
headers["Access-Control-Allow-Credentials"] = "true"
if self.expose_headers:
headers["Access-Control-Expose-Headers"] = ", ".join(self.expose_headers)
logger.debug(f"CORS preflight approved for origin: {origin}")
return Response(
status_code=200,
headers=headers
)
def _get_cors_headers(self, request: Request) -> dict:
"""Get CORS headers for actual request."""
origin = request.headers.get("origin")
headers = {}
if self._is_origin_allowed(origin):
headers["Access-Control-Allow-Origin"] = origin
if self.allow_credentials:
headers["Access-Control-Allow-Credentials"] = "true"
if self.expose_headers:
headers["Access-Control-Expose-Headers"] = ", ".join(self.expose_headers)
return headers
def _is_origin_allowed(self, origin: Optional[str]) -> bool:
"""Check if origin is allowed."""
if not origin:
return not self.strict_origin_check
# Allow all origins in development
if not self.settings.is_production and "*" in self.allow_origins:
return True
# Check exact matches
if origin in self.allow_origins:
return True
# Check wildcard patterns
for allowed_origin in self.allow_origins:
if allowed_origin == "*":
return not self.strict_origin_check
if self._match_origin_pattern(origin, allowed_origin):
return True
return False
def _match_origin_pattern(self, origin: str, pattern: str) -> bool:
"""Match origin against pattern with wildcard support."""
if "*" not in pattern:
return origin == pattern
# Simple wildcard matching
if pattern.startswith("*."):
domain = pattern[2:]
parsed_origin = urlparse(origin)
origin_host = parsed_origin.netloc
# Check if origin ends with the domain
return origin_host.endswith(domain) or origin_host == domain[1:] if domain.startswith('.') else origin_host == domain
return False
def setup_cors_middleware(app: ASGIApp, settings: Settings) -> ASGIApp:
"""Setup CORS middleware for the application."""
if settings.cors_enabled:
logger.info("Setting up CORS middleware")
# Use FastAPI's built-in CORS middleware for basic functionality
app = FastAPICORSMiddleware(
app,
allow_origins=settings.cors_origins,
allow_credentials=settings.cors_allow_credentials,
allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS", "PATCH"],
allow_headers=[
"Accept",
"Accept-Language",
"Content-Language",
"Content-Type",
"Authorization",
"X-Requested-With",
"X-Request-ID",
"X-User-Agent",
],
expose_headers=[
"X-Request-ID",
"X-Response-Time",
"X-Rate-Limit-Remaining",
"X-Rate-Limit-Reset",
],
max_age=600,
)
logger.info(f"CORS enabled for origins: {settings.cors_origins}")
else:
logger.info("CORS middleware disabled")
return app
class CORSConfig:
"""CORS configuration helper."""
@staticmethod
def development_config() -> dict:
"""Get CORS configuration for development."""
return {
"allow_origins": ["*"],
"allow_credentials": True,
"allow_methods": ["*"],
"allow_headers": ["*"],
"expose_headers": [
"X-Request-ID",
"X-Response-Time",
"X-Rate-Limit-Remaining",
"X-Rate-Limit-Reset",
],
"max_age": 600,
}
@staticmethod
def production_config(allowed_origins: List[str]) -> dict:
"""Get CORS configuration for production."""
return {
"allow_origins": allowed_origins,
"allow_credentials": True,
"allow_methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS", "PATCH"],
"allow_headers": [
"Accept",
"Accept-Language",
"Content-Language",
"Content-Type",
"Authorization",
"X-Requested-With",
"X-Request-ID",
"X-User-Agent",
],
"expose_headers": [
"X-Request-ID",
"X-Response-Time",
"X-Rate-Limit-Remaining",
"X-Rate-Limit-Reset",
],
"max_age": 3600, # 1 hour for production
}
@staticmethod
def api_only_config(allowed_origins: List[str]) -> dict:
"""Get CORS configuration for API-only access."""
return {
"allow_origins": allowed_origins,
"allow_credentials": False,
"allow_methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
"allow_headers": [
"Accept",
"Content-Type",
"Authorization",
"X-Request-ID",
],
"expose_headers": [
"X-Request-ID",
"X-Rate-Limit-Remaining",
"X-Rate-Limit-Reset",
],
"max_age": 3600,
}
@staticmethod
def websocket_config(allowed_origins: List[str]) -> dict:
"""Get CORS configuration for WebSocket connections."""
return {
"allow_origins": allowed_origins,
"allow_credentials": True,
"allow_methods": ["GET", "OPTIONS"],
"allow_headers": [
"Accept",
"Authorization",
"Sec-WebSocket-Protocol",
"Sec-WebSocket-Extensions",
],
"expose_headers": [],
"max_age": 86400, # 24 hours for WebSocket
}
def validate_cors_config(settings: Settings) -> List[str]:
"""Validate CORS configuration and return issues."""
issues = []
if not settings.cors_enabled:
return issues
# Check origins
if not settings.cors_origins:
issues.append("CORS is enabled but no origins are configured")
# Check for wildcard in production
if settings.is_production and "*" in settings.cors_origins:
issues.append("Wildcard origin (*) should not be used in production")
# Validate origin formats
for origin in settings.cors_origins:
if origin != "*" and not origin.startswith(("http://", "https://")):
issues.append(f"Invalid origin format: {origin}")
# Check credentials with wildcard
if settings.cors_allow_credentials and "*" in settings.cors_origins:
issues.append("Cannot use credentials with wildcard origin")
return issues
def get_cors_headers_for_origin(origin: str, settings: Settings) -> dict:
"""Get appropriate CORS headers for a specific origin."""
headers = {}
if not settings.cors_enabled:
return headers
# Check if origin is allowed
cors_middleware = CORSMiddleware(None, settings)
if cors_middleware._is_origin_allowed(origin):
headers["Access-Control-Allow-Origin"] = origin
if settings.cors_allow_credentials:
headers["Access-Control-Allow-Credentials"] = "true"
return headers
archive/v1/src/middleware/error_handler.py
+504
View File
@@ -0,0 +1,504 @@
"""
Global error handling middleware for WiFi-DensePose API
"""
import logging
import traceback
import time
from typing import Dict, Any, Optional, Callable, Union
from datetime import datetime
from fastapi import Request, Response, HTTPException, status
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from pydantic import ValidationError
from src.config.settings import Settings
from src.logger import get_request_context
logger = logging.getLogger(__name__)
class ErrorResponse:
"""Standardized error response format."""
def __init__(
self,
error_code: str,
message: str,
details: Optional[Dict[str, Any]] = None,
status_code: int = 500,
request_id: Optional[str] = None,
):
self.error_code = error_code
self.message = message
self.details = details or {}
self.status_code = status_code
self.request_id = request_id
self.timestamp = datetime.utcnow().isoformat()
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary for JSON response."""
response = {
"error": {
"code": self.error_code,
"message": self.message,
"timestamp": self.timestamp,
}
}
if self.details:
response["error"]["details"] = self.details
if self.request_id:
response["error"]["request_id"] = self.request_id
return response
def to_response(self) -> JSONResponse:
"""Convert to FastAPI JSONResponse."""
headers = {}
if self.request_id:
headers["X-Request-ID"] = self.request_id
return JSONResponse(
status_code=self.status_code,
content=self.to_dict(),
headers=headers
)
class ErrorHandler:
"""Central error handler for the application."""
def __init__(self, settings: Settings):
self.settings = settings
self.include_traceback = settings.debug and settings.is_development
self.log_errors = True
def handle_http_exception(self, request: Request, exc: HTTPException) -> ErrorResponse:
"""Handle HTTP exceptions."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.warning(
f"HTTP {exc.status_code}: {exc.detail} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}"
)
# Determine error code
error_code = self._get_error_code_for_status(exc.status_code)
# Build error details
details = {}
if hasattr(exc, "headers") and exc.headers:
details["headers"] = exc.headers
if self.include_traceback and hasattr(exc, "__traceback__"):
details["traceback"] = traceback.format_exception(
type(exc), exc, exc.__traceback__
)
return ErrorResponse(
error_code=error_code,
message=str(exc.detail),
details=details,
status_code=exc.status_code,
request_id=request_id
)
def handle_validation_error(self, request: Request, exc: RequestValidationError) -> ErrorResponse:
"""Handle request validation errors."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.warning(
f"Validation error: {exc.errors()} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}"
)
# Format validation errors
validation_details = []
for error in exc.errors():
validation_details.append({
"field": ".".join(str(loc) for loc in error["loc"]),
"message": error["msg"],
"type": error["type"],
"input": error.get("input"),
})
details = {
"validation_errors": validation_details,
"error_count": len(validation_details)
}
if self.include_traceback:
details["traceback"] = traceback.format_exception(
type(exc), exc, exc.__traceback__
)
return ErrorResponse(
error_code="VALIDATION_ERROR",
message="Request validation failed",
details=details,
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
request_id=request_id
)
def handle_pydantic_error(self, request: Request, exc: ValidationError) -> ErrorResponse:
"""Handle Pydantic validation errors."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.warning(
f"Pydantic validation error: {exc.errors()} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}"
)
# Format validation errors
validation_details = []
for error in exc.errors():
validation_details.append({
"field": ".".join(str(loc) for loc in error["loc"]),
"message": error["msg"],
"type": error["type"],
})
details = {
"validation_errors": validation_details,
"error_count": len(validation_details)
}
return ErrorResponse(
error_code="DATA_VALIDATION_ERROR",
message="Data validation failed",
details=details,
status_code=status.HTTP_400_BAD_REQUEST,
request_id=request_id
)
def handle_generic_exception(self, request: Request, exc: Exception) -> ErrorResponse:
"""Handle generic exceptions."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.error(
f"Unhandled exception: {type(exc).__name__}: {exc} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}",
exc_info=True
)
# Determine error details
details = {}
if self.include_traceback:
details["exception_type"] = type(exc).__name__
details["traceback"] = traceback.format_exception(
type(exc), exc, exc.__traceback__
)
# Don't expose internal error details in production
if self.settings.is_production:
message = "An internal server error occurred"
else:
message = str(exc) or "An unexpected error occurred"
return ErrorResponse(
error_code="INTERNAL_SERVER_ERROR",
message=message,
details=details,
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
request_id=request_id
)
def handle_database_error(self, request: Request, exc: Exception) -> ErrorResponse:
"""Handle database-related errors."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.error(
f"Database error: {type(exc).__name__}: {exc} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}",
exc_info=True
)
details = {
"exception_type": type(exc).__name__,
"category": "database"
}
if self.include_traceback:
details["traceback"] = traceback.format_exception(
type(exc), exc, exc.__traceback__
)
return ErrorResponse(
error_code="DATABASE_ERROR",
message="Database operation failed" if self.settings.is_production else str(exc),
details=details,
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
request_id=request_id
)
def handle_external_service_error(self, request: Request, exc: Exception) -> ErrorResponse:
"""Handle external service errors."""
request_context = get_request_context()
request_id = request_context.get("request_id")
# Log the error
if self.log_errors:
logger.error(
f"External service error: {type(exc).__name__}: {exc} - "
f"{request.method} {request.url.path} - "
f"Request ID: {request_id}",
exc_info=True
)
details = {
"exception_type": type(exc).__name__,
"category": "external_service"
}
return ErrorResponse(
error_code="EXTERNAL_SERVICE_ERROR",
message="External service unavailable" if self.settings.is_production else str(exc),
details=details,
status_code=status.HTTP_502_BAD_GATEWAY,
request_id=request_id
)
def _get_error_code_for_status(self, status_code: int) -> str:
"""Get error code for HTTP status code."""
error_codes = {
400: "BAD_REQUEST",
401: "UNAUTHORIZED",
403: "FORBIDDEN",
404: "NOT_FOUND",
405: "METHOD_NOT_ALLOWED",
409: "CONFLICT",
422: "UNPROCESSABLE_ENTITY",
429: "TOO_MANY_REQUESTS",
500: "INTERNAL_SERVER_ERROR",
502: "BAD_GATEWAY",
503: "SERVICE_UNAVAILABLE",
504: "GATEWAY_TIMEOUT",
}
return error_codes.get(status_code, "HTTP_ERROR")
class ErrorHandlingMiddleware:
"""Error handling middleware for FastAPI."""
def __init__(self, app, settings: Settings):
self.app = app
self.settings = settings
self.error_handler = ErrorHandler(settings)
async def __call__(self, scope, receive, send):
"""Process request through error handling middleware."""
if scope["type"] != "http":
await self.app(scope, receive, send)
return
start_time = time.time()
try:
await self.app(scope, receive, send)
except Exception as exc:
# Create a mock request for error handling
from starlette.requests import Request
request = Request(scope, receive)
# Handle different exception types
if isinstance(exc, HTTPException):
error_response = self.error_handler.handle_http_exception(request, exc)
elif isinstance(exc, RequestValidationError):
error_response = self.error_handler.handle_validation_error(request, exc)
elif isinstance(exc, ValidationError):
error_response = self.error_handler.handle_pydantic_error(request, exc)
else:
# Check for specific error types
if self._is_database_error(exc):
error_response = self.error_handler.handle_database_error(request, exc)
elif self._is_external_service_error(exc):
error_response = self.error_handler.handle_external_service_error(request, exc)
else:
error_response = self.error_handler.handle_generic_exception(request, exc)
# Send the error response
response = error_response.to_response()
await response(scope, receive, send)
finally:
# Log request processing time
processing_time = time.time() - start_time
logger.debug(f"Error handling middleware processing time: {processing_time:.3f}s")
def _is_database_error(self, exc: Exception) -> bool:
"""Check if exception is database-related."""
database_exceptions = [
"sqlalchemy",
"psycopg2",
"pymongo",
"redis",
"ConnectionError",
"OperationalError",
"IntegrityError",
]
exc_module = getattr(type(exc), "__module__", "")
exc_name = type(exc).__name__
return any(
db_exc in exc_module or db_exc in exc_name
for db_exc in database_exceptions
)
def _is_external_service_error(self, exc: Exception) -> bool:
"""Check if exception is external service-related."""
external_exceptions = [
"requests",
"httpx",
"aiohttp",
"urllib",
"ConnectionError",
"TimeoutError",
"ConnectTimeout",
"ReadTimeout",
]
exc_module = getattr(type(exc), "__module__", "")
exc_name = type(exc).__name__
return any(
ext_exc in exc_module or ext_exc in exc_name
for ext_exc in external_exceptions
)
def setup_error_handling(app, settings: Settings):
"""Setup error handling for the application."""
logger.info("Setting up error handling middleware")
error_handler = ErrorHandler(settings)
# Add exception handlers
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException):
error_response = error_handler.handle_http_exception(request, exc)
return error_response.to_response()
@app.exception_handler(StarletteHTTPException)
async def starlette_http_exception_handler(request: Request, exc: StarletteHTTPException):
# Convert Starlette HTTPException to FastAPI HTTPException
fastapi_exc = HTTPException(status_code=exc.status_code, detail=exc.detail)
error_response = error_handler.handle_http_exception(request, fastapi_exc)
return error_response.to_response()
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
error_response = error_handler.handle_validation_error(request, exc)
return error_response.to_response()
@app.exception_handler(ValidationError)
async def pydantic_exception_handler(request: Request, exc: ValidationError):
error_response = error_handler.handle_pydantic_error(request, exc)
return error_response.to_response()
@app.exception_handler(Exception)
async def generic_exception_handler(request: Request, exc: Exception):
error_response = error_handler.handle_generic_exception(request, exc)
return error_response.to_response()
# Add middleware for additional error handling
# Note: We use exception handlers instead of custom middleware to avoid ASGI conflicts
# The middleware approach is commented out but kept for reference
# middleware = ErrorHandlingMiddleware(app, settings)
# app.add_middleware(ErrorHandlingMiddleware, settings=settings)
logger.info("Error handling configured")
class CustomHTTPException(HTTPException):
"""Custom HTTP exception with additional context."""
def __init__(
self,
status_code: int,
detail: str,
error_code: Optional[str] = None,
context: Optional[Dict[str, Any]] = None,
headers: Optional[Dict[str, str]] = None,
):
super().__init__(status_code=status_code, detail=detail, headers=headers)
self.error_code = error_code
self.context = context or {}
class BusinessLogicError(CustomHTTPException):
"""Exception for business logic errors."""
def __init__(self, message: str, context: Optional[Dict[str, Any]] = None):
super().__init__(
status_code=status.HTTP_400_BAD_REQUEST,
detail=message,
error_code="BUSINESS_LOGIC_ERROR",
context=context
)
class ResourceNotFoundError(CustomHTTPException):
"""Exception for resource not found errors."""
def __init__(self, resource: str, identifier: str):
super().__init__(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"{resource} not found",
error_code="RESOURCE_NOT_FOUND",
context={"resource": resource, "identifier": identifier}
)
class ConflictError(CustomHTTPException):
"""Exception for conflict errors."""
def __init__(self, message: str, context: Optional[Dict[str, Any]] = None):
super().__init__(
status_code=status.HTTP_409_CONFLICT,
detail=message,
error_code="CONFLICT_ERROR",
context=context
)
class ServiceUnavailableError(CustomHTTPException):
"""Exception for service unavailable errors."""
def __init__(self, service: str, reason: Optional[str] = None):
detail = f"{service} service is unavailable"
if reason:
detail += f": {reason}"
super().__init__(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=detail,
error_code="SERVICE_UNAVAILABLE",
context={"service": service, "reason": reason}
)
archive/v1/src/middleware/rate_limit.py
+477
View File
@@ -0,0 +1,477 @@
"""
Rate limiting middleware for WiFi-DensePose API
"""
import asyncio
import logging
import time
from typing import Dict, Any, Optional, Callable, Set, Tuple
from datetime import datetime, timedelta
from collections import defaultdict, deque
from dataclasses import dataclass
from fastapi import Request, Response, HTTPException, status
from starlette.types import ASGIApp
from src.config.settings import Settings
logger = logging.getLogger(__name__)
@dataclass
class RateLimitInfo:
"""Rate limit information."""
requests: int
window_start: float
window_size: int
limit: int
@property
def remaining(self) -> int:
"""Get remaining requests in current window."""
return max(0, self.limit - self.requests)
@property
def reset_time(self) -> float:
"""Get time when window resets."""
return self.window_start + self.window_size
@property
def is_exceeded(self) -> bool:
"""Check if rate limit is exceeded."""
return self.requests >= self.limit
class TokenBucket:
"""Token bucket algorithm for rate limiting."""
def __init__(self, capacity: int, refill_rate: float):
self.capacity = capacity
self.tokens = capacity
self.refill_rate = refill_rate
self.last_refill = time.time()
self._lock = asyncio.Lock()
async def consume(self, tokens: int = 1) -> bool:
"""Try to consume tokens from bucket."""
async with self._lock:
now = time.time()
# Refill tokens based on time elapsed
time_passed = now - self.last_refill
tokens_to_add = time_passed * self.refill_rate
self.tokens = min(self.capacity, self.tokens + tokens_to_add)
self.last_refill = now
# Check if we have enough tokens
if self.tokens >= tokens:
self.tokens -= tokens
return True
return False
def get_info(self) -> Dict[str, Any]:
"""Get bucket information."""
return {
"capacity": self.capacity,
"tokens": self.tokens,
"refill_rate": self.refill_rate,
"last_refill": self.last_refill
}
class SlidingWindowCounter:
"""Sliding window counter for rate limiting."""
def __init__(self, window_size: int, limit: int):
self.window_size = window_size
self.limit = limit
self.requests = deque()
self._lock = asyncio.Lock()
async def is_allowed(self) -> Tuple[bool, RateLimitInfo]:
"""Check if request is allowed."""
async with self._lock:
now = time.time()
window_start = now - self.window_size
# Remove old requests outside the window
while self.requests and self.requests[0] < window_start:
self.requests.popleft()
# Check if limit is exceeded
current_requests = len(self.requests)
allowed = current_requests < self.limit
if allowed:
self.requests.append(now)
rate_limit_info = RateLimitInfo(
requests=current_requests + (1 if allowed else 0),
window_start=window_start,
window_size=self.window_size,
limit=self.limit
)
return allowed, rate_limit_info
class RateLimiter:
"""Rate limiter with multiple algorithms."""
def __init__(self, settings: Settings):
self.settings = settings
self.enabled = settings.enable_rate_limiting
# Rate limit configurations
self.default_limit = settings.rate_limit_requests
self.authenticated_limit = settings.rate_limit_authenticated_requests
self.window_size = settings.rate_limit_window
# Trusted proxy IPs — only trust X-Forwarded-For/X-Real-IP from these
self.trusted_proxies: Set[str] = set(
getattr(settings, "trusted_proxies", [])
)
# Storage for rate limit data
self._sliding_windows: Dict[str, SlidingWindowCounter] = {}
self._token_buckets: Dict[str, TokenBucket] = {}
# Cleanup task
self._cleanup_task: Optional[asyncio.Task] = None
self._cleanup_interval = 300 # 5 minutes
async def start(self):
"""Start rate limiter background tasks."""
if self.enabled:
self._cleanup_task = asyncio.create_task(self._cleanup_loop())
logger.info("Rate limiter started")
async def stop(self):
"""Stop rate limiter background tasks."""
if self._cleanup_task:
self._cleanup_task.cancel()
try:
await self._cleanup_task
except asyncio.CancelledError:
pass
logger.info("Rate limiter stopped")
async def _cleanup_loop(self):
"""Background task to cleanup old rate limit data."""
while True:
try:
await asyncio.sleep(self._cleanup_interval)
await self._cleanup_old_data()
except asyncio.CancelledError:
break
except Exception as e:
logger.error(f"Error in rate limiter cleanup: {e}")
async def _cleanup_old_data(self):
"""Remove old rate limit data."""
now = time.time()
cutoff = now - (self.window_size * 2) # Keep data for 2 windows
# Cleanup sliding windows
keys_to_remove = []
for key, window in self._sliding_windows.items():
# Remove old requests
while window.requests and window.requests[0] < cutoff:
window.requests.popleft()
# Remove empty windows
if not window.requests:
keys_to_remove.append(key)
for key in keys_to_remove:
del self._sliding_windows[key]
logger.debug(f"Cleaned up {len(keys_to_remove)} old rate limit windows")
def _get_client_identifier(self, request: Request) -> str:
"""Get client identifier for rate limiting."""
# Try to get user ID from authenticated request
user = getattr(request.state, "user", None)
if user:
return f"user:{user.get('username', 'unknown')}"
# Fall back to IP address
client_ip = self._get_client_ip(request)
return f"ip:{client_ip}"
def _get_client_ip(self, request: Request) -> str:
"""Get client IP address.
Only trusts X-Forwarded-For / X-Real-IP when the direct connection
originates from a known trusted proxy. This prevents clients from
spoofing forwarded headers to bypass rate limiting.
"""
connection_ip = request.client.host if request.client else "unknown"
# Only honour forwarded headers from trusted proxies
if connection_ip in self.trusted_proxies:
forwarded_for = request.headers.get("X-Forwarded-For")
if forwarded_for:
return forwarded_for.split(",")[0].strip()
real_ip = request.headers.get("X-Real-IP")
if real_ip:
return real_ip
return connection_ip
def _get_rate_limit(self, request: Request) -> int:
"""Get rate limit for request."""
# Check if user is authenticated
user = getattr(request.state, "user", None)
if user:
return self.authenticated_limit
return self.default_limit
def _get_rate_limit_key(self, request: Request) -> str:
"""Get rate limit key for request."""
client_id = self._get_client_identifier(request)
endpoint = f"{request.method}:{request.url.path}"
return f"{client_id}:{endpoint}"
async def check_rate_limit(self, request: Request) -> Tuple[bool, RateLimitInfo]:
"""Check if request is within rate limits."""
if not self.enabled:
# Return dummy info when rate limiting is disabled
return True, RateLimitInfo(
requests=0,
window_start=time.time(),
window_size=self.window_size,
limit=float('inf')
)
key = self._get_rate_limit_key(request)
limit = self._get_rate_limit(request)
# Get or create sliding window counter
if key not in self._sliding_windows:
self._sliding_windows[key] = SlidingWindowCounter(self.window_size, limit)
window = self._sliding_windows[key]
# Update limit if it changed (e.g., user authenticated)
window.limit = limit
return await window.is_allowed()
async def check_token_bucket(self, request: Request, tokens: int = 1) -> bool:
"""Check rate limit using token bucket algorithm."""
if not self.enabled:
return True
key = self._get_client_identifier(request)
limit = self._get_rate_limit(request)
# Get or create token bucket
if key not in self._token_buckets:
# Refill rate: limit per window size
refill_rate = limit / self.window_size
self._token_buckets[key] = TokenBucket(limit, refill_rate)
bucket = self._token_buckets[key]
return await bucket.consume(tokens)
def get_rate_limit_headers(self, rate_limit_info: RateLimitInfo) -> Dict[str, str]:
"""Get rate limit headers for response."""
return {
"X-RateLimit-Limit": str(rate_limit_info.limit),
"X-RateLimit-Remaining": str(rate_limit_info.remaining),
"X-RateLimit-Reset": str(int(rate_limit_info.reset_time)),
"X-RateLimit-Window": str(rate_limit_info.window_size),
}
async def get_stats(self) -> Dict[str, Any]:
"""Get rate limiter statistics."""
return {
"enabled": self.enabled,
"default_limit": self.default_limit,
"authenticated_limit": self.authenticated_limit,
"window_size": self.window_size,
"active_windows": len(self._sliding_windows),
"active_buckets": len(self._token_buckets),
}
class RateLimitMiddleware:
"""Rate limiting middleware for FastAPI."""
def __init__(self, settings: Settings):
self.settings = settings
self.rate_limiter = RateLimiter(settings)
self.enabled = settings.enable_rate_limiting
async def __call__(self, request: Request, call_next: Callable) -> Response:
"""Process request through rate limiting middleware."""
if not self.enabled:
return await call_next(request)
# Skip rate limiting for certain paths
if self._should_skip_rate_limit(request):
return await call_next(request)
try:
# Check rate limit
allowed, rate_limit_info = await self.rate_limiter.check_rate_limit(request)
if not allowed:
# Rate limit exceeded
logger.warning(
f"Rate limit exceeded for {self.rate_limiter._get_client_identifier(request)} "
f"on {request.method} {request.url.path}"
)
headers = self.rate_limiter.get_rate_limit_headers(rate_limit_info)
headers["Retry-After"] = str(int(rate_limit_info.reset_time - time.time()))
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="Rate limit exceeded",
headers=headers
)
# Process request
response = await call_next(request)
# Add rate limit headers to response
headers = self.rate_limiter.get_rate_limit_headers(rate_limit_info)
for key, value in headers.items():
response.headers[key] = value
return response
except HTTPException:
raise
except Exception as e:
logger.error(f"Rate limiting middleware error: {e}")
# Continue without rate limiting on error
return await call_next(request)
def _should_skip_rate_limit(self, request: Request) -> bool:
"""Check if rate limiting should be skipped for this request."""
path = request.url.path
# Skip rate limiting for these paths
skip_paths = [
"/health",
"/metrics",
"/docs",
"/redoc",
"/openapi.json",
"/static",
]
return any(path.startswith(skip_path) for skip_path in skip_paths)
async def start(self):
"""Start rate limiting middleware."""
await self.rate_limiter.start()
async def stop(self):
"""Stop rate limiting middleware."""
await self.rate_limiter.stop()
# Global rate limit middleware instance
_rate_limit_middleware: Optional[RateLimitMiddleware] = None
def get_rate_limit_middleware(settings: Settings) -> RateLimitMiddleware:
"""Get rate limit middleware instance."""
global _rate_limit_middleware
if _rate_limit_middleware is None:
_rate_limit_middleware = RateLimitMiddleware(settings)
return _rate_limit_middleware
def setup_rate_limiting(app: ASGIApp, settings: Settings) -> ASGIApp:
"""Setup rate limiting middleware for the application."""
if settings.enable_rate_limiting:
logger.info("Setting up rate limiting middleware")
middleware = get_rate_limit_middleware(settings)
# Add middleware to app
@app.middleware("http")
async def rate_limit_middleware(request: Request, call_next):
return await middleware(request, call_next)
logger.info(
f"Rate limiting enabled - Default: {settings.rate_limit_requests}/"
f"{settings.rate_limit_window}s, Authenticated: "
f"{settings.rate_limit_authenticated_requests}/{settings.rate_limit_window}s"
)
else:
logger.info("Rate limiting disabled")
return app
class RateLimitConfig:
"""Rate limiting configuration helper."""
@staticmethod
def development_config() -> dict:
"""Get rate limiting configuration for development."""
return {
"enable_rate_limiting": False, # Disabled in development
"rate_limit_requests": 1000,
"rate_limit_authenticated_requests": 5000,
"rate_limit_window": 3600, # 1 hour
}
@staticmethod
def production_config() -> dict:
"""Get rate limiting configuration for production."""
return {
"enable_rate_limiting": True,
"rate_limit_requests": 100, # 100 requests per hour for unauthenticated
"rate_limit_authenticated_requests": 1000, # 1000 requests per hour for authenticated
"rate_limit_window": 3600, # 1 hour
}
@staticmethod
def api_config() -> dict:
"""Get rate limiting configuration for API access."""
return {
"enable_rate_limiting": True,
"rate_limit_requests": 60, # 60 requests per minute
"rate_limit_authenticated_requests": 300, # 300 requests per minute
"rate_limit_window": 60, # 1 minute
}
@staticmethod
def strict_config() -> dict:
"""Get strict rate limiting configuration."""
return {
"enable_rate_limiting": True,
"rate_limit_requests": 10, # 10 requests per minute
"rate_limit_authenticated_requests": 100, # 100 requests per minute
"rate_limit_window": 60, # 1 minute
}
def validate_rate_limit_config(settings: Settings) -> list:
"""Validate rate limiting configuration."""
issues = []
if settings.enable_rate_limiting:
if settings.rate_limit_requests <= 0:
issues.append("Rate limit requests must be positive")
if settings.rate_limit_authenticated_requests <= 0:
issues.append("Authenticated rate limit requests must be positive")
if settings.rate_limit_window <= 0:
issues.append("Rate limit window must be positive")
if settings.rate_limit_authenticated_requests < settings.rate_limit_requests:
issues.append("Authenticated rate limit should be higher than default rate limit")
return issues
archive/v1/src/models/__init__.py
View File
archive/v1/src/models/densepose_head.py
+279
View File
@@ -0,0 +1,279 @@
"""DensePose head for WiFi-DensePose system."""
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Dict, Any, Tuple, List
class DensePoseError(Exception):
"""Exception raised for DensePose head errors."""
pass
class DensePoseHead(nn.Module):
"""DensePose head for body part segmentation and UV coordinate regression."""
def __init__(self, config: Dict[str, Any]):
"""Initialize DensePose head.
Args:
config: Configuration dictionary with head parameters
"""
super().__init__()
self._validate_config(config)
self.config = config
self.input_channels = config['input_channels']
self.num_body_parts = config['num_body_parts']
self.num_uv_coordinates = config['num_uv_coordinates']
self.hidden_channels = config.get('hidden_channels', [128, 64])
self.kernel_size = config.get('kernel_size', 3)
self.padding = config.get('padding', 1)
self.dropout_rate = config.get('dropout_rate', 0.1)
self.use_deformable_conv = config.get('use_deformable_conv', False)
self.use_fpn = config.get('use_fpn', False)
self.fpn_levels = config.get('fpn_levels', [2, 3, 4, 5])
self.output_stride = config.get('output_stride', 4)
# Feature Pyramid Network (optional)
if self.use_fpn:
self.fpn = self._build_fpn()
# Shared feature processing
self.shared_conv = self._build_shared_layers()
# Segmentation head for body part classification
self.segmentation_head = self._build_segmentation_head()
# UV regression head for coordinate prediction
self.uv_regression_head = self._build_uv_regression_head()
# Initialize weights
self._initialize_weights()
def _validate_config(self, config: Dict[str, Any]):
"""Validate configuration parameters."""
required_fields = ['input_channels', 'num_body_parts', 'num_uv_coordinates']
for field in required_fields:
if field not in config:
raise ValueError(f"Missing required field: {field}")
if config['input_channels'] <= 0:
raise ValueError("input_channels must be positive")
if config['num_body_parts'] <= 0:
raise ValueError("num_body_parts must be positive")
if config['num_uv_coordinates'] <= 0:
raise ValueError("num_uv_coordinates must be positive")
def _build_fpn(self) -> nn.Module:
"""Build Feature Pyramid Network."""
return nn.ModuleDict({
f'level_{level}': nn.Conv2d(self.input_channels, self.input_channels, 1)
for level in self.fpn_levels
})
def _build_shared_layers(self) -> nn.Module:
"""Build shared feature processing layers."""
layers = []
in_channels = self.input_channels
for hidden_dim in self.hidden_channels:
layers.extend([
nn.Conv2d(in_channels, hidden_dim,
kernel_size=self.kernel_size,
padding=self.padding),
nn.BatchNorm2d(hidden_dim),
nn.ReLU(inplace=True),
nn.Dropout2d(self.dropout_rate)
])
in_channels = hidden_dim
return nn.Sequential(*layers)
def _build_segmentation_head(self) -> nn.Module:
"""Build segmentation head for body part classification."""
final_hidden = self.hidden_channels[-1] if self.hidden_channels else self.input_channels
return nn.Sequential(
nn.Conv2d(final_hidden, final_hidden // 2,
kernel_size=self.kernel_size,
padding=self.padding),
nn.BatchNorm2d(final_hidden // 2),
nn.ReLU(inplace=True),
nn.Dropout2d(self.dropout_rate),
# Upsampling to increase resolution
nn.ConvTranspose2d(final_hidden // 2, final_hidden // 4,
kernel_size=4, stride=2, padding=1),
nn.BatchNorm2d(final_hidden // 4),
nn.ReLU(inplace=True),
nn.Conv2d(final_hidden // 4, self.num_body_parts + 1, kernel_size=1),
# +1 for background class
)
def _build_uv_regression_head(self) -> nn.Module:
"""Build UV regression head for coordinate prediction."""
final_hidden = self.hidden_channels[-1] if self.hidden_channels else self.input_channels
return nn.Sequential(
nn.Conv2d(final_hidden, final_hidden // 2,
kernel_size=self.kernel_size,
padding=self.padding),
nn.BatchNorm2d(final_hidden // 2),
nn.ReLU(inplace=True),
nn.Dropout2d(self.dropout_rate),
# Upsampling to increase resolution
nn.ConvTranspose2d(final_hidden // 2, final_hidden // 4,
kernel_size=4, stride=2, padding=1),
nn.BatchNorm2d(final_hidden // 4),
nn.ReLU(inplace=True),
nn.Conv2d(final_hidden // 4, self.num_uv_coordinates, kernel_size=1),
)
def _initialize_weights(self):
"""Initialize network weights."""
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
if m.bias is not None:
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def forward(self, x: torch.Tensor) -> Dict[str, torch.Tensor]:
"""Forward pass through the DensePose head.
Args:
x: Input feature tensor of shape (batch_size, channels, height, width)
Returns:
Dictionary containing:
- segmentation: Body part logits (batch_size, num_parts+1, height, width)
- uv_coordinates: UV coordinates (batch_size, 2, height, width)
"""
# Validate input shape
if x.shape[1] != self.input_channels:
raise DensePoseError(f"Expected {self.input_channels} input channels, got {x.shape[1]}")
# Apply FPN if enabled
if self.use_fpn:
# Simple FPN processing - in practice this would be more sophisticated
x = self.fpn['level_2'](x)
# Shared feature processing
shared_features = self.shared_conv(x)
# Segmentation branch
segmentation_logits = self.segmentation_head(shared_features)
# UV regression branch
uv_coordinates = self.uv_regression_head(shared_features)
uv_coordinates = torch.sigmoid(uv_coordinates) # Normalize to [0, 1]
return {
'segmentation': segmentation_logits,
'uv_coordinates': uv_coordinates
}
def compute_segmentation_loss(self, pred_logits: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
"""Compute segmentation loss.
Args:
pred_logits: Predicted segmentation logits
target: Target segmentation masks
Returns:
Computed cross-entropy loss
"""
return F.cross_entropy(pred_logits, target, ignore_index=-1)
def compute_uv_loss(self, pred_uv: torch.Tensor, target_uv: torch.Tensor) -> torch.Tensor:
"""Compute UV coordinate regression loss.
Args:
pred_uv: Predicted UV coordinates
target_uv: Target UV coordinates
Returns:
Computed L1 loss
"""
return F.l1_loss(pred_uv, target_uv)
def compute_total_loss(self, predictions: Dict[str, torch.Tensor],
seg_target: torch.Tensor,
uv_target: torch.Tensor,
seg_weight: float = 1.0,
uv_weight: float = 1.0) -> torch.Tensor:
"""Compute total loss combining segmentation and UV losses.
Args:
predictions: Dictionary of predictions
seg_target: Target segmentation masks
uv_target: Target UV coordinates
seg_weight: Weight for segmentation loss
uv_weight: Weight for UV loss
Returns:
Combined loss
"""
seg_loss = self.compute_segmentation_loss(predictions['segmentation'], seg_target)
uv_loss = self.compute_uv_loss(predictions['uv_coordinates'], uv_target)
return seg_weight * seg_loss + uv_weight * uv_loss
def get_prediction_confidence(self, predictions: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
"""Get prediction confidence scores.
Args:
predictions: Dictionary of predictions
Returns:
Dictionary of confidence scores
"""
seg_logits = predictions['segmentation']
uv_coords = predictions['uv_coordinates']
# Segmentation confidence: max probability
seg_probs = F.softmax(seg_logits, dim=1)
seg_confidence = torch.max(seg_probs, dim=1)[0]
# UV confidence: inverse of prediction variance
uv_variance = torch.var(uv_coords, dim=1, keepdim=True)
uv_confidence = 1.0 / (1.0 + uv_variance)
return {
'segmentation_confidence': seg_confidence,
'uv_confidence': uv_confidence.squeeze(1)
}
def post_process_predictions(self, predictions: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
"""Post-process predictions for final output.
Args:
predictions: Raw predictions from forward pass
Returns:
Post-processed predictions
"""
seg_logits = predictions['segmentation']
uv_coords = predictions['uv_coordinates']
# Convert logits to class predictions
body_parts = torch.argmax(seg_logits, dim=1)
# Get confidence scores
confidence = self.get_prediction_confidence(predictions)
return {
'body_parts': body_parts,
'uv_coordinates': uv_coords,
'confidence_scores': confidence
}
archive/v1/src/models/modality_translation.py
+301
View File
@@ -0,0 +1,301 @@
"""Modality translation network for WiFi-DensePose system."""
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Dict, Any, List
class ModalityTranslationError(Exception):
"""Exception raised for modality translation errors."""
pass
class ModalityTranslationNetwork(nn.Module):
"""Neural network for translating CSI data to visual feature space."""
def __init__(self, config: Dict[str, Any]):
"""Initialize modality translation network.
Args:
config: Configuration dictionary with network parameters
"""
super().__init__()
self._validate_config(config)
self.config = config
self.input_channels = config['input_channels']
self.hidden_channels = config['hidden_channels']
self.output_channels = config['output_channels']
self.kernel_size = config.get('kernel_size', 3)
self.stride = config.get('stride', 1)
self.padding = config.get('padding', 1)
self.dropout_rate = config.get('dropout_rate', 0.1)
self.activation = config.get('activation', 'relu')
self.normalization = config.get('normalization', 'batch')
self.use_attention = config.get('use_attention', False)
self.attention_heads = config.get('attention_heads', 8)
# Encoder: CSI -> Feature space
self.encoder = self._build_encoder()
# Decoder: Feature space -> Visual-like features
self.decoder = self._build_decoder()
# Attention mechanism
if self.use_attention:
self.attention = self._build_attention()
# Initialize weights
self._initialize_weights()
def _validate_config(self, config: Dict[str, Any]):
"""Validate configuration parameters."""
required_fields = ['input_channels', 'hidden_channels', 'output_channels']
for field in required_fields:
if field not in config:
raise ValueError(f"Missing required field: {field}")
if config['input_channels'] <= 0:
raise ValueError("input_channels must be positive")
if not config['hidden_channels'] or len(config['hidden_channels']) == 0:
raise ValueError("hidden_channels must be a non-empty list")
if config['output_channels'] <= 0:
raise ValueError("output_channels must be positive")
def _build_encoder(self) -> nn.ModuleList:
"""Build encoder network."""
layers = nn.ModuleList()
# Initial convolution
in_channels = self.input_channels
for i, out_channels in enumerate(self.hidden_channels):
layer_block = nn.Sequential(
nn.Conv2d(in_channels, out_channels,
kernel_size=self.kernel_size,
stride=self.stride if i == 0 else 2,
padding=self.padding),
self._get_normalization(out_channels),
self._get_activation(),
nn.Dropout2d(self.dropout_rate)
)
layers.append(layer_block)
in_channels = out_channels
return layers
def _build_decoder(self) -> nn.ModuleList:
"""Build decoder network."""
layers = nn.ModuleList()
# Start with the last hidden channel size
in_channels = self.hidden_channels[-1]
# Progressive upsampling (reverse of encoder)
for i, out_channels in enumerate(reversed(self.hidden_channels[:-1])):
layer_block = nn.Sequential(
nn.ConvTranspose2d(in_channels, out_channels,
kernel_size=self.kernel_size,
stride=2,
padding=self.padding,
output_padding=1),
self._get_normalization(out_channels),
self._get_activation(),
nn.Dropout2d(self.dropout_rate)
)
layers.append(layer_block)
in_channels = out_channels
# Final output layer
final_layer = nn.Sequential(
nn.Conv2d(in_channels, self.output_channels,
kernel_size=self.kernel_size,
padding=self.padding),
nn.Tanh() # Normalize output
)
layers.append(final_layer)
return layers
def _get_normalization(self, channels: int) -> nn.Module:
"""Get normalization layer."""
if self.normalization == 'batch':
return nn.BatchNorm2d(channels)
elif self.normalization == 'instance':
return nn.InstanceNorm2d(channels)
elif self.normalization == 'layer':
return nn.GroupNorm(1, channels)
else:
return nn.Identity()
def _get_activation(self) -> nn.Module:
"""Get activation function."""
if self.activation == 'relu':
return nn.ReLU(inplace=True)
elif self.activation == 'leaky_relu':
return nn.LeakyReLU(0.2, inplace=True)
elif self.activation == 'gelu':
return nn.GELU()
else:
return nn.ReLU(inplace=True)
def _build_attention(self) -> nn.Module:
"""Build attention mechanism."""
return nn.MultiheadAttention(
embed_dim=self.hidden_channels[-1],
num_heads=self.attention_heads,
dropout=self.dropout_rate,
batch_first=True
)
def _initialize_weights(self):
"""Initialize network weights."""
for m in self.modules():
if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
if m.bias is not None:
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def forward(self, x: torch.Tensor) -> torch.Tensor:
"""Forward pass through the network.
Args:
x: Input CSI tensor of shape (batch_size, channels, height, width)
Returns:
Translated features tensor
"""
# Validate input shape
if x.shape[1] != self.input_channels:
raise ModalityTranslationError(f"Expected {self.input_channels} input channels, got {x.shape[1]}")
# Encode CSI data
encoded_features = self.encode(x)
# Decode to visual-like features
decoded = self.decode(encoded_features)
return decoded
def encode(self, x: torch.Tensor) -> List[torch.Tensor]:
"""Encode input through encoder layers.
Args:
x: Input tensor
Returns:
List of feature maps from each encoder layer
"""
features = []
current = x
for layer in self.encoder:
current = layer(current)
features.append(current)
return features
def decode(self, encoded_features: List[torch.Tensor]) -> torch.Tensor:
"""Decode features through decoder layers.
Args:
encoded_features: List of encoded feature maps
Returns:
Decoded output tensor
"""
# Start with the last encoded feature
current = encoded_features[-1]
# Apply attention if enabled
if self.use_attention:
batch_size, channels, height, width = current.shape
# Reshape for attention: (batch, seq_len, embed_dim)
current_flat = current.view(batch_size, channels, -1).transpose(1, 2)
attended, _ = self.attention(current_flat, current_flat, current_flat)
current = attended.transpose(1, 2).view(batch_size, channels, height, width)
# Apply decoder layers
for layer in self.decoder:
current = layer(current)
return current
def compute_translation_loss(self, predicted: torch.Tensor, target: torch.Tensor, loss_type: str = 'mse') -> torch.Tensor:
"""Compute translation loss between predicted and target features.
Args:
predicted: Predicted feature tensor
target: Target feature tensor
loss_type: Type of loss ('mse', 'l1', 'smooth_l1')
Returns:
Computed loss tensor
"""
if loss_type == 'mse':
return F.mse_loss(predicted, target)
elif loss_type == 'l1':
return F.l1_loss(predicted, target)
elif loss_type == 'smooth_l1':
return F.smooth_l1_loss(predicted, target)
else:
return F.mse_loss(predicted, target)
def get_feature_statistics(self, features: torch.Tensor) -> Dict[str, float]:
"""Get statistics of feature tensor.
Args:
features: Feature tensor to analyze
Returns:
Dictionary of feature statistics
"""
with torch.no_grad():
return {
'mean': features.mean().item(),
'std': features.std().item(),
'min': features.min().item(),
'max': features.max().item(),
'sparsity': (features == 0).float().mean().item()
}
def get_intermediate_features(self, x: torch.Tensor) -> Dict[str, Any]:
"""Get intermediate features for visualization.
Args:
x: Input tensor
Returns:
Dictionary containing intermediate features
"""
result = {}
# Get encoder features
encoder_features = self.encode(x)
result['encoder_features'] = encoder_features
# Get decoder features
decoder_features = []
current = encoder_features[-1]
if self.use_attention:
batch_size, channels, height, width = current.shape
current_flat = current.view(batch_size, channels, -1).transpose(1, 2)
attended, attention_weights = self.attention(current_flat, current_flat, current_flat)
current = attended.transpose(1, 2).view(batch_size, channels, height, width)
result['attention_weights'] = attention_weights
for layer in self.decoder:
current = layer(current)
decoder_features.append(current)
result['decoder_features'] = decoder_features
return result
archive/v1/src/sensing/__init__.py
+58
View File
@@ -0,0 +1,58 @@
"""
Commodity WiFi Sensing Module (ADR-013)
=======================================
RSSI-based presence and motion detection using standard Linux WiFi metrics.
This module provides real signal processing from commodity WiFi hardware,
extracting presence and motion features from RSSI time series.
Components:
- rssi_collector: Data collection from Linux WiFi interfaces
- feature_extractor: Time-domain and frequency-domain feature extraction
- classifier: Presence and motion classification from features
- backend: Common sensing backend interface
Capabilities:
- PRESENCE: Detect whether a person is present in the sensing area
- MOTION: Classify motion level (absent / still / active)
Note: This module uses RSSI only. For higher-fidelity sensing (respiration,
pose estimation), CSI-capable hardware and the full DensePose pipeline
are required.
"""
from v1.src.sensing.rssi_collector import (
LinuxWifiCollector,
SimulatedCollector,
WindowsWifiCollector,
WifiSample,
)
from v1.src.sensing.feature_extractor import (
RssiFeatureExtractor,
RssiFeatures,
)
from v1.src.sensing.classifier import (
PresenceClassifier,
SensingResult,
MotionLevel,
)
from v1.src.sensing.backend import (
SensingBackend,
CommodityBackend,
Capability,
)
__all__ = [
"LinuxWifiCollector",
"SimulatedCollector",
"WindowsWifiCollector",
"WifiSample",
"RssiFeatureExtractor",
"RssiFeatures",
"PresenceClassifier",
"SensingResult",
"MotionLevel",
"SensingBackend",
"CommodityBackend",
"Capability",
]
archive/v1/src/sensing/backend.py
+165
View File
@@ -0,0 +1,165 @@
"""
Common sensing backend interface.
Defines the ``SensingBackend`` protocol and the ``CommodityBackend`` concrete
implementation that wires together the RSSI collector, feature extractor, and
classifier into a single coherent pipeline.
The ``Capability`` enum enumerates all possible sensing capabilities. The
``CommodityBackend`` honestly reports that it supports only PRESENCE and MOTION.
"""
from __future__ import annotations
import logging
from enum import Enum, auto
from typing import List, Optional, Protocol, Set, runtime_checkable
from v1.src.sensing.classifier import MotionLevel, PresenceClassifier, SensingResult
from v1.src.sensing.feature_extractor import RssiFeatureExtractor, RssiFeatures
from v1.src.sensing.rssi_collector import (
LinuxWifiCollector,
SimulatedCollector,
WindowsWifiCollector,
WifiCollector,
WifiSample,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Capability enum
# ---------------------------------------------------------------------------
class Capability(Enum):
"""All possible sensing capabilities across backend tiers."""
PRESENCE = auto()
MOTION = auto()
RESPIRATION = auto()
LOCATION = auto()
POSE = auto()
# ---------------------------------------------------------------------------
# Backend protocol
# ---------------------------------------------------------------------------
@runtime_checkable
class SensingBackend(Protocol):
"""Protocol that all sensing backends must implement."""
def get_features(self) -> RssiFeatures:
"""Extract current features from the sensing pipeline."""
...
def get_capabilities(self) -> Set[Capability]:
"""Return the set of capabilities this backend supports."""
...
# ---------------------------------------------------------------------------
# Commodity backend
# ---------------------------------------------------------------------------
class CommodityBackend:
"""
RSSI-based commodity sensing backend.
Wires together:
- A WiFi collector (real or simulated)
- An RSSI feature extractor
- A presence/motion classifier
Capabilities: PRESENCE and MOTION only.
Parameters
----------
collector : WifiCollector-compatible object
The data source (LinuxWifiCollector or SimulatedCollector).
extractor : RssiFeatureExtractor, optional
Feature extractor (created with defaults if not provided).
classifier : PresenceClassifier, optional
Classifier (created with defaults if not provided).
"""
SUPPORTED_CAPABILITIES: Set[Capability] = frozenset(
{Capability.PRESENCE, Capability.MOTION}
)
def __init__(
self,
collector: LinuxWifiCollector | SimulatedCollector | WindowsWifiCollector,
extractor: Optional[RssiFeatureExtractor] = None,
classifier: Optional[PresenceClassifier] = None,
) -> None:
self._collector = collector
self._extractor = extractor or RssiFeatureExtractor()
self._classifier = classifier or PresenceClassifier()
@property
def collector(self) -> LinuxWifiCollector | SimulatedCollector | WindowsWifiCollector:
return self._collector
@property
def extractor(self) -> RssiFeatureExtractor:
return self._extractor
@property
def classifier(self) -> PresenceClassifier:
return self._classifier
# -- SensingBackend protocol ---------------------------------------------
def get_features(self) -> RssiFeatures:
"""
Get current features from the latest collected samples.
Uses the extractor's window_seconds to determine how many samples
to pull from the collector's ring buffer.
"""
window = self._extractor.window_seconds
sample_rate = self._collector.sample_rate_hz
n_needed = int(window * sample_rate)
samples = self._collector.get_samples(n=n_needed)
return self._extractor.extract(samples)
def get_capabilities(self) -> Set[Capability]:
"""CommodityBackend supports PRESENCE and MOTION only."""
return set(self.SUPPORTED_CAPABILITIES)
# -- convenience methods -------------------------------------------------
def get_result(self) -> SensingResult:
"""
Run the full pipeline: collect -> extract -> classify.
Returns
-------
SensingResult
Classification result with motion level and confidence.
"""
features = self.get_features()
return self._classifier.classify(features)
def start(self) -> None:
"""Start the underlying collector."""
self._collector.start()
logger.info(
"CommodityBackend started (capabilities: %s)",
", ".join(c.name for c in self.SUPPORTED_CAPABILITIES),
)
def stop(self) -> None:
"""Stop the underlying collector."""
self._collector.stop()
logger.info("CommodityBackend stopped")
def is_capable(self, capability: Capability) -> bool:
"""Check whether this backend supports a specific capability."""
return capability in self.SUPPORTED_CAPABILITIES
def __repr__(self) -> str:
caps = ", ".join(c.name for c in sorted(self.SUPPORTED_CAPABILITIES, key=lambda c: c.value))
return f"CommodityBackend(capabilities=[{caps}])"
archive/v1/src/sensing/classifier.py
+201
View File
@@ -0,0 +1,201 @@
"""
Presence and motion classification from RSSI features.
Uses rule-based logic with configurable thresholds to classify the current
sensing state into one of three motion levels:
ABSENT -- no person detected
PRESENT_STILL -- person present but stationary
ACTIVE -- person present and moving
Confidence is derived from spectral feature strength and optional
cross-receiver agreement.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from enum import Enum
from typing import List, Optional
from v1.src.sensing.feature_extractor import RssiFeatures
logger = logging.getLogger(__name__)
class MotionLevel(Enum):
"""Classified motion state."""
ABSENT = "absent"
PRESENT_STILL = "present_still"
ACTIVE = "active"
@dataclass
class SensingResult:
"""Output of the presence/motion classifier."""
motion_level: MotionLevel
confidence: float # 0.0 to 1.0
presence_detected: bool
rssi_variance: float
motion_band_energy: float
breathing_band_energy: float
n_change_points: int
details: str = ""
class PresenceClassifier:
"""
Rule-based presence and motion classifier.
Classification rules
--------------------
1. **Presence**: RSSI variance exceeds ``presence_variance_threshold``.
2. **Motion level**:
- ABSENT if variance < presence threshold
- ACTIVE if variance >= presence threshold AND motion band energy
exceeds ``motion_energy_threshold``
- PRESENT_STILL otherwise (variance above threshold but low motion energy)
Confidence model
----------------
Base confidence comes from how far the measured variance / energy exceeds
the respective thresholds. Cross-receiver agreement (when multiple
receivers report results) can boost confidence further.
Parameters
----------
presence_variance_threshold : float
Minimum RSSI variance (dBm^2) to declare presence (default 0.5).
motion_energy_threshold : float
Minimum motion-band spectral energy to classify as ACTIVE (default 0.1).
max_receivers : int
Maximum number of receivers for cross-receiver agreement (default 1).
"""
def __init__(
self,
presence_variance_threshold: float = 0.5,
motion_energy_threshold: float = 0.1,
max_receivers: int = 1,
) -> None:
self._var_thresh = presence_variance_threshold
self._motion_thresh = motion_energy_threshold
self._max_receivers = max_receivers
@property
def presence_variance_threshold(self) -> float:
return self._var_thresh
@property
def motion_energy_threshold(self) -> float:
return self._motion_thresh
def classify(
self,
features: RssiFeatures,
other_receiver_results: Optional[List[SensingResult]] = None,
) -> SensingResult:
"""
Classify presence and motion from extracted RSSI features.
Parameters
----------
features : RssiFeatures
Features extracted from the RSSI time series of one receiver.
other_receiver_results : list of SensingResult, optional
Results from other receivers for cross-receiver agreement.
Returns
-------
SensingResult
"""
variance = features.variance
motion_energy = features.motion_band_power
breathing_energy = features.breathing_band_power
# -- presence decision ------------------------------------------------
presence = variance >= self._var_thresh
# -- motion level -----------------------------------------------------
if not presence:
level = MotionLevel.ABSENT
elif motion_energy >= self._motion_thresh:
level = MotionLevel.ACTIVE
else:
level = MotionLevel.PRESENT_STILL
# -- confidence -------------------------------------------------------
confidence = self._compute_confidence(
variance, motion_energy, breathing_energy, level, other_receiver_results
)
# -- detail string ----------------------------------------------------
details = (
f"var={variance:.4f} (thresh={self._var_thresh}), "
f"motion_energy={motion_energy:.4f} (thresh={self._motion_thresh}), "
f"breathing_energy={breathing_energy:.4f}, "
f"change_points={features.n_change_points}"
)
return SensingResult(
motion_level=level,
confidence=confidence,
presence_detected=presence,
rssi_variance=variance,
motion_band_energy=motion_energy,
breathing_band_energy=breathing_energy,
n_change_points=features.n_change_points,
details=details,
)
def _compute_confidence(
self,
variance: float,
motion_energy: float,
breathing_energy: float,
level: MotionLevel,
other_results: Optional[List[SensingResult]],
) -> float:
"""
Compute a confidence score in [0, 1].
The score is composed of:
- Base (60%): how clearly the variance exceeds (or falls below) the
presence threshold.
- Spectral (20%): strength of the relevant spectral band.
- Agreement (20%): cross-receiver consensus (if available).
"""
# -- base confidence (0..1) ------------------------------------------
if level == MotionLevel.ABSENT:
# Confidence in absence increases as variance shrinks relative to threshold
if self._var_thresh > 0:
base = max(0.0, 1.0 - variance / self._var_thresh)
else:
base = 1.0
else:
# Confidence in presence increases as variance exceeds threshold
ratio = variance / self._var_thresh if self._var_thresh > 0 else 10.0
base = min(1.0, ratio)
# -- spectral confidence (0..1) --------------------------------------
if level == MotionLevel.ACTIVE:
spectral = min(1.0, motion_energy / max(self._motion_thresh, 1e-12))
elif level == MotionLevel.PRESENT_STILL:
# For still, breathing band energy is more relevant
spectral = min(1.0, breathing_energy / max(self._motion_thresh, 1e-12))
else:
spectral = 1.0 # No spectral requirement for absence
# -- cross-receiver agreement (0..1) ---------------------------------
agreement = 1.0 # default: single receiver
if other_results:
same_level = sum(
1 for r in other_results if r.motion_level == level
)
agreement = (same_level + 1) / (len(other_results) + 1)
# Weighted combination
confidence = 0.6 * base + 0.2 * spectral + 0.2 * agreement
return max(0.0, min(1.0, confidence))
archive/v1/src/sensing/feature_extractor.py
+331
View File
@@ -0,0 +1,331 @@
"""
Signal feature extraction from RSSI time series.
Extracts both time-domain statistical features and frequency-domain spectral
features using real mathematics (scipy.fft, scipy.stats). Also implements
CUSUM change-point detection for abrupt RSSI transitions.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from typing import List, Optional, Tuple
import numpy as np
from numpy.typing import NDArray
from scipy import fft as scipy_fft
from scipy import stats as scipy_stats
from v1.src.sensing.rssi_collector import WifiSample
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Feature dataclass
# ---------------------------------------------------------------------------
@dataclass
class RssiFeatures:
"""Container for all extracted RSSI features."""
# -- time-domain --------------------------------------------------------
mean: float = 0.0
variance: float = 0.0
std: float = 0.0
skewness: float = 0.0
kurtosis: float = 0.0
range: float = 0.0
iqr: float = 0.0 # inter-quartile range
# -- frequency-domain ---------------------------------------------------
dominant_freq_hz: float = 0.0
breathing_band_power: float = 0.0 # 0.1 - 0.5 Hz
motion_band_power: float = 0.0 # 0.5 - 3.0 Hz
total_spectral_power: float = 0.0
# -- change-point -------------------------------------------------------
change_points: List[int] = field(default_factory=list)
n_change_points: int = 0
# -- metadata -----------------------------------------------------------
n_samples: int = 0
duration_seconds: float = 0.0
sample_rate_hz: float = 0.0
# ---------------------------------------------------------------------------
# Feature extractor
# ---------------------------------------------------------------------------
class RssiFeatureExtractor:
"""
Extract time-domain and frequency-domain features from an RSSI time series.
Parameters
----------
window_seconds : float
Length of the analysis window in seconds (default 30).
cusum_threshold : float
CUSUM threshold for change-point detection (default 3.0 standard deviations
of the signal).
cusum_drift : float
CUSUM drift allowance (default 0.5 standard deviations).
"""
def __init__(
self,
window_seconds: float = 30.0,
cusum_threshold: float = 3.0,
cusum_drift: float = 0.5,
) -> None:
self._window_seconds = window_seconds
self._cusum_threshold = cusum_threshold
self._cusum_drift = cusum_drift
@property
def window_seconds(self) -> float:
return self._window_seconds
def extract(self, samples: List[WifiSample]) -> RssiFeatures:
"""
Extract features from a list of WifiSample objects.
Only the most recent ``window_seconds`` of data are used.
At least 4 samples are required for meaningful features.
"""
if len(samples) < 4:
logger.warning(
"Not enough samples for feature extraction (%d < 4)", len(samples)
)
return RssiFeatures(n_samples=len(samples))
# Trim to window
samples = self._trim_to_window(samples)
if len(samples) < 4:
return RssiFeatures(n_samples=len(samples))
rssi = np.array([s.rssi_dbm for s in samples], dtype=np.float64)
timestamps = np.array([s.timestamp for s in samples], dtype=np.float64)
# Estimate sample rate from actual timestamps
dt = np.diff(timestamps)
if len(dt) == 0 or np.mean(dt) <= 0:
sample_rate = 10.0 # fallback
else:
sample_rate = 1.0 / np.mean(dt)
duration = timestamps[-1] - timestamps[0] if len(timestamps) > 1 else 0.0
# Build features
features = RssiFeatures(
n_samples=len(rssi),
duration_seconds=float(duration),
sample_rate_hz=float(sample_rate),
)
self._compute_time_domain(rssi, features)
self._compute_frequency_domain(rssi, sample_rate, features)
self._compute_change_points(rssi, features)
return features
def extract_from_array(
self, rssi: NDArray[np.float64], sample_rate_hz: float
) -> RssiFeatures:
"""
Extract features directly from a numpy array (useful for testing).
Parameters
----------
rssi : ndarray
1-D array of RSSI values in dBm.
sample_rate_hz : float
Sampling rate in Hz.
"""
if len(rssi) < 4:
return RssiFeatures(n_samples=len(rssi))
duration = len(rssi) / sample_rate_hz
features = RssiFeatures(
n_samples=len(rssi),
duration_seconds=float(duration),
sample_rate_hz=float(sample_rate_hz),
)
self._compute_time_domain(rssi, features)
self._compute_frequency_domain(rssi, sample_rate_hz, features)
self._compute_change_points(rssi, features)
return features
# -- window trimming -----------------------------------------------------
def _trim_to_window(self, samples: List[WifiSample]) -> List[WifiSample]:
"""Keep only samples within the most recent ``window_seconds``."""
if not samples:
return samples
latest_ts = samples[-1].timestamp
cutoff = latest_ts - self._window_seconds
trimmed = [s for s in samples if s.timestamp >= cutoff]
return trimmed
# -- time-domain ---------------------------------------------------------
@staticmethod
def _compute_time_domain(rssi: NDArray[np.float64], features: RssiFeatures) -> None:
features.mean = float(np.mean(rssi))
features.variance = float(np.var(rssi, ddof=1)) if len(rssi) > 1 else 0.0
features.std = float(np.std(rssi, ddof=1)) if len(rssi) > 1 else 0.0
features.range = float(np.ptp(rssi))
# Guard against constant signals where higher moments are undefined
if features.std < 1e-12:
features.skewness = 0.0
features.kurtosis = 0.0
else:
features.skewness = float(scipy_stats.skew(rssi, bias=False)) if len(rssi) > 2 else 0.0
features.kurtosis = float(scipy_stats.kurtosis(rssi, bias=False)) if len(rssi) > 3 else 0.0
q75, q25 = np.percentile(rssi, [75, 25])
features.iqr = float(q75 - q25)
# -- frequency-domain ----------------------------------------------------
@staticmethod
def _compute_frequency_domain(
rssi: NDArray[np.float64],
sample_rate: float,
features: RssiFeatures,
) -> None:
"""Compute one-sided FFT power spectrum and extract band powers."""
n = len(rssi)
if n < 4:
return
# Remove DC (subtract mean)
signal = rssi - np.mean(rssi)
# Apply Hann window to reduce spectral leakage
window = np.hanning(n)
windowed = signal * window
# Compute real FFT
fft_vals = scipy_fft.rfft(windowed)
freqs = scipy_fft.rfftfreq(n, d=1.0 / sample_rate)
# Power spectral density (magnitude squared, normalised by N)
psd = (np.abs(fft_vals) ** 2) / n
# Skip DC component (index 0)
if len(freqs) > 1:
freqs_no_dc = freqs[1:]
psd_no_dc = psd[1:]
else:
return
# Total spectral power
features.total_spectral_power = float(np.sum(psd_no_dc))
# Dominant frequency
if len(psd_no_dc) > 0:
peak_idx = int(np.argmax(psd_no_dc))
features.dominant_freq_hz = float(freqs_no_dc[peak_idx])
# Band powers
features.breathing_band_power = float(
_band_power(freqs_no_dc, psd_no_dc, 0.1, 0.5)
)
features.motion_band_power = float(
_band_power(freqs_no_dc, psd_no_dc, 0.5, 3.0)
)
# -- change-point detection (CUSUM) --------------------------------------
def _compute_change_points(
self, rssi: NDArray[np.float64], features: RssiFeatures
) -> None:
"""
Detect change points using the CUSUM algorithm.
The CUSUM statistic tracks cumulative deviations from the mean,
flagging points where the signal mean shifts abruptly.
"""
if len(rssi) < 4:
return
mean_val = np.mean(rssi)
std_val = np.std(rssi, ddof=1)
if std_val < 1e-12:
features.change_points = []
features.n_change_points = 0
return
threshold = self._cusum_threshold * std_val
drift = self._cusum_drift * std_val
change_points = cusum_detect(rssi, mean_val, threshold, drift)
features.change_points = change_points
features.n_change_points = len(change_points)
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def _band_power(
freqs: NDArray[np.float64],
psd: NDArray[np.float64],
low_hz: float,
high_hz: float,
) -> float:
"""Sum PSD within a frequency band [low_hz, high_hz]."""
mask = (freqs >= low_hz) & (freqs <= high_hz)
return float(np.sum(psd[mask]))
def cusum_detect(
signal: NDArray[np.float64],
target: float,
threshold: float,
drift: float,
) -> List[int]:
"""
CUSUM (cumulative sum) change-point detection.
Detects both upward and downward shifts in the signal mean.
Parameters
----------
signal : ndarray
The 1-D signal to analyse.
target : float
Expected mean of the signal.
threshold : float
Decision threshold for declaring a change point.
drift : float
Allowable drift before accumulating deviation.
Returns
-------
list of int
Indices where change points were detected.
"""
n = len(signal)
s_pos = 0.0
s_neg = 0.0
change_points: List[int] = []
for i in range(n):
deviation = signal[i] - target
s_pos = max(0.0, s_pos + deviation - drift)
s_neg = max(0.0, s_neg - deviation - drift)
if s_pos > threshold or s_neg > threshold:
change_points.append(i)
# Reset after detection to find subsequent changes
s_pos = 0.0
s_neg = 0.0
return change_points
archive/v1/src/sensing/mac_wifi.swift
+34
View File
@@ -0,0 +1,34 @@
import Foundation
import CoreWLAN
// Output format: JSON lines for easy parsing by Python
// {"timestamp": 1234567.89, "rssi": -50, "noise": -90, "tx_rate": 866.0}
func main() {
guard let interface = CWWiFiClient.shared().interface() else {
fputs("{\"error\": \"No WiFi interface found\"}\n", stderr)
exit(1)
}
// Flush stdout automatically to prevent buffering issues with Python subprocess
setbuf(stdout, nil)
// Run at ~10Hz
let interval: TimeInterval = 0.1
while true {
let timestamp = Date().timeIntervalSince1970
let rssi = interface.rssiValue()
let noise = interface.noiseMeasurement()
let txRate = interface.transmitRate()
let json = """
{"timestamp": \(timestamp), "rssi": \(rssi), "noise": \(noise), "tx_rate": \(txRate)}
"""
print(json)
Thread.sleep(forTimeInterval: interval)
}
}
main()
archive/v1/src/sensing/rssi_collector.py
+843
View File
@@ -0,0 +1,843 @@
"""
RSSI data collection from Linux WiFi interfaces.
Provides two concrete collectors:
- LinuxWifiCollector: reads real RSSI from /proc/net/wireless and iw commands
- SimulatedCollector: produces deterministic synthetic signals for testing
Both share the same WifiSample dataclass and thread-safe ring buffer.
"""
from __future__ import annotations
import logging
import math
import os
import platform
import re
import subprocess
import threading
import time
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, List, Optional, Protocol, Union
import numpy as np
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Data types
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class WifiSample:
"""A single WiFi measurement sample."""
timestamp: float # UNIX epoch seconds (time.time())
rssi_dbm: float # Received signal strength in dBm
noise_dbm: float # Noise floor in dBm
link_quality: float # Link quality 0-1 (normalised)
tx_bytes: int # Cumulative TX bytes
rx_bytes: int # Cumulative RX bytes
retry_count: int # Cumulative retry count
interface: str # WiFi interface name
# ---------------------------------------------------------------------------
# Thread-safe ring buffer
# ---------------------------------------------------------------------------
class RingBuffer:
"""Thread-safe fixed-size ring buffer for WifiSample objects."""
def __init__(self, max_size: int) -> None:
self._buf: Deque[WifiSample] = deque(maxlen=max_size)
self._lock = threading.Lock()
def append(self, sample: WifiSample) -> None:
with self._lock:
self._buf.append(sample)
def get_all(self) -> List[WifiSample]:
"""Return a snapshot of all samples (oldest first)."""
with self._lock:
return list(self._buf)
def get_last_n(self, n: int) -> List[WifiSample]:
"""Return the most recent *n* samples."""
with self._lock:
items = list(self._buf)
return items[-n:] if n < len(items) else items
def __len__(self) -> int:
with self._lock:
return len(self._buf)
def clear(self) -> None:
with self._lock:
self._buf.clear()
# ---------------------------------------------------------------------------
# Collector protocol
# ---------------------------------------------------------------------------
class WifiCollector(Protocol):
"""Protocol that all WiFi collectors must satisfy."""
def start(self) -> None: ...
def stop(self) -> None: ...
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]: ...
@property
def sample_rate_hz(self) -> float: ...
# ---------------------------------------------------------------------------
# Linux WiFi collector (real hardware)
# ---------------------------------------------------------------------------
class LinuxWifiCollector:
"""
Collects real RSSI data from a Linux WiFi interface.
Data sources:
- /proc/net/wireless (RSSI, noise, link quality)
- iw dev <iface> station dump (TX/RX bytes, retry count)
Parameters
----------
interface : str
WiFi interface name, e.g. ``"wlan0"``.
sample_rate_hz : float
Target sampling rate in Hz (default 10).
buffer_seconds : int
How many seconds of history to keep in the ring buffer (default 120).
"""
def __init__(
self,
interface: str = "wlan0",
sample_rate_hz: float = 10.0,
buffer_seconds: int = 120,
) -> None:
self._interface = interface
self._rate = sample_rate_hz
self._buffer = RingBuffer(max_size=int(sample_rate_hz * buffer_seconds))
self._running = False
self._thread: Optional[threading.Thread] = None
# -- public API ----------------------------------------------------------
@property
def sample_rate_hz(self) -> float:
return self._rate
def start(self) -> None:
"""Start the background sampling thread."""
if self._running:
return
self._validate_interface()
self._running = True
self._thread = threading.Thread(
target=self._sample_loop, daemon=True, name="wifi-rssi-collector"
)
self._thread.start()
logger.info(
"LinuxWifiCollector started on %s at %.1f Hz",
self._interface,
self._rate,
)
def stop(self) -> None:
"""Stop the background sampling thread."""
self._running = False
if self._thread is not None:
self._thread.join(timeout=2.0)
self._thread = None
logger.info("LinuxWifiCollector stopped")
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]:
"""
Return collected samples.
Parameters
----------
n : int or None
If given, return only the most recent *n* samples.
"""
if n is not None:
return self._buffer.get_last_n(n)
return self._buffer.get_all()
def collect_once(self) -> WifiSample:
"""Collect a single sample right now (blocking)."""
return self._read_sample()
# -- availability check --------------------------------------------------
@classmethod
def is_available(cls, interface: str = "wlan0") -> tuple[bool, str]:
"""Check if Linux WiFi collection is possible without raising.
Returns
-------
(available, reason) : tuple[bool, str]
``available`` is True when /proc/net/wireless exists and lists
the requested interface. ``reason`` is a human-readable
explanation when unavailable.
"""
if not os.path.exists("/proc/net/wireless"):
return False, (
"/proc/net/wireless not found. "
"This environment has no Linux wireless subsystem "
"(common in Docker, WSL, or headless servers)."
)
try:
with open("/proc/net/wireless", "r") as f:
content = f.read()
except OSError as exc:
return False, f"Cannot read /proc/net/wireless: {exc}"
if interface not in content:
names = cls._parse_interface_names(content)
return False, (
f"Interface '{interface}' not listed in /proc/net/wireless. "
f"Available: {names or '(none)'}. "
f"Ensure the interface is up and associated with an AP."
)
return True, "ok"
# -- internals -----------------------------------------------------------
def _validate_interface(self) -> None:
"""Check that the interface exists on this machine."""
available, reason = self.is_available(self._interface)
if not available:
raise RuntimeError(reason)
@staticmethod
def _parse_interface_names(proc_content: str) -> List[str]:
"""Extract interface names from /proc/net/wireless content."""
names: List[str] = []
for line in proc_content.splitlines()[2:]: # skip header lines
parts = line.split(":")
if len(parts) >= 2:
names.append(parts[0].strip())
return names
def _sample_loop(self) -> None:
interval = 1.0 / self._rate
while self._running:
t0 = time.monotonic()
try:
sample = self._read_sample()
self._buffer.append(sample)
except Exception:
logger.exception("Error reading WiFi sample")
elapsed = time.monotonic() - t0
sleep_time = max(0.0, interval - elapsed)
if sleep_time > 0:
time.sleep(sleep_time)
def _read_sample(self) -> WifiSample:
"""Read one sample from the OS."""
rssi, noise, quality = self._read_proc_wireless()
tx_bytes, rx_bytes, retries = self._read_iw_station()
return WifiSample(
timestamp=time.time(),
rssi_dbm=rssi,
noise_dbm=noise,
link_quality=quality,
tx_bytes=tx_bytes,
rx_bytes=rx_bytes,
retry_count=retries,
interface=self._interface,
)
def _read_proc_wireless(self) -> tuple[float, float, float]:
"""Parse /proc/net/wireless for the configured interface."""
try:
with open("/proc/net/wireless", "r") as f:
for line in f:
if self._interface in line:
# Format: iface: status quality signal noise ...
parts = line.split()
# parts[0] = "wlan0:", parts[2]=quality, parts[3]=signal, parts[4]=noise
quality_raw = float(parts[2].rstrip("."))
signal_raw = float(parts[3].rstrip("."))
noise_raw = float(parts[4].rstrip("."))
# Normalise quality to 0..1 (max is typically 70)
quality = min(1.0, max(0.0, quality_raw / 70.0))
return signal_raw, noise_raw, quality
except (FileNotFoundError, IndexError, ValueError) as exc:
raise RuntimeError(
f"Failed to read /proc/net/wireless for {self._interface}: {exc}"
) from exc
raise RuntimeError(
f"Interface {self._interface} not found in /proc/net/wireless"
)
def _read_iw_station(self) -> tuple[int, int, int]:
"""Run ``iw dev <iface> station dump`` and parse TX/RX/retries."""
try:
result = subprocess.run(
["iw", "dev", self._interface, "station", "dump"],
capture_output=True,
text=True,
timeout=2.0,
)
text = result.stdout
tx_bytes = self._extract_int(text, r"tx bytes:\s*(\d+)")
rx_bytes = self._extract_int(text, r"rx bytes:\s*(\d+)")
retries = self._extract_int(text, r"tx retries:\s*(\d+)")
return tx_bytes, rx_bytes, retries
except (FileNotFoundError, subprocess.TimeoutExpired):
# iw not installed or timed out -- degrade gracefully
return 0, 0, 0
@staticmethod
def _extract_int(text: str, pattern: str) -> int:
m = re.search(pattern, text)
return int(m.group(1)) if m else 0
# ---------------------------------------------------------------------------
# Simulated collector (deterministic, for testing)
# ---------------------------------------------------------------------------
class SimulatedCollector:
"""
Deterministic simulated WiFi collector for testing.
Generates a synthetic RSSI signal composed of:
- A constant baseline (-50 dBm default)
- An optional sinusoidal component (configurable frequency/amplitude)
- Optional step-change injection (for change-point testing)
- Deterministic noise from a seeded PRNG
This is explicitly a test/development tool and makes no attempt to
appear as real hardware.
Parameters
----------
seed : int
Random seed for deterministic output.
sample_rate_hz : float
Target sampling rate in Hz (default 10).
buffer_seconds : int
Ring buffer capacity in seconds (default 120).
baseline_dbm : float
RSSI baseline in dBm (default -50).
sine_freq_hz : float
Frequency of the sinusoidal RSSI component (default 0.3 Hz, breathing band).
sine_amplitude_dbm : float
Amplitude of the sinusoidal component (default 2.0 dBm).
noise_std_dbm : float
Standard deviation of additive Gaussian noise (default 0.5 dBm).
step_change_at : float or None
If set, inject a step change of ``step_change_dbm`` at this time offset
(seconds from start).
step_change_dbm : float
Magnitude of the step change (default -10 dBm).
"""
def __init__(
self,
seed: int = 42,
sample_rate_hz: float = 10.0,
buffer_seconds: int = 120,
baseline_dbm: float = -50.0,
sine_freq_hz: float = 0.3,
sine_amplitude_dbm: float = 2.0,
noise_std_dbm: float = 0.5,
step_change_at: Optional[float] = None,
step_change_dbm: float = -10.0,
) -> None:
self._rate = sample_rate_hz
self._buffer = RingBuffer(max_size=int(sample_rate_hz * buffer_seconds))
self._rng = np.random.default_rng(seed)
self._baseline = baseline_dbm
self._sine_freq = sine_freq_hz
self._sine_amp = sine_amplitude_dbm
self._noise_std = noise_std_dbm
self._step_at = step_change_at
self._step_dbm = step_change_dbm
self._running = False
self._thread: Optional[threading.Thread] = None
self._start_time: float = 0.0
self._sample_index: int = 0
# -- public API ----------------------------------------------------------
@property
def sample_rate_hz(self) -> float:
return self._rate
def start(self) -> None:
if self._running:
return
self._running = True
self._start_time = time.time()
self._sample_index = 0
self._thread = threading.Thread(
target=self._sample_loop, daemon=True, name="sim-rssi-collector"
)
self._thread.start()
logger.info("SimulatedCollector started at %.1f Hz (seed reused from init)", self._rate)
def stop(self) -> None:
self._running = False
if self._thread is not None:
self._thread.join(timeout=2.0)
self._thread = None
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]:
if n is not None:
return self._buffer.get_last_n(n)
return self._buffer.get_all()
def generate_samples(self, duration_seconds: float) -> List[WifiSample]:
"""
Generate a batch of samples without the background thread.
Useful for unit tests that need a known signal without timing jitter.
Parameters
----------
duration_seconds : float
How many seconds of signal to produce.
Returns
-------
list of WifiSample
"""
n_samples = int(duration_seconds * self._rate)
samples: List[WifiSample] = []
base_time = time.time()
for i in range(n_samples):
t = i / self._rate
sample = self._make_sample(base_time + t, t, i)
samples.append(sample)
return samples
# -- internals -----------------------------------------------------------
def _sample_loop(self) -> None:
interval = 1.0 / self._rate
while self._running:
t0 = time.monotonic()
now = time.time()
t_offset = now - self._start_time
sample = self._make_sample(now, t_offset, self._sample_index)
self._buffer.append(sample)
self._sample_index += 1
elapsed = time.monotonic() - t0
sleep_time = max(0.0, interval - elapsed)
if sleep_time > 0:
time.sleep(sleep_time)
def _make_sample(self, timestamp: float, t_offset: float, index: int) -> WifiSample:
"""Build one deterministic sample."""
# Sinusoidal component
sine = self._sine_amp * math.sin(2.0 * math.pi * self._sine_freq * t_offset)
# Deterministic Gaussian noise (uses the seeded RNG)
noise = self._rng.normal(0.0, self._noise_std)
# Step change
step = 0.0
if self._step_at is not None and t_offset >= self._step_at:
step = self._step_dbm
rssi = self._baseline + sine + noise + step
return WifiSample(
timestamp=timestamp,
rssi_dbm=float(rssi),
noise_dbm=-95.0,
link_quality=max(0.0, min(1.0, (rssi + 100.0) / 60.0)),
tx_bytes=index * 1500,
rx_bytes=index * 3000,
retry_count=max(0, index // 100),
interface="sim0",
)
# ---------------------------------------------------------------------------
# Windows WiFi collector (real hardware via netsh)
# ---------------------------------------------------------------------------
class WindowsWifiCollector:
"""
Collects real RSSI data from a Windows WiFi interface.
Data source: ``netsh wlan show interfaces`` which provides RSSI in dBm,
signal quality percentage, channel, band, and connection state.
Parameters
----------
interface : str
WiFi interface name (default ``"Wi-Fi"``). Must match the ``Name``
field shown by ``netsh wlan show interfaces``.
sample_rate_hz : float
Target sampling rate in Hz (default 2.0). Windows ``netsh`` is slow
(~200-400ms per call) so rates above 2 Hz may not be achievable.
buffer_seconds : int
Ring buffer capacity in seconds (default 120).
"""
def __init__(
self,
interface: str = "Wi-Fi",
sample_rate_hz: float = 2.0,
buffer_seconds: int = 120,
) -> None:
self._interface = interface
self._rate = sample_rate_hz
self._buffer = RingBuffer(max_size=int(sample_rate_hz * buffer_seconds))
self._running = False
self._thread: Optional[threading.Thread] = None
self._cumulative_tx: int = 0
self._cumulative_rx: int = 0
# -- public API ----------------------------------------------------------
@property
def sample_rate_hz(self) -> float:
return self._rate
def start(self) -> None:
if self._running:
return
self._validate_interface()
self._running = True
self._thread = threading.Thread(
target=self._sample_loop, daemon=True, name="win-rssi-collector"
)
self._thread.start()
logger.info(
"WindowsWifiCollector started on '%s' at %.1f Hz",
self._interface,
self._rate,
)
def stop(self) -> None:
self._running = False
if self._thread is not None:
self._thread.join(timeout=2.0)
self._thread = None
logger.info("WindowsWifiCollector stopped")
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]:
if n is not None:
return self._buffer.get_last_n(n)
return self._buffer.get_all()
def collect_once(self) -> WifiSample:
return self._read_sample()
# -- internals -----------------------------------------------------------
def _validate_interface(self) -> None:
try:
result = subprocess.run(
["netsh", "wlan", "show", "interfaces"],
capture_output=True, text=True, timeout=5.0,
)
if self._interface not in result.stdout:
raise RuntimeError(
f"WiFi interface '{self._interface}' not found. "
f"Check 'netsh wlan show interfaces' for the correct name."
)
if "disconnected" in result.stdout.lower().split(self._interface.lower())[1][:200]:
raise RuntimeError(
f"WiFi interface '{self._interface}' is disconnected. "
f"Connect to a WiFi network first."
)
except FileNotFoundError:
raise RuntimeError(
"netsh not found. This collector requires Windows."
)
def _sample_loop(self) -> None:
interval = 1.0 / self._rate
while self._running:
t0 = time.monotonic()
try:
sample = self._read_sample()
self._buffer.append(sample)
except Exception:
logger.exception("Error reading WiFi sample")
elapsed = time.monotonic() - t0
sleep_time = max(0.0, interval - elapsed)
if sleep_time > 0:
time.sleep(sleep_time)
def _read_sample(self) -> WifiSample:
result = subprocess.run(
["netsh", "wlan", "show", "interfaces"],
capture_output=True, text=True, timeout=5.0,
)
rssi = -80.0
signal_pct = 0.0
for line in result.stdout.splitlines():
stripped = line.strip()
# "Rssi" line contains the raw dBm value (available on Win10+)
if stripped.lower().startswith("rssi"):
try:
rssi = float(stripped.split(":")[1].strip())
except (IndexError, ValueError):
pass
# "Signal" line contains percentage (always available)
elif stripped.lower().startswith("signal"):
try:
pct_str = stripped.split(":")[1].strip().rstrip("%")
signal_pct = float(pct_str)
# If RSSI line was missing, estimate from percentage
# Signal% roughly maps: 100% ≈ -30 dBm, 0% ≈ -90 dBm
except (IndexError, ValueError):
pass
# Normalise link quality from signal percentage
link_quality = signal_pct / 100.0
# Estimate noise floor (Windows doesn't expose it directly)
noise_dbm = -95.0
# Track cumulative bytes (not available from netsh; increment synthetic counter)
self._cumulative_tx += 1500
self._cumulative_rx += 3000
return WifiSample(
timestamp=time.time(),
rssi_dbm=rssi,
noise_dbm=noise_dbm,
link_quality=link_quality,
tx_bytes=self._cumulative_tx,
rx_bytes=self._cumulative_rx,
retry_count=0,
interface=self._interface,
)
# ---------------------------------------------------------------------------
# macOS WiFi collector (real hardware via Swift CoreWLAN utility)
# ---------------------------------------------------------------------------
class MacosWifiCollector:
"""
Collects real RSSI data from a macOS WiFi interface using a Swift utility.
Data source: A small compiled Swift binary (`mac_wifi`) that polls the
CoreWLAN `CWWiFiClient.shared().interface()` at a high rate.
"""
def __init__(
self,
sample_rate_hz: float = 10.0,
buffer_seconds: int = 120,
) -> None:
self._rate = sample_rate_hz
self._buffer = RingBuffer(max_size=int(sample_rate_hz * buffer_seconds))
self._running = False
self._thread: Optional[threading.Thread] = None
self._process: Optional[subprocess.Popen] = None
self._interface = "en0" # CoreWLAN automatically targets the active Wi-Fi interface
# Compile the Swift utility if the binary doesn't exist
import os
base_dir = os.path.dirname(os.path.abspath(__file__))
self.swift_src = os.path.join(base_dir, "mac_wifi.swift")
self.swift_bin = os.path.join(base_dir, "mac_wifi")
# -- public API ----------------------------------------------------------
@property
def sample_rate_hz(self) -> float:
return self._rate
def start(self) -> None:
if self._running:
return
# Ensure binary exists
import os
if not os.path.exists(self.swift_bin):
logger.info("Compiling mac_wifi.swift to %s", self.swift_bin)
try:
subprocess.run(["swiftc", "-O", "-o", self.swift_bin, self.swift_src], check=True, capture_output=True)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Failed to compile macOS WiFi utility: {e.stderr.decode('utf-8')}")
except FileNotFoundError:
raise RuntimeError("swiftc is not installed. Please install Xcode Command Line Tools to use native macOS WiFi sensing.")
self._running = True
self._thread = threading.Thread(
target=self._sample_loop, daemon=True, name="mac-rssi-collector"
)
self._thread.start()
logger.info("MacosWifiCollector started at %.1f Hz", self._rate)
def stop(self) -> None:
self._running = False
if self._process:
self._process.terminate()
try:
self._process.wait(timeout=1.0)
except subprocess.TimeoutExpired:
self._process.kill()
self._process = None
if self._thread is not None:
self._thread.join(timeout=2.0)
self._thread = None
logger.info("MacosWifiCollector stopped")
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]:
if n is not None:
return self._buffer.get_last_n(n)
return self._buffer.get_all()
# -- internals -----------------------------------------------------------
def _sample_loop(self) -> None:
import json
# Start the Swift binary
self._process = subprocess.Popen(
[self.swift_bin],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1 # Line buffered
)
while self._running and self._process and self._process.poll() is None:
try:
line = self._process.stdout.readline()
if not line:
continue
line = line.strip()
if not line:
continue
if line.startswith("{"):
data = json.loads(line)
if "error" in data:
logger.error("macOS WiFi utility error: %s", data["error"])
continue
rssi = float(data.get("rssi", -80.0))
noise = float(data.get("noise", -95.0))
link_quality = max(0.0, min(1.0, (rssi + 100.0) / 60.0))
sample = WifiSample(
timestamp=time.time(),
rssi_dbm=rssi,
noise_dbm=noise,
link_quality=link_quality,
tx_bytes=0,
rx_bytes=0,
retry_count=0,
interface=self._interface,
)
self._buffer.append(sample)
except Exception as e:
logger.error("Error reading macOS WiFi stream: %s", e)
time.sleep(1.0)
# Process exited unexpectedly
if self._running:
logger.error("macOS WiFi utility exited unexpectedly. Collector stopped.")
self._running = False
# ---------------------------------------------------------------------------
# Collector factory (ADR-049)
# ---------------------------------------------------------------------------
CollectorType = Union[LinuxWifiCollector, WindowsWifiCollector, MacosWifiCollector, SimulatedCollector]
def create_collector(
preferred: str = "auto",
interface: str = "wlan0",
sample_rate_hz: float = 10.0,
) -> CollectorType:
"""Create the best available WiFi collector for the current platform.
Resolution order (when ``preferred="auto"``):
1. Platform-native WiFi:
- Linux: LinuxWifiCollector (requires /proc/net/wireless + active interface)
- Windows: WindowsWifiCollector (netsh wlan)
- macOS: MacosWifiCollector (CoreWLAN)
2. SimulatedCollector (always available)
This function never raises -- it always returns a usable collector.
Parameters
----------
preferred : str
``"auto"`` for platform detection, or one of ``"linux"``,
``"windows"``, ``"macos"``, ``"simulated"`` to force a specific
collector.
interface : str
WiFi interface name (Linux/Windows only).
sample_rate_hz : float
Target sampling rate.
"""
_VALID_PREFERRED = {"auto", "linux", "windows", "macos", "simulated"}
if preferred not in _VALID_PREFERRED:
logger.warning(
"WiFi collector: unknown preferred=%r (valid: %s). Falling back to auto.",
preferred, ", ".join(sorted(_VALID_PREFERRED)),
)
preferred = "auto"
system = platform.system()
if preferred == "auto":
if system == "Linux":
available, reason = LinuxWifiCollector.is_available(interface)
if available:
logger.info("WiFi collector: using LinuxWifiCollector on %s", interface)
return LinuxWifiCollector(interface=interface, sample_rate_hz=sample_rate_hz)
logger.warning("WiFi collector: LinuxWifiCollector unavailable (%s).", reason)
elif system == "Windows":
try:
win_iface = interface if interface != "wlan0" else "Wi-Fi"
collector = WindowsWifiCollector(interface=win_iface, sample_rate_hz=min(sample_rate_hz, 2.0))
collector.collect_once()
logger.info("WiFi collector: using WindowsWifiCollector on '%s'", interface)
return collector
except Exception as exc:
logger.warning("WiFi collector: WindowsWifiCollector unavailable (%s).", exc)
elif system == "Darwin":
try:
collector = MacosWifiCollector(sample_rate_hz=sample_rate_hz)
logger.info("WiFi collector: using MacosWifiCollector")
return collector
except Exception as exc:
logger.warning("WiFi collector: MacosWifiCollector unavailable (%s).", exc)
elif preferred == "linux":
return LinuxWifiCollector(interface=interface, sample_rate_hz=sample_rate_hz)
elif preferred == "windows":
return WindowsWifiCollector(interface=interface, sample_rate_hz=min(sample_rate_hz, 2.0))
elif preferred == "macos":
return MacosWifiCollector(sample_rate_hz=sample_rate_hz)
elif preferred == "simulated":
return SimulatedCollector(seed=42, sample_rate_hz=sample_rate_hz)
logger.info(
"WiFi collector: falling back to SimulatedCollector. "
"For real sensing, connect ESP32 nodes via UDP:5005 or install platform WiFi drivers."
)
return SimulatedCollector(seed=42, sample_rate_hz=sample_rate_hz)
archive/v1/src/sensing/ws_server.py
+519
View File
@@ -0,0 +1,519 @@
"""
WebSocket sensing server.
Lightweight asyncio server that bridges the WiFi sensing pipeline to the
browser UI. Runs the RSSI feature extractor + classifier on a 500 ms
tick and broadcasts JSON frames to all connected WebSocket clients on
``ws://localhost:8765``.
Usage
-----
pip install websockets
python -m v1.src.sensing.ws_server # or python v1/src/sensing/ws_server.py
Data sources (tried in order):
1. ESP32 CSI over UDP port 5005 (ADR-018 binary frames)
2. Windows WiFi RSSI via netsh
3. Linux WiFi RSSI via /proc/net/wireless
4. Simulated collector (fallback)
"""
from __future__ import annotations
import asyncio
import json
import logging
import math
import signal
import socket
import struct
import sys
import threading
import time
from collections import deque
from typing import Dict, List, Optional, Set
import numpy as np
# Sensing pipeline imports
from v1.src.sensing.rssi_collector import (
WifiSample,
RingBuffer,
)
from v1.src.sensing.feature_extractor import RssiFeatureExtractor, RssiFeatures
from v1.src.sensing.classifier import MotionLevel, PresenceClassifier, SensingResult
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
HOST = "localhost"
PORT = 8765
TICK_INTERVAL = 0.5 # seconds between broadcasts
SIGNAL_FIELD_GRID = 20 # NxN grid for signal field visualization
ESP32_UDP_PORT = 5005
# ---------------------------------------------------------------------------
# ESP32 UDP Collector — reads ADR-018 binary frames
# ---------------------------------------------------------------------------
class Esp32UdpCollector:
"""
Collects real CSI data from ESP32 nodes via UDP (ADR-018 binary format).
Parses I/Q pairs, computes mean amplitude per frame, and stores it as
an RSSI-equivalent value in the standard WifiSample ring buffer so the
existing feature extractor and classifier work unchanged.
Also keeps the last parsed CSI frame for the UI to show subcarrier data.
"""
# ADR-018 header: magic(4) node_id(1) n_ant(1) n_sc(2) freq(4) seq(4) rssi(1) noise(1) reserved(2)
MAGIC = 0xC5110001
HEADER_SIZE = 20
HEADER_FMT = '<IBBHIIBB2x'
def __init__(
self,
bind_addr: str = "0.0.0.0",
port: int = ESP32_UDP_PORT,
sample_rate_hz: float = 10.0,
buffer_seconds: int = 120,
) -> None:
self._bind = bind_addr
self._port = port
self._rate = sample_rate_hz
self._buffer = RingBuffer(max_size=int(sample_rate_hz * buffer_seconds))
self._running = False
self._thread: Optional[threading.Thread] = None
self._sock: Optional[socket.socket] = None
# Last CSI frame for enhanced UI
self.last_csi: Optional[Dict] = None
self._frames_received = 0
@property
def sample_rate_hz(self) -> float:
return self._rate
@property
def frames_received(self) -> int:
return self._frames_received
def start(self) -> None:
if self._running:
return
self._sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._sock.settimeout(1.0)
self._sock.bind((self._bind, self._port))
self._running = True
self._thread = threading.Thread(
target=self._recv_loop, daemon=True, name="esp32-udp-collector"
)
self._thread.start()
logger.info("Esp32UdpCollector listening on %s:%d", self._bind, self._port)
def stop(self) -> None:
self._running = False
if self._thread:
self._thread.join(timeout=2.0)
self._thread = None
if self._sock:
self._sock.close()
self._sock = None
logger.info("Esp32UdpCollector stopped (%d frames received)", self._frames_received)
def get_samples(self, n: Optional[int] = None) -> List[WifiSample]:
if n is not None:
return self._buffer.get_last_n(n)
return self._buffer.get_all()
def _recv_loop(self) -> None:
while self._running:
try:
data, addr = self._sock.recvfrom(4096)
self._parse_and_store(data, addr)
except socket.timeout:
continue
except Exception:
if self._running:
logger.exception("Error receiving ESP32 UDP packet")
def _parse_and_store(self, raw: bytes, addr) -> None:
if len(raw) < self.HEADER_SIZE:
return
magic, node_id, n_ant, n_sc, freq_mhz, seq, rssi_u8, noise_u8 = \
struct.unpack_from(self.HEADER_FMT, raw, 0)
if magic != self.MAGIC:
return
rssi = rssi_u8 if rssi_u8 < 128 else rssi_u8 - 256
noise = noise_u8 if noise_u8 < 128 else noise_u8 - 256
# Parse I/Q data if available
iq_count = n_ant * n_sc
iq_bytes_needed = self.HEADER_SIZE + iq_count * 2
amplitude_list = []
if len(raw) >= iq_bytes_needed and iq_count > 0:
iq_raw = struct.unpack_from(f'<{iq_count * 2}b', raw, self.HEADER_SIZE)
i_vals = np.array(iq_raw[0::2], dtype=np.float64)
q_vals = np.array(iq_raw[1::2], dtype=np.float64)
amplitudes = np.sqrt(i_vals ** 2 + q_vals ** 2)
mean_amp = float(np.mean(amplitudes))
amplitude_list = amplitudes.tolist()
else:
mean_amp = 0.0
# Store enhanced CSI info for UI
self.last_csi = {
"node_id": node_id,
"n_antennas": n_ant,
"n_subcarriers": n_sc,
"freq_mhz": freq_mhz,
"sequence": seq,
"rssi_dbm": rssi,
"noise_floor_dbm": noise,
"mean_amplitude": mean_amp,
"amplitude": amplitude_list[:56], # cap for JSON size
"source_addr": f"{addr[0]}:{addr[1]}",
}
# Use RSSI from the ESP32 frame header as the primary signal metric.
# If RSSI is the default -80 placeholder, derive a pseudo-RSSI from
# mean amplitude to keep the feature extractor meaningful.
effective_rssi = float(rssi)
if rssi == -80 and mean_amp > 0:
# Map amplitude (typically 1-20) to dBm range (-70 to -30)
effective_rssi = -70.0 + min(mean_amp, 20.0) * 2.0
sample = WifiSample(
timestamp=time.time(),
rssi_dbm=effective_rssi,
noise_dbm=float(noise),
link_quality=max(0.0, min(1.0, (effective_rssi + 100.0) / 60.0)),
tx_bytes=seq * 1500,
rx_bytes=seq * 3000,
retry_count=0,
interface=f"esp32-node{node_id}",
)
self._buffer.append(sample)
self._frames_received += 1
# ---------------------------------------------------------------------------
# Probe for ESP32 UDP
# ---------------------------------------------------------------------------
def probe_esp32_udp(port: int = ESP32_UDP_PORT, timeout: float = 2.0) -> bool:
"""Return True if an ESP32 is actively streaming on the UDP port."""
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.settimeout(timeout)
try:
sock.bind(("0.0.0.0", port))
data, _ = sock.recvfrom(256)
if len(data) >= 20:
magic = struct.unpack_from('<I', data, 0)[0]
return magic == 0xC5110001
return False
except (socket.timeout, OSError):
return False
finally:
sock.close()
# ---------------------------------------------------------------------------
# Signal field generator
# ---------------------------------------------------------------------------
def generate_signal_field(
features: RssiFeatures,
result: SensingResult,
grid_size: int = SIGNAL_FIELD_GRID,
csi_data: Optional[Dict] = None,
) -> Dict:
"""
Generate a 2-D signal-strength field for the Gaussian splat visualization.
When real CSI amplitude data is available, it modulates the field.
"""
field = np.zeros((grid_size, grid_size), dtype=np.float64)
# Base noise floor
rng = np.random.default_rng(int(abs(features.mean * 100)) % (2**31))
field += rng.uniform(0.02, 0.08, size=(grid_size, grid_size))
cx, cy = grid_size // 2, grid_size // 2
# Radial attenuation from router
for y in range(grid_size):
for x in range(grid_size):
dist = math.sqrt((x - cx) ** 2 + (y - cy) ** 2)
attenuation = max(0.0, 1.0 - dist / (grid_size * 0.7))
field[y, x] += attenuation * 0.3
# If we have real CSI subcarrier amplitudes, paint them along one axis
if csi_data and csi_data.get("amplitude"):
amps = np.array(csi_data["amplitude"][:grid_size], dtype=np.float64)
if len(amps) > 0:
max_a = np.max(amps) if np.max(amps) > 0 else 1.0
norm_amps = amps / max_a
# Spread subcarrier energy as vertical stripes
for ix, a in enumerate(norm_amps):
col = int(ix * grid_size / len(norm_amps))
col = min(col, grid_size - 1)
field[:, col] += a * 0.4
if result.presence_detected:
body_x = cx + int(3 * math.sin(time.time() * 0.2))
body_y = cy + int(2 * math.cos(time.time() * 0.15))
sigma = 2.0 + features.variance * 0.5
for y in range(grid_size):
for x in range(grid_size):
dx = x - body_x
dy = y - body_y
blob = math.exp(-(dx * dx + dy * dy) / (2.0 * sigma * sigma))
intensity = 0.3 + 0.7 * min(1.0, features.motion_band_power * 5)
field[y, x] += blob * intensity
if features.breathing_band_power > 0.01:
breath_phase = math.sin(2 * math.pi * 0.3 * time.time())
breath_radius = 3.0 + breath_phase * 0.8
for y in range(grid_size):
for x in range(grid_size):
dist_body = math.sqrt((x - body_x) ** 2 + (y - body_y) ** 2)
ring = math.exp(-((dist_body - breath_radius) ** 2) / 1.5)
field[y, x] += ring * features.breathing_band_power * 2
field = np.clip(field, 0.0, 1.0)
return {
"grid_size": [grid_size, 1, grid_size],
"values": field.flatten().tolist(),
}
# ---------------------------------------------------------------------------
# WebSocket server
# ---------------------------------------------------------------------------
class SensingWebSocketServer:
"""Async WebSocket server that broadcasts sensing updates."""
def __init__(self) -> None:
self.clients: Set = set()
self.collector = None
self.extractor = RssiFeatureExtractor(window_seconds=10.0)
self.classifier = PresenceClassifier()
self.source: str = "unknown"
self._running = False
def _create_collector(self):
"""Auto-detect data source: ESP32 UDP > platform WiFi > simulated.
Uses the ``create_collector`` factory (ADR-049) for platform WiFi
detection, which never raises and logs actionable fallback messages.
"""
from .rssi_collector import create_collector
# 1. Try ESP32 UDP first
print(" Probing for ESP32 on UDP :5005 ...")
if probe_esp32_udp(ESP32_UDP_PORT, timeout=2.0):
logger.info("ESP32 CSI stream detected on UDP :%d", ESP32_UDP_PORT)
self.source = "esp32"
return Esp32UdpCollector(port=ESP32_UDP_PORT, sample_rate_hz=10.0)
# 2. Platform-specific WiFi (auto-detect with graceful fallback)
collector = create_collector(preferred="auto", sample_rate_hz=10.0)
# Map collector class to source label
source_map = {
"LinuxWifiCollector": "linux_wifi",
"WindowsWifiCollector": "windows_wifi",
"MacosWifiCollector": "macos_wifi",
"SimulatedCollector": "simulated",
}
self.source = source_map.get(type(collector).__name__, "unknown")
return collector
def _build_message(self, features: RssiFeatures, result: SensingResult) -> str:
"""Build the JSON message to broadcast."""
# Get CSI-specific data if available
csi_data = None
if isinstance(self.collector, Esp32UdpCollector):
csi_data = self.collector.last_csi
signal_field = generate_signal_field(features, result, csi_data=csi_data)
node_info = {
"node_id": 1,
"rssi_dbm": features.mean,
"position": [2.0, 0.0, 1.5],
"amplitude": [],
"subcarrier_count": 0,
}
# Enrich with real CSI data
if csi_data:
node_info["node_id"] = csi_data.get("node_id", 1)
node_info["rssi_dbm"] = csi_data.get("rssi_dbm", features.mean)
node_info["amplitude"] = csi_data.get("amplitude", [])
node_info["subcarrier_count"] = csi_data.get("n_subcarriers", 0)
node_info["mean_amplitude"] = csi_data.get("mean_amplitude", 0)
node_info["freq_mhz"] = csi_data.get("freq_mhz", 0)
node_info["sequence"] = csi_data.get("sequence", 0)
node_info["source_addr"] = csi_data.get("source_addr", "")
msg = {
"type": "sensing_update",
"timestamp": time.time(),
"source": self.source,
"nodes": [node_info],
"features": {
"mean_rssi": features.mean,
"variance": features.variance,
"std": features.std,
"motion_band_power": features.motion_band_power,
"breathing_band_power": features.breathing_band_power,
"dominant_freq_hz": features.dominant_freq_hz,
"change_points": features.n_change_points,
"spectral_power": features.total_spectral_power,
"range": features.range,
"iqr": features.iqr,
"skewness": features.skewness,
"kurtosis": features.kurtosis,
},
"classification": {
"motion_level": result.motion_level.value,
"presence": result.presence_detected,
"confidence": round(result.confidence, 3),
},
"signal_field": signal_field,
}
return json.dumps(msg)
async def _handler(self, websocket):
"""Handle a single WebSocket client connection."""
self.clients.add(websocket)
remote = websocket.remote_address
logger.info("Client connected: %s", remote)
try:
async for _ in websocket:
pass
finally:
self.clients.discard(websocket)
logger.info("Client disconnected: %s", remote)
async def _broadcast(self, message: str) -> None:
"""Send message to all connected clients."""
if not self.clients:
return
disconnected = set()
for ws in self.clients:
try:
await ws.send(message)
except Exception:
disconnected.add(ws)
self.clients -= disconnected
async def _tick_loop(self) -> None:
"""Main sensing loop."""
while self._running:
try:
window = self.extractor.window_seconds
sample_rate = self.collector.sample_rate_hz
n_needed = int(window * sample_rate)
samples = self.collector.get_samples(n=n_needed)
if len(samples) >= 4:
features = self.extractor.extract(samples)
result = self.classifier.classify(features)
message = self._build_message(features, result)
await self._broadcast(message)
# Print status every few ticks
if isinstance(self.collector, Esp32UdpCollector):
csi = self.collector.last_csi
if csi and self.collector.frames_received % 20 == 0:
print(
f" [{csi['source_addr']}] node:{csi['node_id']} "
f"seq:{csi['sequence']} sc:{csi['n_subcarriers']} "
f"rssi:{csi['rssi_dbm']}dBm amp:{csi['mean_amplitude']:.1f} "
f"=> {result.motion_level.value} ({result.confidence:.0%})"
)
else:
logger.debug("Waiting for samples (%d/%d)", len(samples), n_needed)
except Exception:
logger.exception("Error in sensing tick")
await asyncio.sleep(TICK_INTERVAL)
async def run(self) -> None:
"""Start the server and run until interrupted."""
try:
import websockets
except ImportError:
print("ERROR: 'websockets' package not found.")
print("Install it with: pip install websockets")
sys.exit(1)
self.collector = self._create_collector()
self.collector.start()
self._running = True
print(f"\n Sensing WebSocket server on ws://{HOST}:{PORT}")
print(f" Source: {self.source}")
print(f" Tick: {TICK_INTERVAL}s | Window: {self.extractor.window_seconds}s")
print(" Press Ctrl+C to stop\n")
async with websockets.serve(self._handler, HOST, PORT):
await self._tick_loop()
def stop(self) -> None:
"""Stop the server gracefully."""
self._running = False
if self.collector:
self.collector.stop()
logger.info("Sensing server stopped")
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main():
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
server = SensingWebSocketServer()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
def _shutdown(sig, frame):
print("\nShutting down...")
server.stop()
loop.stop()
signal.signal(signal.SIGINT, _shutdown)
try:
loop.run_until_complete(server.run())
except KeyboardInterrupt:
pass
finally:
server.stop()
loop.close()
if __name__ == "__main__":
main()
archive/v1/src/services/__init__.py
+19
View File
@@ -0,0 +1,19 @@
"""
Services package for WiFi-DensePose API
"""
from .orchestrator import ServiceOrchestrator
from .health_check import HealthCheckService
from .metrics import MetricsService
from .pose_service import PoseService
from .stream_service import StreamService
from .hardware_service import HardwareService
__all__ = [
'ServiceOrchestrator',
'HealthCheckService',
'MetricsService',
'PoseService',
'StreamService',
'HardwareService'
]
archive/v1/src/services/hardware_service.py
+482
View File
@@ -0,0 +1,482 @@
"""
Hardware interface service for WiFi-DensePose API
"""
import logging
import asyncio
import time
from typing import Dict, List, Optional, Any
from datetime import datetime, timedelta
import numpy as np
from src.config.settings import Settings
from src.config.domains import DomainConfig
from src.core.router_interface import RouterInterface
logger = logging.getLogger(__name__)
class HardwareService:
"""Service for hardware interface operations."""
def __init__(self, settings: Settings, domain_config: DomainConfig):
"""Initialize hardware service."""
self.settings = settings
self.domain_config = domain_config
self.logger = logging.getLogger(__name__)
# Router interfaces
self.router_interfaces: Dict[str, RouterInterface] = {}
# Service state
self.is_running = False
self.last_error = None
# Data collection statistics
self.stats = {
"total_samples": 0,
"successful_samples": 0,
"failed_samples": 0,
"average_sample_rate": 0.0,
"last_sample_time": None,
"connected_routers": 0
}
# Background tasks
self.collection_task = None
self.monitoring_task = None
# Data buffers
self.recent_samples = []
self.max_recent_samples = 1000
async def initialize(self):
"""Initialize the hardware service."""
await self.start()
async def start(self):
"""Start the hardware service."""
if self.is_running:
return
try:
self.logger.info("Starting hardware service...")
# Initialize router interfaces
await self._initialize_routers()
self.is_running = True
# Start background tasks
if not self.settings.mock_hardware:
self.collection_task = asyncio.create_task(self._data_collection_loop())
self.monitoring_task = asyncio.create_task(self._monitoring_loop())
self.logger.info("Hardware service started successfully")
except Exception as e:
self.last_error = str(e)
self.logger.error(f"Failed to start hardware service: {e}")
raise
async def stop(self):
"""Stop the hardware service."""
self.is_running = False
# Cancel background tasks
if self.collection_task:
self.collection_task.cancel()
try:
await self.collection_task
except asyncio.CancelledError:
pass
if self.monitoring_task:
self.monitoring_task.cancel()
try:
await self.monitoring_task
except asyncio.CancelledError:
pass
# Disconnect from routers
await self._disconnect_routers()
self.logger.info("Hardware service stopped")
async def _initialize_routers(self):
"""Initialize router interfaces."""
try:
# Get router configurations from domain config
routers = self.domain_config.get_all_routers()
for router_config in routers:
if not router_config.enabled:
continue
router_id = router_config.router_id
# Create router interface
router_interface = RouterInterface(
router_id=router_id,
host=router_config.ip_address,
port=getattr(router_config, 'ssh_port', 22),
username=getattr(router_config, 'ssh_username', None) or self.settings.router_ssh_username,
password=getattr(router_config, 'ssh_password', None) or self.settings.router_ssh_password,
interface=router_config.interface,
mock_mode=self.settings.mock_hardware
)
# Connect to router (always connect, even in mock mode)
await router_interface.connect()
self.router_interfaces[router_id] = router_interface
self.logger.info(f"Router interface initialized: {router_id}")
self.stats["connected_routers"] = len(self.router_interfaces)
if not self.router_interfaces:
self.logger.warning("No router interfaces configured")
except Exception as e:
self.logger.error(f"Failed to initialize routers: {e}")
raise
async def _disconnect_routers(self):
"""Disconnect from all routers."""
for router_id, interface in self.router_interfaces.items():
try:
await interface.disconnect()
self.logger.info(f"Disconnected from router: {router_id}")
except Exception as e:
self.logger.error(f"Error disconnecting from router {router_id}: {e}")
self.router_interfaces.clear()
self.stats["connected_routers"] = 0
async def _data_collection_loop(self):
"""Background loop for data collection."""
try:
while self.is_running:
start_time = time.time()
# Collect data from all routers
await self._collect_data_from_routers()
# Calculate sleep time to maintain polling interval
elapsed = time.time() - start_time
sleep_time = max(0, self.settings.hardware_polling_interval - elapsed)
if sleep_time > 0:
await asyncio.sleep(sleep_time)
except asyncio.CancelledError:
self.logger.info("Data collection loop cancelled")
except Exception as e:
self.logger.error(f"Error in data collection loop: {e}")
self.last_error = str(e)
async def _monitoring_loop(self):
"""Background loop for hardware monitoring."""
try:
while self.is_running:
# Monitor router connections
await self._monitor_router_health()
# Update statistics
self._update_sample_rate_stats()
# Wait before next check
await asyncio.sleep(30) # Check every 30 seconds
except asyncio.CancelledError:
self.logger.info("Monitoring loop cancelled")
except Exception as e:
self.logger.error(f"Error in monitoring loop: {e}")
async def _collect_data_from_routers(self):
"""Collect CSI data from all connected routers."""
for router_id, interface in self.router_interfaces.items():
try:
# Get CSI data from router
csi_data = await interface.get_csi_data()
if csi_data is not None:
# Process the collected data
await self._process_collected_data(router_id, csi_data)
self.stats["successful_samples"] += 1
self.stats["last_sample_time"] = datetime.now().isoformat()
else:
self.stats["failed_samples"] += 1
self.stats["total_samples"] += 1
except Exception as e:
self.logger.error(f"Error collecting data from router {router_id}: {e}")
self.stats["failed_samples"] += 1
self.stats["total_samples"] += 1
async def _process_collected_data(self, router_id: str, csi_data: np.ndarray):
"""Process collected CSI data."""
try:
# Create sample metadata
metadata = {
"router_id": router_id,
"timestamp": datetime.now().isoformat(),
"sample_rate": self.stats["average_sample_rate"],
"data_shape": csi_data.shape if hasattr(csi_data, 'shape') else None
}
# Add to recent samples buffer
sample = {
"router_id": router_id,
"timestamp": metadata["timestamp"],
"data": csi_data,
"metadata": metadata
}
self.recent_samples.append(sample)
# Maintain buffer size
if len(self.recent_samples) > self.max_recent_samples:
self.recent_samples.pop(0)
# Notify other services (this would typically be done through an event system)
# For now, we'll just log the data collection
self.logger.debug(f"Collected CSI data from {router_id}: shape {csi_data.shape if hasattr(csi_data, 'shape') else 'unknown'}")
except Exception as e:
self.logger.error(f"Error processing collected data: {e}")
async def _monitor_router_health(self):
"""Monitor health of router connections."""
healthy_routers = 0
for router_id, interface in self.router_interfaces.items():
try:
is_healthy = await interface.check_health()
if is_healthy:
healthy_routers += 1
else:
self.logger.warning(f"Router {router_id} is unhealthy")
# Try to reconnect if not in mock mode
if not self.settings.mock_hardware:
try:
await interface.reconnect()
self.logger.info(f"Reconnected to router {router_id}")
except Exception as e:
self.logger.error(f"Failed to reconnect to router {router_id}: {e}")
except Exception as e:
self.logger.error(f"Error checking health of router {router_id}: {e}")
self.stats["connected_routers"] = healthy_routers
def _update_sample_rate_stats(self):
"""Update sample rate statistics."""
if len(self.recent_samples) < 2:
return
# Calculate sample rate from recent samples
recent_count = min(100, len(self.recent_samples))
recent_samples = self.recent_samples[-recent_count:]
if len(recent_samples) >= 2:
# Calculate time differences
time_diffs = []
for i in range(1, len(recent_samples)):
try:
t1 = datetime.fromisoformat(recent_samples[i-1]["timestamp"])
t2 = datetime.fromisoformat(recent_samples[i]["timestamp"])
diff = (t2 - t1).total_seconds()
if diff > 0:
time_diffs.append(diff)
except Exception:
continue
if time_diffs:
avg_interval = sum(time_diffs) / len(time_diffs)
self.stats["average_sample_rate"] = 1.0 / avg_interval if avg_interval > 0 else 0.0
async def get_router_status(self, router_id: str) -> Dict[str, Any]:
"""Get status of a specific router."""
if router_id not in self.router_interfaces:
raise ValueError(f"Router {router_id} not found")
interface = self.router_interfaces[router_id]
try:
is_healthy = await interface.check_health()
status = await interface.get_status()
return {
"router_id": router_id,
"healthy": is_healthy,
"connected": status.get("connected", False),
"last_data_time": status.get("last_data_time"),
"error_count": status.get("error_count", 0),
"configuration": status.get("configuration", {})
}
except Exception as e:
return {
"router_id": router_id,
"healthy": False,
"connected": False,
"error": str(e)
}
async def get_all_router_status(self) -> List[Dict[str, Any]]:
"""Get status of all routers."""
statuses = []
for router_id in self.router_interfaces:
try:
status = await self.get_router_status(router_id)
statuses.append(status)
except Exception as e:
statuses.append({
"router_id": router_id,
"healthy": False,
"error": str(e)
})
return statuses
async def get_recent_data(self, router_id: Optional[str] = None, limit: int = 100) -> List[Dict[str, Any]]:
"""Get recent CSI data samples."""
samples = self.recent_samples[-limit:] if limit else self.recent_samples
if router_id:
samples = [s for s in samples if s["router_id"] == router_id]
# Convert numpy arrays to lists for JSON serialization
result = []
for sample in samples:
sample_copy = sample.copy()
if isinstance(sample_copy["data"], np.ndarray):
sample_copy["data"] = sample_copy["data"].tolist()
result.append(sample_copy)
return result
async def get_status(self) -> Dict[str, Any]:
"""Get service status."""
return {
"status": "healthy" if self.is_running and not self.last_error else "unhealthy",
"running": self.is_running,
"last_error": self.last_error,
"statistics": self.stats.copy(),
"configuration": {
"mock_hardware": self.settings.mock_hardware,
"wifi_interface": self.settings.wifi_interface,
"polling_interval": self.settings.hardware_polling_interval,
"buffer_size": self.settings.csi_buffer_size
},
"routers": await self.get_all_router_status()
}
async def get_metrics(self) -> Dict[str, Any]:
"""Get service metrics."""
total_samples = self.stats["total_samples"]
success_rate = self.stats["successful_samples"] / max(1, total_samples)
return {
"hardware_service": {
"total_samples": total_samples,
"successful_samples": self.stats["successful_samples"],
"failed_samples": self.stats["failed_samples"],
"success_rate": success_rate,
"average_sample_rate": self.stats["average_sample_rate"],
"connected_routers": self.stats["connected_routers"],
"last_sample_time": self.stats["last_sample_time"]
}
}
async def reset(self):
"""Reset service state."""
self.stats = {
"total_samples": 0,
"successful_samples": 0,
"failed_samples": 0,
"average_sample_rate": 0.0,
"last_sample_time": None,
"connected_routers": len(self.router_interfaces)
}
self.recent_samples.clear()
self.last_error = None
self.logger.info("Hardware service reset")
async def trigger_manual_collection(self, router_id: Optional[str] = None) -> Dict[str, Any]:
"""Manually trigger data collection."""
if not self.is_running:
raise RuntimeError("Hardware service is not running")
results = {}
if router_id:
# Collect from specific router
if router_id not in self.router_interfaces:
raise ValueError(f"Router {router_id} not found")
interface = self.router_interfaces[router_id]
try:
csi_data = await interface.get_csi_data()
if csi_data is not None:
await self._process_collected_data(router_id, csi_data)
results[router_id] = {"success": True, "data_shape": csi_data.shape if hasattr(csi_data, 'shape') else None}
else:
results[router_id] = {"success": False, "error": "No data received"}
except Exception as e:
results[router_id] = {"success": False, "error": str(e)}
else:
# Collect from all routers
await self._collect_data_from_routers()
results = {"message": "Manual collection triggered for all routers"}
return results
async def health_check(self) -> Dict[str, Any]:
"""Perform health check."""
try:
status = "healthy" if self.is_running and not self.last_error else "unhealthy"
# Check router health
healthy_routers = 0
total_routers = len(self.router_interfaces)
for router_id, interface in self.router_interfaces.items():
try:
if await interface.check_health():
healthy_routers += 1
except Exception:
pass
return {
"status": status,
"message": self.last_error if self.last_error else "Hardware service is running normally",
"connected_routers": f"{healthy_routers}/{total_routers}",
"metrics": {
"total_samples": self.stats["total_samples"],
"success_rate": (
self.stats["successful_samples"] / max(1, self.stats["total_samples"])
),
"average_sample_rate": self.stats["average_sample_rate"]
}
}
except Exception as e:
return {
"status": "unhealthy",
"message": f"Health check failed: {str(e)}"
}
async def is_ready(self) -> bool:
"""Check if service is ready."""
return self.is_running and len(self.router_interfaces) > 0

Some files were not shown because too many files have changed in this diff Show More