Astrobee Computer Vision System

Astrobee is a collection of free-flying robots developed by NASA to operate inside the International Space Station (ISS). As a perception engineer on this project, I developed and optimized production computer vision algorithms that enable autonomous navigation, marker-based docking, and real-time localization in the challenging microgravity environment of space.

Perception Pipeline Architecture

The Astrobee perception system is built on a multi-layered architecture that processes visual data from three onboard cameras (Nav, Sci, and Haz cams) at 30 FPS while operating under strict computational constraints of space-grade hardware.

System Overview

• Input: 1280x960 grayscale images from navigation camera
• Processing: Real-time feature detection, tracking, and pose estimation
• Output: 6-DOF pose estimates at 30 Hz for flight control
• Compute: ARM-based processor with limited GPU acceleration
• Reliability: Must operate for years in radiation-heavy space environment

The perception pipeline consists of:

1. Camera calibration and undistortion
2. Feature detection and descriptor extraction
3. Feature matching against pre-built map
4. Robust pose estimation with outlier rejection
5. Temporal filtering and state estimation

Camera Calibration & Intrinsics

Each Astrobee robot undergoes extensive camera calibration before deployment. This is critical because traditional assumptions (gravity-aligned cameras, fixed mounting) don't apply in microgravity.

Calibration Process

• Method: Zhang's calibration method with checkerboard patterns
• Intrinsic Parameters:
  • Focal length (fx, fy)
  • Principal point (cx, cy)
  • Radial distortion coefficients (k1, k2, k3)
  • Tangential distortion coefficients (p1, p2)
• Extrinsic Calibration: Hand-eye calibration between camera frame and robot body frame
• Validation: Sub-pixel reprojection error < 0.5 pixels

Production Considerations

In space, thermal expansion, radiation damage, and mechanical vibration can affect camera calibration over time. The system includes:

• Online calibration refinement: Periodic bundle adjustment to update intrinsics
• Degradation detection: Monitoring reprojection errors to detect calibration drift
• Fallback modes: Safe navigation using degraded perception when calibration quality drops

Visual Feature Detection & Tracking

Astrobee uses sparse feature-based visual odometry and localization, a classical computer vision approach optimized for real-time performance and reliability.

Feature Detection

• Primary: BRISK (Binary Robust Invariant Scalable Keypoints)
  • Binary descriptors for fast matching
  • Scale and rotation invariant
  • Optimized for embedded ARM processors
• Fallback: SURF (Speeded-Up Robust Features)
  • More distinctive features
  • Higher computational cost
  • Used for initial map building

Feature Matching Strategy

1. Detect 300-500 keypoints per frame
2. Extract binary descriptors (256-512 bits)
3. Match against vocabulary database using Bag-of-Words
4. Hamming distance for binary descriptor matching
5. Lowe's ratio test (0.8 threshold) for outlier rejection
6. Geometric verification using RANSAC

Performance Metrics

• Detection Speed: 15-20ms per frame (ARM Cortex-A9)
• Matching Speed: 10-15ms against 50K map features
• Matching Rate: 150-200 successful matches per frame in nominal conditions
• Tracking Success: >95% localization success rate during normal operations

Sparse Mapping & 3D Reconstruction

The ISS environment is pre-mapped using Structure from Motion (SfM) to create a sparse 3D map that Astrobee uses for localization.

Map Building Process

1. Image Collection: Astronauts capture 500-1000 images per module using calibrated cameras
2. Feature Extraction: Detect SURF features in all images (more distinctive for mapping)
3. Feature Matching: Match features across overlapping image pairs
4. Epipolar Geometry: Estimate fundamental matrices, filter outliers with RANSAC
5. Triangulation: Compute 3D positions of matched features
6. Bundle Adjustment: Global optimization to minimize reprojection error
7. Map Optimization: Remove redundant features, ensure uniform coverage

Vocabulary Database

The vocabulary database enables fast image retrieval and feature matching:

• Training: K-means clustering on 1M+ feature descriptors
• Vocabulary Size: 10K-100K visual words (trade-off between speed and accuracy)
• Image Representation: TF-IDF weighted histogram of visual words
• Query Speed: <5ms to find similar images in database with 10K+ images
• Implementation: DBoW2 (Database of Binary Words) library

Map Storage & Format

• Format: Protocol buffers (.map files)
• Size: 50-100 MB per ISS module
• Contents:
  • 3D feature positions (50K-200K landmarks)
  • Feature descriptors
  • Image metadata and camera poses
  • Vocabulary database
• Distribution: Maps uploaded to ISS and loaded into robot memory

Pose Estimation Algorithms

The core of Astrobee's perception system is robust 6-DOF pose estimation from 2D-3D correspondences.

PnP (Perspective-n-Point) Solver

Given 2D image features and their corresponding 3D map positions, estimate camera pose:

• Algorithm: EPnP (Efficient Perspective-n-Point)
  • Linear solution for n ≥ 4 points
  • Solves for camera pose using 4 control points
  • Non-iterative, faster than iterative methods
• Minimum Points: 4 correspondences (3 + 1 for validation)
• Typical Input: 150-200 matched 2D-3D correspondences

RANSAC for Outlier Rejection

Feature matching produces many outliers due to repetitive textures and symmetric structures in the ISS:

• Method: RANSAC (Random Sample Consensus)
• Iterations: 100-200 iterations
• Inlier Threshold: 2-3 pixels reprojection error
• Consensus Set: Typically 60-80% of matches are inliers
• Pose Refinement: Levenberg-Marquardt optimization on inlier set

Temporal Filtering

Raw pose estimates are noisy and may contain occasional outliers:

• Kalman Filter: Fuses pose estimates with IMU data
• State Vector: Position, velocity, orientation, angular velocity
• Prediction: IMU integration between vision updates
• Correction: Vision pose estimates at 30 Hz
• Smoothing: Reduces jitter while maintaining responsiveness

Marker-Based Docking

Astrobee uses AprilTag fiducial markers for precise docking to charging stations and berths.

Marker Detection Pipeline

1. Preprocessing: Adaptive thresholding to handle varying lighting
2. Contour Detection: Find quadrilateral contours in binary image
3. Perspective Correction: Unwarp marker to canonical view
4. Decoding: Read marker ID from binary pattern
5. Pose Estimation: Compute 6-DOF pose from marker corners

Docking Accuracy

• Detection Range: 0.5m to 3.0m
• Position Accuracy: ±5mm at docking
• Orientation Accuracy: ±2 degrees
• Reliability: >99% detection rate when marker is in view

Multi-Marker Fusion

Charging docks have multiple markers for redundancy:

• Fusion Strategy: Weighted average of pose estimates from all detected markers
• Outlier Rejection: Chi-squared test to remove bad marker detections
• Degraded Mode: Can dock with single marker if others are occluded

Production Constraints & Optimization

Developing computer vision for space-grade hardware requires careful optimization and reliability engineering.

Computational Constraints

• Processor: ARM Cortex-A9 (1.7 GHz, quad-core)
• Memory: Limited RAM budget for perception (512 MB allocated)
• GPU: Limited OpenCL acceleration
• Power: Thermal constraints limit sustained compute

Real-Time Optimizations

• Early Rejection: Hierarchical matching to quickly discard bad candidates
• Binary Descriptors: Hamming distance computed using SIMD instructions
• Sparse Processing: Process only salient regions, skip featureless areas
• Adaptive Processing: Reduce feature count if running behind schedule
• Memory Pooling: Pre-allocated buffers to avoid runtime allocation overhead

Validation & Testing

Safety-critical space systems require extensive validation:

• Unit Tests: Test individual CV components (feature detectors, pose solvers)
• Integration Tests: Full pipeline testing with recorded ISS data
• Simulation: Ray-traced ISS simulation for synthetic test data
• Hardware-in-Loop: Testing on flight hardware before deployment
• On-Orbit Validation: Supervised operations during initial deployment

Failure Modes & Recovery

The system is designed to handle perception failures gracefully:

• Localization Loss: Switch to IMU-only mode, beacon-based fallback
• Feature Tracking Loss: Re-initialize from vocabulary database
• Map Staleness: Astronauts can update maps if environment changes
• Degraded Lighting: Adaptive exposure control, optional flashlight activation

Classical Computer Vision Techniques

This project demonstrates the power of classical CV techniques when applied correctly:

Epipolar Geometry

• Fundamental Matrix: Constrains feature correspondences across views
• Essential Matrix: Encodes relative rotation and translation
• Epipolar Lines: Reduces correspondence search to 1D problem
• 5-Point Algorithm: Minimal solver for essential matrix estimation

Structure from Motion

• Incremental SfM: Builds map by progressively adding images
• Two-View Geometry: Initialize with epipolar geometry
• Triangulation: Linear methods for 3D point estimation
• Bundle Adjustment: Sparse Levenberg-Marquardt optimization
• Loop Closure: Detect revisited locations and global optimization

Image Processing

• Undistortion: Remap images to eliminate lens distortion
• Histogram Equalization: Improve contrast in low-light areas
• Gaussian Pyramids: Multi-scale feature detection
• Non-Maximum Suppression: Ensure well-distributed features

Technical Stack

• C++17: Core perception algorithms
• OpenCV 3.4: Computer vision library
• Eigen: Linear algebra and geometry
• ROS (Robot Operating System): Middleware and system integration
• Protocol Buffers: Map serialization format
• DBoW2: Bag-of-Words database
• CMake: Build system
• GTest: Unit testing framework

Key Takeaways

This project demonstrates expertise in:

1. Classical Computer Vision: Feature detection, matching, epipolar geometry, SfM
2. 3D Geometry: Camera calibration, pose estimation, coordinate transforms
3. Production Engineering: Real-time optimization, resource constraints, reliability
4. Systems Integration: ROS, sensor fusion, state estimation
5. Safety-Critical Systems: Extensive testing, failure handling, validation

The Astrobee perception system has been operating successfully on the ISS since 2019, demonstrating that well-engineered classical CV techniques can provide robust, reliable performance in production environments.

Resources

• GitHub Repository
• NASA Astrobee Project
• Technical Papers:
  • "Astrobee: A New Platform for Free-Flying Robotics" (IEEE 2018)
  • "Visual-Inertial Localization for Astrobee Robots" (IROS 2019)