Skip to main content

Showcase

HySDS powers some of NASA's most critical Earth Science missions and data processing systems. This showcase highlights key implementations, achievements, and project successes across the HySDS ecosystem.

NISAR​

  • Processing capacity of > 300TB/day
  • Large-scale data production system
  • Hybrid cloud and on-premise processing capabilities
  • State-of-the-art SAR data processing

SWOT​

  • Processed 2PB in first year of operations
  • Routinely uses 2,000 parallel nodes for bulk reprocessing
  • Operational since December 2022
  • Large-scale water surface monitoring

SMAP with HySDS (SWH)​

  • Operational since November 2023
  • Soil moisture data processing
  • Advanced microwave radiometry processing

SNWG OPERA​

  • Operational since March 2023
  • Advanced SAR processing
  • Large-scale data production

OCO-2/3 Reprocessing​

  • Full physics production in AWS
  • Atmospheric CO2 measurement processing
  • Large-scale reprocessing capabilities

Key Performance Metrics​

Processing Scale​

  • Up to 8,200+ parallel processing nodes
  • Capability to handle 3-million processing jobs per day
  • Demonstrated large-scale hybrid cloud operations

Cost Efficiency​

  • First NASA SDS to utilize AWS spot market
  • Fault tolerant in volatile compute environments
  • Optimized hybrid cloud/on-premise processing

Processing Capabilities​

  • ML and GPU processing at scale
  • Multi-core processing optimization
  • Low-latency urgent response processing
  • On-demand processing capabilities

Project Success Stories​

MAAP (Multi-Mission Algorithm and Analysis Platform)​

  • Reduced biomass harmonization processing time from months to hours
  • Successfully running 4,000 parallel nodes
  • Integration with multiple data sources and processing systems

PO.DAAC SWODLR​

  • On-demand raster generation
  • Advanced data visualization capabilities
  • Integrated with oceanographic data processing

ASTER Volcano Archive (AVA)​

  • Long-term volcanic monitoring
  • Large-scale image processing
  • Automated data analysis

Timeline of Achievements​

Major Milestones​

  • 2008: ACCESS - Service-based science data processing
  • 2014: Automated Urgent Response processing in AWS
  • 2017: GRFN Sentinel-1 production scaled to 8,200+ parallel nodes
  • 2022: SWOT SDS operations begin
  • 2023: OPERA SDS & SMAP operations begin
  • 2025: Planned NISAR SDS operations

Platform Innovations​

Cloud & Infrastructure​

  • First NASA EO Science Data System for cloud operations
  • Multi-cloud support across AWS, GCP, Azure
  • Integration with NASA HECC Super Computing
  • Hybrid cloud processing capabilities

Technical Achievements​

  • Real-time faceted analytics with operations
  • Cost-production modeling for estimation
  • Advanced data processing orchestration
  • Fault-tolerant processing architecture

Implementation Statistics​

Current Usage (as of 2024)​

  • 13 active projects
  • 33 total NASA-funded projects to date
  • 50+ developers
  • 30+ contributors
  • 78 repositories
  • 83 releases

Urgent Response Processing​

  • Low-latency processing capabilities
  • Automated triggering and scaling
  • Real-time data analysis

Large-Scale Reprocessing​

  • Bulk data processing capabilities
  • Efficient resource utilization
  • Automated workflow management

On-Demand Processing​

  • User-triggered processing
  • Dynamic resource allocation
  • Custom processing pipelines

System Capabilities​

Architecture Features​

  • Hybrid cloud deployment options
  • Multi-cloud support
  • Scalable processing framework
  • Advanced job management
  • Real-time metrics and monitoring

Processing Options​

  • AWS cloud processing
  • On-premise processing
  • HECC integration
  • Hybrid processing modes

Impact Metrics​

Data Processing Volume​

  • Multiple petabytes processed
  • Millions of jobs per day capability
  • Support for hundreds of terabytes of daily ingest

Community Impact​

  • Open source development model
  • Multi-mission benefits
  • Shared operational procedures
  • Active developer community

Recognition and Citations​

For academic and technical citations, please reference: DOI: 10.5281/zenodo.11118142

The implementations and achievements listed in this showcase represent significant contributions to Earth Science data processing and demonstrate HySDS's capabilities as a scalable, efficient, and versatile processing system.