Skip to main content

What is HySDS?

Welcome to the Hybrid-Cloud Science Data Processing System (HySDS) documentation! HySDS is an open source science data processing system used across many large-scale Earth Science missions, data production, and analysis systems.

HySDS is a powerful, scalable system designed to handle the growing demands of Earth science data processing. It provides:

  • Large-scale science data processing and management of geospatial data
  • Hybrid cloud deployment capabilities across multiple platforms
  • Community-driven development and reuse of common processing services

Key Features​

Unmatched Scalability & Performance​

Industry-leading processing capacity with 8,000+ parallel nodes handling over 3 million daily jobs and hundreds of terabytes of scientific data.

True Hybrid-Cloud Flexibility​

First-ever science data system to seamlessly operate across multiple platforms including AWS, GCP, Azure, and NASA HECC with built-in cost optimization.

Proven Multi-Mission Excellence​

Trusted by 33+ NASA projects including NISAR, SWOT, and SMAP, with a thriving community of 50+ developers advancing Earth Science missions.

Who Uses HySDS?​

HySDS is actively used by major Earth Science missions and projects:

  • Missions: NISAR, SWOT, SMAP
  • Data Production: SNWG OPERA, OCO-2/3 reprocessing
  • Projects: Multi-Mission Algorithm and Analysis Platform (MAAP), PO.DAAC SWODLR, ASTER AVA, and more

Why Choose HySDS?​

⚡ Exceptional Performance​

  • Support for processing > 300TB/day
  • Data processing scalability to over 8,000 parallel nodes
  • Capability to handle > 3-million processing jobs per day

🌐 Hybrid Deployment​

  • Seamless processing across AWS, HECC, and on-premise systems
  • Fault tolerant in low cost & volatile AWS spot market
  • Multi-cloud support spanning AWS, GCP, and Azure

🛠 Rich Feature Set​

  • ML, GPU, and multi-core processing at large scales
  • Low-latency urgent response & on-demand processing
  • Cost-production model estimation

Getting Started​

To begin using HySDS:

  1. Check out our Quick Start Guide
  2. Review the Architecture Overview
  3. Explore our Tutorials

Community and Support​

HySDS is backed by a strong open source community:

  • 50+ active developers and integrators
  • 78+ repositories in the organization
  • Regular community contributions and updates

Resources​

Next Steps​

Ready to dive deeper? Check out our:

Note: This documentation is maintained by the HySDS community. For the latest updates and information, please check our GitHub repository.