Back to Dashboard
Supabase PostgreSQLScripps Pier Station
Data Collection Pipeline
Monitor and manage automated data scrapers
2 Raw Data Collectors•6 Data Processors•Updating 15+ Tables
Real-time Collection
IFCB data processed every 30 minutes, sensor data hourly
Multi-stage Processing
Raw data → Hourly stats → Daily summaries → Analytics
Automated Pipeline
Cron-scheduled jobs with dependency management
Data Flow Overview
IFCB Plankton Pipeline
IFCB API→Raw PIDs→Hourly Stats→Species Summary→Web Dashboard
Environmental Sensor Pipeline
AXDS API→Raw Observations→Hourly Aggregation→Sensor Charts
Analytics Pipeline
Hourly Data→Daily Stats→Diversity Metrics→Functional Groups