Adding wrf components to production

README.md

@@ -13,7 +13,7 @@ app_port: 7860
 
 Wildfire occurrence modeling using Terrestrial Ecosystem Models and Artificial Intelligence
 
-[CG Lightning Probability Forecast](https://jordancaraballo-alaska-wildfire-occurrence.hf.space/)
+[CG Lightning Probability Forecast](https://huggingface.co/spaces/jordancaraballo/alaska-wildfire-occurrence)
 
 ## Objectives
 
@@ -23,50 +23,27 @@ Wildfire occurrence modeling using Terrestrial Ecosystem Models and Artificial I
 - 30m local Alaska models, 1km circumpolar models
 - Integration of precipitation, temperature and lightning datasets
 
-## Datasets
-
-1. Daily Fire Ignition Points
-
-```bash
-```
-
-2. Daily Area Burned
-
-The dataset comes from https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1559 for 2001-2019. This dataset
-will be extended for 2020-2025. Dataset is located under /explore/nobackup/projects/ilab/projects/LobodaTFO/data/raw_data/ABoVE_DoB.
-
-```bash
-python DAACDataDownload.py -dir /explore/nobackup/projects/ilab/projects/LobodaTFO/data/raw_data/ABoVE_DoB -f URL_FROM_ORDER
-```
-
-3. Annual Fuel Composition
-
-```bash
-```
-
-4. Human Accesibility
-
-```bash
-```
+## Containers
 
-5. Topographic Influence
+### Python Container
 
 ```bash
+module load singularity
+singularity build --sandbox /lscratch/$USER/container/wildfire-occurrence docker://nasanccs/wildfire-occurrence:latest
 ```
 
-All datasets described above will be delivered in the 1 km modeling grid for tundra ecoregions.
-
-## Containers
+## Quickstart
 
-### Python Container
+### Executing WRF
 
 ```bash
-module load singularity
-singularity build --sandbox /lscratch/$USER/container/wildfire-occurrence docker://nasanccs/wildfire-occurrence:latest
+singularity exec --env PYTHONPATH="/explore/nobackup/people/$USER/development/wildfire-occurrence" --nv -B /explore/nobackup/projects/ilab,$NOBACKUP,/lscratch,/explore/nobackup/people /lscratch/$USER/container/wildfire-occurrence python /explore/nobackup/people/$USER/development/wildfire-occurrence/wildfire_occurrence/view/wrf_pipeline_cli.py -c /explore/nobackup/people/$USER/development/wildfire-occurrence/wildfire_occurrence/templates/config.yaml --pipeline-step all --start-date 2023-06-05 --forecast-lenght 10
 ```
 
 ## Extracting variables from WRF
 
+Running this script to extract variables from WRF and perform lightning inference
+
 ```bash
 singularity shell --nv -B /explore/nobackup/projects/ilab,/explore/nobackup/projects/3sl,$NOBACKUP,/lscratch,/explore/nobackup/people /lscratch/jacaraba/container/wildfire-occurrence/
 python wrf_analysis.py 
@@ -78,8 +55,11 @@ python wrf_analysis.py
 singularity exec --env PYTHONPATH="/explore/nobackup/people/jacaraba/development/wildfire-occurrence" --nv -B /explore/nobackup/projects/ilab,/explore/nobackup/projects/3sl,$NOBACKUP,/lscratch,/explore/nobackup/people /lscratch/jacaraba/container/wildfire-occurrence python /explore/nobackup/people/jacaraba/development/wildfire-occurrence/wildfire_occurrence/model/lightning/lightning_model.py
 ```
 
-(base) [jacaraba@gpu021 ~]$ singularity exec --env PYTHONPATH="/explore/nobackup/people/jacaraba/development/wildfire-occurrence" --nv -B /explore/nobackup/projects/ilab,/explore/nobackup/projects/3sl,$NOBACKUP,/lscratch,/explore/nobackup/people /lscratch/jacaraba/container/wildfire-occurrence python /explore/nobackup/people/jacaraba/development/wildfire-occurrence/wildfire_occurrence/model/lightning/lightning_model.py 
+Full Data Pipeline Command
 
+```bash
+singularity exec --env PYTHONPATH="/explore/nobackup/people/jacaraba/development/wildfire-occurrence" --nv -B /explore/nobackup/projects/ilab,/explore/nobackup/projects/3sl,$NOBACKUP,/lscratch,/explore/nobackup/people /lscratch/jacaraba/container/wildfire-occurrence python /explore/nobackup/people/jacaraba/development/wildfire-occurrence/wildfire_occurrence/model/lightning/lightning_model.py 
+```
 
 ## Contributors
 

wildfire_occurrence/model/data_download/ncep_fnl.py

@@ -0,0 +1,222 @@
+import os
+import re
+import sys
+import logging
+import requests
+import datetime
+import pandas as pd
+from datetime import date
+from typing import List, Literal
+from multiprocessing import Pool, cpu_count
+
+__data_source__ = 'https://rda.ucar.edu/datasets/ds083.2'
+
+
+class NCEP_FNL(object):
+
+    def __init__(
+                self,
+                output_dir: str,
+                start_date: str = date.today(),
+                end_date: str = date.today(),
+                hour_intervals: List = ['00', '06', '12', '18'],
+                n_procs: int = cpu_count()
+            ):
+
+        # output directory
+        self.output_dir = output_dir
+
+        # define start and end data of download
+        if isinstance(start_date, str):
+            self.start_date = datetime.datetime.strptime(
+                start_date, '%Y-%m-%d').date()
+        else:
+            self.start_date = start_date
+
+        # define start and end data of download
+        if isinstance(end_date, str):
+            self.end_date = datetime.datetime.strptime(
+                end_date, '%Y-%m-%d').date()
+        else:
+            self.end_date = end_date
+
+        # define hour intervals
+        self.hour_intervals = hour_intervals
+
+        # make sure we do not download data into the future
+        if self.end_date > datetime.datetime.now():
+            self.end_date = datetime.datetime.now()
+            self.hour_intervals = [
+                d for d in self.hour_intervals
+                if int(d) <= self.end_date.hour - 6]
+        logging.info(
+            f'Downloading data from {self.start_date} to {self.end_date}')
+
+        # check for email and password environment variables
+        if "NCEP_FNL_EMAIL" not in os.environ \
+                or "NCEP_FNL_KEY" not in os.environ:
+            sys.exit(
+                "ERROR: You need to set NCEP_FNL_EMAIL and NCEP_FNL_KEY " +
+                "to enable data downloads. If you do not have an " +
+                "account, go to https://rda.ucar.edu/ and create one."
+            )
+
+        # define email and password fields
+        self.email = os.environ['NCEP_FNL_EMAIL']
+        assert re.search(r'[\w.]+\@[\w.]+', self.email), \
+            f'{self.email} is not a valid email.'
+
+        self.password = os.environ['NCEP_FNL_KEY']
+
+        # define cookie filename to store auth
+        self.cookie_filename = f'/home/{os.environ["USER"]}/.ncep_cookie'
+
+        # define login url
+        self.auth_url = 'https://rda.ucar.edu/cgi-bin/login'
+        self.auth_request = {
+            'email': self.email,
+            'passwd': self.password,
+            'action': 'login'
+        }
+
+        # define data url
+        self.data_url = 'https://rda.ucar.edu'
+
+        if self.start_date.year < 2008:
+            self.grib_format = 'grib1'
+        else:
+            self.grib_format = 'grib2'
+
+        self.dataset_path = f'/data/OS/ds083.2/{self.grib_format}'
+
+        # nnumber of processors to use
+        self.n_procs = n_procs
+
+    def _authenticate(self, action: Literal["auth", "cleanup"] = "auth"):
+
+        if action == "cleanup":
+            # cleanup cookie filename
+            os.remove(self.cookie_filename)
+        else:
+            # attempt to authenticate
+            ret = requests.post(self.auth_url, data=self.auth_request)
+            if ret.status_code != 200:
+                sys.exit('Bad Authentication. Check email and password.')
+
+            logging.info('Authenticated')
+
+            os.system(
+                f'wget --save-cookies {self.cookie_filename} ' +
+                '--delete-after --no-verbose ' +
+                f'--post-data="email={self.email}&' +
+                f'passwd={self.password}&action=login" {self.auth_url}'
+            )
+        return
+
+    def _download_file(self, wget_request: str):
+        logging.info(wget_request)
+        os.system(wget_request)
+        return
+
+    def download(self):
+
+        # authenticate against NCEP
+        self._authenticate(action="auth")
+
+        # get list of filenames to download
+        filenames = self._get_filenames()
+
+        # setup list for parallel downloads
+        download_requests = []
+        for filename in filenames:
+
+            # get year from the filename
+            year = re.search(r'\d{4}', filename).group(0)
+
+            # set full output directory and create it
+            output_dir = os.path.join(self.output_dir, year)
+            os.makedirs(output_dir, exist_ok=True)
+
+            # set full url and output filename
+            full_url = self.data_url + filename
+            output_filename = os.path.join(
+                output_dir, os.path.basename(filename))
+            logging.info(f'Downloading {full_url} to {output_filename}')
+
+            # download request for parallel download
+            if not os.path.isfile(output_filename) or \
+                    os.path.getsize(output_filename) == 0:
+                download_requests.append(
+                    f'wget --load-cookies {self.cookie_filename} ' +
+                    f'--no-verbose -O {output_filename} {full_url}'
+                )
+
+        # Set pool, start parallel multiprocessing
+        p = Pool(processes=self.n_procs)
+        p.map(self._download_file, download_requests)
+        p.close()
+        p.join()
+
+        # authenticate against NCEP
+        self._authenticate(action="cleanup")
+
+        return
+
+    def _get_filenames(self):
+        filenames_list = []
+        daterange = pd.date_range(self.start_date, self.end_date)
+        for single_date in daterange:
+            year = single_date.strftime("%Y")
+            for hour in self.hour_intervals:
+                filename = os.path.join(
+                    self.dataset_path,
+                    f'{year}/{single_date.strftime("%Y.%m")}',
+                    f'fnl_{single_date.strftime("%Y%m%d")}_' +
+                    f'{hour}_00.{self.grib_format}'
+                )
+                filenames_list.append(filename)
+        return filenames_list
+
+
+# -----------------------------------------------------------------------------
+# Invoke the main
+# -----------------------------------------------------------------------------
+if __name__ == "__main__":
+
+    dates = [
+        '2003-06-23',
+        '2005-06-11',
+        '2005-06-29',
+        '2005-08-16',
+        '2007-07-04',
+        '2007-07-11',
+        '2008-06-25',
+        '2009-06-09',
+        '2010-07-01',
+        '2013-06-20',
+        '2013-08-16',
+        '2015-07-14',
+        '2015-06-21',
+        '2015-07-23',
+        '2016-07-11',
+        '2022-01-10',
+        '2022-07-03',
+        '2018-02-25',
+        '2019-08-04',
+        '2019-08-19',
+        '2020-09-03',
+        '2022-05-09',
+        '2023-06-04'
+    ]
+
+    for init_date in dates:
+
+        start_date = datetime.datetime.strptime(init_date, "%Y-%m-%d")
+        end_date = (start_date + datetime.timedelta(days=10))
+
+        downloader = NCEP_FNL(
+            output_dir='/explore/nobackup/projects/ilab/projects/LobodaTFO/data/WRF_Data/NCEP_FNL',
+            start_date=start_date.strftime('%Y-%m-%d'),
+            end_date=end_date.strftime('%Y-%m-%d')
+        )
+        downloader.download()

wildfire_occurrence/model/pipelines/wrf_pipeline.py

@@ -0,0 +1,71 @@
+import os
+import logging
+import datetime
+from wildfire_occurrence.model.config import Config
+from wildfire_occurrence.model.common import read_config
+from wildfire_occurrence.model.data_download.ncep_fnl import NCEP_FNL
+
+
+class WRFPipeline(object):
+
+    def __init__(
+                self,
+                config_filename: str,
+                start_date: str,
+                forecast_lenght: str
+            ):
+
+        # Configuration file intialization
+        self.conf = read_config(config_filename, Config)
+        logging.info(f'Loaded configuration from {config_filename}')
+
+        # Set value for forecast start and end date
+        self.start_date = start_date
+        self.end_date = self.start_date + datetime.timedelta(
+            days=forecast_lenght)
+        logging.info(f'WRF start: {self.start_date}, end: {self.end_date}')
+
+        # Generate working directories
+        os.makedirs(self.conf.working_dir, exist_ok=True)
+        logging.info(f'Created working directory {self.conf.working_dir}')
+
+        # Setup working directories and dates
+        self.output_dir = os.path.join(
+            self.conf.working_dir,
+            f'{self.start_date.strftime("%Y-%m-%d")}_' +
+            f'{self.start_date.strftime("%Y-%m-%d")}'
+        )
+        os.makedirs(self.output_dir, exist_ok=True)
+        logging.info(f'Created output directory {self.output_dir}')
+
+        # Setup data_dir
+        self.data_dir = os.path.join(self.output_dir, 'data')
+
+    # -------------------------------------------------------------------------
+    # download
+    # -------------------------------------------------------------------------
+    def download(self):
+
+        # Working on the setup of the project
+        logging.info('Starting download pipeline step')
+
+        # Generate subdirectories to work with WRF
+        os.makedirs(self.data_dir, exist_ok=True)
+        logging.info(f'Created data directory {self.data_dir}')
+
+        # Generate data downloader
+        data_downloader = NCEP_FNL(
+            self.data_dir,
+            self.start_date,
+            self.end_date
+        )
+        data_downloader.download()
+
+        return
+
+    # -------------------------------------------------------------------------
+    # download
+    # -------------------------------------------------------------------------
+    def geogrid(self):
+        logging.info('Running geogrid')
+        return

wildfire_occurrence/view/wrf_pipeline_cli.py

@@ -0,0 +1,91 @@
+import sys
+import time
+import logging
+import argparse
+from datetime import date
+from wildfire_occurrence.model.common import valid_date
+from wildfire_occurrence.model.pipelines.wrf_pipeline import WRFPipeline
+
+
+# -----------------------------------------------------------------------------
+# main
+#
+# python wrf_pipeline_cli.py -c config.yaml -sd 2023-04-05 \
+#   -ed 2023-04-05 -s all
+# -----------------------------------------------------------------------------
+def main():
+
+    # Process command-line args.
+    desc = 'Use this application to perform CNN regression.'
+    parser = argparse.ArgumentParser(description=desc)
+
+    parser.add_argument('-c',
+                        '--config-file',
+                        type=str,
+                        required=True,
+                        dest='config_file',
+                        help='Path to the configuration file')
+
+    parser.add_argument('-d',
+                        '--start-date',
+                        type=valid_date,
+                        required=False,
+                        default=date.today(),
+                        dest='start_date',
+                        help='Start date for WRF')
+
+    parser.add_argument('-l',
+                        '--forecast-lenght',
+                        type=int,
+                        required=False,
+                        default=10,
+                        dest='forecast_lenght',
+                        help='Lenght of WRF forecast')
+
+    parser.add_argument(
+                        '-s',
+                        '--pipeline-step',
+                        type=str,
+                        nargs='*',
+                        required=True,
+                        dest='pipeline_step',
+                        help='Pipeline step to perform',
+                        default=[
+                            'download', 'geogrid',
+                            'ubgrib', 'real', 'wrf', 'all'],
+                        choices=[
+                            'download', 'geogrid',
+                            'ubgrib', 'real', 'wrf', 'all'])
+
+    args = parser.parse_args()
+
+    # Setup logging
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+    ch = logging.StreamHandler(sys.stdout)
+    ch.setLevel(logging.INFO)
+    formatter = logging.Formatter(
+        "%(asctime)s; %(levelname)s; %(message)s", "%Y-%m-%d %H:%M:%S"
+    )
+    ch.setFormatter(formatter)
+    logger.addHandler(ch)
+
+    # Setup timer to monitor script execution time
+    timer = time.time()
+
+    # Initialize pipeline object
+    pipeline = WRFPipeline(
+        args.config_file, args.start_date, args.forecast_lenght)
+
+    # Regression CHM pipeline steps
+    if "download" in args.pipeline_step or "all" in args.pipeline_step:
+        pipeline.download()
+
+    logging.info(f'Took {(time.time()-timer)/60.0:.2f} min.')
+
+
+# -----------------------------------------------------------------------------
+# Invoke the main
+# -----------------------------------------------------------------------------
+if __name__ == "__main__":
+    sys.exit(main())