Skip to content
Snippets Groups Projects

Combine LargeFire Workflow Addiitons

Merged gcorradini requested to merge feature/combine_lf_archive into conus-dps

Changes

  1. add MAAP to VEDA s3 copy
  2. add dask worker parallelism around years loop
  3. write merge function for each layer across years
  4. make sure writes before merge always happen on disk (as opposed to s3) to avoid race conditions

Testing

  1. log into ADE
  2. cd ./fireatlas_nrt/
  3. run the only year we have data for:
python3 combine_largefire.py -s 2023 -e 2023
# or run the dask worker version in parallel (doesn't do much until we can run multiple years)
python3 combine_largefire.py -s 2023 -e 2023 -p
  1. after it finishes running check that outputs in /projects/shared-buckets/gsfc_landslides/FEDSoutput-s3-conus/CONUS_NRT_DPS/LargeFire_Outputs/merged were updated
  2. grok the tail of the logs (running.log) to make sure there are no errors

Merge request reports

Merged by gcorradinigcorradini 2 years ago (Apr 24, 2023 5:00pm UTC)

Loading

Pipeline #5225 passed

Pipeline passed for 6fe2a6dd on conus-dps

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading