Skip to content
Snippets Groups Projects
Commit 23b4f0d8 authored by ranchodeluxe's avatar ranchodeluxe
Browse files

dask is stupid

parent 26b9fd4c
No related branches found
No related tags found
2 merge requests!106Parallelize Uploads to S3 Where Necessary,!61Additions: primary keys, speed ups (preprocessing and postprocessing), tests, scalene, reorg entire fireatlas
Hide whitespace changes
Inline Side-by-side
fireatlas/FireRunDaskCoordinator.py
+ 1
− 2
View file @ 23b4f0d8
......@@ -132,8 +132,7 @@ def Run(region: Region, tst: TimeStep, ted: TimeStep):
# blocks on region_and_t_results and then runs fire forward algorithm (which cannot be run in parallel)
fire_forward_results = delayed(job_fire_forward)(region_and_t_results, region, tst, ted)
# for some reason we can't run this step as part of the above DAG?
# uploads all snapshots/largefire outputs in parallel
# block on fire forward output and upload all snapshots/largefire outputs in parallel
data_dir = os.path.join(settings.LOCAL_PATH, settings.OUTPUT_DIR, region[0], str(tst[0]))
fgb_upload_results = [
delayed(concurrent_copy_outputs_from_local_to_s3)([fire_forward_results,], local_filepath)
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment