@@ -112,7 +112,7 @@ The result of the download will be a directory structure under your 'output-dir'
...
@@ -112,7 +112,7 @@ The result of the download will be a directory structure under your 'output-dir'
find . -name "*.gpkg.bz2" | bunzip2
find . -name "*.gpkg.bz2" | bunzip2
```
```
Or, if there are thousands of files and you're on a powerful computer, do it in parallel:
Or, if there are thousands of files and you're on a powerful computer with GNU parallel, you could do the following. Python based-solutions like dask or futures also work, of course.
```
```
find . -name "*.gpkg.bz2" | parallel bunzip2
find . -name "*.gpkg.bz2" | parallel bunzip2
...
@@ -135,8 +135,8 @@ The script accepts these required arguments:
...
@@ -135,8 +135,8 @@ The script accepts these required arguments:
| `-u/--username` | MAAP username for job tracking |
| `-u/--username` | MAAP username for job tracking |
| `-t/--tag` | Unique identifier for the processing run |
| `-t/--tag` | Unique identifier for the processing run |
| `-c/--config` | Path to YAML config file with filters/pipeline |
| `-c/--config` | Path to YAML config file with filters/pipeline |
| `--hse` | Path to Height Scaling Exponent GeoTIFF |
| `--hse` | Path to height scaling exponent GeoTIFF |
| `--k_allom` | Path to Allometric Coefficient GeoTIFF |
| `--k_allom` | Path to allometric coefficient GeoTIFF |
| `-a/--algo_id` | MAAP algorithm ID to execute |
| `-a/--algo_id` | MAAP algorithm ID to execute |
| `-v/--algo_version` | Version of the algorithm to run |
| `-v/--algo_version` | Version of the algorithm to run |
...
@@ -179,54 +179,3 @@ Uses NASA CMR temporal syntax:
...
@@ -179,54 +179,3 @@ Uses NASA CMR temporal syntax:
All job outputs are stored in the MAAP workspace S3 bucket under a directory structure that includes your username and the specified job tag. For example, if your username is "jsmith" and you used the tag "biomass_2020", the outputs would be in "s3://maap-ops-workspace/jsmith/dps_output/nmbim_biomass_index/main/biomass_2020". You can use AWS CLI tools to locate and download all GeoPackage files within this directory:
```bash
# Get a list of all the output files in the bucket
aws s3 ls s3://maap-ops-workspace/jsmith/dps_output/nmbim_biomass_index/main/biomass_2020/ --recursive | grep'.gpkg.bz2$'
The output of a single model run through run_on_maap.py is a hierarchical directory structure containing output GeoPackage files. The method chosen to process these files will depend on the end goal, but one workflow is outlined below. Future work could automate this process.
1. Download compressed GeoPackages from S3 to your local machine:
```bash
# Get a list of all the output files in the bucket
aws s3 ls s3://maap-ops-workspace/jsmith/dps_output/nmbim_biomass_index/main/biomass_2020/ --recursive | grep'.gpkg.bz2$'