Skip to content
Snippets Groups Projects
Commit 338ec421 authored by Neha Hunka's avatar Neha Hunka
Browse files

Classifying the world into IPCC Table 4.7

parent 55decbc4
No related branches found
Tags IPCC_classes_ADE_v4
No related merge requests found
Hide whitespace changes
Inline Side-by-side
country_summaries/IPCC_classes_DPS/Boreal/.ipynb_checkpoints/boreal_biomass_in_IPCC-checkpoint.ipynb
+ 51
− 60
View file @ 338ec421
%% Cell type:code id:142b1675-bca8-4bb8-b62a-30e6abef26c9 tags:
``` python
!pip install geopandas
!pip install backoff
!pip install statsmodels
!pip install seaborn
!pip install scipy
!pip install mapclassify
!pip install unidecode
!pip install rasterio
!pip install seaborn
!pip install mpl_scatter_density
!pip install rasterstats
!pip install xmltodict
!pip install pandas
```
%% Output
Requirement already satisfied: geopandas in /opt/conda/lib/python3.10/site-packages (0.14.1)
Requirement already satisfied: fiona>=1.8.21 in /opt/conda/lib/python3.10/site-packages (from geopandas) (1.9.5)
Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from geopandas) (23.2)
Requirement already satisfied: pandas>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (2.1.3)
Requirement already satisfied: pyproj>=3.3.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (3.6.1)
Requirement already satisfied: shapely>=1.8.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (2.0.2)
Requirement already satisfied: attrs>=19.2.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (2023.11.17)
Requirement already satisfied: click~=8.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (8.1.7)
Requirement already satisfied: click-plugins>=1.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (1.1.1)
Requirement already satisfied: cligj>=0.5 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (0.7.2)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (1.16.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (68.2.2)
Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (1.26.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2023.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: backoff in /opt/conda/lib/python3.10/site-packages (2.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: statsmodels in /opt/conda/lib/python3.10/site-packages (0.14.0)
Requirement already satisfied: numpy>=1.18 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (1.26.1)
Requirement already satisfied: scipy!=1.9.2,>=1.4 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (1.11.4)
Requirement already satisfied: pandas>=1.0 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (2.1.3)
Requirement already satisfied: patsy>=0.5.2 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (0.5.3)
Requirement already satisfied: packaging>=21.3 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (23.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2023.3)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from patsy>=0.5.2->statsmodels) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: seaborn in /opt/conda/lib/python3.10/site-packages (0.13.0)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /opt/conda/lib/python3.10/site-packages (from seaborn) (1.26.1)
Requirement already satisfied: pandas>=1.2 in /opt/conda/lib/python3.10/site-packages (from seaborn) (2.1.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.3 in /opt/conda/lib/python3.10/site-packages (from seaborn) (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.3->seaborn) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: scipy in /opt/conda/lib/python3.10/site-packages (1.11.4)
Requirement already satisfied: numpy<1.28.0,>=1.21.6 in /opt/conda/lib/python3.10/site-packages (from scipy) (1.26.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: mapclassify in /opt/conda/lib/python3.10/site-packages (2.6.1)
Requirement already satisfied: networkx>=2.7 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (3.2.1)
Requirement already satisfied: numpy>=1.23 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.26.1)
Requirement already satisfied: pandas!=1.5.0,>=1.4 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (2.1.3)
Requirement already satisfied: scikit-learn>=1.0 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.3.2)
Requirement already satisfied: scipy>=1.8 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.11.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2023.3)
Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn>=1.0->mapclassify) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn>=1.0->mapclassify) (3.2.0)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas!=1.5.0,>=1.4->mapclassify) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: unidecode in /opt/conda/lib/python3.10/site-packages (1.3.7)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: rasterio in /opt/conda/lib/python3.10/site-packages (1.3.9)
Requirement already satisfied: affine in /opt/conda/lib/python3.10/site-packages (from rasterio) (2.4.0)
Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from rasterio) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from rasterio) (2023.11.17)
Requirement already satisfied: click>=4.0 in /opt/conda/lib/python3.10/site-packages (from rasterio) (8.1.7)
Requirement already satisfied: cligj>=0.5 in /opt/conda/lib/python3.10/site-packages (from rasterio) (0.7.2)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.26.1)
Requirement already satisfied: snuggs>=1.4.1 in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.4.7)
Requirement already satisfied: click-plugins in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.1.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from rasterio) (68.2.2)
Requirement already satisfied: pyparsing>=2.1.6 in /opt/conda/lib/python3.10/site-packages (from snuggs>=1.4.1->rasterio) (3.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: seaborn in /opt/conda/lib/python3.10/site-packages (0.13.0)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /opt/conda/lib/python3.10/site-packages (from seaborn) (1.26.1)
Requirement already satisfied: pandas>=1.2 in /opt/conda/lib/python3.10/site-packages (from seaborn) (2.1.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.3 in /opt/conda/lib/python3.10/site-packages (from seaborn) (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.3->seaborn) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: mpl_scatter_density in /opt/conda/lib/python3.10/site-packages (0.7)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (1.26.1)
Requirement already satisfied: matplotlib>=3.0 in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (3.8.0)
Requirement already satisfied: fast-histogram>=0.3 in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (0.12)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (2.8.2)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib>=3.0->mpl_scatter_density) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: rasterstats in /opt/conda/lib/python3.10/site-packages (0.19.0)
Requirement already satisfied: affine in /opt/conda/lib/python3.10/site-packages (from rasterstats) (2.4.0)
Requirement already satisfied: click>7.1 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (8.1.7)
Requirement already satisfied: cligj>=0.4 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (0.7.2)
Requirement already satisfied: fiona in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.9.5)
Requirement already satisfied: numpy>=1.9 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.26.1)
Requirement already satisfied: rasterio>=1.0 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.3.9)
Requirement already satisfied: simplejson in /opt/conda/lib/python3.10/site-packages (from rasterstats) (3.19.2)
Requirement already satisfied: shapely in /opt/conda/lib/python3.10/site-packages (from rasterstats) (2.0.2)
Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (2023.11.17)
Requirement already satisfied: snuggs>=1.4.1 in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (1.4.7)
Requirement already satisfied: click-plugins in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (1.1.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (68.2.2)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from fiona->rasterstats) (1.16.0)
Requirement already satisfied: pyparsing>=2.1.6 in /opt/conda/lib/python3.10/site-packages (from snuggs>=1.4.1->rasterio>=1.0->rasterstats) (3.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: xmltodict in /opt/conda/lib/python3.10/site-packages (0.13.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (2.1.3)
Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/lib/python3.10/site-packages (from pandas) (1.26.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

%% Cell type:code id:b52092ac-f150-4ea5-ba54-b5279cd24c2e tags:
``` python
import os
import subprocess
import geopandas
import pandas as pd
import json
import datetime
import backoff
import xmltodict
import fileinput
import xml.etree.ElementTree as ET
from urllib.parse import urlparse
import fileinput
import glob
import matplotlib.pyplot as plt
# import numpy as np
import warnings
warnings.filterwarnings("ignore")
warnings.filterwarnings( "ignore", module = "matplotlib\..*" )
import statsmodels.api as sm
import statsmodels.formula.api as smf
import seaborn as sns
import scipy.stats
import scipy.stats as stats
from statsmodels import graphics
import shutil
from mpl_toolkits.axes_grid1 import make_axes_locatable
from unidecode import unidecode
import rasterio
from rasterio.plot import show
import seaborn
# import mpl_scatter_density # adds projection='scatter_density'
from matplotlib.colors import LinearSegmentedColormap
import matplotlib.colors
from textwrap import wrap
import matplotlib.patches as mpatches
import pylab as py
import rasterstats
from rasterstats import zonal_stats
import scipy
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
import pandas as pd
import subprocess
import glob
import shutil
from maap.maap import MAAP
# maap=MAAP(maap_host="api.ops.maap-project.org")
maap=MAAP()
def BUILD_TABLE_JOBSTATUS(submit_results_df, status_col = 'status'):
import xmltodict
# If jobs failed to submit, then they have a NaN for jobid, which makes the merge (join) fail
submit_results_df = submit_results_df.fillna('')
job_status_df = pd.concat([pd.DataFrame({'job_id': [job_id], 'status':[maap.getJobStatus(job_id)]}) for job_id in submit_results_df.job_id.to_list()])
job_status_df = submit_results_df.merge(job_status_df, how='left', left_on='job_id', right_on='job_id')
print(f'Count total jobs:\t{len(job_status_df)}')
print(f"Count pending jobs:\t{job_status_df[job_status_df[status_col] =='Accepted'].shape[0]}")
print(f"Count running jobs:\t{job_status_df[job_status_df[status_col] =='Running'].shape[0]}")
NUM_FAILS = job_status_df[job_status_df[status_col] =='Failed'].shape[0]
NUM_SUCCEEDS = job_status_df[job_status_df[status_col] =='Succeeded'].shape[0]
print(f"Count succeeded jobs:\t{NUM_SUCCEEDS}")
print(f"Count failed jobs:\t{NUM_FAILS}")
if NUM_FAILS > 0:
print(f"% of failed jobs:\t{round(NUM_FAILS / ( NUM_FAILS + NUM_SUCCEEDS ), 4) * 100}\n")
else:
print(f"% of failed jobs:\tNothing has failed...yet\n")
return job_status_df
```
%% Cell type:code id:3a2b480f-a23e-49fa-b26b-392f27bd9cef tags:
``` python
# cd /projects/ADE_biomass_harmonization/biomass_harmonization/
# git add country_summaries/IPCC_classes_DPS/Boreal
# git commit -m 'Boreal - Classifying the world into IPCC Table 4.7'
# git tag -f Boreal_IPCC_classes_ADE
# git push origin -f Boreal_IPCC_classes_ADE
# git push
# git push origin -f Boreal_IPCC_classes_ADE
########### REGISTER ALGORITHM ON NEW ADE ##############
# maap.register_algorithm_from_yaml_file("/projects/ADE_biomass_harmonization/biomass_harmonization/country_summaries/IPCC_classes_DPS/Boreal/boreal_run_IPCC_classes_ADE.yml").text
```
%% Cell type:code id:63d7cc00-e8bd-47c6-8394-db3c8ef6bfec tags:
``` python
Desired_output_folder = '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/BOREAL_compiled'
if not os.path.exists(Desired_output_folder):
os.mkdir(Desired_output_folder)
###############################################################################################################################
############ SUPPLY A GEOPACKAGE FILE THAT HOLDS THE UNIQUE ID OF THE VECTORS #################################################
# THIS VECTOR IS SPLIT BY THE GEDI - ICESat-2 BORDER AND RENAMED
# ONE FILE IS USED FOR PROCESSING ON MAAP, THE OTHER GIVEN TO THE GEDI L4 TEAM TO GENERATE HYBRID ESTIMATES ###################
###############################################################################################################################
# CUTLINE = geopandas.read_file('/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/GEDI_ICESat_CUTLINE.gpkg')
# VECTOR = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/NE_countries_HNL_DISS_v2.gpkg' #MUST BE A GPKG
# VECTOR = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/Global_IPCC_classes_ADE_v6_GPKG/IPCC_classes_globe_v6_DISS.gpkg'
# VECTOR_EDIT = geopandas.read_file(VECTOR)
# VECTOR_BOREAL = geopandas.clip(VECTOR_EDIT, CUTLINE)
# VECTOR_GEDI = geopandas.overlay(VECTOR_EDIT, CUTLINE, how='difference')
# VECTOR_BOREAL.to_file("/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_BOREAL.gpkg", driver="GPKG")
# VECTOR_GEDI.to_file("/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_GEDI.gpkg", driver="GPKG")
# VECTOR_GEDI.plot()
# VECTOR_BOREAL.plot()
VECTOR_BOREAL = "/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_BOREAL.gpkg"
ADM = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg'
if not os.path.exists(ADM): shutil.copy(VECTOR_BOREAL,ADM)
TILES = ["CHN","FRA","PRK","UKR","BLR","UZB","KAZ","TJK","LTU","MNG","DEU","EST","LVA","NOR","SWE","FIN","LUX","BEL","KGZ","DNK","ROU","HUN","SVK","POL","IRL","GBR","NLD","MDA","USA","GRL","JPN","ISL"]
TILES = ["4232","4332","4132","4333","4133","4135","4233","4235","4335","2135","2235","2335","2233","2333","2133","7131","7135","7231","7235","7331","7335","7132","7232","7332","7133","7233","7333","2141","2241","2341","4143","4343","4243","4141","4241","4341","7141","7143","7241","7243","7341","7343","2142","2242","2342","2143","2243","2343","7142","7242","7342","4142","4242","4342"]
TILES = ["4232","4332","4132","4333","4133","4135","4233","4235","2135","2235","2335","2233","2333","2133","7131","7135","7231","7235","7331","7335","7232","7133","7233","7333","2141","2241","2341","4343","4243","4141","4241","4341","7141","7143","7241","7243","7341","7343","2142","2342","2143","2243","2343","7142","7242","7342","4142","4242","4342"]
i = 0
count = -1
submit_results_df_list = []
len_input_list = len(TILES)
list_jobIDs = [""]*len(TILES)
for TILE in TILES:
if not os.path.exists(os.path.join(Desired_output_folder,TILE + "_ISB_AGB_1KM_C.tif")):
print("submitting " + TILE)
count +=1
submit_result = maap.submitJob(identifier="Boreal_IPCC_classes_ADE",
algo_id="run_Boreal_IPCC_classes_ADE",
version="Boreal_IPCC_classes_ADE",
username="nehajo88",
queue="maap-dps-worker-32vcpu-64gb",
queue="maap-dps-worker-16gb", #32vcpu-64
IPCC_class=TILE,
ISB_AGB_1KM= TILE + "_ISB_AGB_1KM_C.tif",
ISB_VAR_1KM= TILE + "_ISB_VAR_1KM_C.tif",
AOI_stats_table = TILE + "_STATS.csv",
boreal_tiles="https://maap-ops-workspace.s3.amazonaws.com/shared/leitoldv/boreal_tiles_v003.gpkg",
adm="https://maap-ops-workspace.s3.amazonaws.com/shared/nehajo88/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg",
AGB_tindex="https://maap-ops-workspace.s3.amazonaws.com/shared/nehajo88/Data/ICESat-2-TILES/AGB_tindex_master.csv")
job_id = str(submit_result).split(': ')[1].split(',')[0].replace("'","")
print(job_id)
```
%% Output
submitting 4232
4cd17e2e-9f38-4cb3-91eb-7abcdfee2352
32abde50-b026-43b5-bf88-44c3b172a5d3
submitting 4332
cb8ca91d-2f94-47f7-a29d-3a425a5cbe9d
a44380f3-6d98-4e29-8183-46f6f493af09
submitting 4132
8fe25adb-a88f-4f30-9877-cead9516a13a
1276c39e-703b-42ff-91f3-e47f3a4142ed
submitting 4333
18c4dbc6-1f2e-4d64-9336-d91ce1c3fee5
de05e00e-9957-413e-b2b6-4ea6f293a1c7
submitting 4133
17276f78-b107-43f2-bda9-c81a63199bcb
095a044a-cdf0-46d8-9f94-1aa9026953db
submitting 4135
9df75878-7269-4376-ac1d-b2ada6cabe95
3e6eec58-10d0-4fd4-b37e-cd93f49a0997
submitting 4233
77508423-c3cb-4e5a-a666-7a8aa7c53f0b
29560900-9de3-463f-ac7c-32b7053e5511
submitting 4235
8c1b244c-281d-4b58-a49e-035edc831d61
submitting 4335
f7b1aa12-d940-4d60-a533-66ee2c2867e2
submitting 7131
02bd3437-9513-430b-8f0c-8942a8248576
submitting 7135
3bb09fd1-52c4-4c98-a12e-549d45c58907
submitting 7231
147e24d1-4915-4135-a75e-8274e6d71ba4
bc98436b-1254-46e1-9195-26920ae6e92e
submitting 2133
35f59bb8-a891-46ee-9f96-c55d551af33f
submitting 7235
36485303-bf3c-4c66-b173-9468ca5d79bc
submitting 7331
43058bef-c076-485c-8b94-6c4cbd249d11
ddf9e774-8438-4384-977d-89dcebf0bda9
submitting 7335
e2ded0d3-7c72-441a-b9b0-4e1664d086aa
submitting 7132
3f2cc849-64fa-41d8-89d9-4a82e5c308a1
c959d17b-bb5d-46cb-afd0-02629fca8800
submitting 7232
cdc1f767-467b-41f2-af78-74a959472a3c
submitting 7332
9e092c8e-db0a-428d-a108-75b0fe60a2d9
submitting 7133
5476afdd-2678-46f9-b932-8f0e29a9c800
submitting 7233
f2b4cae0-6017-4cab-9d75-d848ff66dea7
submitting 7333
4c6712af-2362-4712-81be-55271a87dd87
submitting 4143
d0195b9d-7c80-4d70-b4cd-2fedda3fc58c
25fcb409-58dc-42ef-a965-d10fcbb9f006
submitting 2141
61499601-c880-4951-9c30-0e1c1e25d9bd
submitting 2241
6b0a9822-87eb-48e8-ab51-03ea4d1e19df
submitting 2341
164fe631-e098-4c79-9c27-cfbcaecf9f8d
submitting 4343
28b881d1-64c9-4c2c-b49c-7e03772fc6d3
7b69b651-4b09-4eb8-8ad3-e883389ed709
submitting 4243
eb5e52bc-a6c7-4df1-85d7-0c0f473e52b5
f661fa41-4de4-4f72-baee-adebe1b172a3
submitting 4141
89587323-c51d-4b24-bd4e-3ac9896d4a8b
63a7442f-a0ef-46d7-b25d-83bcee096b7f
submitting 4241
1926a178-e2b8-47b2-a98c-82cad4bf5edb
8233dc57-f9a3-4ca8-b3d7-b04a8b3f035b
submitting 4341
dfd66493-0748-4031-8d21-9b73c25da7ea
c6491ce4-1b1e-4791-bac6-7e2264cb0953
submitting 7141
b9b0a286-d8e7-4e47-b58a-415560b9b31f
6413abac-5159-473d-824f-c6cac37b3664
submitting 7143
ded0998b-7f8a-4df7-8db1-62e50b4c3df3
ceb63089-b756-42e6-a068-002e35760221
submitting 7241
b96a1d74-a3ef-4d1f-b996-36a10613f762
f68ce7ed-1390-416a-9cce-a4a8ac43193b
submitting 7243
dfd7c5ae-03b2-42b7-9d67-ed9aa921aeb1
9f7ea488-b643-455c-baec-ac6997a91819
submitting 7341
32edf307-49db-481e-9093-9c46366d2bba
b0b57fbc-aeb8-40a5-b332-b7146eea1542
submitting 7343
6a79c67f-5883-47c1-a88d-eb24a7dc065d
submitting 7142
0d7aeddb-9ace-4cbf-97af-6c68828e774d
submitting 7242
7c0d162f-e64d-4d7c-b041-5095e8b61752
submitting 7342
95e5d9c6-6485-4c5c-abcc-4df16a9b78e8
f8b96ef1-5c30-475f-816d-9697a3a7c37d
submitting 2142
506c505c-5a29-4278-98ab-8025a5131d27
submitting 2342
8827cfe4-6278-4a89-ac0e-52d531cd1fd8
submitting 2143
3d437654-08b8-4040-8dd2-f9e325b77c6c
submitting 2243
97009f4f-2f41-492c-817e-a8e88db7a9ee
submitting 2343
9cea5a99-f03c-4864-b5b3-09021058f02f
submitting 4142
10313297-abc0-48db-8def-06ee45d17b10
9546865b-1a6a-456b-a0e6-acc6d46d9f7f
submitting 4242
ce58ac75-9b78-4395-b6f9-614defa53350
2f8fb61e-12a2-4a69-848e-f24ce4eef904
submitting 4342
01e1243f-0854-4e62-bb91-35d9a972a722
ccaa3459-ae0d-4693-a576-e026c2a7625f
%% Cell type:code id:d7007b5e-0b3f-440d-b8cf-02eb330a5fef tags:
``` python
maap.getJobResult('4c427afd-2ef6-4258-aac2-5672c66aaa7b')
```
%% Output
['http://maap-ops-workspace.s3-website-us-west-2.amazonaws.com/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b',
's3://s3-us-west-2.amazonaws.com:80/maap-ops-workspace/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b',
'https://s3.console.aws.amazon.com/s3/buckets/maap-ops-workspace/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b/?region=us-east-1&tab=overview',
'++ python /app/get_username.py\n+ USERNAME=nehajo88\n+ DPS_MACHINE_TOKEN=Y6hERQKLKs0oH0h5Kgi3d-gVVyHgW_DR77YeARdzd-a3KJDE-gYldJzhB94s635KySUXi4r9WOtD6Vo_F9cEbfNI_QUhOG8N47heXHVcW1k3vEw3d0StttXYMOnQPtTzSQLAYKQwJLgbtX-R8ug1XGVeMolh97P9nHcW8awN7yjyaR7g\n++ curl -sb -H \'Accept: application/json\' -H \'Content-Type: application/json\' -H \'dps-token: Y6hERQKLKs0oH0h5Kgi3d-gVVyHgW_DR77YeARdzd-a3KJDE-gYldJzhB94s635KySUXi4r9WOtD6Vo_F9cEbfNI_QUhOG8N47heXHVcW1k3vEw3d0StttXYMOnQPtTzSQLAYKQwJLgbtX-R8ug1XGVeMolh97P9nHcW8awN7yjyaR7g\' https://api.maap-project.org/api/members/nehajo88\n++ jq -r .session_key\n+ export MAAP_PGT=PGT-12-JV4ZKLoVctV2kSSpfHn6OOXIossEXUR7CbydSW4SR8TNr7Mv9KRTov2nz-jslvcFnio-8445019b059c\n+ MAAP_PGT=PGT-12-JV4ZKLoVctV2kSSpfHn6OOXIossEXUR7CbydSW4SR8TNr7Mv9KRTov2nz-jslvcFnio-8445019b059c\n+ unset DPS_MACHINE_TOKEN\n+ /app/biomass_harmonization/country_summaries/IPCC_classes_DPS/Boreal/boreal_run_IPCC_classes_ADE.sh POL POL_ISB_AGB_1KM_C.tif POL_ISB_VAR_1KM_C.tif POL_STATS.csv\nLoading required package: raster\nLoading required package: sp\nThe legacy packages maptools, rgdal, and rgeos, underpinning the sp package,\nwhich was just loaded, will retire in October 2023.\nPlease refer to R-spatial evolution reports for details, especially\nhttps://r-spatial.org/r/2023/05/15/evolution4.html.\nIt may be desirable to make the sf package available;\npackage maintainers should consider adding sf to Suggests:.\nThe sp package is now running under evolution status 2\n (status 2 uses the sf package in place of rgdal)\nLoading required package: rgdal\nPlease note that rgdal will be retired by the end of 2023,\nplan transition to sf/stars/terra functions using GDAL and PROJ\nat your earliest convenience.\n\nrgdal: version: 1.5-32, (SVN revision 1176)\nGeospatial Data Abstraction Library extensions to R successfully loaded\nLoaded GDAL runtime: GDAL 3.6.2, released 2023/01/02\nPath to GDAL shared files: \n GDAL does not use iconv for recoding strings.\nGDAL binary built with GEOS: TRUE \nLoaded PROJ runtime: Rel. 9.1.0, September 1st, 2022, [PJ_VERSION: 910]\nPath to PROJ shared files: /opt/conda/share/proj\nPROJ CDN enabled: FALSE\nLinking to sp version:1.5-1\nTo mute warnings of possible GDAL/OSR exportToProj4() degradation,\nuse options("rgdal_show_exportToProj4_warnings"="none") before loading sp or rgdal.\nLoading required package: terra\nterra 1.7.29\n\nAttaching package: ‘terra’\n\nThe following object is masked from ‘package:rgdal’:\n\n project\n\nLoading required package: sf\nLinking to GEOS 3.11.1, GDAL 3.6.2, PROJ 9.1.0; sf_use_s2() is TRUE\nLoading required package: RCurl\nError in library(x, character.only = TRUE, quietly = TRUE) : \n there is no package called ‘RCurl’\nCalls: lapply -> FUN -> library\nIn addition: Warning message:\nIn library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :\n there is no package called ‘RCurl’\nExecution halted\n+ cp _stderr.txt _alt_traceback.txt\n']
%% Cell type:code id:a91aad0d-86d8-421f-80a7-5951d3f5b239 tags:
``` python
Desired_output_folder = '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/BOREAL_compiled'
if not os.path.exists(Desired_output_folder):
os.mkdir(Desired_output_folder)
SUMMARY_files = glob.glob('/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/**/*.tif', recursive=True)
print("Moving files...")
print(SUMMARY_files)
for each_file in SUMMARY_files:
if os.path.exists(each_file) and not os.path.exists(os.path.join(Desired_output_folder,os.path.basename(each_file))):
# os.remove(os.path.join(Desired_output_folder,os.path.basename(each_file)))
shutil.move(each_file, Desired_output_folder)
SUMMARY_files = glob.glob('/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/**/*.csv', recursive=True)
print("Moving files...")
print(SUMMARY_files)
for each_file in SUMMARY_files:
if os.path.exists(each_file) and not os.path.exists(os.path.join(Desired_output_folder,os.path.basename(each_file))):
# os.remove(os.path.join(Desired_output_folder,os.path.basename(each_file)))
shutil.move(each_file, Desired_output_folder)
```
%% Output
Moving files...
[]
['/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_ISB_AGB_1KM_C.tif', '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_ISB_VAR_1KM_C.tif']
Moving files...
[]
['/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_STATS.csv', '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/30/04/44/03/257748/4332_STATS.csv']
%% Cell type:code id:visible-tampa tags:
``` python
##############################################################################
### FIRST, TO INSTALL PACKAGES, PLEASE OPEN A TERMINAL AND RUN THE FOLLOWING LINE:
### bash /projects/my-public-bucket/_ENV/env/build_r_env.sh
##############################################################################
packages <- c("raster", "rgdal", "terra", "sf", "sp")
package.check <- lapply(packages, FUN = function(x) {
if (!require(x, character.only = TRUE)) {
# install.packages(x, dependencies = TRUE)
library(x, character.only = TRUE, quietly=TRUE)
library(geojsonio)
}
})
```
%% Output
Loading required package: raster
Loading required package: sp
The legacy packages maptools, rgdal, and rgeos, underpinning the sp package,
which was just loaded, were retired in October 2023.
Please refer to R-spatial evolution reports for details, especially
https://r-spatial.org/r/2023/05/15/evolution4.html.
It may be desirable to make the sf package available;
package maintainers should consider adding sf to Suggests:.
Loading required package: rgdal
Please note that rgdal will be retired during 2023,
plan transition to sf/stars/terra functions using GDAL and PROJ
at your earliest convenience.
See https://r-spatial.org/r/2022/04/12/evolution.html and https://github.com/r-spatial/evolution
rgdal: version: 1.6-6, (SVN revision 1201)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.6.2, released 2023/01/02
Path to GDAL shared files: /opt/conda/share/gdal
GDAL does not use iconv for recoding strings.
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 9.1.0, September 1st, 2022, [PJ_VERSION: 910]
Path to PROJ shared files: /projects/.local/share/proj:/opt/conda/share/proj
PROJ CDN enabled: FALSE
Linking to sp version:1.6-0
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading sp or rgdal.
Loading required package: terra
terra 1.7.29
Attaching package: ‘terra’
The following object is masked from ‘package:rgdal’:
project
Loading required package: sf
Linking to GEOS 3.11.1, GDAL 3.6.2, PROJ 9.1.0; sf_use_s2() is TRUE
%% Cell type:code id:9fcec9f5-be89-45bc-a90b-c4ba23ec183d tags:
``` python
urlExists <- function(address) {
tryCatch( {
con <- url(address)
a <- capture.output(suppressWarnings(readLines(con)))
close(con)
TRUE;
},
error = function(err) {
occur <- grep("cannot open the connection", capture.output(err));
if(length(occur) > 0) FALSE;
} )
}
urlExists("https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/boreal_agb_202302061675666484_26022.tif")
```
%% Output
FALSE
FALSE
%% Cell type:code id:protecting-dynamics tags:
``` python
Out_folder <- "/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results"
if (!dir.exists(Out_folder)){dir.create(Out_folder)}
```
%% Cell type:code id:significant-interaction tags:
``` python
## read in boreal tiles grid (save projection information for later)
boreal_tiles <- readOGR('/projects/shared-buckets/leitoldv/boreal_tiles_v003.gpkg',verbose=F)
boreal_proj4 <- crs(boreal_tiles)
```
%% Output
Warning message in OGRSpatialRef(dsn, layer, morphFromESRI = morphFromESRI, dumpSRS = dumpSRS, :
“Discarded datum unknown in Proj4 definition: +proj=aea +lat_0=40 +lon_0=180 +lat_1=50 +lat_2=70 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs,
but +towgs84= values preserved”
%% Cell type:code id:lesbian-driver tags:
``` python
IPCC_class <- "DEU"
adm <- readOGR("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg",verbose=F)
adm <- adm[adm$IPCC_classes_1km == IPCC_class,]
adm_poly <- polygons(adm)
adm_proj <- spTransform(adm_poly, crs(boreal_tiles))
if (!dir.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))){dir.create(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))}
```
%% Cell type:code id:80589b3c-f1e5-485f-b2bf-a26bc270ec38 tags:
``` python
intersecting <- st_intersection(st_as_sf(boreal_tiles), st_convex_hull(st_as_sf(adm_proj)), byid=TRUE)
if (nrow(intersecting) > 0) {
ICESat_tiles <- intersecting
} else {stop('no intersecting CCI Biomass tiles found')}
plot(adm_proj)
plot(ICESat_tiles, add=T, border="blue")
#ICESat_file <- file.path(Out_folder,paste0(IPCC_class,'_icesat_json.shp'))
#if (!file.exists(ICESat_file)) {writeOGR(obj=ICESat_tiles, dsn=Out_folder, layer=paste0(IPCC_class,'_icesat_json'), driver="ESRI Shapefile")}
```
%% Output
Warning message:
“attribute variables are assumed to be spatially constant throughout all geometries”
Warning message in plot.sf(ICESat_tiles, add = T, border = "blue"):
“ignoring all but the first attribute”
%% Cell type:code id:9d62c649-6ee8-49f8-987a-28d94aaf5451 tags:
``` python
tilelist = as.list(ICESat_tiles$tile_num)
summary(boreal_tiles)
```
%% Output
%% Cell type:code id:united-celebration tags:
``` python
## create tileindex with ICESat2 file paths to selected tiles
AGB_tindex <- read.csv("/projects/shared-buckets/lduncanson/DPS_tile_lists/2023/AGB_tindex_master.csv")
AGB_tindex_out <- c()
for (i in 1:length(tilelist)){
tile_num <- tilelist[i]
print(tile_num)
AGB_tindex_out <- rbind(AGB_tindex_out, AGB_tindex[AGB_tindex[,"tile_num"]==tile_num,])
}
```
%% Output
[[1]]
[1] "23776"
[[1]]
[1] "23777"
[[1]]
[1] "23778"
[[1]]
[1] "24055"
[[1]]
[1] "24056"
[[1]]
[1] "24057"
[[1]]
[1] "24058"
[[1]]
[1] "24059"
[[1]]
[1] "24335"
[[1]]
[1] "24336"
[[1]]
[1] "24337"
[[1]]
[1] "24338"
[[1]]
[1] "24339"
[[1]]
[1] "24340"
[[1]]
[1] "24341"
[[1]]
[1] "24615"
[[1]]
[1] "24616"
[[1]]
[1] "24617"
[[1]]
[1] "24618"
[[1]]
[1] "24619"
[[1]]
[1] "24620"
[[1]]
[1] "24621"
[[1]]
[1] "24622"
[[1]]
[1] "24896"
[[1]]
[1] "24897"
[[1]]
[1] "24898"
[[1]]
[1] "24899"
[[1]]
[1] "24900"
[[1]]
[1] "24901"
[[1]]
[1] "24902"
[[1]]
[1] "24903"
[[1]]
[1] "25177"
[[1]]
[1] "25178"
[[1]]
[1] "25179"
[[1]]
[1] "25180"
[[1]]
[1] "25181"
[[1]]
[1] "25182"
[[1]]
[1] "25183"
[[1]]
[1] "25184"
[[1]]
[1] "25459"
[[1]]
[1] "25460"
[[1]]
[1] "25461"
[[1]]
[1] "25462"
[[1]]
[1] "25463"
[[1]]
[1] "25464"
[[1]]
[1] "25465"
[[1]]
[1] "25741"
[[1]]
[1] "25742"
[[1]]
[1] "25743"
[[1]]
[1] "25744"
[[1]]
[1] "25745"
[[1]]
[1] "25746"
[[1]]
[1] "26020"
[[1]]
[1] "26021"
[[1]]
[1] "26022"
%% Cell type:code id:d05ddc0d-d864-451a-854a-62fa29af14a1 tags:
``` python
##### DOWNLOAD NECESSARY FILES FROM NASA ORNL DAAC
PATHS <- AGB_tindex_out$s3_path
ORNL_username <- 'nhunka'
ORNL_password <- 'Boreal_tiles1'
for (i in 1:length(PATHS)){
DOWNLOAD_LOCATION <- paste0("/projects/my-public-bucket/Data/ICESat-2-boreal/",basename(PATHS[i]))
if (!file.exists(DOWNLOAD_LOCATION)){
PATH_i = paste("wget --user",ORNL_username, "--password ",ORNL_password, paste0("https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/",basename(PATHS[i])), "-O", DOWNLOAD_LOCATION)
print(paste("Downloading...",DOWNLOAD_LOCATION))
system(PATH_i)
}
}
###### NOW ALL THE NECCESARY FILES SHOULD HAVE DOWNLOADED INTO MY FOLDER "/projects/my-public-bucket/Data/ICESat-2-boreal/"
```
%% Cell type:code id:dc70fe87-a735-43e6-9b51-1943fb432143 tags:
``` python
PATH_i
```
%% Output
<span style=white-space:pre-wrap>'wget --user nhunka --password Boreal_tiles1 https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/boreal_agb_202302061675666484_26022.tif -O /projects/my-public-bucket/Data/ICESat-2-boreal/boreal_agb_202302061675666484_26022.tif'</span>
'wget --user nhunka --password Boreal\_tiles1 https://daac.ornl.gov/daacdata/above/Boreal\_AGB\_Density\_ICESat2/data/boreal\_agb\_202302061675666484\_26022.tif -O /projects/my-public-bucket/Data/ICESat-2-boreal/boreal\_agb\_202302061675666484\_26022.tif'
%% Cell type:code id:51079f67-9cea-4717-bc4a-fafacf3d3f9e tags:
``` python
if (!dir.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))){dir.create(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))}
for (i in 1:length(PATHS)){
DOWNLOAD_LOCATION <- paste0("/projects/my-public-bucket/Data/ICESat-2-boreal/",basename(PATHS[i]))
if (file.exists(DOWNLOAD_LOCATION) & (file.info(DOWNLOAD_LOCATION)$size > 0)){
ISB_tile_resampled <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_AGBD_1km.tif")
ISB_tile_SDsquared <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_SDsquared.tif")
ISB_tile_AGBrooted <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_AGBrooted.tif")
ISB_tile_VAR30m <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_VAR30m.tif")
ISB_tile_Variance <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_VAR_1km.tif")
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance))){
print(paste("creating ",ISB_tile_Variance))
##### FIRST - SQUARE THE RASTER - THIS CONVERTS ALL ND VALUES TO 0 AND ESTIMATES THE VARIANCE AT 30 M PIXEL SIZE
r <- rast(DOWNLOAD_LOCATION)
s <- r^2
writeRaster(s, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
##### SECOND - SQUARE ROOT AND WRITE THE AGBD RASTER
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))){
v <- rast(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))
m <- sqrt(v)
writeRaster(m[[1]], file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
}
##### THIRD - WRITE THE VAR RASTER SEPARATELY
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))){
v <- rast(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))
writeRaster(v[[2]], file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))}
}
##### THIRD - RESAMPLE THE AGBD RASTER TO 1 KM
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted))){
# gdalwarp(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted),file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_resampled),b=1,ot="Float32", co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=FALSE, tr=c(1000,1000),r='average')
gdal_utils(util = "warp", source = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted),destination=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_resampled),options=c("-ot","Float32","-co","c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)", "-tr",1000,1000,"-r","average"))
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))}
}
##### FOURTH - RESAMPLE THE VARIANCE RASTER TO 1 KM
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m))){
# gdalwarp(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m),file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance),b=1,ot="Float32", co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=FALSE, tr=c(1000,1000),r='average')
gdal_utils(util = "warp", source = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m),destination=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance),options=c("-ot","Float32","-co","c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)", "-tr",1000,1000,"-r","average"))
}
#### FINAL - REMOVE INTERMEDIATE FILES
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_VAR30m)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_VAR30m)))}
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))}
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))}
}
}
}
#-------------------------------------------------------------------------------
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM_C.tif"))){
tiles <- list.files(path = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')),pattern="*_AGBD_1km.tif",full.names=TRUE)
# mosaic_rasters(gdalfile=tiles,dst_dataset=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"),of="GTiff", gdalwarp_params = list(r = "average",ot="Float32"), co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=TRUE)
mosaic(sprc(tiles), fun="mean", filename=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif")), adm_proj)
tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif")), adm_proj, inverse=FALSE, maskvalues=NA, updatevalue=NA)
writeRaster(tile_stack, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM_C.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"))}
}
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM_C.tif"))){
tiles <- list.files(path = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')),pattern="*_VAR_1km.tif",full.names=TRUE)
# mosaic_rasters(gdalfile=tiles,dst_dataset=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"),of="GTiff", gdalwarp_params = list(r = "average",ot="Float32"), co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=TRUE)
mosaic(sprc(tiles), fun="mean", filename=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif")), adm_proj)
tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif")), adm_proj, inverse=FALSE, maskvalues=NA, updatevalue=NA)
writeRaster(tile_stack, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM_C.tif"), overwrite=TRUE)
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"))}
}
```
%% Output
[1] "creating IPCC_Class_GBR_boreal_agb_202302061675665310_24951_VAR_1km.tif"
Warning message in new_CppObject_xp(fields$.module, fields$.pointer, ...):
“GDAL Message 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Message 1: All options related to creation ignored in update mode”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Message 6: driver GTiff does not support creation option c(COMPRESS”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Error in gdal_utils(util = "warp", source = file.path(Out_folder, paste0(IPCC_class, : gdal_utils warp: an error occured
Traceback:
1. gdal_utils(util = "warp", source = file.path(Out_folder, paste0(IPCC_class,
. "_Product_ISB"), ISB_tile_VAR30m), destination = file.path(Out_folder,
. paste0(IPCC_class, "_Product_ISB"), ISB_tile_Variance), options = c("-ot",
. "Float32", "-co", "c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)",
. "-tr", 1000, 1000, "-r", "average"))
2. stop(paste0("gdal_utils ", util, ": an error occured"))
%% Cell type:code id:84b19785-1ae3-4685-afb9-7b1333f93951 tags:
``` python
(length(sprc(tiles)) > 1)
```
%% Output
TRUE
TRUE
%% Cell type:code id:5d53c3fd-bfc1-43c9-a309-a0b340f435fc tags:
``` python
AOI_in_agbd_1km <- raster("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results/2101_Product_ISB/ISB_AGB_1KM_C.tif")
AOI_in_var_1km <- raster("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results/2101_Product_ISB/ISB_VAR_1KM_C.tif")
```
%% Cell type:code id:7951d8bc-d725-4321-8c0d-26f4d0d51742 tags:
``` python
# par(mfrow=c(1,2))
# plot(AOI_in_agbd_1km, main="agbd")
# plot(AOI_in_var_1km, main="var")
```
%% Output
%% Cell type:code id:f386823e-dfff-48e8-9dde-c1e3c14c1bf8 tags:
``` python
total_convert_Mg <- function(x){(x*100)}
```
%% Cell type:code id:f678956d-8367-4c5f-9d97-47e7c17e7e5e tags:
``` python
## convert AGBD (Mg/ha) to AGB (Mg) for 1km resolution AOIs:
AOI_in_AGB_1km <- total_convert_Mg(AOI_in_agbd_1km)
## extract raster values from given AOI
AOI_rastervalues_AGB <- getValues(AOI_in_AGB_1km)[!is.na(getValues(AOI_in_AGB_1km))]
## calculate summary stats for AOI
AOI_stats <- vector(mode = "list", length = 5)
names(AOI_stats) <- c("AGB_total", "AGB_sd", "AGB_mean", "AGB_se", "AREA")
AOI_stats$AGB_total <- sum(AOI_rastervalues_AGB) ##units: Mg
AOI_stats$AGB_sd <- total_convert_Mg(sqrt(sum(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))]))) ##units: Mg
AOI_stats$AREA <- length(AOI_rastervalues_AGB)*100 ##units: ha
AOI_stats$AGB_mean <- AOI_stats$AGB_total / AOI_stats$AREA ##units: Mg/ha
AOI_stats$AGB_se <- sqrt(sum(AOI_rastervalues_VAR))/sqrt(length(AOI_rastervalues_VAR)-1)
AOI_stats$AGB_mean <- mean(getValues(AOI_in_agbd_1km)[!is.na(getValues(AOI_in_agbd_1km))])
AOI_stats$AGB_se <- sqrt(sum(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))]))/sqrt(length(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))])-1)
AOI_stats_table <- matrix(0, nrow=5, ncol=2)
row.names(AOI_stats_table) <- c("AREA", "AGB_total", "AGB_sd", "AGB_mean", "AGB_se")
colnames(AOI_stats_table) <- c("value", "unit")
AOI_stats_table["AREA", "value"] <- round(AOI_stats$AREA, digits=3) / 10^6
AOI_stats_table["AGB_total", "value"] <- round(AOI_stats$AGB_total, digits=3) / 10^9
AOI_stats_table["AGB_sd", "value"] <- round(AOI_stats$AGB_sd, digits=3) / 10^9
AOI_stats_table["AGB_mean", "value"] <- round(AOI_stats$AGB_mean, digits=3)
AOI_stats_table["AGB_se", "value"] <- round(AOI_stats$AGB_se, digits=3)
AOI_stats_table[,"unit"] <- c("Mha", "Pg", "Pg", "Mg/ha", "Mg")
write.csv(AOI_stats_table, file=paste("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_IPCC_stats",IPCC_class,"_agb_stats_v2023SEP28.csv",sep=""))
```
%% Output
$AGB_total
: 3010633.90445709
$AGB_sd
: 22465.6677379406
$AGB_mean
: 29.1445682909689
$AGB_se
: 6.99325690523318
$AREA
: 103300
\begin{description}
\item[\$AGB\_total] 3010633.90445709
\item[\$AGB\_sd] 22465.6677379406
\item[\$AGB\_mean] 29.1445682909689
\item[\$AGB\_se] 6.99325690523318
\item[\$AREA] 103300
\end{description}
......
country_summaries/IPCC_classes_DPS/Boreal/boreal_biomass_in_IPCC.ipynb
+ 41
− 41
View file @ 338ec421
%% Cell type:code id:142b1675-bca8-4bb8-b62a-30e6abef26c9 tags:
``` python
!pip install geopandas
!pip install backoff
!pip install statsmodels
!pip install seaborn
!pip install scipy
!pip install mapclassify
!pip install unidecode
!pip install rasterio
!pip install seaborn
!pip install mpl_scatter_density
!pip install rasterstats
!pip install xmltodict
!pip install pandas
```
%% Output
Requirement already satisfied: geopandas in /opt/conda/lib/python3.10/site-packages (0.14.1)
Requirement already satisfied: fiona>=1.8.21 in /opt/conda/lib/python3.10/site-packages (from geopandas) (1.9.5)
Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from geopandas) (23.2)
Requirement already satisfied: pandas>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (2.1.3)
Requirement already satisfied: pyproj>=3.3.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (3.6.1)
Requirement already satisfied: shapely>=1.8.0 in /opt/conda/lib/python3.10/site-packages (from geopandas) (2.0.2)
Requirement already satisfied: attrs>=19.2.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (2023.11.17)
Requirement already satisfied: click~=8.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (8.1.7)
Requirement already satisfied: click-plugins>=1.0 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (1.1.1)
Requirement already satisfied: cligj>=0.5 in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (0.7.2)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (1.16.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from fiona>=1.8.21->geopandas) (68.2.2)
Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (1.26.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.4.0->geopandas) (2023.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: backoff in /opt/conda/lib/python3.10/site-packages (2.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: statsmodels in /opt/conda/lib/python3.10/site-packages (0.14.0)
Requirement already satisfied: numpy>=1.18 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (1.26.1)
Requirement already satisfied: scipy!=1.9.2,>=1.4 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (1.11.4)
Requirement already satisfied: pandas>=1.0 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (2.1.3)
Requirement already satisfied: patsy>=0.5.2 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (0.5.3)
Requirement already satisfied: packaging>=21.3 in /opt/conda/lib/python3.10/site-packages (from statsmodels) (23.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.0->statsmodels) (2023.3)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from patsy>=0.5.2->statsmodels) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: seaborn in /opt/conda/lib/python3.10/site-packages (0.13.0)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /opt/conda/lib/python3.10/site-packages (from seaborn) (1.26.1)
Requirement already satisfied: pandas>=1.2 in /opt/conda/lib/python3.10/site-packages (from seaborn) (2.1.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.3 in /opt/conda/lib/python3.10/site-packages (from seaborn) (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.3->seaborn) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: scipy in /opt/conda/lib/python3.10/site-packages (1.11.4)
Requirement already satisfied: numpy<1.28.0,>=1.21.6 in /opt/conda/lib/python3.10/site-packages (from scipy) (1.26.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: mapclassify in /opt/conda/lib/python3.10/site-packages (2.6.1)
Requirement already satisfied: networkx>=2.7 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (3.2.1)
Requirement already satisfied: numpy>=1.23 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.26.1)
Requirement already satisfied: pandas!=1.5.0,>=1.4 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (2.1.3)
Requirement already satisfied: scikit-learn>=1.0 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.3.2)
Requirement already satisfied: scipy>=1.8 in /opt/conda/lib/python3.10/site-packages (from mapclassify) (1.11.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas!=1.5.0,>=1.4->mapclassify) (2023.3)
Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn>=1.0->mapclassify) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn>=1.0->mapclassify) (3.2.0)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas!=1.5.0,>=1.4->mapclassify) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: unidecode in /opt/conda/lib/python3.10/site-packages (1.3.7)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: rasterio in /opt/conda/lib/python3.10/site-packages (1.3.9)
Requirement already satisfied: affine in /opt/conda/lib/python3.10/site-packages (from rasterio) (2.4.0)
Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from rasterio) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from rasterio) (2023.11.17)
Requirement already satisfied: click>=4.0 in /opt/conda/lib/python3.10/site-packages (from rasterio) (8.1.7)
Requirement already satisfied: cligj>=0.5 in /opt/conda/lib/python3.10/site-packages (from rasterio) (0.7.2)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.26.1)
Requirement already satisfied: snuggs>=1.4.1 in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.4.7)
Requirement already satisfied: click-plugins in /opt/conda/lib/python3.10/site-packages (from rasterio) (1.1.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from rasterio) (68.2.2)
Requirement already satisfied: pyparsing>=2.1.6 in /opt/conda/lib/python3.10/site-packages (from snuggs>=1.4.1->rasterio) (3.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: seaborn in /opt/conda/lib/python3.10/site-packages (0.13.0)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /opt/conda/lib/python3.10/site-packages (from seaborn) (1.26.1)
Requirement already satisfied: pandas>=1.2 in /opt/conda/lib/python3.10/site-packages (from seaborn) (2.1.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.3 in /opt/conda/lib/python3.10/site-packages (from seaborn) (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.3->seaborn) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: mpl_scatter_density in /opt/conda/lib/python3.10/site-packages (0.7)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (1.26.1)
Requirement already satisfied: matplotlib>=3.0 in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (3.8.0)
Requirement already satisfied: fast-histogram>=0.3 in /opt/conda/lib/python3.10/site-packages (from mpl_scatter_density) (0.12)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (4.43.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (10.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib>=3.0->mpl_scatter_density) (2.8.2)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib>=3.0->mpl_scatter_density) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: rasterstats in /opt/conda/lib/python3.10/site-packages (0.19.0)
Requirement already satisfied: affine in /opt/conda/lib/python3.10/site-packages (from rasterstats) (2.4.0)
Requirement already satisfied: click>7.1 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (8.1.7)
Requirement already satisfied: cligj>=0.4 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (0.7.2)
Requirement already satisfied: fiona in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.9.5)
Requirement already satisfied: numpy>=1.9 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.26.1)
Requirement already satisfied: rasterio>=1.0 in /opt/conda/lib/python3.10/site-packages (from rasterstats) (1.3.9)
Requirement already satisfied: simplejson in /opt/conda/lib/python3.10/site-packages (from rasterstats) (3.19.2)
Requirement already satisfied: shapely in /opt/conda/lib/python3.10/site-packages (from rasterstats) (2.0.2)
Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (23.1.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (2023.11.17)
Requirement already satisfied: snuggs>=1.4.1 in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (1.4.7)
Requirement already satisfied: click-plugins in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (1.1.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from rasterio>=1.0->rasterstats) (68.2.2)
Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from fiona->rasterstats) (1.16.0)
Requirement already satisfied: pyparsing>=2.1.6 in /opt/conda/lib/python3.10/site-packages (from snuggs>=1.4.1->rasterio>=1.0->rasterstats) (3.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: xmltodict in /opt/conda/lib/python3.10/site-packages (0.13.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (2.1.3)
Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/lib/python3.10/site-packages (from pandas) (1.26.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

%% Cell type:code id:b52092ac-f150-4ea5-ba54-b5279cd24c2e tags:
``` python
import os
import subprocess
import geopandas
import pandas as pd
import json
import datetime
import backoff
import xmltodict
import fileinput
import xml.etree.ElementTree as ET
from urllib.parse import urlparse
import fileinput
import glob
import matplotlib.pyplot as plt
# import numpy as np
import warnings
warnings.filterwarnings("ignore")
warnings.filterwarnings( "ignore", module = "matplotlib\..*" )
import statsmodels.api as sm
import statsmodels.formula.api as smf
import seaborn as sns
import scipy.stats
import scipy.stats as stats
from statsmodels import graphics
import shutil
from mpl_toolkits.axes_grid1 import make_axes_locatable
from unidecode import unidecode
import rasterio
from rasterio.plot import show
import seaborn
# import mpl_scatter_density # adds projection='scatter_density'
from matplotlib.colors import LinearSegmentedColormap
import matplotlib.colors
from textwrap import wrap
import matplotlib.patches as mpatches
import pylab as py
import rasterstats
from rasterstats import zonal_stats
import scipy
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
import pandas as pd
import subprocess
import glob
import shutil
from maap.maap import MAAP
# maap=MAAP(maap_host="api.ops.maap-project.org")
maap=MAAP()
def BUILD_TABLE_JOBSTATUS(submit_results_df, status_col = 'status'):
import xmltodict
# If jobs failed to submit, then they have a NaN for jobid, which makes the merge (join) fail
submit_results_df = submit_results_df.fillna('')
job_status_df = pd.concat([pd.DataFrame({'job_id': [job_id], 'status':[maap.getJobStatus(job_id)]}) for job_id in submit_results_df.job_id.to_list()])
job_status_df = submit_results_df.merge(job_status_df, how='left', left_on='job_id', right_on='job_id')
print(f'Count total jobs:\t{len(job_status_df)}')
print(f"Count pending jobs:\t{job_status_df[job_status_df[status_col] =='Accepted'].shape[0]}")
print(f"Count running jobs:\t{job_status_df[job_status_df[status_col] =='Running'].shape[0]}")
NUM_FAILS = job_status_df[job_status_df[status_col] =='Failed'].shape[0]
NUM_SUCCEEDS = job_status_df[job_status_df[status_col] =='Succeeded'].shape[0]
print(f"Count succeeded jobs:\t{NUM_SUCCEEDS}")
print(f"Count failed jobs:\t{NUM_FAILS}")
if NUM_FAILS > 0:
print(f"% of failed jobs:\t{round(NUM_FAILS / ( NUM_FAILS + NUM_SUCCEEDS ), 4) * 100}\n")
else:
print(f"% of failed jobs:\tNothing has failed...yet\n")
return job_status_df
```
%% Cell type:code id:3a2b480f-a23e-49fa-b26b-392f27bd9cef tags:
``` python
# cd /projects/ADE_biomass_harmonization/biomass_harmonization/
# git add country_summaries/IPCC_classes_DPS/Boreal
# git commit -m 'Boreal - Classifying the world into IPCC Table 4.7'
# git tag -f Boreal_IPCC_classes_ADE
# git push origin -f Boreal_IPCC_classes_ADE
# git push
# git push origin -f Boreal_IPCC_classes_ADE
########### REGISTER ALGORITHM ON NEW ADE ##############
# maap.register_algorithm_from_yaml_file("/projects/ADE_biomass_harmonization/biomass_harmonization/country_summaries/IPCC_classes_DPS/Boreal/boreal_run_IPCC_classes_ADE.yml").text
```
%% Cell type:code id:63d7cc00-e8bd-47c6-8394-db3c8ef6bfec tags:
``` python
Desired_output_folder = '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/BOREAL_compiled'
if not os.path.exists(Desired_output_folder):
os.mkdir(Desired_output_folder)
###############################################################################################################################
############ SUPPLY A GEOPACKAGE FILE THAT HOLDS THE UNIQUE ID OF THE VECTORS #################################################
# THIS VECTOR IS SPLIT BY THE GEDI - ICESat-2 BORDER AND RENAMED
# ONE FILE IS USED FOR PROCESSING ON MAAP, THE OTHER GIVEN TO THE GEDI L4 TEAM TO GENERATE HYBRID ESTIMATES ###################
###############################################################################################################################
# CUTLINE = geopandas.read_file('/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/GEDI_ICESat_CUTLINE.gpkg')
# VECTOR = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/NE_countries_HNL_DISS_v2.gpkg' #MUST BE A GPKG
# VECTOR = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/Global_IPCC_classes_ADE_v6_GPKG/IPCC_classes_globe_v6_DISS.gpkg'
# VECTOR_EDIT = geopandas.read_file(VECTOR)
# VECTOR_BOREAL = geopandas.clip(VECTOR_EDIT, CUTLINE)
# VECTOR_GEDI = geopandas.overlay(VECTOR_EDIT, CUTLINE, how='difference')
# VECTOR_BOREAL.to_file("/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_BOREAL.gpkg", driver="GPKG")
# VECTOR_GEDI.to_file("/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_GEDI.gpkg", driver="GPKG")
# VECTOR_GEDI.plot()
# VECTOR_BOREAL.plot()
VECTOR_BOREAL = "/projects/my-public-bucket/Data/Harris_et_al_PAPER/GEDI_ICESAT_cutline/VECTOR_BOREAL.gpkg"
ADM = '/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg'
if not os.path.exists(ADM): shutil.copy(VECTOR_BOREAL,ADM)
TILES = ["CHN","FRA","PRK","UKR","BLR","UZB","KAZ","TJK","LTU","MNG","DEU","EST","LVA","NOR","SWE","FIN","LUX","BEL","KGZ","DNK","ROU","HUN","SVK","POL","IRL","GBR","NLD","MDA","USA","GRL","JPN","ISL"]
TILES = ["4232","4332","4132","4333","4133","4135","4233","4235","4335","2135","2235","2335","2233","2333","2133","7131","7135","7231","7235","7331","7335","7132","7232","7332","7133","7233","7333","2141","2241","2341","4143","4343","4243","4141","4241","4341","7141","7143","7241","7243","7341","7343","2142","2242","2342","2143","2243","2343","7142","7242","7342","4142","4242","4342"]
TILES = ["4232","4332","4132","4333","4133","4135","4233","4235","2135","2235","2335","2233","2333","2133","7131","7135","7231","7235","7331","7335","7232","7133","7233","7333","2141","2241","2341","4343","4243","4141","4241","4341","7141","7143","7241","7243","7341","7343","2142","2342","2143","2243","2343","7142","7242","7342","4142","4242","4342"]
i = 0
count = -1
submit_results_df_list = []
len_input_list = len(TILES)
list_jobIDs = [""]*len(TILES)
for TILE in TILES:
if not os.path.exists(os.path.join(Desired_output_folder,TILE + "_ISB_AGB_1KM_C.tif")):
print("submitting " + TILE)
count +=1
submit_result = maap.submitJob(identifier="Boreal_IPCC_classes_ADE",
algo_id="run_Boreal_IPCC_classes_ADE",
version="Boreal_IPCC_classes_ADE",
username="nehajo88",
queue="maap-dps-worker-32vcpu-64gb",
queue="maap-dps-worker-16gb", #32vcpu-64
IPCC_class=TILE,
ISB_AGB_1KM= TILE + "_ISB_AGB_1KM_C.tif",
ISB_VAR_1KM= TILE + "_ISB_VAR_1KM_C.tif",
AOI_stats_table = TILE + "_STATS.csv",
boreal_tiles="https://maap-ops-workspace.s3.amazonaws.com/shared/leitoldv/boreal_tiles_v003.gpkg",
adm="https://maap-ops-workspace.s3.amazonaws.com/shared/nehajo88/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg",
AGB_tindex="https://maap-ops-workspace.s3.amazonaws.com/shared/nehajo88/Data/ICESat-2-TILES/AGB_tindex_master.csv")
job_id = str(submit_result).split(': ')[1].split(',')[0].replace("'","")
print(job_id)
```
%% Output
submitting 4232
bbc92b0f-1e15-4dd7-9b3a-5c8ad2d410fe
32abde50-b026-43b5-bf88-44c3b172a5d3
submitting 4332
7d384d12-3d86-4326-9f9c-698976da0c38
a44380f3-6d98-4e29-8183-46f6f493af09
submitting 4132
28c00a24-bda6-4353-adf5-c36df6daf423
1276c39e-703b-42ff-91f3-e47f3a4142ed
submitting 4333
b064a873-f13e-4695-b0f6-74db53824ad4
de05e00e-9957-413e-b2b6-4ea6f293a1c7
submitting 4133
142cdfee-a8b3-461c-84d9-7b154f7e72dd
095a044a-cdf0-46d8-9f94-1aa9026953db
submitting 4135
811bdea8-02c4-4f15-9bb2-010c0681dc8c
3e6eec58-10d0-4fd4-b37e-cd93f49a0997
submitting 4233
f08eb498-5d2f-41e2-ba50-981564202c77
29560900-9de3-463f-ac7c-32b7053e5511
submitting 4235
943c9a8d-f4be-414e-b3bd-a0f09d2ecb49
bc98436b-1254-46e1-9195-26920ae6e92e
submitting 2133
2d8b4efe-7a45-425c-bd8c-946a71412133
35f59bb8-a891-46ee-9f96-c55d551af33f
submitting 7235
be6ce556-0275-4d55-bc40-2fdd8d7b84cc
ddf9e774-8438-4384-977d-89dcebf0bda9
submitting 7335
6561d5e5-a612-4a71-a9bb-b71d99a67237
c959d17b-bb5d-46cb-afd0-02629fca8800
submitting 7232
bddcff95-ef73-4532-b4e7-a0abf1b19a5a
25fcb409-58dc-42ef-a965-d10fcbb9f006
submitting 2141
683b7564-5fad-4a13-a72e-70644a0de8aa
61499601-c880-4951-9c30-0e1c1e25d9bd
submitting 2241
3cda8bb2-8576-4f08-8127-b01de8cf8aae
6b0a9822-87eb-48e8-ab51-03ea4d1e19df
submitting 2341
4eea52a2-62a1-40cd-92dc-cb42aac11a21
164fe631-e098-4c79-9c27-cfbcaecf9f8d
submitting 4343
978c30a6-5b5b-445a-9144-d04f438785f9
7b69b651-4b09-4eb8-8ad3-e883389ed709
submitting 4243
cfed2f03-0734-47c6-8666-7e5a09d27132
f661fa41-4de4-4f72-baee-adebe1b172a3
submitting 4141
1fd5ef30-44ed-44b3-8d7e-3e9da865c10a
63a7442f-a0ef-46d7-b25d-83bcee096b7f
submitting 4241
1da2e1d0-6a51-4fd0-8e66-fdfb8a48798b
8233dc57-f9a3-4ca8-b3d7-b04a8b3f035b
submitting 4341
e86ce4b3-f324-4c8f-871d-31ffd37f9dd3
c6491ce4-1b1e-4791-bac6-7e2264cb0953
submitting 7141
2ac99e2a-fdc3-4395-af19-ef06120762dc
6413abac-5159-473d-824f-c6cac37b3664
submitting 7143
86cd1f72-5469-45c5-b3bf-72e1c8aa2819
ceb63089-b756-42e6-a068-002e35760221
submitting 7241
e5b333a5-4694-4bd7-aebe-62895c43d5be
f68ce7ed-1390-416a-9cce-a4a8ac43193b
submitting 7243
118c0861-f23f-4e6f-bd30-e050226b4279
9f7ea488-b643-455c-baec-ac6997a91819
submitting 7341
217586ad-8e84-42e6-a471-ef9c23f92b8b
b0b57fbc-aeb8-40a5-b332-b7146eea1542
submitting 7343
1e99853e-e448-4767-aefd-d527012fb8e5
f8b96ef1-5c30-475f-816d-9697a3a7c37d
submitting 2142
ddafa311-1b81-408c-9b95-98eba25f689b
506c505c-5a29-4278-98ab-8025a5131d27
submitting 2342
e18c40ee-5ce2-4806-a44b-d10abfbce588
8827cfe4-6278-4a89-ac0e-52d531cd1fd8
submitting 2143
9b0d99dc-e9fd-4885-ad97-22301c9461f4
3d437654-08b8-4040-8dd2-f9e325b77c6c
submitting 2243
fb8b553f-eb14-4310-a686-b352b2159f09
97009f4f-2f41-492c-817e-a8e88db7a9ee
submitting 2343
5368fea9-b9ae-414b-8fec-e2311dba415f
9cea5a99-f03c-4864-b5b3-09021058f02f
submitting 4142
5c54bb30-161a-41b7-8192-45135a5c265c
9546865b-1a6a-456b-a0e6-acc6d46d9f7f
submitting 4242
a95c7cb6-f0df-46d6-8b47-12d99b5f51ab
2f8fb61e-12a2-4a69-848e-f24ce4eef904
submitting 4342
6b5635fd-b867-4c4a-816d-ccff287e6d22
ccaa3459-ae0d-4693-a576-e026c2a7625f
%% Cell type:code id:d7007b5e-0b3f-440d-b8cf-02eb330a5fef tags:
``` python
maap.getJobResult('4c427afd-2ef6-4258-aac2-5672c66aaa7b')
```
%% Output
['http://maap-ops-workspace.s3-website-us-west-2.amazonaws.com/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b',
's3://s3-us-west-2.amazonaws.com:80/maap-ops-workspace/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b',
'https://s3.console.aws.amazon.com/s3/buckets/maap-ops-workspace/dataset/triaged_job/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/10/10/4c427afd-2ef6-4258-aac2-5672c66aaa7b/?region=us-east-1&tab=overview',
'++ python /app/get_username.py\n+ USERNAME=nehajo88\n+ DPS_MACHINE_TOKEN=Y6hERQKLKs0oH0h5Kgi3d-gVVyHgW_DR77YeARdzd-a3KJDE-gYldJzhB94s635KySUXi4r9WOtD6Vo_F9cEbfNI_QUhOG8N47heXHVcW1k3vEw3d0StttXYMOnQPtTzSQLAYKQwJLgbtX-R8ug1XGVeMolh97P9nHcW8awN7yjyaR7g\n++ curl -sb -H \'Accept: application/json\' -H \'Content-Type: application/json\' -H \'dps-token: Y6hERQKLKs0oH0h5Kgi3d-gVVyHgW_DR77YeARdzd-a3KJDE-gYldJzhB94s635KySUXi4r9WOtD6Vo_F9cEbfNI_QUhOG8N47heXHVcW1k3vEw3d0StttXYMOnQPtTzSQLAYKQwJLgbtX-R8ug1XGVeMolh97P9nHcW8awN7yjyaR7g\' https://api.maap-project.org/api/members/nehajo88\n++ jq -r .session_key\n+ export MAAP_PGT=PGT-12-JV4ZKLoVctV2kSSpfHn6OOXIossEXUR7CbydSW4SR8TNr7Mv9KRTov2nz-jslvcFnio-8445019b059c\n+ MAAP_PGT=PGT-12-JV4ZKLoVctV2kSSpfHn6OOXIossEXUR7CbydSW4SR8TNr7Mv9KRTov2nz-jslvcFnio-8445019b059c\n+ unset DPS_MACHINE_TOKEN\n+ /app/biomass_harmonization/country_summaries/IPCC_classes_DPS/Boreal/boreal_run_IPCC_classes_ADE.sh POL POL_ISB_AGB_1KM_C.tif POL_ISB_VAR_1KM_C.tif POL_STATS.csv\nLoading required package: raster\nLoading required package: sp\nThe legacy packages maptools, rgdal, and rgeos, underpinning the sp package,\nwhich was just loaded, will retire in October 2023.\nPlease refer to R-spatial evolution reports for details, especially\nhttps://r-spatial.org/r/2023/05/15/evolution4.html.\nIt may be desirable to make the sf package available;\npackage maintainers should consider adding sf to Suggests:.\nThe sp package is now running under evolution status 2\n (status 2 uses the sf package in place of rgdal)\nLoading required package: rgdal\nPlease note that rgdal will be retired by the end of 2023,\nplan transition to sf/stars/terra functions using GDAL and PROJ\nat your earliest convenience.\n\nrgdal: version: 1.5-32, (SVN revision 1176)\nGeospatial Data Abstraction Library extensions to R successfully loaded\nLoaded GDAL runtime: GDAL 3.6.2, released 2023/01/02\nPath to GDAL shared files: \n GDAL does not use iconv for recoding strings.\nGDAL binary built with GEOS: TRUE \nLoaded PROJ runtime: Rel. 9.1.0, September 1st, 2022, [PJ_VERSION: 910]\nPath to PROJ shared files: /opt/conda/share/proj\nPROJ CDN enabled: FALSE\nLinking to sp version:1.5-1\nTo mute warnings of possible GDAL/OSR exportToProj4() degradation,\nuse options("rgdal_show_exportToProj4_warnings"="none") before loading sp or rgdal.\nLoading required package: terra\nterra 1.7.29\n\nAttaching package: ‘terra’\n\nThe following object is masked from ‘package:rgdal’:\n\n project\n\nLoading required package: sf\nLinking to GEOS 3.11.1, GDAL 3.6.2, PROJ 9.1.0; sf_use_s2() is TRUE\nLoading required package: RCurl\nError in library(x, character.only = TRUE, quietly = TRUE) : \n there is no package called ‘RCurl’\nCalls: lapply -> FUN -> library\nIn addition: Warning message:\nIn library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :\n there is no package called ‘RCurl’\nExecution halted\n+ cp _stderr.txt _alt_traceback.txt\n']
%% Cell type:code id:a91aad0d-86d8-421f-80a7-5951d3f5b239 tags:
``` python
Desired_output_folder = '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/BOREAL_compiled'
if not os.path.exists(Desired_output_folder):
os.mkdir(Desired_output_folder)
SUMMARY_files = glob.glob('/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/**/*.tif', recursive=True)
print("Moving files...")
print(SUMMARY_files)
for each_file in SUMMARY_files:
if os.path.exists(each_file) and not os.path.exists(os.path.join(Desired_output_folder,os.path.basename(each_file))):
# os.remove(os.path.join(Desired_output_folder,os.path.basename(each_file)))
shutil.move(each_file, Desired_output_folder)
SUMMARY_files = glob.glob('/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/**/*.csv', recursive=True)
print("Moving files...")
print(SUMMARY_files)
for each_file in SUMMARY_files:
if os.path.exists(each_file) and not os.path.exists(os.path.join(Desired_output_folder,os.path.basename(each_file))):
# os.remove(os.path.join(Desired_output_folder,os.path.basename(each_file)))
shutil.move(each_file, Desired_output_folder)
```
%% Output
Moving files...
[]
['/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_ISB_AGB_1KM_C.tif', '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_ISB_VAR_1KM_C.tif']
Moving files...
[]
['/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/29/07/20/03/056878/7242_STATS.csv', '/projects/my-private-bucket/dps_output/run_Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/Boreal_IPCC_classes_ADE/2023/11/30/04/44/03/257748/4332_STATS.csv']
%% Cell type:code id:visible-tampa tags:
``` python
##############################################################################
### FIRST, TO INSTALL PACKAGES, PLEASE OPEN A TERMINAL AND RUN THE FOLLOWING LINE:
### bash /projects/my-public-bucket/_ENV/env/build_r_env.sh
##############################################################################
packages <- c("raster", "rgdal", "terra", "sf", "sp")
package.check <- lapply(packages, FUN = function(x) {
if (!require(x, character.only = TRUE)) {
# install.packages(x, dependencies = TRUE)
library(x, character.only = TRUE, quietly=TRUE)
library(geojsonio)
}
})
```
%% Output
Loading required package: raster
Loading required package: sp
The legacy packages maptools, rgdal, and rgeos, underpinning the sp package,
which was just loaded, were retired in October 2023.
Please refer to R-spatial evolution reports for details, especially
https://r-spatial.org/r/2023/05/15/evolution4.html.
It may be desirable to make the sf package available;
package maintainers should consider adding sf to Suggests:.
Loading required package: rgdal
Please note that rgdal will be retired during 2023,
plan transition to sf/stars/terra functions using GDAL and PROJ
at your earliest convenience.
See https://r-spatial.org/r/2022/04/12/evolution.html and https://github.com/r-spatial/evolution
rgdal: version: 1.6-6, (SVN revision 1201)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.6.2, released 2023/01/02
Path to GDAL shared files: /opt/conda/share/gdal
GDAL does not use iconv for recoding strings.
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 9.1.0, September 1st, 2022, [PJ_VERSION: 910]
Path to PROJ shared files: /projects/.local/share/proj:/opt/conda/share/proj
PROJ CDN enabled: FALSE
Linking to sp version:1.6-0
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading sp or rgdal.
Loading required package: terra
terra 1.7.29
Attaching package: ‘terra’
The following object is masked from ‘package:rgdal’:
project
Loading required package: sf
Linking to GEOS 3.11.1, GDAL 3.6.2, PROJ 9.1.0; sf_use_s2() is TRUE
%% Cell type:code id:9fcec9f5-be89-45bc-a90b-c4ba23ec183d tags:
``` python
urlExists <- function(address) {
tryCatch( {
con <- url(address)
a <- capture.output(suppressWarnings(readLines(con)))
close(con)
TRUE;
},
error = function(err) {
occur <- grep("cannot open the connection", capture.output(err));
if(length(occur) > 0) FALSE;
} )
}
urlExists("https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/boreal_agb_202302061675666484_26022.tif")
```
%% Output
FALSE
FALSE
%% Cell type:code id:protecting-dynamics tags:
``` python
Out_folder <- "/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results"
if (!dir.exists(Out_folder)){dir.create(Out_folder)}
```
%% Cell type:code id:significant-interaction tags:
``` python
## read in boreal tiles grid (save projection information for later)
boreal_tiles <- readOGR('/projects/shared-buckets/leitoldv/boreal_tiles_v003.gpkg',verbose=F)
boreal_proj4 <- crs(boreal_tiles)
```
%% Output
Warning message in OGRSpatialRef(dsn, layer, morphFromESRI = morphFromESRI, dumpSRS = dumpSRS, :
“Discarded datum unknown in Proj4 definition: +proj=aea +lat_0=40 +lon_0=180 +lat_1=50 +lat_2=70 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs,
but +towgs84= values preserved”
%% Cell type:code id:lesbian-driver tags:
``` python
IPCC_class <- "DEU"
adm <- readOGR("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/IPCC_classes_BOREAL_DISS_TEST.gpkg",verbose=F)
adm <- adm[adm$IPCC_classes_1km == IPCC_class,]
adm_poly <- polygons(adm)
adm_proj <- spTransform(adm_poly, crs(boreal_tiles))
if (!dir.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))){dir.create(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))}
```
%% Cell type:code id:80589b3c-f1e5-485f-b2bf-a26bc270ec38 tags:
``` python
intersecting <- st_intersection(st_as_sf(boreal_tiles), st_convex_hull(st_as_sf(adm_proj)), byid=TRUE)
if (nrow(intersecting) > 0) {
ICESat_tiles <- intersecting
} else {stop('no intersecting CCI Biomass tiles found')}
plot(adm_proj)
plot(ICESat_tiles, add=T, border="blue")
#ICESat_file <- file.path(Out_folder,paste0(IPCC_class,'_icesat_json.shp'))
#if (!file.exists(ICESat_file)) {writeOGR(obj=ICESat_tiles, dsn=Out_folder, layer=paste0(IPCC_class,'_icesat_json'), driver="ESRI Shapefile")}
```
%% Output
Warning message:
“attribute variables are assumed to be spatially constant throughout all geometries”
Warning message in plot.sf(ICESat_tiles, add = T, border = "blue"):
“ignoring all but the first attribute”
%% Cell type:code id:9d62c649-6ee8-49f8-987a-28d94aaf5451 tags:
``` python
tilelist = as.list(ICESat_tiles$tile_num)
summary(boreal_tiles)
```
%% Output
%% Cell type:code id:united-celebration tags:
``` python
## create tileindex with ICESat2 file paths to selected tiles
AGB_tindex <- read.csv("/projects/shared-buckets/lduncanson/DPS_tile_lists/2023/AGB_tindex_master.csv")
AGB_tindex_out <- c()
for (i in 1:length(tilelist)){
tile_num <- tilelist[i]
print(tile_num)
AGB_tindex_out <- rbind(AGB_tindex_out, AGB_tindex[AGB_tindex[,"tile_num"]==tile_num,])
}
```
%% Output
[[1]]
[1] "23776"
[[1]]
[1] "23777"
[[1]]
[1] "23778"
[[1]]
[1] "24055"
[[1]]
[1] "24056"
[[1]]
[1] "24057"
[[1]]
[1] "24058"
[[1]]
[1] "24059"
[[1]]
[1] "24335"
[[1]]
[1] "24336"
[[1]]
[1] "24337"
[[1]]
[1] "24338"
[[1]]
[1] "24339"
[[1]]
[1] "24340"
[[1]]
[1] "24341"
[[1]]
[1] "24615"
[[1]]
[1] "24616"
[[1]]
[1] "24617"
[[1]]
[1] "24618"
[[1]]
[1] "24619"
[[1]]
[1] "24620"
[[1]]
[1] "24621"
[[1]]
[1] "24622"
[[1]]
[1] "24896"
[[1]]
[1] "24897"
[[1]]
[1] "24898"
[[1]]
[1] "24899"
[[1]]
[1] "24900"
[[1]]
[1] "24901"
[[1]]
[1] "24902"
[[1]]
[1] "24903"
[[1]]
[1] "25177"
[[1]]
[1] "25178"
[[1]]
[1] "25179"
[[1]]
[1] "25180"
[[1]]
[1] "25181"
[[1]]
[1] "25182"
[[1]]
[1] "25183"
[[1]]
[1] "25184"
[[1]]
[1] "25459"
[[1]]
[1] "25460"
[[1]]
[1] "25461"
[[1]]
[1] "25462"
[[1]]
[1] "25463"
[[1]]
[1] "25464"
[[1]]
[1] "25465"
[[1]]
[1] "25741"
[[1]]
[1] "25742"
[[1]]
[1] "25743"
[[1]]
[1] "25744"
[[1]]
[1] "25745"
[[1]]
[1] "25746"
[[1]]
[1] "26020"
[[1]]
[1] "26021"
[[1]]
[1] "26022"
%% Cell type:code id:d05ddc0d-d864-451a-854a-62fa29af14a1 tags:
``` python
##### DOWNLOAD NECESSARY FILES FROM NASA ORNL DAAC
PATHS <- AGB_tindex_out$s3_path
ORNL_username <- 'nhunka'
ORNL_password <- 'Boreal_tiles1'
for (i in 1:length(PATHS)){
DOWNLOAD_LOCATION <- paste0("/projects/my-public-bucket/Data/ICESat-2-boreal/",basename(PATHS[i]))
if (!file.exists(DOWNLOAD_LOCATION)){
PATH_i = paste("wget --user",ORNL_username, "--password ",ORNL_password, paste0("https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/",basename(PATHS[i])), "-O", DOWNLOAD_LOCATION)
print(paste("Downloading...",DOWNLOAD_LOCATION))
system(PATH_i)
}
}
###### NOW ALL THE NECCESARY FILES SHOULD HAVE DOWNLOADED INTO MY FOLDER "/projects/my-public-bucket/Data/ICESat-2-boreal/"
```
%% Cell type:code id:dc70fe87-a735-43e6-9b51-1943fb432143 tags:
``` python
PATH_i
```
%% Output
<span style=white-space:pre-wrap>'wget --user nhunka --password Boreal_tiles1 https://daac.ornl.gov/daacdata/above/Boreal_AGB_Density_ICESat2/data/boreal_agb_202302061675666484_26022.tif -O /projects/my-public-bucket/Data/ICESat-2-boreal/boreal_agb_202302061675666484_26022.tif'</span>
'wget --user nhunka --password Boreal\_tiles1 https://daac.ornl.gov/daacdata/above/Boreal\_AGB\_Density\_ICESat2/data/boreal\_agb\_202302061675666484\_26022.tif -O /projects/my-public-bucket/Data/ICESat-2-boreal/boreal\_agb\_202302061675666484\_26022.tif'
%% Cell type:code id:51079f67-9cea-4717-bc4a-fafacf3d3f9e tags:
``` python
if (!dir.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))){dir.create(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')))}
for (i in 1:length(PATHS)){
DOWNLOAD_LOCATION <- paste0("/projects/my-public-bucket/Data/ICESat-2-boreal/",basename(PATHS[i]))
if (file.exists(DOWNLOAD_LOCATION) & (file.info(DOWNLOAD_LOCATION)$size > 0)){
ISB_tile_resampled <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_AGBD_1km.tif")
ISB_tile_SDsquared <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_SDsquared.tif")
ISB_tile_AGBrooted <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_AGBrooted.tif")
ISB_tile_VAR30m <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_VAR30m.tif")
ISB_tile_Variance <- paste0("IPCC_Class_",IPCC_class,"_",strsplit(basename(DOWNLOAD_LOCATION), ".tif")[1],"_VAR_1km.tif")
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance))){
print(paste("creating ",ISB_tile_Variance))
##### FIRST - SQUARE THE RASTER - THIS CONVERTS ALL ND VALUES TO 0 AND ESTIMATES THE VARIANCE AT 30 M PIXEL SIZE
r <- rast(DOWNLOAD_LOCATION)
s <- r^2
writeRaster(s, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
##### SECOND - SQUARE ROOT AND WRITE THE AGBD RASTER
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))){
v <- rast(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))
m <- sqrt(v)
writeRaster(m[[1]], file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
}
##### THIRD - WRITE THE VAR RASTER SEPARATELY
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))){
v <- rast(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_SDsquared))
writeRaster(v[[2]], file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m), overwrite=TRUE, gdal=c("COMPRESS=NONE", "TFW=YES"), datatype='FLT4S')
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))}
}
##### THIRD - RESAMPLE THE AGBD RASTER TO 1 KM
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted))){
# gdalwarp(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted),file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_resampled),b=1,ot="Float32", co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=FALSE, tr=c(1000,1000),r='average')
gdal_utils(util = "warp", source = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_AGBrooted),destination=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_resampled),options=c("-ot","Float32","-co","c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)", "-tr",1000,1000,"-r","average"))
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))}
}
##### FOURTH - RESAMPLE THE VARIANCE RASTER TO 1 KM
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m))){
# gdalwarp(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m),file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance),b=1,ot="Float32", co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=FALSE, tr=c(1000,1000),r='average')
gdal_utils(util = "warp", source = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_VAR30m),destination=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),ISB_tile_Variance),options=c("-ot","Float32","-co","c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)", "-tr",1000,1000,"-r","average"))
}
#### FINAL - REMOVE INTERMEDIATE FILES
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_VAR30m)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_VAR30m)))}
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_AGBrooted)))}
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),basename(ISB_tile_SDsquared)))}
}
}
}
#-------------------------------------------------------------------------------
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM_C.tif"))){
tiles <- list.files(path = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')),pattern="*_AGBD_1km.tif",full.names=TRUE)
# mosaic_rasters(gdalfile=tiles,dst_dataset=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"),of="GTiff", gdalwarp_params = list(r = "average",ot="Float32"), co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=TRUE)
mosaic(sprc(tiles), fun="mean", filename=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif")), adm_proj)
tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif")), adm_proj, inverse=FALSE, maskvalues=NA, updatevalue=NA)
writeRaster(tile_stack, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM_C.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_AGB_1KM.tif"))}
}
if (!file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM_C.tif"))){
tiles <- list.files(path = file.path(Out_folder,paste0(IPCC_class,'_Product_ISB')),pattern="*_VAR_1km.tif",full.names=TRUE)
# mosaic_rasters(gdalfile=tiles,dst_dataset=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"),of="GTiff", gdalwarp_params = list(r = "average",ot="Float32"), co=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9"), overwrite=TRUE, VERBOSE=TRUE)
mosaic(sprc(tiles), fun="mean", filename=file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"), wopt= list(gdal=c("COMPRESS=DEFLATE","PREDICTOR=2","ZLEVEL=9")), overwrite=TRUE)
# tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif")), adm_proj)
tile_stack <- mask(raster(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif")), adm_proj, inverse=FALSE, maskvalues=NA, updatevalue=NA)
writeRaster(tile_stack, file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM_C.tif"), overwrite=TRUE)
if (file.exists(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"))) {file.remove(file.path(Out_folder,paste0(IPCC_class,'_Product_ISB'),"ISB_VAR_1KM.tif"))}
}
```
%% Output
[1] "creating IPCC_Class_GBR_boreal_agb_202302061675665310_24951_VAR_1km.tif"
Warning message in new_CppObject_xp(fields$.module, fields$.pointer, ...):
“GDAL Message 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in x@ptr$writeRaster(opt):
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Message 1: All options related to creation ignored in update mode”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Message 6: driver GTiff does not support creation option c(COMPRESS”
Warning message in CPL_gdalwarp(source, destination, options, oo, doo, quiet, "-overwrite" %in% :
“GDAL Error 1: PROJ: proj_create_from_name: Open of /opt/conda/share/proj failed”
Error in gdal_utils(util = "warp", source = file.path(Out_folder, paste0(IPCC_class, : gdal_utils warp: an error occured
Traceback:
1. gdal_utils(util = "warp", source = file.path(Out_folder, paste0(IPCC_class,
. "_Product_ISB"), ISB_tile_VAR30m), destination = file.path(Out_folder,
. paste0(IPCC_class, "_Product_ISB"), ISB_tile_Variance), options = c("-ot",
. "Float32", "-co", "c(COMPRESS=DEFLATE,PREDICTOR=2,ZLEVEL=9)",
. "-tr", 1000, 1000, "-r", "average"))
2. stop(paste0("gdal_utils ", util, ": an error occured"))
%% Cell type:code id:84b19785-1ae3-4685-afb9-7b1333f93951 tags:
``` python
(length(sprc(tiles)) > 1)
```
%% Output
TRUE
TRUE
%% Cell type:code id:5d53c3fd-bfc1-43c9-a309-a0b340f435fc tags:
``` python
AOI_in_agbd_1km <- raster("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results/2101_Product_ISB/ISB_AGB_1KM_C.tif")
AOI_in_var_1km <- raster("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_results/2101_Product_ISB/ISB_VAR_1KM_C.tif")
```
%% Cell type:code id:7951d8bc-d725-4321-8c0d-26f4d0d51742 tags:
``` python
# par(mfrow=c(1,2))
# plot(AOI_in_agbd_1km, main="agbd")
# plot(AOI_in_var_1km, main="var")
```
%% Output
%% Cell type:code id:f386823e-dfff-48e8-9dde-c1e3c14c1bf8 tags:
``` python
total_convert_Mg <- function(x){(x*100)}
```
%% Cell type:code id:f678956d-8367-4c5f-9d97-47e7c17e7e5e tags:
``` python
## convert AGBD (Mg/ha) to AGB (Mg) for 1km resolution AOIs:
AOI_in_AGB_1km <- total_convert_Mg(AOI_in_agbd_1km)
## extract raster values from given AOI
AOI_rastervalues_AGB <- getValues(AOI_in_AGB_1km)[!is.na(getValues(AOI_in_AGB_1km))]
## calculate summary stats for AOI
AOI_stats <- vector(mode = "list", length = 5)
names(AOI_stats) <- c("AGB_total", "AGB_sd", "AGB_mean", "AGB_se", "AREA")
AOI_stats$AGB_total <- sum(AOI_rastervalues_AGB) ##units: Mg
AOI_stats$AGB_sd <- total_convert_Mg(sqrt(sum(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))]))) ##units: Mg
AOI_stats$AREA <- length(AOI_rastervalues_AGB)*100 ##units: ha
AOI_stats$AGB_mean <- AOI_stats$AGB_total / AOI_stats$AREA ##units: Mg/ha
AOI_stats$AGB_se <- sqrt(sum(AOI_rastervalues_VAR))/sqrt(length(AOI_rastervalues_VAR)-1)
AOI_stats$AGB_mean <- mean(getValues(AOI_in_agbd_1km)[!is.na(getValues(AOI_in_agbd_1km))])
AOI_stats$AGB_se <- sqrt(sum(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))]))/sqrt(length(getValues(AOI_in_var_1km)[!is.na(getValues(AOI_in_var_1km))])-1)
AOI_stats_table <- matrix(0, nrow=5, ncol=2)
row.names(AOI_stats_table) <- c("AREA", "AGB_total", "AGB_sd", "AGB_mean", "AGB_se")
colnames(AOI_stats_table) <- c("value", "unit")
AOI_stats_table["AREA", "value"] <- round(AOI_stats$AREA, digits=3) / 10^6
AOI_stats_table["AGB_total", "value"] <- round(AOI_stats$AGB_total, digits=3) / 10^9
AOI_stats_table["AGB_sd", "value"] <- round(AOI_stats$AGB_sd, digits=3) / 10^9
AOI_stats_table["AGB_mean", "value"] <- round(AOI_stats$AGB_mean, digits=3)
AOI_stats_table["AGB_se", "value"] <- round(AOI_stats$AGB_se, digits=3)
AOI_stats_table[,"unit"] <- c("Mha", "Pg", "Pg", "Mg/ha", "Mg")
write.csv(AOI_stats_table, file=paste("/projects/my-public-bucket/Data/Harris_et_al_PAPER/IPCC_Classes/Boreal_IPCC_stats",IPCC_class,"_agb_stats_v2023SEP28.csv",sep=""))
```
%% Output
$AGB_total
: 3010633.90445709
$AGB_sd
: 22465.6677379406
$AGB_mean
: 29.1445682909689
$AGB_se
: 6.99325690523318
$AREA
: 103300
\begin{description}
\item[\$AGB\_total] 3010633.90445709
\item[\$AGB\_sd] 22465.6677379406
\item[\$AGB\_mean] 29.1445682909689
\item[\$AGB\_se] 6.99325690523318
\item[\$AREA] 103300
\end{description}
......
country_summaries/IPCC_classes_DPS/IPCC_GEDI_Table4.7.ipynb
+ 4
− 0
View file @ 338ec421
......@@ -5,6 +5,10 @@
"execution_count": 1,
"id": "champion-language",
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
country_summaries/IPCC_classes_DPS/IPCC_raster_post_processing.ipynb
+ 1220
− 324
View file @ 338ec421
source diff could not be displayed: it is too large. Options to address this: view the blob.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment