Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • david.m.giles/aos_maap_dps
  • sshah/aos_maap_dps
  • dschuck/aos_maap_dps
3 results
Show changes
View open merge request
Commits on Source (16)
Hide whitespace changes
Inline Side-by-side
.gitignore 0 → 100644
Edit View file @ ac0fa6ab
# Created by https://www.toptal.com/developers/gitignore/api/python
# Edit at https://www.toptal.com/developers/gitignore?templates=python
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml
# ruff
.ruff_cache/
# LSP config files
pyrightconfig.json
# End of https://www.toptal.com/developers/gitignore/api/python
# Created by https://www.toptal.com/developers/gitignore/api/macos
# Edit at https://www.toptal.com/developers/gitignore?templates=macos
### macOS ###
# General
.DS_Store
.AppleDouble
.LSOverride
# Icon must end with two \r
Icon
# Thumbnails
._*
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
### macOS Patch ###
# iCloud generated files
*.icloud
# End of https://www.toptal.com/developers/gitignore/api/macos
# Created by https://www.toptal.com/developers/gitignore/api/linux
# Edit at https://www.toptal.com/developers/gitignore?templates=linux
### Linux ###
*~
# temporary files which can be created if a process still has a handle open of a deleted file
.fuse_hidden*
# KDE directory preferences
.directory
# Linux trash folder which might appear on any partition or disk
.Trash-*
# .nfs files are created when an open file is removed but is still being accessed
.nfs*
# End of https://www.toptal.com/developers/gitignore/api/linux
# Created by https://www.toptal.com/developers/gitignore/api/windows
# Edit at https://www.toptal.com/developers/gitignore?templates=windows
### Windows ###
# Windows thumbnail cache files
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db
# Dump file
*.stackdump
# Folder config file
[Dd]esktop.ini
# Recycle Bin used on file shares
$RECYCLE.BIN/
# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp
# Windows shortcuts
*.lnk
# End of https://www.toptal.com/developers/gitignore/api/windows
# Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode
# Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode
### VisualStudioCode ###
.vscode/
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
# End of https://www.toptal.com/developers/gitignore/api/visualstudiocode
# Created by https://www.toptal.com/developers/gitignore/api/jetbrains
# Edit at https://www.toptal.com/developers/gitignore?templates=jetbrains
### JetBrains ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser
### JetBrains Patch ###
# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
# *.iml
# modules.xml
# .idea/misc.xml
# *.ipr
# Sonarlint plugin
# https://plugins.jetbrains.com/plugin/7973-sonarlint
.idea/**/sonarlint/
# SonarQube Plugin
# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin
.idea/**/sonarIssues.xml
# Markdown Navigator plugin
# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced
.idea/**/markdown-navigator.xml
.idea/**/markdown-navigator-enh.xml
.idea/**/markdown-navigator/
# Cache file creation bug
# See https://youtrack.jetbrains.com/issue/JBR-2257
.idea/$CACHE_FILE$
# CodeStream plugin
# https://plugins.jetbrains.com/plugin/12206-codestream
.idea/codestream.xml
# Azure Toolkit for IntelliJ plugin
# https://plugins.jetbrains.com/plugin/8053-azure-toolkit-for-intellij
.idea/**/azureSettings.xml
# End of https://www.toptal.com/developers/gitignore/api/jetbrains
# Created by https://www.toptal.com/developers/gitignore/api/vim
# Edit at https://www.toptal.com/developers/gitignore?templates=vim
### Vim ###
# Swap
[._]*.s[a-v][a-z]
!*.svg # comment out if you don't need vector files
[._]*.sw[a-p]
[._]s[a-rt-v][a-z]
[._]ss[a-gi-z]
[._]sw[a-p]
# Session
Session.vim
Sessionx.vim
# Temporary
.netrwhist
*~
# Auto-generated tag files
tags
# Persistent undo
[._]*.un~
# End of https://www.toptal.com/developers/gitignore/api/vim
algorithm_config.yaml 0 → 100644
Edit View file @ ac0fa6ab
algorithm_description: Run AOS Tracking paper plots on DPS
algorithm_name: aos_maap_dps
algorithm_version: main
build_command: aos_maap_dps/build.sh
disk_space: 40GB
docker_container_url: mas.maap-project.org/david.m.giles/aos_test/aos_dps:develop
inputs:
config: []
file: []
positional:
- default: ''
description: ''
name: analysis_date
required: false
queue: maap-dps-sandbox
repository_url: https://repo.maap-project.org/sshah/aos_maap_dps.git
run_command: aos_maap_dps/run.sh
\ No newline at end of file
build.sh 0 → 100644
Edit View file @ ac0fa6ab
#!/bin/bash
set -euo pipefail
# Install maap-py so we can use MAAP Secrets in run.sh
conda run --no-capture-output --name mytobac python -m pip install maap-py
notebooks/register-algorithm.ipynb 0 → 100644
Edit
View file @ ac0fa6ab
%% Cell type:markdown id:8a2d8d86-2187-44d7-9f44-d5e6583622b5 tags:
# Register the Algorithm
%% Cell type:code id:3561be3b-2600-4075-b177-0d79c78e334a tags:
``` python
import os
import os.path
import sys
import yaml
from maap.maap import MAAP
```
%% Cell type:markdown id:eb2e6f45-47cb-48c5-9fe2-feccc07d5c17 tags:
## Create MAAP Instance for Initiating Registration
%% Cell type:code id:38f966fd-9f1e-4f57-a930-14c0f471ea19 tags:
``` python
maap = MAAP()
```
%% Cell type:markdown id:ca846531-802e-4f23-8e5d-df84b812a590 tags:
## Initiate a Registration Process
Running the following cell will _initiate_ a registration process to register the algorithm configured in the file `algorith_config.yaml` at the root of this repository. If _initiation_ succeeds, the URL of the initiated registration _process_ will appear in the cell output, otherwise an error message will appear.
If an error appears in the cell output, it likely indicates that there is a problem with the DPS itself that is preventing the initiation of any registration processes, and you should use Slack to request assistance.
If a registration process URL appears in the cell output. Open a browser to the given URL to check the status of the regsitration process. This process will likely take 5-6 minutes to complete successfully, but might fail much more quickly than that.
When the registration process completes successfully will you be able to run the new version of the registered algorithm.
When the registration process fails, it might be a transient problem, so you may simply need to rerun the following cell to get past the transient error. If failure of registration processes persists after 3 or 4 attempts, use Slack to request assistance, as there may be a problem with DPS.
%% Cell type:code id:eb743993-1098-497e-bc17-94bcc81b1eaa tags:
``` python
base_dir = os.path.dirname(os.getcwd())
algorithm_config_yaml = os.path.join(base_dir, "algorithm_config.yaml")
with open(algorithm_config_yaml) as f:
algorithm_config = yaml.safe_load(f)
algorithm_name = algorithm_config["algorithm_name"]
algorithm_version = algorithm_config["algorithm_version"]
if response := maap.register_algorithm_from_yaml_file(algorithm_config_yaml):
job_web_url = response.json()["message"]["job_web_url"]
print(
f"Initiated algorithm registration for {algorithm_name}:{algorithm_version}."
f" Check progress at {job_web_url}."
)
else:
print("ERROR:", response.json()["message"], file=sys.stderr)
```
notebooks/set-secrets.ipynb 0 → 100644
Edit
View file @ ac0fa6ab
%% Cell type:markdown id:331918b2-c03c-4d14-9737-d65b94ed7e73 tags:
# Add MAAP Secrets for Earthdata Login Credentials
Earthdata Login credentials are required for downloading files from NASA DAACs. In order for a DPS job to be able to use such credentials in a safe manner, it looks for them in the MAAP Secrets store. Therefore, you must add the necessary secrets before a DPS job that requires them is submitted, otherwise the job will fail due to the missing secrets.
%% Cell type:code id:cbdb9c92-6219-4d58-b9f4-206ee507ce51 tags:
``` python
from maap.maap import MAAP
```
%% Cell type:code id:e2f0c041-646c-47e3-9c9f-4937ed9f7907 tags:
``` python
def mask(value: str, *, show_last: int = 4) -> str:
"""Mask a value.
Parameters
----------
value:
Value to mask.
show_last:
Number of trailing characters to leave unmasked.
Returns
-------
Masked value, where all but `show_last` characters of the specified value are
replaced with an asterisk (`*`). If necessary, the value is further left-padded
with asterisks to ensure there are at least as many asterisks as there are
unmasked trailing characters.
"""
# Make sure the returned value contains at least the same number of masked
# characters as unmasked characters for a bit of added "security."
n_masked = max(len(value), 2 * show_last) - show_last
return f"{'*' * n_masked}{value[-show_last:]}"
def prompt_user(
prompt: str,
*,
value: str | None = None,
sensitive: bool = True,
) -> str | None:
"""Prompt user for a possibly sensitive input value.
Parameters
----------
prompt:
Human-readable prompt to present to the user, which is combined with `value`
when prompting the user.
value:
Current value (if any) of what to prompt the user for, which is combined with
`prompt` when prompting the user, and masked, if `senstive` is `True`.
sensitive:
If `True`, mask the current value (if any) when prompting the user, and also
avoid echoing user input. Otherwise, the current value is left unmasked and
user input is echoed.
Returns
-------
Value entered by the user at the prompt, or `None` if either the user pressed the
Return or Enter key without supplying any input, or a KeyboardInterrupt occurred.
"""
from getpass import getpass
from IPython.display import clear_output
masked_value = mask(value) if value and sensitive else value
get_user_input = getpass if sensitive else input
try:
return get_user_input(f"{prompt} [{masked_value}]:") or None
except KeyboardInterrupt:
clear_output() # Remove prompt from cell output
raise
def get_secret(maap: MAAP, name: str) -> str | None:
"""Get the value of a MAAP secret, or `None` if the secret does not exist.
Parameters
----------
maap:
MAAP instance to use for secrets management.
name:
Name of the secret to get the value of.
Returns
-------
Value of the secret, if it exists; otherwise `None`.
"""
# Calling maap.secrets.get_secret returns the value of the requested secret, if it
# exists, but oddly returns an object with an error code if it does not exist,
# rather than simply (and conveniently) returning `None`.
return value if isinstance(value := maap.secrets.get_secret(name), str) else None
def set_secret_interactively(
maap: MAAP,
*,
prompt: str,
name: str,
sensitive: bool = True,
) -> str | None:
"""Set the value of a secret by prompting the user for a value.
If the secret already exists, the user prompt will include the existing value,
which will be masked if `sensitive` is `True` (showing only the last 4
characters of the value).
If an empty input is provided (i.e., either the Enter or Return key is pressed at
the prompt, without any other input, or a KeyboardInterrupt occurs), the secret is
not set and `None` is returned. Otherwise, the secret is set to the user-supplied
value and that value is returned.
Parameters
----------
maap:
MAAP instance to use for secrets management.
prompt:
Human-readable prompt to display to user when prompting for input.
name:
Name of the secret.
sensitive:
If `True`, the prompt will mask the current secret value, if there is one.
Further, user input will not be echoed. Otherwise, the prompt will show the
full secret value, if there is one, and user input will be echoed.
Returns
-------
User-supplied input provided at the prompt, or `None` if the Enter or Return key
was pressed without input.
"""
if (value := prompt_user(prompt, value=get_secret(maap, name), sensitive=sensitive)):
maap.secrets.add_secret(name, value)
masked_value = mask(value) if sensitive else value
print(f"The secret {name} was set to {masked_value}.")
return value
print(f"The value for secret {name} was left unchanged.")
```
%% Cell type:markdown id:97611ae6-837e-4021-9e4e-bee447749d5e tags:
## Create MAAP Instance for Managing Secrets
%% Cell type:code id:10612951-3392-4e32-983e-6e9a85c8c938 tags:
``` python
maap = MAAP()
```
%% Cell type:markdown id:98d52db0-7187-4aa3-a24a-7f3cbfce63c5 tags:
## Store Earthdata Login Credentials as MAAP Secrets
%% Cell type:code id:986a47cb-0586-4f62-827b-6df6fcb609d4 tags:
``` python
set_secret_interactively(
maap, prompt="Earthdata Login username", name="EARTHDATA_USERNAME", sensitive=False
)
set_secret_interactively(
maap, prompt="Earthdata Login password", name="EARTHDATA_PASSWORD"
);
```
notebooks/submit-job.ipynb 0 → 100644
Edit
View file @ ac0fa6ab
%% Cell type:markdown id:778cecaf-495e-4ab9-b8a7-7329b74fb49b tags:
# Submit an AOS Job
%% Cell type:code id:2af023f9-90f5-4749-add8-bc81d566014a tags:
``` python
from datetime import date
import ipywidgets as widgets
from maap.maap import MAAP
DEFAULT_QUEUE = "maap-dps-aos-worker-32gb"
```
%% Cell type:code id:50a503fc-8115-4b98-a5dc-4bd438b0f254 tags:
``` python
def get_job_queues(maap: MAAP) -> list[str]:
response = maap.getQueues()
response.raise_for_status()
return response.json()["queues"]
```
%% Cell type:code id:abd6f889-bfda-4592-b096-1b6bff1e120a tags:
``` python
def submit_aos_job(
maap: MAAP,
analysis_date: date,
queue: str = DEFAULT_QUEUE,
) -> str | None:
"""Submit an AOS job for an analysis date.
Returns
-------
A MAAP job ID, if a job was successfully submitted; otherwise `None`.
"""
import re
import sys
if analysis_date is None:
print("Specify a date", file=sys.stderr)
return None
analysis_date_str = analysis_date.strftime("%Y-%m-%d")
job = maap.submitJob(
identifier=analysis_date_str, # custom job tag/label
algo_id="aos_maap_dps", # algorithm name
version="main", # algorithm version
queue=queue, # job queue
username=maap.profile.account_info()["username"],
# Algorithm inputs:
analysis_date=analysis_date_str,
)
if not job.id:
match = re.search(
r"<ows:ExceptionText>(?P<msg>.*)</ows:ExceptionText>", job.error_details
)
print(match["msg"] if match else job.error_details, file=sys.stderr)
return None
print(f"Submitted job with job ID {job.id}", file=sys.stderr)
return job.id
```
%% Cell type:markdown id:bb0a6014-8fcd-474f-94c7-bbeb5f35f50c tags:
## Specify Inputs and Submit a Job
%% Cell type:code id:987727e8-0cdb-4613-a9ef-071a7872abd4 tags:
``` python
maap = MAAP()
```
%% Cell type:code id:689c9988-ff4b-41a0-aa2b-427e96d2a16e tags:
``` python
date_picker = widgets.DatePicker(
description="Analysis Date:",
style=dict(description_width="initial"),
)
queues = get_job_queues(maap)
queue_picker = widgets.Dropdown(
description="Job Queue:",
options=queues,
value=DEFAULT_QUEUE if DEFAULT_QUEUE in queues else None,
style=dict(description_width="initial"),
)
interactively = widgets.interact_manual.options(manual_name="Submit AOS Job")
interactively(
submit_aos_job,
maap=widgets.fixed(maap),
analysis_date=date_picker,
queue=queue_picker
);
```
%% Output
run.sh 0 → 100644
Edit View file @ ac0fa6ab
#!/bin/bash
set -euo pipefail
readarray -d " " -t credentials <<<"$(
conda run --no-capture-output --name mytobac python -c '
from maap.maap import MAAP
maap = MAAP()
username = maap.secrets.get_secret("EARTHDATA_USERNAME")
password = maap.secrets.get_secret("EARTHDATA_PASSWORD")
print(username, password)
'
)"
EARTHDATA_USERNAME=$(echo -e "${credentials[0]}" | tr -d '[:space:]')
EARTHDATA_PASSWORD=$(echo -e "${credentials[1]}" | tr -d '[:space:]')
export EARTHDATA_USERNAME
export EARTHDATA_PASSWORD
# All of the scripts below will use AOS_WORKING_DIR as the root output directory.
AOS_WORKING_DIR="${PWD}/output"
export AOS_WORKING_DIR
analysis_date=$1
conda run --no-capture-output --name mytobac python /root/aos_test/src/acquire_and_convert_calipso.py "${analysis_date}"
conda run --no-capture-output --name mytobac python /root/aos_test/src/acquire_goes.py "${analysis_date}"
conda run --no-capture-output --name mytobac python /root/aos_test/src/process_object_tracking.py "${analysis_date}"