### Set Up Directories for Raster Vision Quickstart Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/quickstart.md Export environment variables for code and output directories and create them. Ensure these directories exist before proceeding with Docker setup. ```bash > export RV_QUICKSTART_CODE_DIR=`pwd`/code > export RV_QUICKSTART_OUT_DIR=`pwd`/output > mkdir -p ${RV_QUICKSTART_CODE_DIR} ${RV_QUICKSTART_OUT_DIR} ``` -------------------------------- ### Install necessary Python packages Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/stac_plus_osm.ipynb Installs the osm2geojson and pystac_client libraries. Ensure these are installed before running the notebook. ```python %pip install osm2geojson==0.2 pystac_client ``` -------------------------------- ### Upload All Example Outputs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Uploads outputs for all examples to the model zoo. This command is used to share results from multiple examples. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ upload ``` -------------------------------- ### Install documentation dependencies Source: https://github.com/azavea/raster-vision/blob/master/docs/README.md Run this command to install the necessary packages for building the documentation. ```sh > pip install -r requirements.txt ``` -------------------------------- ### Collect All Example Outputs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Downloads outputs for all examples. The `--remote` flag should be used if the outputs are stored remotely. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ collect --remote ``` -------------------------------- ### Compare All Example Runs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Compares `eval.json` outputs across all examples between two different versions. Specify the root directories for old and new example outputs. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ compare \ --examples_root_old "s3://raster-vision/examples/0.30.0/output/" \ --examples_root_new "s3://raster-vision/examples/0.31.0/output/" ``` -------------------------------- ### Install PyTorch Lightning Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/lightning_workflow.ipynb Installs the required PyTorch Lightning library for model training. ```bash %pip install -q pytorch-lightning==2.* ``` -------------------------------- ### Install Twine for PyPI publishing Source: https://github.com/azavea/raster-vision/blob/master/docs/release.md Install the tool required to upload packages to PyPI. ```console pip install twine ``` -------------------------------- ### Initialize Tensorboard Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/train.ipynb Load the Tensorboard extension and start the server to monitor training logs. ```python %load_ext tensorboard ``` ```python %tensorboard --bind_all --logdir "data/train-demo/tb-logs" --reload_interval 10 ``` -------------------------------- ### Run All Examples Remotely Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Executes all available examples remotely. Ensure the `--remote` flag is included for remote execution. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ run --remote ``` -------------------------------- ### Install Raster Vision via pip Source: https://github.com/azavea/raster-vision/blob/master/README.md Install the Raster Vision library directly using pip. This is the simplest way to get started. ```sh pip install rastervision ``` -------------------------------- ### Collect Example Outputs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Downloads the output of one or more commands (e.g., `train`, `eval`) for a specific example run. Use the `--remote` flag if outputs are stored remotely. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ collect "spacenet-rio-cc" \ --remote ``` -------------------------------- ### Install third-party license generator Source: https://github.com/azavea/raster-vision/blob/master/docs/release.md Install the utility required to generate the project's third-party license file. ```console pip install third_party_license_file_generator ``` -------------------------------- ### Install Dependencies Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_raster_data.ipynb Install the seaborn library for data visualization. ```python %pip install -q seaborn ``` -------------------------------- ### Run an Example Remotely Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Executes a specific example remotely. Use the `--remote` flag for remote execution. Optional overrides can be provided using the `-o` flag. ```sh python \ "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ run "spacenet-rio-cc" \ -o "remote.processed_uri" "s3://raster-vision-ahassan/rv/0.20/examples/processed/spacenet/rio" \ -o "remote.root_uri" "s3://raster-vision-ahassan/rv/0.20/examples/output/cc/spacenet-rio/" ``` ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ run "spacenet-rio-cc" \ --remote ``` ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ run "isprs-potsdam-ss" \ -o "remote.raw_uri" "s3://raster-vision-ahassan/potsdam/data/raw" \ --remote ``` -------------------------------- ### Install SageMaker support Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/aws.md Install the required package to enable AWS SageMaker integration. ```console > pip install rastervision_aws_sagemaker=={{ version }} ``` -------------------------------- ### Compare Example Runs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Compares the `eval.json` outputs of two different runs for a specific example. Provide URIs for both the old and new runs. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ compare \ --root_uri_old "s3://raster-vision/examples/0.30.0/output/spacenet-rio-cc/" \ --root_uri_new "s3://raster-vision/examples/0.31.0/output/spacenet-rio-cc/" ``` -------------------------------- ### Display Project Directory Structure Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/quickstart.md Use the `tree` command to visualize the project's file and directory hierarchy. You may need to install the `tree` utility first. ```console > tree -L 3 ├── Makefile ├── bundle │ └── model-bundle.zip ├── eval │ └── validation_scenes │ └── eval.json ├── pipeline-config.json ├── predict │ └── scene_25 │ └── labels.tif └── train ├── dataloaders │ ├── train.png │ └── valid.png ├── last-model.pth ├── learner-config.json ├── log.csv ├── model-bundle.zip ├── tb-logs │ ├── events.out.tfevents.1670510483.c5c1c7621fb7.1830.0 │ ├── events.out.tfevents.1670511197.c5c1c7621fb7.2706.0 │ └── events.out.tfevents.1670595249.986730ccbe70.7507.0 ├── valid_metrics.json └── valid_preds.png ``` -------------------------------- ### Install pystac_client dependency Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/temporal.ipynb Install the required pystac_client library for STAC API interactions. ```python %pip install -q pystac_client ``` -------------------------------- ### Launch Docker Console Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/examples.md Starts the Docker container environment with optional support for AWS, GPU, and Tensorboard. ```console docker/run [--aws] [--gpu] [--tensorboard] ``` -------------------------------- ### Upload Example Outputs Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Uploads model bundles, evaluation results, sample images, predictions, and training logs to the model zoo for a specific example. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ upload "spacenet-rio-cc" ``` -------------------------------- ### Install individual Raster Vision packages Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Install specific Raster Vision packages individually. This is equivalent to installing the main rastervision package. ```console pip install rastervision_pipeline=={{ version }} pip install rastervision_aws_s3=={{ version }} pip install rastervision_aws_batch=={{ version }} pip install rastervision_core=={{ version }} pip install rastervision_pytorch_learner=={{ version }} pip install rastervision_pytorch_backend=={{ version }} ``` -------------------------------- ### Install Raster Vision from TestPyPI Source: https://github.com/azavea/raster-vision/blob/master/docs/release.md Test your published package by installing it from the TestPyPI index using pip. Ensure you use the --index-url flag. ```console pip install --index-url https://test.pypi.org/simple/ rastervision ``` -------------------------------- ### Distributed Training with torchrun Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/gpu.md Utilize the `torchrun` CLI command for distributed training across multiple GPUs on a single machine. This example shows how to configure it for 4 GPUs. ```bash torchrun --standalone --nnodes=1 --nproc-per-node=4 --no-python \ rastervision run local rastervision_pytorch_backend/rastervision/pytorch_backend/examples/tiny_spacenet.py ``` -------------------------------- ### Launch Jupyter Notebook Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/examples.md Starts a Jupyter notebook server within the Docker container for data preprocessing. ```console docker/run --jupyter [--aws] ``` -------------------------------- ### Install rastervision.gdal_vsi plugin Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Install the GDAL VSI plugin for Raster Vision to add VsiFileSystem for GDAL IO. ```console pip install rastervision_gdal_vsi=={{ version }} ``` -------------------------------- ### Install Folium for Interactive Maps Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/onnx_inference.ipynb Install the folium library, which is used for creating interactive maps to visualize geospatial data. ```bash %pip install -q folium ``` -------------------------------- ### Configure Pipeline with Custom Greeting Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Configures SamplePipeline2Config with a custom greeting for the MessageMaker. This demonstrates how to override default configurations in hierarchical setups. ```python from rastervision.pipeline_example_plugin1.sample_pipeline2 import ( SamplePipeline2Config, MessageMakerConfig) def get_config(runner, root_uri): names = ['alice', 'bob', 'susan'] # Same as before except we can set the greeting to be # 'hola' instead of 'hello'. message_maker = MessageMakerConfig(greeting='hola') return SamplePipeline2Config( root_uri=root_uri, names=names, message_maker=message_maker) ``` -------------------------------- ### Install Raster Vision via pip Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Install the Raster Vision library using pip. Ensure you specify the desired version. ```console pip install rastervision=={{ version }} ``` -------------------------------- ### Configure Semantic Segmentation LabelStore with RGB Output Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/pipelines.md This example configures a SemanticSegmentationLabelStoreConfig to output predictions in RGB format and also as polygons for a specific class ID (0). ```python label_store = SemanticSegmentationLabelStoreConfig( rgb=True, vector_output=[PolygonVectorOutputConfig(class_id=0)]) scene = SceneConfig( id=id, raster_source=raster_source, label_source=label_source, label_store=label_store) ``` -------------------------------- ### Download and Unzip EORSSD Dataset Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/prechipped_datasets.ipynb Use wget to download the dataset and apt to install unzip, then unzip the dataset into the specified directory. ```python !wget "https://github.com/rmcong/EORSSD-dataset/raw/master/EORSSD.zip" -P "data/" !apt install unzip -y !unzip -q "data/EORSSD.zip" -d "data/EORSSD" ``` -------------------------------- ### Predict with a Model Bundle Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Downloads a model bundle for a specific example and uses it to make predictions on a sample image. Use `--remote` for remote model bundles. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ predict "spacenet-rio-cc" \ --remote ``` -------------------------------- ### Create a DatasetConfig with Training and Validation Scenes Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/pipelines.md Use DatasetConfig to define training, validation, and test splits. Each split is a list of SceneConfigs. This example shows a dataset with a single training and a single validation scene. ```python train_scene = make_scene('scene_205', train_image_uri, train_label_uri, class_config) val_scene = make_scene('scene_25', val_image_uri, val_label_uri, class_config) scene_dataset = DatasetConfig( class_config=class_config, train_scenes=[train_scene], validation_scenes=[val_scene]) ``` -------------------------------- ### Load Image and Get Shape Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_raster_data.ipynb Instantiates a RasterioSource and retrieves its shape and data type. This example shows the initial shape of a 191-band hyperspectral image. ```python from rastervision.core.data import RasterioSource, MinMaxTransformer img_uri = 'data/Hyperspectral_Project/dc.tif' raster_source_hsi = RasterioSource(img_uri) raster_source_hsi.shape, raster_source_hsi.dtype ``` -------------------------------- ### Download and prepare EuroSat dataset Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/prechipped_datasets.ipynb Downloads the EuroSat dataset, installs unzip utility, extracts the data, and reorganizes the directory structure for easier access. This is a prerequisite for using the ClassificationImageDataset with this data. ```bash !wget "https://madm.dfki.de/files/sentinel/EuroSAT.zip" -P "data/" --no-check-certificate !apt install unzip -y !unzip -q "data/EuroSAT.zip" -d "data/EuroSAT" !mv "data/EuroSAT/2750/"* "data/EuroSAT/" && rm -rf "data/EuroSAT/2750" ``` -------------------------------- ### Import dependencies and initialize environment Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/temporal.ipynb Import necessary Raster Vision, PyTorch, and visualization modules, and set the compute device. ```python from rastervision.core.box import Box from rastervision.core.data import (RasterioCRSTransformer, StatsTransformer, XarraySource) from rastervision.core.data.raster_source import XarraySource from rastervision.core.data import Scene from rastervision.pytorch_learner import ( SemanticSegmentationRandomWindowGeoDataset) from tqdm.auto import tqdm import math import torch import pystac_client from shapely.geometry import mapping from matplotlib import pyplot as plt import seaborn as sns sns.reset_defaults() DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu' ``` -------------------------------- ### Install GDAL via conda Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md If pip installation of GDAL fails, install it using conda. Replace with the specific version required. ```console conda install -c conda-forge gdal== ``` -------------------------------- ### Build documentation Source: https://github.com/azavea/raster-vision/blob/master/docs/README.md Commands to generate the HTML documentation or run a live local server for development. ```sh > make html ``` ```sh > make livehtml ``` -------------------------------- ### Initialize project with cookiecutter Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/bootstrap.md Run the cookiecutter command against the template directory to generate the project skeleton. ```console [lfishgold@monoshone ~/projects] $ cookiecutter raster-vision/cookiecutter_template/ caps_project_name [MY_PROJECT]: project_name [my_project]: docker_image [my_project]: parent_docker_image [quay.io/azavea/raster-vision:pytorch-{{ version }}]: version [{{ version }}]: description [A Raster Vision plugin]: url [https://github.com/azavea/raster-vision]: author [Azavea]: author_email [info@azavea.com]: ``` -------------------------------- ### Create a random window dataset Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/sampling_training_data.ipynb Initializes a SemanticSegmentationRandomWindowGeoDataset with class configuration and window sampling parameters. ```python from rastervision.core.data import ClassConfig from rastervision.pytorch_learner import SemanticSegmentationRandomWindowGeoDataset import albumentations as A class_config = ClassConfig( names=['background', 'building'], colors=['lightgray', 'darkred'], null_class='background') ds = SemanticSegmentationRandomWindowGeoDataset.from_uris( class_config=class_config, image_uri=image_uri, label_vector_uri=label_uri, label_vector_default_class_id=class_config.get_class_id('building'), image_raster_source_kw=dict(allow_streaming=True), # window sizes will randomly vary from 100x100 to 300x300 size_lims=(100, 300), # resize chips to 256x256 before returning out_size=256, # allow windows to overflow the extent by 100 pixels padding=100, max_windows=10 ) img_full = ds.scene.raster_source[:, :] windows = [ds.sample_window() for _ in range(50)] show_windows(img_full, windows, title='Random windows') ``` -------------------------------- ### Install rastervision.aws_sagemaker plugin Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Install the AWS SageMaker plugin for Raster Vision to enable running jobs on AWS SageMaker. ```console pip install rastervision_aws_sagemaker=={{ version }} ``` -------------------------------- ### Configure Sample Pipeline with Deluxe Message Maker Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Sets up the SamplePipeline2Config to use the custom DeluxeMessageMakerConfig, specifying the greeting and the new 'level' parameter. ```python from rastervision.pipeline.config import register_config from rastervision.pipeline_example_plugin1.sample_pipeline2 import ( SamplePipeline2Config) from rastervision.pipeline_example_plugin2.deluxe_message_maker import ( DeluxeMessageMakerConfig) def get_config(runner, root_uri): names = ['alice', 'bob', 'susan'] # Note that we use the DeluxeMessageMakerConfig and set the level to 3. message_maker = DeluxeMessageMakerConfig(greeting='hola', level=3) return SamplePipeline2Config( root_uri=root_uri, names=names, message_maker=message_maker) ``` -------------------------------- ### Display docker/run Help Information Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Use the `--help` flag to display usage instructions, available environment variables, and options for the `docker/run` script. This is useful for understanding how to configure and execute commands within the Raster Vision Docker environment. ```console > docker/run --help Usage: run Run a console in a Raster Vision Docker image locally. By default, the raster-vision-pytorch image is used in the CPU runtime. Environment variables: RASTER_VISION_DATA_DIR (directory for storing data; mounted to /opt/data) RASTER_VISION_NOTEBOOK_DIR (optional directory for Jupyter notebooks; mounted to /opt/notebooks) AWS_PROFILE (optional AWS profile) Options: --aws forwards AWS credentials (sets AWS_PROFILE env var and mounts ~/.aws to /root/.aws) --tensorboard maps port 6006 --name sets the name of the running container --jupyter forwards port 8888, mounts RASTER_VISION_NOTEBOOK_DIR to /opt/notebooks, and runs Jupyter --docs runs the docs server and forwards port 8000 --debug forwards port 3000 for use with remote debugger --gpu use nvidia runtime --arm64 uses image built for arm64 architecture All arguments after above options are passed to 'docker run'. ``` -------------------------------- ### Configure label store Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/onnx_inference.ipynb Set up the label store with class configurations and output settings for semantic segmentation. ```python class_config = ClassConfig(names=['bg', 'cloud'], null_class='bg') label_store = SemanticSegmentationLabelStore( uri=out_dir, class_config=class_config, bbox=raster_source.bbox, crs_transformer=raster_source.crs_transformer, vector_outputs=[PolygonVectorOutputConfig(class_id=1)], smooth_output=True, ) ``` -------------------------------- ### Create Scene via Utility Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/scenes_and_aois.ipynb Uses a helper function to simplify the creation of a scene. ```python from rastervision.core.data.utils import make_ss_scene scene = make_ss_scene( class_config=class_config, image_uri=image_uri, label_vector_uri=label_uri, label_vector_default_class_id=class_config.get_class_id('building'), image_raster_source_kw=dict(allow_streaming=True)) ``` -------------------------------- ### Create Random Window Dataset for Training Source: https://context7.com/azavea/raster-vision/llms.txt Initializes a SemanticSegmentationRandomWindowGeoDataset with image and label URIs, applying augmentations via Albumentations. ```python ds_random = SemanticSegmentationRandomWindowGeoDataset.from_uris( image_uri='image.tif', label_vector_uri='buildings.geojson', class_config=class_config, label_vector_default_class_id=0, out_size=256, size_lims=(200, 300), # Random window sizes between 200-300 max_windows=1000, transform=A.Compose([ A.HorizontalFlip(p=0.5), A.VerticalFlip(p=0.5), A.RandomRotate90(p=0.5), A.RandomBrightnessContrast(p=0.2) ]) ) # Use with PyTorch DataLoader train_loader = DataLoader( ds_random, batch_size=8, shuffle=True, num_workers=4 ) for x, y in train_loader: print(f"Batch x: {x.shape}, y: {y.shape}") # x: (8, 3, 256, 256), y: (8, 256, 256) break ``` -------------------------------- ### Configure Dataset Padding Options Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/sampling_training_data.ipynb Demonstrates creating `SemanticSegmentationSlidingWindowGeoDataset` instances with different padding configurations. `padding=None` allows automatic calculation, while `padding=0` disables padding. ```python ds_pad_all_sides = SemanticSegmentationSlidingWindowGeoDataset.from_uris( class_config=class_config, image_uri=image_uri, label_vector_uri=label_uri, label_vector_default_class_id=class_config.get_class_id('building'), image_raster_source_kw=dict(allow_streaming=True), size=200, stride=200, out_size=256, padding=None, # will be automatically calculated if None pad_direction='both', ) ds_no_padding = SemanticSegmentationSlidingWindowGeoDataset.from_uris( class_config=class_config, image_uri=image_uri, label_vector_uri=label_uri, label_vector_default_class_id=class_config.get_class_id('building'), image_raster_source_kw=dict(allow_streaming=True), size=200, stride=200, out_size=256, padding=0, ) ``` -------------------------------- ### Define a simple pipeline and configuration Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Demonstrates the basic structure of a Pipeline and its associated PipelineConfig. Requires registration of the configuration class. ```python from os.path import join from rastervision.pipeline.pipeline import Pipeline from rastervision.pipeline.file_system import str_to_file, file_to_str from rastervision.pipeline.pipeline_config import PipelineConfig from rastervision.pipeline.config import register_config from rastervision.pipeline.utils import split_into_groups # Each Config needs to be registered with a type hint which is used for ``` -------------------------------- ### Get Class ID Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_labels.ipynb Retrieve the integer ID for a given class name from a ClassConfig object. ```python print('background', class_config.get_class_id('background')) print('foreground', class_config.get_class_id('foreground')) ``` -------------------------------- ### Build Raster Vision Packages Source: https://github.com/azavea/raster-vision/blob/master/docs/release.md Run this script from the repository's root directory to build the packages for distribution. ```console scripts/build_packages ``` -------------------------------- ### Initialize Object Detection Visualizer Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/prechipped_datasets.ipynb Sets up an ObjectDetectionVisualizer for plotting images with bounding boxes and class labels. Requires class names and optionally class colors. The 'scale' attribute can adjust the plot size. ```python from rastervision.pytorch_learner import ObjectDetectionVisualizer viz = ObjectDetectionVisualizer( class_names=['Airplane', 'Truncated_airplane'], class_colors=['red', 'green']) viz.scale = 8 viz.plot_batch(x.unsqueeze(0), [y], show=True) ``` -------------------------------- ### Visualize Validation Data Batch Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/train.ipynb Get and plot a batch of data from the validation dataset using the visualizer. ```python x, y = viz.get_batch(val_ds, 4) viz.plot_batch(x, y, show=True) ``` -------------------------------- ### Implement Sample Pipeline Logic Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Implements the logic for the sample pipeline, including saving and printing messages. Commands are defined in the 'commands' list. ```python from rastervision.core.pipeline import Pipeline from rastervision.core.backend import register_config from rastervision.core.utils.misc import split_into_groups from rastervision.core.file_system import str_to_file, file_to_str class SamplePipeline(Pipeline): # The order in which commands run. Each command correspond to a method. commands: list[str] = ['save_messages', 'print_messages'] # Split commands can be split up and run in parallel. split_commands = ['save_messages'] # GPU commands are run using GPUs if available. There are no commands worth running # on a GPU in this pipeline. gpu_commands = [] def save_messages(self, split_ind=0, num_splits=1): # Save a file for each name with a message. # The num_splits is the number of parallel jobs to use and # split_ind tracks the index of the parallel job. In this case # we are splitting on the names/message_uris. split_groups = split_into_groups( list(zip(self.config.names, self.config.message_uris)), num_splits) split_group = split_groups[split_ind] for name, message_uri in split_group: message = 'hello {}!'.format(name) # str_to_file and most functions in the file_system package can # read and write transparently to different file systems based on # the URI pattern. str_to_file(message, message_uri) print('Saved message to {}'.format(message_uri)) def print_messages(self): # Read all the message files and print them. for message_uri in self.config.message_uris: message = file_to_str(message_uri) print(message) ``` -------------------------------- ### Visualize Training Data Batch Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/train.ipynb Get and plot a batch of data from the training dataset using the visualizer. ```python x, y = viz.get_batch(train_ds, 4) viz.plot_batch(x, y, show=True) ``` -------------------------------- ### Download and Unzip Hyperspectral Data Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_raster_data.ipynb Downloads and extracts a hyperspectral image dataset. Ensure you have `wget` and `unzip` installed. ```bash !wget "http://cobweb.ecn.purdue.edu/~biehl/Hyperspectral_Project.zip" -P "data" !apt install unzip !unzip "data/Hyperspectral_Project.zip" -d "data/" ``` -------------------------------- ### Visualize Prediction Scores Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/lightning_workflow.ipynb Use matplotlib to display prediction scores for building and background classes. Requires matplotlib to be installed. ```python from matplotlib import pyplot as plt scores_building = scores[0] scores_background = scores[1] fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5)) fig.tight_layout(w_pad=-2) ax1.imshow(scores_building, cmap='plasma') ax1.axis('off') ax1.set_title('building') ax2.imshow(scores_background, cmap='plasma') ax2.axis('off') ax2.set_title('background') plt.show() ``` -------------------------------- ### Initialize Scene Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/onnx_inference.ipynb Combine the raster source and label store into a single Scene object. ```python scene = Scene(id='', raster_source=raster_source, label_store=label_store) ``` -------------------------------- ### Display Raster Vision CLI Help Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/cli.md Use `rastervision --help` to display the main command and its top-level options, including available subcommands. ```console > rastervision --help Usage: python -m rastervision.pipeline.cli [OPTIONS] COMMAND [ARGS]... The main click command. Sets the profile, verbosity, and tmp_dir in RVConfig. Options: -p, --profile TEXT Sets the configuration profile name to use. -v, --verbose Increment the verbosity level. --tmpdir TEXT Root of temporary directories to use. --help Show this message and exit. Commands: predict Use a model bundle to predict on new images. predict_scene Use a model bundle to predict on a new scene. run Run sequence of commands within pipeline(s). run_command Run an individual command within a pipeline. ``` -------------------------------- ### Get Score Array for Visualization Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/onnx_inference.ipynb Extract the score array for a specific extent from the predicted labels. This array can be used for visualization purposes. ```python score = labels.get_score_arr(labels.extent) ``` -------------------------------- ### Initialize Classification Visualizer Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/prechipped_datasets.ipynb Sets up a ClassificationVisualizer for displaying images and their predicted class labels. It requires the list of class names. The plot_batch method visualizes a batch of data. ```python from rastervision.pytorch_learner import ClassificationVisualizer viz = ClassificationVisualizer(class_names=ds.orig_dataset.classes) viz.plot_batch(x.unsqueeze(0), y.unsqueeze(0), show=True) ``` -------------------------------- ### Visualize Object Detection Dataset Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/visualize_data_samples.ipynb Initialize a sliding window dataset and visualizer for object detection tasks. ```python ds = ObjectDetectionSlidingWindowGeoDataset.from_uris( class_config=class_config, image_uri=image_uri, label_vector_uri=label_uri, size=chip_sz, stride=chip_stride, label_vector_default_class_id=class_config.get_class_id('building'), image_raster_source_kw=dict(allow_streaming=True)) vis = ObjectDetectionVisualizer( class_names=class_config.names, class_colors=class_config.colors, channel_display_groups=channel_display_groups) x, y = vis.get_batch(ds, 4) vis.plot_batch(x, y, show=True) ``` -------------------------------- ### Predict with All Model Bundles Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Downloads and uses model bundles from all examples for prediction. The `--remote` flag is required for remote model bundles. ```sh python "rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py" \ predict \ --remote ``` -------------------------------- ### Retrieve Labels for a Specific Window Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_labels.ipynb Get the class labels for a given window from the label source. This is a direct query after labels have potentially been populated. ```python labels = label_source.get_labels(window) ``` -------------------------------- ### Define Sample Pipeline Configuration Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Defines a configuration schema for a sample pipeline. Use this to set parameters like names and message URIs for the pipeline. ```python from rastervision.core.config import PipelineConfig from rastervision.core.backend import register_config from os.path import join @register_config('pipeline_example_plugin1.sample_pipeline') class SamplePipelineConfig(PipelineConfig): # Config classes are configuration schemas. Each field is an attributes # with a type and optional default value. names: list[str] = ['alice', 'bob'] message_uris: list[str] | None = None def build(self, tmp_dir): # The build method is used to instantiate the corresponding object # using this configuration. return SamplePipeline(self, tmp_dir) def update(self): # The update method is used to set default values as a function of # other values. if self.message_uris is None: self.message_uris = [ join(self.root_uri, '{}.txt'.format(name)) for name in self.names ] ``` -------------------------------- ### Set GDAL_DATA environment variable Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/index.md Set the GDAL_DATA environment variable, which is required by rasterio. This command finds the GDAL data directory within your rasterio installation. ```console export GDAL_DATA=$(pip show rasterio | grep Location | awk '{print $NF"/rasterio/gdal_data/"}') ``` -------------------------------- ### Synchronize Processed Data Source: https://github.com/azavea/raster-vision/blob/master/rastervision_pytorch_backend/rastervision/pytorch_backend/examples/README.md Synchronizes processed data from one S3 location to another using the AWS CLI. This is useful for updating processed data for examples. ```sh aws s3 sync \ s3://raster-vision/examples/0.30.0/processed-data/ \ s3://raster-vision/examples/0.31.0/processed-data/ ``` -------------------------------- ### Set training parameters and output directory Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/lightning_workflow.ipynb Configures hyperparameters such as batch size, learning rate, and number of epochs, and creates the output directory. ```python batch_size = 8 lr = 1e-4 epochs = 3 output_dir = 'data/lightning-demo/' make_dir(output_dir) fast_dev_run = False ``` -------------------------------- ### Run all unit tests Source: https://github.com/azavea/raster-vision/blob/master/tests/README.md Executes the full test suite and generates coverage reports if the coverage package is available. ```sh scripts/unit_tests ``` ```sh python -m unittest discover -t . tests -vf ``` -------------------------------- ### Aggregate requirements files Source: https://github.com/azavea/raster-vision/blob/master/docs/release.md Combine all project requirements into a single file for license generation. ```console cat `find **/requirements.txt` requirements.txt requirements-dev.txt > requirements-all.txt; ``` -------------------------------- ### Get city bounding box Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/stac_plus_osm.ipynb Calculates the bounding box for the specified city using previously defined functions. The result is a Box object and its corresponding Shapely geometry. ```python # get city bounding box from OSM bbox = get_city_bbox(country, city) bbox_geometry = mapping(bbox.to_shapely()) bbox, bbox_geometry ``` -------------------------------- ### Display `run` Subcommand Help Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/cli.md Use `rastervision run --help` to view detailed usage information for the `run` subcommand, including its arguments and options. ```console > rastervision run --help Usage: rastervision run [OPTIONS] RUNNER CFG_MODULE [COMMANDS]... Run COMMANDS within pipelines in CFG_MODULE using RUNNER. RUNNER: name of the Runner to use CFG_MODULE: the module with `get_configs` function that returns PipelineConfigs. This can either be a Python module path or a local path to a .py file. COMMANDS: space separated sequence of commands to run within pipeline. The order in which to run them is based on the Pipeline.commands attribute. If this is omitted, all commands will be run. Options: -a, --arg KEY VALUE Arguments to pass to get_config function -s, --splits INTEGER Number of processes to run in parallel for splittable commands --pipeline-run-name TEXT The name for this run of the pipeline. --help Show this message and exit. ``` -------------------------------- ### Extract a chip from a cropped RasterSource Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_raster_data.ipynb Extract a specific chip from the already cropped RasterSource. The chip dimensions are determined by the slicing applied to the source. This example visualizes the extracted chip. ```python from rastervision.core.box import Box from matplotlib import pyplot as plt chip = raster_source_cropped[:400, :400] print(chip.shape) fig, ax = plt.subplots(figsize=(5, 5)) ax.matshow(chip) plt.show() ``` -------------------------------- ### Configure and Create ChipClassificationLabelSource Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_labels.ipynb Set up the ChipClassificationLabelSourceConfig with parameters like intersection-over-area threshold and whether to infer cells. Then, instantiate the ChipClassificationLabelSource using the configuration, vector source, and raster bounding box. ```python from rastervision.core.data import (ChipClassificationLabelSource, ChipClassificationLabelSourceConfig) cfg = ChipClassificationLabelSourceConfig( ioa_thresh=0.5, use_intersection_over_cell=False, pick_min_class_id=False, background_class_id=class_config.null_class_id, infer_cells=True) label_source = ChipClassificationLabelSource( cfg, vector_source, bbox=raster_source.bbox, lazy=True) ``` -------------------------------- ### Visualize bands from MultiRasterSource chip Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_raster_data.ipynb Display different band combinations from the extracted chip using Matplotlib. This example visualizes the RGB, SWIR, VH, and VV bands separately. ```python from matplotlib import pyplot as plt fig, (ax_rgb, ax_swir, ax_vh, ax_vv) = plt.subplots(1, 4, figsize=(15, 4)) ax_rgb.matshow(chip[..., :3]) ax_rgb.set_title('RGB') ax_swir.matshow(chip[..., 3:6]) ax_swir.set_title('SWIR') ax_vh.matshow(chip[..., 6]) ax_vh.set_title('VH') ax_vv.matshow(chip[..., 7]) ax_vv.set_title('VV') plt.show() ``` -------------------------------- ### Get Predictions from DataLoader Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/lightning_workflow.ipynb Defines a generator function to iterate through a dataloader, perform inference using the model, and yield predictions. It ensures predictions are in numpy format and applies a softmax function. ```python def get_predictions(dataloader): for x, _ in tqdm(dataloader): with torch.inference_mode(): out_batch = model(x) out_batch = out_batch.softmax(dim=1) # This needs to yield a single prediction, not a whole batch of them. for out in out_batch: yield out.numpy() ``` -------------------------------- ### Configure Solver Settings Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/train.ipynb Configure solver parameters such as batch size, learning rate, and class loss weights. ```python from rastervision.pytorch_learner import SolverConfig solver_cfg = SolverConfig( batch_sz=8, lr=3e-2, class_loss_weights=[1., 10.] ) ``` -------------------------------- ### Run Raster Vision Docker Container Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/quickstart.md Execute the Raster Vision Docker container, mounting local directories for code and output. This command starts an interactive bash session inside the container. ```bash > docker run --rm -it \ -v ${RV_QUICKSTART_CODE_DIR}:/opt/src/code \ -v ${RV_QUICKSTART_OUT_DIR}:/opt/data/output \ quay.io/azavea/raster-vision:pytorch-{{ version }} /bin/bash ``` -------------------------------- ### Configure PyTorch Lightning Trainer and Logger Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/lightning_workflow.ipynb Sets up the PyTorch Lightning Trainer with specified accelerators, epochs, logging, and other training configurations. Includes TensorBoardLogger for metrics. ```python from pytorch_lightning.loggers import CSVLogger, TensorBoardLogger deep_lab = deeplabv3_resnet50(num_classes=len(class_config)) model = SemanticSegmentation(deep_lab, lr=lr) tb_logger = TensorBoardLogger(save_dir=output_dir, flush_secs=10) trainer = pl.Trainer( accelerator='auto', min_epochs=1, max_epochs=epochs+1, default_root_dir=output_dir, logger=[tb_logger], fast_dev_run=fast_dev_run, log_every_n_steps=1, ) ``` -------------------------------- ### Create Folium Map with Vector Predictions Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/onnx_inference.ipynb Reads vector data from a GeoJSON file, calculates the centroid, and creates an interactive Folium map displaying the vector predictions. Ensure GeoPandas and Folium are installed. ```python import geopandas as gpd gdf = gpd.read_file(join(out_dir, 'vector_output', 'class-1-cloud.json')) loc = np.array(gdf.unary_union.centroid.xy).squeeze().tolist()[::-1] m = folium.Map(location=loc, zoom_start=12) folium.GeoJson(gdf).add_to(m) m ``` -------------------------------- ### Query Chip Class ID (With Buildings) Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/reading_labels.ipynb Extract a different chip and query its class ID. This example shows a chip that contains buildings, and the label source correctly returns the building class ID. ```python from matplotlib import pyplot as plt chip = raster_source[:200, 200:400] label = label_source[:200, 200:400] fig, ax = plt.subplots(figsize=(5, 5)) ax.matshow(chip) ax.set_title(f'Class ID: {label}') plt.show() ``` -------------------------------- ### Create Pipeline Configuration Function Source: https://github.com/azavea/raster-vision/blob/master/docs/framework/architecture.md Provides a function to instantiate the SamplePipelineConfig. This function acts like a config file, allowing for dynamic configuration using Python code. ```python from rastervision.pipeline_example_plugin1.sample_pipeline import ( SamplePipelineConfig) def get_config(runner, root_uri): # The get_config function returns an instantiated PipelineConfig and # plays a similar role as a typical "config file" used in other systems. # It's different in that it can have loops, conditionals, local variables, # etc. The runner argument is the name of the runner used to run the # pipeline (eg. local or batch). Any other arguments are passed from the # CLI using the -a option. names = ['alice', 'bob', 'susan'] # Note that root_uri is a field that is inherited from PipelineConfig, # the parent class of SamplePipelineConfig, and specifies the root URI # where any output files are saved. return SamplePipelineConfig(root_uri=root_uri, names=names) ``` -------------------------------- ### Configure Label Sources Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/scenes_and_aois.ipynb Sets up class configurations and transforms vector labels into a rasterized source aligned with the raster source. ```python from rastervision.core.data import ( ClassConfig, ClassInferenceTransformer, GeoJSONVectorSource, RasterizedSource, Scene) class_config = ClassConfig( names=['background', 'building'], colors=['lightgray', 'darkred'], null_class='background') class_inf_tf = ClassInferenceTransformer( default_class_id=class_config.get_class_id('building')) vector_source = GeoJSONVectorSource( label_uri, crs_transformer=raster_source.crs_transformer, vector_transformers=[class_inf_tf]) label_raster_source = RasterizedSource( vector_source=vector_source, background_class_id=class_config.null_class_id, bbox=raster_source.bbox) ``` -------------------------------- ### Initialize Semantic Segmentation Visualizer Source: https://github.com/azavea/raster-vision/blob/master/docs/usage/tutorials/train.ipynb Initialize a visualizer for semantic segmentation tasks using class names and colors. ```python import albumentations as A from rastervision.pytorch_learner import ( SemanticSegmentationRandomWindowGeoDataset, SemanticSegmentationSlidingWindowGeoDataset, SemanticSegmentationVisualizer) viz = SemanticSegmentationVisualizer( class_names=class_config.names, class_colors=class_config.colors) ``` -------------------------------- ### Run Docker Container with GPU Access Source: https://github.com/azavea/raster-vision/blob/master/docs/setup/gpu.md When running Raster Vision within a Docker container, use the `--gpus=all` flag to grant the container access to all available GPUs. Ensure the NVIDIA Docker runtime is installed. ```bash docker run --gpus=all --rm -it quay.io/azavea/raster-vision:pytorch-{{ version }} /bin/bash ```