Bulk Image Processing

Disclaimer

The example Python source code is meant merely as a starting point to quickly interact and retrieve insights from ADMA. It is in no way intended for production workloads.

Overview

Once you have successfully executed tutorial.example.imagery, you can use the same Party, Farm and Field to execute this example. This example is similar to the tutorial.example.imagery, but allows you to retrieve all types of supported imagery for a specific field over a range of days.

Background

This example is slightly more in depth in that it starts considering the details of Sentinel-2 satellite imagery. The Sentinel-2 constellation does not provide daily imagery and on those days when imagery is available, the images may be too cloudy to be useful. For the Sentinel-2 constellation, a location is revisited every five days.

The Imagery service supports three types of imagery: TRUE_COLOR, SCOUTING_MAP, and VEGETATION_MAP. True color may be generated on any day containing imagery and is not constrained by the amount of cloud cover. Scouting and vegetation imagery on the other hand are constrained by cloud cover. If the cloud cover is greater than 80% then the day is too cloudy and no useful SCOUTING_MAP or VEGETATION_MAP imagery can be generated.

There are other common error conditions to consider, too. The example assumes not all HTTP calls are successful and performs up to five retries for the following HTTP status codes: 429, 502, 503, 504.

Bulk Image Processing

Approach

The example, tutorial.example.bulk_imagery, uses the following approach. Note that this approach is for demonstration purposes only.

1. Given a Party, Field, starting date, and a range of days, attempt to generate a true color image for each day. If the request is successful, then you may be assured that an image is available. Since an image was generated, the necessary bands will have been ingested and cached for further use.
2. Step 1 provides a list of days for which images are available. Using this much smaller list, attempt to generate scouting imagery. Scouting map image generation will either succeed or fail depending on the amount of cloud cover. Scouting and vegetation imagery will only generate if the cloud cover percentage is 80% or less.
3. Step 2 provides a list of days for which an image has 80% or less cloud cover. With this even smaller list, generate the remaining vegetation imagery.

E.g.

The approach attempts to reduce the number of calls to the Imagery service in order reduce the costs associated with using SentinelHub and the Imagery service. A superior method might be to request only the SCL cloud band, as that band contains less variable pixel data and is only a single band. This exercise is left up to the reader, but more inventive methods to reduce costs may be useful for high volume pipelines that look for daily imagery over thousands of Fields.

It is worth noting in the example image above, that for the specified month, only two days provided satellite imagery that was conducive to analytical image generation. So, any methods to reduce unnecessary calls are useful for performance and cost savings.

Logging

Due to the long-running nature of this example, logging was enabled. The logging is configured to log to stdout in JSON format, ensure that the timestamp is in ISO 8601 using the UTC time zone, and is backed by a queue. The log configuration file is resource/logging.yml and all logging code exists in tutorial/util/logging.py.

Log Example Download

A sample run of the log output from this example may be downloaded as bulk-imagery-log.

Job IDs

In this example, job ids are generated using the following snippet. While this is sufficient for a first run, each separate request to the Imagery service must have a unique job id, or you will receive an error like the one below the snippet.

job_id = f'{image_date.isoformat()}-{layer}'

Error for 2023-08-28-VEGETATION_MAP
{ "error": {
  "code": "Conflict",
  "message": "Job JobId [https://production.farmbeats.azure.net, 2023-08-28-VEGETATION_MAP-] already exists."
  },
  "traceId": "4a7ca2c78d313f18e0bf40eb7fe4c5ae"
}

So a potential workaround for this issue is to provide a suffix to the generated job ids, but this may not be sufficient for your use case.

job_id = f'{image_date.isoformat()}-{layer}-{number:03d}'

Whatever your use case, keep in mind that you will need to keep track of these job ids for subsequent requests to the Imagery service and for retrieving the generated insight attachments. The approach for this tutorial is to prefix each output with the generated job id.

Additional Details

Before we execute the command, a little explanation is in order. A new directory will be created in the root of the adma directory named output. Within the output directory, the following directory structure will be created: party_id/resource_id/solution_id. This directory will hold a log files of the submitted requests and their subsequent responses. In the following example the file is named 2023-08-28-VEGETATION_MAP.txt. This directory will also hold the generated insights. JSON files are a JSON representation of the legend that accompanies the TIFF images. Each type of insight, TRUE_COLOR, SCOUTING_MAP, and VEGETATION_MAP, has an associated JSON file.

Output Directory Structure

output
└── 551c7fb3-8b9c-4b5c-8ecc-d6830325ce12
    └── b159f824-4d12-4b40-aa38-b98f0c876edf
        └── bayerAgPowered.imagery
            ├── 2023-08-28-VEGETATION_MAP-5ae18207-947c-4b87-8bb7-04dd1aad4c74.tiff
            ├── 2023-08-28-VEGETATION_MAP-643e4515-29d2-440c-8afb-22900aff826a.json
            └── 2023-08-28-VEGETATION_MAP.txt

Sample Output Directory Download

A sample output directory from this example may be downloaded as bulk-imagery-output.

Generate Bulk Imagery

Note that you will need to modify lines 202-205 in tutorial/example/bulk_imagery.py to change the number_of_days, image_date, and add the UUID of the Party and Field in order to generate and retrieve the insights and their attachments.

Note this command will take a while to complete. You may want to limit the number of days for it to complete sooner. The command is almost complete when it prints the message "Attempting to generate 'VEGETATION_MAP' for days found with non-cloudy imagery.".

Note there is a secondary Python file tutorial/example/concurrent/bulk_imagery.py that works very similarly but executes image generation in a concurrent manner. This will execute in a much more timely manner.

python -m tutorial.example.bulk_imagery