diff --git a/docs/cli/cli-tips-tricks.md b/docs/cli/cli-tips-tricks.md index 81d4bfb8..c13b220b 100644 --- a/docs/cli/cli-tips-tricks.md +++ b/docs/cli/cli-tips-tricks.md @@ -5,24 +5,24 @@ title: More CLI Tips & Tricks ## About This document shows off a range of more advanced command-line workflows, making use of a wider range -of tools in the command-line & geospatial ecosystem. Some of them can be a pain to install, like -GDAL/OGR, and several pop in and out of web tools, so these are kept out of the main tutorial -section. +of tools in the command-line & geospatial ecosystem. Some of them can be a pain to install, like +GDAL/OGR, and several pop in and out of web tools, so these are kept out of the main tutorial +section. **WORK IN PROGRESS**: This document is still under construction, with a number of TODO’s remaining, but we are publishing as there’s a lot of good information here. ## Tools used -* **[GDAL/OGR](https://gdal.org)** - We’ll mostly use OGR, the vector tooling. +* **[GDAL/OGR](https://gdal.org)** - We’ll mostly use OGR, the vector tooling. Great for things like format conversion and basic simplification. * **[Keplergl_cli](https://github.com/kylebarron/keplergl_cli#usage)** - Nice tool to call the -awesome kepler.gl library from the commandline. Useful for visualization of large amounts of +awesome kepler.gl library from the commandline. Useful for visualization of large amounts of geojson. * **[GeoJSON.io](https://geojson.io/)** - Simple tool to do editing of geojson, useful for creating AOI’s. It integrates with github, but the ability to save a GeoJSON to github doesn't seem to work so well. * **[Placemark.io](https://placemark.io)** - More advanced tool from the creator of GeoJSON.io, very -nice for creating AOI’s and piping them in, with lots of rich geometry editing features. +nice for creating AOI’s and piping them in, with lots of rich geometry editing features. * **[MapShaper](https://github.com/mbloch/mapshaper)** - Tool to do interactive simplification of GeoJSON, has a nice CLI. * **[STACTools](https://github.com/stac-utils/stactools)** - CLI for working with STAC data. There @@ -34,15 +34,15 @@ future. ### Geometry Inputs -While the command-line can often be quicker than using a UI, one place that can be slower is +While the command-line can often be quicker than using a UI, one place that can be slower is getting the geometry input for searching or clipping. Hand-editing GeoJSON is a huge pain, so most -people will open up a desktop tool like QGIS or ArcGIS Pro and save the file. But there are a few +people will open up a desktop tool like QGIS or ArcGIS Pro and save the file. But there are a few tools that can get you back into the CLI workflow more quickly. #### Use the Features API Rather than using GeoJSON in the SDK, upload your GeoJSON to the [Features API](https://developers.planet.com/docs/apis/features/) and use references across the system with the sdk. -References are used in the geometry block of our services like: +References are used in the geometry block of our services in a GeoJSON blob like: ```json "geometry": { @@ -50,10 +50,12 @@ References are used in the geometry block of our services like: "type": "ref" } ``` +Or as a string in a geometry option like `"pl:features/my/[collection-id]/[feature-id]"` + #### Draw with GeoJSON.io -One great tool for quickly drawing on a map and getting GeoJSON output is +One great tool for quickly drawing on a map and getting GeoJSON output is [GeoJSON.io](https://geojson.io). You can draw and save the file, but an even faster workflow is to use your operating system’s clipboard to command-line tools. @@ -74,7 +76,7 @@ pbpaste | planet data filter --geom - | planet data search SkySatCollect --filt A really fantastic tool for working with GeoJSON is [Placemark](https://placemark.io). It is a commercial tool that you’ll have to pay for, but it’s got a really nice feature that makes it very -compatible with command-line workflows. You can easily grab the URL of any individual GeoJSON +compatible with command-line workflows. You can easily grab the URL of any individual GeoJSON feature and stream it in as your geometry using `curl`: ![Stream from Placemark](https://user-images.githubusercontent.com/407017/179412209-2365d79a-9260-47e5-9b08-9bc5b84b6ddc.gif) @@ -92,8 +94,8 @@ let you pipe (`|`) the output more directly. #### Copy GeoJSON to clipboard -One of the quicker routes to visualizing search output is to copy the output to your clipboard and paste into a -tool that will take GeoJSON and visualize it. +One of the quicker routes to visualizing search output is to copy the output to your clipboard and paste into a +tool that will take GeoJSON and visualize it. You can do this on GeoJSON.io: @@ -113,7 +115,7 @@ planet data filter --string-in strip_id 5743669 | planet data search PSScene --f #### Post to Github as gist -Another easy option that is a bit more persistent is to post to Github using the +Another easy option that is a bit more persistent is to post to Github using the [`gh` cli tool](https://github.com/cli/cli). Specifically using the `gist create` command. The following command will get the latest SkySat image captured, upload to github, and open @@ -147,13 +149,13 @@ planet data filter --string-in strip_id $stripid | planet data search PSScene -- One of the best tools to visualize large numbers of imagery footprints is a tool called [kepler.gl](https://kepler.gl/), which has a really awesome command-line version which is perfect for working with Planet’s CLI. To get the CLI go to -[keplergl_cli](https://github.com/kylebarron/keplergl_cli) and follow the +[keplergl_cli](https://github.com/kylebarron/keplergl_cli) and follow the [installation instructions](https://github.com/kylebarron/keplergl_cli#install). Be sure to get a Mapbox API key (from the [access tokens](https://account.mapbox.com/access-tokens/) page) - just sign up for a free account if you don't have one already. The kepler CLI won't work at all without getting one and setting it as the `MAPBOX_API_KEY` environment variable. -Once it’s set up you can just pipe any search command directly to `kepler` (it usually does fine even without +Once it’s set up you can just pipe any search command directly to `kepler` (it usually does fine even without `planet collect` to go from ndgeojson to geojson). For example: ```console @@ -191,9 +193,9 @@ curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a073 #### Large Dataset Visualization -Oftentimes it can be useful to visualize a large amount of data, to really get a sense of the -coverage and then do some filtering of the output. For this we recommend downloading the output -to disk. Getting 20,000 skysat collects will take at least a couple of minutes, and will be over +Oftentimes it can be useful to visualize a large amount of data, to really get a sense of the +coverage and then do some filtering of the output. For this we recommend downloading the output +to disk. Getting 20,000 skysat collects will take at least a couple of minutes, and will be over 100 megabytes of GeoJSON on disk. ```console @@ -267,7 +269,7 @@ Smaller ratios preserve the character of concave features better. ##### Simplification with OGR -The other thing you’ll likely want to do to visualize large amounts of data is to simplify it +The other thing you’ll likely want to do to visualize large amounts of data is to simplify it some. Many simplification tools call for a 'tolerance', often set in degrees. For SkySat some useful values are: | tolerance | result | @@ -276,7 +278,7 @@ some. Many simplification tools call for a 'tolerance', often set in degrees. Fo | 0.01 | Messes with the shape a bit, but the footprint generally looks the same, with a couple vertices off. | | 0.1 | Mashes the shape, often into a triangle, but still useful for understanding broad coverage. | -It’s worth experimenting with options between these as well. The more simplification the easier it is for programs to +It’s worth experimenting with options between these as well. The more simplification the easier it is for programs to render the results. `ogr2ogr` includes the ability to simplify any output: ```console @@ -289,14 +291,14 @@ Alternative - use convex hull. TODO: test this, write it up ogr2ogr skysat-convex.gpkg skysat.geojson ogr2ogr -sql "select st_convexhull(geometry) from skysat" -dialect sqlite ``` -Other alternative for really big ones, centroid. GDAL should be able to do this, need to figure out the similar +Other alternative for really big ones, centroid. GDAL should be able to do this, need to figure out the similar sql. #### Simplification with Mapshaper -Another great tool is [Mapshaper](https://github.com/mbloch/mapshaper), which excels at simplification. It offers a -web-based user interface to see the results of simplification, and also a command-line tool you can use if you -find a simplification percentage you’re happy with. After you get it +Another great tool is [Mapshaper](https://github.com/mbloch/mapshaper), which excels at simplification. It offers a +web-based user interface to see the results of simplification, and also a command-line tool you can use if you +find a simplification percentage you’re happy with. After you get it [installed](https://github.com/mbloch/mapshaper#installation) you can fire up the UI with: ```console @@ -312,13 +314,13 @@ interface, or you can also run the command-line program: mapshaper -i footprints.geojson -simplify 15% -o simplified.geojson ``` -Once you find a simplification amount you’re happy with you can use it as a piped output. +Once you find a simplification amount you’re happy with you can use it as a piped output. ```console planet data search --limit 20 SkySatCollect - | planet collect - | mapshaper -i - -simplify 15% -o skysat-ms2.geojson ``` -Mapshaper also has more simplification algorithms to try out, so we recommend diving into the +Mapshaper also has more simplification algorithms to try out, so we recommend diving into the [CLI options](https://github.com/mbloch/mapshaper/wiki/Command-Reference). #### Simplification with QGIS @@ -328,7 +330,7 @@ Another good tool for simplification is QGIS. TODO: Flesh out this section, add in command-line qgis_processing option. Other simplification options for large datasets: - + * Use QGIS, run 'convex hull' (Vector -> Geoprocessing -> Convex Hull). Good idea to convert to gpkg or shapefile before you open in qgis if large. ### Advanced jq @@ -339,7 +341,7 @@ Other simplification options for large datasets: - get id by array number ```console -planet orders list | jq -rs '.[3] | "\(.id) \(.created_on) \(.name) \(.state)"' +planet orders list | jq -rs '.[3] | "\(.id) \(.created_on) \(.name) \(.state)"' ``` (limit can get the most recent, but not a second or third) diff --git a/planet/cli/data.py b/planet/cli/data.py index 8000927f..42889983 100644 --- a/planet/cli/data.py +++ b/planet/cli/data.py @@ -15,7 +15,6 @@ from typing import List, Optional from contextlib import asynccontextmanager from pathlib import Path - import click from planet.reporting import AssetStatusBar @@ -81,9 +80,15 @@ def check_item_types(ctx, param, item_types) -> Optional[List[dict]]: raise click.BadParameter(str(e)) -def check_geom(ctx, param, geometry: Optional[dict]) -> Optional[dict]: +def check_geom(ctx, param, geometry) -> Optional[dict]: """Validates geometry as GeoJSON or feature ref(s).""" - return geojson.as_geom_or_ref(geometry) if geometry else None + if isinstance(geometry, dict): + return geojson.as_geom_or_ref(geometry) + geoms = {} + if geometry: + for geom in geometry: + geoms.update(geojson.as_geom_or_ref(geom)) + return geoms if geoms else None def check_item_type(ctx, param, item_type) -> Optional[List[dict]]: @@ -286,7 +291,7 @@ def filter(ctx, @click.argument("item_types", type=types.CommaSeparatedString(), callback=check_item_types) -@click.option("--geom", type=types.JSON(), callback=check_geom) +@click.option("--geom", type=types.Geometry(), callback=check_geom) @click.option('--filter', type=types.JSON(), help="""Apply specified filter to search. Can be a json string, @@ -332,7 +337,7 @@ async def search(ctx, item_types, geom, filter, limit, name, sort, pretty): @click.argument("item_types", type=types.CommaSeparatedString(), callback=check_item_types) -@click.option("--geom", type=types.JSON(), callback=check_geom) +@click.option("--geom", type=types.Geometry(), callback=check_geom) @click.option( '--filter', type=types.JSON(), @@ -500,7 +505,10 @@ async def search_delete(ctx, search_id): type=str, required=True, help='Name of the saved search.') -@click.option("--geom", type=types.JSON(), callback=check_geom, default=None) +@click.option("--geom", + type=types.Geometry(), + callback=check_geom, + default=None) @click.option('--daily-email', is_flag=True, help='Send a daily email when new results are added.') diff --git a/planet/cli/subscriptions.py b/planet/cli/subscriptions.py index 26df617b..f9e8d169 100644 --- a/planet/cli/subscriptions.py +++ b/planet/cli/subscriptions.py @@ -13,6 +13,7 @@ from .. import subscription_request from ..subscription_request import sentinel_hub from ..specs import get_item_types, validate_item_type, SpecificationException +from planet import geojson ALL_ITEM_TYPES = get_item_types() valid_item_string = "Valid entries for ITEM_TYPES: " + "|".join(ALL_ITEM_TYPES) @@ -29,6 +30,17 @@ def check_item_types(ctx, param, item_types) -> Optional[List[dict]]: raise click.BadParameter(str(e)) +def check_geom(ctx, param, geometry) -> Optional[dict]: + """Validates geometry as GeoJSON or feature ref(s).""" + if isinstance(geometry, dict): + return geojson.as_geom_or_ref(geometry) + geoms = {} + if geometry: + for geom in geometry: + geoms.update(geojson.as_geom_or_ref(geom)) + return geoms if geoms else None + + def check_item_type(ctx, param, item_type) -> Optional[List[dict]]: """Validates the item type provided by comparing it to all supported item types.""" @@ -346,7 +358,8 @@ def request(name, @click.option( '--geometry', required=True, - type=types.JSON(), + type=types.Geometry(), + callback=check_geom, help="""Geometry of the area of interest of the subscription that will be used to determine matches. Can be a string, filename, or - for stdin.""") @click.option('--start-time', @@ -418,7 +431,8 @@ def request_catalog(item_types, @click.option( '--geometry', required=True, - type=types.JSON(), + type=types.Geometry(), + callback=check_geom, help="""Geometry of the area of interest of the subscription that will be used to determine matches. Can be a string, filename, or - for stdin.""") @click.option('--start-time', @@ -437,6 +451,7 @@ def request_pv(var_type, var_id, geometry, start_time, end_time, pretty): Variables](https://developers.planet.com/docs/subscriptions/pvs-subs/#planetary-variables-types-and-ids) for details. """ + # print("Geom is", geometry) res = subscription_request.planetary_variable_source( var_type, var_id, diff --git a/planet/cli/types.py b/planet/cli/types.py index 94ced235..6032fe70 100644 --- a/planet/cli/types.py +++ b/planet/cli/types.py @@ -93,6 +93,20 @@ def convert(self, value, param, ctx) -> dict: return convdict +class Geometry(click.ParamType): + name = 'geom' + + def __init__(self): + self.types = [JSON(), CommaSeparatedString()] + + def convert(self, value, param, ctx): + for type in self.types: + try: + return type.convert(value, param, ctx) + except click.BadParameter: + continue + + class Field(click.ParamType): """Clarify that this entry is for a field""" name = 'field' diff --git a/planet/geojson.py b/planet/geojson.py index 77bb3247..2e0813a6 100644 --- a/planet/geojson.py +++ b/planet/geojson.py @@ -19,7 +19,6 @@ import geojson as gj from jsonschema import Draft7Validator - from .constants import DATA_DIR from .exceptions import GeoJSONError, FeatureError @@ -28,7 +27,7 @@ LOGGER = logging.getLogger(__name__) -def as_geom_or_ref(data: dict) -> dict: +def as_geom_or_ref(data) -> dict: """Extract the geometry from GeoJSON and validate. Parameters: @@ -42,6 +41,8 @@ def as_geom_or_ref(data: dict) -> dict: or FeatureCollection or if more than one Feature is in a FeatureCollection. """ + if isinstance(data, str): + return as_ref(data) geom_type = data['type'] if geom_type == 'ref': return as_ref(data) @@ -51,16 +52,49 @@ def as_geom_or_ref(data: dict) -> dict: return geom -def as_ref(data: dict) -> dict: - geom_type = data['type'] - if geom_type.lower() != 'ref': - raise FeatureError( - f'Invalid geometry reference: {geom_type} is not a reference (the type should be "ref").' - ) - if "content" not in data: - raise FeatureError( - 'Invalid geometry reference: Missing content block that contains the reference.' - ) +def validate_ref(uri) -> bool: + if uri is None: + raise FeatureError("Expected str, not None") + parts = uri.split("/", 4) + if parts[0] != "pl:features": + raise FeatureError("Expected scheme pl:features") + path = parts[1:] + if len(path) < 2: + raise FeatureError("Expceted dataset/collection path") + return True + + +def convert_ref_to_dict(data: str) -> dict: + """ Ensure geom reference is in the expected format + Then convert it into a geometry block + + Parameters: + data: str, a feature reference + Returns: + GeoJSON geometry reference + """ + if validate_ref(data): + geom = { + "type": "ref", + "content": data, + } + return geom + return None + + +def as_ref(data) -> dict: + if isinstance(data, str): + data = convert_ref_to_dict(data) + if isinstance(data, dict): + geom_type = data['type'] + if geom_type.lower() != 'ref': + raise FeatureError( + f'Invalid geometry reference: {geom_type} is not a reference (the type should be "ref").' + ) + if "content" not in data: + raise FeatureError( + 'Invalid geometry reference: Missing content block that contains the reference.' + ) return data diff --git a/planet/subscription_request.py b/planet/subscription_request.py index 361b3d71..88d13df7 100644 --- a/planet/subscription_request.py +++ b/planet/subscription_request.py @@ -167,7 +167,7 @@ def build_request(name: str, def catalog_source( item_types: List[str], asset_types: List[str], - geometry: Mapping, + geometry: dict, start_time: datetime, filter: Optional[Mapping] = None, end_time: Optional[datetime] = None, @@ -250,7 +250,7 @@ def catalog_source( parameters = { "item_types": item_types, "asset_types": asset_types, - "geometry": geojson.as_geom_or_ref(dict(geometry)), + "geometry": geojson.as_geom_or_ref(geometry), } try: @@ -287,7 +287,7 @@ def planetary_variable_source( "forest_carbon_diligence_30m", "field_boundaries_sentinel_2_p1m"], var_id: str, - geometry: Mapping, + geometry: dict, start_time: datetime, end_time: Optional[datetime] = None, ) -> dict: @@ -355,7 +355,7 @@ def planetary_variable_source( parameters = { "id": var_id, - "geometry": geojson.as_geom_or_ref(dict(geometry)), + "geometry": geojson.as_geom_or_ref(geometry), } try: diff --git a/tests/conftest.py b/tests/conftest.py index 79f851ba..028cb17d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -121,6 +121,11 @@ def geom_geojson(): } # yapf: disable +@pytest.fixture +def str_geom_reference(): + return "pl:features/my/water-fields-RqB0NZ5/rmQEGqm" + + @pytest.fixture def geom_reference(): return { diff --git a/tests/integration/test_data_cli.py b/tests/integration/test_data_cli.py index aff36c2f..bc6cb0f5 100644 --- a/tests/integration/test_data_cli.py +++ b/tests/integration/test_data_cli.py @@ -436,7 +436,8 @@ def test_data_search_cmd_item_types(item_types, expect_success, invoke): @respx.mock @pytest.mark.parametrize("geom_fixture", [('geom_geojson'), ('feature_geojson'), - ('featurecollection_geojson'), ('geom_reference')]) + ('featurecollection_geojson'), ('geom_reference'), + ("str_geom_reference")]) def test_data_search_cmd_top_level_geom(geom_fixture, request, invoke): """Ensure that all GeoJSON forms of describing a geometry are handled and all result in the same, valid GeometryFilter being created""" @@ -446,8 +447,10 @@ def test_data_search_cmd_top_level_geom(geom_fixture, request, invoke): }]}) respx.post(TEST_QUICKSEARCH_URL).return_value = mock_resp geom = request.getfixturevalue(geom_fixture) + if isinstance(geom, dict): + geom = json.dumps(geom) - result = invoke(["search", 'PSScene', f"--geom={json.dumps(geom)}"]) + result = invoke(["search", 'PSScene', f"--geom={geom}"]) assert result.exit_code == 0 diff --git a/tests/integration/test_subscriptions_cli.py b/tests/integration/test_subscriptions_cli.py index fd276e6c..56e9d5d5 100644 --- a/tests/integration/test_subscriptions_cli.py +++ b/tests/integration/test_subscriptions_cli.py @@ -321,12 +321,17 @@ def test_request_base_success(invoke, geom_geojson): assert result.exit_code == 0 # success. -def test_request_base_clip_to_source(invoke, geom_geojson): +@pytest.mark.parametrize("geom_fixture", + [('geom_geojson'), ('geom_reference'), + ("str_geom_reference")]) +def test_request_base_clip_to_source(geom_fixture, request, invoke): """Clip to source using command line option.""" + geom = request.getfixturevalue(geom_fixture) + print("geom is", geom) source = json.dumps({ "type": "catalog", "parameters": { - "geometry": geom_geojson, + "geometry": geom, "start_time": "2021-03-01T00:00:00Z", "item_types": ["PSScene"], "asset_types": ["ortho_analytic_4b"] @@ -344,7 +349,7 @@ def test_request_base_clip_to_source(invoke, geom_geojson): req = json.loads(result.output) tool = req["tools"][0] assert tool["type"] == "clip" - assert tool["parameters"]["aoi"] == geom_geojson + assert tool["parameters"]["aoi"] == geom def test_request_catalog_success(invoke, geom_geojson): @@ -378,15 +383,18 @@ def test_subscriptions_results_csv(invoke): assert result.output.splitlines() == ["id,status", "1234-abcd,SUCCESS"] -@pytest.mark.parametrize("geom", ["geom_geojson", "geom_reference"]) +@pytest.mark.parametrize( + "geom", ["geom_geojson", "geom_reference", "str_geom_reference"]) def test_request_pv_success(invoke, geom, request): """Request-pv command succeeds""" geom = request.getfixturevalue(geom) + if isinstance(geom, dict): + geom = json.dumps(geom) result = invoke([ "request-pv", "--var-type=biomass_proxy", "--var-id=BIOMASS-PROXY_V3.0_10", - f"--geometry={json.dumps(geom)}", + f"--geometry={geom}", "--start-time=2021-03-01T00:00:00", ]) diff --git a/tests/unit/test_geojson.py b/tests/unit/test_geojson.py index 9ed1a4dc..38d9c6b6 100644 --- a/tests/unit/test_geojson.py +++ b/tests/unit/test_geojson.py @@ -99,7 +99,7 @@ def test_validate_geom_as_geojson_empty_coordinates(geom_geojson): _ = geojson.validate_geom_as_geojson(geom_geojson) -def test_as_geom_or_ref(geom_geojson): +def test_as_geojson(geom_geojson): assert geojson.as_geom_or_ref(geom_geojson) == geom_geojson @@ -107,10 +107,23 @@ def test_as_polygon(geom_geojson): assert geojson.as_polygon(geom_geojson) == geom_geojson -def test_as_reference(geom_reference): +def test_as_ref(geom_reference): assert geojson.as_ref(geom_reference) == geom_reference +def test_as_str_ref(str_geom_reference): + geomify_ref = { + "type": "ref", + "content": str_geom_reference, + } + assert geojson.as_ref(str_geom_reference) == geomify_ref + + +def test_as_invalid_ref(): + with pytest.raises(exceptions.FeatureError): + geojson.as_ref("some:nonesense/with/nothing") + + def test_as_polygon_wrong_type(point_geom_geojson): with pytest.raises(exceptions.GeoJSONError): _ = geojson.as_polygon(point_geom_geojson)