allow loose format in Google Cloud Functions & AWS Lambda targets (#2…

…0789)

Because it's a (not) common use-case for people to deploy google cloud
functions via something like

```
gcloud functions deploy abc --trigger-http --runtime=python311 --entry-point=handler 
```

When omitted the `--source` flag will default to the current directory.

This is often easier than uploading to an explicit bucket and specifying
`--source=gs://bucket`

docs/docs/python/integrations/aws-lambda.mdx

@@ -7,10 +7,10 @@ Create a Lambda with Python code.
 
 ---
 
-Pants can create a Lambda-compatible zip file from your Python code, allowing you to develop your Lambda functions and layers in your repository instead of using the online Cloud9 editor.
+Pants can create a Lambda-compatible zip file or directory from your Python code, allowing you to develop your Lambda functions and layers in your repository instead of using the online Cloud9 editor.
 
 :::note FYI: how Pants does this
-Under-the-hood, Pants uses the [PEX](https://github.com/pex-tool/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file, in the format recommended by AWS.
+Under-the-hood, Pants uses the [PEX](https://github.com/pex-tool/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file or directory, in the format recommended by AWS.
 :::
 
 ## Step 1: Activate the Python AWS Lambda backend
@@ -58,6 +58,10 @@ Pants will use [dependency inference](../../using-pants/key-concepts/targets-and
 
 You can optionally set the `output_path` field to change the generated zip file's path.
 
+:::tip Using layout
+Use [layout](../../../reference/targets/python_aws_lambda_function.mdx#layout) to determine whether to build a `.zip` file or a directory
+:::
+
 :::caution Use `resource` instead of `file`
 `file` / `files` targets will not be included in the built AWS Lambda artifacts because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` and `resources` target. See [Assets and archives](../../using-pants/assets-and-archives.mdx) for further explanation.
 :::

docs/docs/python/integrations/google-cloud-functions.mdx

@@ -7,10 +7,10 @@ Create a Cloud Function with Python.
 
 ---
 
-Pants can create a Google Cloud Function-compatible zip file from your Python code, allowing you to develop your functions in your repository.
+Pants can create a Google Cloud Function-compatible zip file or directory from your Python code, allowing you to develop your functions in your repository.
 
 :::note FYI: how Pants does this
-Under-the-hood, Pants uses the [PEX](https://github.com/pex-tool/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file, in the format recommended by Google Cloud Functions.
+Under-the-hood, Pants uses the [PEX](https://github.com/pex-tool/pex) project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file or directory, in the format recommended by Google Cloud Functions.
 :::
 
 ## Step 1: Activate the Python Google Cloud Function backend
@@ -59,6 +59,10 @@ Pants will use [dependency inference](../../using-pants/key-concepts/targets-and
 
 You can optionally set the `output_path` field to change the generated zip file's path.
 
+:::tip Using layout
+Use [layout](../../../reference/targets/python_google_cloud_function.mdx#layout) to determine whether to build a `.zip` file or a directory
+:::
+
 :::caution Use `resource` instead of `file`
 `file` / `files` targets will not be included in the built Cloud Function because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` / `resources` target. See [Assets and archives](../../using-pants/assets-and-archives.mdx) for further explanation.
 :::
@@ -91,9 +95,15 @@ A better solution is to work with the dependencies to stop them from packaging f
 
 ## Step 4: Upload to Google Cloud
 
-You can use any of the various Google Cloud methods to upload your zip file, such as the Google Cloud console or the [Google Cloud CLI](https://cloud.google.com/functions/docs/deploying/filesystem#deploy_using_the_gcloud_tool).
+You can use any of the various Google Cloud methods to upload your zip file or directory, such as the Google Cloud console or the [Google Cloud CLI](https://cloud.google.com/functions/docs/deploying/filesystem#deploy_using_the_gcloud_tool).
+
+You must specify the `--entry-point` as `handler`. This is a re-export of the function referred to by the `handler` field of the target.
 
-You must specify the handler as `handler`. This is a re-export of the function referred to by the `handler` field of the target.
+For example, if using `layout="flat"`:
+
+```
+gcloud functions deploy --source=dist/project/cloud_function --entry-point=handler --trigger-topic=<TOPIC> --runtime=python38 <FUNCTION_NAME>
+```
 
 ## Advanced: Using PEX directly
 

src/python/pants/backend/awslambda/python/rules.py

@@ -18,6 +18,7 @@
 from pants.backend.python.util_rules.faas import (
     BuildPythonFaaSRequest,
     PythonFaaSCompletePlatforms,
+    PythonFaaSLayoutField,
     PythonFaaSPex3VenvCreateExtraArgsField,
 )
 from pants.backend.python.util_rules.faas import rules as faas_rules
@@ -36,6 +37,7 @@ class _BaseFieldSet(PackageFieldSet):
     runtime: PythonAwsLambdaRuntime
     complete_platforms: PythonFaaSCompletePlatforms
     pex3_venv_create_extra_args: PythonFaaSPex3VenvCreateExtraArgsField
+    layout: PythonFaaSLayoutField
     output_path: OutputPathField
     environment: EnvironmentField
 
@@ -71,6 +73,7 @@ async def package_python_aws_lambda_function(
             include_requirements=field_set.include_requirements.value,
             include_sources=True,
             pex3_venv_create_extra_args=field_set.pex3_venv_create_extra_args,
+            layout=field_set.layout,
             reexported_handler_module=PythonAwsLambdaHandlerField.reexported_handler_module,
         ),
     )
@@ -91,6 +94,7 @@ async def package_python_aws_lambda_layer(
             include_requirements=field_set.include_requirements.value,
             include_sources=field_set.include_sources.value,
             pex3_venv_create_extra_args=field_set.pex3_venv_create_extra_args,
+            layout=field_set.layout,
             # See
             # https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
             #

src/python/pants/backend/awslambda/python/rules_test.py

@@ -384,6 +384,7 @@ def test_pex3_venv_create_extra_args_are_passed_through(
         complete_platforms=Mock(),
         output_path=Mock(),
         environment=Mock(),
+        layout=Mock(),
         **{arg: Mock() for arg in extra_field_set_args},
         pex3_venv_create_extra_args=extra_args_field,
     )

src/python/pants/backend/awslambda/python/target_types.py

@@ -13,6 +13,7 @@
     PythonFaaSDependencies,
     PythonFaaSHandlerField,
     PythonFaaSKnownRuntime,
+    PythonFaaSLayoutField,
     PythonFaaSPex3VenvCreateExtraArgsField,
     PythonFaaSRuntimeField,
 )
@@ -152,6 +153,7 @@ class _AWSLambdaBaseTarget(Target):
         PythonAwsLambdaRuntime,
         PythonFaaSCompletePlatforms,
         PythonFaaSPex3VenvCreateExtraArgsField,
+        PythonFaaSLayoutField,
         PythonResolveField,
         EnvironmentField,
     )

src/python/pants/backend/google_cloud_function/python/rules.py

@@ -15,6 +15,7 @@
 from pants.backend.python.util_rules.faas import (
     BuildPythonFaaSRequest,
     PythonFaaSCompletePlatforms,
+    PythonFaaSLayoutField,
     PythonFaaSPex3VenvCreateExtraArgsField,
 )
 from pants.backend.python.util_rules.faas import rules as faas_rules
@@ -35,6 +36,7 @@ class PythonGoogleCloudFunctionFieldSet(PackageFieldSet):
     runtime: PythonGoogleCloudFunctionRuntime
     complete_platforms: PythonFaaSCompletePlatforms
     pex3_venv_create_extra_args: PythonFaaSPex3VenvCreateExtraArgsField
+    layout: PythonFaaSLayoutField
     type: PythonGoogleCloudFunctionType
     output_path: OutputPathField
     environment: EnvironmentField
@@ -53,6 +55,7 @@ async def package_python_google_cloud_function(
             runtime=field_set.runtime,
             handler=field_set.handler,
             pex3_venv_create_extra_args=field_set.pex3_venv_create_extra_args,
+            layout=field_set.layout,
             output_path=field_set.output_path,
             include_requirements=True,
             include_sources=True,

src/python/pants/backend/google_cloud_function/python/rules_test.py

@@ -260,6 +260,7 @@ def test_pex3_venv_create_extra_args_are_passed_through() -> None:
         output_path=Mock(),
         environment=Mock(),
         pex3_venv_create_extra_args=extra_args_field,
+        layout=Mock(),
     )
 
     observed_calls = []

src/python/pants/backend/google_cloud_function/python/target_types.py

@@ -10,6 +10,7 @@
     PythonFaaSCompletePlatforms,
     PythonFaaSDependencies,
     PythonFaaSHandlerField,
+    PythonFaaSLayoutField,
     PythonFaaSPex3VenvCreateExtraArgsField,
     PythonFaaSRuntimeField,
 )
@@ -119,6 +120,7 @@ class PythonGoogleCloudFunction(Target):
         PythonFaaSCompletePlatforms,
         PythonGoogleCloudFunctionType,
         PythonFaaSPex3VenvCreateExtraArgsField,
+        PythonFaaSLayoutField,
         PythonResolveField,
         EnvironmentField,
     )

src/python/pants/backend/python/util_rules/faas.py

@@ -69,13 +69,27 @@
 logger = logging.getLogger(__name__)
 
 
+class PythonFaaSLayoutField(StringField):
+    alias = "layout"
+    valid_choices = PexVenvLayout
+    expected_type = str
+    default = PexVenvLayout.FLAT_ZIPPED.value
+    help = help_text(
+        """
+        Control the layout of the final artifact: `flat` creates a directory with the
+        source and requirements at the top level, as recommended by cloud vendors,
+        while `flat-zipped` (the default) wraps this up into a single zip file.
+        """
+    )
+
+
 class PythonFaaSPex3VenvCreateExtraArgsField(StringSequenceField):
     alias = "pex3_venv_create_extra_args"
     default = ()
     help = help_text(
         """
         Any extra arguments to pass to the `pex3 venv create` invocation that is used to create the
-        final zip file.
+        final zip file or directory.
 
         For example, `pex3_venv_create_extra_args=["--collisions-ok"]`, if using packages that have
         colliding files that aren't required at runtime (errors like "Encountered collisions
@@ -422,6 +436,7 @@ class BuildPythonFaaSRequest:
     output_path: OutputPathField
     runtime: PythonFaaSRuntimeField
     pex3_venv_create_extra_args: PythonFaaSPex3VenvCreateExtraArgsField
+    layout: PythonFaaSLayoutField
 
     include_requirements: bool
     include_sources: bool
@@ -503,13 +518,17 @@ async def build_python_faas(
 
     pex_result = await Get(Pex, PexFromTargetsRequest, pex_request)
 
-    output_filename = request.output_path.value_or_default(file_ending="zip")
+    layout = PexVenvLayout(request.layout.value)
+
+    output_filename = request.output_path.value_or_default(
+        file_ending="zip" if layout is PexVenvLayout.FLAT_ZIPPED else None
+    )
 
     result = await Get(
         PexVenv,
         PexVenvRequest(
             pex=pex_result,
-            layout=PexVenvLayout.FLAT_ZIPPED,
+            layout=layout,
             platforms=platforms.pex_platforms,
             complete_platforms=platforms.complete_platforms,
             extra_args=request.pex3_venv_create_extra_args.value or (),

src/python/pants/backend/python/util_rules/faas_test.py

@@ -24,6 +24,7 @@
     PythonFaaSHandlerField,
     PythonFaaSHandlerInferenceFieldSet,
     PythonFaaSKnownRuntime,
+    PythonFaaSLayoutField,
     PythonFaaSPex3VenvCreateExtraArgsField,
     PythonFaaSRuntimeField,
     ResolvedPythonFaaSHandler,
@@ -34,7 +35,7 @@
 )
 from pants.backend.python.util_rules.pex import CompletePlatforms, Pex, PexPlatforms
 from pants.backend.python.util_rules.pex_from_targets import PexFromTargetsRequest
-from pants.backend.python.util_rules.pex_venv import PexVenv, PexVenvRequest
+from pants.backend.python.util_rules.pex_venv import PexVenv, PexVenvLayout, PexVenvRequest
 from pants.build_graph.address import Address
 from pants.core.goals.package import OutputPathField
 from pants.core.target_types import FileTarget
@@ -414,6 +415,7 @@ def test_venv_create_extra_args_are_passed_through() -> None:
         output_path=OutputPathField(None, addr),
         runtime=Mock(),
         pex3_venv_create_extra_args=extra_args_field,
+        layout=PythonFaaSLayoutField(PexVenvLayout.FLAT_ZIPPED.value, addr),
         include_requirements=False,
         include_sources=False,
         reexported_handler_module=None,
@@ -454,3 +456,60 @@ def mock_get_pex_venv(request: PexVenvRequest) -> PexVenv:
     # Verify
     assert len(observed_extra_args) == 1
     assert observed_extra_args[0] == extra_args
+
+
+@pytest.mark.parametrize(
+    "input_layout,expected_output",
+    [
+        (PexVenvLayout.FLAT_ZIPPED, "x.zip"),
+        (PexVenvLayout.FLAT, "x"),
+        # PexVenvLayout.VENV is semi-supported: if a user can get it to work, that's fine, but we don't explicitly support it.
+    ],
+)
+def test_layout_should_be_passed_through_and_adjust_filename(input_layout, expected_output) -> None:
+    # Setup
+    addr = Address("x")
+    request = BuildPythonFaaSRequest(
+        address=addr,
+        target_name="x",
+        complete_platforms=Mock(),
+        handler=None,
+        output_path=OutputPathField(None, addr),
+        runtime=Mock(),
+        pex3_venv_create_extra_args=Mock(),
+        layout=input_layout,
+        include_requirements=False,
+        include_sources=False,
+        reexported_handler_module=None,
+    )
+
+    mock_build = Mock()
+
+    # Exercise
+    run_rule_with_mocks(
+        build_python_faas,
+        rule_args=[request],
+        mock_gets=[
+            MockGet(
+                output_type=RuntimePlatforms,
+                input_types=(RuntimePlatformsRequest,),
+                mock=lambda _: RuntimePlatforms(interpreter_version=None),
+            ),
+            MockGet(
+                output_type=ResolvedPythonFaaSHandler,
+                input_types=(ResolvePythonFaaSHandlerRequest,),
+                mock=lambda _: Mock(),
+            ),
+            MockGet(output_type=Digest, input_types=(CreateDigest,), mock=lambda _: EMPTY_DIGEST),
+            MockGet(
+                output_type=Pex,
+                input_types=(PexFromTargetsRequest,),
+                mock=lambda _: Pex(digest=EMPTY_DIGEST, name="pex", python=None),
+            ),
+            MockGet(output_type=PexVenv, input_types=(PexVenvRequest,), mock=mock_build),
+        ],
+    )
+
+    args = mock_build.mock_calls[0].args[0]
+    assert args.layout == input_layout
+    assert args.output_path.name == expected_output