[SPARK-50613][CONNECT] Support Configurable Idle Timeout of Connect Server

[soumasish]

What changes were proposed in this pull request?

This PR introduces a new configurable idle timeout feature for the Spark Connect server. Specifically, it adds a configuration option (spark.connect.server.idleTimeout) that allows administrators to specify a duration after which the Spark Connect server will automatically shut down if no new sessions or requests are received. By default, this value is set to 0 (disabled), preserving current behavior.
When enabled (i.e., set to a positive value), the Spark Connect service periodically checks for activity based on recorded session access times. If the server remains idle beyond the configured timeout, it triggers a graceful shutdown of the service.

Why are the changes needed?

Currently, the Spark Connect server remains running indefinitely until manually terminated. In managed environments (e.g., Amazon EMR), the lack of a natural stopping condition means the cluster may never be marked as idle and thus never shut down, leading to unnecessary resource consumption and cost.
By introducing this idle timeout:

1. Clusters can automatically terminate when no active Spark Connect sessions are ongoing, leading to better resource utilization.
2. Administrators have more control over lifecycle management, improving operational efficiency in hosted or on-demand environments.

Does this PR introduce any user-facing change?

Yes. A new configuration parameter spark.connect.server.idleTimeout is introduced. When set to a positive duration (e.g., 10m), the Spark Connect server will monitor inactivity and stop after the specified period with no requests or active sessions.
By default, it is disabled (0).

How was this patch tested?

• Added unit tests for PySpark

Was this patch authored or co-authored using generative AI tooling?

No