<#115 Add a default interpreter_path and entrypoin...
# flyte-github
c
#115 Add a default interpreter_path and entrypoint_path Pull request opened by pingsutw This pull request introduces improvements and fixes for Spark task configuration, testing, and code organization. The most significant changes include enhancements to the Spark plugin's configuration logic, the addition of comprehensive tests for Spark tasks, and refactoring for better code clarity and maintainability. ### Spark plugin configuration improvements • Updated
custom_config
in
Spark
to provide sensible defaults for
mainApplicationFile
and
executorPath
when not explicitly set, using the serialization context to determine these paths. This ensures more robust and predictable Spark job configuration. • Refactored the
SerializationContext
class to introduce an
interpreter_path
field and improved the logic for determining the entrypoint path, making path resolution clearer and more flexible. ### Testing enhancements • Added new tests in
plugins/spark/tests/test_task.py
to cover Spark plugin initialization, pre-cluster setup, and custom configuration logic, increasing test coverage and reliability of Spark-related code. flyteorg/flyte-sdk