cuddly-jelly-27016
08/22/2025, 7:46 AMcustom_config in Spark to provide sensible defaults for mainApplicationFile and executorPath when not explicitly set, using the serialization context to determine these paths. This ensures more robust and predictable Spark job configuration.
• Refactored the SerializationContext class to introduce an interpreter_path field and improved the logic for determining the entrypoint path, making path resolution clearer and more flexible.
### Testing enhancements
• Added new tests in plugins/spark/tests/test_task.py to cover Spark plugin initialization, pre-cluster setup, and custom configuration logic, increasing test coverage and reliability of Spark-related code.
flyteorg/flyte-sdkcuddly-jelly-27016
08/22/2025, 9:28 PM