Evan Sadler
03/08/2023, 5:08 PMpyflyte register
to pick up environment variables when I set them like this?
@task(
task_config=Databricks(databricks_token=os.environ.get("DATABRICKS_TOKEN"))
)
def etl():
Ketan (kumare3)
Evan Sadler
03/08/2023, 5:46 PMenvironment
options for @dynamic. Setting it in @dynamic
seemed to propagate the env variable to the inner task.
This worked great!
@task(
task_config=Databricks(
databricks_conf={
"run_name": "rfy_etl",
"existing_cluster_id": "0302-183318-vi1yrbkk"
},
databricks_token=os.environ.get("DATABRICKS_TOKEN"),
),
)
def test_task() -> int:
return 10
@dynamic(environment={"DATABRICKS_TOKEN": os.environ.get("DATABRICKS_TOKEN")})
def wrapper() -> int:
return test_task()
@workflow
def wf() -> int:
return wrapper()
Ketan (kumare3)
Evan Sadler
03/09/2023, 3:59 PM