https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
e

Evan Sadler

03/08/2023, 5:08 PM
Is it possible for 
pyflyte register
to pick up environment variables when I set them like this?
Copy code
@task(
    task_config=Databricks(databricks_token=os.environ.get("DATABRICKS_TOKEN"))
)
def etl():
k

Ketan (kumare3)

03/08/2023, 5:20 PM
absolutely
is it not picking it up ?
e

Evan Sadler

03/08/2023, 5:46 PM
Great to know it should pick up on them. I thought it was picking up and then it stopped! I will debug on my end. Thanks for the help.
I believe the reason that flyte stopped picking up the environment variables is because I started using a @dynamic workflow. I just tested 2 versions of the workflow: 1. Not using a dynamic workflow 2. Wrapping the task in a dynamic workflow Scenario 1 worked and scenario 2 failed. I searched the docs and found the 
environment
options for @dynamic. Setting it in 
@dynamic
seemed to propagate the env variable to the inner task. This worked great!
Copy code
@task(
    task_config=Databricks(
        databricks_conf={
            "run_name": "rfy_etl",
            "existing_cluster_id": "0302-183318-vi1yrbkk"
        },
        databricks_token=os.environ.get("DATABRICKS_TOKEN"),
    ),
)
def test_task() -> int:
    return 10


@dynamic(environment={"DATABRICKS_TOKEN": os.environ.get("DATABRICKS_TOKEN")})
def wrapper() -> int:
    return test_task()

@workflow
def wf() -> int:
    return wrapper()
k

Ketan (kumare3)

03/09/2023, 3:53 PM
Ya this is interesting- Flyte does not capture env vars by default
Maybe in environment setting we can use delegate. But this is potentially dangerous. As you may have secrets etc
e

Evan Sadler

03/09/2023, 3:59 PM
I think it works great as is, I just needed to a do a bit more research.
Thanks for your help!!!!!!
Powered by