https://flyte.org
Join Slack
Is it possible for `pyflyte register` to pick up environment variables when I set them like this? `...
# flyte-support
a
Is it possible for 
pyflyte register
to pick up environment variables when I set them like this?
Copy code
@task(
    task_config=Databricks(databricks_token=os.environ.get("DATABRICKS_TOKEN"))
)
def etl():
f
absolutely
is it not picking it up ?
a
Great to know it should pick up on them. I thought it was picking up and then it stopped! I will debug on my end. Thanks for the help.
I believe the reason that flyte stopped picking up the environment variables is because I started using a @dynamic workflow. I just tested 2 versions of the workflow: 1. Not using a dynamic workflow 2. Wrapping the task in a dynamic workflow Scenario 1 worked and scenario 2 failed. I searched the docs and found the 
environment
options for @dynamic. Setting it in 
@dynamic
seemed to propagate the env variable to the inner task. This worked great!
Copy code
@task(
    task_config=Databricks(
        databricks_conf={
            "run_name": "rfy_etl",
            "existing_cluster_id": "0302-183318-vi1yrbkk"
        },
        databricks_token=os.environ.get("DATABRICKS_TOKEN"),
    ),
)
def test_task() -> int:
    return 10


@dynamic(environment={"DATABRICKS_TOKEN": os.environ.get("DATABRICKS_TOKEN")})
def wrapper() -> int:
    return test_task()

@workflow
def wf() -> int:
    return wrapper()
f
Ya this is interesting- Flyte does not capture env vars by default
Maybe in environment setting we can use delegate. But this is potentially dangerous. As you may have secrets etc
a
I think it works great as is, I just needed to a do a bit more research.
Thanks for your help!!!!!!
🔥 1
151 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by