Is it possible for `pyflyte register` to pick up e...
# ask-the-communitye
Evan Sadler
03/08/2023, 5:08 PMIs it possible for
pyflyte register Copy code
@task(
task_config=Databricks(databricks_token=os.environ.get("DATABRICKS_TOKEN"))
)
def etl():k
Ketan (kumare3)
03/08/2023, 5:20 PM
absolutely
Ketan (kumare3)
03/08/2023, 5:20 PM
is it not picking it up
?
e
Evan Sadler
03/08/2023, 5:46 PMGreat to know it should pick up on them. I thought it was picking up and then it stopped! I will debug on my end. Thanks for the help.
Evan Sadler
03/09/2023, 3:49 PMI believe the reason that flyte stopped picking up the environment variables is because I started using a @dynamic workflow. I just tested 2 versions of the workflow:
1. Not using a dynamic workflow
2. Wrapping the task in a dynamic workflow
Scenario 1 worked and scenario 2 failed. I searched the docs and found the
environment@dynamic Copy code
@task(
task_config=Databricks(
databricks_conf={
"run_name": "rfy_etl",
"existing_cluster_id": "0302-183318-vi1yrbkk"
},
databricks_token=os.environ.get("DATABRICKS_TOKEN"),
),
)
def test_task() -> int:
return 10
@dynamic(environment={"DATABRICKS_TOKEN": os.environ.get("DATABRICKS_TOKEN")})
def wrapper() -> int:
return test_task()
@workflow
def wf() -> int:
return wrapper()k
Ketan (kumare3)
03/09/2023, 3:53 PM
Ya this is interesting- Flyte does not capture env vars by default
Ketan (kumare3)
03/09/2023, 3:54 PM
Maybe in environment setting we can use delegate. But this is potentially dangerous. As you may have secrets etc
e
Evan Sadler
03/09/2023, 3:59 PMI think it works great as is, I just needed to a do a bit more research.
Evan Sadler
03/09/2023, 4:00 PMThanks for your help!!!!!!
150 Views