Apologies if I'm missing something in the docs, bu...
# ask-the-community
d
Apologies if I'm missing something in the docs, but I'm trying to do a MVP with a custom docker container task. What I've tried: 1.
flytectl demo start
2. Build the container with the tag
localhost:30000/custom_container:v1
3. exported the config
export FLYTECTL_CONFIG=/home/user/.flyte/config-sandbox.yaml
4. Tried to run the workflow
pyflyte run --remote workflows/mvp.py mvp --input_s3_path="<s3://data/path>"
The error I get is that it cannot pull the image. Am a I missing something very basic here?
n
hi Dan! just to double check, but in step (2) you’re doing
docker push localhost:30000/custom_container:v1
correct?
d
maybe that's my idiot mistake...
give me a sec to try that
n
and you’ll have to reference the image in
pyflyte run
as well, with the
--image
flag
d
the image tag is going to make it the default image, correct?
n
right, using
pyflyte run --remote --image localhost:30000/custom_container:v1 ...
makes the workflow run with that image
d
perfect. Yes, I was trying that and also doing it at the task level.
n
whereas
docker pushlocalhost:30000/…
should make it available in the local Flyte cluster docker registry
d
I was forgetting to push. Somehow I missed the fact that it was a registry and that I was going to need to push it.
bravo for setting this up so nicely. This is super slick.
n
thanks Dan! glad it’s been (fairly) smooth so far 😉 feel free to keep this thread open if you run into anything else
d
Now I'm happily running into errors that I expected from my end 🙂
Honestly, this is exceptionally smooth so far. Waiting to find the brick wall that we've run into with other frameworks.
n
we’re happy to help out if you do! 🧱
d
ok, maybe another simple question. Is there a way to remove the
_FSEC_
prefix from the environment variables? It seems like there should be a way to set it for this config (https://github.com/flyteorg/flytekit/blob/9d313429c577a919ec0ad4cd397a5db356a1df0d/flytekit/configuration/internal.py#L141-L159) object but I'm not sure where it should be set. Is that in config-sandbox.yaml?
n
are you trying to access secrets within a task?
d
yes
I've gotten it to work but the secret name is prefixed with
_fsec_
That throws off our configuration tool that is looking for them without the prefix. I can change it within the code but I wasn't sure if there was an easy way to modify it within flyte.
n
hmm, can you try
FLYTE_SECRET_ENV_PREFIX=""
? (docs here)
d
Where would I put that?
n
you can
export
it before running pyflyte run
d
Gave that a shot. No luck. I realized I'd still have to do something on our end because of the group. FWIW, I totally understand why the variables are named the way they are and think it makes a lot of sense.
n
@Eduardo Apolinario (eapolinario) @Kevin Su there should be a way to override the
FLYTE_SECRET_ENV_PREFIX
configuration right?
e
unfortunately, this is not possible today.
Another user reached out some time ago asking about that, here's the issue: https://github.com/flyteorg/flyte/issues/3053
n
is there any recourse or workaround for this? @Dan Corbiani is this a blocker for you, or are there workarounds on your side you can implement?
d
totally not a blocker. I can make this work on our side.
n
also, I may not have full context here, but Flyte does come with a way to inject secrets into your tasks: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#how-to-use-secrets-injection-in-a-task Which you can access via a flytekit-specific API:
Copy code
@task(
    secret_requests=[
        Secret(key=USERNAME_SECRET, group=SECRET_GROUP),
        Secret(key=PASSWORD_SECRET, group=SECRET_GROUP),
    ]
)
def user_info_task() -> Tuple[str, str]:
    secret_username = flytekit.current_context().secrets.get(
        SECRET_GROUP, USERNAME_SECRET
    )
    secret_pwd = flytekit.current_context().secrets.get(SECRET_GROUP, PASSWORD_SECRET)
    # use secrets here
d
Saw I didn't reply to this. That was what we were using. We were running into some issues since certain tools in our workflow require environment variables to be set. We were able to pretty quickly get all of the pieces fit together and basically used something similar to what you posted.
n
Awesome @Dan Corbiani! glad you were able to get things working.
106 Views