https://flyte.org
Join Slack
Hey team is there a way to import my own modules in imagespec? or is can I only include pypi package...
# flyte-support
c
Hey team is there a way to import my own modules in imagespec? or is can I only include pypi packages
d
You can use gitsha such as this. (Just an overview)
Copy code
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@62bd03b6eab075470e17932a268fc566f643fc8e#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
a
you can also set 
pip_index
on 
ImageSpec
, we use AWS CodeArtifact and it works well
c
@damp-lion-88352 how do I get the git sha for a py file in a repo? @abundant-laptop-64153 do you have an example of how this is done?
a
Copy code
image_spec = ImageSpec(
    # need to aws codeartifact login first, we use direnv to set CODEARTIFACT_AUTH_TOKEN locally to pass into the image
    pip_index="<https://aws>:{0}@DOMAIN-ACCOUNT.d.codeartifact.REGION.amazonaws.com/pypi/REPOSITORY/simple/".format(
        os.environ["CODEARTIFACT_AUTH_TOKEN"]
    ),
    requirements="requirements.txt",
    # we push to ECR so docker login is required prior to running, I'd recommend <https://github.com/awslabs/amazon-ecr-credential-helper>
    registry="<http://ACCOUNT.dkr.ecr.REGION.amazonaws.com|ACCOUNT.dkr.ecr.REGION.amazonaws.com>",
)
d
c
Copy code
#10 0.731 Collecting git+<https://github.com/blabla/bla.git@606c967f#subdirectory=flyteingest/plugins/mm-tokenizer>
#10 0.732   Cloning <https://github.com/blabla/bla.git> (to revision 606c967f) to /tmp/pip-req-build-by17ht1e
#10 0.737   Running command git clone --filter=blob:none --quiet <https://github.com/blabla/bla.git> /tmp/pip-req-build-by17ht1e
#10 1.505   fatal: could not read Username for '<https://github.com>': No such device or address
#10 1.520   error: subprocess-exited-with-error
@damp-lion-88352 when running 
pyflyte run --remote ..
getting a could not read Username error .. I've followed some of the solutions here https://stackoverflow.com/questions/22147574/github-fatal-could-not-read-username-for-https-github-com-no-such-file-o but cant seem to work it out Did you do something additional beyond this?
10 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by