https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • q

    Quinn Romanek

    12/10/2021, 4:03 PM
    hey all - I'm having an issue launching a task execution on my bare metal k8s cluster - when flyteadmin attempts to create a new FlyteWorkflow resource, it fails because the default namespace ({{project}}-{{domain}}) doesn't exist. Have I misconfigured something? I'm wondering how it works in the sandbox
    👋 1
    k
    • 2
    • 34
  • q

    Quinn Romanek

    12/17/2021, 6:50 PM
    how do you handle a task failure in a workflow? can you just use try-catch?
    y
    k
    g
    • 4
    • 10
  • e

    Endre Karlson

    12/21/2021, 4:53 PM
    Does the sandbox cluster support running in a podman rootless cluster?
    k
    y
    • 3
    • 18
  • e

    Eugene Cha

    12/27/2021, 2:43 AM
    Hi all. I originally had flyte working, but had to delete the cluster and reinstall. I passed the deployment stage and set it using this console (https://birdejo.live). Unfortunately the console doesnt seem to work. Any ideas on how to diagnose what the issue is?
    k
    p
    y
    • 4
    • 68
  • k

    Ketan (kumare3)

    12/27/2021, 8:34 PM
    @Niels Bantilan / @Eduardo Apolinario (eapolinario) can you point @Khuyen Tran to an example of flytekit remote
    k
    e
    • 3
    • 6
  • k

    Khuyen Tran

    12/27/2021, 9:30 PM
    So I created a sandbox using:
    flytectl sandbox start --source .
    . After doing that, I can access to Flyte UI. However, after stopping the container, I tried to run
    flytectl sandbox start --source .
    to resume the existing sandbox cluster, but I can't access to Flyte UI. Here is the specific outputs I got:
    ➜ flytectl sandbox start                                       
    🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
    delete existing sandbox cluster [y/n]: n
    Existing details of your sandbox👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉 
    Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable 
    export KUBECONFIG=$KUBECONFIG:/home/khuyen/.kube/config:/home/khuyen/.flyte/k3s/k3s.yaml 
    export FLYTECTL_CONFIG=/home/khuyen/.flyte/config-sandbox.yaml
    I wonder what is the best way to resume the existing cluster?
    h
    y
    • 3
    • 11
  • b

    Brian Lorenz

    02/04/2022, 3:51 PM
    I would like to run sandbox locally on an offline machine. I'm encountering the issue above preventing me from restarting the sandbox. Ultimately I want to be able to run the sandbox offline. Is this supported out of the box or will I need to modify?
    k
    • 2
    • 9
  • r

    Rodrigo Baron

    02/04/2022, 5:30 PM
    I've deployed flyte using kubeadm along with spark-k8s-operator which run successful this example, did try run the example from flytesnacks and got some issues (logs in 🧵). Someone know about
    Non-spark-on-k8s command provided
    ?
    h
    y
    • 3
    • 21
  • j

    Jonas Tischer

    02/22/2022, 6:08 PM
    I’m currently testing out flyte and still have a few issues. In the sandbox UI I can not see any workflows. When I try to register my own workflow with
    python deploy.py
    I get the following error message:
    Serializing Flyte workflows
    Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
    Try 'pyflyte --help' for help.
    
    Error: Invalid value for '-k' / '--pkgs': Illegal package value booster-team for parameter: <Option pkgs>. Expected for the form [a.b.c]
    Registering Flyte workflows
    Error: open flyte-package.tgz: no such file or directory
    What is the best way to diagnose what’s going wrong?
    h
    y
    • 3
    • 3
  • j

    Joseph Curtin

    02/23/2022, 3:34 PM
    Hi, I'm trying to understand concurrency using Flyte. I've implemented this example, • https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/subworkflows.html#example-2 Under which or what circumstances does concurrency occur?
    y
    k
    • 3
    • 5
  • h

    Hafsa Junaid

    03/05/2022, 2:44 AM
    Heyy! I am learning and getting hands-on in flyte. following https://docs.flyte.org/en/latest/getting_started/deploy.html running $flytectl sandbox start --source . Don't know exactly why the services of the sandbox here are remaining in pending state
    h
    k
    • 3
    • 4
  • m

    Matheus Moreno

    03/11/2022, 6:43 PM
    Hey, everyone! I need some help understanding what I'm doing wrong. I'm trying out Flyte using the sandbox and flytectl. I'm following the workflow: build the container, serialize tasks, register the tasks. I'm avoiding fast serialization because I'm using a custom Dockerfile that sets the user as non-root, and apparently that breaks the fast registration system (by the way, is this expected? Will this also happen in a remote cluster?). Now, I just created a new task to test Google Cloud integration, called
    test_google_cloud
    , but the system won't find it: if I try to run it, I'm met with the error
    AttributeError: module 'src.flyte.tasks' has no attribute 'test_google_cloud'
    . I ran
    docker build
    again, serialized the tasks again, registered them again, and still nothing. The task is registered, since I can see it on the UI, but the worker can't find it. What is happening?
    k
    g
    • 3
    • 9
  • e

    Erbene Castro

    03/15/2022, 2:40 PM
    Hello everyone! I'm testing out flyte and running it locally in sandbox mode. I have created a simple task that loads a 2 GB file in memory and prints some useful information about the file. At this point, I'm just loading the file in memory and not doing anything with it. As I try to run the task from the console, my pod gets killed with OOMKilled. How can I increase the resources available for pod created by Flyte? Thank you!
    y
    • 2
    • 2
  • w

    Weide Zhang

    03/22/2022, 12:48 AM
    Hi, I’m new to flyte but I found this project interesting. i was building a training pipeline that uses multiple k8s cluster (one is public aws eks cluster and another is private on-premise k8s cluster). The training dataset (petastorm format) needs to be generated by aws eks and stores in s3 path and copied over to local premise. And then local premise cluster will kick off distributed horovod training consuming the generated dataset (if the dataset already exists (already synced across cluster), no copy is needed). In order to achieve that, what’s the best practice in Flyte ? How many workflows is needed ?
    h
    k
    • 3
    • 4
  • s

    Stephen McGroarty

    03/22/2022, 4:11 AM
    Hey! I'm trying out flyte but I'm running into some issues with authentication. I've set up a new GKE cluster following https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual and then added Oauth2 authentication using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html. I can see that it is working on the browser
    https://${MY_DOMAIN}/console
    however I can't seem to get it to work on the command line, when running a command I get:
    Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"
    My config looks something like (i've tried a few variations):
    admin:
      endpoint: dns:///${MY_DOMAIN}
      authType: Pkce
      insecure: false
      clientId: ${MY_ID}
    logger:
      show-source: true
      level: 0
    Any hints on how to get around this?
    h
    k
    +2
    • 5
    • 23
  • e

    Erbene Castro

    03/25/2022, 2:46 PM
    Hello,
  • e

    Erbene Castro

    03/25/2022, 2:48 PM
    Hello! Does Min.io deployment inside the single container Flyte sandbox need additional setup to get the web interface working? I try to access it through localhost:30084 but it just redirects me to localhost:9001 and shows "can't be reached" error.
    y
    p
    +2
    • 5
    • 7
  • m

    Matheus Moreno

    03/25/2022, 3:56 PM
    Hello, everyone! I have some questions regarding launch plans and scheduling. 1. I'm not sure how I'm supposed to create new launch plans for a project. Do they have to be defined in the same file as the workflows? Can I create a separate file, and Flyte will find/run them? 2. To set a cron schedule to a launch plan, do I necessarily need to define an input on the workflow that will be solely used as the kickoff time? Can't I just define a CronSchedule of monthy execution (for instance), so that the scheduler will run the launch plan a month from now? 3. Is it possible to define scheduled launch plans as YAML files, like it's possible for executions? If so, what fields do I need to define? I feel like saving the launch plans as YAML files will result in a more organized code.
    y
    • 2
    • 2
  • d

    dowstreet

    03/25/2022, 6:32 PM
    Hi All - I've been doing a little testing with a sandbox install, and would like to try installing something closer to 'production', but on an on-prem k8s cluster. I'm a little confused as to the recommended approach here - would I start with the sandbox helm charts, and modify those for my environment? (The README in flyte/charts/flyte seems to indicate that these are sandbox deployments) Thanks!
    y
    • 2
    • 3
  • m

    Matheus Moreno

    03/25/2022, 8:06 PM
    Hi, everyone! Another question: is it possible to start the sandbox without using flytectl? A member of my team is using Windows and wants to try Flyte. I was trying to accomplish this by starting the sandbox with Docker Compose, and having another container to communicate with it. I was able to run simple commands such as
    flytectl create
    and
    flytectl get
    from the container, but I can't build images, and I think the problem is on the sandbox side, since I can't do that even on my host machine. I get the error:
    OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "docker": executable file not found in $PATH: unknown
    This is the configuration for the sandbox. What am I missing?
    services:
      flyte-sandbox:
        image: <http://cr.flyte.org/flyteorg/flyte-sandbox:latest|cr.flyte.org/flyteorg/flyte-sandbox:latest>
        container_name: flyte-sandbox
        privileged: true
        volumes:
          - ..:/usr/src           # The docker-compose file is inside a sandbox/ dir
        ports:
          - "30081:30081"         # Console at <http://localhost:30081/console>
          - "30084:30084"
          - "30088:30088"
    s
    y
    • 3
    • 7
  • j

    Johnson Huynh

    03/30/2022, 7:10 PM
    Hi guys, I am trying to deploy Flyte to our Kubernetes stack but it seems like our FlyteScheduler deployment does not seem to want to start. I keep noticing this error in the Flyte Scheduler logs.
    Error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized
    
    Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
    This is the log from FlyteAdmin logs:
    {
      "json": {
        "src": "token.go:37"
      },
      "level": "info",
      "msg": "Error occurred in NewAccessRequest: invalid_client",
      "ts": "2022-03-30T19:38:33Z"
    }
    My Flyte Scheduler configmap is listed below. Am I perhaps missing a step?
    data:
      admin.yaml: |
        admin:
          clientId: ExternalODICclientID
          clientSecretLocation: /etc/secrets/flyte-secret-auth
          endpoint: flyteadmin.flyte.svc:81
          insecure: true
        event:
          capacity: 1000
          rate: 500
          type: admin
      db.yaml: |
        database:
          dbname: postgres
          host: 'postgres-postgresql.flyte.svc'
          passwordPath: /etc/db/pass.txt
          port: 5432
          username: xxxx
      logger.yaml: |
        logger:
          level: 4
          show-source: true
      server.yaml: |
        scheduler:
          metricsScope: 'flyte:'
          profilerPort: 10254
    This is our Flyte Admin configmap:
    auth:
      appAuth:
        openId:
          baseUrl: <https://ExternalODICbaseurl.com>
          clientId: OurODICClientID
          scopes:
          - profile
          - openid
          - email
        thirdPartyConfig:
          flyteClient:
            clientId: ExternalODICclientID
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://our.domain.com>
      userAuth:
        openId:
          baseUrl: <https://ExternalODICbaseurl.com>
          clientId: ExternalODICclientID
          scopes:
          - profile
          - openid
          - email
      authorizedUris:
      - <https://our.domain.com>
    k
    y
    h
    • 4
    • 23
  • m

    Matheus Moreno

    03/31/2022, 2:32 PM
    Hey, everyone! I was wondering: what is the best approach to deal with persistent files on Flyte tasks/workflows? For instance, what if I train a model and I want to save this model on my Flyte server to be used every time a certain task is called, is it possible? In this example on the User Guide, the data returned is a saved file, but will it be persistent? And can it have a human-friendly name, or will it be random letters and numbers, like how it is with DataFrames outputs?
  • t

    Tiansu Yu

    04/07/2022, 9:39 AM
    Hi, everyone. I have a naive question: How do I load a file from a GCS bucket inside a spark_task? Is there some idiomatic ways to do it in Flyte? (Normally I would add some maven coordinates to my spark config, but I noticed that this does not work for Flyte in a cluster where spark images has to be pre-built.)
    y
    s
    k
    • 4
    • 28
  • h

    Henri Palacci

    04/10/2022, 1:48 AM
    HI all! After playing around with the sandbox I'm trying to set up flyte for my team so they can try it out as well. We're on EKS so I followed the AWS guide and got everything working without auth. Then I followed the auth setup guide and got the console behind Google IdP OAuth (still using self-signed certs at this stage). However, I can't seem to get
    flytectl
    to work now - I get redirected to
    <http://localhost:53593/callback>
    with the error message:
    Flyte Authentication
    Couldn't get access token due to error: Post "<https://flyte>.<mydomain.com>:443/oauth2/token": x509: "flyte.<mydomain.com>" certificate is not standards compliant
    I also set
    insecureSkipVerify: true
    in
    ~/.flyte/config.yaml
    but to no avail.
    k
    h
    +2
    • 5
    • 10
  • k

    Katrina P

    04/14/2022, 4:01 PM
    Hello! I got a sandbox up and running locally and have been trying out all the examples in the cookbook. I've gone through the Spark plugin setup here and have registered the latest version of the plugin example (0.3.66) but I've run into some issues with executing them as-is and the spark context can't be initialized. I re-ran through all the steps on the plugin install page again, but I'm wondering if something needs to be changed in the configuration, restarted, or if the issue is in the container from the registered workflows. Any guidance on how to best debug would be very much appreciated!
    Traceback (most recent call last):
    
          File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
            return wrapped(*args, **kwargs)
          File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 464, in dispatch_execute
            new_user_params = self.pre_execute(ctx.user_space_params)
          File "/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py", line 123, in pre_execute
            self.sess = sess_builder.getOrCreate()
          File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
            sc = SparkContext.getOrCreate(sparkConf)
          File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 392, in getOrCreate
            SparkContext(conf=conf or SparkConf())
          File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 146, in __init__
            self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
          File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 209, in _do_init
            self._jsc = jsc or self._initialize_context(self._conf._jconf)
          File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 329, in _initialize_context
            return self._jvm.JavaSparkContext(jconf)
          File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1585, in __call__
            return_value = get_return_value(
          File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
            raise Py4JError(
    
    Message:
    
        An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext
    
    SYSTEM ERROR! Contact platform administrators.
    The full log from the pod:
    ++ id -u
    + myuid=0
    ++ id -g
    + mygid=0
    + set +e
    ++ getent passwd 0
    + uidentry=root:x:0:0:root:/root:/bin/bash
    + set -e
    + '[' -z root:x:0:0:root:/root:/bin/bash ']'
    + SPARK_CLASSPATH=':/opt/spark/jars/*'
    + env
    + grep SPARK_JAVA_OPT_
    + sed 's/[^=]*=\(.*\)/\1/g'
    + sort -t_ -k4 -n
    + readarray -t SPARK_EXECUTOR_JAVA_OPTS
    + '[' -n '' ']'
    + '[' '' == 2 ']'
    + '[' '' == 3 ']'
    + '[' -n '' ']'
    + '[' -z ']'
    + case "$1" in
    + echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
    + CMD=("$@")
    + exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/1> --raw-output-data-prefix <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1> --checkpoint-path <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1/_flytecheckpoints> --prev-checkpoint <s3://my-s3-bucket/vc/nw0xm6ntwz-n0-0/_flytecheckpoints> --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
    Non-spark-on-k8s command provided, proceeding in pass-through mode...
    22/04/14 15:19:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    ERROR:root:Exception while sending command.
    Traceback (most recent call last):
      File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 480, in send_command
        raise Py4JNetworkError("Answer from Java side is empty")
    py4j.protocol.Py4JNetworkError: Answer from Java side is empty
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1038, in send_command
        response = connection.send_command(command)
      File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 503, in send_command
        raise Py4JNetworkError(
    py4j.protocol.Py4JNetworkError: Error while sending or receiving
    {"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
    {"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 464, in dispatch_execute\n        new_user_params = self.pre_execute(ctx.user_space_params)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 123, in pre_execute\n        self.sess = sess_builder.getOrCreate()\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n        sc = SparkContext.getOrCreate(sparkConf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n        SparkContext(conf=conf or SparkConf())\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 146, in __init__\n        self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 209, in _do_init\n        self._jsc = jsc or self._initialize_context(self._conf._jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 329, in _initialize_context\n        return self._jvm.JavaSparkContext(jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1585, in __call__\n        return_value = get_return_value(\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/protocol.py\", line 334, in get_return_value\n        raise Py4JError(\n\nMessage:\n\n    An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext\n\nSYSTEM ERROR! Contact platform administrators."}
    {"asctime": "2022-04-14 15:20:06,730", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}
    s
    k
    • 3
    • 8
  • y

    Yee

    04/15/2022, 5:51 PM
    @jeev when you get a chance - mind giving the new getting started guide a whirl? would like to see what you think ofit
    👍 1
    j
    • 2
    • 3
  • s

    Sebastian

    04/20/2022, 9:17 AM
    Hi! I'm porting some stuff to Flyte to see how it compares to our current solution, but I get stuck at a very simple thing. I need to do datetime arithmetic in numpy, so how to I pass datetype parameters to a workflow? np.datetime64 is not supported and trying to parse a string errors. Thanks!
    import numpy as np
    from flytekit import workflow
    
    @workflow
    def wf(date: str = "2022-01-01"):
        date = np.datetime64(date) # this errors
    s
    n
    • 3
    • 7
  • s

    Sören Brunk

    04/20/2022, 12:25 PM
    You should be able to use a python datetime as input type: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/flyte_python_types.html Also the actual processing needs to be done inside a task. Workflows are basically a DSL to compose tasks. I.e. something like this (not tested):
    from datetime import datetime
    @task
    def t(date: datetime = datetime(2022,1,1)):
        date = np.datetime64(date)
        ...
    
    @workflows
    def wf(date: datetime):
        t(date=date)
  • h

    Han

    04/20/2022, 5:21 PM
    Searching for
    flyte run workflow
    on Google, the first result is a 404 page on the documentation site.
    k
    y
    • 3
    • 5
  • c

    caioau

    04/20/2022, 6:18 PM
    Hello! I'm starting with flyte, I'm trying to understand how flyte isolates resources, especially how to define the domains and their resourceQuotas, can I change it on the fly? For now, I believe this is defined in the helm values and cannot be changed without changing the helm, is this true? or is there another way?
    k
    k
    • 3
    • 4
Powered by Linen
Title
c

caioau

04/20/2022, 6:18 PM
Hello! I'm starting with flyte, I'm trying to understand how flyte isolates resources, especially how to define the domains and their resourceQuotas, can I change it on the fly? For now, I believe this is defined in the helm values and cannot be changed without changing the helm, is this true? or is there another way?
k

Ketan (kumare3)

04/20/2022, 6:56 PM
cc @katrina
k

katrina

04/20/2022, 7:08 PM
hey @caioau you can absolutely change them on the fly and flyte was built with first-class support for just this use-case. The values defined in helm are the defaults that get applied to every project, but you can set-per project overrides by following the instructions here: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#cluster-resources
✅ 1
if you're interested in changing these default values across all projects we don't have a programmatic api for that just yet, so you will need to update your config and roll your deployment
c

caioau

04/20/2022, 7:14 PM
thanks very much @katrina!
View count: 10