happy-bird-19790
01/27/2022, 9:03 PMflyte-prod-tasks-logs
and updated the values-eks.yaml
file accordingly. In opta/aws/flyte.yaml
I added the following.
task_logs:
plugins:
logs:
cloudwatch-enabled: true
cloudwatch-log-group: flyte-prod-tasks-logs
cloudwatch-region: "{vars.region}"
My workflow runs successfully but I don't see anything in the log group. My ShellTask prints to standard out and I have a PythonTask that uses logging. Is there anything special I need to do? Or some configuration I missed?happy-bird-19790
01/27/2022, 9:07 PM> opta version
v0.22.1
> terraform --version
Terraform v1.0.11
on linux_amd64
freezing-airport-6809
freezing-airport-6809
kubectl get -n cm flyte-propeller-config -o yaml
freezing-airport-6809
happy-bird-19790
01/27/2022, 9:34 PMapiVersion: v1
data:
admin.yaml: |
admin:
clientId: 'flytepropeller'
clientSecretLocation: /etc/secrets/client_secret
endpoint: flyteadmin:81
insecure: true
event:
capacity: 1000
rate: 500
type: admin
cache.yaml: |
cache:
max_size_mbs: 1024
target_gc_percent: 70
catalog.yaml: |
catalog-cache:
endpoint: datacatalog:89
insecure: true
type: datacatalog
copilot.yaml: |
plugins:
k8s:
co-pilot:
image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.24|cr.flyte.org/flyteorg/flytecopilot:v0.0.24>
name: flyte-copilot-
start-timeout: 30s
core.yaml: |
manager:
pod-application: flytepropeller
pod-template-container-name: flytepropeller
pod-template-name: flytepropeller-template
propeller:
downstream-eval-duration: 30s
enable-admin-launcher: true
gc-interval: 12h
kube-client-config:
burst: 25
qps: 100
timeout: 30s
leader-election:
enabled: true
lease-duration: 15s
lock-config-map:
name: propeller-leader
namespace: flyte
renew-deadline: 10s
retry-period: 2s
limit-namespace: all
max-workflow-retries: 50
metadata-prefix: metadata/propeller
metrics-prefix: flyte
prof-port: 10254
queue:
batch-size: -1
batching-interval: 2s
queue:
base-delay: 5s
capacity: 1000
max-delay: 120s
rate: 100
type: maxof
sub-queue:
capacity: 1000
rate: 100
type: bucket
type: batch
rawoutput-prefix: <s3://flyte-prod-service-flyte>
workers: 40
workflow-reeval-duration: 30s
webhook:
certDir: /etc/webhook/certs
serviceName: flyte-pod-webhook
enabled_plugins.yaml: |
tasks:
task-plugins:
default-for-task-types:
container: container
container_array: k8s-array
sidecar: sidecar
enabled-plugins:
- container
- sidecar
- k8s-array
k8s.yaml: |
plugins:
k8s:
default-cpus: 100m
default-env-vars: []
default-memory: 100Mi
logger.yaml: |
logger:
level: 5
show-source: true
resource_manager.yaml: |
propeller:
resourcemanager:
type: noop
storage.yaml: |
storage:
type: s3
container: "flyte-prod-service-flyte"
connection:
auth-type: iam
region: us-west-2
limits:
maxDownloadMBs: 10
task_logs.yaml: |
plugins:
logs:
cloudwatch-enabled: true
cloudwatch-log-group: flyte-prod-tasks-logs
cloudwatch-region: us-west-2
kubernetes-enabled: false
kind: ConfigMap
metadata:
annotations:
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: service-flyte-helmchart
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
creationTimestamp: "2022-01-26T19:57:43Z"
labels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: service-flyte-helmchart
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
<http://app.kubernetes.io/name|app.kubernetes.io/name>: flyteadmin
<http://helm.sh/chart|helm.sh/chart>: flyte-core-v0.1.10
name: flyte-propeller-config
namespace: flyte
resourceVersion: "8963"
selfLink: /api/v1/namespaces/flyte/configmaps/flyte-propeller-config
uid: 3fdd593b-ee78-476b-84fc-8c8a8c9b3502
freezing-airport-6809
high-park-82026
task_logs.yaml: |
plugins:
logs:
cloudwatch-enabled: true
cloudwatch-log-group: flyte-prod-tasks-logs
cloudwatch-region: us-west-2
kubernetes-enabled: false
k8s-array:
logs:
config:
cloudwatch-enabled: true
cloudwatch-log-group: flyte-prod-tasks-logs
cloudwatch-region: us-west-2
kubernetes-enabled: false
great-school-54368
01/28/2022, 2:48 PMhappy-bird-19790
01/28/2022, 4:43 PM@task
def get_uuid() -> str:
id = str(uuid.uuid4())
<http://logging.info|logging.info>(f'got new id {id}')
print(f'got new id {id}')
return id
freezing-airport-6809
happy-bird-19790
01/28/2022, 5:06 PMfreezing-airport-6809
happy-bird-19790
01/28/2022, 5:17 PMfreezing-airport-6809
freezing-airport-6809
high-park-82026
freezing-airport-6809
high-park-82026
freezing-airport-6809
freezing-airport-6809
most-sunset-30029
01/28/2022, 5:19 PMmost-sunset-30029
01/28/2022, 5:19 PMmost-sunset-30029
01/28/2022, 5:19 PMhigh-park-82026
Every organization potentially uses different log aggregators, making it hard to create a one-size-fits-all solution. Some examples of the log aggregators include cloud-hosted solutions like AWS CloudWatch, GCP Stackdriver, Splunk, Datadog, etc.
Flyte does not have an opinion here and provides a simplified interface to configure your log provider. Flyte-sandbox ships with the Kubernetes dashboard to visualize the logs. This may not be safe for production; hence we recommend users explore other log aggregators.
high-park-82026
happy-bird-19790
01/28/2022, 5:21 PMhappy-bird-19790
01/28/2022, 5:21 PMmost-sunset-30029
01/28/2022, 5:22 PMhappy-bird-19790
01/28/2022, 6:07 PMhappy-bird-19790
01/28/2022, 6:31 PMfreezing-airport-6809
high-park-82026
blue-eve-64591
01/29/2022, 1:27 AMblue-eve-64591
01/29/2022, 1:30 AMhigh-park-82026
kubectl get pods -n <namespace>
Should list you all the pods in, say, flyte-snacks namespace…
Unless nodes are deeply nested, the pod names will have the structure of:
<execId>-n0-0
Where <execId>
is the flyte execution id
and n0
is the node id (unless you explicitly name it)
and 0
is the retry attempt
Then:
kubectl logs -n <namespace> <pod name>
Or to watch logs as they come:
kubectl logs -n <namespace> <pod name> --follow
most-sunset-30029
01/29/2022, 1:36 AMhappy-bird-19790
01/31/2022, 6:10 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
happy-bird-19790
01/31/2022, 7:38 PMhappy-bird-19790
01/31/2022, 7:46 PMfreezing-airport-6809