https://flyte.org
Join Slack
<@U016RNZDZRS> Propeller workload pod is still crashing. Annotation with a namespace is done. Also, ...
# announcements
c
@great-school-54368 Propeller workload pod is still crashing. Annotation with a namespace is done. Also, the actual problem statement is still there. As I need to access external IP for UI services. @thankful-minister-83577
g
Nice, Can you paste your propeller logs ?
c
message has been deleted
g
Cool, I saw this before. In your helm values file remove spark from here https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-gcp.yaml#L279-L288 and then re deploy the application
cc: @hallowed-mouse-14616 Any idea ? What is the issue with spark plugin ?
c
Re-deployed! After commenting
message has been deleted
g
what is the logs ?
c
1. enabled-plugins: 2. - container 3. - sidecar 4. # - spark 5. - k8s-array 6. default-for-task-types: 7. container: container 8. sidecar: sidecar 9. spark: spark 10. container_array: k8s-array Line # 4 is the required change right?
g
You also need to comment out line 9
c
It's done. Thanks, @great-school-54368 But the external IP is still not accessible.
🎉 1
g
Can you post the ingress spec ?
Did you setup the DNS ?
Also paste the output of 
kubectl get ManagedCertificate
, Did you install the cert manager ?
c
I have the domain name but mapping IP to requires external IP but we don't have that for flyteconsole. (as per my understanding)
message has been deleted
I created the SslCert client certificate
for database instance
g
do you think maybe we can discuss this on a google meet ?
Looks like your certificates are wrong
Copy code
➜  ~ curl -L --header "Host: <http://flyte.openaimp.com|flyte.openaimp.com>"  <https://34.70.204.255> -v
*   Trying 34.70.204.255:443...
* Connected to 34.70.204.255 (34.70.204.255) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I also didn’t find any A record for your dns
c
Sure we can meet, I really need to understand this.
g
@high-park-82026 Do we need any change in spark config for GCP ?
Copy code
sparkoperator:
  enabled: true
  plugin_config:
    plugins:
      spark:
        # -- Spark default configuration
        spark-config-default:
          # We override the default credentials chain provider for Hadoop so that
          # it can use the serviceAccount based IAM role or ec2 metadata based.
          # This is more in line with how AWS works
          - spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
          - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2"
          - spark.kubernetes.allocation.batch.size: "50"
          - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl"
          - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
          - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A"
          - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
          - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A"
          - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
          - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
          - spark.hadoop.fs.s3a.multipart.threshold: "536870912"
          - spark.blacklist.enabled: "true"
          - spark.blacklist.timeout: "5m"
          - spark.task.maxfailures: "8"
b
Google managed certificates, 
kubectl get <http://managedcertificates.networking.gke.io|managedcertificates.networking.gke.io>
only work with the GKE ingress. If you're using another ingress controller (like nginx, contour) with cert-manager you will have a different type of certificate: 
kubectl get <http://certificates.cert-manager.io|certificates.cert-manager.io>
🙌 1
169 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by