You may add further methods by overriding the relevant class methods. (message Body filtered using a JSONPath expression). None (no filtering, default), 'literal' (message Body literal match) or 'jsonpath' Message_filtering ( Optional ]) - Specified how received messages should be filtered. Visibility_timeout ( Optional ) - Visibility timeout, a period of time during whichĪmazon SQS prevents other consumers from receiving and processing the message. Wait_time_seconds ( int) - The time in seconds to wait for receiving messages (default: 1 second) Max_messages ( int) - The maximum number of messages to retrieve for each poke (templated) Sqs_queue - The SQS queue url (templated) Read messages from an Amazon SQS queue Parameters apiVersion: v1 kind: Service metadata: name: airflow namespace: airflow spec: ports: - port: 80 targetPort: 8080 protocol: TCP type: NodePort selector: app: airflow - apiVersion: /v1 kind: Ingress metadata: name: airflow namespace: airflow annotations: /ttl: "300" /hostname: kubernetes.io/ingress.class: alb /scheme: internal /target-type: ip /certificate-arn: arn:aws:acm:eu-west-1:1234567890:certificate/a01d4f1d-1009-42f4-9f04-c0f98bccffbf /listen-ports: '[-vpc" cidr = var.vpc_cidr azs = var.azs private_subnets = var.private_subnets public_subnets = var.For more information on how to use this sensor, take a look at the guide: However, as I mentioned previously we are no longer using Istio in the new EKS cluster so instead we configure service and ingress resources, and use the AWS Load Balancer Controller annotations on the ingress resource to provision an internal load balancer (because this service should not be accessible outside of our organisation) within AWS for our application. The majority of the helm release config is identical, although re-factored slightly to conform to the chart specific templates. In my opinion this is one of the downsides of using helm to deploy applications in a smaller environment, however discussing the pros and cons of deploying to Kubernetes using helm is not within the scope of this post. The configuration for the migration was very similar however we opted to use the official helm chart for the deployment rather than the community chart, which meant re-factoring many of the parameters and values to conform to the official helm chart specs. apiVersion: /v1alpha3 kind: VirtualService metadata: name: airflow namespace: airflow spec: hosts: - gateways: - istio-system/limejump-gateway-external http: - route: - destination: host: airflow-webĪll is working as expected, and when navigating in a web browser to you are routed to the Airflow web GUI and are prompted to login using Google SSO identity provider. Our web base URL is configured with the HTTPS protocol: web: baseUrl: Īnd we have an Istio virtual service for ingress using our configured Istio external ingress gateway. extraEnv: - name: AIRFLOW_WEBSERVER_AUTHENTICATE value: "True" - name: AIRFLOW_WEBSERVER_AUTH_BACKEND value: .google_auth - name: AIRFLOW_GOOGLE_CLIENT_ID value: - name: AIRFLOW_GOOGLE_CLIENT_SECRET valueFrom: secretKeyRef: name: airflow-google-client-secret key: client_secret optional: false - name: AIRFLOW_GOOGLE_OAUTH_CALLBACK_ROUTE value: "/oauth2callback" - name: AIRFLOW_GOOGLE_DOMAIN value: - name: AIRFLOW_GOOGLE_PROMPT value: "select_account consent" Note: The AIRFLOW_GOOGLE_CLIENT_ID, and AIRFLOW_GOOGLE_DOMAIN values below have been replaced with dummy data. You can see we are retrieving Google credentials from a Kubernetes secret which is configured via ExternalSecrets. OAuth values were set in the helm chart and the callback URLs were configured on the Google GCP side for the application. Authentication for the web GUI was handled via Google OAuth, over HTTPS - a fairly standard setup. Our original implementation of Apache Airflow was deployed onto EKS using the community maintained helm chart. Self-hosted Implementation The original configuration
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |