You must migrate all of the jobs that are still valid as the result of the migration, and create new Job IDs for new jobs. The Job ID is from the CA7 Job Definition table.
AIRFLOW STREAMING PLUS
Job_Definition: the main fields include Job ID and Job description, plus the new field for Airflow – DAG Template.Therefore, only a small list of tables from CA7 were carried over: We excluded a big chunk of the CA7 database, such as the tables for automatic recovery and Job Control Language, because we won’t have the applications in the scheduler database. Apache Spark is used in the applications to take advantage of Spark’s interface for programming the clusters with data parallelism and fault tolerance. We implemented Airflow’s SparkSubmitOperator in the DAGs to launch Spark jobs. The application also publishes the processing status by sending out Kafka topics to notify the Airflow scheduler and Prometheus monitor. The application consumes the topic and starts to process data. When the S3 Poller detects a data file, Airflow runs a job that publishes a Kafka topic. We implemented Kafka Producer in Airflow. Implementations to trigger streaming jobsĪlthough AWS has Amazon Managed Streaming for Apache Kafka service, this deployment used the original Apache Kafka due to the AWS Outposts requirement.
DAGs that are cross-dependent between them must be run in the same instance, or one after the other in a constant amount of time. This sensor will look up past executions of DAGs and tasks, and it will match those DAGs that share the same execution_date as our DAG. The ExternalTaskSensor sensor lets your task (B1) in DAG(B) be scheduled and wait for a success on task (A1) in DAG(A). TriggerDagRunOperator triggers another DAG from a DAG. SubDagOperator lets us run a DAG with a separate set of tasks within another DAG. Implementations to handle job dependenciesĭepending on the use cases, you may try three different Airflow operators to handle job dependencies as shown in the following diagram. Since Apache Airflow doesn’t have a “Move Object” operator, we implemented Apache Airflow’s S3CopyObjectOperator and S3DeleteObjectOperator to move the S3 Object, so that an incoming file can be moved to a different folder to avoid repeated processing. We implemented Apache Airflow’s S3KeySensor as our S3 poller to respond to S3 events. Although it delivers object storage on the customer’s premises, Amazon S3 is still a fully managed service and designed to provide high durability and redundancy. Amazon S3 is available on AWS Outposts as a new storage class called “S3 Outposts”. We used Amazon S3 for the storage of Airflow DAGs and data files. We used the open-source workflow management platform Apache Airflow to achieve the scheduling capabilities, including the abilities to run, retry, pause, kill, and override jobs to run concurrent jobs to define the dependency of jobs to view the job execution status and to allow values to be passed between tasks and the templating of jobs.Īs mentioned above, we have a use case of AWS Outposts deployment, so Amazon MWAA is not used. Run, Rerun, Pause/Hold, Kill, or Overrideįor capabilities not covered in this post, such as monitoring and alerts, client onboarding, and reporting, we will cover it in a future blog.To understand the scope of migration, we assessed the functionalities offered by mainframe CA7 scheduler and the capabilities noted below are covered in this post: In turn, the database and calendar management service can send schedule changes to schedulers. From the dashboard, the administrator can interface with the database and work on a calendar management service. Prometheus sends metrics/logs to a Grafana dashboard. The Airflow worker nodes have Apache Kafka, Apache Spark, and Prometheus built in. From left to right, you see the Airflow web server, Airflow schedulers, and the Airflow workers. A Kubernetes cluster of Apache Airflow is deployed on a subnet. Amazon Simple Storage Service (Amazon S3) on Outposts is used to store Apache Airflow’s Directed Acyclic Graph (DAG) objects and data files. The architecture diagram shown in the following figure depicts that an AWS Outposts rack is deployed in a customer’s corporate network due to the data residency requirement of the customer for which the solution was provided. You can use either Amazon Managed Workflows for Apache Airflow (MWAA), or run Apache Airflow in an Amazon Elastic Compute Cloud instance, or even deploy it in an instance within AWS Outposts for a hybrid cloud solution.
AIRFLOW STREAMING HOW TO
In this post, I’ll show you how to migrate mainframe CA7 job schedules to a cloud native job scheduler in AWS, how to trigger off event-based jobs, how to run streaming jobs, how to migrate CA7 database, and how to use external calendar management services to manage job schedules. When you migrate mainframe applications to the cloud, you will usually have to migrate mainframe job schedules too.