????? ??????(Feat. gcp) - ??????(GCP) ?? ??
11 likes2,404 views
This document provides instructions for installing and using Embulk and Airflow. It explains how to install Embulk and common Embulk plugins, run Embulk tasks to load data from MySQL to BigQuery, and configure an Embulk config file to specify the input, output, and load options. It also explains how to install Airflow, initialize the metadata database, run the Airflow web server, list and test DAGs and tasks, and write a simple DAG with BashOperator tasks to print the date and sleep for demonstration purposes.
![# airflow
pip3 install apache-airflow
airflow initdb
airflow webserver -p 8080
localhost:8080?
?
airflow list_dags
- airflow dags *.py , DAGs
- dags DAG
airflow list_tasks test
- test dags tasks
airflow list_tasks test --tree
- test dags tasks tree
airflow test [DAG id] [Task id] [date]
) airflow test test print_date 2017-10-01
- DAG Task test
airflow scheduler
- Test , . DAG](https://image.slidesharecdn.com/feat-180118115120/85/Feat-gcp-GCP-36-320.jpg)
![from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2017, 10, 1),
'email': ['airflow@airflow.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
# 'queue': 'bash_queue',
# 'pool': 'backfill', # Only celery option
# 'priority_weight': 10,
# 'end_date': datetime(2016, 1, 1),
}
# dag
dag = DAG('test', description='First DAG',
schedule_interval = '55 14 * * *',
default_args=default_args)
t1 = BashOperator(
task_id='print_date',
bash_command='date',
dag=dag)
# BashOperator
# task_id unique
# bash_command bash date
t2 = BashOperator(
task_id='sleep',
bash_command='sleep 5',
retries=3,
dag=dag)
t3 = BashOperator(
task_id='templated',
bash_command=templated_command,
params={'my_param': 'Parameter I passed in'},
dag=dag)
# set_upstream t1 t2
t2.set_upstream(t1)
# t1.set_downstream(t2)
# t1 >> t2
t3.set_upstream(t1)](https://image.slidesharecdn.com/feat-180118115120/85/Feat-gcp-GCP-37-320.jpg)
![[MLOps KR ??] MLOps ?? ?? ?? ??(210605)](https://cdn.slidesharecdn.com/ss_thumbnails/mlops-basic-210605064957-thumbnail.jpg?width=560&fit=bounds)
![[NDC18] ??? ? ???? ??? ????? ???: ?? ??? ?? ?? ??](https://cdn.slidesharecdn.com/ss_thumbnails/ndc2018-1-180430181759-thumbnail.jpg?width=560&fit=bounds)
![Naver???, ??? ??, ??? ?? ??DB (??? ?????? ??DB) [Naver]](https://cdn.slidesharecdn.com/ss_thumbnails/naver-190916181334-thumbnail.jpg?width=560&fit=bounds)
![[NDC18] ??? ? ???? ??? ????? ???: ?? ??? ?? ?? ?? (2?)](https://cdn.slidesharecdn.com/ss_thumbnails/ndc2018-2-180430180517-thumbnail.jpg?width=560&fit=bounds)
![[KAIST ?????] ??? ????? ?? ?? ????](https://cdn.slidesharecdn.com/ss_thumbnails/temp-180502031907-thumbnail.jpg?width=560&fit=bounds)
![[261] ???????? ??????????? ???????????? ?????????](https://cdn.slidesharecdn.com/ss_thumbnails/216-150915054828-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
![[???] ??? ????](https://cdn.slidesharecdn.com/ss_thumbnails/woowahan-oo-190624161343-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to ????? ??????(Feat. gcp) - ??????(GCP) ?? ?? (20)
Recently uploaded (20)
????? ??????(Feat. gcp) - ??????(GCP) ?? ??
- 2. ? ? ?
- 3. ? ? ?
- 4. ? ? ?
- 8. ? ? ? ? ? ?
- 13. ?
- 15. ? ?
- 19. ? ? ? ? ? ?
- 24. ? ?
- 25. ? ?
- 30. # embulk curl --create-dirs -o ~/.embulk/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar" chmod +x ~/.embulk/bin/embulk echo 'export PATH="$HOME/.embulk/bin:$PATH"' >> ~/.bashrc source ~/.bashrc? ? # ? embulk gem install embulk-input-mysql embulk gem install embulk-output-bigquery? ? # embulk ? embulk example - embulk guess embulk-example/seed.yml - seed.yml embulk guess embulk-example/seed.yml -o config.yml - seed.yml config.yml embulk preview config.yml - config.yml embulk run config.yml - config.yml
- 31. # config.yml in: type: mysql host: localhost port: 3306 user: root password: root database: database_name table: table_name select: "col1, col2, col3" where: "col4 != 'a'" order_by: "col1 DESC" out: type: bigquery mode: replace auth_method: json_key json_keyfile: /path/to/json_keyfile.json project: my-project dataset: reservation table: reservation gcs_bucket: seongyun auto_create_gcs_bucket: true auto_create_table: true auto_create_dataset: true embulk run config.yml
- 35. ? ?
- 36. # airflow pip3 install apache-airflow airflow initdb airflow webserver -p 8080 localhost:8080? ? airflow list_dags - airflow dags *.py , DAGs - dags DAG airflow list_tasks test - test dags tasks airflow list_tasks test --tree - test dags tasks tree airflow test [DAG id] [Task id] [date] ) airflow test test print_date 2017-10-01 - DAG Task test airflow scheduler - Test , . DAG
- 37. from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2017, 10, 1), 'email': ['airflow@airflow.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), # 'queue': 'bash_queue', # 'pool': 'backfill', # Only celery option # 'priority_weight': 10, # 'end_date': datetime(2016, 1, 1), } # dag dag = DAG('test', description='First DAG', schedule_interval = '55 14 * * *', default_args=default_args) t1 = BashOperator( task_id='print_date', bash_command='date', dag=dag) # BashOperator # task_id unique # bash_command bash date t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag) t3 = BashOperator( task_id='templated', bash_command=templated_command, params={'my_param': 'Parameter I passed in'}, dag=dag) # set_upstream t1 t2 t2.set_upstream(t1) # t1.set_downstream(t2) # t1 >> t2 t3.set_upstream(t1)
- 41. ? Project Id :? Project Id ? Key?le Path :?json key?le ? Scopes : https://www.googleapis.com/auth/cloud-platform
- 42. ? ?