Airflow

[AIRFLOW-159] Add cloud integration section + GCP documentation
[AIRFLOW-477][AIRFLOW-478] Restructure security section for clarity
[AIRFLOW-467] Allow defining of project_id in BigQueryHook
[AIRFLOW-483] Change print to logging statement
[AIRFLOW-475] make the segment granularity in Druid hook configurable

[AIRFLOW-886] Pass result to post_execute() hook
[AIRFLOW-871] change logging.warn() into warning()
[AIRFLOW-882] Remove unnecessary dag>>op assignment in docs
[AIRFLOW-861] make pickle_info endpoint be login_required
[AIRFLOW-869] Refactor mark success functionality
[AIRFLOW-877] Remove .sql template extension from GCS download operator
[AIRFLOW-826] Add Zendesk hook
[AIRFLOW-842] do not query the DB with an empty IN clause
[AIRFLOW-834] change raise StopIteration into return
[AIRFLOW-832] Let debug server run without SSL
[AIRFLOW-862] Add DaskExecutor
[AIRFLOW-858] Configurable database name for DB operators
[AIRFLOW-863] Example DAGs should have recent start dates
[AIRFLOW-853] use utf8 encoding for stdout line decode
[AIRFLOW-857] Use library assert statements instead of conditionals
[AIRFLOW-856] Make sure execution date is set for local client
[AIRFLOW-854] Add OKI as Airflow user
[AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce Travis log verbosity
[AIRFLOW-814] Fix Presto*CheckOperator.__init__
[AIRFLOW-793] Enable compressed loading in S3ToHiveTransfer
[AIRFLOW-844] Fix cgroups directory creation
[AIRFLOW-831] Restore import to fix broken tests
[AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN exclusively from settings
[AIRFLOW-694] Fix config behaviour for empty envvar
[AIRFLOW-365] Set dag.fileloc explicitly and use for Code view
[AIRFLOW-781] Allow DataFlowOperators to accept jobs stored in GCS

[AIRFLOW-2381] Fix the flaky ApiPasswordTests test
[AIRFLOW-2378] Add Groupon to list of current users
[AIRFLOW-2382] Fix wrong description for delimiter
[AIRFLOW-2380] Add support for environment variables in Spark submit operator.
[AIRFLOW-2377] Improve Sendgrid sender support
[AIRFLOW-2331] Support init action timeout on dataproc cluster create
[AIRFLOW-1835] Update docs: Variable file is json
[AIRFLOW-1781] Make search case-insensitive in LDAP group
[AIRFLOW-2042] Fix browser menu appearing over the autocomplete menu
[AIRFLOW-XXX] Remove wheelhouse files from travis not owned by travis
[AIRFLOW-2336] Use hmsclient in hive_hook
[AIRFLOW-2041] Correct Syntax in python examples
[AIRFLOW-74] SubdagOperators can consume all celeryd worker processes
[AIRFLOW-2369] Fix gcs tests
[AIRFLOW-2365] Fix autocommit attribute check
[AIRFLOW-2068] MesosExecutor allows optional Docker image
[AIRFLOW-1652] Push DatabricksRunSubmitOperator metadata into XCOM
[AIRFLOW-2234] Enable insert_rows for PrestoHook
[AIRFLOW-2208][Airflow-22208] Link to same DagRun graph from TaskInstance view
[AIRFLOW-1153] Allow HiveOperators to take hiveconfs
[AIRFLOW-775] Fix autocommit settings with Jdbc hook
[AIRFLOW-2364] Warn when setting autocommit on a connection which does not support it
[AIRFLOW-2357] Add persistent volume for the logs
[AIRFLOW-766] Skip conn.commit() when in Auto-commit
[AIRFLOW-2351] Check for valid default_args start_date
[AIRFLOW-1433] Set default rbac to initdb
[AIRFLOW-2270] Handle removed tasks in backfill
[AIRFLOW-2344] Fix `connections -l` to work with pipe/redirect
[AIRFLOW-2300] Add S3 Select functionarity to S3ToHiveTransfer
[AIRFLOW-1314] Cleanup the config
[AIRFLOW-1314] Polish some of the Kubernetes docs/config
[AIRFLOW-1314] Improve error handling
[AIRFLOW-1999] Add per-task GCP service account support
[AIRFLOW-1314] Rebasing against master
[AIRFLOW-1314] Small cleanup to address PR comments (24)
[AIRFLOW-1314] Add executor_config and tests
[AIRFLOW-1314] Improve k8s support
[AIRFLOW-1314] Use VolumeClaim for transporting DAGs
[AIRFLOW-1314] Create integration testing environment
[AIRFLOW-1314] Git Mode to pull in DAGs for Kubernetes Executor
[AIRFLOW-1314] Add support for volume mounts & Secrets in Kubernetes Executor
[AIRFLOW=1314] Basic Kubernetes Mode
[AIRFLOW-2326][AIRFLOW-2222] remove contrib.gcs_copy_operator
[AIRFLOW-2328] Fix empty GCS blob in S3ToGoogleCloudStorageOperator
[AIRFLOW-2350] Fix grammar in UPDATING.md
[AIRFLOW-2302] Fix documentation
[AIRFLOW-2345] pip is not used in this setup.py
[AIRFLOW-2347] Add Banco de Formaturas to Readme
[AIRFLOW-2346] Add Investorise as official user of Airflow
[AIRFLOW-2330] Do not append destination prefix if not given
[AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.
[AIRFLOW-2309] Fix duration calculation on TaskFail
[AIRFLOW-2335] fix issue with jdk8 download for ci
[AIRFLOW-2184] Add druid_checker_operator
[AIRFLOW-2299] Add S3 Select functionarity to S3FileTransformOperator
[AIRFLOW-2254] Put header as first row in unload
[AIRFLOW-610] Respect _cmd option in config before defaults
[AIRFLOW-2287] Fix incorrect ASF headers
[AIRFLOW-XXX] Add Zego as an Apache Airflow user
[AIRFLOW-952] fix save empty extra field in UI
[AIRFLOW-1325] Add ElasticSearch log handler and reader
[AIRFLOW-2301] Sync files of an S3 key with a GCS path
[AIRFLOW-2293] Fix S3FileTransformOperator to work with boto3
[AIRFLOW-3212][AIRFLOW-2314] Remove only leading slash in GCS path
[AIRFLOW-1509][AIRFLOW-442] SFTP Sensor
[AIRFLOW-2291] Add optional params to ML Engine
[AIRFLOW-1774] Allow consistent templating of arguments in MLEngineBatchPredictionOperator
[AIRFLOW-2302] Add missing operators and hooks
[AIRFLOW-2312] Docs Typo Correction: Corresponding
[AIRFLOW-1623] Trigger on_kill method in operators
[AIRFLOW-2162] When impersonating another user, pass env variables to sudo
[AIRFLOW-2304] Update quickstart doc to mention scheduler part
[AIRFLOW-1633] docker_operator needs a way to set shm_size
[AIRFLOW-1340] Add S3 to Redshift transfer operator
[AIRFLOW-2303] Lists the keys inside an S3 bucket
[AIRFLOW-2209] restore flask_login imports
[AIRFLOW-2306] Add Bonnier Broadcasting to list of current users
[AIRFLOW-2305][AIRFLOW-2027] Fix CI failure caused by []
[AIRFLOW-2281] Add support for Sendgrid categories
[AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
[AIRFLOW-2256] SparkOperator: Add Client Standalone mode and retry mechanism
[AIRFLOW-2284] GCS to S3 operator
[AIRFLOW-2287] Update license notices
[AIRFLOW-2296] Add Cinimex DataLab to Readme
[AIRFLOW-2298] Add Kalibrr to who uses airflow
[AIRFLOW-2292] Fix docstring for S3Hook.get_wildcard_key
[AIRFLOW-XXX] Update PR template
[AIRFLOW-XXX] Remove outdated migrations.sql
[AIRFLOW-2287] Add license header to docs/Makefile
[AIRFLOW-2286] Add tokopedia to the readme
[AIRFLOW-2273] Add Discord webhook operator/hook
[AIRFLOW-2282] Fix grammar in UPDATING.md
[AIRFLOW-2200] Add snowflake operator with tests
[AIRFLOW-2178] Add handling on SLA miss errors
[AIRFLOW-2169] Fix type 'bytes' is not JSON serializable in python3
[AIRFLOW-2215] Pass environment to subproces.Popen in base_task_runner
[AIRFLOW-2253] Add Airflow CLI instrumentation
[AIRFLOW-2274] Fix Dataflow tests
[AIRFLOW-2269] Add Custom Ink as an Airflow user
[AIRFLOW-2259] Dataflow Hook Index out of range
[AIRFLOW-2233] Update updating.md to include the info of hdfs_sensors renaming
[AIRFLOW-2217] Add Slack webhook operator
[AIRFLOW-1729] improve dagBag time
[AIRFLOW-2264] Improve create_user cli help message
[AIRFLOW-2260] [AIRFLOW-2260] SSHOperator add command template .sh files
[AIRFLOW-2261] Check config/env for remote base log folder
[AIRFLOW-2258] Allow import of Parquet-format files into BigQuery
[AIRFLOW-1430] Include INSTALL instructions to avoid GPL
[AIRFLOW-1430] Solve GPL dependency
[AIRFLOW-2251] Add Thinknear as an Airflow user
[AIRFLOW-2244] bugfix: remove legacy LongText code from models.py
[AIRFLOW-2247] Fix RedshiftToS3Transfer not to fail with ValueError
[AIRFLOW-2249] Add side-loading support for Zendesk Hook
[AIRFLOW-XXX] Add Qplum to Airflow users
[AIRFLOW-2228] Enhancements in ValueCheckOperator
[AIRFLOW-1206] Typos

[AIRFLOW-2592] Bump Bleach dependency
[AIRFLOW-2622] Add "confirm=False" option to SFTPOperator
[AIRFLOW-2662] support affinity & nodeSelector policies for kubernetes executor/operator
[AIRFLOW-2709] Improve error handling in Databricks hook
[AIRFLOW-2723] Update lxml dependancy to >= 4.0.
[AIRFLOW-2763] No precheck mechanism in place during worker initialisation for the connection to metadata database
[AIRFLOW-2789] Add ability to create single node cluster to DataprocClusterCreateOperator
[AIRFLOW-2797] Add ability to create Google Dataproc cluster with custom image
[AIRFLOW-2854] kubernetes_pod_operator add more configuration items
[AIRFLOW-2855] Need to Check Validity of Cron Expression When Process DAG File/Zip File
[AIRFLOW-2904] Clean an unnecessary line in airflow/executors/celery_executor.py
[AIRFLOW-2921] A trivial incorrectness in CeleryExecutor()
[AIRFLOW-2922] Potential deal-lock bug in CeleryExecutor()
[AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
[AIRFLOW-2949] Syntax Highlight for Single Quote
[AIRFLOW-2951] dag_run end_date Null after a dag is finished
[AIRFLOW-2956] Kubernetes tolerations for pod operator
[AIRFLOW-2997] Support for clustered tables in Bigquery hooks/operators
[AIRFLOW-3006] Fix error when schedule_interval="None"
[AIRFLOW-3008] Move Kubernetes related example DAGs to contrib/example_dags
[AIRFLOW-3025] Allow to specify dns and dns-search parameters for DockerOperator
[AIRFLOW-3067] (www_rbac) Flask flash messages are not displayed properly (no background color)
[AIRFLOW-3069] Decode output of S3 file transform operator
[AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer role
[AIRFLOW-3090] INFO logs are too verbose
[AIRFLOW-3103] Update Flask-Login
[AIRFLOW-3112] Align SFTP hook with SSH hook
[AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg
[AIRFLOW-3137] Make ProxyFix middleware optional
[AIRFLOW-3173] Add _cmd options for more password config options
[AIRFLOW-3177] Change scheduler_heartbeat metric from gauge to counter
[AIRFLOW-3193] Pin docker requirement version to v3
[AIRFLOW-3195] Druid Hook: Log ingestion spec and task id
[AIRFLOW-3197] EMR Hook is missing some parameters to valid on the AWS API
[AIRFLOW-3232] Make documentation for GCF Functions operator more readable
[AIRFLOW-3262] Can't get log containing Response when using SimpleHttpOperator
[AIRFLOW-3265] Add support for "unix_socket" in connection extra for Mysql Hook

Doc-only changes:

[AIRFLOW-1441] Tutorial Inconsistencies Between Example Pipeline Definition and Recap
[AIRFLOW-2682] Add how-to guide(s) for how to use basic operators like BashOperator and PythonOperator
[AIRFLOW-3104] .airflowignore feature is not mentioned at all in documentation
[AIRFLOW-3237] Refactor example DAGs
[AIRFLOW-3187] Update airflow.gif file with a slower version
[AIRFLOW-3159] Update Airflow documentation on GCP Logging
[AIRFLOW-3030] Command Line docs incorrect subdir
[AIRFLOW-2990] Docstrings for Hooks/Operators are in incorrect format
[AIRFLOW-3127] Celery SSL Documentation is out-dated
[AIRFLOW-2779] Add license headers to doc files
[AIRFLOW-2779] Add project version to license

Bug fixes:

[AIRFLOW-839] docker_operator.py attempts to log status key without first checking existence
[AIRFLOW-1104] Concurrency check in scheduler should count queued tasks as well as running
[AIRFLOW-1163] Add support for x-forwarded-* headers to support access behind AWS ELB
[AIRFLOW-1195] Cleared tasks in SubDagOperator do not trigger Parent dag_runs
[AIRFLOW-1508] Skipped state not part of State.task_states
[AIRFLOW-1762] Use key_file in SSHHook.create_tunnel()
[AIRFLOW-1837] Differing start_dates on tasks not respected by scheduler.
[AIRFLOW-1874] Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
[AIRFLOW-1917] print() from python operators end up with extra new line
[AIRFLOW-1970] Database cannot be initialized if an invalid fernet key is provided
[AIRFLOW-2145] Deadlock after clearing a running task
[AIRFLOW-2216] Cannot specify a profile for AWS Hook to load with s3 config file
[AIRFLOW-2574] initdb fails when mysql password contains percent sign
[AIRFLOW-2707] Error accessing log files from web UI
[AIRFLOW-2716] Replace new Python 3.7 keywords
[AIRFLOW-2744] RBAC app doesn't integrate plugins (blueprints etc)
[AIRFLOW-2772] BigQuery hook does not allow specifying both the partition field name and table name at the same time
[AIRFLOW-2778] Bad Import in collect_dag in DagBag
[AIRFLOW-2786] Variables view fails to render if a variable has an empty key
[AIRFLOW-2799] Filtering UI objects by datetime is broken
[AIRFLOW-2800] Remove airflow/ low-hanging linting errors
[AIRFLOW-2825] S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3
[AIRFLOW-2848] dag_id is missing in metadata table "job" for LocalTaskJob
[AIRFLOW-2860] DruidHook: time variable is not updated correctly when checking for timeout
[AIRFLOW-2865] Race condition between on_success_callback and LocalTaskJob's cleanup
[AIRFLOW-2893] Stuck dataflow job due to jobName mismatch.
[AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs
[AIRFLOW-2900] Code not visible for Packaged DAGs
[AIRFLOW-2905] Switch to regional dataflow job service.
[AIRFLOW-2907] Sendgrid - Attachments - ERROR - Object of type 'bytes' is not JSON serializable
[AIRFLOW-2938] Invalid 'extra' field in connection can raise an AttributeError when attempting to edit
[AIRFLOW-2979] Deprecated Celery Option not in Options list
[AIRFLOW-2981] TypeError in dataflow operators when using GCS jar or py_file
[AIRFLOW-2984] Cannot convert naive_datetime when task has a naive start_date/end_date
[AIRFLOW-2994] flatten_results in BigQueryOperator/BigQueryHook should default to None
[AIRFLOW-3002] ValueError in dataflow operators when using GCS jar or py_file
[AIRFLOW-3012] Email on sla miss is send only to first address on the list
[AIRFLOW-3046] ECS Operator mistakenly reports success when task is killed due to EC2 host termination
[AIRFLOW-3064] No output from `airflow test` due to default logging config
[AIRFLOW-3072] Only admin can view logs in RBAC UI
[AIRFLOW-3079] Improve initdb to support MSSQL Server
[AIRFLOW-3089] Google auth doesn't work under http
[AIRFLOW-3099] Errors raised when some blocs are missing in airflow.cfg
[AIRFLOW-3109] Default user permission should contain 'can_clear'
[AIRFLOW-3111] Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
[AIRFLOW-3124] Broken webserver debug mode (RBAC)
[AIRFLOW-3136] Scheduler Failing the Task retries run while processing Executor Events
[AIRFLOW-3138] Migration cc1e65623dc7 creates issues with postgres
[AIRFLOW-3161] Log Url link does not link to task instance logs in RBAC UI
[AIRFLOW-3162] HttpHook fails to parse URL when port is specified
[AIRFLOW-3183] Potential Bug in utils/dag_processing/DagFileProcessorManager.max_runs_reached()
[AIRFLOW-3203] Bugs in DockerOperator & Some operator test scripts were named incorrectly
[AIRFLOW-3238] Dags, removed from the filesystem, are not deactivated on initdb
[AIRFLOW-3268] Cannot pass SSL dictionary to mysql connection via URL
[AIRFLOW-3277] Invalid timezone transition handling for cron schedules
[AIRFLOW-3295] Require encryption in DaskExecutor when certificates are configured.
[AIRFLOW-3297] EmrStepSensor marks cancelled step as successful

[AIRFLOW-261] Add bcc and cc fields to EmailOperator
[AIRFLOW-348] Fix code style warnings
[AIRFLOW-349] Add metric for number of zombies killed
[AIRFLOW-340] Remove unused dependency on Babel
[AIRFLOW-339]: Ability to pass a flower conf file
[AIRFLOW-341][operators] Add resource requirement attributes to operators
[AIRFLOW-335] Fix simple style errors/warnings
[AIRFLOW-337] Add __repr__ to VariableAccessor and VariableJsonAccessor
[AIRFLOW-334] Fix using undefined variable
[AIRFLOW-315] Fix blank lines code style warnings
[AIRFLOW-306] Add Spark-sql Hook and Operator
[AIRFLOW-327] Add rename method to the FTPHook
[AIRFLOW-321] Fix a wrong code example about tests/dags
[AIRFLOW-316] Always check DB state for Backfill Job execution
[AIRFLOW-264] Adding workload management for Hive
[AIRFLOW-297] support exponential backoff option for retry delay
[AIRFLOW-31][AIRFLOW-200] Add note to updating.md
[AIRFLOW-307] There is no __neq__ python magic method.
[AIRFLOW-309] Add requirements of develop dependencies to docs
[AIRFLOW-307] Rename __neq__ to __ne__ python magic method.
[AIRFLOW-313] Fix code style for sqoop_hook.py
[AIRFLOW-311] Fix wrong path in CONTRIBUTING.md
[AIRFLOW-24] DataFlow Java Operator
[AIRFLOW-308] Add link to refresh DAG within DAG view header
[AIRFLOW-314] Fix BigQuery cursor run_table_upsert method
[AIRFLOW-298] fix incubator diclaimer in docs
[AIRFLOW-284] HiveServer2Hook fix for cursor scope for get_results
[AIRFLOW-260] More graceful exit when issues can't be closed
[AIRFLOW-260] Handle case when no version is found
[AIRFLOW-228] Handle empty version list in PR tool
[AIRFLOW-302] Improve default squash commit message
[AIRFLOW-187] Improve prompt styling
[AIRFLOW-187] Fix typo in argument name
[AIRFLOW-187] Move "Close XXX" message to end of squash commit
[AIRFLOW-247] Add EMR hook, operators and sensors. Add AWS base hook
[AIRFLOW-301] Fix broken unit test
[AIRFLOW-100] Add execution_date_fn to ExternalTaskSensor
[AIRFLOW-282] Remove PR Tool logic that depends on version formatting
[AIRFLOW-291] Add index for state in TI table
[AIRFLOW-269] Add some unit tests for PostgreSQL
[AIRFLOW-296] template_ext is being treated as a string rather than a tuple in qubole operator
[AIRFLOW-286] Improve FTPHook to implement context manager interface
[AIRFLOW-243] Create NamedHivePartitionSensor
[AIRFLOW-246] Improve dag_stats endpoint query
[AIRFLOW-189] Highlighting of Parent/Child nodes in Graphs
[ARFLOW-255] Check dagrun timeout when comparing active runs
[AIRFLOW-281] Add port to mssql_hook
[AIRFLOW-285] Use Airflow 2.0 style imports for all remaining hooks/operators
[AIRFLOW-40] Add LDAP group filtering feature.
[AIRFLOW-277] Multiple deletions does not work in Task Instances view if using SQLite backend
[AIRFLOW-200] Make hook/operator imports lazy, and print proper exceptions
[AIRFLOW-283] Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator
[AIRFLOW-278] Support utf-8 ecoding for SQL
[AIRFLOW-280] clean up tmp druid table no matter if an ingestion job succeeds or not
[AIRFLOW-274] Add XCom functionality to GoogleCloudStorageDownloadOperator
[AIRFLOW-273] Create an svg version of the airflow logo.
[AIRFLOW-275] Update contributing guidelines
[AIRFLOW-244] Modify hive operator to inject analysis data
[AIRFLOW-162] Allow variable to be accessible into templates
[AIRFLOW-248] Add Apache license header to all files
[AIRFLOW-263] Remove temp backtick file
[AIRFLOW-252] Raise Sqlite exceptions when deleting tasks instance in WebUI
[AIRFLOW-180] Fix timeout behavior for sensors
[AIRFLOW-262] Simplify commands in MANIFEST.in
[AIRFLOW-31] Add zope dependency
[AIRFLOW-6] Remove dependency on Highcharts
[AIRFLOW-234] make task that aren't `running` self-terminate
[AIRFLOW-256] Fix test_scheduler_reschedule heartrate
Add Python 3 compatibility fix
[AIRFLOW-31] Use standard imports for hooks/operators
[AIRFLOW-173] Initial implementation of FileSensor
[AIRFLOW-224] Collect orphaned tasks and reschedule them
[AIRFLOW-239] Fix tests indentation
[AIRFLOW-225] Better units for task duration graph
[AIRFLOW-241] Add testing done section to PR template
[AIRFLOW-222] Show duration of task instances in ui
[AIRFLOW-231] Do not eval user input in PrestoHook
[AIRFLOW-216] Add Sqoop Hook and Operator
[AIRFLOW-171] Add upgrade notes on email and S3 to 1.7.1.2