This configuration is now more flexible and can be easily be maintained in a single file. A it is impractical to modify the config value after an Airflow instance is running for a while, since all existing task logs have be saved under the previous format and cannot be found with the new config value. specifying the service account. variable if you need to use a non default value for this. Refer to test_sftp_operator.py for usage info. kubectl create job --namespace astronomer --from=cronjob/astronomer-houston-update-airflow-check airflow-update-check-first-run, astro deployment airflow upgrade --deployment-id=, FROM quay.io/astronomer/astro-runtime:5.0.4, astro deployment airflow upgrade --cancel --deployment-id=, astro deployment airflow upgrade --cancel --deployment-id, Airflow upgrade process has been successfully canceled. [AIRFLOW-307] There is no __neq__ Python magic method. to historical reasons. If its desirable to let the sensor continue running for longer time, set a larger timeout instead. Airflow 2.3.0 restores the original meaning to the parameter. the number of tasks that is running concurrently for a DAG, add up the number of running A deprecation warning has also been raised for paths (#7319), [AIRFLOW-6491] Improve handling of Breeze parameters (#7084), [AIRFLOW-6470] Avoid pipe to file when do curl (#7063), [AIRFLOW-6471] Add pytest-instafail plugin (#7064), [AIRFLOW-6462] Limit exported variables in Dockerfile/Breeze (#7057), [AIRFLOW-6465] Add bash autocomplete for Airflow in Breeze (#7060), [AIRFLOW-6464] Add cloud providers CLI tools in Breeze (#7059), [AIRFLOW-6461] Remove silent flags in Dockerfile (#7052), [AIRFLOW-6459] Increase verbosity of pytest (#7049), [AIRFLOW-6370] Skip Cassandra tests if cluster is not up (#6926), [AIRFLOW-6511] Remove BATS docker containers (#7103), [AIRFLOW-6475] Remove duplication of volume mount specs in Breeze.. (#7065), [AIRFLOW-6489] Add BATS support for Bash unit testing (#7081), [AIRFLOW-6387] print details of success/skipped task (#6956), [AIRFLOW-6568] Add Emacs related files to .gitignore (#7175), [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking (#7185), [AIRFLOW-6496] Separate integrations in tests (#7091), [AIRFLOW-6634] Set PYTHONPATH in interactive Breeze, [AIRFLOW-6564] Additional diagnostics information on CI check failure (#7172), [AIRFLOW-6383] Add no trailing-whitespace pre-commit hook (#6941), [AIRFLOW-XXXX] Consistency fixes in new documentation (#7207), [AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291), [AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311), [AIRFLOW-XXXX] Update docs with new BranchPythonOperator behaviour (#4682), [AIRFLOW-XXXX] Fix Typo in scripts/ci/ci_run_airflow_testing.sh (#7235), [AIRFLOW-XXXX] Screenshot showing disk space configuration for OSX (#7226), [AIRFLOW-XXXX] Add mentoring information to contributing docs (#7202), [AIRFLOW-XXXX] Add rebase info to contributing (#7201), [AIRFLOW-XXXX] Increase verbosity of static checks in CI (#7200), [AIRFLOW-XXXX] Adds branching strategy to documentation (#7193), [AIRFLOW-XXXX] Move email configuration from the concept page (#7189), [AIRFLOW-XXXX] Update task lifecycle diagram (#7161), [AIRFLOW-XXXX] Fix reference in concepts doc (#7135), [AIRFLOW-XXXX] Fix typos and broken links in development docs (#7086), [AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date (#6999), [AIRFLOW-XXXX] Add airflow dags show command guide (#7014), [AIRFLOW-XXXX] Update operation chaining documentation (#7018), [AIRFLOW-XXXX] Add .autoenv_leave.zsh to .gitignore (#6986), [AIRFLOW-XXXX] Fix development packages installation instructions (#6942), [AIRFLOW-XXXX] Update committers list (#7212), [AIRFLOW-XXXX] Move UPDATING changes into correct versions (#7166), [AIRFLOW-XXXX] Add Documentation for check_slas flag (#6974), [AIRFLOW-XXXX] Fix gcp keyfile_dict typo (#6962), [AIRFLOW-XXXX] Add tips for writing a note in UPDATIND.md (#6960), [AIRFLOW-XXXX] Add note warning that bash>4.0 is required for docs build script (#6947), [AIRFLOW-XXXX] Add autoenv to gitignore (#6946), [AIRFLOW-XXXX] Fix GCSTaskHandler Comment Typo (#6928), [AIRFLOW-XXXX] Fix broken DAG Serialization Link (#6891), [AIRFLOW-XXXX] Add versions_added field to configs, The default behavior was to strip the microseconds (and milliseconds, etc) off of all dag runs triggered by (#4340), [AIRFLOW-2156] Parallelize Celery Executor task state fetching (#3830), [AIRFLOW-3702] Add backfill option to run backwards (#4676), [AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662), [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347), [AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467), [AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647), [AIRFLOW-3817] Corrected task ids returned by BranchPythonOperator to match the dummy operator ids (#4659), [AIRFLOW-3782] Clarify docs around celery worker_autoscale in default_airflow.cfg (#4609), [AIRFLOW-1945] Add Autoscale config for Celery workers (#3989), [AIRFLOW-3590] Change log message of executor exit status (#4616), [AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502), [AIRFLOW-3709] Validate allowed_states for ExternalTaskSensor (#4536), [AIRFLOW-3522] Add support for sending Slack attachments (#4332), [AIRFLOW-3569] Add Trigger DAG button in DAG page (#4373), [AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887), [AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779), [AIRFLOW-2988] Run specifically python2 for dataflow (#3826), [AIRFLOW-3697] Vendorize nvd3 and slugify (#4513), [AIRFLOW-3692] Remove ENV variables to avoid GPL (#4506), [AIRFLOW-3907] Upgrade flask and set cookie security flags. treat the pod parameter as a kubernetes.client.models.V1Pod object. Previously, there were defined in various places, example as ID_PREFIX class variables for pass Log the number of errors when importing DAGs, Log dagbag metrics duplicate messages in queue into StatsD (#1406), correct missed arg.foreground to arg.daemon in cli, Reinstate imports for github enterprise auth, Use os.execvp instead of subprocess.Popen for the webserver, Revert from using foreground to daemon, Fixed a bug in the scheduler: num_runs used where runs intended, Add multiprocessing support to the scheduler, Partial fix to make sure next_run_date cannot be None, Support list/get/set variables in the CLI. (#5307), [AIRFLOW-4519] Optimise operator classname sorting in views (#5282), [AIRFLOW-4503] Support fully pig options (#5271), [AIRFLOW-4468] add sql_alchemy_max_overflow parameter (#5249), [AIRFLOW-4467] Add dataproc_jars to templated fields in Dataproc oper (#5248), [AIRFLOW-4381] Use get_direct_relative_ids get task relatives (#5147), [AIRFLOW-3624] Add masterType parameter to MLEngineTrainingOperator (#4428), [AIRFLOW-3143] Support Auto-Zone in DataprocClusterCreateOperator (#5169), [AIRFLOW-3874] Improve BigQueryHook.run_with_configurations location support (#4695), [AIRFLOW-4399] Avoid duplicated os.path.isfile() check in models.dagbag (#5165), [AIRFLOW-4031] Allow for key pair auth in snowflake hook (#4875), [AIRFLOW-3901] add role as optional config parameter for SnowflakeHook (#4721), [AIRFLOW-3455] add region in snowflake connector (#4285), [AIRFLOW-4073] add template_ext for AWS Athena operator (#4907), [AIRFLOW-4093] AWSAthenaOperator: Throw exception if job failed/cancelled/reach max retries (#4919), [AIRFLOW-4356] Add extra RuntimeEnvironment keys to DataFlowHook (#5149), [AIRFLOW-4337] Fix docker-compose deprecation warning in CI (#5119), [AIRFLOW-3603] QuboleOperator: Remove SQLCommand from SparkCmd documentation (#4411), [AIRFLOW-4328] Fix link to task instances from Pool page (#5124), [AIRFLOW-4255] Make GCS Hook Backwards compatible (#5089), [AIRFLOW-4103] Allow uppercase letters in dataflow job names (#4925), [AIRFLOW-4255] Replace Discovery based api with client based for GCS (#5054), [AIRFLOW-4311] Remove sleep in localexecutor (#5096), [AIRFLOW-2836] Minor improvement-contrib.sensors.FileSensor (#3674), [AIRFLOW-4104] Add type annotations to common classes. The default value for the aws_conn_id was accidentally set to s3_default instead of aws_default in some of the emr operators in previous the scheduler can create in a dag. importing with the suffix. (~/airflow by default): the AIRFLOW_HOME environment variable, and the Weve improved masking for sensitive data in Web UI and logs. * for auth backends (#8072), Fix Example in config_templates for Secrets Backend (#8074), Add backticks in IMAGES.rst command description (#8075), Change version_added for store_dag_code config (#8076), [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716), [AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049), [AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047), [AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048), [AIRFLOW-XXXX] Simplify GCP operators listing, [AIRFLOW-XXXX] Simplify Qubole operators listing, [AIRFLOW-XXXX] Add autogenerated TOC (#6038), [AIRFLOW-XXXX] Create Using the CLI page (#5823), [AIRFLOW-XXXX] Group references in one section (#5776), [AIRFLOW-XXXX] Add S3 Logging section (#6039), [AIRFLOW-XXXX] Move Azure Logging section above operators (#6040), [AIRFLOW-XXXX] Update temp link to a fixed link (#7715), [AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385), [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321), [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466), [AIRFLOW-XXXX] Small BREEZE.rst update (#7487), [AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461), [AIRFLOW-XXXX] Remove travis config warnings (#7467), [AIRFLOW-XXXX] Add communication chapter to contributing (#7204), [AIRFLOW-XXXX] Add known issue - example_dags/__init__.py (#7444), [AIRFLOW-XXXX] Fix breeze build-docs (#7445), [AIRFLOW-XXXX] Less verbose docker builds, [AIRFLOW-XXXX] Speed up mypy runs (#7421), [AIRFLOW-XXXX] Fix location of kubernetes tests (#7373), [AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226), [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318), [AIRFLOW-XXXX] Fix typo in the word committer (#7392), [AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662), Fix reference to KubernetesPodOperator (#8100), [AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377), When task is marked failed by user or task fails due to system failures - on failure call back will be called as part of clean up, [AIRFLOW-4026] Add filter by DAG tags (#6489), [AIRFLOW-6613] Center dag on graph view load (#7238), [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281), [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038), [AIRFLOW-6438] Filter DAGs returned by blocked (#7019), [AIRFLOW-6666] Resolve js-yaml advisory (#7283), [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280), [AIRFLOW-6667] Resolve serialize-javascript advisory (#7282), [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skipable (#7134), [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090), [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871), [AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289), [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007), [AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210), [AIRFLOW-6620] Mock celery in worker cli test (#7243), [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports, [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633), [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978), [AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909), [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868), [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113), [AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886), [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825), [AIRFLOW-6238] Filter dags returned by dag_stats, [AIRFLOW-5616] Switch PrestoHook from pyhive to presto-python-client, [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236), [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162), [AIRFLOW-6584] Pin cassandra driver (#7194), [AIRFLOW-6537] Fix backticks in RST files (#7140), [AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948), [AIRFLOW-6330] Show cli help when param blank or typo (#6883), [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097), [AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036), [AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024), [AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015), [AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143), [AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844), [AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords, [AIRFLOW-6358] Log details of failed task (#6908), [AIRFLOW-5149] Skip SLA checks config (#6923), [AIRFLOW-6057] Update template_fields of the PythonSensor (#6656), [AIRFLOW-4445] Mushroom cloud errors too verbose (#6952), [AIRFLOW-6394] Simplify github PR template (#6955), [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976), [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901), [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187), [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298), [AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296), [AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159), [AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904), [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329), [AIRFLOW-6627] Email with incorrect DAG not delivered (#7250), [AIRFLOW-6637] Fix Airflow test command in 1.10.x, [AIRFLOW-6636] Avoid exceptions when printing task instance, [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120), [AIRFLOW-5501] Make default in_cluster value in KubernetesPodOperator respect config (#6124), [AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367), [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985), [AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013), [AIRFLOW-2516] Fix mysql deadlocks (#6988), [AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124), [AIRFLOW-6517] Make merge_dicts function recursive (#7111), [AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025), [AIRFLOW-6353] Security - ui - add click jacking defense (#6995), [AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915), [AIRFLOW-6323] Remove non-ascii letters from default config (#6878), [AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122), [AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109), [AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005), [AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022), [AIRFLOW-6347] BugFix: Cant get task logs when serialization is enabled (#7092), [AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859), [AIRFLOW-6366] Fix migrations for MS SQL Server (#6920), [AIRFLOW-5406] Allow spark without kubernetes (#6921), [AIRFLOW-6229] SparkSubmitOperator polls forever if status JSON cant (#6918), [AIRFLOW-6352] Security - ui - add login timeout (#6912), [AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958), [AIRFLOW-6400] Fix pytest not working on Windows (#6964), [AIRFLOW-6418] Remove SystemTest.skip decorator (#6991), [AIRFLOW-6425] Serialization: Add missing DAG parameters to JSON Schema (#7002), [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067), [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083), [AIRFLOW-5814] Implementing Presto hook tests (#6491), [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516), [AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108), [AIRFLOW-XXXX] Improve clarity of confirm message (#7110), [AIRFLOW-6705] One less chatty message at breeze initialisation (#7326), [AIRFLOW-6705] Less chatty integration/backend checks (#7325), [AIRFLOW-6662] Switch to init docker flag for signal propagation (#7278), [AIRFLOW-6661] Fail after 50 failing tests (#7277), [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225), [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203), [AIRFLOW-6592] Doc build is moved to test phase (#7208), [AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261), [AIRFLOW-6642] Make local task job test less flaky (#7262), [AIRFLOW-6643] Fix flakiness of kerberos tests, [AIRFLOW-6638] Remove flakiness test from test_serialized_db remove, [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323), [AIRFLOW-6702] Dumping kind logs to file.io. Properly handle BigQuery booleans in BigQuery hook. The client also handles retries automatically. This document describes the changes that have been made, and what you need to do to update your usage. This section describes the changes that have been made, and what you need to do to update your script. This patch allows for testing of hive operators and hooks. The following configurations have been moved from [core] to the new [logging] section. This section describes the major changes that have been made in this release. Note: Due to the deprecation of Dockershim, Azure does not support private CAs starting with Kubernetes 1.19. (#13249), Add read only REST API endpoints for users (#14735), Add files to generate Airflows Python SDK (#14739), Add dynamic fields to snowflake connection (#14724), Add read only REST API endpoint for roles and permissions (#14664), Add new datetime branch operator (#11964), Add Google leveldb hook and operator (#13109) (#14105), Add plugins endpoint to the REST API (#14280), Add worker_pod_pending_timeout support (#15263), Add support for labeling DAG edges (#15142), Add CUD REST API endpoints for Roles (#14840), A bunch of template_fields_renderers additions (#15130), Add REST API query sort and order to some endpoints (#14895), Add different modes to sort dag files for parsing (#15046), BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963), Clear tasks by task ids in REST API (#14500), Support jinja2 native Python types (#14603), Allow celery workers without gossip or mingle modes (#13880), Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519), Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720), Add optional result handler callback to DbApiHook (#15581), Update Flask App Builder limit to recently released 3.3 (#15792), Prevent creating flask sessions on REST API requests (#15295), Sync DAG specific permissions when parsing (#15311), Increase maximum length of pool name on Tasks to 256 characters (#15203), Enforce READ COMMITTED isolation when using mysql (#15714), Auto-apply apply_default to subclasses of BaseOperator (#15667), Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669), More verbose logs when running airflow db check-migrations (#15662), When one_success mark task as failed if no success (#15467), Add an option to trigger a dag w/o changing conf (#15591), Add Airflow UI instance_name configuration option (#10162), Add a decorator to retry functions with DB transactions (#14109), Add return to PythonVirtualenvOperators execute method (#14061), Add verify_ssl config for kubernetes (#13516), Add description about secret_key when Webserver > 1 (#15546), Add Traceback in LogRecord in JSONFormatter (#15414), Add support for arbitrary json in conn uri format (#15100), Adds description field in variable (#12413) (#15194), Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134), Execute on_failure_callback when SIGTERM is received (#15172), Allow hiding of all edges when highlighting states (#15281), Display explicit error in case UID has no actual username (#15212), Serve logs with Scheduler when using Local or Sequential Executor (#15557), Deactivate trigger, refresh, and delete controls on dag detail view. Note that this is only a change to the Web UI behavior; when updating your hook in this way, you must make sure that when your hook reads the extra field, it will also check for the prefixed value for backward compatibility. This behaviour is now changed. The WasbHook in Apache Airflow use a legacy version of Azure library. This section describes the changes that have been made, and what you need to do to update your if This setting changes that behavior to only execute the latest interval. The UID to run the first process of the Worker PODs when using has been changed to 50000 Allow changing Task States Colors (#9520), Add support for AWS Secrets Manager as Secrets Backend (#8186), Add Airflow info command to the CLI (#8704), Add Local Filesystem Secret Backend (#8596), Add Support for Python 3.8 (#8836)(#8823), Allow K8S worker pod to be configured from JSON/YAML file (#6230), Add support for ephemeral storage on KubernetesPodOperator (#6337), Add AirflowFailException to fail without any retry (#7133), Use NULL as dag.description default value (#7593), BugFix: DAG trigger via UI error in RBAC UI (#8411), Fix logging issue when running tasks (#9363), Fix JSON encoding error in DockerOperator (#8287), Fix alembic crash due to typing import (#6547), Correctly restore upstream_task_ids when deserializing Operators (#8775), Correctly store non-default Nones in serialized tasks/dags (#8772), Correctly deserialize dagrun_timeout field on DAGs (#8735), Fix tree view if config contains (#9250), Fix Dag Run UI execution date with timezone cannot be saved issue (#8902), RBAC ui: Fix missing Y-axis labels with units in plots (#8252), RBAC ui: Fix missing task runs being rendered as circles instead (#8253), Fix: DagRuns page renders the state column with artifacts in old UI (#9612), Fix task and dag stats on home page (#8865), Fix the trigger_dag api in the case of nested subdags (#8081), UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912), Fix connection add/edit for spark (#8685), Fix retries causing constraint violation on MySQL with DAG Serialization (#9336), [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253), Restrict google-cloud-texttospeech to
Sitemap 17