Localstack

Major Changes

This release introduces three major changes

* A Debian-based multi-platform Docker image of LocalStack, with full support for amd64 and experimental support for aarch64 (see 4921 for known limitations and progress)
* Upgrade to Python 3.8
* Lazy-loading of services as the default behavior

Debian-based multi-platform (amd64, arm64) docker image

4754 introduced a multi-platform docker build to support both amd64 and arm64. Alpine and musl made things unnecessarily difficult, so we switched to a Debian buster base image. Post-init (e.g., in `/docker-entrypoint-initaws.d`) scripts that expect an Alpine environment (e.g., use `apk` commands to install things into the LocalStack container) will stop working.

Lazy-loading of services

As part of an ongoing effort to improve startup performance, we have introduced a new code loading and service startup mechanism to allow lazy-loading of services. After LocalStack starts, services used to be in the state `running`, indicating that they are loaded and ready to serve requests. To allow more fine-grained state handling for services with lazy loading, we have introduced a new type of state called `available`.
When starting LocalStack, services are by default in the state `available`, which means that clients can start making requests to the services, and at the first time the service is hit, all the necessary code is loaded and the backend service is started, at which point the service after which the service will be in the `running` state. The environment variable `EAGER_SERVICE_LOADING` controls this behavior.

Scripts that depend on the output of `/health` being `running` will require setting setting `EAGER_SERVICE_LOADING=1`, which will start all services eagerly. This works together with the `SERVICES` variable to create the behavior from localstack <= 0.13.0.

(An alternative solution is to update your startup logic to accept either `running` or `available` as the healthy service state - e.g., see details in [this issue](https://github.com/localstack/localstack/issues/4904)).

Change log

1. New Features

* make lazy service loading the default behavior
* debian-based multi-platform support
- multi-platform support in localstack codebase
- finalize multi-platform build
- initial version of multi-platform-build
* support multiple concurrent elasticsearch clusters:
- implement one cluster per domain-support for elasticsearch
- implement custom endpoint routing for elasticsearch clusters
* add simple UI for deploying CloudFormation stacks from public template URLs
* add delete support for AWS::SSM::Parameter CloudFormation resources
* add initial support for Firehose processors to transform records written to targets
* add ability to short circuit wait/retries

2. Enhancements

* use serving.Server abstraction for ElasticMQ server to resolve startup issues
* add Werkzeug>=2.0 as requirement
* use werkzeug Request object in asf
* use werkzeug Response object in asf
* invalidate pro test cache for changes in both
* bump moto version to 2.2.5
* minor: correct typos in README
* add support for {"exists": false} in SNS message filtering
* add support for CFN resource AWS::Logs::LogStream
* minor: adjust API GW logic and tests for latest upstream moto changes
* convert boolean strings to lowercase in IAM responses for JS SDK compatibility
* add LEGACY_DOCKER_CLIENT and EAGER_SERVICE_LOADING to CONFIG_ENV_VARS
* minor: add property to API GW resource context class to extract identity information
* refactor logic for pulling specific Docker image tags, add more tests
* implement generic approach for routing internal REST resources
* extend logic and add more tests for escapeJavaScript(..) velocity template function
* remove custom parameter by label filtering, leverag logic from upstream
* upgrade moto-ext dependency to latest version
* Format JSON stack template for better readability in UI
* minor: remove print(..) statement
* support ranges of host ports to be mapped to a container port
* add missing attributes in Firehose records sent to destinations

3. Bug Fixes

* minor fix to deal with missing Tags attribute in EC2::RouteTable
* fix filtering by tags when fetching state of EC2::RouteTable CloudFormation resources
* fix URL encoding to properly handle '(none)' base paths in API Gateway
* fix arn parsing in aws_stack
* minor: fix API GW test assertion after recent moto upstream changes (part 2)
* minor: fix API GW test assertion after recent moto upstream changes
* minor fixes in Lambda and API Gateway logging / error handling
* fix extracting attributes from Firehose records
* revert moto to version 2.2.4 to fix builds temporarily

**1. New Features**

* major: implement lazy-loading of services
* introduce ApiInvocationContext class to encapsulate API Gateway request context information
* add AWS Service Framework parsers, serializers, and tests
* add AWS Service Framework API and scaffolding CLI
* add initial lifecycle hook abstraction for services
* add `OUTBOUND_HTTP_PROXY` option for proxies when downloading external resources
* add temporary support for python3.9 runtimes via mlupin/docker-lambda images
* add support for more Elasticsearch versions

**2. Enhancements**

* add object-oriented moto server abstraction
* update moto patch to fix CDATA wrapper for SQS message attribute values
* add option to set permissions for save_file(..), save local config file with 0o600 permissions
* upgrade Terraform version to fix CI builds
* refactor API GW logic for easier extraction of resource/method details from invocation context
* add test-invoke-method to test API Gateway endpoints
* add requestId attribute to Lambda event from API Gateway integration
* add more SQS tests to prepare for new SQS implementation
* refactor DynamoDB to use serving.Server
* update Elasticsearch default version in es_api.py
* support version qualifiers for managing Lambda policy statements
* correctly format Lambda function inline code on CloudFormation update
* refactor serving.Server to fix ElasticsearchCluster
* add headers from S3 object and correct codes to S3 website responses
* add defaults for SSM parameter name in CloudFormation model
* extend create_zip_file(..) to support 'append' mode and specifying root directory
* update default/cached Elasticsearch version to 7.10
* extract correct region from target ARN for S3 notifications
* add ability to skip content gzipping for individual services responses
* make boto client creation in aws_stack.connect_to_service thread safe
* add SSM->EventBridge integration to send notifications about parameter changes
* add concurrency control for multiserver startup
* add multi-region support for EventSourceListenerSQS
* add setup.cfg and update build configuration
* pin version of "pip" to fix setup step in CI
* add support for "::" to specify a custom Java Lambda handler method in local executor mode
* fix health endpoint to support feature states
* refactor extract_region_from_headers to make it more reusable
* add generic fallback health check for local APIs
* adjust exposed ports in docker-compose.yml
* add logic and tests to gracefully handle zipping of empty directories
* use Python time for do_run cache time comparison
* add support for routing events to log groups, fix event pattern matching/mapping for prefix filter
* prefix Lambda container names with MAIN_CONTAINER_NAME to support multiple instances in parallel
* allow partial SSE specification and auto-create managed KMS keys for DynamoDB tables
* update troubleshooting guide to not set DOCKER_HOST when using Lambda remote mode in CI
* minor: use existing util function to extract region from ARN
* minor: add generate_default_name_without_stack utility method

**3. Bug Fixes**

* fix docker_utils import in localstack status cli command
* fix AWS::SecretsManager::Secret CloudFormation ARN handling and tests
* fix restarting of DynamoDB process
* fix local execution of Java Lambdas if handler method is implementing `RequestHandler` interface
* fix issues in AWS service framework dispatching
* fix error response when attempting to delete non-existing Lambda
* fix scaffolding for remaining AWS services
* fix Elasticsearch plugin download when building docker base image
* fix check for None values in response multi_value_headers
* fix overwrite behavior for PUT requests to the health endpoint
* fix updates for AWS::IAM::Role in CloudFormation
* fix state checking for active service provider
* fix statement/policy handling for Events::EventBusPolicy CFN resource
* upgrade version of Terraform in base image to fix recent CI build issues

**1. New Features**

* add initial support for KMS asymmetric keys
* add initial support for KMS grant operations
* add support for and/or/condition CFN intrinsic functions
* add support for states:::events:putEvents to publish events from StepFunction
* add wildcard support for `EXTRA_CORS_ALLOWED_ORIGINS` configuration

**2. Enhancements**

* move attribute defaults from template_deployer to service models
* cache single instance of request context to enable mutability
* refactor Lambda API delete_function logic for better extensibility
* pass ENABLE_CONFIG_UPDATES environment variable to Docker container from CLI
* support case-sensitive response headers, add ETag to CORS exposed headers for S3 JS SDK compatibility
* add pagination to get_lambda_log_events(..) util function to fix Lambda tests
* log emails sent via SES send_raw_email into local log file
* add ability to pass invocation handler to test HTTP server for API GW integration tests
* add missing event attributes and increase timeout in Java Lambda tests
* rename docker->docker_utils to avoid import/aliasing issues
* minor refactoring of inter-service message forwarding for better extensibility
* add simple util classes ObjectIdHashComparator and ArbitraryAccessObj
* add helper function to mock a request context with a particular region
* refactor and remove fix_resource_props_for_sdk_deployment(..) in CFN deployer
* move delete patches in template deployer to CFN service models
* refactor error handling and add MOCK_UNIMPLEMENTED config to gracefully handle unimplemented APIs
* add ability to pass --workdir to launched containers in Docker client
* allow Lambda plugins to return invocation results directly, create plugin to handle LAMBDA_FORWARD_URL
* increase ElasticSearch cluster startup timeout
* skip forwarding Lambda invocation result if on_successful_invocation is None
* add missing edge routing logic for AWS SDK v2
* add support for s3:ObjectCreated:Copy event in the presence of X-Amz-Copy-Source headers
* add Docker client methods to inspect networks
* wait until port is closed on DynamoDB shutdown/restart
* pass CORS configuration options and Thundra Node.js/Python agent config from CLI to the container
* add util function to restart StepFunctions service process
* minor: check for empty files in download_and_extract(..) util function
* implement PluginDisabled exception to handle disabled plugins
* refactor DDBStreams/ES/Firehose APIs to use RegionBackend classes

**3. Bug Fixes**

* fix potential race condition in Docker container start/wait logic
* fix CFN SecretsManager::Secret to properly return secret ARN as Ref
* fix CFN resource type lookup
* fix deploying Lambdas with colliding names due to common prefixes
* fix issue with single quotes in payload for Node.js Lambda in local execution mode
* fix deployment of DynamoDB table with SSE disabled in Terraform definition
* fix selection of sub-elements in SQS listener XML responses
* fix Lambda SNS event source attribute as `aws:sns` instead of `localstack:sns`
* minor fix to avoid running multiple edge proxies on the same port

**1. New Features**

* add generic LocalStack plugin framework
* implement CLI using click, rich, and stevedore
* add Node.js/Python/java8.al2 runtime support for Thundra plugin
* add extensible class abstraction for Lambda event source listeners
* introduce Lambda executor plugin mechanism to hook in custom extensions at runtime
* add support for local execution of Go Lambda functions
* add support for generated secrets in CFN SecretsManager::Secret
* add initial version of Config Service API

**2. Enhancements**

* make some integration tests more lenient to environment changes
* add attributes to SQS messages from Lambda event source mapping
* add support for LS_LOG=trace-internal log level
* add check if DNS_ADDRESS is zero before starting the DNS server
* consider function alias when resolving Lambda function or layer ARN
* refactor Lambda executor methods to use InvocationContext as input param
* move pre- and post-create CFN actions from template_deployer to resource models
* extend util function for creating simple SSL proxy servers
* add plugin_spec to signatures of PluginLifecycleListener
* move API imports in infra.py to respective start methods
* add missing SSEDescription and TableId entries in DynamoDB DescribeTable calls
* add ability to specify --network via DOCKER_FLAGS config
* add util function to manually configure region for current request context
* enable sync/async SFN invocations in API Gateway integration
* remove obsolete patches for Logs metric_filters
* add test for deletion of SQS messages with changed visibility
* parse volume mount flag with regex to allow windows paths and additional arguments
* support `<apiId>.execute-api.*` hostnames for API Gateway invocation URLs
* distinguish polling and reactive mode in EventSourceListener
* add CloudFormation support for AWS::Events::EventBusPolicy
* create SQS queue policy for AWS::SQS::QueuePolicy CFN resource
* add support for DefinitionSubstitutions in CFN StateMachine resources
* add support for ApiGateway::DomainName and ::BasePathMapping CFN resources
* add get_physical_resource_id() for ApiGateway::Deployment
* add ability to specify SelfManagedEventSource for CFN Lambda::EventSourceMapping
* update kinesis-mock to 0.2.0, adding built-in multi-region support
* add startup logic to install prebuilt SSL cert if available
* update setup.py with additional information
* run Golang local test only if executed in Alpine
* run LocalStack tests against LocalStack Pro in Github Action
* rework some integration tests to make them more robust
* restructure Python requirements, Dockerfile and Makefile
* add `LAMBDA_CODE_EXTRACT_TIME` to configure max Lambda code extract wait time
* add default name for IAM::ManagedPolicy CFN resource
* aggregate valid regions accross all available partitions, to support regions like us-gov-*
* disable cli banner when running infra in docker
* move code in RESOURCE_TO_FUNCTION into service models, split into separate files
* add exception handling to EventBridge notifications if targets are unaccessible
* inject AWS_ENDPOINT_URL into Lambda container environment
* add plug points for code/config initializations in LambdaExecutorPlugin
* refactor `testutil.py` to deploy large Lambda zip files via S3
* add separate section with service provider configurations to README
* disable shell mode in run(..) util if cmd is passed as a list
* pass Cognito Identity info through HTTP headers, return `InvocationResult` with content instead of wrapping as Flask response for LAMBDA_FORWARD_URL
* add util functions get_edge_port_http() and is_provided_runtime(..)
* add localstack infra cli command for backwards compatibility
* add test assertion to verify value of maxReceiveCount in SQS RedrivePolicy
* extend logic to apply request/response templates in API GW messages
* add support for synchronous invocations of StepFunctions via API GW
* support `--add-host` argument in `LAMBDA_DOCKER_FLAGS`
* add simple integration tests for Support API
* add utility function to parse properties/INI config files
* add restart method for Kinesis and DynamoDB processes
* add util function to inject specific region into AWS Authorization header
* inject missing Lambda environment variables in local Lambda executor
* add resource id to auto-generated Lambda name in CFN stacks
* add warning message for LAMBDA_EXECUTOR=docker* if Docker is not accessible
* consolidate .coveragerc and .flake8 config int pyproject.toml
* minor: remove references to PORT_WEB_UI in docs
* minor: add util function to extract access_key_id from auth header, add type hints to utils
* minor: expose private Docker client methods as static methods in Docker Util class

**3. Bug Fixes**

* fix logic and add test for edge invocations from AWS Go SDK v2
* fix CFN list_stack_resources() for stacks with deleted resources
* fix ApiKeyRequired in API Gateway CloudFormation resources
* fix code updates of Lambda functions for Docker executor
* fix EC2/Logs tests after upstream changes
* fix raw message delivery attribute for SNS/SQS subscriptions
* fix documenting TMPDIR and HOST_TMP_FOLDER in docker-compose
* fix getting IP addresses of Docker containers with non-default networks
* fix template deployer and tests to retrieve updated PhysicalResourceId for EC2::Instance
* fix spaces in JSON events passed to custom runtime
* fix AWS::ApiGateway::Account regression
* fix argument error in exception handling in lambda_api.py
* fix adding tags to SQS queues when value is an empty string
* fix missing request parameters in CFN ApiGateway integration
* fix responding to S3 website requests with proper object content-type
* minor fix in regex to parse Docker volume flags
* restructure and clean up README.md

**1. New Features**

* security fixes: replace Docker CLI commands with Docker Python SDK client
* add support for multi-region request handling
* add ability to specify custom IDs for API GW REST APIs via tags
* add CloudFormation support for Events::Connection
* add EDGE_FORWARD_URL option to forward all edge requests

**2. Enhancements**

* update README and coverage docs with latest available services
* canonicalize boolean strings in SNS FIFO topic creation via CloudFormation
* canonicalize HTTP header names in API GW Lambda events to lower-case
* add basic interface for StateSerializer to be used in persistence mechanism
* add support for Route53 GetHealthCheck
* minor refactoring of imports to fix CLI dependency issues
* deploy extended attributes for IAM::User CloudFormation resources
* add ability to run privileged ports 53/443 in local dev mode
* return S3 bucket hostname for "DomainName" CF attribute
* restructure Thundra plugin code and fetch latest Java agent version from metatata xml in repository
* add support for IAM::User CloudFormation resources
* add missing CloudFormation properties for SNS::Topic
* update download URL for patched DynamoDB in base Docker image
* add fallback for boolean type conversion in CloudFormation deployments
* update analytics event logging system
* add Thundra config variables to `config.CONFIG_ENV_VARS` to pass them to Docker instance
* refactor logic across the codebase to make use of the Docker client utils
* extend merge_recursive(..) util function
* introduce KMS_PROVIDER env variable and switch to moto as default KMS provider
* add cleanup of docker-reuse Lambda containers if code/configuration is updated

**3. Bug Fixes**

* fix logic and tests for decimals formatting
* fix removing X-amz-content-sha256 header after stripping off chunk signatures
* fix/revert lookup of OS user inside Docker container
* fix patches in Logs API to accommodate recent upstream changes
* fix KMS listener initialization
* fix extracting event bus name as Ref attribute from CF Events::EventBus resources
* fix Kinesis CBOR response encoding for Java SDK compatibility, extend tests
* minor fix for handling symlink targets in cp_r(..)
* minor fixes to enable deployment of arc.codes sample app
* minor fix: consider Flask&Requests responses in aws_responses utils

**1. New Features**

* add integration for Thundra tracing via `THUNDRA_APIKEY`
* add support for local execution of Node.js lambdas
* add support for DeleteAlias and TracingConfig in Lambda API
* support sequential Lambda execution for Kinesis streams, using ParallelizationFactor
* support resolving SSM parameter values in CloudFormation templates
* add possibility for services to subscribe to runtime config changes
* add `TEST_TARGET=AWS_CLOUD` option to run tests against real AWS

**2. Enhancements**

* update README with instructions to build the Docker image
* add integration test for EC2 route table association
* adjust updated download URL for patched DynamoDBLocal.zip
* update kinesis-mock version to fix missing Logger classes
* store SES templated email messages to DATA_DIR or tmp folder for easier debugging
* refactor and extend AWS response parsing utils
* remove Web dashboard process from supervisord.conf
* extend test utils to pass in an auth_creator_func(..) for creating API GW authorizers
* minor: remove deprecated web UI documentation
* add option to disable boto3 retries/timeouts
* remove EventBus proxy listener patches for Tags operations handled upstream
* integrate and use the util function to remove Docker env files
* print version string if Terraform version check fails in integration tests
* replace `unparse` with custom XML serializer for AWS responses
* support CloudFormation string placeholders with triples like `"${resource.attr.subattr}"`
* improve kinesis startup routine and add KINESIS_MOCK_FORCE_JAVA config option
* replace botostubs with boto3-stubs in test fixtures
* add retry logic for downloading Elasticsearch plugins to fix temporary network issues
* extend ACM ARN integration test when importing certificates
* improve infra graph code, extend tests, add new fixtures
* add tests for Lambda invocation destinations
* add unit tests for common and config
* add JSON safe parsing when processing API GW event in local Javascript Lambda
* patch moto to include region header for GET and HEAD requests
* update CI workflow to aggregate coverage report
* use JAVA_TOOL_OPTIONS to apply LAMBDA_JAVA_OPTS in Java Lambdas
* cleanup and refactor CI configurations
* add default account_id for API GW Lambda invocation events
* add ability to define UDP Docker ports in PortMappings class
* strip off warning messages when extracting Docker container ID from stdout
* add exception response when calling DeletePolicy on non-existent IAM policy
* refactor Lambda executor to use unified container API and break apart container/Lambda logic
* pin version of "requests" library to fix import errors in Lambda tests
* add feature that saves LocalStack version info into data dir
* add pre-commit hook that runs linting and formatting
* add support for CloudFormation Fn::GetAtt WebsiteURL for S3 buckets
* refactor JSONPatch implementation for API GW to handle edge cases more reliably
* add make targets for linting and formatting only modified files
* add proper headers when triggering API destination target
* add CORS headers required for AWS SDK v3
* refactor Firehose and properly return "ExtendedS3DestinationDescription" with extended attributes
* add support for API Gateway documentation parts (tf-compat)

**3. Bug Fixes**

* fix regex to extract API ID in hostname from API GW endpoint URL
* fix logic in inject_test_credentials_into_env(..) to set custom env parameters for Lambda functions
* fix DynamoDB to Kinesis stream event format - add "tableName", remove "StreamViewType"
* fix Docker detection on hosts with cgroup v2 (unified hierachy)
* fix running multiple API Gateway JSON Patch operations in sequence (tf-compat)
* fix timestamp format of Lambda event source mappings for TF compatibility
* fix returning error response when requesting ACL for non-existing S3 bucket
* fix edge routing to match rules if request host contains a dot "."
* fix EXTRA_CORS_ALLOWED_ORIGINS environment variable to match docs
* fix ARN structure for EventBridge resources
* fix extraction of EventBridge API destination name using new ARN format
* fix S3 host-based routing detection
* fix accessing missing "body" from Lambda response in API GW
* fix JSON Patch for API GW method settings
* fix some data types for API GW patch operations
* fix "application/xml" content-type header for S3 responses
* minor: fix file path in "config validate" CLI command