Deployment

The service runs as one container on the AWS Lightsail container service diaspora-web-service in us-east-1. The deployment workflow (.github/workflows/deployment.yml) builds the image from web_service/Dockerfile and deploys it. Parts of the workflow are adapted from fdiesel/github-action-deploy-aws-lightsail-container. Rollback, logs, triage and credential rotation are in the runbook.

In this page a deploy is one run of the workflow, and a deployment is what Lightsail runs: a numbered version that names the container, its image and its environment.

What triggers a deploy

• A push to main, which includes merging a pull request. A push whose changed files are all under docs/, are Markdown files, or are mkdocs.yml does not deploy; a push that changes code and docs together does.
• A manual run of the workflow (see Deploy by hand).

Nothing else deploys: not pull requests, other branches, tags or a schedule.

Deploys run one at a time in the production concurrency group. A deploy in progress is never cancelled. While one runs, GitHub keeps only the newest waiting run and cancels older waiting runs. After several quick merges, the running deploy finishes, and then the newest waiting run deploys the latest main.

What the workflow does

1. The unit job runs the unit tests (the reusable unit-tests.yml workflow, without secrets; see Testing). If a test fails, nothing is deployed.
2. The deploy-lightsail-containers job then runs these steps:
   1. It configures AWS credentials from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY repository secrets for us-east-1, and checks them with aws sts get-caller-identity.
   2. It installs the Lightsail CLI plugin (lightsailctl), which aws lightsail push-container-image needs.
   3. It checks out the commit and builds the image from web_service/Dockerfile, tagged docker.
   4. It creates the container service with power micro (0.25 vCPU, 1 GB of memory) and scale 1. When the service already exists, this step fails and the job continues.
   5. It pushes the image with the label docker. Lightsail stores it as :diaspora-web-service.docker.<N>, where <N> grows by one with each push.
   6. It creates a deployment with one container named flask that runs the new image, listens on port 8000 over HTTP and is the public endpoint. The container's environment comes from the DOCKER_* secrets (Secrets). Each deployment carries its whole environment, so a variable the workflow does not pass is unset in the container.
   7. It waits for the deployment to become active (The wait step).
   8. It deletes the stored images that no deployment still needs. It keeps the images of the active deployment, of any deployment still activating, and of the most recent inactive deployment, which is the rollback target. This step runs only after the wait step succeeds, and an error in it does not fail the run.

The wait step

The step reads the version number of the new deployment, then checks the service every 30 seconds. It succeeds when that version is the current deployment and its state is ACTIVE. While a new deployment activates, Lightsail keeps the previous one serving; if the new one fails, the previous one stays active.

Error in the run log	What it usually means	What to do
No pending deployment found for diaspora-web-service	Right after the create call, Lightsail showed no pending deployment.	Check the service's state (Check a deploy) and the previous step's log.
Deployment <version> failed; check the container logs	Lightsail marked the deployment FAILED: the container did not start, or did not pass Lightsail's health check, which requests /. A missing environment variable is one cause.	Read the container log and triage the failure.
Deployment <version> is no longer pending and did not become active	The version is neither current nor pending. Another deployment, made from the console or the CLI while the workflow waited, can replace it; workflow runs do not overlap, so it was not another run.	List the deployments and their states (Roll back a deploy shows the command).
Deployment <version> is not active after 20 minutes	Lightsail was still activating the deployment when the step stopped waiting. It can still become active or fail.	Check the service's state and the container log. The image cleanup did not run.

Secrets

The deploy uses two sets of repository secrets. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are the workflow's own AWS key. It needs access to the Lightsail container service only, and the container never sees it. The DOCKER_* secrets become the container's environment, without the prefix. The live job of tests.yml, whose last step is the nightly sweep, reads the same DOCKER_* secrets (the AWS key only while the live tests' own key is not set), so a change to one of them also reaches the live tests.

Secret	Container variable	Notes
AWS_ACCESS_KEY_ID	none	The deploy's AWS access key ID.
AWS_SECRET_ACCESS_KEY	none	The deploy's AWS secret access key.
DOCKER_AWS_ACCESS_KEY_ID	AWS_ACCESS_KEY_ID	Required.
DOCKER_AWS_SECRET_ACCESS_KEY	AWS_SECRET_ACCESS_KEY	Required.
DOCKER_SERVER_CLIENT_ID	SERVER_CLIENT_ID	Required.
DOCKER_SERVER_SECRET	SERVER_SECRET	Required.
DOCKER_AWS_ACCOUNT_ID	AWS_ACCOUNT_ID	Required.
DOCKER_AWS_ACCOUNT_REGION	AWS_ACCOUNT_REGION	Required.
DOCKER_MSK_CLUSTER_NAME	MSK_CLUSTER_NAME	Required.
DOCKER_DEFAULT_SERVERS	DEFAULT_SERVERS	Needed for any Kafka call.
DOCKER_CLIENT_ID	CLIENT_ID	Unused by the service. tests.yml runs the live tests only when this secret is set, so runs without the repository's secrets skip them.
DOCKER_CLIENT_SECRET	CLIENT_SECRET	Unused.
DOCKER_CLIENT_SCOPE	CLIENT_SCOPE	Unused.
DOCKER_MSK_CLUSTER_ARN_SUFFIX	MSK_CLUSTER_ARN_SUFFIX	Unused.

Container environment

The service checks seven variables when web_service.main is imported, which happens when the gunicorn worker starts. Each must be set and non-empty. A missing one stops the worker with AssertionError: <NAME> environment variable is not set., and Lightsail marks the deployment failed.

Variable	Required	Default	What it sets
AWS_ACCESS_KEY_ID	yes		The AWS key the service uses for IAM, DynamoDB and MSK IAM authentication.
AWS_SECRET_ACCESS_KEY	yes		The secret of that key.
SERVER_CLIENT_ID	yes		The ID of the service's Globus confidential client. Every token must name it in its audience and carry the scope https://auth.globus.org/scopes/<SERVER_CLIENT_ID>/action_all.
SERVER_SECRET	yes		The secret of that client, used to introspect tokens.
AWS_ACCOUNT_ID	yes		The account in the ARNs of the IAM policies the service writes.
AWS_ACCOUNT_REGION	yes		The region of the DynamoDB tables and the MSK cluster.
MSK_CLUSTER_NAME	yes		The cluster named in those IAM policies.
DEFAULT_SERVERS	no	none	The MSK bootstrap brokers for IAM authentication, as comma-separated host:port pairs. Without it the service starts, but topic creates and deletes change only the service's own records and not the cluster, and topic recreates and consumer creates fail with Kafka endpoint is not configured.
KEYS_TABLE_NAME	no	diaspora-keys-table	The DynamoDB table of access keys.
USERS_TABLE_NAME	no	diaspora-users-table	The DynamoDB table of users.
NAMESPACE_TABLE_NAME	no	diaspora-namespace-table	The DynamoDB table of namespaces.
MAX_TOPICS_PER_NAMESPACE	no	50	The most topics a namespace may hold. A value that is not a whole number of at least 1 is logged and replaced by 50.
CONSUMER_INSTANCE_TIMEOUT_MS	no	300000 (5 minutes)	How long a consumer instance may sit idle before the service closes it. The service looks for idle instances every 30 seconds. A value that is not an integer stops the worker at start with a ValueError.
IAM_USER_PERMISSIONS_BOUNDARY_ARN	no	none	The ARN of a managed policy that the service sets as the permissions boundary of every IAM user it creates; unset or empty, it sets none. When the service runs as its own identity (The service's AWS identity), it must be that identity's boundary, or IAM denies every user the service creates; the workflow sets it from a repository variable (The boundary in the container). The live tests set it to theirs (The live tests' AWS identity).

The workflow passes none of the other optional variables, so production runs with their defaults. To set one, add it to the environment block in deployment.yml. It sets IAM_USER_PERMISSIONS_BOUNDARY_ARN only when the repository variable DOCKER_IAM_USER_PERMISSIONS_BOUNDARY_NAME is set (The boundary in the container).

The service's AWS identity

The container's AWS key must belong to an IAM user whose policy lets the service reach only what it creates and uses: the IAM users and policies it names, its three DynamoDB tables (diaspora-keys-table, diaspora-users-table and diaspora-namespace-table) and the MSK cluster. Two templates in deploy/iam/ hold that user's policies, with <account-id>, <region>, <cluster-name> and <cluster-uuid> in place of the account's values:

• service-policy.json, the policy of the IAM user /diaspora-service/diaspora-web-service-container, created as the managed policy /diaspora-service/diaspora-web-service-container.
• user-boundary-policy.json, created as /diaspora-service/msk-user-boundary: the permissions boundary of every IAM user the service creates. It allows the kafka-cluster actions that the user policy (IAMService.generate_user_policy) grants, on the cluster, and nothing else. A user with this boundary can do no more than the boundary allows, whatever policy is attached to the user.

In the table, <subject> is a name shaped like a Globus subject, a UUID (????????-????-????-????-???????????? in the policy). The service names each user and its policy after the user's subject, so the policy reaches no other name, and no user or policy under a deeper path such as /msk-users/x/<subject>. The live tests' test-subject-* users are out of its reach too. The service's policy grants these actions, each used by the code named (in web_service/services.py unless another file is named):

Actions	On	Used by
iam:CreateUser, only with the boundary as the new user's permissions boundary	IAM users /msk-users/<subject>	IAMService.create_user_and_policy
iam:CreatePolicy	managed policies /msk-policy/<subject>	IAMService.create_user_and_policy
iam:AttachUserPolicy, only of a policy /msk-policy/<subject> and only to a user whose boundary is the boundary	IAM users /msk-users/<subject>	IAMService.create_user_and_policy
iam:CreateAccessKey	IAM users /msk-users/<subject>	IAMService.create_access_key
iam:ListAccessKeys, iam:DeleteAccessKey	IAM users /msk-users/<subject>	IAMService.delete_access_keys
iam:DetachUserPolicy, only of a policy /msk-policy/<subject>; iam:DeleteUser	IAM users /msk-users/<subject>	IAMService.delete_user_and_policy
iam:ListPolicyVersions, iam:DeletePolicyVersion, iam:DeletePolicy	managed policies /msk-policy/<subject>	IAMService.delete_user_and_policy
dynamodb:GetItem	the three tables	DynamoDBService._get_item, which every read goes through
dynamodb:PutItem	the three tables	DynamoDBService.store_key
dynamodb:UpdateItem	the three tables	DynamoDBService.add_user_namespace, remove_user_namespace, add_global_namespace, remove_global_namespace, add_namespace_topic, remove_namespace_topic
dynamodb:DeleteItem	the three tables	DynamoDBService.delete_key, delete_namespace_topics
dynamodb:CreateTable, dynamodb:DescribeTable	the three tables	DynamoDBService._create_keys_table, _create_users_table, _create_namespace_table, when a table is missing; DescribeTable is boto3's table_exists waiter
kafka-cluster:Connect	the cluster	every Kafka client: KafkaService._admin_client, each consumer instance (consumer_service.py) and the producer (producer_service.py)
kafka-cluster:CreateTopic, kafka-cluster:DeleteTopic	every topic	KafkaService.create_topic, delete_topic, recreate_topic
kafka-cluster:DescribeTopic	every topic	the topic deletes, the producer's sends and metadata, and the consumer instances' subscribes, fetches, committed offsets and seeks to the beginning or end
kafka-cluster:WriteData	every topic	the produce routes (ProducerService)
kafka-cluster:ReadData	every topic	the consumer instances' fetches and commits (consumer_service.py)
kafka-cluster:DescribeGroup, kafka-cluster:AlterGroup	every consumer group	the consumer instances, which join groups, commit offsets, read committed offsets and leave their group when closed (consumer_service.py)

A namespace may have any valid name, so the service's topics and groups (<namespace>.<name>) can have any name, and the Kafka actions cover every topic and group on the cluster. The service makes no other AWS call: the MSK token is signed locally from the key, with no STS call, and the brokers come from DEFAULT_SERVERS. The template names the production tables; a deployment with other *_TABLE_NAME values needs those names in its copy of the template.

deploy/iam/service-actions.json lists each of these actions with an example resource and the calls in web_service/ that need it. tests/service_identity_test.py checks that the policy allows each listed use and grants nothing more, that the IAM and DynamoDB uses are the calls in web_service/services.py, and that each Kafka call in the code has its actions listed.

The policy grants no right to change a user's permissions boundary, add an inline policy, add or select a version of a policy, or touch anything under /diaspora-service/, its own policy and the boundary included. So with the container's key alone, no one can give a user more than the boundary allows, or give the service more than this policy.

The boundary in the container

The workflow sets IAM_USER_PERMISSIONS_BOUNDARY_ARN only when the repository variable DOCKER_IAM_USER_PERMISSIONS_BOUNDARY_NAME is set. Its value is a policy name under /diaspora-service/, msk-user-boundary, and the workflow builds the ARN arn:aws:iam::<account-id>:policy/diaspora-service/msk-user-boundary from it and the account ID in the DOCKER_AWS_ACCOUNT_ID secret. While the variable is not set, the container gets no such variable and creates users without a boundary. A value that is not a valid policy name fails the deploy before anything is pushed.

Users created before the boundary

IAM users the service created while it set no boundary have none. The service's identity can still delete them, since detaching a policy and deleting a user do not require the boundary. It can't attach a policy to them, and POST /api/v3/key attaches the user's policy again whenever it creates a key. So before the container switches to this identity, every user under /msk-users/ needs the boundary. The boundaries step of scripts/update_user_policies.py (Update the users' IAM) gives it to the service's users. Delete the live tests' leftover users, which it skips, with the sweep (scripts/sweep_test_resources.py --apply). A user under /msk-users/ named neither as a subject nor as a test user is skipped by both: delete it, or give it the boundary by hand with aws iam put-user-permissions-boundary.

The boundaries step checks only that the boundary exists. Check first that its default version matches deploy/iam/user-boundary-policy.json.

Switch the service to its identity

Run these with an administrator's credentials. Fill in the values as in Create the identity, which also shows the render function used below.

1. Create the boundary, the service's policy, the user and its access key:

   aws iam create-policy --path /diaspora-service/ \
     --policy-name msk-user-boundary \
     --policy-document "$(render deploy/iam/user-boundary-policy.json)"
   aws iam create-policy --path /diaspora-service/ \
     --policy-name diaspora-web-service-container \
     --policy-document "$(render deploy/iam/service-policy.json)"
   aws iam create-user --path /diaspora-service/ \
     --user-name diaspora-web-service-container
   aws iam attach-user-policy --user-name diaspora-web-service-container \
     --policy-arn "arn:aws:iam::$ACCOUNT_ID:policy/diaspora-service/diaspora-web-service-container"
   (umask 077; aws iam create-access-key \
     --user-name diaspora-web-service-container \
     --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text \
     > ~/web-service-key.txt)
   

   The file holds the key ID and its secret, which AWS returns only here; nothing is printed. Step 6 stores them, and you delete the file after it.

2. Set the boundary variable and deploy. From then on every new user gets the boundary.

   gh variable set DOCKER_IAM_USER_PERMISSIONS_BOUNDARY_NAME --body msk-user-boundary
   gh workflow run deployment.yml --ref main
   

3. Give the users created before it the boundary. The first command only lists what would change. The second makes the changes and records each one in a new file outside the repository. The third must report 0 users without a boundary.

   BOUNDARY="arn:aws:iam::$ACCOUNT_ID:policy/diaspora-service/msk-user-boundary"
   python scripts/update_user_policies.py boundaries --boundary "$BOUNDARY"
   python scripts/update_user_policies.py boundaries --boundary "$BOUNDARY" \
     --apply --record ~/boundaries-$(date -u +%Y%m%dT%H%M%SZ).json
   python scripts/update_user_policies.py boundaries --boundary "$BOUNDARY"
   

4. If the live tests' own key is not set yet, set it first (The live tests' AWS identity). Without it, the live job of tests.yml uses the container's key, and the live tests fail, since this identity cannot reach the test tables.

5. Note the container's current key ID, for the rollback and for retiring that key later. The query prints the key ID only:

   aws lightsail get-container-services --region us-east-1 \
       --service-name diaspora-web-service \
       --query 'containerServices[0].currentDeployment.{version: version, key: containers.flask.environment.AWS_ACCESS_KEY_ID}'
   

6. Store the new key as DOCKER_AWS_ACCESS_KEY_ID and DOCKER_AWS_SECRET_ACCESS_KEY, delete the file, deploy, and check the deploy:

   cut -f1 ~/web-service-key.txt | tr -d '\n' | gh secret set DOCKER_AWS_ACCESS_KEY_ID
   cut -f2 ~/web-service-key.txt | tr -d '\n' | gh secret set DOCKER_AWS_SECRET_ACCESS_KEY
   rm ~/web-service-key.txt
   gh workflow run deployment.yml --ref main
   

   Then make one call that creates a key, such as POST /api/v3/key for a user without one, and read the log for AccessDenied.

To change what the identity may do, edit the template and apply it as a new default version of its policy, as the live tests' guide shows for theirs. To rotate its key, follow Rotate an AWS access key with the user diaspora-web-service-container.

Roll back the switch

Put the previous key back in the two secrets and deploy. The boundary variable may stay: the boundary allows everything a user's policy grants, so the previous key works with it. GitHub does not show a secret's value, but the deployment noted in step 5 still holds the previous key, so these commands copy it from there without printing it (replace <version>):

previous() {
  aws lightsail get-container-service-deployments --region us-east-1 \
      --service-name diaspora-web-service \
      --query "deployments[?version==\`<version>\`].containers.flask.environment.$1 | [0]" \
      --output text
}
key_id=$(previous AWS_ACCESS_KEY_ID)
secret=$(previous AWS_SECRET_ACCESS_KEY)
if [ -z "$key_id" ] || [ "$key_id" = None ] || [ -z "$secret" ] || [ "$secret" = None ]; then
  echo "Deployment <version> holds no key; nothing was changed" >&2
else
  printf '%s' "$key_id" | gh secret set DOCKER_AWS_ACCESS_KEY_ID
  printf '%s' "$secret" | gh secret set DOCKER_AWS_SECRET_ACCESS_KEY
  gh workflow run deployment.yml --ref main
fi

The AWS CLI prints None when the deployment is not found, for example when <version> is mistyped or has left Lightsail's history, so the commands check both values before they change either secret.

If the previous key has been deactivated since, make it active first (aws iam update-access-key --status Active, with its user and key ID).

Retire the previous key

Once the service has run as its identity without an AccessDenied in the log, for at least a day that included user and key creations, deactivate the container's previous key, and delete it later. The deploy workflow's own key (AWS_ACCESS_KEY_ID) may be the same key; check before you deactivate it.

In the diaspora-web-service deployment it is the same key: the deploy's AWS_ACCESS_KEY_ID is the container's previous key, which belongs to an administrator. It stays active as the deploy's key and is not replaced, deactivated or deleted; only the container stopped using it.

1. Show the previous key's user and last use, with the key ID from step 5: aws iam get-access-key-last-used --access-key-id <previous-key-id>. No secret is printed.
2. Run a deploy (gh workflow run deployment.yml --ref main) and wait for it to finish. Then run step 1 again. If the key's LastUsedDate moved to the deploy's time, with ServiceName lightsail or sts, the deploy uses it too. Then give the deploy its own key first (Rotate an AWS access key), and repeat this check.
3. Deactivate the key: aws iam update-access-key --user-name <user> --access-key-id <previous-key-id> --status Inactive. Run a deploy and check it.
4. After a week without a failure, delete it: aws iam delete-access-key --user-name <user> --access-key-id <previous-key-id>.

One worker

The Dockerfile starts gunicorn without --workers, so it runs one Uvicorn worker, and the service runs at scale 1. Consumer instances live in that one process's memory, so the service must stay at one worker on one node. With a second worker or node, a request can reach a process that does not hold its instance and gets 404 with error code 40403 (Consumer instance not found). Do not pass --workers to gunicorn, do not set WEB_CONCURRENCY (gunicorn reads its worker count from it) or GUNICORN_CMD_ARGS (gunicorn reads extra options from it), and keep the scale at 1.

Every deploy or restart stops the process and drops every consumer instance. Clients then get 40403 and must create a new instance. A new instance in the same consumer group starts from the offsets the group last committed to Kafka.

When the container gets SIGTERM, gunicorn gives the worker 30 s (--graceful-timeout) to stop and then kills it. The worker, web_service.worker.Worker, stops accepting connections and waits up to 20 s for open requests, then cancels any still running. It then runs the service's shutdown, which closes the consumer instances and the producer in the remaining 10 s or so. A shutdown that takes longer is cut off when gunicorn kills the worker.

Deploy by hand

From GitHub

gh workflow run deployment.yml --ref main
gh run list --workflow deployment.yml --event workflow_dispatch --limit 1   # the run's ID
gh run watch <run-id>

A manual run goes through the same unit tests, queue, wait step and cleanup as a push. It deploys the branch named by --ref; a run on another branch is replaced by the next deploy of main.

From your machine

Use this only when GitHub Actions cannot run the workflow. It skips the unit tests, the wait step and the image cleanup. You need Docker, the AWS CLI with the Lightsail container services plugin, jq, and AWS credentials with the deploy's Lightsail access.

The steps copy the current deployment's container, environment included, and change only its image, so they need no secret values. If the container service does not exist, create it before step 1:

aws lightsail create-container-service --region us-east-1 \
    --service-name diaspora-web-service --power micro --scale 1

A new service has no deployment to copy, so step 3 must then set the environment by hand (Set the environment by hand). Its public URL is also likely to change: AWS builds a service's default domain from a randomly generated identifier of the service, and clients and these docs use the current domain.

1. Build the image from the repository root. The Dockerfile pins linux/amd64, the platform Lightsail runs.

   docker build -t docker -f web_service/Dockerfile \
       --build-arg GIT_SHA="$(git rev-parse HEAD)" .
   

2. Push it and keep the name Lightsail gives it:

   output=$(aws lightsail push-container-image --region us-east-1 \
       --service-name diaspora-web-service --label docker --image docker)
   image=$(echo "$output" | sed -n 's/.*Refer to this image as "\(.*\)" in deployments.*/\1/p')
   echo "$image"   # for example :diaspora-web-service.docker.<N>
   

3. Copy the current deployment's container and set its image. $containers then holds the container's secrets, so do not print it.

   containers=$(aws lightsail get-container-services --region us-east-1 \
       --service-name diaspora-web-service \
       --query 'containerServices[0].currentDeployment.containers' --output json |
     jq --arg image "$image" '.flask.image = $image')
   jq -e '.flask.environment' > /dev/null <<< "$containers" ||
     echo "No current deployment: set the environment by hand (below)."
   

4. Create the deployment. The command prints the deployment with its environment, so the example discards its output.

   aws lightsail create-container-service-deployment --region us-east-1 \
       --service-name diaspora-web-service \
       --containers "$containers" \
       --public-endpoint '{"containerName": "flask", "containerPort": 8000}' > /dev/null
   

   Keep the container name flask: the public endpoint and the logs refer to it.

5. Check the deploy.

Set the environment by hand

To change a variable, or to deploy to a service with no current deployment, read the values into the shell first. read -rs keeps each value off the screen and out of shell history. The SVC_ prefix keeps the values apart from the AWS CLI's own credentials. A value set this way lasts only until the next workflow deploy, which rebuilds the environment from the DOCKER_* secrets, deployment.yml and the repository variable DOCKER_IAM_USER_PERMISSIONS_BOUNDARY_NAME; update the matching secret or variable, too.

for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY SERVER_CLIENT_ID SERVER_SECRET \
    AWS_ACCOUNT_ID AWS_ACCOUNT_REGION MSK_CLUSTER_NAME DEFAULT_SERVERS; do
  printf '%s: ' "$var"; IFS= read -rs value; echo
  export "SVC_$var=$value"
done
unset value

To change one variable of the current deployment, run the loop with only that name (for var in MAX_TOPICS_PER_NAMESPACE; do) and extend the filter of step 3, for example jq --arg image "$image" '.flask.image = $image | .flask.environment.MAX_TOPICS_PER_NAMESPACE = env.SVC_MAX_TOPICS_PER_NAMESPACE'. For a service with no current deployment, build the whole container instead of step 3:

containers=$(jq -n --arg image "$image" '{
  flask: {
    image: $image,
    ports: {"8000": "HTTP"},
    environment: {
      AWS_ACCESS_KEY_ID: env.SVC_AWS_ACCESS_KEY_ID,
      AWS_SECRET_ACCESS_KEY: env.SVC_AWS_SECRET_ACCESS_KEY,
      SERVER_CLIENT_ID: env.SVC_SERVER_CLIENT_ID,
      SERVER_SECRET: env.SVC_SERVER_SECRET,
      AWS_ACCOUNT_ID: env.SVC_AWS_ACCOUNT_ID,
      AWS_ACCOUNT_REGION: env.SVC_AWS_ACCOUNT_REGION,
      MSK_CLUSTER_NAME: env.SVC_MSK_CLUSTER_NAME,
      DEFAULT_SERVERS: env.SVC_DEFAULT_SERVERS
    }
  }
}')

After the deployment is created, remove the values from the shell with unset "${!SVC_@}" in bash or unset -m 'SVC_*' in zsh.

Check a deploy

aws lightsail get-container-services --region us-east-1 \
    --service-name diaspora-web-service \
    --query 'containerServices[0].{state: state, current: currentDeployment.[version, state], image: currentDeployment.containers.flask.image, next: nextDeployment.[version, state]}'

A finished deploy shows the service RUNNING, the current deployment ACTIVE with the new image, and next as null. Keep the --query: without it, the command prints the container's environment, secrets included.

Then check that the service answers. It serves its API documentation at /, so this prints 200:

curl -sS -o /dev/null -w '%{http_code}\n' https://diaspora-web-service.qpp943wkvr7b2.us-east-1.cs.amazonlightsail.com/