Container Orchestration For Developers

Container orchestration for developers turns on a single property: each component of the suite survives being killed on one node and restarted on another, with no operator in the loop and no state lost. Satisfy that property and the remainder is configuration. Fail it and the scheduler does precisely what you asked, restarting a process that cannot rebuild the context it needs.

Ten steps take a running application to a suite a cluster can hold at a declared state. Each step ends with a check. Run the check before starting the next one.

1. Rebuild every component as a process with a port, a log stream and environment-driven configuration

The container interface is narrower than most teams assume. The SKAO container orchestration guidelines, written against the Container Network Interface, the Container Storage Interface, the Cloud Native Computing Foundation reference stack and the Open Container Initiative specifications, fix the contract: inputs arrive through a configurable port, outputs go to a configurable network service, logs go to stdout and stderr as structured JSON, metrics are advertised through a Prometheus exporter or as JSON over TCP, and configuration arrives as environment variables and simple key-value files. Anything read from a hardcoded absolute path, an unmounted host directory or a local SQLite file breaks the contract, because the process cannot reconstruct its own context after rescheduling. The same source defines the requirement that makes orchestration possible at all:

applications should have builtin recoverability so that prior state and context is automatically discovered on restart

Check: start the image with docker run --rm -p 8080:8080 -e LOG_LEVEL=debug image:tag, stop it, start it again with the same environment, and confirm it serves the same responses without a manual migration step.

2. Take state off the container filesystem

State belongs in a database, an object store or a mounted volume, never in the writable layer. Once the container is empty, two artefacts provide portability, and neither is the process inside it.

The image, packaged to the Open Container Initiative image specification and moved through a registry that speaks the OCI distribution specification, so a build produced by one tool is understood by another.
The descriptor, which names the image by digest, so the reference stays fixed between two rollouts of the same file.

The OCI Runtime Specification fixes the lifecycle operations of create, start, stop and delete, with runc as the reference implementation. That is why an image built with Podman runs unchanged under containerd and under the runtime inside a Kubernetes node. Pin the digest in every descriptor, since a tag points at whatever the registry currently contains.

Check: docker run --read-only --tmpfs /tmp image:tag starts and serves traffic. Any write that fails under a read-only root filesystem is a write that would be lost on rescheduling.

3. Write one descriptor per component in generic API types

The SKAO guidelines prefer YAML over JSON for human readability, and require the namespace to stay out of the metadata block, because placing it there reduces the flexibility of any templating layer applied on top of the file. Namespace selection happens at apply time: kubectl --namespace test apply -f resource-file.yaml. Descriptors stay declarative, stating the end state, and the scheduler moves the cluster towards it. Keep to generic API types. The same guidelines warn that third-party custom resource types reduce portability and increase dependency on integrations outside your control, which becomes upgrade paralysis when the software development life cycle of the operator is not your team's to schedule.

Check: apply the identical file into two namespaces and diff the resulting object sets. The only differences should be names that incorporate the namespace.

4. Declare Services before the Pods that consume them

Service names are permanent, predictable and scoped to a namespace: a MySQL instance named tangodb is reachable as tangodb.test from the test namespace and tangodb.qa from qa, with no change to application configuration. Declaring the Service first matters because environment variables for service discovery are generated at Pod creation time. A Pod created before its Service exists has no environment variables for it.

Check: kubectl exec -it deploy/api -- getent hosts billing returns an address from inside a running Pod.

5. Choose the controller from the guarantee you need

Kubernetes is the default target. The CNCF survey that Splunk's beginner's guide reports found 91% of respondents using Kubernetes, with 83% of those running it in production, against 78% a year earlier and 58% in 2018. The same guide sets the ceiling of one cluster at 5,000 nodes and 150,000 pods. The guarantees below come from the SKAO container orchestration guidelines, which use Kubernetes v1.16.2 with Helm v3.1.2 as their reference implementation.

Controller	Workload shape it fits	Naming guarantee	Volume guarantee	Ordering guarantee
Bare Pod	one instance bound to a specific device or node	the exact name you choose	mount only, no claim	none, replacement is a new Pod
Deployment	stateless services behind a Service	random suffix per replica	shared volumes only	none; replicas scale independently
StatefulSet	databases, queues, anything with a stable identity	stable ordinal names	stable persistent volume per replica	ordered graceful deployment and ordered rolling updates
DaemonSet	one instance per node, node agents, batch workers	per node	node-local	one Pod per scheduled node
Job and CronJob	run-to-completion work and scheduled launches	per run	per run	completion counted

Check: kubectl get pods -o custom-columns=NAME:.metadata.name. A random suffix on a database Pod means the controller choice was wrong.

6. Set requests and limits from measured usage

Google Cloud's account describes scaling as driven by CPU utilisation or custom metrics, with replica counts moving with load. A request with no limit lets one container starve its neighbours on the same node. A limit equal to the request converts every burst into CPU throttling. Measure steady-state usage over at least 14 days, set the request at the 95th percentile of that measurement, set the limit at twice the request, and point the autoscaler at whichever metric saturates first; for most request-serving services that metric is queue depth or latency.

Check: under peak load, kubectl top pod shows no Pod above 80% of its limit and no Pod below half its request, the headroom band the SKAO container orchestration guidelines set for scheduling.

7. Gate traffic on the readiness probe

Red Hat's account of pod behaviour notes that probes hold traffic from containers still booting while restarting frozen ones, and an aggressive liveness probe during startup restarts a process that was still initialising. Point the readiness probe at the port the Service targets. Set the initial delay above the 99th percentile of observed start time measured over 100 cold starts.

Check: kubectl rollout status deployment/api --timeout=180s completes, and kubectl get endpoints api shows at least one ready address at every point during the rollout.

8. Inject configuration and secrets at run time

Google Cloud documents ConfigMaps for configuration data and Secrets for passwords and API keys, both injected into the container environment so a configuration change arrives without an image rebuild. Configuration that only changes through a registry push turns a five-minute change into a release with a review, a build and a rollout. Wiz's container orchestration guide points to a secrets manager injecting values into the Pod at runtime, with the credential absent from every image layer.

Check: kubectl exec deploy/api -- env displays every key the manifest declares, and docker history --no-trunc image:tag shows no layer that wrote a credential to the filesystem.

9. Bound each workload's permissions and network reach

Wiz's guide identifies the controls that determine how far one workload can reach: least privilege enforced through identity and access management policies and role-based access control, network policies that restrict Pod-to-Pod traffic to the ingress and egress a service needs, mutual TLS between services, and secrets injected at runtime. Role-based access control is the mechanism that assigns specific permissions to users and service accounts, which is the knob that determines how much of the cluster a compromised Pod can read.

Check: kubectl auth can-i --list --as=system:serviceaccount:prod/api returns only the verbs the workload uses, and a connection from a Pod outside the network policy times out.

10. Scan the image before the scheduler admits it

Wiz recommends image scanning inside the continuous integration pipeline, with tools such as Trivy or Clair, plus a policy that blocks deployment of an image carrying a critical vulnerability. The PCI Security Standards Council published an information supplement from its 2021 Special Interest Group on containers and orchestration tools, covering the threats aimed at payment systems and the practices that answer them. Run image scanning in the pipeline and admission control on the cluster, with the image digest as the point where the two meet.

Check: kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u shows only digests the pipeline produced, and a push of an unscanned tag to the registry is rejected.

The stack a developer actually touches narrows to a registry, a cluster and a pipeline. An IBM study found 70% of surveyed developers using a container orchestration solution, with 70% of that group running a fully managed service. Every check above measures the application's behaviour under a scheduler that is free to move it, and each one returns a yes or a no before the next step starts.