DOCUMENTATION / OPERATIONS

Run it with
clear failure boundaries.

Storage pressure, TLS configuration, peer failures, and private-image access are operational design decisions. Test them before relying on SIC for a production cluster.

CAPACITY MANAGEMENT

Protect the node's free space.

SIC checks configured free-space reserves and optional cache-size caps before admitting downloads. It evicts least-recently-used blobs that are not actively in use; if enough space cannot be recovered, the request fails rather than filling the filesystem indefinitely.

Defaults documented in the repository include SIC_MIN_FREE_BYTES=1GiB, SIC_MIN_FREE_PERCENT=20, and no hard cache-size cap unless SIC_MAX_CACHE_BYTES is set. The larger of the byte and percentage reserves applies. Choose values with awareness of the node's kubelet eviction thresholds and the filesystem shared with containerd.

Useful signals include sic_cache_bytes, sic_evicted_blobs_total, sic_evicted_bytes_total, and sic_disk_full_total.

OBSERVABILITY

Health and Prometheus metrics.

SIC exposes Prometheus-format metrics at /metrics and a health endpoint at /healthz. When mutual TLS protects the primary listener, the sample manifest uses a separate admin listener bound to loopback so kubelet probes do not need client certificates. Expose that listener only as intended by your deployment.

Monitor cache size and eviction, corrupt blob counts, store errors, disk-full events, stream aborts, upstream auth challenges, and containerd import errors. Review metric labels and cardinality before adding custom instrumentation.

SECURITY MODEL

Understand what SIC does — and does not — authorize.

Cache reads are not registry re-authentication. SIC forwards upstream registry authentication challenges and does not persist the Authorization header. But once content is cached, SIC does not check the original registry credentials on each read. A client accepted by the node's SIC access boundary that knows a tag or digest may retrieve that content. Treat the client CA and network reachability as the content-sharing boundary; keep upstream fallback off for registries whose images must remain isolated.

Use mutual TLS for containerd-to-SIC and SIC-to-SIC traffic; use a trusted CA and plan certificate rotation.
Upstream fallback is opt-in and should be restricted with SIC_UPSTREAM_REGISTRIES.
Peer requests should never redirect. Registry redirects must remain HTTPS-only and bounded.
The sample DaemonSet uses root to access the host containerd socket. It drops Linux capabilities and uses a read-only root filesystem, but host networking, host mounts, socket access, and scheduling priority require a deployment-specific threat-model review.
Pin a release tag or immutable image digest. Do not use a moving main image in production.

TROUBLESHOOTING

Start with the failure class.

Symptom	What to inspect
TLS handshake fails	DNS/IP SANs, client/server extended key usages, CA chain, certificate expiry, and Secret rotation.
Pull fails on node restart	Check the bootstrap sequence: SIC image must already be available before the registry mirror routes image pulls through SIC.
HTTP 507 / disk-full metric rises	Inspect filesystem free space, byte/percentage reserves, active readers/writers, and cache cap.
Peer miss or delay	Confirm peer readiness, network reachability, matching CA/certificates, and peer timeout/backoff metrics or logs.
Corrupt content / stream abort	Inspect digest mismatch, source interruption, peer health, and recovery from a known-good source.
Private image appears accessible	Review which client certificates and networks can reach SIC; cached reads do not re-check registry credentials.

For exact environment variables and current failure semantics, see the README failure section.

RELEASE READINESS

Production checklist

Release-wide formatting, vet, unit, race, image-build, and Kubernetes manifest checks pass on the exact candidate commit.
Two-node cold/warm pull and peer-reuse behavior is validated on a disposable cluster.
Peer loss, interrupted downloads, corruption repair, disk pressure, and cache eviction are exercised.
Private-image access boundary and certificate rotation/failure behavior are reviewed and documented.
Install, upgrade, rollback, and node-restart bootstrap procedures are tested.
Published images are pinned by release tag or digest, with release notes and known limitations.

See releases for published artifacts and Actions for workflow evidence. A passing PR workflow is not proof that every check passed on the exact stable release commit.