Serverless #
The name “serverless” is one of the most misleading names in technology. Servers still exist — you just don’t manage them. What genuinely disappears is the responsibility for provisioning, configuring, scaling, and maintaining server infrastructure. You just write functions, deploy, and the platform handles the rest: scaling, availability, operating system patching, and resource allocation.
This model changes how we think about backends: no longer “how many servers are needed” but “what functions need to execute and when”. A function only needed when a specific event happens — file uploads, incoming webhooks, daily schedules — doesn’t need an always-running server. With serverless, you pay only for executions that actually happen, not for waiting capacity.
But serverless isn’t a silver bullet. It introduces new constraints — cold starts, execution timeouts, strict statelessness, harder debugging — that need to be understood before choosing it as an architecture.
How Function as a Service (FaaS) Works #
FaaS is the core of serverless. Each “function” is an independent deployment unit — writable in various languages, triggerable by various events, and independently scalable.
sequenceDiagram
participant E as Event Source
participant P as Platform\n(AWS Lambda, GCP Functions)
participant C as Container
participant F as Function Code
E->>P: Event arrives\n(HTTP request, S3 upload, SQS message)
alt Cold Start (no active container)
P->>C: Provision a new container
C->>C: Download and initialize the runtime
C->>F: Load function code
F->>F: Initialize (global scope)
Note over C,F: Cold start: 100ms - 3 seconds
else Warm Start (container already exists)
P->>C: Route to the existing container
Note over C: Warm start: < 10ms
end
F->>F: Execute the handler function
F->>E: Return the responseTraditional server vs serverless models:
Traditional Servers:
Servers always running 24/7
→ Pay for server uptime, not just usage
→ Manual or slow auto-scaling
→ Responsibilities: OS updates, security patches, capacity planning
→ Suitable for: constant, predictable workloads
Serverless (FaaS):
Functions only run when events occur
→ Pay per execution (and per GB-second of memory used)
→ Auto-scales from 0 to thousands of concurrent executions in seconds
→ No infrastructure responsibilities
→ Suitable for: spiky, intermittent, or event-driven workloads
Cold Starts: The Main Serverless Challenge #
Cold starts happen when no “warm” container exists to handle a request. The platform must prepare a new environment from scratch — which takes time users can feel if not handled well.
Cold start anatomy:
1. The platform provisions a new container (~50-200ms)
2. Download and extract the deployment package (~50-500ms depending on size)
3. Runtime initialization (Node.js, Python, Java, etc.) (~50-500ms)
4. Code outside the handler executed (global scope) (~10-1000ms)
5. Handler function execution (varies)
Total cold start:
→ Node.js/Python: usually 100-500ms
→ Java/C#: can be 1-5 seconds (heavy JVM startups)
→ Go/Rust: usually < 100ms (compiled binaries)
Warm start (existing container):
→ Straight to step 5
→ Usually < 10ms overhead
Cold Start Mitigation Strategies #
// Go: cold-start mitigation — initialize AWS clients ONCE at package scope
// Anti-pattern: creating clients inside the handler (runs on every invocation)
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"sync"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
// ✗ Anti-pattern: expensive initialization inside the handler
func handlerBad(ctx context.Context, event map[string]any) error {
dbClient := dynamodb.NewFromConfig(cfg) // overhead on every call
s3Client := s3.NewFromConfig(cfg)
config := loadConfigFromS3(ctx, s3Client) // a network call every time!
_ = dbClient
// ... process the event ...
return nil
}
// ✓ Correct: clients initialized ONCE at package scope (global scope)
// Executed once at cold start, reused across warm invocations
var (
cfg aws.Config
dbClient *dynamodb.Client
s3Client *s3.Client
config *AppConfig // lazy load
)
func init() {
cfg = loadAWSConfig()
dbClient = dynamodb.NewFromConfig(cfg)
s3Client = s3.NewFromConfig(cfg)
}
var configOnce sync.Once
func getConfig(ctx context.Context) *AppConfig {
configOnce.Do(func() {
out, err := s3Client.GetObject(ctx, &s3.GetObjectInput{
Bucket: aws.String("config"), Key: aws.String("app.json"),
})
if err != nil {
return
}
defer out.Body.Close()
config = &AppConfig{}
_ = json.NewDecoder(out.Body).Decode(config)
})
return config
}
func handler(ctx context.Context, event map[string]any) error {
// Handler runs on every invocation, but dbClient/s3Client already exist
cfg := getConfig(ctx) // only loads the first time per container
// ... process the event using the existing clients ...
return processEvent(ctx, event, cfg, dbClient)
}
func loadAWSConfig() aws.Config { return aws.Config{} }
func loadConfigFromS3(ctx context.Context, c *s3.Client) string { return "" }
func processEvent(ctx context.Context, e map[string]any, c *AppConfig, d *dynamodb.Client) error { return nil }
type AppConfig struct{ URL string }
// Other cold start mitigation techniques:
// 1. Reduce the deployment package size
// Smaller packages → faster downloads → shorter cold starts
// requirements.txt: only include what's truly needed
// Use Lambda Layers for rarely-changing dependencies
// 2. Provisioned Concurrency (AWS Lambda)
// Pre-warm N containers before requests arrive
// Pay extra but cold starts are eliminated for predictable traffic
// aws lambda put-provisioned-concurrency-config \
// --function-name my-function \
// --qualifier LIVE \
// --provisioned-concurrent-executions 5
// 3. Scheduled pings for keep-alive (cheap workaround)
// CloudWatch Events every 5 minutes → invoke the Lambda
// → Containers stay warm
// But this doesn't scale for functions needing many concurrent instances
// 4. Choose runtimes with faster cold starts
// Cold start ranking (fastest first):
// Go/Rust > Node.js ≈ Python > C# > Java
// For latency-sensitive functions, avoid Java without GraalVM native images
Stateless by Design #
Serverless enforces strict statelessness — there’s no guarantee the next invocation uses the same container. Every invocation must treat local state as temporary.
// Go: stateless by design — all state lives in external storage
// Anti-pattern: storing state in container memory
// ✗ Anti-pattern: storing state in container memory
// This state is inconsistent across invocations!
var requestCounter = 0 // container A: 5, container B: 3, container C: 0
func handlerBad(ctx context.Context, event map[string]any) {
requestCounter++
// requestCounter differs in every container
// Can't be relied on for anything
}
// ✓ Correct: all state in external storage
// For accurate counters (DynamoDB atomic ADD)
func incrementRequestCounter(ctx context.Context, functionName string) (int64, error) {
out, err := dbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String("counters"),
Key: map[string]types.AttributeValue{"name": &types.AttributeValueMemberS{Value: functionName}},
UpdateExpression: aws.String("ADD #count :one"),
ExpressionAttributeNames: map[string]string{"#count": "count"},
ExpressionAttributeValues: map[string]types.AttributeValue{":one": &types.AttributeValueMemberN{Value: "1"}},
ReturnValues: types.ReturnValueUpdatedNew,
})
if err != nil {
return 0, err
}
return out.Attributes["count"].(*types.AttributeValueMemberN).Value, nil
}
// For sessions/cache (Redis)
var redisClient = redis.NewClient(&redis.Options{
Addr: os.Getenv("REDIS_HOST") + ":6379",
})
func handler(ctx context.Context, event map[string]any) {
// State always from external storage
counter, _ := incrementRequestCounter(ctx, "my-function")
session, _ := redisClient.Get(ctx, "session:"+event["session_id"].(string)).Result()
_ = counter
_ = session
// ... process ...
}
Where to store state in serverless:
Temporary (within a single invocation only):
→ Local variables in the function
→ /tmp filesystems (max 512MB in AWS Lambda, but lost when containers recycle)
Persistent state:
→ Databases (DynamoDB, RDS via connection pooling)
→ Caches (ElastiCache/Redis via connections reused in global scope)
→ Object storage (S3 for files)
→ Message queues (SQS, SNS for async communication)
State to avoid:
→ Local files expected to persist across invocations
→ In-memory caches expected to stay consistent across containers
→ Unclean connections (database connection leaks)
Event-Driven Architecture in Serverless #
Serverless shines brightest in event-driven architectures — functions triggered by events from various sources.
// Go: example Lambda functions handling various triggers
// 1. HTTP requests via API Gateway
func apiHandler(ctx context.Context, event map[string]any) (map[string]any, error) {
// Handler for HTTP requests.
path := event["path"].(string)
method := event["httpMethod"].(string)
body := map[string]any{}
if raw, ok := event["body"].(string); ok && raw != "" {
_ = json.Unmarshal([]byte(raw), &body)
}
if method == "POST" && path == "/orders" {
return createOrder(ctx, body)
}
return map[string]any{
"statusCode": 404,
"body": `{"error": "Not found"}`,
}, nil
}
// 2. S3 events — when files are uploaded
func s3EventHandler(ctx context.Context, event map[string]any) {
// Process newly uploaded files.
for _, rec := range event["Records"].([]any) {
r := rec.(map[string]any)
bucket := r["s3"].(map[string]any)["bucket"].(map[string]any)["name"].(string)
key := r["s3"].(map[string]any)["object"].(map[string]any)["key"].(string)
// Process the file: resize images, parse CSVs, scan viruses, etc.
processUploadedFile(ctx, bucket, key)
}
}
// 3. SQS queues — process messages from queues
func sqsHandler(ctx context.Context, event map[string]any) (map[string]any, error) {
// Process messages from an SQS queue.
failed := []map[string]string{}
for _, rec := range event["Records"].([]any) {
r := rec.(map[string]any)
messageID := r["messageId"].(string)
var body map[string]any
_ = json.Unmarshal([]byte(r["body"].(string)), &body)
if err := processMessage(ctx, body); err != nil {
// Mark as failed — will return to the queue or go to the DLQ
failed = append(failed, map[string]string{"itemIdentifier": messageID})
}
}
// Return batch item failures — only the failed ones get retried
return map[string]any{"batchItemFailures": failed}, nil
}
// 4. CloudWatch Events (scheduled)
func scheduledHandler(ctx context.Context, event map[string]any) (map[string]any, error) {
// Runs every day at 08:00.
sendDailyReport(ctx)
cleanupExpiredSessions(ctx)
return map[string]any{"status": "completed"}, nil
}
// 5. DynamoDB Streams — react to database changes
func dynamodbStreamHandler(ctx context.Context, event map[string]any) {
// Triggered when new or changed records appear in DynamoDB.
for _, rec := range event["Records"].([]any) {
r := rec.(map[string]any)
if r["eventName"] == "INSERT" {
newItem := r["dynamodb"].(map[string]any)["NewImage"].(map[string]any)
userID := newItem["user_id"].(map[string]any)["S"].(string)
sendWelcomeEmail(ctx, userID)
}
}
}
Serverless Limitations Worth Understanding #
Serverless isn’t without limitations. Understanding them matters before choosing serverless for specific use cases.
Main AWS Lambda limits (numbers may differ per provider):
Execution timeout:
→ Maximum 15 minutes per invocation
→ Not suitable for long-running processes (video encoding, ML training)
Memory:
→ 128MB up to 10GB (more memory = more CPU)
→ No GPUs (for ML/AI workloads)
Storage:
→ /tmp: 512MB (configurable up to 10GB)
→ Not persistent storage
Concurrent executions:
→ Default limit: 1000 concurrent per region (can be raised)
→ Important to understand for burst workloads
Deployment packages:
→ 50MB compressed, 250MB uncompressed
→ Layers help but have total limits too
Networking:
→ Longer cold starts inside VPCs (ENI provisioning)
→ NAT Gateways needed for internet access from VPCs
Payloads:
→ API Gateway: 10MB requests/responses
→ Synchronous invocations: 6MB payloads
→ SQS: 256KB per message
When serverless is NOT suitable:
✗ Long-running workloads (> 15 minutes)
→ ETL processing large datasets
→ Video transcoding
→ ML model training
→ Solution: ECS Fargate, EC2, AWS Batch
✗ Latency-sensitive workloads intolerant of cold starts
→ Real-time gaming backends
→ HFT (High Frequency Trading)
→ Solution: Provisioned Concurrency or traditional servers
✗ Heavy stateful workloads
→ Long-persisting WebSocket connections
→ Stateful streaming (usually needs persistent connections)
→ Solution: EC2/ECS with WebSocket servers
✗ Workloads needing GPUs
→ Deep learning inference
→ Solution: EC2 GPU instances, SageMaker
✗ Very high constant traffic
→ If always 1000+ concurrent executions 24/7
→ Costs can exceed EC2
→ Calculate first before deciding
Serverless Cost Models #
The serverless cost model differs fundamentally from traditional servers — it can be very economical for some use cases, but can also exceed expectations.
AWS Lambda pricing (2025):
Compute:
→ $0.0000166667 per GB-second
→ Example: a 256MB function running for 1 second
= 0.25 GB × 1 second × $0.0000166667 = $0.0000041667
Requests:
→ $0.20 per 1 million requests
Free tier (monthly):
→ 1 million free requests
→ 400,000 free GB-seconds
Cost calculations:
Scenario A — Low traffic:
100 requests/day × 30 days = 3,000 requests/month
Still in the free tier → FREE
vs EC2 t3.micro: ~$8.5/month
Scenario B — Medium traffic:
100,000 requests/day × 30 days = 3,000,000 requests/month
Functions at 256MB, averaging 100ms
Compute: 3M × 0.25GB × 0.1s = 75,000 GB-s × $0.0000166667 = $1.25
Requests: (3M - 1M) × $0.2/1M = $0.40
Total: ~$1.65/month vs EC2 t3.micro $8.5/month → SAVINGS
Scenario C — High traffic:
10 million requests/day = 300 million requests/month
Functions at 512MB, averaging 200ms
Compute: 300M × 0.5GB × 0.2s = 30M GB-s × $0.0000166667 = $500
Requests: 300M × $0.2/1M = $60
Total: ~$560/month vs an EC2 fleet that may be cheaper
→ Needs more detailed calculations before deciding
Observability in Serverless #
Debugging and monitoring in serverless is more challenging than traditional servers — no SSH to servers, logs scattered across functions, and distributed tracing becomes more important.
// Go: best practices for Lambda observability
// Structured logging — easier to parse in CloudWatch Logs Insights
package main
import (
"encoding/json"
"log"
"time"
)
func logEvent(eventType string, fields map[string]any) {
// Log structured events for observability.
entry := map[string]any{"event_type": eventType, "timestamp": time.Now().Unix()}
for k, v := range fields {
entry[k] = v
}
log.Println(string(mustJSON(entry)))
}
// tracedHandler — wrapper (the Go equivalent of a decorator) for tracing and error handling
func tracedHandler(handler func(event, context any) (any, error)) func(event, context any) (any, error) {
return func(event, context any) (any, error) {
startTime := time.Now()
// Log invocation metadata
logEvent("invocation_start", map[string]any{
"function_name": context.(map[string]any)["function_name"],
"request_id": context.(map[string]any)["request_id"],
"remaining_time_ms": context.(map[string]any)["remaining_time_ms"],
"memory_limit_mb": context.(map[string]any)["memory_limit_mb"],
})
result, err := handler(event, context)
durationMs := time.Since(startTime).Milliseconds()
if err != nil {
logEvent("invocation_error", map[string]any{
"error_type": fmt.Sprintf("%T", err),
"error_message": err.Error(),
"duration_ms": durationMs,
"request_id": context.(map[string]any)["request_id"],
})
return nil, err // re-raise so Lambda knows this was an error
}
logEvent("invocation_success", map[string]any{
"duration_ms": durationMs,
"request_id": context.(map[string]any)["request_id"],
})
return result, nil
}
}
var handler = tracedHandler(func(event, context any) (any, error) {
// The main handler function
return nil, nil
})
// Go: AWS X-Ray for distributed tracing
import (
"context"
"github.com/aws/aws-xray-sdk-go/xray"
)
func init() {
// Patch all AWS SDK calls for auto-tracing
xray.Configure(xray.Config{})
}
// processOrder — this function appears as a subsegment in X-Ray traces
func processOrder(ctx context.Context, orderData map[string]any) (map[string]any, error) {
ctx, seg := xray.BeginSubsegment(ctx, "process_order")
defer seg.Close(nil)
seg.AddAnnotation("order.id", orderData["id"])
seg.AddAnnotation("order.total", orderData["total"])
ctx, sub := xray.BeginSubsegment(ctx, "validate_order")
err := validateOrder(ctx, orderData)
sub.Close(err)
if err != nil {
return nil, err
}
ctx, sub2 := xray.BeginSubsegment(ctx, "save_to_database")
order, err := saveOrder(ctx, orderData)
sub2.AddMetadata("saved_order", order)
sub2.Close(err)
return order, err
}
Infrastructure as Code for Serverless #
# serverless.yml — using the Serverless Framework
service: my-app
provider:
name: aws
runtime: python3.12
region: ap-southeast-1
memorySize: 256 # MB, default for all functions
timeout: 30 # seconds, default
environment:
APP_ENV: ${opt:stage, 'development'}
DATABASE_URL: ${env:DATABASE_URL}
# IAM role — least privilege
iam:
role:
statements:
- Effect: Allow
Action:
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
Resource: !GetAtt OrdersTable.Arn
functions:
# HTTP API
api:
handler: src/api.handler
events:
- httpApi:
path: /orders
method: POST
- httpApi:
path: /orders/{id}
method: GET
memorySize: 512 # override the default for functions needing more
# Scheduled jobs
dailyReport:
handler: src/reports.daily_handler
timeout: 300 # 5 minutes for report processing
events:
- schedule:
rate: cron(0 8 * * ? *) # Every day at 08:00 UTC
enabled: true
# SQS consumers
orderProcessor:
handler: src/processor.sqs_handler
reservedConcurrency: 10 # Limit concurrency to protect downstreams
events:
- sqs:
arn: !GetAtt OrderQueue.Arn
batchSize: 10
functionResponseType: ReportBatchItemFailures
resources:
Resources:
OrdersTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${self:service}-${opt:stage}-orders
BillingMode: PAY_PER_REQUEST # On-demand — no capacity planning
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
OrderQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:service}-${opt:stage}-orders
VisibilityTimeout: 60
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrderDLQ.Arn
maxReceiveCount: 3 # Retry 3 times before going to the DLQ
OrderDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:service}-${opt:stage}-orders-dlq
MessageRetentionPeriod: 1209600 # 14 days
Anti-Patterns to Avoid #
// Go: anti-patterns
// ✗ Anti-pattern 1: expensive initialization inside the handler
func handlerBad(ctx context.Context, event map[string]any) {
db := connectDatabase(ctx) // a new connection every invocation!
config := loadFromS3(ctx) // a network call every invocation!
}
// ✓ Solution: global scope for reusable initialization
var (
db = connectDatabase(nil) // once per container
config = loadFromS3(nil) // once per container
)
func handler(ctx context.Context, event map[string]any) {
// reused from the global scope
process(ctx, event, db, config)
}
// ✗ Anti-pattern 2: storing state in /tmp and relying on its persistence
func handlerTmp(ctx context.Context, event map[string]any) {
os.WriteFile("/tmp/data.json", data, 0644)
// The next invocation may use a different container — the file doesn't exist!
}
// ✓ Solution: use S3 or databases for persistent state
// ✗ Anti-pattern 3: timeouts too short without retry logic
// Default Lambda timeout: 3 seconds
// If the downstream service is slow → timeout before finishing
// ✓ Solution: set realistic timeouts + implement retries in SQS
// ✗ Anti-pattern 4: no Dead Letter Queues
// Failed messages → lost forever
// ✓ Solution: always configure DLQs for SQS consumers and async invocations
// ✗ Anti-pattern 5: concurrency not limited for downstream protection
// Lambdas scaling to thousands of concurrent executions → flooding databases or third-party APIs
// ✓ Solution: set reservedConcurrency or use SQS for natural throttling
// ✗ Anti-pattern 6: functions doing too much (monolithic Lambdas)
// One function handling everything: auth + business logic + data access + notifications
// ✓ Solution: single responsibility per function, composition via events/messages
Serverless Checklist #
FUNCTION DESIGN:
□ Each function has one clear responsibility
□ Handler functions thin — business logic in separate modules
□ Expensive initialization in global scope, not inside handlers
□ No state stored in memory or /tmp for persistence across invocations
COLD STARTS:
□ Deployment packages as small as possible
□ Unneeded dependencies not included
□ Runtimes chosen per latency needs (Go/Node.js for low latency)
□ Provisioned Concurrency configured for latency-critical functions
ERROR HANDLING:
□ Dead Letter Queues configured for async invocations and SQS consumers
□ Proper retry logic (exponential backoff for transient errors)
□ Idempotent handlers — executing twice produces the same result
□ Partial batch failures handled correctly (reportBatchItemFailures)
SECURITY:
□ IAM roles with least privilege — only needed permissions
□ Secrets from AWS Secrets Manager or Parameter Store, not hardcoded env vars
□ No sensitive data in logs
□ VPCs if functions need private resource access
OBSERVABILITY:
□ Structured logging (JSON) for easy CloudWatch Logs Insights queries
□ Custom metrics for important business events
□ X-Ray tracing enabled for distributed tracing
□ Alerts installed for error rates, durations, throttling
COST:
□ Memory sizes configured per needs (more memory = more expensive)
□ Timeouts configured per needs (not set to maximums)
□ Reserved concurrency set to prevent runaway costs
□ Cost estimates made before going live
OPERATIONAL:
□ Infrastructure as Code (Serverless Framework, AWS CDK, SAM)
□ CI/CD pipelines for deployments
□ Separate environments (dev, staging, production)
□ Fast rollbacks possible
Summary #
- Serverless isn’t without servers — it’s without server management — the platform handles provisioning, scaling, and availability. You focus on code and business logic.
- Cold starts are the main trade-off — new containers take time to initialize. Minimize them with global-scope initialization, small packages, and the right runtimes.
- Statelessness is a requirement, not a choice — no guarantee the next invocation uses the same container. All state must be in external storage (DynamoDB, Redis, S3).
- Event-driven is serverless’s best context — file uploads, webhooks, queue messages, periodic schedules — all fit. Constant and latency-critical workloads fit traditional servers better.
- Global-scope initialization for warm invocations — database connections, configuration, and SDK clients initialized outside handler functions are reused across invocations in the same container.
- Dead Letter Queues are mandatory safety nets — failed messages must not disappear silently. DLQs enable investigation and reprocessing.
- IAM least privilege is mandatory — each function may only access genuinely needed resources. One compromised function must not access everything.
- Idempotency is a must — Lambda can execute functions more than once (exactly-once delivery isn’t guaranteed). Idempotent handlers are safe to retry.
- Cost models differ from traditional setups — very economical for low-to-medium traffic, potentially more expensive than EC2 for very high constant traffic. Always calculate before deciding.
- Observability is more challenging — structured logging, X-Ray tracing, and proper alerts are mandatory because there’s no SSH and logs are scattered across many functions.
#
- Serverless isn’t without servers — it’s without server management — the platform handles provisioning, scaling, and availability. You focus on code and business logic.
- Cold starts are the main trade-off — new containers take time to initialize. Minimize them with global-scope initialization, small packages, and the right runtimes.
- Statelessness is a requirement, not a choice — no guarantee the next invocation uses the same container. All state must be in external storage (DynamoDB, Redis, S3).
- Event-driven is serverless’s best context — file uploads, webhooks, queue messages, periodic schedules — all fit. Constant and latency-critical workloads fit traditional servers better.
- Global-scope initialization for warm invocations — database connections, configuration, and SDK clients initialized outside handler functions are reused across invocations in the same container.
- Dead Letter Queues are mandatory safety nets — failed messages must not disappear silently. DLQs enable investigation and reprocessing.
- IAM least privilege is mandatory — each function may only access genuinely needed resources. One compromised function must not access everything.
- Idempotency is a must — Lambda can execute functions more than once (exactly-once delivery isn’t guaranteed). Idempotent handlers are safe to retry.
- Cost models differ from traditional setups — very economical for low-to-medium traffic, potentially more expensive than EC2 for very high constant traffic. Always calculate before deciding.
- Observability is more challenging — structured logging, X-Ray tracing, and proper alerts are mandatory because there’s no SSH and logs are scattered across many functions.