Operations & Deployment Guide - CAMT-CSV Project

Overview

This document provides comprehensive guidance for building, deploying, monitoring, and maintaining the CAMT-CSV application in production environments.

Build Process

1. Local Development Build

# Using Makefile (recommended)
make build            # Build the application
make all              # Lint, test, and build
make install-tools    # Install dev tools (golangci-lint, gosec, cyclonedx-gomod)

# Direct commands (for specific cases)
# Clean build
go clean -cache
go mod tidy
go mod verify

# Build for current platform
go build -o bin/camt-csv cmd/camt-csv/main.go

# Build with version information
VERSION=$(git describe --tags --always --dirty)
BUILD_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
COMMIT=$(git rev-parse --short HEAD)

go build -ldflags "\
  -X main.version=${VERSION} \
  -X main.commit=${COMMIT} \
  -X main.date=${BUILD_TIME}" \
  -o camt-csv .

2. Cross-Platform Builds

# Build for multiple platforms
PLATFORMS="darwin/amd64 darwin/arm64 linux/amd64 linux/arm64 windows/amd64"

for platform in $PLATFORMS; do
    GOOS=${platform%/*}
    GOARCH=${platform#*/}

    echo "Building for $GOOS/$GOARCH..."

    output="bin/camt-csv-${GOOS}-${GOARCH}"
    if [ "$GOOS" = "windows" ]; then
        output="${output}.exe"
    fi

    GOOS=$GOOS GOARCH=$GOARCH go build \
        -ldflags "-s -w -X main.version=${VERSION}" \
        -o $output .
done

3. Docker Build

# See Dockerfile in project root for the full version
FROM golang:1.24-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN CGO_ENABLED=0 GOOS=linux go build \
    -ldflags="-s -w -X main.version=${VERSION}" \
    -o camt-csv .

FROM alpine:3.21
RUN apk --no-cache add ca-certificates poppler-utils
RUN adduser -D -g '' appuser
WORKDIR /app
COPY --from=builder /app/camt-csv .
COPY --from=builder /app/database ./database
RUN chown -R appuser:appuser /app
USER appuser
ENTRYPOINT ["./camt-csv"]

# Build Docker image
docker build -t camt-csv:latest .

# Multi-platform build
docker buildx build --platform linux/amd64,linux/arm64 -t camt-csv:latest .

Release Process

1. Version Management

Semantic Versioning: Follow semver (MAJOR.MINOR.PATCH)

• MAJOR: Breaking changes to CLI interface or file formats
• MINOR: New features, new parser support
• PATCH: Bug fixes, performance improvements

2. Release Workflow

# 1. Prepare release
git checkout main
git pull origin main

# 2. Update version
VERSION="v1.2.3"
echo $VERSION > VERSION

# 3. Update CHANGELOG.md
# Add release notes, breaking changes, new features

# 4. Commit and tag
git add VERSION CHANGELOG.md
git commit -m "Release $VERSION"
git tag -a $VERSION -m "Release $VERSION"

# 5. Push
git push origin main
git push origin $VERSION

3. Automated Release (GoReleaser + GitHub Actions)

Releases are fully automated via GoReleaser v2. When you push a v* tag, the workflow at .github/workflows/goreleaser.yml runs and produces:

1. Multi-platform binaries (linux/darwin/windows, amd64/arm64) as tar.gz/zip archives
2. Docker images pushed to ghcr.io/fjacquet/camt-csv (multi-arch manifest for amd64+arm64)
3. Homebrew formula pushed to fjacquet/homebrew-tap (requires TAP_GITHUB_TOKEN secret)
4. GitHub Release with archives, checksums (SHA-256), and auto-generated changelog
5. SLSA provenance — the SLSA workflow (.github/workflows/go-ossf-slsa3-publish.yml) triggers on release: [created] to add provenance attestation and SBOM

Configuration is in .goreleaser.yaml. Version info is injected via ldflags (main.version, main.commit, main.date).

Required secrets:

Secret	Purpose
GITHUB_TOKEN	Provided automatically — used for release creation and GHCR push
TAP_GITHUB_TOKEN	PAT with Contents:Write on fjacquet/homebrew-tap — for Homebrew formula push

To create a release:

# 1. Update CHANGELOG.md
# 2. Commit and tag
git tag -a v2.3.0 -m "Release v2.3.0"
git push origin v2.3.0
# GoReleaser handles the rest

To test locally without publishing:

goreleaser build --snapshot --clean

Deployment Strategies

1. Homebrew (macOS / Linux)

brew tap fjacquet/homebrew-tap
brew install camt-csv

# Verify
camt-csv --version

# Upgrade
brew upgrade camt-csv

2. Standalone Binary Deployment

Download pre-built binaries from GitHub Releases.

# Example: Linux amd64
curl -L https://github.com/fjacquet/camt-csv/releases/latest/download/camt-csv_$(curl -s https://api.github.com/repos/fjacquet/camt-csv/releases/latest | grep tag_name | cut -d'"' -f4 | sed 's/v//')_linux_amd64.tar.gz | tar xz
chmod +x camt-csv
sudo mv camt-csv /usr/local/bin/

# Verify installation
camt-csv --version

3. Container Deployment (GHCR)

Multi-arch Docker images (amd64/arm64) are published to GitHub Container Registry on every release.

# Pull the latest image
docker pull ghcr.io/fjacquet/camt-csv:latest

# Run with Docker
docker run --rm \
  -v $(pwd)/data:/data \
  -e GEMINI_API_KEY=$GEMINI_API_KEY \
  ghcr.io/fjacquet/camt-csv:latest camt -i /data/input.xml -o /data/output.csv

# Docker Compose for batch processing
version: '3.8'
services:
  camt-csv:
    image: ghcr.io/fjacquet/camt-csv:latest
    volumes:
      - ./input:/input
      - ./output:/output
    environment:
      - GEMINI_API_KEY=${GEMINI_API_KEY}
      - LOG_LEVEL=info
    command: convert --input /input/statements.xml --output /output/transactions.csv

4. Kubernetes Deployment

Use Cases: Large-scale processing, enterprise environments

# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: camt-csv-processor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: camt-csv
  template:
    metadata:
      labels:
        app: camt-csv
    spec:
      containers:
      - name: camt-csv
        image: ghcr.io/fjacquet/camt-csv:latest
        env:
        - name: GEMINI_API_KEY
          valueFrom:
            secretKeyRef:
              name: ai-credentials
              key: gemini-api-key
        - name: LOG_LEVEL
          value: "info"
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Configuration Management

1. Environment-Specific Configuration

# config/production.yaml
log:
  level: "warn"
  format: "json"

ai:
  enabled: true
  requests_per_minute: 60

csv:
  delimiter: ","

# config/development.yaml  
log:
  level: "debug"
  format: "text"

ai:
  enabled: false

2. Secret Management

Environment Variables:

# Production secrets
export GEMINI_API_KEY="$(cat /etc/secrets/gemini-api-key)"
export LOG_LEVEL="warn"

Kubernetes Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: ai-credentials
type: Opaque
data:
  gemini-api-key: <base64-encoded-key>

Docker Secrets:

# Create secret
echo "your-api-key" | docker secret create gemini_api_key -

# Use in service
docker service create \
  --secret gemini_api_key \
  --env GEMINI_API_KEY_FILE=/run/secrets/gemini_api_key \
  camt-csv:latest

Monitoring & Observability

1. Logging Strategy

Structured Logging:

log.WithFields(logrus.Fields{
    "file":         filePath,
    "parser":       "camt",
    "transactions": len(transactions),
    "duration":     time.Since(start),
}).Info("File processing completed")

Log Aggregation:

# docker-compose.yml with ELK stack
version: '3.8'
services:
  camt-csv:
    image: ghcr.io/fjacquet/camt-csv:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    environment:
      - LOG_FORMAT=json

  filebeat:
    image: elastic/filebeat:7.15.0
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml

Note: This is a CLI tool without an HTTP server, so there are no Prometheus metrics endpoints or health check endpoints. Monitoring is done through:

• Structured logging (JSON format for machine processing)
• Exit codes (0 for success, non-zero for errors)
• File-based audit logs when needed
• CI/CD pipeline checks (golangci-lint, gosec, tests with coverage via codecov)
• SBOM generation via cyclonedx-gomod
• GitHub Pages documentation at https://fjacquet.github.io/camt-csv/

CI/CD Pipeline uses:

• Go 1.24.2
• golangci-lint for code quality
• gosec for security scanning (with SARIF output)
• cyclonedx-gomod for SBOM generation (CycloneDX format)
• codecov for coverage reporting

Makefile Targets:

make build         # Build the application
make test          # Run all tests
make test-race     # Run tests with race detector
make coverage      # Generate HTML coverage report
make lint          # Run golangci-lint
make security      # Run gosec security scan
make sbom          # Generate SBOM
make all           # Lint, test, and build
make install-tools # Install dev tools

Performance Optimization

1. Resource Management

Memory Optimization:

// Stream processing for large files
func processLargeFile(filePath string) error {
    file, err := os.Open(filePath)
    if err != nil {
        return err
    }
    defer file.Close()

    decoder := xml.NewDecoder(file)

    for {
        token, err := decoder.Token()
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }

        // Process tokens incrementally
        if se, ok := token.(xml.StartElement); ok {
            if se.Name.Local == "Ntry" {
                var entry Entry
                if err := decoder.DecodeElement(&entry, &se); err != nil {
                    return err
                }
                processEntry(entry)
            }
        }
    }

    return nil
}

CPU Optimization:

// Parallel processing
func processTransactions(transactions []models.Transaction) []models.Transaction {
    numWorkers := runtime.NumCPU()
    jobs := make(chan models.Transaction, len(transactions))
    results := make(chan models.Transaction, len(transactions))

    // Start workers
    for w := 0; w < numWorkers; w++ {
        go worker(jobs, results)
    }

    // Send jobs
    for _, tx := range transactions {
        jobs <- tx
    }
    close(jobs)

    // Collect results
    processed := make([]models.Transaction, 0, len(transactions))
    for i := 0; i < len(transactions); i++ {
        processed = append(processed, <-results)
    }

    return processed
}

2. Caching Strategy

// Category cache
type CategoryCache struct {
    cache map[string]*models.Category
    mutex sync.RWMutex
    ttl   time.Duration
}

func (c *CategoryCache) Get(key string) (*models.Category, bool) {
    c.mutex.RLock()
    defer c.mutex.RUnlock()

    category, exists := c.cache[key]
    return category, exists
}

Troubleshooting

1. Common Issues

File Processing Errors:

# Check file format
camt-csv validate --input suspicious_file.xml

# Debug with verbose logging
CAMT_LOG_LEVEL=debug camt-csv convert --input file.xml --output file.csv

# Test with minimal example
camt-csv convert --input samples/minimal.xml --output test.csv

Memory Issues:

# Monitor memory usage
top -p $(pgrep camt-csv)

# Use streaming for large files
camt-csv convert --streaming --input large_file.xml --output output.csv

AI Service Issues:

# Test AI connectivity
curl -H "Authorization: Bearer $GEMINI_API_KEY" \
  https://generativelanguage.googleapis.com/v1/models

# Disable AI fallback
CAMT_AI_ENABLED=false camt-csv convert --input file.xml --output file.csv

2. Diagnostic Commands

# System information
camt-csv version --verbose

# Configuration dump
camt-csv config --show

# Performance profiling
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

3. Log Analysis

Error Patterns:

# Find parsing errors
grep "parse error" /var/log/camt-csv.log | tail -20

# Check AI API issues
grep "ai service" /var/log/camt-csv.log | grep ERROR

# Performance analysis
grep "processing_duration" /var/log/camt-csv.log | \
  awk '{print $NF}' | sort -n | tail -10

Backup & Recovery

1. Data Backup

Configuration Backup:

# Backup user configuration
tar -czf camt-csv-config-$(date +%Y%m%d).tar.gz \
  ~/.camt-csv/ \
  ~/.env

Database Backup:

# Backup category mappings
cp -r database/ backup/database-$(date +%Y%m%d)/

2. Disaster Recovery

Recovery Procedures:

# Restore from backup
tar -xzf camt-csv-config-20241219.tar.gz -C ~/

# Verify configuration
camt-csv config --validate

# Test functionality
camt-csv convert --input samples/test.xml --output test.csv --dry-run

Security Considerations

1. Secure Deployment

File Permissions:

# Secure configuration files
chmod 600 ~/.camt-csv/config.yaml
chmod 600 ~/.env

# Secure binary
chmod 755 /usr/local/bin/camt-csv

Network Security:

# Kubernetes NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: camt-csv-policy
spec:
  podSelector:
    matchLabels:
      app: camt-csv
  policyTypes:
  - Egress
  egress:
  - to: []
    ports:
    - protocol: TCP
      port: 443  # HTTPS only

2. Security Monitoring

Audit Logging:

log.WithFields(logrus.Fields{
    "user":      os.Getenv("USER"),
    "file":      filePath,
    "operation": "convert",
    "timestamp": time.Now(),
}).Info("File processing initiated")

Vulnerability Scanning:

# Scan dependencies
go list -json -m all | nancy sleuth

# Container scanning
docker scan camt-csv:latest

This operations guide provides comprehensive coverage for deploying, monitoring, and maintaining the CAMT-CSV application in production environments while ensuring security, performance, and reliability.