This document describes the file storage system in TGM Manager Server, including multi-tenant storage support where each company can use their own storage provider.

Table of Contents¶

• Overview
• Multi-Tenancy Context
• Global Storage Configuration
• Per-Company Storage
• Supported Providers
• Configuration Examples
• API Reference
• Security Considerations
• Troubleshooting

Overview¶

TGM Manager Server supports flexible file storage with multiple levels of configuration:

1. Global Storage: Platform-wide default storage configuration
2. Per-Company Storage: Individual companies can use their own storage providers

This enables scenarios like: - Company A uses AWS S3 with their own bucket - Company B uses the platform's MinIO instance - Company C hosts their own MinIO server - Company D uses Azure Blob Storage

Multi-Tenancy Context¶

Storage configuration operates within TGM's multi-tenant architecture:

┌─────────────────────────────────────────────────────────────────┐
│                    Platform Level                                │
│  Global Storage Config (MinIO/S3)                               │
│  Used when company.storage_type = 'default'                     │
└─────────────────────────────────────────────────────────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Client A    │      │   Client B    │      │   Client C    │
│   Database    │      │   Database    │      │   Database    │
├───────────────┤      ├───────────────┤      ├───────────────┤
│   Company     │      │   Company     │      │   Company     │
│ storage_type: │      │ storage_type: │      │ storage_type: │
│   's3'        │      │   'default'   │      │   'minio'     │
│ (own bucket)  │      │ (platform)    │      │ (self-hosted) │
└───────────────┘      └───────────────┘      └───────────────┘
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   AWS S3      │      │   Platform    │      │  Self-Hosted  │
│   Bucket      │      │   MinIO       │      │  MinIO        │
└───────────────┘      └───────────────┘      └───────────────┘

API Request Headers¶

All storage configuration requests require multi-tenant headers:

curl -X GET "http://localhost:1337/api/companies/1/storage-config" \
  -H "Authorization: Bearer $JWT" \
  -H "X-Client-ID: acme-corp"

*Can be resolved via subdomain: acme-corp.tgm-expert.com

Global Storage Configuration¶

The global storage configuration is used when: - A company's storage_type is set to default - No company-specific configuration is provided

Environment Variables¶

application.yml¶

app:
  storage:
    type: ${APP_STORAGE_TYPE:minio}
    s3:
      endpoint: ${APP_STORAGE_S3_ENDPOINT:http://minio:9000}
      access-key: ${APP_STORAGE_S3_ACCESS_KEY:minioadmin}
      secret-key: ${APP_STORAGE_S3_SECRET_KEY:minioadmin}
      bucket: ${APP_STORAGE_S3_BUCKET:tgm-uploads}
      region: ${APP_STORAGE_S3_REGION:}
      public-url: ${APP_STORAGE_S3_PUBLIC_URL:}
    local:
      path: ${APP_STORAGE_LOCAL_PATH:./uploads}

Per-Company Storage¶

Each company can configure their own storage provider independently of the global configuration.

Database Schema¶

The following columns are added to the companies table:

Storage Types¶

Supported Providers¶

MinIO (Self-Hosted)¶

MinIO is an S3-compatible object storage that can be self-hosted.

Advantages: - Full control over data - No cloud vendor lock-in - Low latency for on-premise deployments - S3 API compatible

Configuration Example:

{
  "storageType": "minio",
  "storageEndpoint": "https://minio.company.com",
  "storageAccessKey": "companyAccessKey",
  "storageSecretKey": "companySecretKey",
  "storageBucket": "company-files",
  "storagePublicUrl": "https://files.company.com"
}

AWS S3¶

Amazon Simple Storage Service for cloud-native storage.

Advantages: - High availability and durability - Global CDN integration (CloudFront) - Fine-grained access control (IAM) - Automatic scaling

Configuration Example:

{
  "storageType": "s3",
  "storageEndpoint": "https://s3.us-east-1.amazonaws.com",
  "storageAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "storageSecretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "storageBucket": "company-tgm-files",
  "storageRegion": "us-east-1",
  "storagePublicUrl": "https://d1234567890.cloudfront.net"
}

S3-Compatible Services¶

Many cloud providers offer S3-compatible storage:

Configuration Examples¶

Example 1: Company Using AWS S3¶

# Update company storage configuration
curl -X PATCH "http://localhost:1337/api/admin/companies/1/storage" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "storageType": "s3",
    "storageEndpoint": "https://s3.eu-west-1.amazonaws.com",
    "storageAccessKey": "AKIAIOSFODNN7EXAMPLE",
    "storageSecretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "storageBucket": "acme-corp-files",
    "storageRegion": "eu-west-1"
  }'

Example 2: Company Using Self-Hosted MinIO¶

curl -X PATCH "http://localhost:1337/api/admin/companies/2/storage" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "storageType": "minio",
    "storageEndpoint": "https://minio.internal.company.com:9000",
    "storageAccessKey": "internalAccessKey",
    "storageSecretKey": "internalSecretKey",
    "storageBucket": "documents",
    "storagePublicUrl": "https://files.company.com"
  }'

Example 3: Reset to Default Storage¶

curl -X PATCH "http://localhost:1337/api/admin/companies/3/storage" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "storageType": "default"
  }'

API Reference¶

Get Company Storage Info¶

GET /api/admin/companies/{companyId}/storage
Authorization: Bearer <jwt_token>

Response:

{
  "type": "s3",
  "endpoint": "https://s3.****",
  "bucket": "company-files",
  "region": "us-east-1",
  "healthy": true
}

Update Company Storage¶

PATCH /api/admin/companies/{companyId}/storage
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "storageType": "s3",
  "storageEndpoint": "https://s3.us-east-1.amazonaws.com",
  "storageAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "storageSecretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "storageBucket": "my-bucket",
  "storageRegion": "us-east-1",
  "storagePublicUrl": "https://cdn.example.com"
}

Test Storage Connectivity¶

POST /api/admin/companies/{companyId}/storage/test
Authorization: Bearer <jwt_token>

Response:

{
  "success": true,
  "message": "Storage connectivity test passed"
}

Security Considerations¶

Credential Storage¶

Storage credentials (access keys, secret keys) should be: - Encrypted at rest in the database - Never logged in plain text - Rotated regularly

IAM Best Practices (AWS S3)¶

When using AWS S3, create dedicated IAM users with minimal permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::company-bucket",
        "arn:aws:s3:::company-bucket/*"
      ]
    }
  ]
}

MinIO Best Practices¶

For self-hosted MinIO: - Use TLS for all connections - Enable server-side encryption - Configure access policies per bucket - Use separate access keys per application

Network Security¶

• Use HTTPS endpoints only in production
• Configure VPC endpoints for AWS S3
• Use private networks for MinIO connections
• Implement IP whitelisting where possible

Troubleshooting¶

Common Issues¶

1. "Access Denied" Errors¶

Symptoms: File upload/download fails with 403 error

Solutions: - Verify access key and secret key are correct - Check bucket policy allows the required operations - Ensure the bucket exists - For AWS S3, verify IAM permissions

2. "Bucket Does Not Exist" Errors¶

Symptoms: Operations fail with bucket not found error

Solutions: - Create the bucket manually before configuring - Verify bucket name is correct (case-sensitive) - Check region is correct for AWS S3

3. Connection Timeout¶

Symptoms: Storage operations timeout

Solutions: - Verify endpoint URL is accessible from the server - Check firewall rules and security groups - Ensure DNS resolution works for the endpoint - For MinIO, verify the server is running

4. SSL Certificate Errors¶

Symptoms: SSL handshake failures

Solutions: - Ensure SSL certificates are valid - For self-signed certificates, configure the Java truststore - Use proper CA-signed certificates in production

Health Check¶

Check company storage health via the API:

curl "http://localhost:1337/api/admin/companies/1/storage" \
  -H "Authorization: Bearer $JWT"

Or test connectivity:

curl -X POST "http://localhost:1337/api/admin/companies/1/storage/test" \
  -H "Authorization: Bearer $JWT"

Debug Logging¶

Enable debug logging for storage operations:

logging:
  level:
    ca.ensolutions.tgm.service.CompanyStorageService: DEBUG
    ca.ensolutions.tgm.service.external.FileStorageService: DEBUG
    io.minio: DEBUG

Migration Guide¶

From Global to Per-Company Storage¶

1. Configure company storage:

   curl -X PATCH "http://localhost:1337/api/admin/companies/1/storage" \
     -H "Authorization: Bearer $JWT" \
     -d '{"storageType": "s3", ...}'

2. Migrate existing files (manual process):

3. Download files from global storage
4. Upload to company-specific storage

5. Update file URLs in database

6. Verify access:

7. Test file upload/download
8. Verify presigned URLs work
9. Check public URL accessibility

Database Migration¶

The storage fields are added by migration V44__company_storage_configuration.sql:

ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_type VARCHAR(20) DEFAULT 'default';
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_endpoint VARCHAR(255);
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_access_key VARCHAR(255);
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_secret_key VARCHAR(255);
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_bucket VARCHAR(255);
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_region VARCHAR(50);
ALTER TABLE companies ADD COLUMN IF NOT EXISTS storage_public_url VARCHAR(255);

Architecture¶

Storage Flow¶

┌─────────────────┐
│   File Upload   │
│   Request       │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────┐
│        CompanyStorageService        │
│  ┌─────────────────────────────┐    │
│  │ Check company.storage_type  │    │
│  └──────────┬──────────────────┘    │
│             │                       │
│    ┌────────┴────────┐              │
│    ▼                 ▼              │
│ default        custom config        │
│    │                 │              │
│    ▼                 ▼              │
│ ┌──────────┐  ┌──────────────┐      │
│ │  Global  │  │  Company     │      │
│ │ Storage  │  │  MinioClient │      │
│ └────┬─────┘  └──────┬───────┘      │
└──────┼───────────────┼──────────────┘
       │               │
       ▼               ▼
┌──────────────┐ ┌─────────────┐
│   Platform   │ │  Company    │
│   MinIO/S3   │ │  S3/MinIO   │
└──────────────┘ └─────────────┘

Client Caching¶

MinioClient instances are cached per company to avoid creating new connections for each request:

// Cache maintained in CompanyStorageService
Map<Long, MinioClient> companyMinioClients = new ConcurrentHashMap<>();

// Cache is cleared when storage configuration changes
companyStorageService.clearClientCache(companyId);