Infisical Secrets Management ============================= .. verified:: 2025-10-13 :reviewer: Christof Buchbender Overview -------- Infisical provides centralized secrets management for the observatory, replacing scattered ``.env`` files and manual credential sharing. Access is controlled via GitHub organization membership, with secrets pulled dynamically at deployment time and available to developers through Web UI or CLI. **Key Benefits:** - Single source of truth for all credentials - GitHub OAuth single sign-on - Role-based access via GitHub teams - Audit trail of all access - No secrets in git repositories - Centralized credential rotation Architecture ------------ .. code-block:: text GitHub Organization (Authentication) ↓ Infisical (Central Storage) ↓ ┌───────┴───────┬──────────┬──────────┐ │ │ │ │ Web UI CLI Ansible Jenkins/GH Actions (humans) (humans) (deploy) (CI/CD) **Components:** - **GitHub OAuth:** Authentication via existing GitHub accounts - **Infisical Server:** Hosted at ``infisical.data.ccat.uni-koeln.de`` - **Projects:** Logical groupings of secrets (Science, Infrastructure, Development) - **Environments:** Separate secrets for Production, Staging, Development - **Service Tokens:** Machine credentials for automated systems Projects and Access Control ---------------------------- Access is determined by GitHub team membership TO BE UPDATED AND DISCUSSED EXAMPLES: .. list-table:: :header-rows: 1 :widths: 30 30 40 * - GitHub Team - Infisical Project - Access Level * - ``@observatory/datacener`` - Data Center - Admin (Full Control) * - ``@observatory/developers`` - Developer - Admin (Full Control) * - ``@observatory/chai`` - CHAI - Member (Read/Write) * - ``@observatory/primecam`` - PrimeCam - Member (Read/Write) Each project contains secrets scoped to different environments (Production, Staging, Development). Accessing Secrets ----------------- Web UI Access ^^^^^^^^^^^^^ For quick lookups and browsing: 1. Navigate to ``https://infisical.data.ccat.uni-koeln.de`` 2. Click **"Login with GitHub"** 3. Authorize the application (first time only) 4. Browse to your project (e.g., "Science APIs") 5. Select environment (Production, Staging, Development) 6. Click on a secret to reveal its value 7. Copy the value for use **Use cases:** - Looking up API tokens - Finding SSH credentials during incidents - Checking database connection strings - Verifying configuration values CLI Installation ^^^^^^^^^^^^^^^^ For developers who prefer command-line workflows: **Debian/Ubuntu:** .. code-block:: bash curl -1sLf \ 'https://artifacts-cli.infisical.com/setup.deb.sh' \ | sudo -E bash sudo apt-get update && sudo apt-get install -y infisical **macOS:** .. code-block:: bash brew install infisical/get-cli/infisical **RedHat/CentOS:** .. code-block:: bash curl -1sLf \ 'https://dl.cloudsmith.io/public/infisical/infisical-cli/setup.rpm.sh' \ | sudo -E bash sudo yum install -y infisical **One-time authentication:** .. code-block:: bash infisical login This opens a browser for GitHub OAuth and saves a token locally at ``~/.config/infisical/config.json``. CLI Usage Examples ^^^^^^^^^^^^^^^^^^ **Get a specific secret:** .. code-block:: bash infisical secrets get API_TOKEN \ --projectId=64abc123def456 \ --env=development **Export all secrets to .env file:** .. code-block:: bash infisical export \ --projectId=64abc123def456 \ --env=development > .env **Run a command with secrets injected:** .. code-block:: bash infisical run \ --projectId=64abc123def456 \ --env=development \ -- python analysis_script.py The command runs with all project secrets available as environment variables. **List all secrets (without values):** .. code-block:: bash infisical secrets list \ --projectId=64abc123def456 \ --env=development Finding Project IDs ^^^^^^^^^^^^^^^^^^^ Project IDs are available in the Web UI: 1. Navigate to your project 2. Go to **Settings** → **General** 3. Copy the **Project ID** Alternatively, list all accessible projects: .. code-block:: bash infisical projects list Managing Secrets ---------------- Adding Secrets ^^^^^^^^^^^^^^ **Via Web UI:** 1. Navigate to project and environment 2. Click **"Add Secret"** 3. Enter key (e.g., ``DATABASE_PASSWORD``) 4. Enter value 5. Click **"Save"** **Via CLI:** .. code-block:: bash infisical secrets set DATABASE_PASSWORD "newpassword123" \ --projectId=64abc123def456 \ --env=production Updating Secrets ^^^^^^^^^^^^^^^^ **Via Web UI:** 1. Navigate to the secret 2. Click **"Edit"** 3. Update value 4. Click **"Save"** Changes propagate immediately to all systems pulling from Infisical. **Via CLI:** .. code-block:: bash infisical secrets set API_TOKEN "new-token-value" \ --projectId=64abc123def456 \ --env=production Rotating Credentials ^^^^^^^^^^^^^^^^^^^^ When rotating credentials (e.g., database passwords, API tokens): 1. Update the secret in Infisical (Web UI or CLI) 2. Update the credential in the target system (database, API service) 3. Trigger redeployment or restart services to pick up new value Services using ``infisical run`` will pick up changes on next restart. For long-running services, a redeployment is required. Deleting Secrets ^^^^^^^^^^^^^^^^ **Via Web UI:** 1. Navigate to the secret 2. Click **"Delete"** 3. Confirm deletion **Via CLI:** .. code-block:: bash infisical secrets delete OLD_API_KEY \ --projectId=64abc123def456 \ --env=development SSH Access Management --------------------- Store SSH credentials as secrets for centralized management and audit trails. Storing SSH Credentials ^^^^^^^^^^^^^^^^^^^^^^^^ **Username/Password access:** Create secrets in the Infrastructure project: - ``SSH_SERVER01_USER``: ``admin`` - ``SSH_SERVER01_PASSWORD``: ``securepassword`` - ``SSH_SERVER01_HOST``: ``server01.observatory.local`` **SSH Private Keys:** For private keys (multi-line values): 1. Web UI: Paste the entire private key including headers (``-----BEGIN OPENSSH PRIVATE KEY-----``) 2. CLI: Use quotes and preserve newlines: .. code-block:: bash infisical secrets set SSH_DEPLOY_KEY "$(cat ~/.ssh/deploy_key)" \ --projectId=64abc123def456 \ --env=production Retrieving SSH Credentials ^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Interactive lookup (Web UI):** 1. Navigate to Infrastructure project → Production environment 2. Find ``SSH_SERVER01_PASSWORD`` 3. Click to reveal and copy 4. Use in terminal: ``ssh admin@server01.observatory.local`` **CLI retrieval:** .. code-block:: bash # Get password PASSWORD=$(infisical secrets get SSH_SERVER01_PASSWORD \ --projectId=64abc123def456 \ --env=production \ --plain) # Use with sshpass (if needed) sshpass -p "$PASSWORD" ssh admin@server01.observatory.local **SSH key usage:** .. code-block:: bash # Export key to temporary file infisical secrets get SSH_DEPLOY_KEY \ --projectId=64abc123def456 \ --env=production \ --plain > /tmp/deploy_key chmod 600 /tmp/deploy_key ssh -i /tmp/deploy_key user@server.local rm /tmp/deploy_key **Best practices:** - Use SSH keys instead of passwords where possible - Store connection details (host, user, port) alongside credentials - Document which key/credential is for which server - Rotate SSH passwords regularly - Use service accounts for automated access CI/CD Integration ----------------- GitHub Actions ^^^^^^^^^^^^^^ **Self-Hosted Runners:** Our GitHub Actions runners operate on observatory infrastructure with access to Infisical. **Accessing secrets in workflows:** .. code-block:: yaml name: Build and Deploy on: push: branches: [main] jobs: build: runs-on: [self-hosted, linux] steps: - uses: actions/checkout@v4 - name: Pull secrets from Infisical run: | infisical export \ --projectId=${{ secrets.INFISICAL_PROJECT_ID }} \ --env=production \ --token=${{ secrets.INFISICAL_TOKEN }} > .env - name: Build application run: | source .env docker build -t myapp:latest . - name: Run with secrets run: | infisical run \ --projectId=${{ secrets.INFISICAL_PROJECT_ID }} \ --env=production \ --token=${{ secrets.INFISICAL_TOKEN }} \ -- docker-compose up -d **Required GitHub Secrets:** Configure these in your repository settings (Settings → Secrets and variables → Actions): - ``INFISICAL_TOKEN``: Service token for the project - ``INFISICAL_PROJECT_ID``: Project ID from Infisical **Creating service tokens:** 1. In Infisical Web UI, navigate to project 2. Go to **Settings** → **Service Tokens** 3. Click **"Create Service Token"** 4. Set name (e.g., ``github-actions-production``) 5. Set environment (Production, Staging, or Development) 6. Set permissions (Read or Read/Write) 7. Copy the generated token (starts with ``st.``) 8. Add to GitHub repository secrets Jenkins Integration ^^^^^^^^^^^^^^^^^^^ Jenkins jobs SSH to target servers where Infisical CLI is installed. **Deployment flow:** .. code-block:: text GitHub Actions → Build containers → Trigger Jenkins ↓ Jenkins → SSH to server → Execute deploy script ↓ Server → infisical run -- docker-compose up -d **Jenkins pipeline example:** .. code-block:: groovy pipeline { agent any stages { stage('Deploy to Production') { steps { sshagent(['production-ssh-key']) { sh ''' ssh deploy@prod-server-01 "/opt/deploy/deploy.sh" ''' } } } } } **Server deployment script** (``/opt/deploy/deploy.sh``): .. code-block:: bash #!/bin/bash set -e # Load service token export INFISICAL_TOKEN=$(cat /etc/infisical/token) cd /opt/myapp # Pull latest containers docker-compose pull # Deploy with secrets from Infisical infisical run \ --projectId=64abc123def456 \ --env=production \ -- docker-compose up -d echo "Deployment complete at $(date)" Server Setup with Ansible -------------------------- Servers are provisioned with Infisical CLI and service tokens via Ansible. **Ansible role structure:** .. code-block:: text roles/infisical/ ├── tasks/ │ └── main.yml └── templates/ ├── infisical-token.j2 └── deploy.sh.j2 **Service token storage:** Service tokens are stored in Ansible Vault at ``group_vars/production.yml``: .. code-block:: yaml --- infisical_token: "st.prod.abc123..." infisical_project_id: "64abc123def456" **Deploy script template** (``templates/deploy.sh.j2``): .. code-block:: bash #!/bin/bash set -e export INFISICAL_TOKEN=$(cat /etc/infisical/token) cd /opt/{{ app_name }} docker-compose pull infisical run \ --projectId={{ infisical_project_id }} \ --env={{ environment }} \ -- docker-compose up -d echo "Deployment complete" **Ansible playbook execution:** .. code-block:: bash # Setup production servers ansible-playbook -i inventory/production \ playbooks/setup-infisical.yml \ --ask-vault-pass Local Development Workflow --------------------------- Typical developer workflow for local testing: **Option 1: Generate .env file** .. code-block:: bash cd ~/projects/myapp # Pull development secrets infisical export \ --projectId=64abc123def456 \ --env=development > .env # Run application normally docker-compose up **Option 2: Run with injected secrets** .. code-block:: bash cd ~/projects/myapp # Run directly with secrets infisical run \ --projectId=64abc123def456 \ --env=development \ -- docker-compose up **Option 3: Manual lookup** For one-off scripts or notebooks: .. code-block:: bash # Get specific secret API_KEY=$(infisical secrets get OBSERVATION_API_TOKEN \ --projectId=64abc123def456 \ --env=development \ --plain) # Use in script python analyze_data.py --api-key="$API_KEY" Security Best Practices ----------------------- Network Access ^^^^^^^^^^^^^^ Infisical is accessible only from: - University network (direct) - University VPN - Whitelisted collaborator institutions - Observatory on-premises network Access is enforced via nginx-proxy IP whitelisting. Audit Logging ^^^^^^^^^^^^^ All secret access is logged in Infisical: - Who accessed which secret - When it was accessed - From which IP address - What action was taken (read, write, delete) Admins can review logs in **Settings** → **Audit Logs**. Secret Rotation ^^^^^^^^^^^^^^^ Rotate credentials regularly: 1. **Monthly:** API tokens for external services 2. **Quarterly:** Database passwords 3. **Annually:** SSH keys 4. **Immediately:** When team member leaves or credential compromised Update in Infisical first, then update in target system, then redeploy. Least Privilege ^^^^^^^^^^^^^^^ - Use environment-specific secrets (don't use production secrets in development) - Grant minimum required access (Viewer vs Member vs Admin) - Use service tokens with read-only access where possible - Scope service tokens to specific environments Never Commit Secrets ^^^^^^^^^^^^^^^^^^^^ Secrets should never appear in: - Git repositories - Docker images - Log files - Slack messages - Email Use Infisical for all credential storage and retrieval. Troubleshooting --------------- CLI Authentication Issues ^^^^^^^^^^^^^^^^^^^^^^^^^ **Problem:** ``infisical login`` fails **Solution:** 1. Ensure you can access ``https://infisical.data.ccat.uni-koeln.de`` in browser 2. Check VPN connection 3. Verify GitHub OAuth app is authorized in GitHub settings 4. Try re-logging: ``rm ~/.config/infisical/config.json && infisical login`` Service Token Invalid ^^^^^^^^^^^^^^^^^^^^^^ **Problem:** CI/CD fails with "Invalid token" **Solution:** 1. Verify token in Infisical UI (Settings → Service Tokens) 2. Check token hasn't expired 3. Regenerate token and update in GitHub Secrets or Ansible Vault 4. Ensure token has correct environment permissions Secrets Not Updating ^^^^^^^^^^^^^^^^^^^^ **Problem:** Changed secret in Infisical but application still uses old value **Solution:** 1. For ``infisical run`` commands: Restart the service 2. For exported ``.env`` files: Re-run ``infisical export`` 3. For containerized apps: Redeploy (``docker-compose up -d``) 4. Check that service token has read access to the environment Permission Denied ^^^^^^^^^^^^^^^^^ **Problem:** "You don't have permission to access this project" **Solution:** 1. Verify your GitHub team membership at ``https://github.com/orgs/YOUR_ORG/teams`` 2. Ask admin to add your team to the Infisical project 3. Log out and back in to refresh permissions 4. Check you're accessing the correct environment (Production vs Development) Support and Documentation ------------------------- - **Infisical Web UI:** https://infisical.data.ccat.uni-koeln.de - **Official Documentation:** https://infisical.com/docs - **CLI Reference:** https://infisical.com/docs/cli/overview - **Internal Issues:** Contact ``@observatory/infrastructure`` team - **Emergency Access:** Use university VPN + Web UI for credential lookup Appendix: Common Commands -------------------------- .. code-block:: bash # Authentication infisical login infisical logout # List projects infisical projects list # List secrets in project infisical secrets list --projectId= --env= # Get specific secret infisical secrets get --projectId= --env= # Get secret value only (for scripting) infisical secrets get --projectId= --env= --plain # Set secret infisical secrets set "" --projectId= --env= # Delete secret infisical secrets delete --projectId= --env= # Export to .env infisical export --projectId= --env= > .env # Run command with secrets infisical run --projectId= --env= -- # Using service token (CI/CD) infisical export --token= --env= > .env infisical run --token= --env= --