These are my notes about what DevSecOps effectively adds to DevOps.
With DevSecOps, security checks are pushed to the beginning and are part of all development phases (Design/Plan/Develop/Build).
Key Points
- Secure SDLC (aka SSDLC)
- Source-code quality check,
- Dependency management,
- Scan artifacts,
- Threat modeling,
- Unity and integration tests,
- and more.
- Secure CI/CD
- Minimized permissions,
- Defending against pipeline attacks or weaponization (no shared build agent),
- Absolute segmentation for applications/data/systems/environments/etc,
- and more.
- Infrastructure as Code (IaC)
- Secret management,
- Leveraging of GitOps best practices,
- Monitoring and alerting,
- and more.
- Process
- Add security requirements to story templates, review checklists, and refinement sessions.
- A risk assessment decision matrix for features, assets, resources, etc is required.
- Only allow building on top of approved base images.
- Stagged/gradual rollout of new versions with fallback capabilities.
Challenges to Overcome
- Security must not be siloed but a company-wide discipline,
- Visibility (normalized logs, tailored alerts, behavioral monitoring, etc) is paramount,
- Prioritization for security issues should be part of the company culture,
- Due diligence before using any tool or library plus up-to-date software bill of material,
- Mastering and testing on a sandbox before pushing it into production,
- Continuous watch for Developer Bypasses before they leak Dev.
Metrics
- Meantime to Production (MTTP)
- The time it takes to get a whole new app deployed to production from its first commit.
- Meantime to Recover (MTTR)
- It refers to the ability of the infrastructure to recover from a failed deployment (rollback) or a service interruption (self-healing).
- Deployment Agility
- A combination of the speed and frequency of deployments.
- Failure Rate
- Calculates the percentage of deployments that contained a bug in relation to the total of deployments. This reveals the size of the opportunity for improvements.
- Communication of Risks
- In DevOps, the comparison of these metrics over time shows how much impact changes caused during a period. But in DevSecOps, risk means the likelihood of a vulnerability being exploited and its impact on systems.
Automated Tests
- Unit Test
- Tests if the application works (not broken) and if it responds as expected (a few or all end-points of an API) when compared with a source of truth.
- Integration Test
- Focus on testing the integration between applications (connectivity, authentication, etc) and also tests for regression.
- Security Test
- SAST (Static) – Source code analysis.
- Easy to be integrated into the code review at Pull Request (PR) / Merge Request (MR) time.
- DAST (Dynamic) – Fuzzing and automated “pentesting”.
- Can be triggered at promoting to staging, where intrusive tests can safely hammer the application in a safe environment.
- SAST (Static) – Source code analysis.
K8s
DevSecOps approach to Kubernetes
- Integrates security at every stage of the container lifecycle: development, deployment, and operations.
- Applies security through the CI/CD pipeline all the way into Kubernetes runtime (not as an afterthought).
- Scans container image for vulnerable dependencies, usable of minimalistic base images, non-root user and non-privileged container execution whenever possible, and prefer read-only filesystems.
- Policy as Code (PaC) / RBAC / ACL: strict or limit inter-pod communication and access to other resources.
- Leverage Key Vaults for secrets or certificates and use environment-specific variables provided at start time.
- Store logs in an external centralized location for audits and monitoring.
- PSS defines security policies at 3 levels (see below) at a namespace or cluster-wide level.
- Privileged: near unrestricted policy and even allowing privilege escalation.
- Baseline: minimally restricted policy and prevent privilege escalations but allows deployment of pods with default configuration.
- Restricted: heavily restricted policy follows the current pod hardening best practices
- PSA enforces these PSS by intercepting API server requests and applying these policies.
TOOLS
- Source-code scanners:
- Search for credentials:
- Dependency management and SBOM:
- OSV-Scanner [Link]
- Scan CT and VM images for vulnerable packages, secrets, misconfigurations, and more:
- Optimize container images:
- Slim [Link]
- Scans IsC manifests for errors, and misconfigurations, and suggests best practices:
- Automate pentesting in the pipeline:
- OWASP ZAP [Link]
- QA tests for web apps on the browser:
- Cypress [Link]
- Real-time behavioral security analysis:
- Falco [Link]
PROVISIONING INFRASTRUCTURE vs CONFIGURATION MANAGEMENT & AUTOMATION
The overlaps in capabilities of the tools frequently distort the comprehension of the concepts. While both provisioning and configuration management tools aim to automate IT processes, their core purposes differ fundamentally. Understanding these differences helps in choosing the right tool for the right task.
Overlap on both concepts:
- Declarative approach
- Define the desired state, and the tool ensures it is provisioned accordingly.
- Idempotency
- Ensures that applying the same configuration multiple times produces the same result.
- State enforcement
- Ensures that systems remain in the desired configuration.
- Source control
- Provides consistency, version tracking, and collaboration in managing infrastructure and configurations.
Distinct concepts:
Infrastructure as Code (IaC)
Infrastructure provisioning focuses on creating, modifying, and managing infrastructure resources such as virtual machines, networks, storage, and cloud services. The primary goal is to define cloud resources, ensuring deployment repeatability, version control, and scalability.
- Terraform
- Terraform (by Hashicorp), and OpenTofu (a community fork of Terraform), make use of state files (stateful) to remember what the resources are and their previous state.
Configuration Management & Automation
Configuration management and automation tools focus on maintaining the desired state of system and software configurations on existing infrastructure. They are typically used to install and manage software, enforce security policies, and orchestrate system-wide tasks.
- Ansible
- Ansible, on the other hand, has a stateless approach and checks the current state of each component in the scope at runtime with the usage of modules. But it can also be imperative and execute commands (like shell) to the targets.
BONUS
GitOps
A fundamental methodology that uses Git as the single source of truth for declarative infrastructure and application management. It can be used in CI/CD pipelines to automate infrastructure and application deployments, realistically reducing MTTP and MTTR to their minimum (optimum).
- GitLab / GitHub
- No matter the flavor of the service, they wrap around Git and add their own additional features on top. Each commit is a new layer identified with a checksum that cannot be tampered with or removed. Everything can be audited to the exact changes, timestamps, authors, etc.
- Jenkins
- It is responsible for identifying the triggers and running the respective tasks to accomplish the desired output.
- Example 1
- A change to a Terraform module in Git could trigger the infrastructure change by applying it to the respective environment.
- Example 2
- A modification in an Ansible playbook could trigger a task that applies the modification to a target or group of targets.
- Example 3
- A code change in an application being developed in-house could trigger the necessary tasks to make it build and get deployed into the desired environment (dev, stage, test, hotfix, prod, etc.)
- Example 1
- It is responsible for identifying the triggers and running the respective tasks to accomplish the desired output.