Background
GreenSpark Analytics specialises in data-driven energy trading across foreign markets with growing operations in Ireland, the United Kingdom, and the Netherlands. As trading volumes continue to increase, automation and algorithmic trading have become essential to ensure faster decision-making and remove emotional influences from the trading process. The GreenSpark Analytics platform is central to this data-driven business, supported by a skilled engineering team who had already migrated much of their workloads to AWS.
The stuff we tried already took most of last year. We didn’t think anything impactful could be done in that timeframe.
The Challenge
GreenSpark presented their primary technology challenges to fourTheorem, with a clear prioritisation across data engineering, business, and platform activities. These challenges included:
- Complex data integration across multiple systems
- Performance bottlenecks in the existing infrastructure
- Manual operations consume valuable engineering time
- Outdated security practices
- Limited monitoring capabilities
- Scaling constraints as data volumes increased
The Solution
Infrastructure as Code
- Implemented Terraform Cloud with GitHub integration to enable robust GitOps workflows
- Integrated GitHub Actions for CI/CD pipelines
- Built and pushed container images to a shared AWS account
Security
- Established a multi-account AWS Organisation with Service Control Policy (SCP) guardrails
- Integrated IAM Identity Centre with GreenSpark’s Identity Provider (IdP)
- Defined permission sets aligned with teams and job roles
- Centralised monitoring using AWS CloudTrail and Config
- Delegated a security account as admin with detective security controls
- Replaced legacy self-hosted OpenVPN with AWS Client VPN
Containerisation
- Migrated workloads from EC2 to ECS Fargate with automated deployment pipelines
- Implemented ephemeral containerised tasks using ECS Fargate and EventBridge Scheduled Rules
- Developed a hosting pattern for Streamlit apps on ECS Fargate using Application Load Balancer (ALB) and authentication
Analytics
- Deployed read replicas to separate production and analytics workloads
- Integrated Amazon DocumentDB (MongoDB-compatible) with existing analytics tools
The Outcome
Accelerated Maturity
In just weeks, GreenSpark achieved a level of maturity in software development, infrastructure, and platform operations that typically takes 12–18 months to build internally.
Time to Market
Remove Engineers can create the infrastructure and automation for new services or features in minutes or hours.
Landing Zone
Accelerated set up of AWS Landing Zone using best-practices blueprints, saving time, effort, and money required to set up each account manually.
Developer Productivity
Onboarding new team members is streamlined with SSO access to AWS. Infrastructure is well-documented, and new development resources are just a Pull Request away.
Operation Excellence
GreenSpark’s infrastructure is treated as cattle, not pets – automated, scalable, and hands-off. Manual processes are minimised. Deployments are robust and highly available. Changes are made through small, peer-approved Pull Requests.