Next Generation Monitoring Capabilities

Ruter AS is the public transport authority for the Oslo and Akershus counties in Norway. Formally a limited company – 60% of its shares are owned by the Oslo county municipality and 40% by that of Akershus – it is responsible for the administration, funding, and marketing (but not direct operation) of public transport in the two counties, including bus, Metro, Tram and ferry services. Ruter also has agreements with the Norwegian State Railway, concerning the regulation of fares on local and regional train services within the two counties.
The development of digital services at Ruter involves many actors. At the heart of Ruter’s digital platform is Ruter’s Core Mobility Platform, a service which is fully managed by Basefarm. The Core Mobility Platform itself consists of a Fleet Logistics service and a Reporting Service built on Amazon EC2 instances where Basefarm operates the key components Kafka, Cassandra, MQ (Message Queue) and several database services on Amazon RDS.

The Challenge

Ruters Core Mobility Platform, provides services for many consumers; some examples include: suppliers that deliver IOT integration for buses; autonomous teams developing new features on Kubernetes for the public transport business and business operations staff that manage Ruter’s day to day operations.
The customer chose an open systems approach to their solutions architecture, choosing to build the services using open source components, rather than to tightly integrate with AWS specific managed services, this is to allow Ruter the flexibility to re-host the services easily in future, should they choose to do so.
As so many people depend on the availability of public transport services, it is necessary to ensure the platform is available at all times, an effective monitoring capability is therefore essential for ensuring this.

Solution

A number of complex requirements exist for monitoring the Core Mobility Platform, since a number of different consumers each need visibility of the components that they are responsible for. Ruter and Basefarm chose Datadog as the primary monitoring tool, because of its range of plugin extensions and the capability to correlate logs and Application Performance Management data within a single tool.

Datadog’s plugin approach enables Ruter and Basefarm to add-on different modules as needed. In this case study we focus on the elements of the platform where Basefarm has monitoring responsibility, which include the underlying AWS infrastructure, EC2 compute, storage and networking. All infrastructure and monitoring components are defined as IaC using HashiCorp Terraform. This enables Ruter and Basefarm to easily manage and monitor different environments. Resources are tagged as belonging to clusters and groups, which enables monitoring events and logs to be automatically associated with the correct services as scaling events are triggered and new resources are launched.

Monitoring solution for load distribution and anomalies

The services that Basefarm manage for Ruter are all distributed components, each with flexible demand and capacity requirements, requests to the service can originate from a variety of different consumers. Basefarm monitor’s the Apache Cassandra clusters using statistical analysis algorithms, these are used identify outliers and anomalies in metrics to gain improved insight in to how the service is being used and how it is performing. Basefarm has configured an outlier monitor to detect when nodes experience a high load condition. This enables Basefarm to track usage patterns and to optimize the cluster for increased performance.

The graph below shows an example of where the performance of a single node within the cluster has deviated from that of the other member nodes. This may indicate a problem with the node in question or with the way in which data is partitioned across the cluster.

Using Datadog to forecast threshold breach

Datadog collects a wide range of metrics from heterogenous sources, it employs predictive algorithms to predict how specific metrics will continue to rise or fall, based on identified trends.

Basefarm is responsible for managing the different Cassandra Clusters for Ruter, forecasting growth in compute or storage of the cluster enables Basefarm to plan scaling and maintenance activity, before performance begins to degrade, this ensures responsiveness of the cluster, without hugely overprovisioning cluster resources.

Basefarm is using the AWS GuardDuty threat detection service to continuously monitor Ruter’s productions accounts for malicious activity and unauthorized behavior. GuardDuty uses Machine Learning based on a set of AWS log sources, when an anomaly is detected, the issue is displayed in AWS Security Hub for full transparency with the customer. Basefarm has integrated AWS Security Hub alerts with our service management system, and any alerts will be handled 24/7 by Basefarm’s Operations Center.

Advantages

With the many consumers of monitoring data within Ruter’s platform, Datadog has enabled Basefarm and Ruter to provide a single monitoring platform for can be used by all teams. The implementation enables Basefarm to deliver next generation monitoring with advanced statistical analysis that provides anomaly detection, outlier identification, predictions for future growth and the capability to identify potential cost savings.

As the current product stack consists mainly of distributed components, these tools provide better overview of status and overall health of the solution. It eases the process of identifying the root cause of issues during incident management.

About Basefarm

Basefarm is a European managed service provider creating market leaders, by integrating key skills of big data, cloud services and information security into a service offering.
The company provides strategic advice, architecture and implementation together with the management and operation of solutions to several different cloud platforms.

The business was founded in 2000 in the Nordic countries and today there are 550 leading engineers and advisors working in Norway, Sweden, the Netherlands, Germany and Austria.

Basefarm was ranked highest in Whitelane’s IT Outsourcing study Nordics 2018 and was approved by Gartner as cool vendor supplier at the European Cloud Computing Market 2015 for our unique methodology in application operations.

In August 2018, Basefarm was acquired by Orange Group and is now an Orange Business Service subsidiary.