Basefarm brings Next Generation Monitoring Capabilities to Bokbasen

Bokbasen provides a wide range of services to all parts of Norway’s publishing community. The company builds and maintains the Den norske Bokdatabasen catalogue containing data about all Norwegian publishers.  By making use of this single centralized register, the industry can easily make use of the data – in online stores, bookshops, libraries and schools – instead of spending time gathering it, as is the focus in many other countries.  When e-books entered the market, Bokbasen was quick to build the infrastructure to support them, including a streaming service for audiobooks. AWS partner Basefarm has been hosting and managing operations for Bokbasen in its private cloud since 2012.

In 2017 Bokbasen started utilizing AWS for storing and transcoding of audio files. In 2018 Bokbasen migrated their public website to AWS.

In 2019, Bokbasen set out on a new mission to digitize books and learning materials, beginning with Digitalelev, a product for management and procurement of digital learning materials in schools, and Allvit, a platform for distribution of textbooks and resources for higher education.

The Challenge

Bokbasen is a small organization, their primary focus is the development of their services. Bokbasen chose to develop their new services using containers and serverless technologies. All infrastructure required to support the Bokbasen services is defined using Infrastructure as Code.

As the demand of the services is variable depending on the time and day, it was a key design requirement to utilize pay-as-you-go services, that are able to scale resource consumption quickly to meet demand and scale back, during periods of low usage.

It was quickly identified that the traditional monitoring solutions that Bokbasen had in place to manage their traditional workloads, would not meet the requirements of their new architecture. In order to effectively manage and monitor the end to end solution and facilitate both Bokbasen and Basefarm engineering staff to manage the applications, middleware and AWS services, it was necessary to implement new tooling.

Solution

Basefarm and Bokbasen evaluated several tools which could be used to monitor the necessary services in AWS and found Datadog to be the best fit.

Datadog has built in integrations with many AWS services, making it easy to configure monitoring and get access to metrics from a broad spectrum of services from both AWS and third parties. This service is delivered as a managed service by Basefarm and has integrations with reporting and alert mechanisms.

Bokbasen chose to utilize Application Performance Management (APM), to give them better insights into the performance of the application services.

In partnership with Basefarm, Bokbasen designed and implemented monitoring capabilities, addressing the following aspects:

Service Intelligence monitoring
Use of statistical analysis algorithms to identify outliers and anomalies
Use of Machine learning for heterogenous monitoring by identifying trends in data to trigger actions prior to anomalies or threshold breaches being detected

Datadog supports a wide variety of alerting mechanisms, for Bokbasen it was agreed that all alerts should be sent to a common slack channel, alerts for Basefarm managed services should be sent to Basefarm ITSM. This can be summarized as shown below:

Service intelligence monitoring capabilities

Datadog is one of the market leaders for monitoring, with their various integrations, it is easy to enable monitoring for a wide variety of workloads. AWS Fargate manages the scaling of the underlying infrastructure needed to support the container workloads in a highly-available manner. Thousands of containers can be easily launched on demand, within seconds. Utilizing the Datadog modules for Fargate, ECS and Lambda functions, Basefarm is able to provide monitoring for the platform services and infrastructure, as well as the compute resources supporting the Bokbasen services.

Each container workload is tagged as part of its definition, with its cluster and groups, based on this tagging, new containers and functions are automatically added to the correct monitoring groups and configurations are automatically applied when changes to the environment are initiated. Bokbasen has applied APM to their containers, and this enables full traceability of performance and data flows between the distributed microservices.

Anomaly detection based on statistical methods

Bokbasen has currently almost no ability to predict how the new applications will be used. Datadog supports creation of metrics from a wide variety of sources using their Agent software or directly from AWS to identify anomalies and outliers.

Bokbasen and Basefarm are working together on several use cases to identify relevant patterns to detect such outliers and anomalies. One example is addressing response times in their digitalelev-gui application.

Datadog supports a wide set of variables for detecting anomalies, with algorithms which can be tuned to match the needs of the solutions, such as:

Basic – Metrics with no repeating seasonal pattern
Agile – Seasonal metrics expected to shift, the algorithm should quickly adjust to metric level shifts
Robust – Seasonal metrics expected to be stable, slow level shifts are considered anomalies

More details on advanced options for this type of monitoring are available in Datadog’s Anomaly monitor documentation.

Using Machine Learning to identify trends in data and identifying possible problems.

As Bokbasen is continuously developing new services, it is difficult to predict how the application/services will scale. Analysis of trends is therefore important.

Datadog has statistical tools to do forecasting against collected metrics.

Forecasting is especially suited for metrics that have strong trends or recurring patterns.

A use case for forecasting performance of Bokbasens RDS database services has been developed and can easily be adapted to any metric.

Another feature Datadog offers, is Watchdog. Watchdog is an algorithmic feature for APM which automatically detects potential application and infrastructure issues. These features include all of Datadog’s established machine learning features such as anomaly and outlier detection, as well as forecasting. All findings are assigned a “story”, based on these it is easy to create additional monitoring. This feature enables Bokbasen and Basefarm to automatically detect possible problems and address them as needed.

In addition, Basefarm, has enabled GuardDuty on the Bokbasen production account. GuardDuty uses Machine Learning based on a set of AWS log sources. Basefarm has implemented this using Security Hub for visibility to the customer, with integrations to the Basefarm IT service management system. If a new GuardDuty alarm is raised, it will automatically create a ticket to be handled by Basefarms 24/7 Operation Center.

Advantages

Bokbasen is now able to benefit from next generation monitoring through use of native AWS services and Datadog. With its Managed Service Provider Basefarm and its 24/7 Operations Center they now address:

Service Intelligence monitoring, through AWS ECS, Fargate and Lambda with Datadog monitoring services, providing access to current and historical data for all gathered sources, including APM.
Anomaly detection, Datadog captures metrics that is analyzed using statistical methods, giving better insight than basic threshold monitoring.
Using Machine Learning to identify trends in data and identifying possible problems. Watchdog can identify possible performance issues before they arise, GuardDuty can detect possible malicious threats.

The above Monitoring capabilities can raise incidents with Basefarm’s ITSM system, so issues that arise are managed by the 24/7 Operations Center.

About Basefarm

Basefarm is a European managed service provider creating market leaders, by integrating key skills of big data, cloud services and information security into a service offering.
The company provides strategic advice, architecture and implementation together with the management and operation of solutions to several different cloud platforms.

The business was founded in 2000 in the Nordic countries and today there are 550 leading engineers and advisors working in Norway, Sweden, the Netherlands, Germany and Austria.

Basefarm was ranked highest in Whitelane’s IT Outsourcing study Nordics 2018 and was approved by Gartner as cool vendor supplier at the European Cloud Computing Market 2015 for our unique methodology in application operations.

In August 2018, Basefarm was acquired by Orange Group and is now an Orange Business Service subsidiary.