Bokbasen provides a wide range of services to all parts of Norway’s publishing community. The company builds and maintains the Den norske Bokdatabasen catalogue containing data about all Norwegian publishers. By making use of this single centralized register, the industry can easily make use of the data – in online stores, bookshops, libraries and schools – instead of spending time gathering it, as is the focus in many other countries. When e-books entered the market, Bokbasen was quick to build the infrastructure to support them, including a streaming service for audiobooks. AWS partner Basefarm has been hosting and managing operations for Bokbasen in its private cloud since 2012.
In 2017 Bokbasen started utilizing AWS for storing and transcoding of audio files. In 2018 Bokbasen migrated their public website to AWS.
In 2019, Bokbasen set out on a new mission to digitize books and learning materials, beginning with Digitalelev, a product for management and procurement of digital learning materials in schools, and Allvit, a platform for distribution of textbooks and resources for higher education.
Solution
Basefarm and Bokbasen evaluated several tools which could be used to monitor the necessary services in AWS and found Datadog to be the best fit.
Datadog has built in integrations with many AWS services, making it easy to configure monitoring and get access to metrics from a broad spectrum of services from both AWS and third parties. This service is delivered as a managed service by Basefarm and has integrations with reporting and alert mechanisms.
Bokbasen chose to utilize Application Performance Management (APM), to give them better insights into the performance of the application services.
In partnership with Basefarm, Bokbasen designed and implemented monitoring capabilities, addressing the following aspects:
- Service Intelligence monitoring
- Use of statistical analysis algorithms to identify outliers and anomalies
- Use of Machine learning for heterogenous monitoring by identifying trends in data to trigger actions prior to anomalies or threshold breaches being detected
Datadog supports a wide variety of alerting mechanisms, for Bokbasen it was agreed that all alerts should be sent to a common slack channel, alerts for Basefarm managed services should be sent to Basefarm ITSM. This can be summarized as shown below:
Service intelligence monitoring capabilities
Datadog is one of the market leaders for monitoring, with their various integrations, it is easy to enable monitoring for a wide variety of workloads. AWS Fargate manages the scaling of the underlying infrastructure needed to support the container workloads in a highly-available manner. Thousands of containers can be easily launched on demand, within seconds. Utilizing the Datadog modules for Fargate, ECS and Lambda functions, Basefarm is able to provide monitoring for the platform services and infrastructure, as well as the compute resources supporting the Bokbasen services.
Each container workload is tagged as part of its definition, with its cluster and groups, based on this tagging, new containers and functions are automatically added to the correct monitoring groups and configurations are automatically applied when changes to the environment are initiated. Bokbasen has applied APM to their containers, and this enables full traceability of performance and data flows between the distributed microservices.
Anomaly detection based on statistical methods
Bokbasen has currently almost no ability to predict how the new applications will be used. Datadog supports creation of metrics from a wide variety of sources using their Agent software or directly from AWS to identify anomalies and outliers.
Bokbasen and Basefarm are working together on several use cases to identify relevant patterns to detect such outliers and anomalies. One example is addressing response times in their digitalelev-gui application.
Datadog supports a wide set of variables for detecting anomalies, with algorithms which can be tuned to match the needs of the solutions, such as:
- Basic – Metrics with no repeating seasonal pattern
- Agile – Seasonal metrics expected to shift, the algorithm should quickly adjust to metric level shifts
- Robust – Seasonal metrics expected to be stable, slow level shifts are considered anomalies
More details on advanced options for this type of monitoring are available in Datadog’s Anomaly monitor documentation.
Using Machine Learning to identify trends in data and identifying possible problems.
As Bokbasen is continuously developing new services, it is difficult to predict how the application/services will scale. Analysis of trends is therefore important.
Datadog has statistical tools to do forecasting against collected metrics.
Forecasting is especially suited for metrics that have strong trends or recurring patterns.
A use case for forecasting performance of Bokbasens RDS database services has been developed and can easily be adapted to any metric.
Another feature Datadog offers, is Watchdog. Watchdog is an algorithmic feature for APM which automatically detects potential application and infrastructure issues. These features include all of Datadog’s established machine learning features such as anomaly and outlier detection, as well as forecasting. All findings are assigned a “story”, based on these it is easy to create additional monitoring. This feature enables Bokbasen and Basefarm to automatically detect possible problems and address them as needed.
In addition, Basefarm, has enabled GuardDuty on the Bokbasen production account. GuardDuty uses Machine Learning based on a set of AWS log sources. Basefarm has implemented this using Security Hub for visibility to the customer, with integrations to the Basefarm IT service management system. If a new GuardDuty alarm is raised, it will automatically create a ticket to be handled by Basefarms 24/7 Operation Center.
Advantages
Bokbasen is now able to benefit from next generation monitoring through use of native AWS services and Datadog. With its Managed Service Provider Basefarm and its 24/7 Operations Center they now address:
- Service Intelligence monitoring, through AWS ECS, Fargate and Lambda with Datadog monitoring services, providing access to current and historical data for all gathered sources, including APM.
- Anomaly detection, Datadog captures metrics that is analyzed using statistical methods, giving better insight than basic threshold monitoring.
- Using Machine Learning to identify trends in data and identifying possible problems. Watchdog can identify possible performance issues before they arise, GuardDuty can detect possible malicious threats.
The above Monitoring capabilities can raise incidents with Basefarm’s ITSM system, so issues that arise are managed by the 24/7 Operations Center.
About Basefarm
Basefarm is a European managed service provider creating market leaders, by integrating key skills of big data, cloud services and information security into a service offering.
The company provides strategic advice, architecture and implementation together with the management and operation of solutions to several different cloud platforms.
The company provides strategic advice, architecture and implementation together with the management and operation of solutions to several different cloud platforms.
The business was founded in 2000 in the Nordic countries and today there are 550 leading engineers and advisors working in Norway, Sweden, the Netherlands, Germany and Austria.
Basefarm was ranked highest in Whitelane’s IT Outsourcing study Nordics 2018 and was approved by Gartner as cool vendor supplier at the European Cloud Computing Market 2015 for our unique methodology in application operations.
In August 2018, Basefarm was acquired by Orange Group and is now an Orange Business Service subsidiary.
