How we consolidate data in the Spencer cloud layer

profile picture

Kevin Engineer

20 Apr 2018  ·  6 min read


For our product company Spencer – a tool that makes employees’ lives easier by uniting different processes and tools into one app – we had to figure out a way to handle data from different sources.

The main idea behind Spencer is that the product doesn’t provide data – instead, it’s a cloud-based middleware on top of the existing tools a company uses. Spencer consists of different modules, that each tap into data and tools the client already has. The goal is to unite all this data in one mobile interface, visualised and organised consistently. The problem we face a lot is that different clients use different tools for the same data. In this post, I’ll explain how the cloud layer of Spencer tackles this easily!

Let’s dive right into it with an example. Imagine a “News” module in Spencer, designed to display important news and updates to a company’s employees. Now, this module should be implemented for different clients. Let’s say client A’s news feed comes from an RSS feed, while client B’s comes from a Drupal JSON feed, or from a custom-made internal communication platform.

The problem here is that we can’t just connect to those services and proxy the data back raw to the mobile application. This would be a nightmare for maintainability, because every mobile platform would need to know how to process any of those services and integrate it into one feed.

So to do better, we need to solve two problems:

  1. We need to make sure the REST API response of the cloud layer is consistent to the application.
  2. For scalability, we want to provide a plug-and-play solution, even with different kinds of data sources.

So let’s tackle the first problem first.

Problem 1: consistent data output

This is where Spencer’s cloud layer – written in PHP 7.1 & Symfony 2.8 – really comes into play. We first analyse the module to see what kind of data our REST API should return to the application. For our mobile application, it is essential that our REST API does not change its response for different clients or platforms. All the processing and thinking is done on the cloud layer; this allows us to reuse the app for different clients.

So let’s look at a news article in detail (I’ll use a simplified example here):

A news item must have the following properties:

  • title
  • content
  • published_at
  • author

Now, we built a news bundle into our cloud layer. This is a generic bundle that can be used for multiple clients, meaning it must not have any specific client code or configuration. Behind this news bundle are several connector bundles that provide the data. The news bundle contains the following structure:

Controller
        NewsController => Which exposes a consistent REST API for the mobile application
    Chain
        NewsItemProviderChain => This will hold all the connector NewsItemProviders
        NewsItemProviderChainInterface => Interface for the NewsItemProviderChain
    Model
        NewsItem => The defined news item model with the properties discussed above
    Provider
        NewsItemProvider => Provides news items to the NewsController
        NewsItemProviderInterface => A contract for our connector bundles & NewsProvider

And with that, we’ve already tackled our first problem. The mobile applications can rest assured that the API they’re using will output a consistent response.

Problem 2: Handling different data sources

Our second problem was a bit harder to solve, because we want our news bundle to be unaware of the external systems that provide news items. Here’s where Symfony’s tagged services come into play. We came up with the idea to build one big chain that can hold these services. The NewsProvider of the news bundle will use this chain to consolidate all the data back into one single feed.

So let’s start digging into the code, shall we?

You saw above that we currently have one news bundle that only holds a controller, model, and provider. What we’ll build now are all the external connectors as a bundle. Using the example from the beginning of this post, we create an RSS bundle and a Drupal JSON feed bundle. Both of these bundles must have a NewsItemProvider that implements the NewsItemProviderInterface of the news bundle.

When we define those news item providers as a service, we include a tag.

    spencer_rss.news_item_provider:
        class: 'Spencer\RSSBundle\Provider\NewsItemProvider'
        tags:
            - { name: spencer_news.news_item_provider }
    
    spencer_drupal_json_feed.news_item_provider:
        class: 'Spencer\DrupalJSONFeedBundle\Provider\NewsItemProvider'
        tags:
            - { name: spencer_news.news_item_provider }

At this point, we have an empty news bundle and two connector bundles that both have a service tagged as spencer_news.news_item_provider. Before we explain how we chained these services, let’s look at what this Chain class contains:

    <?php

    namespace Spencer\NewsBundle\Chain;
    
    use Spencer\NewsBundle\Provider\NewsItemProviderInterface;
    
    /**
     * This chain will have all the news item providers that are configured with tag spencer_news.news_item_provider
     *
     * @package Spencer\NewsBundle\Chain
     */
    class NewsItemProviderChain
    {
        /**
         * @var array
         */
        private $newsItemProviders;

        public function __construct()
        {
            $this->newsItemProviders = array();
        }
    
        /**
         * @param NewsItemProviderInterface $newsItemProvider
         */
        public function addNewsItemProvider(NewsItemProviderInterface $newsItemProvider)
        {
            $this->newsItemProviders[] = $newsItemProvider;
        }
    
        /**
         * @return array
         */
        public function getNewsItemProviders()
        {
            return $this->newsItemProviders;
        }
    }

Define it as a service:


    spencer_news.news_item_provider_chain:
        class: 'Spencer\NewsBundle\Chain\NewsItemProviderChain'

It’s a very simple class that just has an addNewsItemProvider function to add a news provider to the chain and a getNewsItemProviders to get all the news item providers from the chain.

The next step is to create a CompilerPass That will take the two tagged services and chain them in the NewsItemProviderChain from above.

    <?php
    
    namespace Spencer\NewsBundle\DependencyInjection\Compiler;
    
    use Symfony\Component\DependencyInjection\ContainerBuilder;
    use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
    use Symfony\Component\DependencyInjection\Reference;

    /**
     * @package Spencer\NewsBundle\DependencyInjection\Compiler
     */
    class NewsProviderChainPass implements CompilerPassInterface
    {
        /**
         * @param ContainerBuilder $container
         */
        public function process(ContainerBuilder $container)
        {
            // always first check if the primary service is defined
            if (!$container->has('spencer_news.news_item_provider_chain')) {
                return;
            }
    
            // get chain definition
            $definition = $container->findDefinition('spencer_news.news_item_provider_chain');
    
            // find all service IDs with the spencer_news.news_item_provider tag
            $taggedServices = $container->findTaggedServiceIds('spencer_news.news_item_provider');
    
            foreach ($taggedServices as $id => $tags) {
                $definition->addMethodCall('addNewsItemProvider', new Reference($id));
            }
        }
    }

Add the compiler pass to the bundle:

    <?php

    namespace Spencer\NewsBundle;
    
    use Spencer\NewsBundle\DependencyInjection\Compiler\NewsProviderChainPass;
    use Symfony\Component\DependencyInjection\ContainerBuilder;
    use Symfony\Component\HttpKernel\Bundle\Bundle;
    
    /**
     * @package Spencer\NewsBundle
     */
    class SpencerNewsBundle extends Bundle
    {
        /**
         * @param ContainerBuilder $container
         */
        public function build(ContainerBuilder $container)
        {
            parent::build($container);
    
            // add NewsProviderChainPass
            $container->addCompilerPass(new NewsProviderChainPass());
        }
    }

Now when the Symfony application starts, the NewsProviderChainPass will add all services tagged as spencer_news.news_item_provider to the NewsProviderChain.

Our chain is now ready to be used wherever you need to fetch all the news items. Let’s look at the NewsProvider of the news bundle.

We first need to inject it in our provider:

    spencer_news.news_item_provider:
            class: Spencer\NewsBundle\Provider\NewsItemProvider
            arguments:
                - '@spencer_news.news_item_provider_chain'

Next, we can fetch all the news item from every configured provider:

    /**
     * @var NewsItemProviderChainInterface
     */
    private $newsItemProviderChain;
    
    /**
     * @param NewsItemProviderChainInterface $newsItemProviderChain
     */
    public function __construct(NewsItemProviderChainInterface $newsItemProviderChain)
    {
        $this->newsItemProviderChain = $newsItemProviderChain;
    }
    
    public function getAllNewsItems()
    {
        $configuredProviders = $this->newsItemProviderChain->getNewsItemProviders();
        $newsItems           = array();
        
        foreach ($configuredProviders as $provider) {
            $newsItems[] = $provider->getNewsItems();
        }
        
        return $newsItems;
    }

And that’s it! The NewsItemProvider from the news bundle can get all the news items, and doesn’t need to know where those items are coming from. This gives us the possibility to add a custom news item provider without even touching the news bundle’s code. Other providers can be in a standalone bundle – all the providers need is a tag spencer_news.news_item_provider for the NewsItemProvider to pick it up.

This is how our news bundle merges all news items from all different sources into one consistent API response that will never change for the different clients.

This was just one of the many methods Spencer uses to consolidate data into a consistent data source. Ofcourse this won’t be a silver bullet for every problem, but for us it solved 1 of the problems we face when working with multiple bundles that need a plug and play solution.