Do You Need a Microservice Catalog?

In the last few years, we've seen application architecture paradigms evolve from monolithic to microservice-based models. Now it's common to see both new applications built, as well as legacy ones rearchitected, as a collection of business function-centric microservices that interact with one another using well-defined APIs, and which are developed, deployed, and monitored independently.

The number of microservices required to implement a fully functional application varies with the complexity and scale of the business requirements, however, it's not uncommon to see 10-20 microservices at a time, or even 40-50 in some cases.

One of the most extreme examples of the scale of a microservices deployment is that of Uber, which as of 2020 was a network of 80 microservices across multiple development teams.

As the number of microservices increases, managing development, coordinating deployments, monitoring metrics, and responding to security and performance incidents promptly becomes difficult for developers, architects, and DevOps teams. There are a couple of reasons for this.

Essential knowledge about each microservice's implementation and architecture tends to get compartmentalized within each group with the result that there's no one person with complete knowledge of the end-to-end application across all of its microservices.

A global view of all the microservices documenting their current state of configuration, deployment, and APIs is missing. This is usually due to a distinct lack of communication among the microservice owners, and makes information gathering, especially during incidents, slow and unreliable.

To address these problems, a new category of products called Microservice Catalogs is now emerging. These microservice catalog products aim to help developers, architects, and DevOps teams deal with microservice dependencies, configurations, upgrades, and incident response. They capture information about each microservice in the application's stack, thus creating a catalog of sorts, which can then be used for monitoring and reporting purposes.

A lot of different kinds of information may be captured for each microservice – GitHub repo where the code lives, a link to its API documentation, its current configuration parameters, deployment status and when it was last deployed, the Slack channel where the service developers may be reached, the services it depends on, or the services that depend on it.

All of this seems interesting! Much like data catalog products, which crawl data on file servers, databases, and cloud storage to build a global repository that captures each data type's properties, shape, and sensitivity, microservice catalog products hold promise in creating a global repository of architectural and operational information about microservices.

However, I see three hurdles in the adoption of these products for teams dealing with these problems.

The first is that these products require relevant microservice information to be "pushed into" them using integrations such as webhooks. Developers are required to make changes to their services, or the CICD pipelines, so that things such as configuration parameters, deployment status, etc. may be captured in the microservice catalog. Other information, such as GitHub repo, Slack channel info, etc. must be "manually entered" into the products. This creates a steep barrier to seeing immediate value in the customer's mind.

The second is that it requires ongoing maintenance and good business process hygiene to make sure information in the catalog is never stale, or missing when new microservices are introduced, or old ones are retired.

The last is that the whole micro-service paradigm itself seems to have been pushed a bit too far to the edge, where having too many microservices (more than 20ish) is now looked at with suspicion! What this means is that most companies may be better off with a small number of microservices, in which case the additional burden of managing and maintaining a microservice catalog may not be fully justified. On the other hand, the ones that do need a lot of microservices (the Ubers of the world!) for legitimate reasons, would likely build their own custom solutions to deal with the operational problems, as they have always done.

For these reasons, although the microservice catalog is an interesting concept, I don't believe will likely see huge adoption outside of niche use cases.

Hope you enjoyed reading this!

Subscribe to my Data Management Newsletter for more original content on databases, data management, and data protection!