In the age of cloud computing and particularly within cloud microservices, it has become common practice to carefully consider the team structure so that a desired software architecture emerges (Baiyere, Ross, & Sebastian, 20171; Childers, 20192; Cockcroft, 20143; Vogels, 20064). As mentioned in previous posts discussing Mirroring and Conway’s Law, this is often referred to informally as an ‘inverse Conway manoeuvre’ within professional software engineering circles.

Analysis of the data has revealed that cloud native organizations such as Netflix, Amazon and Spotify (Bäcklander, 20195; Balalaie, Heydarnoori, & Jamshidi, 20166; Cockcroft, 20143; Diaz, 20177; Vogels, 20064) deliberately structure their teams in ways that reinforce the software architectures they want to build, in order to leverage the homomorphic force expressed within the Mirroring Hypothesis. They have a structure for their teams that results in a more optimum software architectural outcome.

This deliberate team configuration approach is particularly prevalent within organizations that use Service Oriented Architecture (SOA) or the related micro-service architectural patterns within their organizations (Balalaie et al., 20166; Villamizar et al., 20158). Amazon AWS, Netflix, Google Cloud Platform, Microsoft Azure cloud are all based on SOA or micro-service architectures (referred collectively as microservices) (Cockcroft, 20143; Shoup, 20159; Vogels, 20064). The data within this study reveals that all major public cloud players have adopted microservices in their cloud offerings (Bäcklander, 20195; Villamizar et al., 20158). Private cloud providers may also use microservices with the possible exception of providers who only provide IaaS virtual machine services and don’t venture into the higher abstractions of PaaS and SaaS.

Micro-service software organizations have many teams that work independently of one another. They communicate through interfaces and each team has full ownership of their own set of microservices. The literature on microservices adoption focuses on silicon-valley based organizations. Interview results (VP, Global financial services) indicate that global scale organizations outside of the silicon-valley tradition, but with a history of software innovation, can struggle with the adoption of microservices, even when the full weight of the organization is behind the move to cloud adoption. This raises interesting questions for our decision processes within Snapfix. In light of this observation, I have placed a strong focus on the cultural and ideological aspects of micro-service adoption and consider it a precursor to successful implemention and adoption of a micro-service architecture within Snapfix. You can read more on this in my series of blog posts on Open Source Software and Open Source Ideology.

Facebook is a notable exception to this trend towards microservices amongst the cloud industry leaders (Spinellis & Gousios, 200910). Facebook has retained its monolithic architecture throughout its evolution, with its organizational structure quite different from the underlying software architecture, thus not supportive of mirroring (Feitelson, Frachtenberg, & Beck, 201311). However, Facebook has had to develop elaborate means of managing this configuration of many teams working on a single monolithic code-base (Adams, 201912).

In my opinion, Facebook are second only to Google in its leadership and development of world class Software Engineering practices and it’s worth keeping in mind they still choose to embrace the monolithic approach to cloud computing. Enterprise Ireland and some VC firms encourage the move towards “micro-services” and “Kubernetes” in the investor pitches of emerging SaaS startups, but the move from monolithic to micro-service should be rooted in the fundamental advantages and disadvantages of each architecture rather than the pressures of VCs to adopt the latest and greatest in cloud architectures. A well thought out monolithic system will always be superior to a poorly considered micro-service architecture.

Different organizations manage this mirroring process in different ways. At Amazon, the teams are structured by the Chief Architecture Officer or member of the team who is responsible for software architecture (Vogels, 20064). At other companies such as Spotify, there is more of a consensus driven approach with the tendency to mirror team structures baked into the engineering culture of the organization and driven by the CTO (Goldsmith, 201613). Spotify has a Chief Software Architect and his role is directed at making sure good engineering principles are being used and examining the reasons when engineers wish to diverge from those principles (Goldsmith, 201613). In this context, good engineering principles means adhering to the Service Oriented Architecture (SOA) approach of mirroring of teams to specific microservices.

Within the data, I noted two common ways in which organizations adhere to mirroring. The first way is to recognize that the existing team structure is working well and if possible to design the new software architecture so that it fits in with the existing team structure. The other way is to do an ‘inverse Conway manoeuvre’ and decide on the software architecture first and design the teams so that it best mirrors the desired architecture. An interviewee (VP, Global financial services) indicated this was the approach taken by their financial services organization. They are strategically attempting to adopt a Service Oriented Architecture (SOA) but with mixed results and slow progress. The progress at the interviewees firm is in contrast to the experiences of the major cloud providers who have been researched within the collective literature.

Netflix (Cockcroft, 20143; Toffetti, Brunner, Blöchlinger, Dudouet, & Edmonds, 201514), Spotify and Amazon (Jamshidi et al., 201815; Vogels, 20064) run their teams as independent units. Each team works on their own set of goals and works autonomously. The data indicates that it is common practice within the software industry to start each project as a monolith with one team working on the same code-base (Baiyere et al., 20171; Balalaie et al., 20166). As the company or project increases in complexity and additional teams start working on it, a decision is made to re-architect the system and transition to a microservices architecture that allows each team to take full ownership of each micro-service, with its own independent code-base (Mazlami, Cito, & Leitner, 201716). The teams mirror the code-base. A single team works on a single code-base that encompasses the entire product or service the company provides to its clients. From this perspective, it could be claimed that the company starts as a single service team mirrored to a monolithic single code-base product. As the company grows it typically forms a second team, but retains its monolithic architecture, and therefore ‘breaks the mirror’. Assuming the company wants to follow current best practice for a cloud native company, they will want to eventually transition to a microservices mirrored organization (Balalaie et al., 20166; Dehghani, 201817). However, there is a period during this transition when the project is in an unmirrored state. If proactive steps are not taken it can continue in this mode and become more unmirrored with corresponding drops in efficiency (like the example of Facebook discussed earlier). The point at which this transition occurs depends on their resources. Ideally, each team will have its distinct designers, developers, quality testers etc. In cases where rapidly growing organizations have adequate capital, they can quickly structure their teams in this manner. However, in cases where the budget of the organization requires sharing of expertise across teams, then it will be more difficult to transition from the monolithic approach to the microservices approach. I have experienced this dynamic during my software engineering career. Projects are started and grow in a monolithic manner with a single code-base and a single team contributing to the code.

This transition from monolith to micro-service can manifest itself in other ways. A good example is provided in the professional literature in the context of emerging data science services (Dehghani, 2019). Many organizations don’t currently have the means to assign a data scientist to each service team. In this case, the data scientist works across teams when their skills are required (Al-Ruithe et al., 201818; Dehghani, 201919). This is another example of ‘breaking the mirror’ to facilitate the adoption of an emerging technology or practice.

There were some Mirroring Hypothesis ‘anti-patterns’ noted in the data. There were cases where mirroring harms specific parts of the organization. For example, a request could come from a customer for a specific new feature. In this case, it may be as simple as updating a single service. But it could involve changes being required of several teams managing different services. In this case, agreement must be reached with each of the teams that help to provide the client function under development. This can slow product development when compared to the monolithic approach.

There are other issues raised in the professional literature related to mirroring within cloud native teams. Josh McKenty, formerly at NASA and now VP at Redis Labs, questions the long term effectiveness of a deliberate mirroring of small team structures to the architectures they strive to produce (Woods, 201720). When empowering smaller teams in this manner, it can result in an intense loyalty within that team that outweighs the loyalty to the overall project or organization and can manifest through organizational, domain or geographic boundaries. This is similar to a phenomenon encountered in the Information Systems open source literature where commercial open source developers foster more of a loyalty towards the open source project than to their employers (Daniel, Maruping, Cataldo, & Herbsleb, 201821). McKenty encourages companies to selectively ‘break the mirror’ to facilitate cross team communication. I briefly discuss this loyalty issue in a post I wrote on Commercial Open Source Ideology.

Communication must regularly occur across teams. When coordination is required across different functional blocks of code the teams can request various changes and wait for teams to respond to the request, but this appears to be overly bureaucratic and ineffective within mirrored organizations. The method which surfaced from the industry data borrows from the open source community and is referred to as ‘inner sourcing’ (Kozłowski, 201922). It team A is dependent on a function within team Bs micro-service, then team A can delve into team Bs code-base and make a change, but with oversight by the team owning the code. In this way, team A doesn’t have to request changes from team B which could slow down new development. They book the code out of the repository, make the change and then book it back in, with team B approving the change upon review. Communication can be in the form of a pull request as described above. It can also be communicated by using tasks within Snapfix Groups and some organizations have what they call ‘guilds’ (Kniberg & Ivarsson, 201223; Smite, Moe, Levinta, & Floryan, 201924). These are groups of people that regularly work across teams to discuss best practices and challenges across the organization. This is another example where problems of limited communication between teams is solved by strategically ‘breaking the mirror’. Other strategies are employed to ensure cross pollination of ideas and skills across siloed teams. For example, companies will encourage team members to join other teams. Within Snapfix we have doubled down on this strategy in anticipation of the move towards micro-services. Even if we ultimately delay or even reverse our plans to embrace this architecture, we will still have levereged this form of open communication in developing our services.

Some papers discuss highly distributed systems but don’t explicitly state they are cloud based. However, based on the content of the papers I judge it highly likely they are describing cloud hosted services. MacCormack, Baldwin, & Rusnak, (2012)25 examine underlying code-bases from open source and privately developed cloud services. They note that artefacts developed through open innovation show increased modularity compared to artefacts developed in closed source, thus supporting mirroring to some extent, but with one notable exception where a single developer contributed all the code to a repository in an unmirrored fashion. Le & Panchal, (2012)26 research of Drupal concludes that technical interfaces between files significantly influence the creation of new communication channels between team members working on the same files. Drupal can potentially be used on physical web services but is designed for cloud hosting on IaaS so I consider it relevant to cloud computing. There were indicators across several cases that point to the reduced time required to carry out tasks that are more mirrored (Cataldo et al., 200827). A study of Wikipedia looks into the performance aspects of article contributions. Articles involving interdependent tasks are of higher quality when authored by smaller teams (Kittur & Kraut, 200828). The authors examined twenty Wikipedia articles and the evolution of their editing over time. When smaller groups of contributors were involved in writing an article, the resultant quality was greater, which supported mirroring.

Next: The Adoption of Cloud Services

  1. Baiyere, A., Ross, J. W., & Sebastian, I. M. (2017). Designing for Digital: Lessons from Spotify. Retrieved from https://research.cbs.dk/en/publications/designing-for-digital-lessons-from-spotify ↩︎

  2. Childers, C. (2019, August 17). Context: Reversing Conway’s Law for Superior Service Management. Retrieved 20 August 2019, from The New Stack website: https://thenewstack.io/context-reversing-conways-law-for-superior-service-management/ ↩︎

  3. Cockcroft, A. (2014). Migrating to Microservices at Netflix. Retrieved 20 August 2019, from InfoQ website: https://www.infoq.com/presentations/migration-cloud-native/ ↩︎

  4. Vogels, W. (2006). A Conversation with Werner Vogels—ACM Queue. Retrieved 20 August 2019, from https://queue.acm.org/detail.cfm?id=1142065 ↩︎

  5. Bäcklander, G. (2019). Doing complexity leadership theory: How agile coaches at Spotify practise enabling leadership. Creativity and Innovation Management, 28(1), 42–60. https://doi.org/10.1111/caim.12303 ↩︎

  6. Balalaie, A., Heydarnoori, A., & Jamshidi, P. (2016). Microservices Architecture Enables DevOps: Migration to a Cloud-Native Architecture. IEEE Software, 33(3), 42–52. https://doi.org/10.1109/MS.2016.64 ↩︎

  7. Diaz, F. (2017). Spotify: Music Access At Scale. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1349–1349. https://doi.org/10.1145/3077136.3096471 ↩︎

  8. Villamizar, M., Garcés, O., Castro, H., Verano, M., Salamanca, L., Casallas, R., & Gil, S. (2015). Evaluating the monolithic and the microservice architecture pattern to deploy web applications in the cloud. 2015 10th Computing Colombian Conference (10CCC), 583–590. https://doi.org/10.1109/ColumbianCC.2015.7333476 ↩︎

  9. Shoup, R. (2015). Microservices, the Reality of Conway’s Law, and Evolutionary Architecture. Retrieved 20 August 2019, from InfoQ website: https://www.infoq.com/interviews/randy-shoup-microservices/ ↩︎

  10. Spinellis, D., & Gousios, G. (2009). Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design. O’Reilly Media, Inc. ↩︎

  11. Feitelson, D. G., Frachtenberg, E., & Beck, K. L. (2013). Development and Deployment at Facebook. IEEE Internet Computing, 17(4), 8–17. https://doi.org/10.1109/MIC.2013.25 ↩︎

  12. Adams, K. (2019, July 15). Facebook PHP with Keith Adams. Retrieved 22 August 2019, from Software Engineering Daily website: https://softwareengineeringdaily.com/2019/07/15/facebook-php-with-keith-adams/ ↩︎

  13. Goldsmith, K. (2016, February 15). Spotify Technology Career Steps. Retrieved 18 August 2019, from Labs website: https://labs.spotify.com/2016/02/15/spotify-technology-career-steps/ ↩︎

  14. Toffetti, G., Brunner, S., Blöchlinger, M., Dudouet, F., & Edmonds, A. (2015). An Architecture for Self-managing Microservices. Proceedings of the 1st International Workshop on Automated Incident Management in Cloud, 19–24. https://doi.org/10.1145/2747470.2747474 ↩︎

  15. Jamshidi, P., Pahl, C., Mendonça, N. C., Lewis, J., & Tilkov, S. (2018). Microservices: The Journey So Far and Challenges Ahead. IEEE Software, 35(3), 24–35. https://doi.org/10.1109/MS.2018.2141039 ↩︎

  16. Mazlami, G., Cito, J., & Leitner, P. (2017). Extraction of Microservices from Monolithic Software Architectures. 2017 IEEE International Conference on Web Services (ICWS), 524–531. https://doi.org/10.1109/ICWS.2017.61 ↩︎

  17. Dehghani, Z. (2018). How to break a Monolith into Microservices. Retrieved 20 August 2019, from Martinfowler.com website: https://martinfowler.com/articles/break-monolith-into-microservices.html ↩︎

  18. Al-Ruithe, M., Benkhelifa, E., & Hameed, K. (2018). A systematic literature review of data governance and cloud data governance. Personal and Ubiquitous Computing. https://doi.org/10.1007/s00779-017-1104-3 ↩︎

  19. Dehghani, Z. (2019). How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. Retrieved 15 August 2019, from Martinfowler.com website: https://martinfowler.com/articles/data-monolith-to-mesh.html ↩︎

  20. Woods, D. (2017). How Platforms Are Neutralizing Conway’s Law. Retrieved 15 August 2019, from Forbes website: https://www.forbes.com/sites/danwoods/2017/08/15/how-platforms-are-neutralizing-conways-law/ ↩︎

  21. Daniel, S. L., Maruping, L. M., Cataldo, M., & Herbsleb, J. (2018). The Impact of Ideology Misfit on Open Source Software Communities and Companies. MIS Quarterly, 42(4), 1069–1096. https://doi.org/10.25300/MISQ/2018/14242 ↩︎

  22. Kozłowski, A. (2019, August 19). How Microservices Architecture Impacted the Culture of Software Development. Retrieved 20 August 2019, from Medium website: https://medium.com/better-programming/how-microservices-architecture-impacted-the-culture-of-software-development-6bba363ecdf1 ↩︎

  23. Kniberg, H., & Ivarsson, A. (2012). Scaling Agile at Spotify with Tribes, Squads, Chapters & Guilds. 14. ↩︎

  24. Smite, D., Moe, N. B., Levinta, G., & Floryan, M. (2019). Spotify Guilds: How to Succeed With Knowledge Sharing in Large-Scale Agile Organizations. IEEE Software, 36(2), 51–57. https://doi.org/10.1109/MS.2018.2886178 ↩︎

  25. MacCormack, A., Baldwin, C., & Rusnak, J. (2012). Exploring the duality between product and organizational architectures: A test of the “mirroring” hypothesis. Research Policy, 41(8), 1309–1324. https://doi.org/10.1016/j.respol.2012.04.011 ↩︎

  26. Le, Q., & Panchal, J. H. (2012). Analysis of the interdependent co-evolution of product structures and community structures using dependency modelling techniques. Journal of Engineering Design, 23(10–11), 807–828. https://doi.org/10.1080/09544828.2012.695014 ↩︎

  27. Cataldo, M., Herbsleb, J. D., & Carley, K. M. (2008). Socio-technical Congruence: A Framework for Assessing the Impact of Technical and Work Dependencies on Software Development Productivity. Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 2–11. https://doi.org/10.1145/1414004.1414008 ↩︎

  28. Kittur, A., & Kraut, R. E. (2008). Harnessing the Wisdom of Crowds in Wikipedia: Quality Through Coordination. Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, 37–46. https://doi.org/10.1145/1460563.1460572 ↩︎