Systematic Literature Review: Exploring the Use of Informal Architecture Methodologies in Industry
This literature review was written as part of the course requirements for ESI 6891. This HAS NOT been peer reviewed and should not be considered a peer-reviewed publication.
I feel that this review needs further exploration and refinement beyond the scope of requirements as a course deliverable. However, I feel that until such time when it is further refined the content of the review may provide value in a self-published format. As such, I have chosen to publish it as-is on this site.
This was originally written in October 2022 and was first published on this site in August 2023.
Abstract
Background. The practice of software engineering and architecture is disconnected from the major body of research on the topic [1
]. This review explores the existing body of literature on software architecture practices with particular emphasis on literature that describes that state of architecture practices in industry.Objectives. The aim of this review is to explore the use cases and challenges of formal architecture methods (e.g. modeling, analysis, etc.) and the use of informal architecture and design methods (e.g. sketches, informal diagrams, natural language documentation) in practice.
Methods A search of existing academic literature was conducted using the Clarivate Web of Science and Elsevier Engineering Village (Compendex) databases. The following keywords were used: informal "software architecture"; formal "software architecture"; and software architecture methodologies. Peer-reviewed journal articles from 2017 through 2022 were considered.
Findings. The current body of research on the practice of software architecture is limited to studies with a small number of participants. Many challenges with existing methodologies are identified, but proposed solutions frequently do not adequately address industry needs. Modeling and analysis of formal architectures have many use cases, but the proposed solutions tend to focus on theoretical or proof-of-concept analyses rather than studies of real-world application. The body of research on informal methods in software architecture is almost non-existent. Only two studies were identified that explores the use of informal architecture artifacts.
Conclusions. The body of research on the use of informal techniques is substantially lacking. While informal communication and design methods are commonplace in practice, the research on the topic of software architecture does not sufficiently address this. This gap establishes open problem for future research.
Background
A clear definition of software architecture does not exists in industry. Richards and Ford still struggle to provide a clear definition [2
] and reference Martin Fowler's refusal to define the term nearly 20 years earlier [3]. As such, they define in it the broadest possible sense. This review will do the same and take a broad and inclusive view of what can be considered software architecture.The gap between the research community and industry is widely accepted [1
]. But the details of this gap are not well explored [4]. To address that gap, this review explores the existing body of literature focused on software architecture methodologies and practices, with a particular emphasis on exploring the extent of the academic literature discussing the more informal approaches described in Fundamentals of Software Architecture [2].In practice software engineers tend to use informal diagrams, sketches, whiteboard drawings, natural language documentation (particularly wikis), email, instant messaging, and similar informal approaches to communicating the design intent of a system. [5
] Frequently this is not documented comprehensively in any centralized format.More formalized methods do exist however such as the use of the Unified Modeling Language (UML) and Systems Modeling Language (SysML). However, these Model-Based Engineering approaches come with specific syntax and complex tools that lead to substantial learning curve leading to a lack of use by most software teams [6
].This review also addresses the fact that in the software industry, practices in industry significantly outpace advances in research. This review explores literature that attempts to address that and proposed potential future work to address that in the Future Works section of this review.
Potential Biases
In any study or review, potentials biases should be appropriately considered. In this review, the author identifies his own potential biases based on industry experience in this field. The author has more than a decade of experience building production software systems and approximately 9 years experience of in-depth engagement in software architecture practices.
The author's industry experience covers a breadth encompassing system modeling methods with UML and SysML (which the author considers to be a formal method), informal methods, software design, software engineering, model-based engineering, and model-based systems engineering. The author was interested in exploring this topic in large part based on an experiential observation of the gap between software engineering, design, and architecture in industry and its study in research and academic contexts.
The author has also observed many teams in industry where the gaps between architectural modeling and the informal processes used by the teams designing and building software vary widely.
Thus, it is of personal and professional interest to explore these topics, but it should be recognized that the basis of defining the objectives of this review are to explore the academic literature around experiential observations.
Methodology
The PRISMA protocol for systematic literature reviews was used to systematically evaluate of existing literature on the topic [7
].Objectives
The aim of this review is to explore the use cases and challenges of formal architecture methods (e.g. modeling, analysis, etc.) and the use of informal architecture and design methods (e.g. sketches, informal diagrams, natural language documentation) in practice.
Research Questions
- RQ1: What is the current body of research on existing architecture practices and what are the challenges associated with existing architecture methodologies in practice?
- RQ2: What are the use cases of formal software architecture models and what are the challenges?
- RQ3: What is the current body of research on the use of informal methods and artifacts in software engineering and software architecture?
Eligibility Criteria
Studies were limited to journal articles published from 2018 to 2022. The following inclusion criteria were used to assess literature:
- IC1. Studies with a focus on software architecture and design methodology.
- IC2. Studies from peer reviewed journal publications.
- IC3. English language studies with full-text access.
Exclusion Criteria
The following exclusion criteria (EC) are used to exclude irrelevant search results from this review:
- EC1 Broad Topic Irrelevance. Studies that did not focus sufficiently on software systems architecture, software architecture, systems architecture of cyber-physical systems, or similar architecture and design methodology of software systems are excluded.
- EC2a Narrow Topic Irrelevance - System or Domain Specific Focus. Studies that focused on specific architecture models or domain-specific architectures were excluded due to their focus on the specific architecture or domain rather than the study or application of architecture methodology.
- EC2b Narrow Topic Irrelevance - Student Project. Studies focused on engineering education and the use of software/systems architecture in a student academic setting due to the significant experiential differences between student teams and experienced software engineers and architects.
- EC2c Narrow Topic Irrelevance - Tooling Development. Studies focused on tool development in a research environment that are not describing tools not describing industry practical application or an industry accepted practice are excluded.
- EC3 Chronological Obsolescence. Records published more than 5 years ago are excluded in the interest of focusing on recent accepted knowledge.
- EC4 Non-Journal Publications. Conference papers, books, and content from other sources that are not peer reviewed journals are excluded.
Information Sources
A systematic search of the Elsevier Engineering Village and Clarivate Web of Science data bases was conducted. This databases were chosen because they index data from several other sources such as IEEE, ACM, Wiley, etc.
Additional Sources Considered
Industry-focused books were also evaluated using the O'Reilly Online Learning platform, published by O'Reilly Media. Search terms such as "architecture" and "design" were. 43 relevant books were identified of which only one is referenced in this review. The remainder of these books were ultimately excluded from this study, but are mentioned due to their potential relevance in future works.
Search Strategy
Preliminary Search
An initial unstructured search was conducted using permutations of the following search terms: architecture, software architecture, informal software architecture, formal software architecture, systems architecture, software architecture methodologies, software design methodologies, model-based systems engineering.
This yielded a broad set of results and was used in conjuction with a preliminary review of papers most relevant to the literature review objectives to identify terms for a more structured search.
It should be noted that results found during this preliminary search were also saved as initial records. Duplicates noted during the structured search and items excluded during chronological screening may refer to articles identified during the pre-systematic search phase.
Search Terms
| Search Topic | Ei Engineering Village (Compendex) Query |
|---|---|
| informal software architecture | (((((informal "software architecture") WN ALL)) AND ({ja} WN DT)) AND ((2022 OR 2021 OR 2020 OR 2019 OR 2018) WN YR)) |
| formal software architecture | (((((formal "software architecture") WN ALL)) AND ({ja} WN DT)) AND ((2022 OR 2021 OR 2020 OR 2019 OR 2018) WN YR)) |
| software architecture methodologies | ((((software architecture methodologies))) AND (({ja} WN DT) AND ({english} WN LA) AND ((2022 OR 2021 OR 2020 OR 2019 OR 2018) WN YR))) |
The term informal "software architecture".
The broadest results from Engineering Village when using the search term
informal software architecture included 24,266 results. This was further
refined by quoting the term "software architecture" as described in the table
above. This search yielded 1,224 results.
An issue encountered with this search was the "autostemming" feature of Engineering Village which reduces search terms to their root words, leading to several false matches where the root for "informal" shares the word-stem "inform" with words like "information" producing irrelevant results. Another search was conducted with "autostemming" turned off. This yielded 15 results.
The Web of Science results initially included 57 results. After limiting to only journal articles and articles since 2018, this was reduced to only 6 results. Of these six, three were duplicates and three were excluded on the basis of this review's eligibility criteria. Ignoring the journal article and last 5 years constraints, 42 of the original 57 remained after checking for duplicates.
The term formal "software architecture".
Using Engineering Village, the search described in the table above was used. This yielded 407 results. Disabling autostemming and limiting results to English language papers reduced this to 343 results. The equivalent Web of Science search produced 39 results, of which 22 were added after checking for duplicates.
The term "software architecture methodologies".
Using Engineering Village, the search described in the table above yielded 317 results, with 87 results included after duplicate removal. The equivalent Web of Science search yielded 796 results. This was further refined by topic area by limiting to the following areas: 4.47 Software Engineering, 4.224 Design & Manufacturing, 6.3 Management, 4.206 Models of Computation, 7.57 Modeling and Simulation. This reduced the result set to 202 results of which 185 results were included after checking for duplicates.
Search Results
After the above searches, 481 records were found to be screened. The figure below demonstrates this workflow visually.
It should be noted that the searches for formal architecture and architecture methodologies yielded substantially more results than the search for informal methods. While evaluating this is, in part, the purpose of this study, the body of knowledge gained from search results is heavily biased towards formal methods.
Selection Process
Title and Abstract Screening
The screening process leveraged a multistage screening pipeline to rule out articles by topic relevance, then chronological relevance, then publication source.
- Exclude articles based on EC1 (Broad Irrelevance).
- Exclude articles based on EC2 (Narrow Irrelevance)
- Exclude articles based on EC3 (Chronology)
- Exclude articles based on EC4 (Non-Journal Articles)
Articles were screened by evaluating, titles, abstracts, and publication information only. 481 records entered the selection process. After EC1, the record set was reduced to 311 records. After EC2, this was further reduced to 125 records. After EC3 and EC4, this was further reduced to 64 records.
Full-Text Screening
A full-text screening of 64 records was conducted. Studies that did not sufficiently address or support the specific topic objectives of this review were then eliminated. A preliminary full-text screening reduced the record set to 52 articles. After deeper read of the remaining 52 papers, 24 were either directly related or provided support for answering the research questions of this literature review.
Synthesis and Analysis
Papers were categorized following the full-text reading into the following groups: Informal Methods (3) - This includes any papers that discuss informal approaches to the design or architecture of software system; Formal Methods and Uses (12) - This includes papers that discuss formal architecture methodologies either in theory or in practice. These tend to be representative of use cases for formal methods; Architecture Challenges (3) - This category includes papers that address specific challenges and proposed solutions within the field of software architecture; Related Literature Reviews (6) - A number of relevant papers were literature reviews on related topics. Those were categorized separately to show the differing methodology of the content of those papers.
The figure above shows the breakdown and categorization of the identified and analyzed papers. It should be noted that only 12.5% of papers identified for this review specifically address the core research topics of informal methods. This is indicative of a lack of sufficient academic literature on the topic.
Data Collection
The author collected data about each publication by exporting BibTeX citations from database and/or publication sources and importing those into a custom developed proprietary citation management tool which was then used to track, search, and correct any missing article metadata. The author then used that citation management tool in conjunction with a proprietary document management tool that compiles this review into its final format with references.
Funding and Other Resources
Database and journal access is provided via the University of Central Florida library. The author's company, Triple Dot Engineering, has provided subscription access to the O'Reilly Online Learning platform for access to industry publications.
Literature Review
There is a significant gap between the state of research and the industry practice of software engineering [1
]. The architecture of a system plays a critical role in the entire software lifecycle. It is a key driver in not only the design of the software itself, but the delivery of that software [8]. In today's DevOps landscape, a software's delivery is critical to project success.Architecture often considers a system's quality attributes such as modifiablity, performance, interoperability, availability, security, testability, usability, etc. Kassab et al. identify modifiability as a top system quality need across most industries [9
].Previous Literature Reviews on Related Topics
Previous systematic literature review have explored similar topics, but have not addressed the specific topic covered here [10
][11][12][13][14]. None of these reviews address the use of informal methodologies in practice in the engineering of software systems. Each of these will be detailed briefly to provide context of the existing literature review landscape.Valle et al. [11
] explore the use of reference architectures in the field software engineering. They address several use cases of reference architectures and the review also identifies primary study contexts as derived from either academic or industry contexts. Of the 19 studies they explored only 4 (21%) came from an industry context. It is also noteworthy that the majority of their sources came from conferences and workshops rather than journals.Araujo et al. reviews the research landscape of formal verification of architectures (discussed below in the sections on formal architecture) [10
]. Similarly, Ciccozzi et al. reviews the research and practice of executable UML models, a comparatively formal approach relative to the informal techniques explored below [13]. Banijamali et al. explores specific architecture patterns in the intersection of the cloud computing and IoT (Internet of Things) spaces [14]. Finally, Tuma et al. explores the use of system and architectural models in the context of analysis of security threats to software systems [12].Several of these reviews also emphasize the use of the backward and forward snowballing technique for software engineering literature reviews [15
]. It is recommended that any follow on work to this review explore that technique.Architecture Methodologies
While UML and SysML are widely used in certain industries, the literature on the topic provides conflicting evidence of its use. While Ciccozzi et al., describe UML as a de facto standard [13
], others describe various architecture description languages (ADLs) as more prevalent [16]. Meanwhile industry literature describes UML as only used by "organizations that mandated its use" and instead describe more informal diagramming approach without standard notations [2].Enterprise architecture visualization methodologies are explored in Zhou et al. They explore the use of ArchiMate, UML, BMM, BPMN, SoaML, SysML, and others in the use of architecture visualization as well as assessing the using of various modeling frameworks such as TOGAF, DoDAF, MODAF, and others as exemplary models for enterprise architecture visualization [17
]. Zhou's review only covers formal or semi-formal methods and modeling languages. They also do not explicitly address the role of architecture visualization in software design.On formal and semi-formal architecture
Formal and semi-formal methods
In their paper, Basili et al. [1
] argue that academic research tends to propose solutions that "do not match real needs" or "are not applicable or scalable".Camara et al. demonstrate one such example of this in their paper on quantitatively synthesizing tradeoff spaces They describe formal methods for exploring and evaluating the tradeoff space of a software system that [18
]. We will show in this review that this formalisms are not representative of practices found in industry.Oueslati et al. are another example where new formal notations are proposed from a research context with no reference to it addressing an identied need. They propose a new set of formalisms on top of UML for developing reconfigurable control systems [19
].Similar formal verification methods may be useful when applied to the verification of safety-critical systems as shown by Hadad et al. in their paper describing a method of formal verification of safety critical systems using AADL [20
].In a review by Araujo et al., only 59% of studies used a visual method for modeling architecture. UML/SysML were used in only one-fifth of the studies. In this review, they find that there is no single notation for specifying a system's architecture [10
].In Ozkaya's comparison of formal architectural languages they also reference three other studies that surveyed a total of 291 participants. Ozkaya's conclusion is that is it difficult to discern practitioner needs of formal architecture languages. They also note another study which produced results noting the rare use of formal architecture languages [21
].Both Kassab et al. and Ali et al. idenfiy significant concerns with architectural drift in which the implemented system does not perfectly match the indented architecture. Kassab's findings show that only 9% of survey participants reported implementing the intended architecture with no changes [9
][5]. Kassab also identifies the top reasons participants provided for this drift as takes lots of time (35%), hard to reuse (26%), too complicated (19%), and too expensive (18%) [9].This topic is further addressed by Schroder et al. in their study on architecture enforcement concerns and activities. In this study the explore two main goals. The first considers what may be termed as architecture validation, they describe it as ensuring "agreement on design decisions with stakeholders". The second concern of architecture enforcement they address is conformance between the design and the implementation [22
]. In other words, architecture drift.Addressing formal architecture use cases without formal architectures
Dependency analysis is one method where formal models do not need to exist in order to achieve similar formal analyses. Wu et al. demonstrate a formal use of graph theory applied to existing open source software projects to analyze dependency networks between modules in source code [23
].Ali et al. [5
] also demonstrates this by identifying the use of tools like SonarQube for static analysis or Crucible for supporting code reviews. In these cases, these types of tools rely only on the existing source code to be of analytical value.On informal architecture and design
Informal architecture methods and their uses
Ozkaya describes three types of informal architecture notations [24
] as:- simple boxes/lines
- natural languages
- modeling languages
There is no industry consensus on the third item. Others have described modeling languages such as UML as "semi-formal" methods [10
]. For the purposes of this literature review, we consider UML in the semi-formal or formal category, particularly when compared to the boxes/lines and natural language approaches to architecture development.Architecture artifacts are often used to communicate the intent of a given approach to a software system [5
]. In their systematic review of Model-Based Systems Engineering (MBSE), Carroll and Malins identified a "a general lack of skilled MBSE engineers as a major hindrance to implementing an MBSE approach successfully" [6]. The common trend is is that most software engineering organizations lack the skills for formal architecture methods or don't value the benefits [6][5].With this it is common to see far more informal approaches to software architecture and design in practice [5
]. Atanasovski et al. demonstrate an example this approach, even when using formal approaches where informal diagrams are used first to communicate architecture and come to consensus before undergoing the added eff ort of more formal modeling approaches [25].Ali and Lai's review of Global Software Development, addresses major architecture challenges. Their findings show that communication & coordination is overwhelmingly one of the biggest architectural challenges faced by globally distributed software engineering teams [26
]. While their review proposes solutions, these solutions are not rooted in industry needs and do address real-world case studies.In Araujo et al., formal validation methods of architecture models are reviewed. One of the notable findings is that of the 39 studies reviewed only 13 of 39 (33.3%) provided real-world case studies rather than illustrating a possible solution [10
].Use cases of informal architecture methods
Ozkaya found that 94% of 11 study participants use informal software modeling methods and of the 6% that use formal methods, only 2% prefer exclusively formal methods. More notable yet is that for making complex architecture decisions 100% of participants use simple boxes and lines diagrams more than 75% of the time. Of these, 79% of participants attribute this to ease or learning and use [24
].A common use case of architecture is the documentation and communication of a system's design. While this is often done via informal methods such as natural language textual documentation, there are many challenges. Theunissen et al. identify several issues with informal documentation as described below [27
]:- Informal documentation can be hard to understand. It is worth noting however that they attribute this to the sparseness of the documentation, not the informal nature of it.
- The perception that documentation effort is considered wasteful.
- Organizational measures of software, not of documentation progress.
- Documentation drifts out of sync with software.
Theunissen's results tend to support the challenges identified by Ali and others. Though it may be noted that item (3) above is poorly correlated with the accepted agile strategies seeing increased adoption in industry [28
].State of research and practice
This review supports Ozkaya's finding that the while the existing literature describes many approaches to specific architectures (excluded from this review during screening), there are no studies that explore the formal and informal approaches to software architecture and practical use of each approach [24
].Sahlabadi et al. also concludes that methods described in the academic literature are not broadly applied in industry [29
].Ozkaya's finding that 100% of survey participants use simple box and line diagrams quantitatively demonstrates the critical gap between the focus of software architecture research areas and the approach to software architecture in practice described by Basili et al.
In Sievi-Korte et al., a study of software architecture design in a global software development context is conducted. They notably introduce the topic however with a discussion of the role that Conway's Law plays in architecture and design . Conway's Law famously describes a relationship in which organizations are likely to design systems with structures that reflect the structure of the organization [30
]. Insights from this observation may be further explored to understand the role of organizational and social dynamics in the design and development of software systems.In a similar regard, Razavian et al. identifies research gaps in the human-centric aspects of software engineering, particularly as it applies to the role of decision making in the context of software architecture.
A critical takeway from this study is the insights of Basili et al. in which they emphasize the need for different approaches to research in the field of software engineering and architecture in order for the research community to keep up with the ever changing and fast-paced industry landscape in this field [1
].This disconnect between the research community and the industry community will be further explored in the Future Works section of this review in which we will describe possible approaches and paths forward in software engineering and architecture research to both evaluate and attempt to bridge that gap.
Future Works
This review demonstrates a significant need for future work in this space. A study of industry literature, conference papers, talks, and workshops, and additional surveys of industry practitioners should be conducted.
The author plans to explore portions of this follow on work as part of ongoing research activities. Immediate next steps include exploring industry literature and industry-focused conference content as a method of more broadly exploring practices in the field rather than studies where a subset of the broader population is surveyed.
Furthermore industry partnerships and real-world case studies appear to be another key approach to bridging the current research gap in this space. Leveraging some of the mechanisms proposed in the studied literature to focus research problem on industry needs will be critical to improving the impact of research in the field of software engineering and software architecture.
Another important factor to consider in future works is the pace of the software field. New technologies and techniques are developed, tested, and established or disregarded in industry more rapidly than a large body of academic literature is able to be developed on the subject. Designing studies that can leverage the more rapidly evolving body of industry literature as a way of gathering industry perspectives may also be an approach to furthering the field.
This review establishes a clear need for future work in this field, but also a need for future work to explore new and alternative approaches in order to effectively impact the field of software engineering and software architecture.
Conclusion
Findings RQ1
What is the current body of research on existing architecture practices and what are the challenges associated with existing architecture methodologies in practice?
The current body of research on the practice of software architecture is limited to studies with a small number of participants. Many challenges with existing methodologies are identified, but proposed solutions frequently do not adequately address industry needs.
Findings RQ2
What are the use cases of formal software architecture models and what are the challenges?
Modeling and analysis of formal architectures have many use cases, but the proposed solutions tend to focus on theoretical or proof-of-concept analyses rather than studies of real-world application.
Findings RQ3
What is the current body of research on the use of informal methods and artifacts in software engineering and software architecture?
The body of research on informal methods in software architecture is almost non-existent. Only two studies were identified that explores the use of informal architecture artifacts.
Closing Comments
The body of research on informal methodologies of software architecture is substantially lacking. While some studies in the the existing literature indicate that informal communication and design methods are commonplace in practice, the research on the topic of software architecture does not broadly tackle this.
A systematic study to explore this disconnect may shed light on the state of industry relative to research literature. When the literature focuses so heavily on formal architecture methods, but some studies suggest that a large number of practitioners are using informal methods in practice, it raises the issue of research bias. For example, if studies are exploring the use of formal architecture languages or the practical challenges of formal methods, survey respondents are highly likely to be users of formal methods rather than informal methods.
It is worth considering, particularly in the field of software architecture if researchers are asking this right questions or if the questions being asked are leading to a biased body of literature.
This gap is open problem for future research as well as research/industry partnerships to shift the current focus of research in this space.