By Embracing the foundations of new enterprise computing
We are currently in a slow rebellion for ‘New World’ Infrastructure, turning over the traditions of enterprise orthodoxy.
In practice, the operating system stays the same, but everything else is changing. Sun Microsystem’s mantra (actually, the remarkable John Gage), the network is the computer, has become the predominant force and the 90s open system transformation is being replayed with genuine distribution of application components.
To describe what’s changed, we need a map to describe the layering of services in the New World.
The New World Stack
The impact of cloud transformation is shown as specific hardware in each venue. While the hyperbole described cloud infrastructure as a revolution, it is, in fact, a very successful but continual transformation to the financing, ownership and operation of equipment. The renewal of enterprise computing is expensive and conservative, which is why it has to be slow. As each system is renewed in the cloud, it becomes incrementally more important.
In the view above, there is no distinction between internal or external cloud, traditional data centres or processor architectures. It just adds to the diversity of hardware.
Sitting on the hardware are the foundations, which we cover below. This layer deals with all support services that applications would need in the new world, like security and naming.
Data is deliberately separated from services in a layer of its own. This is an organisation that we will see increasingly, as data ceases to belong to a single application, effectively becoming its own application. It is also unlikely to be a single state of data, but one where information is transformed over the business processes of the enterprise, from an initial ‘golden source’ to various forms of enrichment. In this case, the data layer may be a virtual tracking of all the data states rather than a single repository. In short, the layer is either the single source of truth or holds references to its transformation journey, like virtual data.
In the long term, the Data layer is likely to be implemented as mostly a set of repositories. Initially, however, applications will hold much of an organisation’s data in aggregate, with a gradual migration to more of a library function. The other important part of future data handling is how ownership is modelled. Instead of a dataset being owned by an application and user rights created by that tool, there will be a switch to data being owned by the person to which it pertains. The ownership is expected to be carried to all subscribing applications, together with usage auditing and increases the role of authorisation. This allows an organisation to locate and centrally manage a person’s information.
Service and client layers are as expected, but joined with ‘things’ to represent the huge numbers of ultra mobile or computer aware objects that participate in the future connected society: the Internet of Things.
The Politics of Data
Data has become political.
The general population (not just the savvy) are starting to understand what can happen when enough of their personal data is held in one place.
Data leakage, weaponising, social targeting, mass influence for ill, shaming or blackmail have certainly taken old laws on data protection and given them an unexpected twist. Open and advanced societies are particularly at risk. So we can bet on more law, processes and accountability for those that hold data.
The politicians that represent us vary in their understanding of the high-level concepts of data management. However, laws about data have arrived, and many more are coming. Some will be good, others bad.
The US has standards around Personally Identifiable Information (PII) and the EU has legally binding data protection for the individual (GDPR). Other countries have standards relating to their nationals, such as Switzerland. The result is that some data can only be kept in particular legal jurisdictions. Metadata (the data about data) is predicted to accumulate similar rights, once a similar public education has taken place. The European E-Privacy proposals are one such indication.
Unless there is an overall agreement between nation-states (and their alliance blocks), it will lead to difficulty in running a data business. However, everyone is now a data business. Working with data is fundamental to most forms of business, even if its only to give customers the interaction they have learned to expect. Companies large and small have to deal with whatever the future hands them, as dealing with personal becomes fundamental to the process of just doing business.
Therefore, data should be handled defensively. Each jurisdiction will have its own contract with the data with rules for its transfer. Bottom line: an organisation has to assume that it cannot own data on a person; it is an asset of the individuals concerned not the enterprise.
To stretch a personal analogy to the nation-state, a piece of data needs passport and visa to represent identity, residency and movement rights. The movement must be tracked by a compliance system, which also needs to know the systems that are using the data to ensure that the purpose is sound and there is no indirect leakage or that the context of the data is inappropriate
Foundations
As shown in the data discussion above, effective management of systems and data require many foundation systems running in context to take care of concepts like identity and authorisation. It is too big and too hard to be handled within each application. Moreover, the trend for bottom-up self-written software becomes more productive once core responsibilities are handled by specialist systems.
New world infrastructure needs foundations to help manage contemporary challenges. It includes new stacks and groupings that represents the current software construction techniques and support for new data awareness.
We can represent the foundations on a four-square grid to help explain the functionality of the infrastructure tools.
On the x-axis, there is spectrum or stack depth which we represent as either basic technology or supporting application. On the other axis is insight that shows the source of data grouped into two: a defining tool or data as one group, the other being discovery where tools use or report on automatically generated data.
Each square can be labelled as follows:-
- Identity and Construction
- Discovery and Observation
- Communication
- Modelling
These squares contain tool groups that enable us to use more familiar software, such as naming systems.
Exclusions
Before we start the listing, it’s worth mentioning that a very key set of tools are not included. That is development in all shapes and sizes, for construction of applications and systems. Whilst it seems a glaring omission, development is actually optional for many people; they just buy-in or rent what they need (SaaS), or consider lightweight development part of client software. Development’s relationship is considered in a future article.
Identity and Construction
The first group in the Technology Definition quadrant provides the basic enablers to the new foundations.
Every component in the new world needs an Identity, and every activity needs Authorisation. The Data layer gains the support of Entitlements to cover ownership and rights.
Everyone needs to know that trust exists in their working group and with their peers. Externalising this function lets software be smaller and more nimble with coordination integrated into business processes.
Completing the defining technology are Orchestration tools that are abstracting the new physical architectures.
There will be greater depth for each tool group in subsequent articles.
Discovery and Observation
In the Technology Discovery quadrant, Observability is now an important field of knowledge in modern operations. It has encompassed all previous forms of monitoring and eclipsed modelling of systems for capacity planning.
As well as being a more comprehensive and general purpose way to monitor, the core problem domain has become greater. As modern systems design becomes more bottom-up and less centrally planned, it becomes harder to be deterministic about the expected shape, costs and performance of a future estate.
The solution is to collect data on the running of everything, make it searchable and then run queries on the metadata. We observe the situations in our system and then decide to take action depending on what we find.
If we have good metadata, observability can address more organisational problems than just development and operation of applications — for example, resource utilisation or corporate responsiveness. We can write scripts, views or applications that use metadata to program our reactions. We can extend the scope of data collection to encompass all types of technically generated information. It merges in with the overall streaming architecture pattern, and we can utilise visualisation techniques in use with Business Intelligence. One might use the phrase ‘technical intelligence’ or ‘operational intelligence’, as it’s a similar concept to BI but it relies on continuous deliveries of data from technical sources: infrastructure and applications.
Observation tooling is broken into Event and Data Logging with Tracing. Discovery is a more passive discovery method for reverse engineering an environment.
Communication
In the Defined Application quadrant sit the communication methods. As distributed systems evolve, the count of independent services (or microservices) has gone up. The protocol relationships between them have also become more complex. Networks are increasingly used for communication within apps rather than external between apps.
This section addresses the common service elements to networking and how they can help us. We assume that everything is attached to a reasonably performant network, ideally locally attached.
Tooling here includes Publish-Subscribe (pub-sub), Enterprise Service Buses (ESB) and Data Streaming services.
Modelling
The Application Discovery quadrant packs several essential areas, generally used in modelling resources and infrastructure.
Data Lineage tools help map information flow over many transformative steps. Cost control and infrastructure strategy setting uses Modelling tools to decide a top-down architecture, into which the computing environment sits.
Then one of the most significant disciplines in modern computing has a place for its tooling — Security.
By freeing it of coordination and control management tasks, development processes have been able to proceed more smoothly. However, this poses a problem with compliance and security, part of the broader responsibilities of computing groups. One approach is the proactive use of tooling during development to prevent threats and the reactive discovery of other situations in running systems.
The Completed Foundation
The foundation view represents the most significant functions that need to exist to support a modern organisation, with current markets and politics. The current view is not complete and should be considered a work in progress. It will apply to any multinational company. Intriguingly, it also applies to users of multinational infrastructure providers: you may outsource to others, but you still own the responsibility or compliance.
It is useful to map against your own provision to see what is missing. In that regard, it can be considered to be a practical, lightweight reference framework. It certainly highlights the new toolsets that are considered common to all and essential.
The New World is a useful label to apply to changes happening to our industry right now.
It reflects the way we buy computing, the way we codify our businesses and our treatment of individuals. It shows how we approach automation and social responsibility.
Each company has its own mission and its own set of applications. However, they will all need the same foundations.
About the Author
Architect, PM and Strategist. Follow me on Twitter @nigelstuckey
System Garden
Agile Infrastructure for Enterprise DevOps
Design from diagrams, document and deploy to your cloud.
systemgarden.com, Twitter @systemgarden
Future articles examine the categories in greater detail.
Originally published on the System Garden Blog. Thanks to Alex Stuckey