From Data Absence to
Signal–Driven Systems
Industrial terminal operations across Europe and Asia with no real-time visibility — data either absent, manually monitored on scheduled cycles, or captured inconsistently with no connected architecture.
Manual monitoring could not detect what was happening in real time. By the time a problem was identified, the operational window to respond proactively had often already closed.
Building a connected data architecture — ingestion from multiple sources, governed pipelines, sensor-based capture where needed, and a unified flow from operational reality to decision — from near-zero, designed for production scale from the start.
A production-grade signal system that moved operations from reactive manual monitoring to continuous, real-time operational awareness.
A major global energy company — one of the world’s largest integrated oil and gas operators. The context was the Trading and Supply division: responsible for the commercial management and physical movement of energy commodities across global markets.
Terminal operations across Europe and Asia, including South Africa — running without digital visibility. Operational states were monitored manually, on scheduled cycles. Data was either not being captured at all, captured inconsistently across systems, or captured in ways that were never designed to connect — different formats, different field definitions, different systems that did not talk to each other.
The system had no mechanism to sense what was happening in real time, no threshold alerts, and no unified architecture to move operational data from where it was generated to where it was needed for decisions.
The gap between what was operationally happening and what the system could detect, consolidate, and act on was the problem. And that gap was costing the organisation in operational risk, decision quality, and response time.
Where This Started —
Two Problems, Not One
Terminal operations in this environment were running on a combination of manual processes, disconnected systems, and inconsistently captured data. The operational landscape was fragmented in specific ways that mattered.
Some data was being captured — but in formats specific to each system, each site, or each team that had built the local process. Different systems held different pieces of operational information. Those systems were not connected to each other, which meant data that existed in one place was invisible everywhere else. There was no unified flow from the moment operational data was generated to the moment it was needed for a decision.
Some data was not being captured at all. Operational states — tank levels, throughput, usage patterns, system conditions — were checked manually on scheduled cycles. Between those cycles, the system was blind. If a condition changed outside of a scheduled check window, the organisation would not know until the next visit.
The consequence for the Trading and Supply operation was direct. Decisions about inventory management, supply planning, and resource allocation were being made on information that was delayed, inconsistently structured, or manually assembled from sources that had never been designed to work together. The quality of the signal feeding those decisions was constrained by the quality — and the absence — of the operational data architecture beneath them.
I worked within Trading and Supply throughout — first on the IT side building the technical foundation, then on the business side as Technical Product Manager. Both phases informed this case.
The Real Constraints —
What Made This Hard to Build
Understanding why this system was hard to build requires understanding the specific constraints that shaped every architectural decision. These were not generic transformation challenges. They were constraints specific to this operational environment, this scale, and this gap between current state and target state.
The system had to be built from near-zero in the data engineering layer. There were no existing pipelines to extend, no established data lake architecture to build on, no scheduled workflows already in place. Every layer of the technical stack had to be designed and built from the ground up — which meant the sequencing of the build mattered enormously. Getting the sequence wrong would mean rework at production cost.
The target geography spanned Europe and Asia including South Africa — terminal operations ranging from highly digitised facilities in mature markets to sites where the gap between current state and target architecture was substantial. A solution designed for one end of that maturity spectrum would not hold across the other. The architecture had to be designed for the full range, not the average.
The raw terminal data did not arrive in clean, consistent formats. Field definitions varied across sites. Units of measurement were applied inconsistently. Time stamps, threshold definitions, and operational state categories meant different things in different parts of the estate. Before any pipeline could be reliable, the data model had to be standardised at the source — which required working across teams and sites to establish shared definitions that people would actually adopt.
The technical output of the system — alerts, dashboards, pipeline feeds — would only change operational behaviour if the people responsible for acting on those outputs trusted what they were seeing. Operational teams that had been running on manual processes and local knowledge for years do not automatically defer to a new system’s readings. If a sensor alert fired and the operator’s instinct said otherwise, the alert would be ignored — regardless of how technically accurate it was.
The system therefore had to be designed to earn trust incrementally — starting with the operational states and sites where the data quality was most reliable, making the system’s reasoning visible rather than opaque, and building adoption through demonstrated accuracy before expanding the scope. Trust was not a change management problem to be solved after deployment. It was a design requirement that shaped how the system was built and rolled out.
This was a global energy company. Security standards, data governance, enterprise architecture compliance, and access control requirements applied from the first line of code — not from the point at which the POC graduated to production. A system built without embedded governance would require a complete rework before it could be deployed in the production environment. The cost of deferring governance is not a saved cost — it is a compounded one.
The Architecture Decisions —
What Was Built and Why
Each architectural decision in this system was made in response to a specific constraint. Understanding the decision without understanding the constraint it was responding to misses the point of why it was made that way.
The target geography was Europe and Asia. The POC started with five terminals. This was not a resource constraint or a lack of ambition. It was a deliberate design choice about where failure is acceptable.
A pipeline that fails in a five-terminal POC costs almost nothing to diagnose and correct. The same failure in a production system serving the full geographic scope costs significantly more — in rework, in operational disruption, in the erosion of stakeholder confidence that is difficult to rebuild once lost.
The instinct in most transformation programmes is to start with the visible output — the dashboard, the report, the executive view. These are the things that demonstrate progress to stakeholders. They are also the things that fail when the foundation beneath them is unreliable.
The first architectural priority was to redesign how data entered the system — at the point of origin. This meant multiple layers simultaneously: structured ingestion from operational systems that already held data but had never been connected; sensor-based capture for operational states that were not being captured at all (tank levels, throughput, usage conditions); and the transformation and staging layers that made raw inputs from both sources usable downstream.
Sensor-based capture was the most significant new component — it enabled real-time tracking and threshold-based alerting for the first time, moving specific operational states from scheduled manual checks to continuous monitoring. But it was one part of a broader data capture redesign, not the whole of it.
Before building pipelines that would carry data across the architecture, the data model had to be standardised. The same field could not mean one thing at one terminal and something slightly different at another. A pipeline that carries inconsistent data at scale simply industrialises the inconsistency — it does not resolve it.
This required working across teams and sites to establish shared definitions for every key field — what a tank level reading represented, what operational states were valid, how throughput was calculated and over what time window. This was governance work, not engineering work. It required conversations, alignment, and the specific discipline of not moving to the build phase until the definitions were genuinely stable.
In a global energy company, governance, security, and architecture compliance are not optional layers added before deployment. They are requirements that apply from the first working system, regardless of whether it is called a POC or a production system.
Building the POC with full production governance requirements embedded from the start meant that when the scope expanded from five terminals toward the full geographic estate, the architecture did not require structural rework. The foundation was designed for what it was intended to become — not for the size it started at.
What Changed
The shifts were visible across both the technical architecture and the operational decisions it was built to support.
Data that had existed in isolated systems — visible only to the teams that held it — became accessible across the operation through a connected architecture. Operational states that had only been observable through scheduled manual checks became continuously tracked through sensor-based monitoring. The window between a condition developing and the organisation detecting it closed from hours to near-zero for the states that mattered most. A class of operational problem that could previously only be managed after it developed could now be intercepted before it reached operational impact.
Cross-site and cross-region comparisons that previously required a manual consolidation cycle — with corresponding uncertainty about whether the numbers being compared actually measured the same thing — became queries against a trusted architecture. Operational planning conversations that had previously required a data preparation cycle running over days could now happen from a live system. The reliability of the data changed the speed and confidence of the conversations it supported.
Supply planning, inventory management, and resource allocation decisions across the Trading and Supply operation moved from being constrained by what had already happened to being grounded in a continuously updated view of operational reality. The system’s ability to surface current conditions — rather than past conditions — changed the nature of the planning conversation. Decisions could be made from a live operational picture rather than a lagged reconstruction of one.
What This Revealed
The foundational layer — how data is generated, captured, standardised, and governed — is what determines whether everything built on top of it is trustworthy. It is also the layer that is most consistently underinvested in, because it does not produce visible outputs while it is being built. The governance work, the field standardisation, the pipeline architecture — none of this looks like progress until it is done. And then, suddenly, everything that was impossible becomes possible.
This experience also revealed something about the relationship between the technical foundation and the operational decisions it was designed to support. A sensing system that feeds an unchanged operational decision process will produce better-informed decisions made the same way. The system creates the possibility of better decisions — but only if the decision process is designed to use what the system produces. Both have to be designed together.
What This Taught Me
The five-terminal POC was not a small ambition. It was a deliberate architecture for learning. Starting at a scope where every failure is recoverable — and designing the POC to surface failure modes rather than to demonstrate polished outputs — is what separates foundations that hold from foundations that require complete rebuilds at production stage. The cost of finding a pipeline failure in a five-terminal POC is an afternoon. The cost of finding the same failure in a production system serving the full geographic estate is measured differently.
Building production-grade governance into the POC from the first working system — not before deployment, but from the beginning — is not over-engineering. It is the decision that determines whether the system can scale without structural rework. A POC that defers governance is not a faster path to production. It is a deferred rework cost that appears at exactly the moment when the organisation is most impatient for deployment. The time saved in the early phase is paid back with interest at the end.
Building the sensing layer, the pipelines, and the governed architecture creates the possibility of better operational decisions. It does not automatically produce them. The operational processes that use the system’s outputs have to be designed to take advantage of what the system now makes possible — the threshold alerts, the real-time state visibility, the cross-site comparability. A system that is technically sound but deployed into an unchanged operational process produces better-informed decisions made the same way. Both the system and the process it serves have to be designed together, from the start, for the investment to produce its full value.
Where this case sits in
the Signal → Perception loop.
This case sits primarily at the Signal stage — with a direct connection to the Execution stage that the signal system was designed to serve.
At the Signal stage: the foundational work was creating the conditions under which reliable operational signal could exist at all. Structured ingestion from disconnected sources, sensor-based capture for operational states not previously tracked, standardised data models, governed pipelines — these are not analytics investments. They are signal investments. Before this architecture existed, what the organisation had was operational history: records of what had happened, assembled manually, arriving too late to change the decisions they were meant to inform. After it, the organisation had operational awareness: a continuously updated view of what was happening, surfaced at the moment when it could still change an outcome.
At the Execution stage: the operational decisions the system was designed to support — threshold alerts, real-time state visibility, supply planning, proactive response — live in the Execution layer. The signal system created the precondition for better execution. A system that senses in real time makes a different category of operational decision possible — not just better-informed versions of the same decisions, but decisions that could not have been made at all when the sensing layer was absent. That is the connection between the Signal investment and the Execution value it was designed to produce.
The organisation described in this case is a major global energy company. Specific identifying details have been kept general deliberately — this is standard practice when documenting real operational work.
The architectural decisions, the constraints, the trust-building challenges, and the lessons described are real. Systems in Practice exists to document how complex systems actually evolve — not how they are designed to evolve in theory. The distance between the two is where the genuine learning lives.
Operations that run —
but cannot yet sense.
If your organisation is navigating a version of this — systems that operate but lack real-time awareness, data that exists across your estate in fragments that cannot be connected or trusted, or a transformation programme where the foundational data layer has not been designed before the analytics layer was commissioned — this is the problem space I work in.