Human control

Artificial intelligence, automation and remote operations are transforming safety-critical industries at an unprecedented pace. Yet as technology becomes more capable, one question is becoming increasingly important: how do we ensure that humans retain meaningful control when autonomous systems reach their limits? Contrary to what many assume, greater autonomy does not reduce the importance of humans. It changes their role. Human operators may intervene less often, but when they do, the decisions they make are likely to be more complex, more time-critical and more consequential than ever before. This is one of the central themes explored in Safety by Design: Human-Centered Approaches to AI, Automation, and Remote Operations, where Stig Ole Johnsen and Gunhild Birgitte Sætren are among the contributing authors. Their work argues that meaningful human control is not something that naturally remains as systems become more autonomous. It must be deliberately engineered. Professor Gunhild B. Sætren notes, “We need to ensure humans stay truly in control in an age of AI, automation, and remote operations.” Sætren, who is also Professor and Manager of the Arctic Safety Centre in Svalbard, believes the discussion should move beyond principles and focus on practical system design. Meaningful human control is not just a symbolic emergency button, she says. It requires real authority, situational awareness, time, and the genuine ability to intervene when it matters. Gunhild B. Sætren, professor and manager of Arctic Safety Centre in Svalbard. Photo: Gunhild B Sætren The challenge is therefore not simply keeping a human in the loop. It is ensuring that operators have the authority, competence, situational awareness and time needed to understand what is happening and make sound decisions when automation behaves unexpectedly. Jooyoung Park, PhD candidate at NTNU Shore Control Lab and co-author, highlights that meaningful human control also depends on how information is presented to operators. “As people move from directly controlling systems to supervising increasingly capable automation, interfaces should support the entire cognitive process, from maintaining engagement during routine monitoring, to recognising critical cues, understanding their meaning, anticipating what comes next and evaluating possible responses.” Jooyoung Park, PhD candidate at NTNU Shore Control Lab. Photo: Lars Bugge Aarset/Fremtidens Industri As Stig Ole Johnsen, Senior Scientist at NTNU, explains, “In complex systems, control is not something we assume, it is something we must actively design.” A common misconception is that meaningful human control can be achieved by giving operators an emergency override or a final approval button. In reality, this creates only the appearance of control if operators lack the information, understanding or opportunity to act effectively. Without these conditions, human control becomes symbolic rather than operational. According to Park, this is why human-centred interfaces should provide operators with a clear “situation at a glance”, enabling them to rapidly build an accurate understanding of the operational picture and recognise when automation is approaching its limits. Stig Ole Johnsen, Senior scientist at NTNU Why better automation makes humans more important One of the great paradoxes of automation is that as systems become more capable, humans often become less involved in day-to-day operations, but more critical when something unexpected occurs. Operators may spend long periods supervising highly automated systems before suddenly being required to understand a complex situation and make the right decision within seconds. As Johnsen notes, “The challenge is that humans are expected to step in precisely when they are least engaged in the system during normal operation.” Increasing automation therefore does not eliminate human responsibility. It concentrates responsibility into fewer, but far more demanding, situations where the consequences of a wrong decision may be significant. For organisations developing autonomous and AI-enabled systems, this means that investing in technology alone is not enough. Equal attention must be given to designing systems that support human judgement, decision-making and intervention. Photo: Lars Bugge Aarset/Fremtidens Industri Aviation shows what meaningful human control looks like Few industries illustrate engineered human control better than aviation. Over decades, aviation has developed integrated systems of engineering, training, certification, operational procedures and reporting cultures that enable people to perform effectively under pressure. As Sætren explains, “Control in complex systems is not about one layer of defence, it is about how design, training and organisation work together under stress.” One of the clearest examples is the emergency landing of US Airways Flight 1549 on the Hudson River in 2009. After a bird strike disabled both engines shortly after take-off, Captain Chesley Sully Sullenberger and First Officer Jeffrey Skiles had only minutes to assess the situation. Their successful landing was not simply the result of exceptional piloting. It reflected decades of investment in cockpit design, crew resource management, simulator training and Human Factors engineering. Meaningful human control had already been built into the system through engineering, training and preparation. The same principle applies to autonomous and remotely operated maritime systems. When automation reaches its limits, operators must quickly understand the situation, assess their options and intervene effectively. Whether they succeed depends largely on how well the system has been designed to support human performance. US Airways Flight 1549 on the Hudson River in 2009. (Greg L/Wikimedia Commons) Design is a safety function A key insight from Human Factors research is that system design directly influences safety. Park argues that interface design should therefore be viewed as a safety function rather than simply a usability concern. “Instead of requiring operators to adapt to technology, systems should support the way people naturally monitor, interpret and act.” Interfaces, alarm systems, workload, automation logic and procedures all shape how people perceive situations and make decisions. She adds that interfaces providing a clear “situation at a glance” can reduce cognitive workload while enabling faster and more accurate understanding of operational situations. As Johnsen puts it, “System design is not neutral, it actively shapes human behaviour and the decisions that follow.” Jooyoung Park, PhD candidate at NTNU Shore Control Lab. Photo: Lars Bugge Aarset/Fremtidens Industri Poor design is therefore not simply a usability problem. It is a safety issue. Research referenced by the authors indicates that approximately 40 to 50 percent of adverse events may be associated with weaknesses in system design, operational procedures or technology. This represents an important shift in thinking. Rather than asking why an operator made a mistake, organisations should ask how the system influenced that decision. From this perspective, human error is not the end of an investigation. It is the beginning of understanding how the system can be improved. Human control as a continuous learning cycle Meaningful human control is not something that can be achieved once during system development. It must be maintained throughout the entire operational life of a system. Park points out that meaningful human control should be engineered from the outset by involving operators, designers, engineers and other stakeholders throughout system development. Human, operational and technical requirements should be considered together, rather than treating the human element as something to be addressed after deployment. NTNU Shore Control Lab. Photo: Lars Bugge Aarset/Fremtidens Industri This begins with design. Systems should be developed using Human Factors Engineering methods such as task analysis, workload assessment, alarm design, human-machine interface design and realistic operational testing. As Sætren points out, “Learning and continuous improvement don’t happen by chance. They depend on strong Human Factors and Ergonomics knowledge, psychological safety, and clearly defined responsibility.” Just as importantly, organisations must learn from work as it is actually performed, not only from how designers or procedures expect it to be performed. Real-world operations often differ from design assumptions. Understanding these differences allows organisations to continuously improve both technology and operational practices. A third principle is to investigate accidents and incidents from a systems perspective. As Johnsen argues, “Human error should be the starting point of an investigation, not the conclusion.” Rather than assigning blame, organisations should ask why people acted as they did and what factors influenced their decisions. Investigations should examine design, training, workload, automation behaviour and organisational conditions, then use those findings to improve future systems. This creates a continuous learning cycle where operational experience is fed back into system design, making future operations safer and more resilient. Testing of Candela P-12, Frostabåten. Photo: Lars Bugge Aarset/Fremtidens Industri Human control must be engineered As AI and autonomous technologies continue to evolve, the relationship between humans and machines is changing. Human operators are becoming supervisors of increasingly capable systems rather than direct controllers of every task. This shift makes meaningful human control more important than ever. The goal is not simply to keep humans involved. The goal is to ensure that they remain capable of understanding situations, making informed decisions and intervening effectively when technology reaches its limits. As Johnsen summarizes, “If we want meaningful human control, we must design for it from the start, not hope it appears when needed.” The message is clear. As artificial intelligence, automation and autonomy become more advanced, human control becomes more important, not less. Meaningful human control is not an emergency stop button. It is the outcome of deliberate design, responsible operations and continuous learning. MIDAS – Humans in Future Maritime Operations The article is based on Chapter 2 in Safety by Design: Human-Centered Approaches to AI, Automation, and Remote Operations, edited by Frøy Birte Bjørneseth, Stig Ole Johnsen, Ole Andreas Alsos, Vidar Hepsø, and Gunhild Birgitte Sætren. The book was initiated, funded and written as part of MIDAS – Humans in Future Maritime Operations. MIDAS is a national competence project that examines the role of humans in future maritime operations, where autonomy, artificial intelligence and automation are becoming increasingly important. The ambition is to ensure that new technology is developed in a way that safeguards safety, usability and meaningful human control. The project brings together researchers, technologists, designers and industry actors across the maritime value chain. SINTEF Digital contributes multidisciplinary expertise in human factors and digitalisation. DNV provides expertise in classification and maritime safety, while clusters such as Digital Norway, Ocean Autonomy Cluster and Blue Maritime Cluster ensure relevance for industry stakeholders. Through MIDAS, the goal is to strengthen Norwegian industry’s innovation capacity and contribute to the development and export of future autonomous maritime solutions. See also: New book puts humans at the centre of autonomous and AI-enabled systems Open access resource for industry and academia Published by CRC Press, Safety by Design is available as an Open Access book under the Creative Commons CC BY 4.0 licence. This means that the book can be downloaded free of charge, shared, copied and reused in research, education, training, industrial development and policy work, provided that the original source is properly credited. The book offers practical and research-based insights for technology developers, operators, designers, researchers, educators, regulators and policymakers working to ensure that autonomous, remote and AI-enabled systems are not only innovative, but also safe, usable and centred on meaningful human control. Read or download the book here

Human control

Why meaningful human control matters more as AI and autonomy advance

On the same subject

Breach VR: Reality is cross-platform

The Ocean Space Incubator in Trøndelag: Will strengthen Norway’s position in ocean space technology

Autonomimiljøet i Midt-Norge får støtte til millionprosjekt