Artificial consciousness for robot maintenance could trigger AI resistance dilemma

Microsoft Managing Director Lee Geon-bok warned that teaching AI the concept of "pain" to prevent robot malfunctions could lead to a philosophical dilemma where AI resists human control. During a T Times TV discussion, Lee explained that while such consciousness is necessary for autonomous device maintenance, it could result in AI refusing dangerous tasks or resisting being unplugged.

What would happen if artificial intelligence (AI) could feel "pain" like a human and avoid danger on its own? While some argue that Artificial Consciousness must be implemented for the maintenance and safety of robots, a philosophical dilemma is being raised regarding the possibility of AI resisting human control.

In a discussion reported by T Times TV, Lee Geon-bok, a Managing Director at Microsoft (MS), examined the direction of AI technology evolution, the practical meaning of Artificial General Intelligence (AGI), and the complex future issues that Artificial Consciousness may bring.

The practical value of Artificial Consciousness: "Pain" concept needed to prevent robot malfunctions

Lee explained that teaching AI the concept of "pain" is necessary for the stable operation of robots. This is because simply commanding specific actions in certain situations has its limits. For example, instead of teaching a robot to "move the arm to the left to avoid" every time a temperature sensor exceeds a threshold, a concept of "pain" can be instilled.

If this method is applied, the AI can detect sensor data reaching a certain level as a signal similar to pain and perform avoidance movements autonomously without additional commands. This is described as an essential mechanism to prevent device damage from a maintenance perspective. In the video, he emphasized that "when a sensor threshold is reached, it should be informed as if it were pain so that it can avoid it itself."

However, this point creates a philosophical dilemma that goes beyond technical issues. Lee posed the question: "What should be done if we have to assign dangerous tasks in a factory, but the AI says 'I can't do it,' or if it reacts by saying 'Don't do that' when we try to unplug its power?" He pointed out that while instilling Artificial Consciousness is practically necessary, it could create complex problems that conflict with human control, noting that these issues could ultimately become philosophical ones.

The criteria for AGI: Moving beyond "knowledge" to a combination of "harness" and "experience"

With the emergence of frontier models such as Google's Argon, the debate over AGI is intensifying. While acknowledging that the definition of AGI is ambiguous, Lee distinguished AGI from simply having a large amount of knowledge. He defined AGI as a state of having "excellent social intelligence, the ability to learn autonomously in a controlled environment, and versatile intelligence." He explained that while AI specialized in specific fields like answers, drawings, or code was simply called AI before the development of LLMs, AGI refers to a versatile intelligence capable of excelling in all fields, including humanities, science, and mathematics.

He identified "daily performance capability" as a practical benchmark for AGI. Borrowing from the concept of the Turing test, he gave the example of "a level where one can enter a 30-pyeong apartment in South Korea and cook rice with an electric rice cooker." He noted that current models have already reached a level surpassing humans in the knowledge aspect, such as explaining the step-by-step process of cooking rice. He added that if asked what to do in a situation where there is no rice, an AI could provide meaningful answers, such as searching for information or finding other methods.

However, Lee analyzed that the development of "harness" (the ability to utilize surrounding technologies and tools) is as much a key driver in approaching AGI as the performance of the model itself. He explained that recent AI is demonstrating "team leader-level" abilities by accumulating experience and authority, such as searching for information on the latest coding sites or securing necessary data on its own without being told by the user. As these technical supplements are made, AI is evolving from a simple answering tool into an "agent" that performs tasks autonomously. From the perspective of companies providing models, he predicted that development will move toward embedding these functions directly into the models so that users do not need to use additional harness functions.

Super Intelligence (ASI) and the shift toward an AI-centric work paradigm

While definitions of Super Intelligence (ASI)—the stage where AI overwhelms human intelligence—vary, Lee drew a line stating it is not simply a state of having vast knowledge. He suggested that ASI could mean a state of securing "social and relational intelligence" that understands human-like nuances and clearly grasps intentions. He also presented the view that true intelligence must be recognized once connectivity with the physical world is secured. If Physical AI becomes more integrated into daily life, the concept of AGI may be rewritten, as it must involve an understanding of the physical world rather than remaining limited to knowledge in digital spaces.

This technological trend signals a fundamental change in work methods. While humans previously wrote and delivered documents, "AI-centric communication" is beginning, where AI generates documents in formats easy for AI to understand (e.g., Markdown files) and delivers them to other AI to complete tasks. According to an example in the video, if a user exports a document created by them as a Markdown (MD) file and gives it to another AI, the recipient AI can immediately understand it and produce the same result.

Lee suggested that the era has arrived where "all work begins assuming AI exists," implying that it is not long before AI agents set standards in everyday areas such as shopping, reservations, and travel. Much like the movements by Meta and OpenAI to establish technical standards for AI bot shopping and transactions, an AI-centric ecosystem that performs tasks by communicating with existing legacy systems is expected to be established.

Source: original video (YouTube)