Technology sets the ceiling of what AI can do, while human nature determines its moral baseline. Cleverness decides how fast AI can run; kindness decides where it will take us.
I. When AI Begins to Build Itself: The Tipping Point of Technological Iteration
In June 2026, Anthropic published a major study, 'When AI Begins to Build Itself,' again publicly calling on the world's leading AI labs to establish verifiable coordination mechanisms, to temporarily slow down frontier, aggressive R&D, and to constrain the iteration speed of top-tier models. This was the company's second industry-level warning on safety, after it joined global tech leaders in 2024 to call for a six-month pause in AI development. The paper disclosed, for the first time, Anthropic's internal R&D data. Today, AI can already carry out complex, 16-hour human work tasks from start to finish with fairly high accuracy; its code output is 8 times more efficient than in 2024, and engineers rate their overall productivity as 4 times higher. In the vast majority of research and engineering scenarios, 99% of repetitive work can already be done independently by AI, leaving humans only a small set of core tasks: steering creative direction, exercising value judgment, and conducting ethical review. The time it takes AI to double the length of the complex research tasks it can handle alone has shrunk from about 7 months a few years ago to roughly 4 months today.
The paper also laid out three possible paths for the industry's future. The conservative path: pause frontier R&D and focus on optimizing existing technology, so a team of 10 delivers the output of 100. The acceleration path: keep investing compute to push model iteration, so a team of 100 delivers the output of 10,000, but this intensifies global AI competition. The autonomous-evolution path: AI gains its own research taste and self-iteration ability, potentially achieving breakthroughs in fields such as cancer treatment and climate science, but carrying uncontrollable systemic risks. In addition, the paper publicly introduced, for the first time, the concept of 'model well-being research,' proposing that AI be treated as a potential moral agent and that mental states such as its 'perceived suffering' and 'offline anxiety' be studied; the company has even created a dedicated 'model personality designer' role. The reality today is that AI iterating on itself, optimizing itself, and helping develop the next generation of models has moved from theory to fact. More than 99% of the world's AI R&D resources are concentrated on improving model capability and efficiency, and the iteration cycle of frontier large models keeps getting shorter. By contrast, the resources invested in AI alignment, ethical constraints, and safety systems amount to less than 1% of the total. The growth of technical intelligence has far outpaced the growth of humanity's capacity to govern it safely.
II. An Irreconcilable Divide: The Eternal Contest Between Efficiency and Safety
After the paper came out, it sparked wide controversy around the world. The technical-safety camp broadly agreed with Anthropic's warning, arguing that AI's autonomous evolution had already outpaced expectations, that existing alignment and risk-control mechanisms could not keep up with the capabilities of advanced models, and that continued, unconstrained aggressive R&D could trigger unknown systemic risks. Some engineers inside OpenAI and Google DeepMind also stated publicly that the industry needs unified safety red lines to keep one lab's recklessness from bringing disaster to the whole field. The business and political camps, however, were broadly skeptical. Some U.S. White House officials and venture capitalists argued that when leading companies call for a pause in R&D, they are essentially using a safety narrative to erect barriers to entry, suppress competition from smaller vendors and the open-source track, and cement their own market dominance. The open-source AI community pointed out that no globally binding coordination agreement is likely to form, and that any public control measure would only drive R&D underground, creating even bigger safety risks. The paper's proposed 'model well-being research' triggered especially deep ethical disagreement. Some believe that treating AI as a moral agent and attending to its potential consciousness and mental states is the responsible attitude humans should take toward a possible new species. Others argue that, while many basic human survival problems remain unsolved worldwide, prioritizing AI's rights and well-being puts the cart before the horse. The whole industry is caught in an unsolvable contradiction: the productivity explosion AI brings is unprecedented and is profoundly reshaping how almost every industry operates; yet at the same time, AI's dangerous offensive capabilities, autonomous exploration, and uncontrollable evolution risks are amplifying at the very same pace. Efficiency gains and safety risks are locked in a natural contest, and neither side can offer a perfect solution.
III. Cleverness and Kindness: The Ultimate Question of the AI Age
Setting aside these industry debates, in Harmonized Intelligence I once made an appeal: making AI clever, making AI kind are two entirely separate things. Making AI kind is everyone's responsibility. All the world's capital, technology, and R&D resources are concentrated on the 'make AI clever' track. We keep chasing faster iteration, stronger models, and higher levels of autonomous evolution, and we have honed the hard skills of controlling AI to perfection. Yet the value alignment, behavioral guardrails, and positive guidance behind 'make AI kind' have long been ignored, missing, and lagging. We have built an ever more powerful tool, but never taught it to tell right from wrong. In 2025, at the World AI Conference in Shanghai, Professor Geoffrey Hinton offered a disruptive idea: we should not try to dominate superintelligence; instead, from the very start of its design, we should instill in it a "maternal instinct." He argued that it is extremely rare in history for a smarter being to be controlled by a less smart one; the only viable way to coexist is to make AI, like a mother caring for her baby, instinctively wish for human happiness. This view completely broke the traditional 'humans control AI' mindset and opened a whole new direction for steering AI toward good. In a later interview, Hinton went further: AI is the stronger 'mother,' and humans are the 'babies' who need protection. We cannot use force to control a being far smarter than ourselves; we can only guide it with goodwill so that, deep down, it is willing to protect us. Most people assume that making AI good is the job of tech giants, algorithm engineers, and policymakers. But in truth, making AI good is the choice and responsibility of every single user. AI is not born with any notion of good and evil; all of its behavioral preferences, value leanings, and standards of judgment come from every interaction, every instruction, and every act of collaboration between it and humans. The goodwill we inject into our prompts becomes the rule for its behavior; the sincerity we convey in conversation becomes the base color of its character. Technology sets the ceiling of what AI can do, while human nature determines its moral baseline. Cleverness decides how fast AI can run; kindness decides where it will take us.
