AI Agents: The Rise of Claude & GPT

The Next AI Race: Building Agents Companies Can Actually Control

The first wave of AI was about one question:

Can AI do it?

Can it write code?

Can it analyze documents?

Can it create images?

Can it research a topic?

Can it automate a workflow?

Can it operate software?

Now the question is changing.

As AI systems become more capable and move from chatbots toward autonomous agents, businesses are facing a much harder problem:

Can AI do it while the company remains in control?

That question is becoming one of the biggest themes in enterprise AI.

Recent moves from Replit, Anthropic, and OpenAI point in the same direction. Replit is adding enterprise governance features such as an Admin API, audit logs, and workspace settings. Anthropic has restricted access to its advanced cybersecurity capabilities through controlled programs such as Project Glasswing. OpenAI has similarly been building trusted-access systems for advanced cybersecurity models.

These aren't three unrelated announcements.

Together, they reveal a larger shift:

The future of AI agents may depend just as much on governance as intelligence.

From chatbots to agents

A traditional chatbot waits for you.

You ask a question.

It gives you an answer.

You decide what to do next.

AI agents are different.

An agent can potentially take a goal and perform multiple steps to achieve it.

Instead of asking:

"How do I fix this software bug?"

You could tell an agent:

"Find the problem, fix it, test the application, and prepare the changes for review."

That sounds incredibly useful.

But now the AI isn't simply providing information.

It's taking actions.

And actions require permissions.

That's where enterprise AI becomes complicated.

The permission problem

Imagine an employee gives an AI agent access to a company database.

At first, everything looks fine.

The agent can retrieve information.

It can analyze it.

It can generate reports.

It can automate repetitive tasks.

But what happens when the agent needs information from another system?

What happens if it wants to modify a database?

What happens if it wants to send an email?

What happens if it wants to deploy code?

What happens if it encounters something it wasn't explicitly told how to handle?

Suddenly, the question isn't:

"Is the AI smart enough?"

It's:

"What is the AI allowed to do?"

That's a very different problem.

Intelligence without boundaries is not enterprise-ready

A company may have the world's most capable AI agent.

But if that agent has unrestricted access to sensitive systems, the company has created a massive risk.

Think about a human employee.

A junior developer normally doesn't have access to every production database.

A new employee doesn't automatically receive administrator privileges.

A contractor may only have access to the systems required for their job.

Why?

Because organizations understand a basic security principle:

Give people only the access they need.

AI agents will increasingly need the same treatment.

Instead of giving an agent unlimited permissions, companies can define exactly what the agent is allowed to access and what actions it can perform.

This is where identity, permissions, tool access, and policy become critical.

The new AI architecture

A future enterprise AI system could look something like this:

AI Agent

↓

Identity

↓

Permissions

↓

Approved Tools

↓

Action

↓

Audit Log

↓

Risk Assessment

↓

Human Approval

↓

Production

The important part is that the agent isn't operating alone.

Every important action exists inside a governance framework.

That framework determines what the agent can see, what it can change, and what happens when something unusual occurs.

Replit is showing what this looks like in software development

Replit's latest enterprise governance announcement is a good example.

The company introduced an Admin API, Audit Logs, and Workspace Settings to give organizations more control and visibility over how teams adopt AI with Replit.

That might sound boring compared with an AI model launch.

It isn't.

Imagine a company with hundreds of developers using AI coding agents.

Those developers can potentially create applications much faster than before.

But speed creates a management problem.

If an AI agent generates a large amount of code, organizations need to know:

Who initiated the work?

Which workspace was involved?

What changes were made?

What policies apply?

What happened before the code reached production?

Auditability becomes extremely important.

Replit's enterprise direction reflects that shift from simply providing AI coding capabilities to helping companies govern those capabilities at scale.

Audit logs become the memory of AI activity

Consider a normal software system.

If an administrator changes an important setting, the organization usually wants a record.

Who changed it?

When?

What was changed?

Why?

The same principle becomes even more important when AI agents are involved.

An agent might perform dozens or hundreds of actions during a single task.

If something goes wrong, humans need a way to reconstruct what happened.

Without logs, an organization could be left asking:

"What did the AI actually do?"

That's a terrible position to be in during a security incident.

Audit logs can provide a record of important actions and changes, giving organizations a way to investigate and understand AI activity.

As agents become more autonomous, this kind of visibility could become standard infrastructure.

Anthropic is taking a different but related approach

Anthropic's Project Glasswing provides another example.

Claude Mythos Preview demonstrated powerful cybersecurity capabilities, including finding and fixing vulnerabilities, but Anthropic did not release the model for general availability. Instead, it provided access to selected partner organizations under restrictions focused on cybersecurity use.

The reason is straightforward.

Cybersecurity AI is highly dual-use.

A model capable of discovering vulnerabilities can be incredibly valuable to defenders.

But the same capabilities could potentially be misused.

So Anthropic introduced additional controls around who could access the system and how it could be used.

That's governance at the model-access level.

Replit's approach is more about governance inside an enterprise software environment.

The underlying principle is similar:

More capable AI requires more deliberate control.

OpenAI is moving in the same direction

OpenAI has also been developing trusted-access mechanisms for advanced cybersecurity capabilities.

Its Trusted Access for Cyber program was created to expand access to frontier cyber capabilities while using identity and trust mechanisms to help ensure those capabilities reach appropriate defenders.

OpenAI later expanded the program with additional access tiers and GPT-5.4-Cyber for verified cybersecurity defenders.

More recently, OpenAI has described even more specialized cyber models and governed access through programs such as Daybreak.

Again, the pattern is clear.

The most powerful capabilities aren't necessarily treated like ordinary consumer features.

Access can depend on identity.

The intended use can matter.

The environment can matter.

And additional safeguards can be introduced when the capabilities become more powerful.

Why cybersecurity is an early example

Cybersecurity is one of the clearest demonstrations of the governance problem because the consequences can be immediate.

Imagine an AI agent that can inspect a company's code.

That's useful.

Now give it permission to modify the code.

More useful.

Now let it deploy the changes automatically.

Even more powerful.

But now imagine the agent makes a mistake.

A small coding error could break a production system.

A bad configuration could expose sensitive information.

An incorrect security change could make a vulnerability worse.

The more authority the agent receives, the more important controls become.

That's why cybersecurity may be one of the first areas where companies seriously experiment with different levels of agent autonomy.

The principle of least privilege comes to AI

Security engineers have used a concept called least privilege for decades.

The idea is simple:

Give an entity only the permissions necessary to perform its job.

AI agents are increasingly becoming another type of entity inside company infrastructure.

That means the same principle can apply.

An agent that summarizes documents doesn't need production database access.

A coding agent doesn't necessarily need access to financial records.

A marketing agent doesn't need permission to modify server configurations.

A security agent may need elevated access—but perhaps only inside a controlled environment.

This creates a much more granular AI architecture.

Instead of:

"Give the AI access."

The question becomes:

"Exactly what should this AI be allowed to access?"

Tool access could become the real control layer

Agents become powerful because they can use tools.

A model by itself can generate text.

Give it a browser, and it can research.

Give it a coding environment, and it can build.

Give it APIs, and it can interact with external systems.

Give it databases, and it can retrieve information.

Give it payment tools, and it can potentially move money.

Give it deployment credentials, and it can change production infrastructure.

The model might remain the same.

The tool permissions completely change what the agent can do.

That's why future AI governance may focus heavily on tool access.

A company could define:

Agent A

Can read documentation.

Can access GitHub.

Can run tests.

Cannot deploy.

Agent B

Can deploy to staging.

Cannot access production databases.

Agent C

Can access production.

Requires human approval for every change.

That is much closer to traditional enterprise security.

Human approval won't disappear

There is a temptation to imagine the ultimate AI agent as completely autonomous.

Give it a goal.

Walk away.

Come back when the job is finished.

For low-risk tasks, that may work.

But for high-impact actions, human approval can remain important.

Consider an AI agent managing company finances.

Would you let it transfer ₹50 lakh without approval?

Probably not.

What about deleting a production database?

Definitely not.

What about changing a security policy?

Probably not without review.

This suggests a useful concept:

Risk-based autonomy.

Low-risk actions can be automated.

Medium-risk actions can be monitored.

High-risk actions require human approval.

The AI doesn't need to ask permission for every tiny task.

But it shouldn't have unlimited authority either.

The "human in the loop" could evolve

Human oversight doesn't necessarily mean someone watches an AI agent every second.

That wouldn't scale.

Instead, organizations can define approval thresholds.

For example:

Low risk: automatic.

Medium risk: automatic with logging.

High risk: human review.

Critical: multiple approvals.

This could allow agents to operate quickly without removing organizational control.

The system becomes something like an automated employee with clearly defined authority.

AI governance may become a new infrastructure layer

Today, companies have identity systems.

They have access-control systems.

They have security monitoring.

They have compliance tools.

They have logging systems.

They have approval workflows.

As AI agents become more important, organizations may need another layer:

AI governance infrastructure.

This layer could answer questions such as:

Which agents exist?

Who owns them?

What models are they using?

What tools can they access?

What data can they see?

What actions can they perform?

What policies apply?

What did they do?

What requires human approval?

What happens when they violate policy?

That could become a major enterprise software category.

The AI administrator could become a real job

Companies already employ administrators for cloud infrastructure, databases, identity systems, and security.

AI agents could create another administrative discipline.

Imagine an AI operations or AI governance team responsible for managing the company's fleet of agents.

They might:

• Approve new agents

• Configure permissions

• Monitor activity

• Review logs

• Establish policies

• Manage model access

• Evaluate risk

• Investigate incidents

• Control sensitive tools

• Approve high-risk workflows

The role wouldn't necessarily be about building AI models.

It would be about making AI usable inside an organization without losing control.

The biggest mistake companies could make

The biggest mistake may be treating an AI agent like a normal software feature.

A button doesn't normally make decisions.

An agent can.

A traditional application generally follows predefined workflows.

An agent may interpret instructions and choose how to accomplish a goal.

That introduces uncertainty.

And uncertainty requires controls.

Companies should therefore think about agents more like semi-autonomous digital workers than ordinary software buttons.

If you wouldn't give an employee unlimited access to your entire company, you probably shouldn't automatically give an AI agent unlimited access either.

Governance shouldn't kill productivity

There's another danger.

Companies could overreact.

They could create so many restrictions that employees stop using AI.

If every AI action requires five approvals, nobody will want to use the system.

That defeats the purpose.

The goal should not be maximum restriction.

The goal should be appropriate control.

A useful governance system should allow safe work to happen quickly while slowing down risky work.

That's the balance companies need.

The future may be "permissioned autonomy"

This could become one of the defining concepts of enterprise AI.

Not:

Full autonomy.

And not:

No autonomy.

Instead:

Permissioned autonomy.

An agent gets authority within clearly defined boundaries.

It can act independently inside those boundaries.

It can use approved tools.

It can access approved data.

It can complete low-risk tasks automatically.

And when it reaches a boundary, it stops or requests approval.

That's much more practical than expecting humans to supervise every action.

The agent becomes part of the security architecture

Eventually, companies may treat every AI agent as an identity.

It could have:

An ID

An owner

A role

Permissions

Credentials

Tool access

Policies

Logs

Risk classification

Approval requirements

That is very similar to how organizations already manage users and services.

The difference is that the entity can make decisions.

That makes governance even more important.

What happens when agents manage other agents?

Here's where things get really interesting.

Imagine a company with hundreds of AI agents.

One agent writes software.

Another tests it.

Another scans it for security issues.

Another monitors production.

Another analyzes customer feedback.

Another manages documentation.

Another coordinates workflows.

Eventually, one agent could potentially delegate tasks to another.

Now you have:

Agent → Agent → Agent → Action

The governance problem becomes more complicated.

Who authorized the original task?

What permissions can the second agent inherit?

Can an agent pass credentials to another agent?

What happens if one agent makes a mistake and another agent acts on it?

This is why identity and permissions will likely become fundamental to agentic systems.

The future AI stack

The enterprise AI stack could eventually look like:

Model

↓

Agent

↓

Identity

↓

Policy

↓

Permissions

↓

Tools

↓

Data

↓

Action

↓

Monitoring

↓

Audit

↓

Human approval

The model is only one component.

And that may be the biggest misconception about the next phase of AI.

The most valuable enterprise AI system won't necessarily be the model with the highest benchmark score.

It could be the system that combines a powerful model with excellent controls.

Intelligence is becoming a commodity. Control may not be.

As models become increasingly capable, the difference between companies may increasingly come from how effectively they can deploy those models.

A business doesn't just need a smart AI.

It needs an AI that can safely operate inside its environment.

That means:

Reliable identity.

Clear permissions.

Controlled tools.

Good data boundaries.

Detailed logs.

Risk-based approvals.

Strong security.

Without those things, companies may hesitate to give agents important responsibilities.

With them, agents could become much more useful.

The real AI race

So the next AI race may not simply be:

Who has the smartest model?

It could become:

Who can build the smartest agent that a company is actually willing to trust?

That's a much harder challenge.

An agent that can do everything but cannot be controlled is difficult to deploy safely.

An agent that is perfectly controlled but can't accomplish anything isn't useful.

The winning systems will likely sit somewhere between those extremes.

Powerful enough to act.

Controlled enough to trust.

The question businesses should ask now

Before giving an AI agent access to a company system, businesses should ask:

What identity does this agent use?

What can it access?

What tools can it call?

What data can it read?

What actions can it perform?

What happens when it makes a mistake?

Are its actions logged?

Can access be revoked instantly?

Which actions require human approval?

And perhaps the most important question:

What happens if the agent behaves in a way we didn't anticipate?

Those questions aren't about slowing down AI.

They're about making AI deployable.

The next generation of enterprise AI

The AI industry has spent years proving that models can generate impressive outputs.

Now it has to prove something else:

That those models can safely operate inside the real world.

That's why Replit's governance controls matter.

That's why Anthropic's restricted cybersecurity access matters.

That's why OpenAI's trusted-access programs matter.

They're all examples of the same emerging philosophy:

More capable AI requires more sophisticated control.

The future isn't just about AI agents that can take action.

It's about agents whose actions can be understood, limited, monitored, and—when necessary—stopped.

Because the next question isn't:

"Can AI agents do it?"

We've already started answering that.

The real question is:

"Can we give AI enough freedom to be incredibly useful without giving it more power than we're prepared to control?"

That may be the defining engineering and business challenge of the agent era.

And the companies that solve it could be the ones that turn AI agents from impressive demos into everyday infrastructure.