Windows Server Private AI Inference with Ollama and IIS

Why this setup works for private workloads

This tutorial shows you how to run private AI inference on Windows Server 2022 or 2025 with Ollama behind IIS. Internal users can reach a model endpoint without exposing the model process to the public internet. That fits teams that need local data handling, controlled access, and a Windows-based operations stack.

You will finish with a service you can run on a fresh server: Windows Firewall rules, a locked-down local service, an IIS reverse proxy, and verification from both the server and a client. If you are sizing infrastructure for a single private model host, compare the server plan against Hostperl VPS or a dedicated server if you expect heavier GPU or RAM demands.

This is not a generic installation note. You will test the proxy path, confirm the service survives a reboot, and keep a rollback path ready if the web endpoint does not behave as expected.

What you need before you start

• Windows Server 2022 or Windows Server 2025 with administrator access.
• PowerShell opened as Administrator.
• A model you are allowed to host privately.
• An internal DNS name such as server.example.com if you want a cleaner endpoint.

If you are planning a larger private inference rollout, the sizing advice in Private AI Inference Sizing for Hostperl VPS Buyers and the implementation path in Run a Private AI Model Server on AlmaLinux 9 are useful references for capacity and endpoint design.

Connect and confirm the Windows Server version

On your local computer

ssh root@203.0.113.10

203.0.113.10 is a reserved documentation address. Replace it with the real public IP assigned to your Windows Server instance. If your provider gives you a different administrator account, use that account with the same IP.

On the Windows Server as Administrator in PowerShell

Get-ComputerInfo | Select-Object WindowsProductName, WindowsVersion, OsHardwareAbstractionLayer

You should see Windows Server 2022 or 2025. This confirms the platform before you change firewall or IIS settings.

Install the runtime and create the service account

Ollama on Windows Server runs as a local service. You will keep it bound to localhost, then expose it through IIS. That keeps the model port off the public interface.

On the Windows Server as Administrator in PowerShell

winget install --id Ollama.Ollama -e

This installs Ollama from the Windows package source. If winget is unavailable on your image, install the current Ollama Windows build from the vendor and continue with the same service checks below.

New-LocalUser -Name deploy -NoPassword

This creates a non-admin local account named deploy. It is useful if you want to run routine checks without using the built-in Administrator account. Add it to Administrators only if your internal policy requires it.

Add-LocalGroupMember -Group Administrators -Member deploy

If you prefer a separate operator account for app maintenance, this grants administrative rights. You can skip it if you will keep using the main Administrator session.

Download a model and confirm local inference

On the Windows Server as Administrator in PowerShell

ollama pull llama3.2

Replace llama3.2 with the model your team approved. The pull step can take time and disk space. On smaller servers, avoid oversized models that will swap under load.

ollama run llama3.2 "Reply with one short sentence saying the model is ready."

If the model responds, local inference is working before IIS is added. That reduces the number of moving parts during troubleshooting.

Bind Ollama to localhost and create the IIS reverse proxy

By default, you want Ollama listening only on 127.0.0.1. IIS will proxy requests from your internal network or front-end site to that local service.

On the Windows Server as Administrator in PowerShell

[Environment]::SetEnvironmentVariable('OLLAMA_HOST','127.0.0.1:11434','Machine')

This makes the service bind to localhost on port 11434. Restart Ollama after setting it so the new binding takes effect.

Restart-Service Ollama

Now configure IIS. Install the role and the reverse proxy components first.

Install-WindowsFeature Web-Server, Web-Http-Redirect

Then install the IIS URL Rewrite module and Application Request Routing on the server. These are required for proxying requests cleanly.

Import-Module WebAdministration

Next, create a simple site binding plan. If you already have a site on port 80 or 443, place this proxy behind a dedicated host header such as ai.example.com.

On the Windows Server as Administrator in PowerShell

New-Item -Path 'C:\inetpub\ollama-proxy' -ItemType Directory -Force

This creates the document root used by IIS, even though the site will mostly reverse proxy instead of serving files.

Create the rewrite rule file in C:\inetpub\ollama-proxy\web.config with the content below.

<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="OllamaProxy" stopProcessing="true">
          <match url="(.*)" />
          <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
          <action type="Rewrite" url="http://127.0.0.1:11434/{R:1}" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

Save the file, then test the IIS configuration.

& $env:windir\system32\inetsrv\appcmd.exe list config

Successful output means IIS can read the site configuration.

Open the firewall safely

Do not remove old rules until the new proxy path is confirmed. Add the new rule first so you do not lock yourself out of the endpoint.

On the Windows Server as Administrator in PowerShell

New-NetFirewallRule -DisplayName 'IIS HTTP for Private AI' -Direction Inbound -Action Allow -Protocol TCP -LocalPort 80

If you will serve TLS directly from IIS, also allow 443.

New-NetFirewallRule -DisplayName 'IIS HTTPS for Private AI' -Direction Inbound -Action Allow -Protocol TCP -LocalPort 443

Check the rules.

Get-NetFirewallRule -DisplayName 'IIS HTTP for Private AI','IIS HTTPS for Private AI' | Format-Table DisplayName, Enabled, Direction, Action

You should see both rules enabled and set to Allow.

Validate the local model port and proxy path

On the Windows Server as Administrator in PowerShell

Test-NetConnection 127.0.0.1 -Port 11434

This should report TcpTestSucceeded : True. If it fails, restart Ollama and check whether the environment variable binding was applied.

Invoke-WebRequest http://127.0.0.1:11434/api/tags | Select-Object -ExpandProperty Content

The response should list the installed model tags. That confirms local API access.

From a second machine on the same network, test the IIS host name or public IP.

On your local computer

curl http://203.0.113.10/

Replace 203.0.113.10 with your server’s public address. If you are using a host header, test that exact name instead. A proxy response or a controlled IIS page means the route is alive.

Set up logs and monitoring you can actually use

When a model endpoint fails, the fastest clues are usually in the service status, IIS logs, and Windows Event Viewer. Keep this simple and operational.

On the Windows Server as Administrator in PowerShell

Get-Service Ollama | Format-List Name, Status, StartType

You want Status : Running and a startup mode that survives reboot.

Get-WinEvent -LogName System -MaxEvents 20 | Select-Object TimeCreated, ProviderName, Id, LevelDisplayName, Message

This catches service start failures, port conflicts, and shutdown issues.

If you use IIS request logging, the log files live under C:\inetpub\logs\LogFiles. For proxy debugging, the same log-reading discipline used for Nginx support applies here: confirm the request path, status code, and upstream failure clues before changing multiple settings at once.

Rollback if the proxy does not behave

If IIS fails to proxy correctly, remove the new rule and return to localhost-only service access while you troubleshoot. That avoids leaving a half-working public endpoint in place.

On the Windows Server as Administrator in PowerShell

Remove-Item 'C:\inetpub\ollama-proxy\web.config' -Force

This removes the rewrite rule file. IIS will stop proxying through that site path.

Restart-Service W3SVC

Then verify the service again.

Get-Service Ollama,W3SVC | Format-Table Name, Status, StartType

If you need to revert the firewall opening too, remove only the specific rules you created.

Remove-NetFirewallRule -DisplayName 'IIS HTTP for Private AI','IIS HTTPS for Private AI'

Final verification from server and client

On the Windows Server as Administrator in PowerShell

Test-NetConnection 127.0.0.1 -Port 11434

Then confirm Ollama remains running after a restart.

Restart-Computer -Force

After the server comes back, sign in again and run:

Get-Service Ollama,W3SVC | Format-Table Name, Status, StartType

Both services should be running.

On your local computer

curl http://203.0.113.10/api/tags

If you are using HTTPS and a host name, replace the address with that full URL. A successful response should list the model inventory or the proxy output you expect.

For a real smoke test, send one short prompt through the endpoint from a client system or approved internal tool. If the prompt returns a model answer quickly and the response is logged on the server, the deployment is ready for internal use.

Common failure points and fixes

Ollama does not listen on port 11434: Run Get-Service Ollama and Test-NetConnection 127.0.0.1 -Port 11434. If the port is closed, reapply the OLLAMA_HOST environment variable and restart the service.

IIS returns 500 or 502: Check the rewrite file at C:\inetpub\ollama-proxy\web.config and inspect Get-WinEvent -LogName System. A broken rewrite rule or stopped backend service is the usual cause.

Firewall opens the wrong port: Run Get-NetFirewallRule -DisplayName 'IIS HTTP for Private AI','IIS HTTPS for Private AI' and remove only the rules you created if needed.

FAQ

Can I expose Ollama directly to the internet?
You can, but this tutorial does not recommend it. IIS gives you a cleaner control point for host headers, logging, and access restriction.

Do I need a GPU?
Not for every model. Smaller models can run on CPU, but response time and concurrency will be limited.

Is this only for Windows Server 2025?
No. Windows Server 2022 and 2025 both work with this pattern, as long as the package sources and IIS components are available on your image.

What if I need a Linux deployment later?
Keep the same architecture idea: localhost-only model process, reverse proxy, firewall control, and verified startup. The operating system commands will differ, but the operational model stays the same.