This isn’t an anti-AI piece. It’s a reality check.

Probabilistic vs. Deterministic — the distinction nobody in the C-suite asks about

A large language model is, by design, a probability engine. Ask it the same question twice and you can get two different answers. That’s not a bug to be patched away — it’s the nature of the thing. Traditional software, by contrast, is deterministic: same input, same output, every time, forever.

Most of the actual work happening inside companies — payroll calculations, leave approvals, financial reconciliations, standard reports — is deterministic. It has a correct answer, a fixed rule set, and zero tolerance for “approximately right.” This is exactly the kind of work traditional software has solved cheaply and reliably for decades.

Yet a huge share of current AI initiatives are aimed squarely at this deterministic work — not because it’s a good technical fit, but because nobody wants to be the department that isn’t “doing AI.”

The result is systems that are probabilistic being asked to do a deterministic job, wrapped in a disclaimer that says “AI can make mistakes.” That disclaimer isn’t boilerplate. It’s an admission that should stop a lot of these projects at the pitch stage. Instead it gets buried in a footer, and the project ships anyway.

The shift nobody validated

A pattern playing out across large service organizations right now: internal data and analytics teams pivoting from dashboards to “agents” — conversational interfaces that answer questions in natural language instead of showing a chart.

The uncomfortable part isn’t the technology. It’s who’s driving the decision. It’s rarely the HR team, the finance team, the ops or sales function — the actual consumers of these dashboards — asking for a conversational agent. It’s the teams building them, motivated by a very reasonable but very different incentive: something demonstrable for a board update, an internal AI showcase, or a slide that proves the org is “in the race.”

Nobody asked the person who actually uses the dashboard every day: do you want to type a question instead of glancing at a chart you’ve read a hundred times? In most cases, the honest answer would be no. But that question rarely gets asked, because the initiative isn’t really being built for the end user. It’s being built for the room upstairs.

The irony at the center of it all

Here’s the part that should worry finance departments more than anyone: after all this build-out, the actual source of truth for most teams is still an Excel sheet. Not because Excel is glamorous, but because it’s free, transparent, and trusted. People can see the formula. They can audit it. There’s no black box and no per-query cost.

Now watch what happens: the same question that used to be answered by opening a spreadsheet — free, instant, deterministic — gets routed through an AI agent instead. The agent burns tokens, calls a model, generates a natural-language summary, and then, often, regenerates the same spreadsheet as its output. The work that used to cost nothing now has a metered price tag attached to it, incurred every single time someone asks. And this was invisible while heavy subsidies and discounted token pricing made it look free. As those subsidies wind down, the actual unit economics are starting to show — and they’re not flattering.

None of this is being tracked at the level it should be. Conversations with people close to large-scale AI rollouts reveal environments where thousands of agents have been spun up over time — proof-of-concepts, pilots, one-off demos — with no consolidated inventory of what’s still running, what it’s costing, or whether anyone is still using it. An agent built for a demo six months ago can still be quietly consuming budget today, and the bill only becomes visible long after it should have been shut down. That’s not an AI strategy. That’s uncontrolled sprawl with a monthly invoice.

Do decision-makers even look?

There’s a harder question buried in all of this: when the agent produces its polished natural-language summary, does the executive actually read it and think it through — or do they just want a junior to hand them a filled Excel sheet, the same as before, so they can move on with their day?

If it’s the latter, the AI layer isn’t informing the decision. It’s theater sitting on top of a decision-making process that never changed. You cannot responsibly base a business call on an output that ships with a disclaimer saying it might be wrong — not when the underlying deterministic data was already sitting right there, for free, before the AI system re-derived it at a cost.

The people cost nobody puts in a deck

There’s also a quieter toll this creates on the people actually building these systems. Someone with a data engineering background — hired to build pipelines and models — ends up asked to produce a demo video to show off in a board meeting or an internal AI event. Not because it’s their job, but because the org needs something to point to, and the people who can actually build the thing are also the ones roped into selling it. That work often goes unrecognized and uncompensated, sitting entirely outside the job description it was layered onto.

This is a symptom of the same root cause: the initiative exists to be shown, not to be used. When the primary audience for a product is a board slide instead of an end user, the people doing real technical work get quietly repurposed into marketing, and nobody adjusts their role, their scope, or their pay to reflect it.

The simplest test of all: the button

Strip away the architecture diagrams and ROI slides and there’s a test simple enough that almost nobody applies it: if the UI already has a button for the task, why would anyone type a sentence to an AI to do it instead?

Applying for leave, raising a support ticket, filling a timesheet — these are one-click actions today. Replacing a click with a typed request to an agent isn’t a productivity gain; it’s added friction wearing an AI badge. End users know this instinctively, which is why adoption of these “agentic” replacements for already-solved workflows tends to be quiet and thin, no matter how good the demo looked.

Where this doesn’t apply

To be fair to the technology itself: none of this means AI has no place in the enterprise. Probabilistic systems are genuinely well-suited to genuinely probabilistic problems — unstructured document synthesis, first-draft generation, anomaly detection in noisy data, coding assistance, customer sentiment analysis, and other work where “highly likely correct with human review” is an acceptable and even valuable standard. The failure isn’t AI. The failure is applying a probabilistic tool to deterministic work purely because leadership wants a story to tell, without asking the people who’d actually use it whether they wanted it in the first place.

Soaring Costs and AI Agent Sprawl

Meanwhile, deploying AI can be expensive and hard to track. After introductory credits and subsidies end, every AI query incurs token charges — and usage can skyrocket unexpectedly. As the Wall Street Journal reports, companies are already “pumping the brakes” as the “soaring cost” of AI compute hits the budget. Unchecked, AI agents can quietly burn millions: for example, one healthcare organization ran up over $6 million in API costs (consuming a trillion tokens) before the finance team even noticed what was happening. A recent survey found 85% of companies routinely miss their AI cost forecasts by more than 10%. Put bluntly, work that was once done for free in a spreadsheet now incurs a token cost, often with no extra benefit — yet companies still feel pressure to keep a fleet of AI agents running. Without strict governance, the AI rollout risks becoming a budget sink rather than a productivity gain.

The reckoning

Token discounts and subsidized pricing masked the real cost of this approach for a while. As that support fades, the unit economics of “AI for everything” are becoming visible, and in a lot of cases the ROI simply isn’t there — because the ROI was never the point. The point was optics: a board deck, a submission, an event demo, proof that the organization is “in the race.”

Eventually the money question comes due. And when it does, the organizations that will have something real to show are the ones that asked a simple question before building anything: does the person who has to use this every day actually want it? Everyone else will be left explaining, in a board meeting, why millions went into a race that never had a finish line — and why the spreadsheet is still, quietly, doing all the real work.

Part of the series AI Cloud and Data Engineering Articles. Every pipeline in this series so far — SharePoint ingestion, AES key management, Snowflake-Azure connectivity — leans on Azure Key Vault to hold the secrets that make it work. This article is about what happens after those pipelines have been running for a year or two: the vault fills up, and there’s no “select all, delete” button for cleaning it out.

Who this article is for: Sections 1–4 are written for anyone who owns or is accountable for a system that uses Key Vault — a business or IT lead who needs to understand the risk and approve a cleanup, not run the commands. Sections 5 onward are written for the developer or operator who will actually run them. Read the parts that match your role; nothing after Section 4 assumes you’re a business reader.

Why this matters

Every application that isn’t hard-coding its passwords stores them somewhere safer — usually Azure Key Vault. That’s the right approach. But a vault that’s been backing an HR, Sales, or Finance system for a couple of years doesn’t stay small. It accumulates secrets the same way an inbox accumulates emails: some belong to systems that were retired last year, some were one-off credentials created for a proof-of-concept that never shipped, and some are old versions left behind by a rotation policy.

Modern enterprise AI systems make this challenge even bigger. Imagine an HR organization with 80,000 employees building a machine learning model to predict attrition, to predict sales, to predict budget vs actuals, recommend training, or analyze workforce trends. For privacy and compliance reasons, the ML model should never receive an employee’s actual ID. Instead, the application generates a unique secret or token for every employee, stores the mapping securely in Azure Key Vault, and sends only the secret to the AI model. Once the model returns its predictions, the application looks up the secret and maps it back to the original employee ID.

This approach protects sensitive information and helps organizations meet privacy and governance requirements. However, it also means the Key Vault may eventually contain tens of thousands of employee-specific secrets. As employees leave the organization, a new employee joins, mergers happen, projects are retired, or an application accidentally generates duplicate or incorrect secrets because of a bug, those entries must be removed. Otherwise, the vault slowly fills with secrets that no longer serve any purpose.

The problem is that Azure provides no native “Select All → Delete” option for secrets. You can delete them one at a time in the Azure Portal, or you can automate the process yourself. For a vault containing only a few secrets, that’s manageable. But for enterprise workloads that may store one secret per employee, customer, device, tenant, or application integration, cleaning up thousands — or even hundreds of thousands — of secrets becomes a significant operational challenge.

That’s a genuine business and security concern, not just housekeeping:

• Security risk — Every unnecessary secret increases the attack surface. Even if the associated employee or application no longer exists, the credential remains a protected asset that must still be governed.

• Operational complexity — When a vault contains tens of thousands of secrets, it becomes increasingly difficult for administrators to distinguish active secrets from obsolete ones, making troubleshooting and maintenance harder.

• Compliance and privacy — Regulations and internal governance policies often require organizations to remove credentials and identifiers associated with users or systems that no longer exist. Retaining unnecessary secrets can create audit findings and increase compliance risk.

• Application reliability — Bugs or failed deployments can accidentally create duplicate or incorrectly generated secrets. Without an efficient bulk cleanup mechanism, correcting these mistakes becomes slow, error-prone, and time-consuming.

This is why understanding how to safely delete, bulk-delete, and automate secret cleanup in Azure Key Vault is an important operational capability for both enterprise teams and developers. It’s not simply about keeping a vault tidy — it’s about maintaining security, ensuring compliance, and keeping large-scale cloud applications manageable as they grow.

What Azure Key Vault actually stores

Azure Key Vault is Microsoft’s managed service for holding the sensitive values an application needs but should never have sitting in its source code or a config file. It manages three kinds of objects:

• Secrets — arbitrary sensitive strings: database passwords, API keys, connection strings, OAuth tokens, employee secret IDs for ML models. This article is about secrets specifically.

• Keys — cryptographic keys used for encryption, signing, and wrapping other keys.

• Certificates — TLS/SSL certificates, with automated renewal support.

The value of centralizing this: developers reference a secret by name and vault URL at runtime, the value itself is never in a repository that could be exposed, access is controlled through Azure identity (RBAC or access policies) rather than a shared password, and every read or write is logged for audit.

Why secrets multiply so quickly

Secrets aren’t created just once — they’re created per user, per application, per environment, or per integration, and that adds up surprisingly fast in an enterprise.

Consider a few common business scenarios:

• HR systems store employee secret tokens, payroll API keys, and benefits portal credentials. As organizations grow, new employees are onboarded, others leave, and new HR applications are integrated, causing the number of secrets to increase continuously.

• Sales platforms rely on CRM API tokens, quote engine credentials, and partner portal secrets. Every new customer integration, marketing campaign, or third-party service introduces additional credentials that must be securely managed.

• Finance applications maintain ERP connection strings, banking or SFTP credentials, and OAuth tokens for reporting platforms. Regular credential rotation, audit requirements, and temporary project environments constantly generate new secrets and new versions of existing ones.

• AI and Machine Learning workloads often use employee-to-token mappings, encryption keys, model access tokens, and inference API credentials. Privacy regulations frequently require organizations to replace sensitive identifiers with secure tokens before sending data to AI models, creating yet another category of secrets that must be maintained.

Consider an enterprise HR analytics platform that processes data for 80,000 employees. Instead of sending an employee’s actual ID to a machine learning model, the application first generates a unique token or secret, stores the mapping securely in Azure Key Vault, and sends only the token to the model. After the model produces its predictions, the application retrieves the original employee ID using that stored mapping.

This design protects sensitive employee information and helps satisfy privacy and compliance requirements. However, it also means the Key Vault may contain one secret for every employee — 80,000 secrets for employee mappings alone. If employees leave the organization, records must be removed, or a software bug accidentally generates incorrect or duplicate mappings, those secrets also need to be cleaned up.

Now combine that with thousands of API keys, connection strings, OAuth tokens, certificates, temporary proof-of-concept credentials, and secrets created by automated rotation policies. Over time, it’s easy for an Azure Key Vault to contain tens or even hundreds of thousands of secrets, many of which are no longer actively used. Managing that lifecycle efficiently becomes just as important as securing the secrets themselves.

The one concept to understand before you delete anything: soft-delete vs. purge

This is the single most important thing to get right, for business owners and developers alike, before approving or running any cleanup.

Deleting a secret in Key Vault does not remove it immediately. Key Vault has soft-delete behavior — think of it as a recycle bin. When you “delete” a secret, it moves into a recoverable, soft-deleted state for a retention period (7–90 days, 90 by default) and can still be restored during that window. Permanently removing it requires a second, separate action: purging it.

Two details worth knowing before you plan a cleanup:

• Soft-delete is no longer optional. As of February 2025, Microsoft enabled soft-delete on all key vaults and removed the ability to turn it off — every vault has it, whether the team configured it or not. If you’re working with an older vault, you may find this behavior is already active even if no one deliberately set it up.

• Purge protection is a separate, optional safeguard, off by default. If it’s turned on for a vault, purging is blocked entirely until the retention period passes — not even an Owner-level account can bypass it. This is a deliberate design choice: it exists specifically so that a compromised or over-permissioned account can’t instantly and irreversibly destroy the vault’s contents.

The practical takeaway: “delete” is safe and reversible for up to 90 days. “Purge” is not reversible, ever. Treat purge as a deliberate, reviewed decision — not something that happens automatically as part of routine cleanup.

Which approach fits your situation?

Depending on the number of secrets you need to remove and how frequently cleanup is required, there are three common approaches:

• Delete a single secret (Azure Portal or one command)
  Best when you only need to remove one or two known secrets. This is the simplest and safest option, with minimal risk of accidentally deleting the wrong secret.

• Scripted bulk delete (Azure PowerShell or Azure CLI)
  Ideal for cleaning up tens or hundreds of secrets on demand. By combining a simple loop with name-based filters (such as a prefix), you can safely remove groups of related secrets. This approach requires more care to ensure only the intended secrets are deleted.

• Automated Azure Function
  The best choice for enterprise environments where cleanup needs to be repeatable, scheduled, or triggered automatically. An Azure Function can filter secrets by prefix, process large batches, generate audit logs, and provide a controlled, scalable solution for ongoing secret lifecycle management. While it requires more initial setup, it offers the highest level of automation, consistency, and governance.

The rest of this article is written for the developer or operator running these commands. If you’re a business stakeholder, the concepts covered so far should help you understand why secret cleanup matters and which approach best fits your organization’s needs. The sections that follow demonstrate how to implement each approach using Azure PowerShell, Azure CLI, and an automated Azure Function.

Prerequisites

• Azure PowerShell (the Az module) or the Azure CLI, installed and signed in (Connect-AzAccount or az login).

• An RBAC role or access policy on the vault that grants secret delete permission (and purge, if you’ll be permanently removing anything) — for example, the built-in Key Vault Secrets Officer role for delete, and Key Vault Purge Operator if purge access needs to be granted separately.

• Throughout the examples below, the placeholder vault name contoso-hr-kv is used — replace it with your own.

cripted bulk delete (Azure PowerShell or Azure CLI)

Delete a single secret (soft-delete)

The simplest case: remove one secret you know by name. This operation performs a soft delete. The secret moves to the deleted state and remains recoverable for the vault’s configured retention period (typically up to 90 days), unless it is later purged.

PowerShell:

$kvName = "contoso-hr-kv"
$secName = "employee-100234-key"
Remove-AzKeyVaultSecret -VaultName $kvName -Name $secName -Force

Azure CLI:

az keyvault secret delete --vault-name contoso-hr-kv --name employee-100000-key

A note on the CLI command above: Note: At the time of writing, Microsoft Learn marks az keyvault secret delete as Deprecated in the Azure CLI reference, but the command continues to function and performs a soft delete as expected. Microsoft hasn't announced a removal date, but because the command is deprecated, it's a good idea to check the latest Azure CLI documentation before building long-lived automation around it.

Portal (point-and-click equivalent): Open the vault → Objects → Secrets → select the secret → Delete.

Bulk delete: Soft-delete every matching secret

Deleting a single secret is straightforward. The challenge comes when you need to remove dozens, hundreds, or even thousands of secrets — for example, after decommissioning an application, cleaning up employee-specific mappings, or removing secrets accidentally created during a failed deployment.

Get-AzKeyVaultSecret retrieves the active secrets in a Key Vault. By piping the results to Remove-AzKeyVaultSecret, you can soft-delete every matching secret in a single operation.

$kvName = "contoso-hr-kv" 
Get-AzKeyVaultSecret -VaultName $kvName | ForEach-Object { 
Write-Host "Deleting $($_.Name)" 
Remove-AzKeyVaultSecret -VaultName $kvName -Name $_.Name -Force 
}

If you’re deleting a large number of secrets, it’s helpful to log your progress. The following example numbers each deletion, making it easier to monitor the operation or review execution logs later.

$kv = "contoso-hr-kv" 
$i = 0 

Get-AzKeyVaultSecret -VaultName $kv | ForEach-Object 
{ 
  $i++ $name = $_.Name 
  Remove-AzKeyVaultSecret -VaultName $kv -Name $name -Force 
  Write-Host ("[{0}] Deleted (soft): {1}" -f $i, $name) 
}

Always filter before bulk deletion

An unfiltered bulk delete against a shared production vault is risky. Enterprise Key Vaults often contain secrets used by multiple applications, environments, or business units. Accidentally deleting secrets that are still in use can lead to application failures or service outages.

Whenever possible, scope the operation to a specific naming convention or prefix. For example, if employee mapping secrets all begin with employee-, delete only those secrets.

Get-AzKeyVaultSecret -VaultName $kv | Where-Object 
{ 
  $_.Name -like "employee-*" } | 
ForEach-Object 
{ 
  Remove-AzKeyVaultSecret -VaultName $kv -Name $_.Name -Force 
}

Using a consistent naming convention for secrets isn’t just good housekeeping — it enables safe, targeted automation. Prefixes such as employee-, sales-, finance-, or ml- allow administrators to confidently clean up only the intended group of secrets.

Azure CLI equivalent

If your organization standardizes on the Azure CLI, the following Bash example performs the same operation. It lists all secrets, filters those beginning with employee-, and soft-deletes each one.

KV="contoso-hr-kv" 
for name in $(az keyvault secret list --vault-name "$KV" --query "[].name" -o tsv); do 
  if [[ "$name" == employee-* ]]; then 
    echo "Deleting $name" 
    az keyvault secret delete \ 
    --vault-name "$KV" \ 
    --name "$name" fi done

Best practice: Before running any bulk delete in a production environment, first execute the listing command by itself and verify the names that match your filter. A quick review can prevent accidental deletion of active secrets.

Purging: permanent removal

Soft-deleted secrets still exist in the vault, in a recoverable state, and they still block reuse of that name. To actually reclaim the name and remove them for good, you purge them — and this step needs the purge permission granted separately from delete.

Purge a single deleted secret

PowerShell:

$kvName = "contoso-hr-kv"
$secName = "sales-100000-key"
Remove-AzKeyVaultSecret -VaultName $kvName -Name $secName -InRemovedState -Force

Azure CLI:

az keyvault secret purge --vault-name contoso-hr-kv --name sales-100000-key

Purge all deleted secrets

$kvName = "contoso-hr-kv"

Get-AzKeyVaultSecret -VaultName $kvName -InRemovedState | ForEach-Object {
    Write-Host "Purging deleted secret $($_.Name)"
    Remove-AzKeyVaultSecret -VaultName $kvName -Name $_.Name -InRemovedState -Force
}

Azure CLI, using the recoveryId from each deleted secret:

KV="contoso-hr-kv"
az keyvault secret list-deleted --vault-name "$KV" --query "[].id" -o tsv | while read -r id; do
    echo "Purging $id"
    az keyvault secret purge --id "$id"
done

Purge is irreversible. Once purged, a secret cannot be recovered under any circumstances. If purge protection is enabled on the vault, the purge command will fail — by design — until the retention window (up to 90 days) has fully elapsed. Treat purge as a deliberate, reviewed action, never something bundled automatically into a routine delete script.

Verify the results

Always confirm state before and after a cleanup pass.

List the current (active) secrets:

Get-AzKeyVaultSecret -VaultName $kvName

List the soft-deleted secrets that are still recoverable:

Get-AzKeyVaultSecret -VaultName $kvName -InRemovedState

Azure CLI equivalents:

az keyvault secret list --vault-name contoso-hr-kv
az keyvault secret list-deleted --vault-name contoso-hr-kv

Automated Azure Function

Automated cleanup with an Azure Function

For repeatable, scheduled, or on-demand cleanup — for example, purging only per-employee HR secrets once a quarter — an HTTP-triggered Python Azure Function is a good fit. It authenticates with a managed identity (no credentials in code), filters by a name prefix so it can never touch secrets outside its intended scope, deletes in one pass, optionally purges in a second pass, and returns a JSON summary you can review or log.

import json
import time
import traceback

import azure.functions as func
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
from azure.core.exceptions import HttpResponseError, ResourceNotFoundError

# Replace with your Azure Key Vault URL
VAULT_URL = "https://contoso-kv.vault.azure.net/"

# Default settings
DEFAULT_PREFIX = "integration-"   # Only secrets starting with this prefix are processed
DEFAULT_LIMIT = 0                 # 0 = No limit
DEFAULT_PURGE = True              # Purge after soft-delete

# Initialize clients once per Function worker
credential = DefaultAzureCredential()
client = SecretClient(
    vault_url=VAULT_URL,
    credential=credential
)


def parse_bool(value, default):
    """Convert query/body parameter to boolean."""
    if value is None:
        return default
    return str(value).strip().lower() in ("1", "true", "yes", "y")


def main(req: func.HttpRequest) -> func.HttpResponse:
    start_time = time.time()

    try:
        body = req.get_json() if req.method == "POST" else {}
    except Exception:
        body = {}

    prefix = body.get("prefix") or req.params.get("prefix") or DEFAULT_PREFIX
    purge = parse_bool(
        body.get("purge") or req.params.get("purge"),
        DEFAULT_PURGE
    )

    try:
        limit = int(
            body.get("limit")
            or req.params.get("limit")
            or DEFAULT_LIMIT
        )
    except Exception:
        limit = DEFAULT_LIMIT

    scanned = 0
    matched = 0
    deleted = 0
    purged = 0
    errors = []

    print(f"Starting cleanup")
    print(f"Prefix : {prefix}")
    print(f"Purge  : {purge}")
    print(f"Limit  : {'No Limit' if limit == 0 else limit}")

    # ------------------------------------------------------------------
    # Pass 1 - Soft Delete
    # ------------------------------------------------------------------

    try:
        for props in client.list_properties_of_secrets():

            scanned += 1
            secret_name = props.name

            if not secret_name.startswith(prefix):
                continue

            matched += 1

            try:
                print(f"Deleting {secret_name}")

                poller = client.begin_delete_secret(secret_name)
                poller.wait()

                deleted += 1

            except ResourceNotFoundError:
                pass

            except HttpResponseError as ex:
                errors.append({
                    "name": secret_name,
                    "stage": "delete",
                    "status": getattr(ex, "status_code", None)
                })

            except Exception as ex:
                errors.append({
                    "name": secret_name,
                    "stage": "delete",
                    "error": str(ex)
                })

            if limit and deleted >= limit:
                break

    except Exception as ex:

        traceback.print_exc()

        return func.HttpResponse(
            json.dumps({
                "error": f"Delete failed: {str(ex)}"
            }),
            status_code=500,
            mimetype="application/json"
        )

    # ------------------------------------------------------------------
    # Pass 2 - Purge
    # ------------------------------------------------------------------

    if purge:

        try:

            for deleted_secret in client.list_deleted_secrets():

                secret_name = deleted_secret.name

                if not secret_name.startswith(prefix):
                    continue

                try:
                    print(f"Purging {secret_name}")

                    client.purge_deleted_secret(secret_name)

                    purged += 1

                except HttpResponseError as ex:

                    errors.append({
                        "name": secret_name,
                        "stage": "purge",
                        "status": getattr(ex, "status_code", None)
                    })

                except Exception as ex:

                    errors.append({
                        "name": secret_name,
                        "stage": "purge",
                        "error": str(ex)
                    })

                if limit and purged >= limit:
                    break

        except Exception as ex:

            traceback.print_exc()

            return func.HttpResponse(
                json.dumps({
                    "error": f"Purge failed: {str(ex)}"
                }),
                status_code=500,
                mimetype="application/json"
            )

    elapsed = round(time.time() - start_time, 2)

    response = {
        "vault": VAULT_URL,
        "prefix": prefix,
        "purge": purge,
        "scanned": scanned,
        "matched": matched,
        "deleted": deleted,
        "purged": purged,
        "elapsed_seconds": elapsed,
        "errors": errors[:50]
    }

    return func.HttpResponse(
        json.dumps(response, indent=2),
        status_code=200,
        mimetype="application/json"
    )

Example secret names

integration-salesforce-api
integration-workday-token
integration-servicenow-oauth
integration-sap-connection
integration-jira-webhook
integration-slack-bot

In the above code, change the secret name to the one that you have created with specific prefix.

How to call it

curl "https://<your-function-app>.azurewebsites.net/api/cleanup?prefix=integration-&limit=25&purge=true"

# Delete + purge only the first 25 HR secrets (a safe, incremental run)
curl "https://<your-func-app>.azurewebsites.net/api/cleanup?prefix=employee-&limit=25&purge=true"

# Soft-delete only, leave everything recoverable
curl "https://<your-func-app>.azurewebsites.net/api/cleanup?prefix=token-&purge=false"

The prefix filter, the optional per-run limit, and the two-pass delete-then-purge design exist specifically so you can dry-run against a small, named batch first, review the JSON response, and only widen the scope once you trust the result.

Best practices and safety checklist

1. List before you delete — verify names and counts first, every time.

2. Filter by prefix or pattern — never run an unqualified “delete everything” against a shared vault.

3. Soft-delete first, review, then purge as a separate, deliberate step — don’t chain them automatically by default.

4. Use a per-run limit on large vaults, so you can validate results incrementally rather than committing to a single massive run.

5. Understand purge protection — if it’s enabled on a vault, purge is blocked until the retention window ends, with no override.

6. Prefer a managed identity (DefaultAzureCredential) over stored credentials for any automation that runs unattended.

7. Keep the audit log on — Key Vault logs every delete and purge; confirm diagnostic logging is enabled if compliance requires an audit trail.

Bottom line

There’s no native “bulk delete secrets” button in Azure Key Vault. A short PowerShell or CLI loop covers on-demand cleanup for a handful of secrets; a prefix-scoped Azure Function turns it into a safe, repeatable, auditable operation for ongoing hygiene. Choose the level of automation that matches the size and sensitivity of the vault — and treat purge, specifically, as a decision that gets reviewed, not automated by default.

References

• Azure Key Vault soft-delete overview (mandatory since February 2025, retention periods) — https://learn.microsoft.com/en-us/azure/key-vault/general/soft-delete-overview

• Azure Key Vault recovery overview (soft-delete vs. purge protection) — https://learn.microsoft.com/en-us/azure/key-vault/general/key-vault-recovery

• Remove-AzKeyVaultSecret (PowerShell, including -InRemovedState) — https://learn.microsoft.com/en-us/powershell/module/az.keyvault/remove-azkeyvaultsecret

• az keyvault secret command reference (including the delete deprecation notice and purge/list-deleted) — https://learn.microsoft.com/en-us/cli/azure/keyvault/secret

• Azure Key Vault Secrets client library for Python (begin_delete_secret, purge_deleted_secret) — https://learn.microsoft.com/en-us/python/api/overview/azure/keyvault-secrets-readme

  ---

  #azurekeyvault #azurefunction #bulkdelete_keyvault_secrets #azurekeyvaultsecrets #purge_and_soft_delete_secrets

The Problem: Your Business Data and Your AI Platform Don’t Live in the Same Place

Modern enterprises rarely keep all of their data on a single platform. Customer information, financial records, invoices, contracts, operational reports, IoT data, and application logs are often generated by dozens of different business systems and stored in Azure services such as Azure Blob Storage or Azure Data Lake Storage. At the same time, organizations increasingly rely on Snowflake to power analytics, AI applications, reporting, and enterprise data sharing.

This creates an important business challenge. The data that drives critical business decisions is stored in one environment, while the platform responsible for analytics and AI operates in another. Moving information between these environments must be secure, reliable, and scalable. As organizations grow, manually building and maintaining separate integration pipelines for every department, application, or new data source quickly becomes expensive, difficult to govern, and operationally complex.

The challenge extends beyond simply loading data into Snowflake. Many enterprise workflows also require Snowflake to communicate with Azure services — for example, triggering Azure Functions to automate business processes, calling APIs exposed through Azure API Management, integrating with ERP or CRM systems, or writing results back to Azure-hosted applications. These interactions are often part of mission-critical business operations rather than simple data movement.

For industries such as banking, healthcare, insurance, manufacturing, and government, the question is no longer just “Can these platforms communicate?” The real question is “Can they communicate in a way that satisfies our security, compliance, governance, and regulatory requirements?” Whether data travels across the public internet or remains entirely within a private network can determine whether an architecture meets internal security standards and regulatory obligations.

The Need: A Secure, Enterprise-Grade Connection Between Snowflake and Azure

As organizations scale their data and AI initiatives, connecting Snowflake with Azure becomes more than a technical requirement — it becomes a governance and security priority. While there are several ways to establish connectivity, enterprise organizations need a solution that is secure, easy to manage, and aligned with corporate compliance policies.

Traditional approaches often rely on shared credentials or access tokens to connect cloud storage with analytics platforms. Although these methods may work for small projects or proof-of-concepts, they introduce operational overhead and security risks as the number of applications, teams, and datasets grows. Credentials expire, require regular rotation, and increase the likelihood of service interruptions or human error if not managed carefully.

From a business perspective, security teams prefer to grant access based on verified organizational identities rather than distributing shared secrets. This allows access to be governed through Azure’s identity and access management policies, providing clear ownership, centralized administration, and a complete audit trail of who was granted access, when it was approved, and what resources can be accessed.

For organizations operating in highly regulated industries such as banking, healthcare, insurance, telecommunications, and government, the requirements extend even further. Sensitive business data may be required to remain within private network boundaries, ensuring that communication between Snowflake and Azure never traverses the public internet. These controls help organizations satisfy internal security standards, customer commitments, and regulatory obligations while reducing their overall security exposure.

To address these enterprise requirements, Snowflake provides Storage Integrations, which enable secure, identity-based access to Azure Storage without embedding long-lived credentials in application configurations. For organizations with stricter networking and compliance requirements, Azure Private Link enables private connectivity between Snowflake and Azure services, helping ensure that data traffic remains on Microsoft’s private backbone instead of the public internet.

The Solution: two connectivity models, chosen by your compliance bar

Snowflake gives you two ways to connect to Azure Blob Storage, and the right one depends on how sensitive the data is and what your security team requires:

1. Storage Integration over the public network — Snowflake authenticates to your storage account as a managed Azure AD service principal (no keys, no tokens in Snowflake), but the actual data transfer still traverses the public internet (over HTTPS/TLS). This is the default, and it’s what the vast majority of Snowflake-Azure pipelines use.

2. Storage Integration with Private Link — the same service-principal-based authentication, but the network path between Snowflake’s VNet and your storage account’s private endpoint never leaves Microsoft’s private backbone. This requires Snowflake Business Critical Edition or higher, costs more (you pay per private connectivity endpoint plus data processed), and takes noticeably more setup — but it’s the right call when a compliance requirement says “no public internet.”

Below is the step-by-step for both, starting with the one nearly everyone needs first.

Part A: Connect Snowflake to Azure Blob Storage over the public network (Storage Integration)

This is the foundational setup — the same pattern used to build the external stage in our SharePoint → Snowflake pipeline article.

Step 1: Create a Storage Integration in Snowflake

CREATE STORAGE INTEGRATION azure_int
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'AZURE'
  ENABLED = TRUE
  AZURE_TENANT_ID = '<your_org_tenant_id>'
  STORAGE_ALLOWED_LOCATIONS = ('azure://<account>.blob.core.windows.net/<container>/<path>/');

A storage integration is a Snowflake object that stores a generated Azure AD identity for your external storage, along with an explicit allow-list of locations it’s permitted to touch. It’s the mechanism that lets you avoid supplying credentials when creating stages or loading/unloading data — the credential lives in Azure identity management, not in a Snowflake stage definition.

<your_tenant_id> is your Microsoft Entra ID (formerly Azure AD) tenant ID, found under Microsoft Entra ID → Overview → Properties → Tenant ID. A single storage integration authenticates to exactly one tenant, so every location in STORAGE_ALLOWED_LOCATIONS must belong to that same tenant.

Step 2: Retrieve the consent URL and service principal name

DESC STORAGE INTEGRATION azure_int;

Note two values from the output:

AZURE_CONSENT_URL — navigate to this URL in a browser and click Accept. This grants the Azure AD application that Snowflake created for your account a permission to request an access token on resources inside your tenant (it does not, by itself, grant access to any data — that's the next step).

AZURE_MULTI_TENANT_APP_NAME — the name of the Snowflake service principal that Snowflake created, formatted as _. You'll search for the part before the underscore in the next step.

Step 3: Grant the Snowflake service principal access in the Azure Portal

1. Go to Azure Portal → Storage Accounts → your storage account.

2. Click Access Control (IAM) → Add role assignment.

3. Assign one of:

• Storage Blob Data Reader — read-only access, for loading data into Snowflake.

• Storage Blob Data Contributor — read + write access, needed if you also want Snowflake to unload data or run REMOVE against files in the container.

4. Under Members, search for the Snowflake service principal using the string before the underscore in AZURE_MULTI_TENANT_APP_NAME.

5. Click Review + assign.

It’s normal — not a sign something broke — for the service principal to take up to an hour (occasionally longer) to appear as searchable in the Azure Portal after you accept the consent URL, and for the role assignment itself to take a few minutes to propagate once saved.

Step 4: Back on Snowflake - Create the external stage

LIST @my_azure_stage;

If the role assignment has propagated and the consent has been accepted, this returns the files sitting in that container/path of your Azure storage. If it comes back empty or errors out, the most common causes are: the role assignment hasn’t propagated yet, the consent URL wasn’t accepted, or the URL in the stage doesn't exactly match a location listed in STORAGE_ALLOWED_LOCATIONS.

SYSTEM$VALIDATE_STORAGE_INTEGRATION is also worth knowing about — it runs a permissions check against the integration and a specific file path and reports back what succeeded or failed, which is faster to interpret than a generic LIST error while you're still debugging the setup.

Part B: Private connectivity between Snowflake and an Azure Blob external stage

Everything in Part A still applies — you still create a storage integration and an external stage. Private connectivity adds a private network path underneath it, so traffic never crosses the public internet. This feature requires Snowflake Business Critical Edition or higher; on lower editions, USE_PRIVATELINK_ENDPOINT isn't available as of july 2026, and you'll need to contact Snowflake to discuss an edition upgrade before proceeding.

Step 1: Provision the private endpoint for Blob storage in Snowflake

USE ROLE ACCOUNTADMIN;

SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT(
  '/subscriptions/<subscription_id>/resourceGroups/<rg_name>/providers/Microsoft.Storage/storageAccounts/<storage_account>',
  '<storage_account>.blob.core.windows.net',
  'blob'
);

This provisions a private endpoint inside Snowflake’s own VNet that points at your storage account. It typically takes a few minutes to complete, since Snowflake is calling Azure’s own APIs behind the scenes to create the endpoint.

Once it’s provisioned, go to Azure Portal → your storage account → Networking → Private endpoint connections, and approve the pending connection request from Snowflake.

Step 2: Check endpoint status until it’s approved

SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO();

Poll this until the relevant entry shows "status": "APPROVED". You can proceed with the next steps while waiting, but the private path won't actually carry traffic until approval goes through on the Azure side.

Step 3: Create the storage integration with private connectivity enabled

CREATE OR REPLACE STORAGE INTEGRATION my_azure_private_int
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'AZURE'
  AZURE_TENANT_ID = '<tenant_id>'
  STORAGE_ALLOWED_LOCATIONS = ('azure://<account>.blob.core.windows.net/<container>/<path>/')
  USE_PRIVATELINK_ENDPOINT = TRUE
  ENABLED = TRUE;

USE_PRIVATELINK_ENDPOINT = TRUE is the property that tells this integration to route through the private endpoint rather than the public network. If you also need public-network access to the same storage account for a different use case, create a second, separate storage integration with USE_PRIVATELINK_ENDPOINT = FALSE — you can run both side by side.

Step 4: Get the consent URL and service principal name

DESC STORAGE INTEGRATION my_azure_private_int;

Same as Part A — note AZURE_CONSENT_URL and AZURE_MULTI_TENANT_APP_NAME.

Step 5: Create the external stage

CREATE OR REPLACE STAGE my_private_stage
  URL = 'azure://<account>.blob.core.windows.net/<container>/<path>/'
  STORAGE_INTEGRATION = my_azure_private_int;

A stage that references a storage integration with USE_PRIVATELINK_ENDPOINT = TRUE automatically inherits that private endpoint configuration — you don't (and can't) set the property again on the stage itself.

Step 6: Test the connection

LIST @my_private_stage;

Step 7: Finish the Azure-side approvals

1. Approve the private endpoint — Azure Portal → Storage Account → Networking → Private endpoint connections → approve the pending Snowflake endpoint (same as Step 1, confirm it’s approved).

2. Accept the consent URL — navigate to AZURE_CONSENT_URL in a browser and click Accept.

3. Grant the role to the Snowflake service principal — Storage Account → Access Control (IAM) → Add role assignment → assign Storage Blob Data Reader (or Storage Blob Data Contributor) to the Snowflake app, searching by the string before the underscore in AZURE_MULTI_TENANT_APP_NAME.

A couple of platform-level limits worth knowing before you plan multiple private endpoints: Snowflake currently caps outbound private connectivity at five private endpoints per account, and you can’t have more than one endpoint pointed at the same Azure subresource — plan your endpoint usage across teams accordingly, and deprovision endpoints you’re no longer using so they don’t count against the limit.

Which Approach Should You Choose?

The right approach depends on your organization’s security requirements, compliance obligations, and business priorities.

For most organizations, Snowflake Storage Integration is the recommended choice. It provides a secure, identity-based connection between Snowflake and Azure Storage without relying on shared credentials or manually managed access keys. It’s simple to set up, cost-effective, and suitable for the majority of analytics, reporting, data engineering, and AI workloads.

If your organization operates in industries such as banking, healthcare, insurance, government, or any environment with strict security and regulatory requirements, Azure Private Link may be the better option. It keeps communication between Snowflake and Azure on Microsoft’s private network instead of the public internet, helping meet compliance requirements and reducing network exposure. The trade-off is additional configuration, the need for Snowflake Business Critical Edition, and extra networking costs.

The good news is that choosing one approach over the other does not change how you build your data pipelines. Whether you use a standard Storage Integration or Azure Private Link, you continue to work with the same Snowflake features — such as external stages, COPY INTO, Snowpipe, and stored procedures. Since the connectivity layer is independent of the data processing layer, organizations can start with a standard Storage Integration and adopt Azure Private Link later if business or compliance requirements evolve, without redesigning their existing pipelines.

References

• Create storage integration for Azure — https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration

• Create an Azure stage — https://docs.snowflake.com/en/user-guide/data-load-azure-create-stage

• Configure an Azure container for loading data (storage integration setup, role assignment) — https://docs.snowflake.com/en/user-guide/data-load-azure-config

• Private connectivity to external stages and Snowpipe automation for Microsoft Azure — https://docs.snowflake.com/en/user-guide/data-load-azure-private

• Private connectivity for outbound network traffic (edition requirement, endpoint limits) — https://docs.snowflake.com/en/user-guide/private-connectivity-outbound

• SYSTEM$PROVISION_PRIVATELINK_ENDPOINT — https://docs.snowflake.com/en/sql-reference/functions/system_provision_privatelink_endpoint

• Snowflake editions (Business Critical Edition) — https://docs.snowflake.com/en/user-guide/intro-editions

  ---

  What’s Next?

  Connecting Snowflake with Azure opens the door to much more than securely moving data. It also enables organizations to automate business processes that extend beyond the Snowflake platform.

  One common challenge many teams encounter is notifications. Snowflake provides built-in email capabilities, but they are designed to send emails only to verified users within the same Snowflake account. This works well for internal administrators and users, but it isn’t sufficient for many real-world business scenarios.

  For example, an organization may need to notify a supplier when a contract is nearing expiration, inform a customer that a document has been processed, alert an external auditor about a compliance report, or send an email to a business partner when a data quality issue is detected. Since these recipients are typically outside the Snowflake account, Snowflake’s native email functionality cannot send notifications to them directly.

  This is where the Snowflake–Azure integration becomes even more valuable. By securely connecting Snowflake with Azure services, you can trigger external business workflows directly from your data pipelines. A Snowflake Task or stored procedure can invoke an Azure Function App to send emails using Microsoft Graph or SendGrid, or trigger a Power Automate flow to deliver notifications through Outlook, Microsoft Teams, Slack, or other enterprise applications.

  In the next article, we’ll build this solution step by step — showing how Snowflake can securely communicate with Azure to send notifications to anyone, whether they’re an internal employee, a customer, a vendor, or another external stakeholder.

  #snowflake #snowflake_azure_connectivity #snowflake_private_link #snowflake_private_connectivity #snowflake_multi_tenant_app_name #storage_integration #external_stage_in_snowflake

The Problem: valuable business knowledge is trapped in SharePoint

Most enterprises store their most business-critical documents in SharePoint — invoices, purchase orders, vendor contracts, legal agreements, HR records, procurement files, financial reports. These documents are rich in business context, but they sit in folders where analytics systems and AI agents can’t easily reason over them. Someone can open a contract and read it, but nothing is answering “which vendor contracts expire next quarter and which of them conflict with the payment terms on file” across ten thousand documents at once.

Microsoft 365 Copilot helps at the individual-productivity layer — Copilot in OneDrive can compare a handful of selected files side by side, for instance. But comparing a few documents interactively is a different problem from what enterprises actually need at scale: matching thousands of invoices against purchase orders, extracting structured fields into governed tables, building a reusable semantic search index, and joining document-derived facts with ERP or finance data. That’s an information-architecture problem, and it needs a platform, not a copilot.

The Need: get documents into Snowflake without giving up control of them

Once the goal becomes “make SharePoint content queryable and AI-ready inside Snowflake,” a new set of requirements shows up that a simple file copy doesn’t satisfy:

• The content is sensitive. Contracts, payroll data, and vendor agreements can’t sit in a landing zone in plain text, even temporarily.

• The encryption key can’t live next to the encrypted files. If the same breach that exposes the storage account also exposes the key, encryption bought nothing.

• Not every file is analytics-ready as-is. ZIP archives need to be unpacked, very long PDFs need to be split so document-AI functions can process them efficiently, and unsupported formats need to be filtered out before they clog the pipeline.

• The ingestion process runs unattended. Nobody wants to be the person who has to remember to run a script every morning; it needs to authenticate as an application, not as a person.

• Processing needs to be idempotent and auditable. The same file shouldn’t get parsed twice, and someone should be able to answer “was this document processed, and when?” without digging through logs.

• The landing zone shouldn’t lock you into one team’s toolchain. Other systems — Databricks, Fabric, Synapse — may want to read from the same encrypted zone later.

That’s the need this article answers: an ingestion and processing pipeline that treats “get the file from SharePoint” and “make the file’s content queryable in Snowflake” as two cleanly separated, independently auditable problems, with encryption and key separation enforced at every hop.

The Solution: SharePoint → encrypted Azure Blob → Snowflake stored procedure

The solution has three layers, each with one job:

1. SharePoint access — a Python ingestion pipeline authenticates to Microsoft Graph as a Service Principal, recursively walks the target document library, validates and pre-processes files (ZIP extraction, long-PDF splitting), and encrypts each one with AES-256-CBC using a key pulled from Azure Key Vault — the same pattern Part 1 walked through in depth.

2. Transport — the encrypted files land in Azure Blob Storage, a neutral landing zone that isn’t tied to any single downstream consumer.

3. Decrypt-and-understand — a Snowflake stored procedure decrypts each file only in memory or files are temporarily stored in internal stage, runs Snowflake Cortex’s AI_PARSE_DOCUMENT and AI_EXTRACT against the plaintext, logs/stores the result in tables, and immediately removes the decrypted copy from the stage. A Snowflake Task runs this on a schedule so the whole thing operates hands-off.

One quick note before we get into it: Snowflake now also ships a generally available Openflow Connector for SharePoint — a managed, low-code path for ingesting SharePoint files and permissions directly into Snowflake. We haven’t covered it yet in this series, and it deserves its own detailed treatment (setup, permission model, and — importantly — how it compares to the custom pipeline below on cost, control, and flexibility). That comparison is coming in the next article. For now, this piece focuses on the custom encrypted pipeline, which remains the right choice whenever you need client-side encryption before data leaves your boundary, custom file handling, or a cloud-agnostic landing zone that other systems can also consume.

Architecture at a glance

SharePoint Document Library (invoices, POs, contracts, HR docs)
        │
        ▼
Microsoft Graph API  (app-only auth via Azure Service Principal)
        │
        ▼
Python Ingestion Pipeline
  • recursive traversal + pagination
  • retry / token-refresh logic
  • file-type validation
  • optional ZIP extraction, long-PDF splitting
  • AES-256-CBC encryption (key pulled from Azure Key Vault)
        │
        ▼
Azure Blob Storage  (encrypted landing zone — AWS S3 / GCP GCS also supported)
        │
        ▼
Snowflake External Stage  (storage integration)
        │
        ▼
Snowflake Internal Stage + Stored Procedure
  • decrypt file with SnowflakeFile + AES key
  • write plaintext to a temporary internal stage
  • AI_PARSE_DOCUMENT / AI_EXTRACT
  • log to FILES_PROCESSED
  • REMOVE decrypted file from stage
        │
        ▼
Structured tables + Cortex Search index
        │
        ▼
Cortex Agent / Streamlit chatbot

This design keeps three concerns cleanly separated: SharePoint access (Graph API + Service Principal), transport and encryption (Python + Azure Blob), and decrypt-and-understand (Snowflake stored procedure + Cortex AI functions). Each layer can be swapped independently — a different cloud storage provider, a different ingestion runtime, a different AI extraction schema — without touching the others.

Solution -

Step 1: Register an Azure AD app for SharePoint access

Because the ingestion pipeline runs unattended (no interactive user logs in), it authenticates as a Service Principal using the OAuth2 client-credentials flow — tenant ID, client ID, and a client secret or certificate. This is the standard app-only pattern for backend access to Microsoft Graph.

In Microsoft Entra ID:

1. App registrations → New registration. Give it a clear name, e.g. sp-sharepoint-snowflake-ingestion_sps.

2. Record the Tenant ID, Client ID, and generate a Client secret.

3. Under API permissions, add these Microsoft Graph application permissions:

• Sites.Selected — scopes the app to specific SharePoint sites only, instead of tenant-wide read access

• Files.Read.All (or the narrower Files.SelectedOperations.Selected if you want per-file granularity)

• GroupMember.Read.All — only if you also need to resolve SharePoint group membership for ACLs

• User.ReadBasic.All — only if you need to resolve user display names/emails from IDs

4. Grant admin consent, then grant the app access to the specific SharePoint site you’re ingesting from (not the whole tenant) — this limits blast radius if the credential ever leaks.

Sites.Selected is worth calling out specifically: it's the right default for any service-to-service integration — grant the app exactly the sites it needs, nothing more.

Step 2: Store configuration and secrets properly

Everything that isn’t code goes into environment variables or a secrets manager — never hardcoded.

# Azure / Microsoft Graph
TENANT_ID=00000000-0000-0000-0000-000000000000
CLIENT_ID=11111111-1111-1111-1111-111111111111
CLIENT_SECRET=your-client-secret
GRAPH_URL=https://graph.microsoft.com/v1.0
SITE_ID=your-sharepoint-site-id
DRIVE_ID=your-drive-id
START_PATH=Shared Documents/Finance Documents

# Azure Blob
AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net
AZURE_CONTAINER_AND_PREFIX=enterprise-documents/encrypted/sharepoint

# Azure Key Vault
AZURE_KEY_VAULT_NAME=kv-enterprise-docs
AZURE_AES_SECRET_NAME=SHAREPOINT_AES_KEY

# Processing
PDF_MAX_PAGES_PER_PART=100
MAX_ZIP_DEPTH=5

In production, don’t even keep CLIENT_SECRET and the blob connection string in a plain .env on disk — pull them from Key Vault at runtime using a managed identity, the same way the pipeline pulls the AES key (see Part 1 for the full Key Vault + RBAC walkthrough). The .env above is fine for local development only.

Step 3: The Python ingestion pipeline — SharePoint → encrypted Azure Blob

At a high level, the script:

1. Authenticates to Microsoft Graph using client-credentials flow, refreshing the token on 401 responses.

2. Recursively walks the configured SharePoint drive/folder, paginating through Graph API responses, with retry/backoff on transient failures.

3. Validates file types against an allow-list (PDF, DOCX, PPTX, images, HTML, TXT, EML, etc.), extracts ZIP archives, and optionally splits very long PDFs into smaller parts so downstream parsing stays efficient.

4. Fetches the AES-256 key from Azure Key Vault (never hardcoded) and encrypts each file with AES-256-CBC before it ever touches Blob Storage.

5. Uploads the encrypted bytes to Azure Blob, writes a small control/metadata file alongside it (source path, category, run ID), and appends to a local “already uploaded” log so re-runs don’t duplicate work.

import os, io, base64, time
from datetime import datetime, timezone
import requests
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
from azure.storage.blob import BlobServiceClient, ContentSettings
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

def get_access_token(tenant_id, client_id, client_secret):
    url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"
    data = {
        "grant_type": "client_credentials",
        "client_id": client_id,
        "client_secret": client_secret,
        "scope": "https://graph.microsoft.com/.default",
    }
    resp = requests.post(url, data=data, timeout=60)
    resp.raise_for_status()
    return resp.json()["access_token"]

def load_aes_key(vault_name: str, secret_name: str) -> bytes:
    vault_url = f"https://{vault_name}.vault.azure.net"
    credential = DefaultAzureCredential(exclude_interactive_browser_credential=True)
    client = SecretClient(vault_url=vault_url, credential=credential)
    encoded_key = client.get_secret(secret_name).value.strip()
    key = base64.b64decode(encoded_key, validate=True)
    if len(key) != 32:
        raise ValueError("AES key must be 32 bytes for AES-256.")
    return key

def aes_encrypt(file_bytes: bytes, key: bytes) -> bytes:
    iv = get_random_bytes(16)
    cipher = AES.new(key, AES.MODE_CBC, iv)
    pad_len = 16 - (len(file_bytes) % 16)
    padded = file_bytes + bytes([pad_len]) * pad_len
    return iv + cipher.encrypt(padded)  # prepend IV so the decrypt step can read it back out

def upload_encrypted_blob(blob_service: BlobServiceClient, container: str,
                           blob_path: str, encrypted_bytes: bytes) -> None:
    container_client = blob_service.get_container_client(container)
    container_client.upload_blob(
        name=blob_path,
        data=encrypted_bytes,
        overwrite=True,
        content_settings=ContentSettings(content_type="application/octet-stream"),
    )

def run_ingestion(config: dict) -> None:
    token = get_access_token(config["tenant_id"], config["client_id"], config["client_secret"])
    aes_key = load_aes_key(config["vault_name"], config["aes_secret_name"])
    blob_service = BlobServiceClient.from_connection_string(config["blob_connection_string"])

    for file_name, file_bytes, source_path in walk_sharepoint_drive(token, config):
        encrypted = aes_encrypt(file_bytes, aes_key)
        blob_path = f"{config['prefix']}/{datetime.now(timezone.utc):%Y/%m/%d}/{file_name}.enc"
        upload_encrypted_blob(blob_service, config["container"], blob_path, encrypted)
        print(f"Uploaded {source_path} -> {blob_path}")

walk_sharepoint_drive(...) is the recursive Graph-API traversal function — pagination via @odata.nextLink, a 401-triggered token refresh, and retry-with-backoff around each request. It's mechanical and mostly boilerplate, so it's omitted here for space, but the shape is: list children of DRIVE_ID/START_PATH, recurse into folders, download file content for anything matching the supported-extension allow-list, and yield (file_name, file_bytes, source_path) tuples.

Where does this script actually run?

You have two solid options, and the right one depends on how “always-on” you need ingestion to be:

• Local machine or company laptop, via VS Code or any IDE. Perfectly fine for development, backfills, and one-off runs. Point-and-click debugging, easy to inspect logs, no cloud deployment overhead. The downside is obvious: if the laptop is asleep, off, or on a different network, ingestion doesn’t run.

• Azure Function App (timer-triggered). The same script, deployed as a Python Azure Function, running serverless on a schedule (e.g., every 15–30 minutes) so new SharePoint files land in Blob without anyone’s laptop needing to be on. The Function App can use a managed identity to pull secrets from Key Vault at runtime instead of a client secret in local.settings.json, which is one credential fewer to rotate and protect.

Most teams prototype locally in VS Code first, confirm the encryption and upload logic behaves correctly against a test SharePoint site, and then redeploy the same code as an Azure Function once it’s stable. Nothing about the script itself changes between the two — only how and where it’s triggered.

Step 4: Snowflake External Stage — pointing at the Blob landing zone

Once encrypted files are landing in Blob, create a storage integration and an external stage in Snowflake so it can see that container:

CREATE STORAGE INTEGRATION my_azure_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'AZURE'
  ENABLED = TRUE
  AZURE_TENANT_ID = '<tenant-id>'
  STORAGE_ALLOWED_LOCATIONS = ('azure://<storage-account>.blob.core.windows.net/<container>/<prefix>/');

CREATE STAGE raw_encrypted_external_stage
  URL = 'azure://<storage-account>.blob.core.windows.net/<container>/<prefix>/'
  STORAGE_INTEGRATION = my_azure_integration;

DESC STORAGE INTEGRATION my_azure_integration returns a consent URL and an Azure AD application object ID — grant that identity Storage Blob Data Reader on the container so Snowflake can list and pull files without any long-lived key sitting in Snowflake itself.

From here, a scheduled COPY FILES (or a directory-table-driven process) moves new encrypted objects from the external stage into an internal stage(optional), where the decrypt-and-parse stored procedure picks them up. Keeping a Snowflake-managed internal stage as the working area — rather than decrypting directly against the external stage — keeps the blast radius of the decrypted plaintext limited to Snowflake's own encrypted storage.

CREATE STAGE IF NOT EXISTS raw_encrypted_stage
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE')
  DIRECTORY = (ENABLE = TRUE);

CREATE STAGE IF NOT EXISTS decrypted_stage
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE')
  DIRECTORY = (ENABLE = TRUE);

We can also directly decrypt files by placing files from external stage -> decryption SP -> Internal Stage.

Step 5: The core piece — decrypt, parse, log, and clean up in one stored procedure

This is the heart of the pipeline: a Python stored procedure that

1. reads the encrypted bytes off the internal/external stage(Azure blob encrypted files) using the SnowflakeFile class,

2. decrypts them with the AES key,

3. writes the plaintext to a temporary internal stage,

4. calls AI_PARSE_DOCUMENT (full text/layout) and AI_EXTRACT (structured fields) on the decrypted file,

5. logs the file as processed into a supporting table, and

6. deletes the decrypted file from the internal stage so nothing sensitive lingers.

First, the supporting tables:

CREATE TABLE IF NOT EXISTS files_processed (
    file_name        STRING,
    status           STRING,           -- 'parsed' | 'failed'
    error_message    STRING,
    processed_time   TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

CREATE TABLE IF NOT EXISTS parsed_results (
    file_name        STRING,
    parsed_content   VARIANT,   -- output of AI_PARSE_DOCUMENT
    extracted_fields VARIANT,   -- output of AI_EXTRACT
    processed_time   TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

Store the AES key as a Snowflake secret object bound to an external access integration, or fetch it through a secure UDF that calls Key Vault — either way, never inline the key value in the procedure body. The example below assumes a Snowflake secret named aes_key_secret holding the same base64-encoded 32-byte key that Key Vault stores.

CREATE OR REPLACE PROCEDURE decrypt_and_parse_sp(encrypted_file_path STRING)
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.11'
PACKAGES = ('snowflake-snowpark-python', 'pycryptodome')
HANDLER = 'run'
EXECUTE AS OWNER
AS
$$
import io
import base64
from Crypto.Cipher import AES
from snowflake.snowpark.files import SnowflakeFile


def get_aes_key(session) -> bytes:
    # Pulls the base64-encoded AES-256 key from a Snowflake SECRET object.
    # The secret itself is populated out-of-band from Azure Key Vault, so the
    # key value is never typed into SQL or checked into source control.
    row = session.sql(
        "SELECT SYSTEM$GET_PRIVATE_SECRET('aes_key_secret')"
    ).collect()
    return base64.b64decode(row[0][0])


def run(session, encrypted_file_path: str) -> str:
    file_name = encrypted_file_path.split('/')[-1]
    decrypted_name = file_name[:-4] if file_name.endswith('.enc') else file_name

    try:
        # 1. Read the encrypted bytes straight from the internal stage
        with SnowflakeFile.open(encrypted_file_path, 'rb') as f:
            raw = f.read()

        iv, ciphertext = raw[:16], raw[16:]
        key = get_aes_key(session)

        # 2. Decrypt (AES-256-CBC) and strip PKCS7 padding
        cipher = AES.new(key, AES.MODE_CBC, iv)
        padded = cipher.decrypt(ciphertext)
        plaintext = padded[:-padded[-1]]

        # 3. Write the plaintext to a temporary internal stage
        session.file.put_stream(
            io.BytesIO(plaintext),
            f"@decrypted_stage/{decrypted_name}",
            auto_compress=False,
            overwrite=True,
        )

        # 4. Parse with Cortex AI functions
        parsed = session.sql(f"""
            SELECT AI_PARSE_DOCUMENT(
                TO_FILE('@decrypted_stage', '{decrypted_name}'),
                {{'mode': 'LAYOUT'}}
            )
        """).collect()[0][0]

        extracted = session.sql(f"""
            SELECT AI_EXTRACT(
                file => TO_FILE('@decrypted_stage', '{decrypted_name}'),
                responseFormat => {{
                    'schema': {{
                        'type': 'object',
                        'properties': {{
                            'document_type': {{'description': 'What type of document is this (invoice, PO, contract, HR record)?', 'type': 'string'}},
                            'key_dates':     {{'description': 'Any important dates mentioned in the document', 'type': 'array'}},
                            'parties':       {{'description': 'Named organizations or individuals referenced', 'type': 'array'}}
                            'other parameters........ TCV, payment terms, bill to, ship to....etc'
                        }}
                    }}
                }}
            )
        """).collect()[0][0]

        # 5. Log the parsed content and mark the file as processed
        session.sql(
            "INSERT INTO parsed_results (file_name, parsed_content, extracted_fields) "
            "SELECT ?, PARSE_JSON(?), PARSE_JSON(?)",
            params=[decrypted_name, parsed, extracted],
        ).collect()

        session.sql(
            "INSERT INTO files_processed (file_name, status) VALUES (?, 'parsed')",
            params=[decrypted_name],
        ).collect()

        # 6. Cleanup — remove the decrypted plaintext and the original encrypted file
        session.sql(f"REMOVE @decrypted_stage/{decrypted_name}").collect()
        session.sql(f"REMOVE {encrypted_file_path}").collect()

        return f"Processed and cleaned up {decrypted_name}"

    except Exception as exc:
        session.sql(
            "INSERT INTO files_processed (file_name, status, error_message) VALUES (?, 'failed', ?)",
            params=[decrypted_name, str(exc)],
        ).collect()
        # Best effort cleanup even on failure, so plaintext never lingers
        session.sql(f"REMOVE @decrypted_stage/{decrypted_name}").collect()
        raise
$$;

A few things worth calling out about this procedure:

• SnowflakeFile.open() is the documented way to stream a file — of any size — out of a stage inside a Python handler, and it works for both internal and external stages. Reading it as bytes ('rb') keeps the AES decryption simple, since AES-CBC operates on raw bytes.

• session.file.put_stream() uploads the decrypted bytes directly to the internal stage from memory, without ever touching local disk inside the sandbox — the plaintext exists only as long as the procedure needs it.

• AI_PARSE_DOCUMENT returns the full text/layout of the document as JSON — this is what you want for building a Cortex Search index or doing general-purpose RAG over the document.

• AI_EXTRACT answers specific questions against a schema you define — this is what you want for pulling structured fields (invoice totals, contract parties, key dates) into governed tables. Use both together: parse for search, extract for structured analytics.

• REMOVE deletes the file from the stage immediately after processing, so the plaintext copy has the shortest possible lifetime. The original encrypted file is also removed once processing succeeds, so nothing double-processes on the next run.

• Wrap the whole thing in a try/except that still logs a 'failed' row and still attempts cleanup — you don't want a single malformed PDF to leave a decrypted file sitting in a stage indefinitely, or to silently break the pipeline for every file after it.

Step 6: Automate it with a Snowflake Task, every 1–2 hours or as per business requirements

The procedure above handles a single file. A thin driver procedure lists whatever hasn’t been processed yet and calls it for each new file, and a Task runs that driver on a schedule.

CREATE OR REPLACE PROCEDURE process_new_files_sp()
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.11'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'run'
AS
$$
def run(session):
    new_files = session.sql("""
        SELECT relative_path
        FROM DIRECTORY(@raw_encrypted_stage)
        WHERE relative_path NOT IN (
            SELECT file_name FROM files_processed WHERE status = 'parsed'
        )
    """).collect()

    processed = 0
    for row in new_files:
        path = f"@raw_encrypted_stage/{row['RELATIVE_PATH']}"
        session.call('decrypt_and_parse_sp', path)
        processed += 1

    return f"Processed {processed} new file(s)."
$$;

-- Runs every hour
CREATE OR REPLACE TASK decrypt_and_parse_task
  WAREHOUSE = my_wh
  SCHEDULE = '60 MINUTE'
AS
  CALL process_new_files_sp();

-- Or, if you'd rather run every 2 hours:
-- SCHEDULE = '120 MINUTE'
-- Or with cron syntax, e.g. on the hour, every 2 hours, UTC:
-- SCHEDULE = 'USING CRON 0 */2 * * * UTC'

-- Tasks are created in a SUSPENDED state — you must explicitly resume them
ALTER TASK decrypt_and_parse_task RESUME;

A couple of operational notes:

DIRECTORY(@raw_encrypted_stage) requires the stage to have been created with DIRECTORY = (ENABLE = TRUE), which gives you a queryable directory table of everything sitting in the stage — handy for exactly this "what's new" pattern. The SCHEDULE parameter accepts either a fixed interval ('60 MINUTE', '120 MINUTE') or a USING CRON expression if you need it to run at a specific time of day rather than a rolling interval. Every newly created task starts suspended — ALTER TASK ... RESUME is what actually turns it on. It's an easy step to forget and then wonder why nothing is happening. SUSPEND_TASK_AFTER_NUM_FAILURES is worth setting so a persistently broken file (or an expired credential) doesn't spin the task forever without anyone noticing — check INFORMATION_SCHEMA.TASK_HISTORY() periodically, or alert off it.

CREATE STORAGE INTEGRATION my_s3_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-role'
  STORAGE_ALLOWED_LOCATIONS = ('s3://my-bucket/encrypted-docs/');

CREATE STAGE raw_encrypted_external_stage
  URL = 's3://my-bucket/encrypted-docs/'
  STORAGE_INTEGRATION = my_s3_integration;

Google Cloud Storage:

CREATE STORAGE INTEGRATION my_gcs_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'GCS'
  ENABLED = TRUE
  STORAGE_ALLOWED_LOCATIONS = ('gcs://my-bucket/encrypted-docs/');

CREATE STAGE raw_encrypted_external_stage
  URL = 'gcs://my-bucket/encrypted-docs/'
  STORAGE_INTEGRATION = my_gcs_integration;

If you’re on AWS or GCP already, the AES key can live in AWS Secrets Manager or Google Secret Manager instead of Azure Key Vault — same principle (key separated from encrypted data, fetched only at decrypt time), different vendor.

Recap: what this pipeline gives you

• Encryption in transit and at rest from the moment a file leaves SharePoint until the instant before it’s parsed.

• Key separation — the AES key never lives next to the encrypted files, in Blob, in an external stage, or in source control.

• Least-lifetime plaintext — the decrypted file exists in an internal Snowflake stage only for the duration of parsing, and is removed immediately after (and even on failure).

• Auditability — files_processed gives you a record of every file, when it ran, and whether it succeeded, without re-processing anything twice.

• Automation — a Snowflake Task keeps the whole thing running hands-off every hour (or two), and the Python ingestion side can run from a developer’s laptop during testing or as an unattended Azure Function in production.

Portability — swap Azure Blob for S3 or GCS, or swap Key Vault for another secrets manager, without touching the Snowflake-side logic at all.

What’s next

• AI_PARSE_DOCUMENT and AI_EXTRACT hand back JSON — a VARIANT sitting in parsed_results. That's a good stopping point for this article, but it's not yet something a business user or a downstream table can consume directly, and there's a real amount of design work between "we have JSON" and "we have a governed, queryable dataset."

  In the next article, we’ll take a close look at the Openflow Connector for SharePoint — how to set it up, its permission model (Sites.Selected, document ACLs), and a head-to-head comparison against the custom pipeline built here: where the managed connector wins on speed of setup, and where the custom pipeline still wins on encryption control, custom file handling, and cross-platform reuse.

  After that, we’ll come back to this pipeline and go deep on turning that raw JSON into something an AI agent and a business user can both actually use:

  • Structured side (AI_EXTRACT output): landing the raw JSON in a staging table as-is, then a clean/cast step that pulls fields out of the VARIANT into proper typed columns (dates as DATE, amounts as NUMBER, not strings) in a target table — and why you want that staging layer instead of casting inline during extraction. From there, building views that join the extracted document fields against your existing business unit, customer, and contract-master tables, so "which vendor contracts expire next quarter" is a normal SQL query, not a document search.

  • Unstructured side (AI_PARSE_DOCUMENT output): why the parsed JSON needs to be flattened before it's usable, how to chunk the flattened text sensibly (page-aware, section-aware, or token-based chunking, and the tradeoffs between them), and how to build a Cortex Search service on top of the chunked content.

  • Putting it together: a full walkthrough of standing up a Cortex Agent that reasons across both the structured tables and the Cortex Search index in one conversation, plus the end-user interface on top of it — whether that’s a Streamlit-in-Snowflake app or an external Angular front end calling the agent through an API.

That’s the article where this pipeline stops being “documents in a stage” and starts being an actual AI application people can talk to.

#snowflake #dataengineering #sharepointtosnowflake #graphapi #AIAGENT #agentic_pipeline #sharepoint_to_snowflake #dataingestion

Streamlit in Snowflake’s Warehouse Runtime was a natural first step for building such interfaces (with dashboards on one page and an AI chat on another). However, Warehouse Runtime has key limitations: it runs all code with the app owner’s privileges (no restricted caller’s rights) and limits Python packages to Snowflake’s curated Conda channel. This makes it nearly impossible to enforce per-user row-level security (RLS) or to install the rich set of third-party libraries (Streamlit releases, AI/ML packages, Azure SDKs, etc.) modern applications need.

To address this, we migrate our Streamlit app to Snowpark Container Services (SPCS) — Snowflake’s container-based runtime for Streamlit (and native apps) — which supports custom Python environments and restricted caller’s rights.

This guide uses the latest Snowflake documentation (as of mid-2026) and flags any preview/region dependencies.

From Dashboards to AI Agents: The Enterprise Motivation

For years, BI tools (Power BI, Tableau, Qlik, Snowsight, etc.) dominated enterprise analytics. HR teams tracked attrition rates, background-check (BGV) compliance, hiring funnel metrics, etc., on dashboards. Finance teams reviewed revenue, PAT, cost breakdowns, and forecasts. These reports are often based on the very data already in Snowflake. If employees can query dashboards, why not ask an AI agent?

A modern enterprise AI assistant might combine traditional analytics and natural-language questioning. For example, imagine a Streamlit app with multiple tabs: one showing key performance indicator (KPI) visuals, and another hosting a Cortex AI agent for chat. An HR manager could open the app, authenticate via SSO, and ask “Show me attrition trends over the last year” or “How many candidates are pending background verification (BGV).” A finance user might ask about “Quarterly revenue by region” or “Top 10 customers by PAT” and get answers in chat form. Behind the scenes, these agents use Snowflake data and AI models (cortex agents), but the user experience is seamless. This is a powerful paradigm shift: from static dashboards to interactive AI-powered data assistants.

To build such apps, developers often start with Streamlit in Snowflake (easy deployment, direct Snowflake access, Snowsight integration). However, the Warehouse Runtime quickly becomes a hindrance for enterprise needs.

Limitations of Streamlit on Warehouse Runtime

Warehouse Runtime runs a new, isolated instance of the Streamlit app for each viewer.

This ensures fresh compute per user, but it has significant constraints:

• Owner’s Rights Only: Warehouse runtime always executes queries with the app owner’s privileges. There is no concept of restricted caller’s rights here. In practice, this means if Rakesh (an HR user with HR Role) and Harsh (a Finance user with Finance Role) use the app, all Snowflake queries run under the single “owner” role that deployed the app. HR User Rakesh with HR role will still able to see Finance Data, because the queries run under the OWNER’s role and not on CALLER’s role(HR role). The app cannot natively act on behalf of Rakesh or Harsh. In particular, if you have any Row Access Policies (RLS) that use CURRENT_ROLE(), they will see the owner’s role, not the viewer’s role. This breaks security: an HR user might see finance data and vice versa. (Snowflake docs explicitly warn that “Warehouse-runtime apps using CURRENT_ROLE() in row access policies will always return the app owner’s role, not the viewer’s role”.)

• Limited Packages: In Warehouse runtime, Python packages are limited to what Snowflake provides via a built-in Anaconda/Conda channel. You can only include dependencies by specifying an environment.yml that references the Snowflake Anaconda repository. Many popular packages (like recent Streamlit versions, Azure SDKs, LangChain, etc.) may not be available or are outdated. For example, Warehouse apps only support Streamlit versions up to ~1.22, whereas enterprise developers often want the latest 1.5x versions for new features. The environment.yml method is also clunky to update.

• No Persistent Server or Caching: Each viewer has a separate app instance. This increases resource usage and latency (each page view starts a new Python process). It also means caching (e.g. @st.cache) does not share data between users. The app can feel less responsive for multiple users.

• Network & API Access: By default, Warehouse runtime has no internet access (except Snowflake). You can’t call external APIs or pip-install at runtime. Even calling an external API requires special support (not natively possible for a warehouse-streamlit app). This severely limits using AI APIs (OpenAI, Azure, etc.) or pulling data from external services.

In summary, Warehouse Runtime is great for rapid prototyping and simple dashboards, but it cannot meet enterprise requirements for security, extensibility, and performance.

Why Row Access Policies Break Under Owner’s Rights

A concrete issue is row-level security (RLS) based on user roles. Suppose we have a table SALES and a row access policy that uses CURRENT_ROLE() to filter rows to only those in the user’s business unit. For example:

CREATE OR REPLACE ROW ACCESS POLICY rls_sales AS ( (CURRENT_ROLE() = 'FINANCE' AND department = 'Finance') OR (CURRENT_ROLE() = 'HR' AND department = 'HR') ); 

ALTER TABLE SALES ADD ROW ACCESS POLICY rls_sales ON (department);

If the Streamlit app runs under owner’s rights with a fixed role (say the “app_service_role”), then CURRENT_ROLE() will always be app_service_role, no matter who is logged in. Thus an HR user or Finance user sees the same filtered data – likely none or all, depending on the policy logic. The user’s own Snowflake role is ignored. This is confirmed in Snowflake’s security docs: “Warehouse-runtime apps using CURRENT_ROLE() in row access policies will always return the app owner’s role, not the viewer’s role”.

Therefore, any fine-grained access control (by role, by user) fails under owner’s rights. The only workaround is to implement your own logic in the app (e.g. passing the Snowflake user token in queries), but that is fragile and shifts security checks to application code.

Why Container Runtime (SPCS) Solves These Issues

Snowflake’s Container Runtime (also known as Snowpark Container Services or “Compute Pool for Streamlit”) was designed to address these limitations. In Container mode:

• Restricted Caller’s Rights: Container runtimes support restricted caller’s rights, a model where an app can run with the caller’s (viewer’s) Snowflake privileges, subject to explicit grants. In other words, if Rakesh opens the app, the Snowflake queries can execute as Rakesh(with his assigned role) rather than as the owner’s role. Crucially, the developer must explicitly ask for each privilege (e.g. SELECT on a table) via GRANT CALLER ... commands. This prevents privilege escalation while enabling true user-based filtering. (Note: Restricted caller’s rights was a preview feature, but as of Aug 2025 it is GA for native apps/SPCS and fully supported in Streamlit container mode.)

• Shared Persistent Server: In container mode, one instance of the app runs and all viewers share it. This means faster connections for new viewers (the app is already running) and support for caching results across users (when appropriate).

• Unlimited Packages: The app runs in a Snowpark-managed Docker container where you control the image. You can use PyPI (and any internet-accessible repository) to install any Python package. There’s no Conda limitation. This unlocks the full Python ecosystem (e.g. the latest Streamlit, langchain, openai, azure-identity, sentence-transformers, etc.). It also allows including OS-level libraries and binaries if needed.

• Network Access via External Integrations: Container apps can access the internet, but you must explicitly allow it via an External Access Integration (EAI). This means you define which network endpoints (PyPI, REST APIs, etc.) the app can reach. This controlled egress is much more flexible than Warehouse runtime, while still meeting security requirements.

• Flexible Config: Container apps can run any Streamlit version (including streamlit-nightly) and any Python 3.11 packages. You can configure ports, compute pool size, autosuspend, etc., via DDL (CREATE/ALTER SERVICE) or the Snowflake UI.

Because of these advantages, enterprise AI apps on Snowflake should use Container Runtime once they move beyond proof-of-concept. The rest of this article explains the key steps to make this transition: setting up restricted caller’s rights, configuring package access, and building the container environment.

Enabling Restricted Caller’s Rights

To run queries under the caller’s credentials, we use Snowflake’s restricted caller’s rights model. Conceptually, this changes the execution flow:

Under the hood, Snowflake restricts the app so it can only use privileges explicitly granted via GRANT CALLER ... (also called caller grants).

In this architecture, your app code runs in a container, but it queries Snowflake using the viewer’s identity.

The App Owner Role (APP_OWNER_ROLE): Owns the container/app and has full CRUD privileges on the application objects (the container itself, the stage, tables, views etc.). The Business Roles (HR_ROLE, FINANCE_ROLE): These roles have read-only privileges on the underlying datasets (Tables, Views, Semantic Views, Stored Procedures, UDFs, Stages, Cortex Analyst/Search/Agents). The Enforcement: By granting “Caller Rights,” you tell Snowflake: “When a user (e.g., Rakesh with HR_ROLE) connects via the app, the app can only execute queries using the privileges that Rakesh already possesses."

Here’s how to set it up:

1. Use Container Runtime: Ensure your Streamlit app is created or migrated to container runtime (Snowflake calls it “FROM” mode in CREATE STREAMLIT). Restricted caller’s rights require container runtime. In warehouse runtime, any attempt to use st.connection("snowflake-callers-rights") will fail.

In Snowflake container runtime streamlit file add-

st.connection("snowflake-callers-rights")

import streamlit as st

# For enabling Snowflake caller rights
conn = st.connection("snowflake-callers-rights")

# This query will now respect RLS policies tied to CURRENT_ROLE()
df = conn.query("SELECT * FROM DATA_DB.PUBLIC.HR_DATA")
st.write(df)

1. Streamlit Version: Use Streamlit library version >= 1.53.0. Older versions do not support the caller’s rights feature.

2. Grant CALLER privileges: A Snowflake admin (or higher) must grant caller privileges on each object the app needs.

NOTE: The caller privileges will be granted only to the app owner role.

To enable this, an Administrator (or a role with MANAGE CALLER GRANTS) must perform the grants. You can either perform these manually or grant the app owner the power to manage their own caller grants.

The admin can grant the manager caller grants to the owner role, this lets the app owner grant caller rights on underlying objects themselves:

-- 1. Enable self-management of caller grants for the App Owner
GRANT MANAGE CALLER GRANTS ON ACCOUNT TO ROLE APP_OWNER_ROLE;

-- 2. Grant Caller Usage/Select on required Databases and Tables
-- HR Data
GRANT CALLER USAGE ON DATABASE hr_db TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE hr_db TO ROLE APP_OWNER_ROLE;
GRANT CALLER SELECT ON TABLE hr_db.hr_schema.employees TO ROLE APP_OWNER_ROLE;

-- Finance Data
GRANT CALLER USAGE ON DATABASE finance_db TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE finance_db TO ROLE APP_OWNER_ROLE;
GRANT CALLER SELECT ON TABLE finance_db.finance_schema.revenue TO ROLE APP_OWNER_ROLE;

-- Sales Data
GRANT CALLER USAGE ON DATABASE sales_db TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE sales_db TO ROLE APP_OWNER_ROLE;
GRANT CALLER SELECT ON TABLE sales_db.sales_schema.pipeline TO ROLE APP_OWNER_ROLE;

-- BGV Data
GRANT CALLER USAGE ON DATABASE bgv_db TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE bgv_db TO ROLE APP_OWNER_ROLE;
GRANT CALLER SELECT ON TABLE bgv_db.bgv_schema.bgv_data TO ROLE APP_OWNER_ROLE;

-- Operations Data
GRANT CALLER USAGE ON DATABASE ops_db TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE ops_db TO ROLE APP_OWNER_ROLE;
GRANT CALLER SELECT ON TABLE ops_db.ops_schema.business_ops TO ROLE APP_OWNER_ROLE;

-- Semantic Views & Stages
GRANT CALLER SELECT ON VIEW DATA_DB.PUBLIC.SEMANTIC_VIEW TO ROLE APP_OWNER_ROLE;
GRANT CALLER READ ON STAGE DATA_DB.PUBLIC.DOC_STAGE TO ROLE APP_OWNER_ROLE;

-- Cortex Search, Agents, and Snowflake Functions
GRANT CALLER USAGE ON CORTEX SEARCH SERVICE DATA_DB.PUBLIC.MY_SEARCH_SERVICE TO ROLE APP_OWNER_ROLE;
GRANT CALLER USAGE ON DATABASE SNOWFLAKE TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL SCHEMAS IN DATABASE SNOWFLAKE TO ROLE APP_OWNER_ROLE;
GRANT INHERITED CALLER USAGE ON ALL FUNCTIONS IN DATABASE SNOWFLAKE TO ROLE APP_OWNER_ROLE;

-- Intelligence/Agents DB
GRANT CALLER USAGE ON DATABASE SNOWFLAKE_INTELLIGENCE TO ROLE APP_OWNER_ROLE;
GRANT CALLER USAGE ON SCHEMA SNOWFLAKE_INTELLIGENCE.AGENTS TO ROLE APP_OWNER_ROLE;

Each caller grant allows the app to use that privilege only if the viewer has it. The database and schema grants use INHERITED CALLER USAGE (to apply to all current/future schemas or objects). We also grant CALLER rights on the SNOWFLAKE system databases so Snowflake’s functions and UDFs will run under the caller’s context. Similarly, if the app calls a semantic view or stored procedure, grant CALLER SELECT or CALLER EXECUTE on those objects. For example, for a semantic view corp_db.corp_schema.kpi_view, use GRANT CALLER SELECT ON VIEW corp_db.corp_schema.kpi_view TO ROLE APP_OWNER_ROLE. For an internal stage or file format, use GRANT INHERITED CALLER USAGE,READ ON ALL STAGES IN SCHEMA ops_db.ops_schema

Make sure the admin has provided the cortex_user role and cortex_agent_user role to the role.

GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO APP_OWNER_ROLE;
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_AGENT_USER TO APP_OWNER_ROLE;

4.Specifying Execution Rights in streamlit_spec.yaml

To fully enable restricted caller’s rights for the container application, you must define the executeAsCaller capability in your streamlit_spec.yaml file within your project folder.

spec:
  containers:
    - name: streamlit-app
      image: <your_image_path>
      env:
        STREAMLIT_SERVER_PORT: 8080
capabilities:
  securityContext:
    executeAsCaller: true

5.Critical Requirement: Default Role Alignment

Even with the correct CALLER grants and executeAsCaller: true configuration, the restricted caller’s rights model relies on the user's default role.

• Role Context: Snowflake’s restricted-caller connections resolve permissions based on the user’s DEFAULT_ROLE rather than the role currently selected in the Snowsight UI at the time of access.

• Enforcement: If a user (e.g., Rakesh) logs into the Streamlit app, the query will be executed using his DEFAULT_ROLE. If his DEFAULT_ROLE is set to PUBLIC (or any role other than HR_ROLE), the RLS policy—which checks CURRENT_ROLE()—will fail to authorize the query, resulting in "no data" or an "insufficient privileges" error.

• Best Practice: Ensure all the end users of this app have their DEFAULT_ROLE correctly configured to match the business role required to view the data for their specific department.

By enforcing these five steps, you ensure that the application is not only functional but also compliant with strict enterprise data governance, preventing “privilege escalation” where the app owner’s rights would otherwise accidentally expose sensitive data to unauthorized users.

Throughout this setup, the app is still “owned” by app_owner_role, but its queries run as the viewer’s role. This is the essence of restricted caller’s rights. (The name highlights: the app is restricted to use only those caller privileges explicitly granted, rather than all of the caller’s powers.)

Now you are all set to use caller rights. This setup will help you to restrict user’s role to access only the data they are allowed to see.

QUICK TEST:

Change your default role to HR_ROLE and test the RLS on the streamlit container app — it will only fetch the data HR_ROLE is allowed to see.

Major Consideration When Using Cortex Search with Restricted Caller Rights

One important point that isn’t immediately obvious is how Cortex Search behaves in an enterprise application.

A Cortex Search Service builds and maintains vector embeddings over the source table to provide semantic search capabilities. However, simply enabling Restricted Caller Rights on your Streamlit application does not automatically guarantee that every Cortex Search request is evaluated using the logged-in user’s security context.

In most enterprise applications, document access is controlled using Row Access Policies, business-unit mappings, customer-level authorization, or department-specific permissions. These policies often rely on functions such as CURRENT_ROLE() or CURRENT_USER() to determine which records a user is allowed to access.

If your Agent invokes the Cortex Search Service directly, you may not get the desired caller-context evaluation for those security policies.

Solution:

Your agent definition should call the SP, that has the cortex search service, instead of calling the search service directly.

To bridge this gap, implement a secure intermediary layer — such as a Stored Procedure or a dedicated Python function — that acts as the gatekeeper between your AI Agent and the Cortex Search Service. Because this intermediary is invoked by your Streamlit app running with Restricted Caller’s Rights, it executes using the logged-in user’s security context rather than the app owner’s.

Your workflow should be:

1. User Request: The user initiates a query via the Streamlit interface.

2. Secure Intermediary: The Agent decides to use cortex analyst or search, based on instructions it calls the intermediary (Stored Procedure/Function) for detailed data or summary that is in tables on which search service is made.

3. Context-Aware Execution: The intermediary executes the search under the CURRENT_ROLE() of the authenticated user.

4. RLS Enforcement: Snowflake’s native Row Access Policies automatically evaluate the user’s role against the indexed documents.

5. Filtered Result: Only the subset of documents authorized for that specific user is retrieved and passed back to the Agent for processing.

Network Rules and External Access Integration for Packages

To install third-party Python packages in SPCS Streamlit apps, you must grant the app internet access at build/deploy time. Snowflake enforces this via network rules and external access integrations (EAI).

Creating Network Rules

A network rule specifies an external host (and port) that the container is allowed to reach. For PyPI (Python package index), we need to allow access to pypi.org and its CDN domain. You can either use Snowflake’s managed PyPI rule or create your own. For illustration, here’s how to create one:

-- Example: allow PyPI domains (HTTPS default port 443)
CREATE OR REPLACE NETWORK RULE pypi_network
  MODE = EGRESS
  TYPE = HOST_PORT
  VALUE_LIST = ('pypi.org:443', 'files.pythonhosted.org:443');

This VALUE_LIST covers the main PyPI endpoints. (Snowflake’s docs note you must list full hostnames; wildcards like *.pypi.org are not supported.) If you omit the port, it defaults to 443 for HTTPS. You could also allow all internet (0.0.0.0:443) but that’s usually too permissive.

Alternatively, Snowflake provides a managed rule SNOWFLAKE.EXTERNAL_ACCESS.PYPI_RULE that already includes the necessary PyPI domains. For brevity, we can use that:

-- Using Snowflake-managed PyPI network rule (easiest)
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION pypi_access_integration
  ALLOWED_NETWORK_RULES = (snowflake.external_access.pypi_rule)
  ENABLED = TRUE;

Under this approach, Snowflake created a network rule object snowflake.external_access.pypi_rule that covers the needed domains. This EAI now allows egress to PyPI.

Creating the External Access Integration

After you have your network rules, create an External Access Integration (EAI) that bundles them. For example:

-- Manual approach with our own network rule
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION pypi_access_integration
  ALLOWED_NETWORK_RULES = (pypi_network)
  ENABLED = TRUE;

-- Grant usage on the integration to the app developer role
GRANT USAGE ON INTEGRATION pypi_access_integration TO ROLE my_streamlit_role;

The first statement makes an integration named pypi_access_integration that allows traffic to the hosts in pypi_network. We then grant USAGE on the integration to the role that owns (or deploys) the Streamlit app.

(If you use the Snowflake-managed rule, you could skip the CREATE NETWORK RULE and just do the first SQL block above with ALLOWED_NETWORK_RULES = (snowflake.external_access.pypi_rule).)

Associating the EAI with the Streamlit App

Finally, tell the Streamlit object to use the integration. You can do this in either CREATE STREAMLIT or ALTER STREAMLIT. For an existing app:

USE ROLE my_streamlit_role;

ALTER STREAMLIT mydb.hr_schema.hr_streamlit_app
  SET EXTERNAL_ACCESS_INTEGRATIONS = (pypi_access_integration);

This DDL instructs Snowflake to attach the EAI to the app. Once set, whenever you deploy or update the app, Snowflake’s container environment will allow pip-install from PyPI. Note: You can also set it at CREATE STREAMLIT time with EXTERNAL_ACCESS_INTEGRATIONS=(...).

Summary of Commands

Putting it all together, a typical sequence might be:

-- 1. (Optional) Use Snowflake-managed PyPI rule
-- CREATE OR REPLACE NETWORK RULE pypi_network MODE = EGRESS TYPE = HOST_PORT VALUE_LIST=('pypi.org:443','files.pythonhosted.org:443');

-- 2. Create external access integration for PyPI
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION pypi_access_int
  ALLOWED_NETWORK_RULES = (snowflake.external_access.pypi_rule)  -- or (pypi_network)
  ENABLED = TRUE;

-- 3. Grant the integration to the app's owner role
GRANT USAGE ON INTEGRATION pypi_access_int TO ROLE my_streamlit_role;

-- 4. Attach EAI to the Streamlit app
ALTER STREAMLIT mydb.hr_schema.my_app
  SET EXTERNAL_ACCESS_INTEGRATIONS = (pypi_access_int);

With this in place, your Streamlit container can pip-install any public package during image build.

For example: If you want to use Plotly, now you can simply add —

Add Plotly to requirements.txt

Inside your project folder, ensure your requirements.txt includes Plotly. When the container starts, it will use the pypi_access_int to fetch these libraries.

# requirements.txt
streamlit>=1.53.0
plotly==5.24.0

End-to-End Example Flow

Putting it all together, the runtime flow looks like this:

1. User Authentication: Rakesh accesses the Streamlit app URL (Snowflake provides a link). She is prompted to log in via Snowflake (or SSO). Snowflake issues a short-lived token for the viewer, including their default role.

2. Streamlit App Execution: Snowflake’s compute pool routes the request to the already-running container instance of the app. The app’s code (from app.py) starts executing for this session.

3. Caller’s Rights Connection: At the top of the Streamlit script, we did conn = st.connection("snowflake-callers-rights"). This uses the secret token to establish a Snowflake session as Rakesh.

4. Query Execution: When Rakesh ask “Show me my department’s data summary”, the code runs a query via conn.query(...). Snowflake sees the query coming from Rakesh’s context, checks that app_owner_role has CALLER SELECT on the table and Rakesh has SELECT through some application role, and then executes it. Any RLS on the table sees CURRENT_ROLE() = Rakesh’s role — HR_ROLE.

5. Return Results: The query results (already filtered by RLS) go back through the connector and are displayed in the app.

6. Display: Streamlit renders the output (charts or text) in Rakesh’s browser. At no point did the app reveal data outside Rakesh’s authorization.

What’s Next

In this article, we covered why enterprise AI apps shift from Streamlit’s Warehouse runtime to Container runtime (SPCS) and how to configure third-party packages and restricted caller’s rights. We enabled per-user data access via caller’s rights and provided the exact SQL for network rules, EAIs, and integration with Streamlit.

Above all, remember that running production apps on Snowflake means designing for least privilege. The combination of Restricted Caller’s Rights and External Access Integrations gives you tight control: the app can only do what you explicitly grant. Your ongoing work will likely involve refining those grants as business needs evolve, updating and scanning dependencies, and monitoring usage (e.g. via Snowflake’s EXTERNAL_ACCESS_HISTORY or SPCS logging).

By following these practices, your organization can leverage AI on its Snowflake data securely and scalably, effectively turning your enterprise data warehouse into an AI-powered insights platform.

However, Restricted Caller Rights are only one part of building a secure enterprise AI application.

The next step is ensuring that every user only sees the data they are authorized to access, regardless of whether they’re interacting through a SQL query, a Cortex Agent, a Cortex Search Service, or a semantic model.

In the next article, we’ll take a deep dive into Row Access Policies and Dynamic Data Masking in Snowflake, where we’ll build a real-world enterprise authorization framework.

We’ll cover topics such as:

• Implementing Row Access Policies using CURRENT_ROLE() and CURRENT_USER()

• Restricting data by Business Unit, Department, or Country

• Applying Dynamic Data Masking to sensitive fields such as salaries, invoice amounts, TCV (Total Contract Value), and confidential financial metrics

• Integrating Row Access Policies with Cortex Agents and enterprise AI applications

• Ensuring AI responses respect the same security policies as traditional SQL queries

• Best practices for designing scalable, maintainable, and auditable security models in Snowflake

By the end of the next article, you’ll have a robust authorization framework that allows AI applications to deliver personalized responses while ensuring sensitive enterprise data remains protected.

#snowflake #snowflakecallerrights #snowflakecortex #cortexagent #restrictedcallerrights #streamlitinsnowflake #enterpriseAI #AIagent

One of the biggest risks in enterprise AI is exposing sensitive information during document storage, processing, and retrieval. Simply storing files in cloud storage or data platforms is not enough. The data should also be secured during transmission.

In this article, I will walk you through a security-first architecture that combines AES encryption, Azure Key Vault, Service Principals, RBAC permissions, and controlled decryption workflows to protect sensitive documents while still enabling AI-powered document extraction, analytics, and intelligent agents.

We'll cover:

✔ The security problem enterprises face

✔ Why encrypted storage alone is not enough

✔ Why encryption keys should never be stored in code or .env files

✔ Azure Key Vault fundamentals

✔ Service Principal authentication

✔ RBAC permissions and access control

✔ Creating and managing Key Vault secrets

✔ Python implementation examples

✔ Snowflake and Databricks integration patterns

✔ Secure document processing for AI workloads

✔ Security benefits and best practices

This architecture can be applied across Snowflake, Databricks, Microsoft Fabric, Azure AI Foundry, Synapse Analytics, and custom AI platforms.

The Growing Security Challenge in Enterprise AI

Organizations are increasingly building AI applications on top of business-critical documents.

Examples include:

• Customer contracts

• Legal agreements

• Invoices

• Financial statements

• Employee records

• Payroll information

• Vendor documents

• HR reports

• Employee attrition predictions

• Compliance reports

• Healthcare records

• Procurement documents

These documents often contain highly sensitive information.

At the same time, organizations want AI systems to:

• Extract information from documents

• Build search experiences

• Power RAG solutions

• Support AI agents

• Generate business insights

• Answer natural language questions

• Generate KPIs, charts and visuals

This creates a difficult challenge:

How can we allow AI systems to use sensitive information without exposing the underlying documents?

The Wrong Approach

Many organizations unknowingly introduce security risks by:

• Storing documents unencrypted

• Storing encryption keys in source code or environment files

• Keeping secrets in configuration files

• Saving credentials inside notebooks

• Embedding keys inside ETL pipelines

For example:

AES_KEY = "MySecretEncryptionKey"

If someone gains access to:

• GitHub repositories

• DevOps pipelines

• Virtual machines

• Databases

• Storage accounts

they may gain access to both the encrypted documents and the encryption key.

At that point, encryption provides little value.

The Security Principle

One of the most important security principles is:

Encrypt the data and protect the encryption key separately.

Even if attackers gain access to encrypted documents, they should not be able to access the encryption key.

This is where Azure Key Vault becomes critical.

The Solution Architecture

The architecture consists of five major layers:

1. Document Encryption

2. Secure Key Storage

3. Controlled Authentication

4. In-Memory Decryption

5. AI Processing

Document Source - The source of document can be anything like SAP, Cloud storage, Azure Blob Storage etc.

When we move them to our storage layer, they should be encrypted.

A) Why We Encrypt Documents

Consider documents such as:

• Client contracts

• Employee compensation data

• Vendor agreements

• Invoices

• Strategic business reports

• Attrition prediction outputs

These files often contain information that should not be accessible to every user or system.

Before storing these files, we encrypt them using AES-256.

This ensures that even if storage is compromised or a developer has RBAC (Role based access control), the document remains unreadable without the encryption key.

B) Creating an AES Encryption Key

Example:

from Crypto.Random import get_random_bytes

AES-256 = 32 bytes

aes_key = get_random_bytes(32)

print(aes_key.hex()) 32 bytes = 256 bits

This creates an AES-256 encryption key.

Why AES?

AES (Advanced Encryption Standard) is one of the most widely used symmetric encryption algorithms in the world.

Advantages:

• Fast

• Highly secure

• Industry standard

• Suitable for large documents

• Supported across cloud platforms

AES-256 is commonly used in enterprise environments.

C) What is Azure Key Vault?

Azure Key Vault is a managed Azure service designed for securely storing and controlling access to secrets, cryptographic keys, and certificates.

Instead of storing sensitive values in application code, organizations place them inside a centralized vault protected by Azure Identity and RBAC controls.

Azure Key Vault supports three primary object types:

Secrets

Examples:

1. API Keys

2. Database Passwords

3. OAuth Client Secrets

4. AES Encryption Keys

Keys

Examples:

1. RSA Keys

2. Cryptographic Signing Keys

3. Customer Managed Encryption Keys

Certificates

Examples:

1. SSL Certificates

2. TLS Certificates

In our use case, the AES encryption key is stored as a Secret.

Why Azure Key Vault is required?

1. Centralized secret management

2. Fine-grained access control

3. Secret rotation support

4. Access auditing

5. Compliance support

6. Reduced insider risk

Creating Azure Key Vault

1.Navigate to: portal.azure.com

Azure Portal -> Create Resource -> Key Vault -> Create

2.Configure:

Subscription

Resource Group

Region (eg. Central India)

Vault Name (eg. projectname-aes-key-kv)

3.For Authorization/Permission Model:

Select: Azure Role Based Access Control (RBAC)

4.For Networking:

Public access: Selected networks

or

Private endpoint

5.Permissions Required to Create a Key Vault

Typically one of the following roles:

1. Owner

2. Contributor

3. Key Vault Contributor

Important:

Key Vault Contributor can manage the Key Vault resource itself but cannot read secrets stored within the vault.

This separation improves security.

6.Storing the AES Key in key vault secret and permission required to create secrets.

IMP: When using Azure RBAC, you will need Key Vault secret officer role to create secret.

Once the role is assigned, Create and Store the generated AES key as a secret.

Example:

Secret Name:

AES-DOCUMENT-ENCRYPTION-KEY : secret _code

The key is now centrally managed and protected.

D) Creating a Service Principal Applications, requires an identity to access Azure resources.

This is where Service Principals are used.

Think of a Service Principal as:

A non-human identity used by applications.

Example:

Snowflake/Databricks/Azure Function/Fabric Pipeline/Custom API

↓

Service Principal

↓

Azure Key Vault

↓

AES Secret

The Service Principal becomes the trusted identity that accesses the vault.

Why Service Principals Are Important

Without Service Principals:

1. Applications would require user credentials

2. Automation becomes difficult

3. Auditing becomes harder

Service Principals provide:

1. Automated authentication

2. Secure access

3. RBAC integration

4. Least privilege access

RBAC access for service principal on the key vault: Application Access

For Snowflake, Databricks, Fabric, or AI platforms:

Role:

Key Vault Secrets User

Can:

1. Read secret values

2. Read secret metadata

E) Python Example: Reading the AES Key

from azure.identity import ClientSecretCredential
from azure.keyvault.secrets import SecretClient

credential = ClientSecretCredential(
    tenant_id="<tenant-id>",
    client_id="<client-id>",
    client_secret="<client-secret>"
)

vault_url = "https://company-kv.vault.azure.net"

client = SecretClient(
    vault_url=vault_url,
    credential=credential
)

aes_key = client.get_secret(
    "AES-DOCUMENT-ENCRYPTION-KEY"
).value

The application retrieves the key only when needed.

F) Encrypting Sensitive Documents

from Crypto.Cipher import AES
from Crypto.Util.Padding import pad
from Crypto.Random import get_random_bytes

# AES key retrieved from Azure Key Vault
key = bytes.fromhex(aes_key_hex)

# Generate random IV (16 bytes for AES)
iv = get_random_bytes(16)

with open("contract.pdf", "rb") as file:
    plaintext = file.read()

cipher = AES.new(key, AES.MODE_CBC, iv)

ciphertext = cipher.encrypt(
    pad(plaintext, AES.block_size)
)

# Store IV along with encrypted content
encrypted_content = {
    "iv": iv,
    "ciphertext": ciphertext
}

The encrypted file can now be stored safely.

G) AI Processing Pattern

When processing begins:

1. Retrieve AES key from Key Vault

2. Decrypt document in memory

3. Extract structured or unstructured chunk information

4. Persist extracted results

5. Discard decrypted content

The key principle is:

Decrypt only when required and only in memory.

H) Platform Integration Examples

I) AI Agents and RAG Systems

Once document information is extracted into structured datasets, AI systems can answer questions such as:

1. Which contracts expire next month?

2. Which invoices exceed $100,000?

3. Which vendors require renewal?

4. What is the predicted attrition risk for the Sales department?

5. Which customers have pending obligations?

The AI model operates on structured business data rather than unrestricted access to sensitive raw documents.

J) Security Benefits Achieved

This architecture provides:

1. Encryption at Rest - Documents remain protected in storage.

2. Key Separation - Encryption keys are separated from encrypted data.

3. Least Privilege Access - Applications receive only required permissions.

4. Auditing - Secret access can be monitored.

5. Key Rotation - Encryption keys can be replaced centrally.

6. Compliance Support - Supports enterprise governance and security requirements.

7. Reduced Attack Surface - Compromised storage does not automatically expose sensitive information.

Final Thoughts

As organizations accelerate AI adoption, securing the data feeding those AI systems becomes just as important as building the models themselves.

A secure AI platform is not only about model accuracy, vector databases, or intelligent agents. It is equally about protecting contracts, invoices, employee information, and other sensitive business assets that power those systems.

By combining AES encryption, Azure Key Vault, Service Principals, RBAC permissions, and controlled in-memory decryption, organizations can build AI platforms that are secure, scalable, compliant, and enterprise-ready.

The goal is simple:

Protect the documents.

Protect the keys.

Allow AI access only when necessary.

What next?

In my next article, I will walk you through how we implemented enterprise-grade access controls for AI-powered platforms using techniques such as:

• Row-Level Security (RLS)

• Row Access Policies

• Dynamic Data Masking

• Role-Based Access Control (RBAC)

• Business Unit-based filtering

• Customer-based entitlements

• Financial data masking

• Secure document access controls

• Time-limited document download links

• Agent-level authorization patterns

• Governance frameworks for AI applications

We'll explore how an AI agent can return different answers to different users for the same question, while ensuring that sensitive information remains protected and compliant with organizational policies.

Because securing the documents is important.

But securing who can access the information extracted from those documents is what truly makes an AI platform enterprise-ready.

#ArtificialIntelligence #DataEngineering #Azure #AzureKeyVault #CyberSecurity #DataSecurity #DocumentAI #Snowflake #Databricks #EnterpriseAI #AIAgents #AI