The data platform is where an organisation's most sensitive information is concentrated, and it is often the least-secured part of the Azure estate.
Azure Data Factory and Azure Databricks move and process that data at scale. They hold credentials to every source system, and they run code against the data itself. That combination is why the security model is not the same as a typical application, and why the getting-started defaults are the wrong place to leave it.
What each one does, quickly
Two tools, two jobs, usually working together.
Azure Data Factory is the mover and scheduler. Pipelines, activities, and triggers move and transform data between systems. Integration runtimes do the actual work: the Azure runtime for cloud-to-cloud, the self-hosted runtime for on-premises and private sources, and the SSIS runtime for lift and shift. Linked services hold the connection details to every source and sink it touches.
Azure Databricks is the processor. Apache Spark, notebooks, jobs, SQL warehouses, and machine learning, with Delta Lake for storage and Unity Catalog as the governance layer. It is where the heavy transformation and analytics happen.
A common pattern uses both: Data Factory lands raw data and handles scheduling, and Databricks runs the transformation through the medallion layers, bronze to silver to gold, before the data is served to a warehouse or to business intelligence.
The use cases
The reason this pairing shows up everywhere:
- Batch ETL and ELT. Ingest from databases, SaaS, files, and on-premises systems into a lake, transform through the medallion layers, and serve to a warehouse or to reporting.
- Streaming and near real-time. Databricks structured streaming from Event Hubs or Kafka, for data that cannot wait for the nightly batch.
- Machine learning and AI. Feature engineering, training, and serving on Databricks, scheduled by Data Factory, with MLflow and Unity Catalog governing the models as well as the data.
- Lakehouse and business intelligence. Delta Lake plus Unity Catalog serving Databricks SQL and Power BI, without copying data into a separate warehouse.
- Migration and hybrid integration. Data Factory's large connector library and the self-hosted integration runtime, pulling from on-premises and private sources.
Every one of these concentrates access. That is the whole security story, so it is worth being explicit about why.
Why the security model is different
- Credentials are concentrated. Linked services and Databricks connections hold the keys to every source and sink. Compromise the platform, and you inherit everything it can reach. The blast radius is the entire data estate.
- It runs arbitrary code by design. Notebooks, jobs, init scripts, and libraries all execute code. A compromised notebook, a malicious dependency, or a rogue init script runs with the compute's identity and its data access. Treat the compute as if it runs untrusted code, because sometimes it does.
- It bridges networks. The self-hosted integration runtime reaches from the cloud into on-premises networks. Serverless compute runs in a Microsoft-managed plane with its own egress to think about.
- Defaults favour speed. Public endpoints, broad tokens, and legacy storage mounts are the path of least resistance to a working pipeline, and they tend to stay long after the demo is over.
The controls that matter
Grouped by domain, because that is how you assign and review them.
Identity and access
- Kill keys and passwords. For Data Factory linked services, authenticate with the factory's managed identity or a service principal, not account keys or connection strings. For Databricks reading storage, use Unity Catalog storage credentials and external locations rather than mounted keys.
- Least privilege on those identities. A managed identity that can read the entire lake is a large blast radius in one object. Scope it to what the workload needs.
- Govern Databricks with Unity Catalog. It is the single governance layer across catalogs, schemas, tables, volumes, models, and functions, with grants down to the row and column. Define principals at the account level and provision them with SCIM from Entra ID, and turn off workspace-level provisioning. Grant to groups, not individuals, and run production jobs as service principals so a person cannot overwrite production by accident.
- Choose the right access mode. Standard access mode, previously called shared, is the recommendation for almost everything and is Unity Catalog enabled. Reserve dedicated access mode, previously single-user, for the cases standard cannot cover. Avoid the legacy no-isolation clusters for sensitive data, because they do not enforce isolation between users.
- Rein in personal access tokens. Long-lived tokens sprawl and over-grant. Prefer single sign-on and short-lived, governed tokens, and disable token creation where you can.
- Separate factories and use the narrow role. Distinct factories for dev, test, and production. Data Factory Contributor is not Owner, so grant the narrower role.
Secrets
- Data Factory: store every credential in Azure Key Vault and reference it through a Key Vault linked service, so nothing sensitive sits inline in the pipeline definition or in your Git history.
- Databricks: use secret scopes backed by Key Vault, and never hardcode a secret in a notebook. Databricks redacts known secrets from output, but that is a backstop, not a control.
Network
- Databricks classic: deploy into your own virtual network, enable secure cluster connectivity so nodes have no public IP, and use Private Link for the control plane and the web application. Route outbound through a NAT gateway or a firewall with a user-defined route where you need egress inspection.
- Databricks serverless: there is no network to inject into, so you govern it with network connectivity configurations and serverless egress policies. Lock outbound to the resources it actually needs, use private endpoints to your storage, and you can now put a firewall in front of the workspace storage account as well.
- Data Factory: use a managed virtual network with managed private endpoints to your sources, so pipeline traffic does not cross the public internet, and Private Link to the factory itself.
- Storage (ADLS Gen2): private endpoints on, storage firewall on, public access off. This is the data. Treat it accordingly.
- Self-hosted integration runtime: it bridges networks, so patch the host, restrict its outbound, and give it the least access it needs on both sides.
Data protection
- Encryption. Transit is TLS by default. For data at rest, use customer-managed keys for the workspace and storage where compliance requires it, and consider Azure confidential computing for protecting data in use on the most sensitive workloads.
- Protect the sensitive columns in Unity Catalog. Column masks and row filters, delivered through dynamic views, so a user sees only what their role allows, enforced at query time rather than by copying redacted extracts around.
- Classify and trace. Use Unity Catalog lineage for impact analysis, and Microsoft Purview for enterprise classification and discovery across the wider estate.
Compute hardening
- Enforce configuration with compute policies. Require Unity Catalog enabled access modes, allowed instance types, auto-termination, mandatory tags, and no public IP. A policy is how you stop a well-meaning engineer from spinning up an open cluster at 5pm on a Friday.
- Govern init scripts and libraries. An init script runs arbitrary code at cluster start, and a library is a supply-chain dependency. Store init scripts in Unity Catalog volumes or the workspace, allow-list them, and control where libraries come from. Restrict Git folders to trusted repositories.
- Restart compute on a schedule, so a long-lived cluster is not carrying stale state or quietly hosting a persistent foothold.
Monitoring and audit
- Turn on verbose audit logging in Databricks and read it through system tables or Azure Log Analytics. Send Data Factory diagnostic logs to Log Analytics too.
- Watch the signals that matter: unusual data egress, failed authentication, privilege changes, and new external locations. Wire it into Microsoft Defender for Cloud and Sentinel.
- Check for drift. Run the Databricks Security Analysis Tool against your workspaces on a schedule, rather than assuming the settings you made on day one still hold six months later.
The gotchas that cause the incidents
- Inline credentials in linked services, committed to Git in the deployment template. Move them to Key Vault.
- Hardcoded secrets in notebooks, then pushed to a repository where they live forever. Same fix.
- Legacy storage mounts and no-isolation clusters sharing one broad identity across every user. Move to Unity Catalog external locations and standard access mode.
- Personal access tokens, long-lived, over-permissioned, handed out and never revoked.
- Public workspace or public storage, because Private Link "was going to be done later".
- Serverless egress left wide open, quietly able to send data anywhere. Lock it with network connectivity configurations and egress policies.
- An init script or an untrusted library as the way in. This is a supply-chain problem, so govern it rather than trusting it.
- A self-hosted integration runtime host that is unpatched and over-privileged, bridging straight into the on-premises network.
Best practices, in short
- No keys or passwords. Managed identities and service principals everywhere, least privileged.
- Unity Catalog as the single governance layer, account-level SCIM, grants to groups, standard access mode.
- Every secret in Key Vault, referenced, never inline.
- Private by default: VNet injection or network connectivity configurations, secure cluster connectivity, private endpoints, storage firewall on, public access off.
- Compute policies to enforce the safe configuration, with governed init scripts and libraries.
- Customer-managed keys and column-level controls where the data demands it.
- Verbose audit logging, system tables, and the Security Analysis Tool, watched rather than just switched on.
- Separate workspaces and factories for production and non-production, and isolate the most sensitive workloads.
The part to take away
A data platform concentrates access to everything an organisation knows, and it runs code against it. That combination is why it deserves top-tier security treatment rather than the getting-started defaults. Govern it centrally with Unity Catalog and Key Vault, put it behind Private Link, assume the compute runs untrusted code, and watch the audit trail.
The controls are all there in the platform. The failure, almost every time, is that the defaults were never changed. If you want two questions to start with: where are your credentials stored, and can your compute reach the public internet. The answers tell you most of what you need to know about how exposed your data platform really is.