In today’s digital age, businesses are constantly seeking innovative solutions to drive growth and stay ahead of the competition. One such solution that has been gaining traction in recent years is Cloud Datalake and AI technologies. These technologies not only offer cost-effective storage solutions but also provide valuable insights that can help businesses make informed decisions and drive business growth.

MI Cloud AI Technologies Pvt Ltd, a leading software consultancy firm specializing in Cloud Adoption, Cloud Governance services, Datalake, and AI solutions, is at the forefront of helping businesses harness the power of these technologies. With over a decade of experience in the industry, the firm has established itself as a trusted partner for clients looking to leverage the benefits of cloud computing and artificial intelligence.

Cloud Datalake solutions offered by MI Cloud AI Technologies provide businesses with a centralized repository for storing and analyzing large volumes of data. By consolidating data from various sources into a single location, businesses can gain a comprehensive view of their operations and customer interactions. This, in turn, enables them to identify trends, make predictions, and optimize business processes for improved efficiency and profitability. In addition to Datalake solutions, MI Cloud AI Technologies also offers AI services that can further enhance business operations. Artificial intelligence algorithms can analyze data patterns, automate repetitive tasks, personalize customer experiences, and even predict future outcomes. By incorporating AI into their operations, businesses can unlock new opportunities for growth and innovation.

With a focus on cloud adoption and governance, MI Cloud AI Technologies ensures that businesses can seamlessly transition to the cloud while maintaining compliance with industry regulations and best practices. This not only streamlines the adoption process but also enhances data security and scalability for future growth. In conclusion, Cloud Datalake and AI solutions have the potential to drive significant business growth by enabling data-driven decision-making and fostering innovation. By partnering with a reputable firm like MI Cloud AI Technologies, businesses can unlock the full potential of these technologies and stay ahead in today’s competitive market landscape.

The Microsoft Azure Cloud Adoption Framework (CAF) is a comprehensive methodology designed to guide organizations through their cloud adoption journey. It encompasses best practices, tools, and documentation to align business and technical strategies, ensuring seamless migration and innovation in the cloud. The framework is structured into eight interconnected phases: Strategy, Plan, Ready, Migrate, Innovate, Govern, Manage, and Secure. Each phase addresses specific aspects of cloud adoption, enabling organizations to achieve their desired business outcomes effectively.

The Strategy phase focuses on defining business justifications and expected outcomes for cloud adoption. In the Plan phase, actionable steps are aligned with business goals. The Ready phase ensures that the cloud environment is prepared for planned changes by setting up foundational infrastructure. The Migrate phase involves transferring workloads to Azure while modernizing them for optimal performance.

Innovation is at the heart of the Innovate phase, where organizations develop new cloud-native or hybrid solutions. The Govern phase establishes guardrails to manage risks and ensure compliance with organizational policies. The Manage phase focuses on operational excellence by maintaining cloud resources efficiently. Finally, the Secure phase emphasizes enhancing security measures to protect data and workloads over time.

This structured approach empowers organizations to navigate the complexities of cloud adoption while maximizing their Azure investments. The Azure CAF is suitable for businesses at any stage of their cloud journey, providing a robust roadmap for achieving scalability, efficiency, and innovation.

Below is a visual representation of the Azure Cloud Adoption Framework lifecycle:

 The diagram illustrates the eight phases of the framework as a continuous cycle, emphasizing their interconnectivity and iterative nature. By following this proven methodology, organizations can confidently adopt Azure’s capabilities to drive business transformation.

What is Azure Cloud Adoption Framework (CAF):

The Azure Cloud Adoption Framework (CAF) is a comprehensive, industry-recognized methodology developed by Microsoft to streamline an organization’s journey to the cloud. It provides a structured approach, combining best practices, tools, and documentation to help organizations align their business and technical strategies while adopting Azure cloud services. The framework is designed to address every phase of the cloud adoption lifecycle, including strategy, planning, readiness, migration, innovation, governance, management, and security.

CAF enables businesses to define clear goals for cloud adoption, mitigate risks, optimize costs, and ensure compliance with organizational policies. By offering actionable guidance and templates such as governance benchmarks and architecture reviews, it simplifies the complexities of cloud adoption.

How Can Azure CAF Help Companies

Azure CAF provides several key benefits to organizations:

• Business Alignment: It ensures that cloud adoption strategies are aligned with broader business objectives for long-term success.
• Risk Mitigation: The framework includes tools and methodologies to identify and address potential risks during the migration process.
• Cost Optimization: CAF offers insights into resource management and cost control to prevent overspending on cloud services.
• Enhanced Governance: It establishes robust governance frameworks to maintain compliance and operational integrity.
• Innovation Enablement: By leveraging cloud-native technologies, companies can innovate faster and modernize their IT infrastructure effectively.

How AUTOMICLOUDAI(AMCA) Can Help You Onboard to Azure CAF

At AMCA, we specialize in making your transition to Azure seamless by leveraging the Azure Cloud Adoption Framework. Here’s how we can assist:

1. Customized Strategy Development: We work with your team to define clear business goals and create a tailored cloud adoption strategy.
2. Comprehensive Planning: Our experts design detailed migration roadmaps while addressing compliance and security requirements.
3. End-to-End Support: From preparing your environment to migrating workloads and optimizing operations, we ensure a smooth transition.
4. Governance & Cost Management: We implement robust governance policies and provide cost optimization strategies for efficient resource utilization.
5. Continuous Monitoring & Innovation: Post-migration, AMCA offers ongoing support to manage workloads and foster innovation using Azure’s advanced capabilities.

With AMCA as your partner, you can confidently adopt Azure CAF while minimizing risks and maximizing returns on your cloud investment. Let us guide you through every step of your cloud journey.

Contact us at myinfo@insight42.com, we provide worldwide services

Why Databricks on GCP Needs a Tool Like HashiCorp Vault

The modern data landscape presents complex security challenges that require sophisticated secrets management solutions. While Databricks on Google Cloud Platform offers powerful data processing capabilities, organizations face significant credential management hurdles that demand tools like HashiCorp Vault for comprehensive security.

The Credential Management Challenge

Databricks environments on GCP create a perfect storm for secrets management complexity. Organizations typically manage hundreds or thousands of sensitive credentials across multiple environments – development, staging, and production – each requiring access to various external services. This proliferation leads to secrets sprawl, where sensitive data becomes scattered across different platforms, making it difficult to track, secure, and manage effectively.

The collaborative nature of Databricks compounds these challenges. Data engineers, data scientists, and analysts frequently share notebooks and code, increasing the risk of inadvertent credential exposure. Without proper safeguards, sensitive information like API keys, database passwords, and service account tokens can easily leak through shared repositories or collaborative workspaces.

Security Vulnerabilities in Default Configurations

Recent security research has exposed critical vulnerabilities in Databricks platform configurations. Researchers discovered that low-privileged users could break cluster isolation and gain remote code execution on all clusters in a workspace. These attacks can lead to credential theft, including the ability to capture administrator API tokens and escalate privileges to workspace administrator levels.

The default Databricks File System (DBFS) configuration poses particular risks, as it’s accessible by every user in a workspace, making all stored files visible to anyone with access. This creates opportunities for malicious actors to modify cluster initialization scripts and establish persistent access to sensitive credentials.

Limitations of Native Databricks Secrets Management

While Databricks on Google Cloud offers Secret storage as Databricks Scoped Secrets or Databricks Secrets Backed by GCP secret manager/ Azure KeyVault as a native solution, it has significant limitations when integrated with complex Databricks workflows. GCP Secret Manager is tightly coupled to the GCP ecosystem, making it challenging to implement consistent secrets management across multi-cloud or hybrid environments. Organizations using Databricks often need to integrate with various external services, databases, and APIs that may not be Google Cloud native. It’s also reachable on a public network. Fine granular access is also a  challenge .

And why would you like to even integrate Azure KeyVault with GCP Databricks if you are on GCP 😀.

HashiCorp Vault: The Strategic Solution

HashiCorp Vault addresses these challenges through several key capabilities that are particularly valuable for Databricks on GCP:

Dynamic Secrets Generation

Vault’s Google Cloud secrets engine generates temporary, short-lived GCP IAM credentials that automatically expire. This eliminates the security risks associated with long-lived static credentials, significantly reducing the window for potential credential misuse. For AI workloads on GCP, including those running on Databricks, this dynamic approach is crucial for maintaining security while enabling automated data processing.

Centralized Secrets Management

Vault provides a unified control plane for managing secrets across different environments and platforms. This centralization addresses the secrets sprawl problem by ensuring all sensitive data is stored in a single, secure location with comprehensive access controls. Development teams can retrieve secrets programmatically without hardcoding them into notebooks or configuration files.

Advanced Access Control and Auditing

Vault implements fine-grained access policies that can be customized based on roles, environments, and specific use cases. Every secret access is logged and auditable, providing the forensic trail necessary for compliance and security incident response. This is particularly important in Databricks environments where data governance and regulatory compliance are critical requirements.

Workload Identity Federation Support(Optional)

Vault now supports Workload Identity Federation (WIF) with Google Cloud, enabling secure authentication without requiring long-lived service account credentials. This integration minimizes credential sprawl and establishes a trust relationship between Vault and GCP services, reducing security concerns associated with manually created service accounts.

Implementation

Lets get to it , here I have provided configuration in Terraform and Bash cli, you can use any other method as well.

Note : Shared Databricks Clusters are not Supported, only dedicated clusters such as personal or job clusters.

Step 1: Configurations on GCP, Create an SA and grant project.viewer and serviceaccount.admin permissions

$ export GCP_PROJECT=<Your GCP project>

$ gcloud services enable --project "${GCP_PROJECT}" \
    cloudresourcemanager.googleapis.com \
    iam.googleapis.com

$ gcloud iam service-accounts create sa-vault \
    --display-name "Vault Authenticator SA" \
    --project "${GCP_PROJECT}"

$ gcloud projects add-iam-policy-binding \
    "${GCP_PROJECT}" --member \
    "serviceAccount:sa-vault@${GCP_PROJECT}.iam.gserviceaccount.com" \
    --role "roles/viewer"

$ gcloud projects add-iam-policy-binding \
    "${GCP_PROJECT}" --member \
    "serviceAccount:sa-vault@${GCP_PROJECT}.iam.gserviceaccount.com" \
    --role "roles/iam.serviceAccountKeyAdmin"

$ gcloud projects add-iam-policy-binding \
    "${GCP_PROJECT}" --member \
    "serviceAccount:sa-vault@${GCP_PROJECT}.iam.gserviceaccount.com" \
    --role "roles/iam.serviceAccountTokenCreator"

$ gcloud iam service-accounts keys create sa-vault.json \
    --iam-account "sa-vault@${GCP_PROJECT}.iam.gserviceaccount.com"

Step 2: Configurations on Vault , I have used terrafrom , but you can use bash/cli as well.

terraform {
  required_providers {
    vault = {
      source = "hashicorp/vault"
      version = "~> 5.0.0"
    }
  }
}

provider "vault" {
  address = "Your vault address"

  # This is the configuration to run it locally with azuread auth - it will
  # automatically login using a browser
  # You can use some other auth method for vault as well
  auth_login_oidc {
    role  = "azuread"
    mount = "azuread"
  }
}

variable "gcp_sa_admins" {
  description = "List of GCP Sevice accounts for Vault admin role"
  type        = list(string)
  default = [ "" ]
}

variable "gcp_sa_contributors" {
  description = "List of GCP Sevice accounts for Vault contributor role"
  type        = list(string)
  default = [ "" ]
}
variable "gcp_databricks_project" {
  description = "List of GCP Sevice accounts for Vault contributor role"
  type        = list(string)
  default = [ "" ]
}


resource "vault_gcp_auth_backend" "gcp" {
  credentials = sa-vault.json 
  #Using all defaults, but you can customize
}

resource "vault_mount" "gcp" {
  path     = "gcp"
  type     = "kv"
  options  = { version = "2" }
}

resource "vault_kv_secret_backend_v2" "gcp" {
  mount                = vault_mount.gcp.path
  max_versions         = 0
  delete_version_after = 0
  cas_required         = false
}

resource "vault_kv_secret_v2" "example_secret" {
  mount = vault_mount.gcp.path
  name  = "common/example"
  data_json = jsonencode({
    "example_secret" = "some-value"
  })
}

resource "vault_policy" "admin" {
  name   = "admin"
  policy = <<-EOF
    path "*" {
      capabilities = ["create", "read", "update", "delete", "list", "sudo"]
    }
  EOF
}

resource "vault_gcp_auth_backend_role" "gce" {
  role                   = "gcp-vault-admin"
  type                   = "iam"
  backend                = vault_gcp_auth_backend.gcp.path
  bound_service_accounts = var.gcp_sa_admins
  token_policies         = [vault_policy.admin.name]
  max_jwt_exp            = "30m"
}

resource "vault_policy" "vault_contributors" {
  for_each = var.use_cases
  name   = "gcp/policies/gcp/databricks/contributors"
  policy = <<EOF
    path "gcp/data/databricks/*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }

    path "gcp/metadata/databricks/*" {
      capabilities = ["list"]
    }

    path "gcp/metadata/" {
      capabilities = ["list"]
    }    
    EOF
}


resource "vault_gcp_auth_backend_role" "gce" {
  role                   = "vault-databrick-contributors"
  type                   = "gce"
  bound_projects         = var.gcp_databricks_project
  backend                = vault_auth_backend.gcp.path
  bound_service_accounts = var.gcp_sa_contributors
  token_policies         = [vault_policy.vault_contributors.name]
} 

Step 3: Configuration On Databricks Cluster , Set Google Service Account on the Databricks cluster

• Via UI : While creating cluster Navigate to Compute -> New compute –> Advanced Settings -> Google Service Account –> <Enter your GCP Service Account>

Via terraform

variable "gcp_sa_contributors" {
  description = "List of GCP Sevice accounts for Vault contributor role"
  type        = list(string)
  default = [ "" ]
}
data "databricks_node_type" "smallest" {
  local_disk = true
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "Shared Autoscaling"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 10
  }
  gcp_attributes {
    google_service_account = var.gcp_sa_contributors
  }
}

Step 4: Accessing Secrets from Vault from databricks notebook or job connected to a dedicated cluster

Below is a sample python notebook code to access secrets, I recommend to write a python library to optimize the usage.

pip install hvac requests

 import requests
import hvac
 
def login_to_vault_with_gcp(role, vault_url):
    # GCP metadata endpoint for the service account token
    metadata_url = "http://metadata/computeMetadata/v1/instance/service-accounts/default/identity"
     
    # Request the JWT token from the metadata server
    headers = {"Metadata-Flavor": "Google"}
    params = {"audience": f"http://vault/{role}", "format": "full"}
     
    try:
        jwt_token = requests.get(metadata_url, headers=headers, params=params).text
    except requests.RequestException as e:
        raise Exception(f"Failed to get JWT token: {e}")
     
    # Log into Vault using the GCP method
    client = hvac.Client(url=vault_url)
    login_response = client.auth.gcp.login(role=role, jwt=jwt_token)
     
    if 'auth' in login_response and 'client_token' in login_response['auth']:
        print("Login successful")
        client.token = login_response['auth']['client_token']
        return client
    else:
        print("Login failed:", login_response)
        return None
 
def list_secrets(client, path):
    try:      
        list_response = client.secrets.kv.v2.list_secrets(mount_point=mount, path=path)
 
        list_response = client.secrets.kv.v2.list_secrets(mount_point=mount, path=path)
        print('The following paths and secrets are available under the path prefix: {keys}'.format(
            keys=','.join(list_response['data']['keys']),
        ))
 
    except hvac.exceptions.InvalidRequest as e:
        print(f"Invalid request: {e}")
    except hvac.exceptions.Forbidden as e:
        print(f"Access denied: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")
 
def create_secrets(client,mount, path,secretname):
    try:
        client.secrets.kv.v2.create_or_update_secret(mount_point=mount, path=path+"/"+secretname,secret=dict(mysecretkey='mysecretvalue'))
    except hvac.exceptions.InvalidRequest as e:
        print(f"Invalid request: {e}")
    except hvac.exceptions.Forbidden as e:
        print(f"Access denied: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")
 
if __name__ == "__main__":
    vault_url = "https://vault.com" # Replace with your vault hostname
    role = "vault-databrick-contributors"
    mount= "gcp" # Base path
    path = "databricks"  # Specify the path to list secrets
    secretname = "test1"
 
    # Log in to Vault and get the client
    client = login_to_vault_with_gcp(role, vault_url)
    print(client.token)
    if client:
        create_secrets(client,mount,path,secretname)
        list_secrets(client, path)

Conclusion: The Future of Secure Data Platforms

The integration of HashiCorp Vault with Databricks on GCP represents a critical evolution in data platform security. As organizations face increasingly sophisticated threats and stringent compliance requirements, traditional approaches to credential management are no longer sufficient.

HCP Vault Secrets and advanced features like Vault Radar are expanding security lifecycle management capabilities, enabling organizations to discover, remediate, and prevent unmanaged secrets across their entire IT estate. These tools help locate and secure credentials that developers often store insecurely in source code, configuration files, and collaboration platforms.

The architectural patterns demonstrated in this implementation provide a foundation for secure, scalable data operations that can grow with your organization’s needs. By adopting dynamic secrets, centralized management, and comprehensive auditing, teams can focus on deriving value from their data rather than managing security vulnerabilities.

The secure approach becomes the easy approach when organizations invest in proper tooling and architectural patterns. As cloud data platforms continue to evolve, the integration of enterprise-grade secrets management will become not just a best practice, but a fundamental requirement for any serious data operation.

Feel free to drop a message to myinfo@insight42.com if you have some questions or comments.