vault.hashicorp.com/agent-init-first does not work with init containers coming from annotations

[josueCarvajal]

Problem Statement

I was working on a specific scenario with init containers, in which I need the agent-init must be executed first before all the other init containers, by doing a little bit of research I've reached the following issue #53 in which the following fix was applied https://github.com/hashicorp/vault-k8s/pull/91/files

So I've decided to try it but it did not work for me. By checking that pull request I've noticed that the way it actually works is by appending the agent-init before all the init containers in the containers := a.Pod.Spec.InitContainers but in my scenario I do not have defined InitContainers in the pod definition, my init containers are coming from other annotations, causing the vault-agent to be executed as last resource instead of the very first one since it does not check those annotations!

In my case, I was trying to run the agent-init to mount a vault token and then have the banzai webhook annotations (https://bank-vaults.dev/docs/mutating-webhook/annotations/) to mount the secret in it's own init container, but it crashes since it requires the agent-init to be executed first!

To Reproduce

Steps to reproduce the behavior:

1. Follow the documentation of integrating a third party webhook, or any tool that injects init containers using podAnnotations, for this example I'm going to use banzai webhook (https://bank-vaults.dev/docs/mutating-webhook/annotations/)
2. Use the following annotations in your deployment, appRole must be enabled in this scenario, in this example my role is user-management-role, here are the hashicorp annotations for the vault-agent

podAnnotations:
    vault.hashicorp.com/agent-init-first: 'true' 
    vault.hashicorp.com/agent-inject-token: "true"
    vault.hashicorp.com/auth-type: 'approle'
    vault.hashicorp.com/auth-path: 'auth/approle'
    vault.hashicorp.com/role: "user-management-role" 
    vault.hashicorp.com/auth-config-role-id-file-path: '/vault/custom/role-id' 
    vault.hashicorp.com/agent-extra-secret: 'approle'
    vault.hashicorp.com/agent-inject:  "true"
    vault.hashicorp.com/tls-skip-verify: "true"

3- For the previous scenario you need to create a k8s secret in this way holding the roleId

 apiVersion: v1
kind: Secret
metadata:
  name: approle
data:
  role-id: <base64EncondedRoleId>

4- Add the following annotations for banzai (or the tool that injects init containers using annotations)
Here the annotation token-auth-mount will use the agent-init mount to read the token

    vault.security.banzaicloud.io/vault-addr: "https://dev-vault.dev-vault:8200"
    vault.security.banzaicloud.io/vault-skip-verify: "true" 
    vault.security.banzaicloud.io/token-auth-mount: "vault-secrets:token"

5. Execute the deployment / pod and check that the order is not applied by the vault.hashicorp.com/agent-init-first
   You can check this using lens and reviewing the execution order

And this causes the following error with the init container that required the access to that token:

Expected behavior
When using vault.hashicorp.com/agent-init-first it should always put the agent first, even though there are init containers coming from annotations.

Environment

• Kubernetes version: 1.27
  • Distribution or cloud vendor EKS (AWS)
  • Other configuration options or runtime services: Banzai webhook (for environment variables)
• vault-k8s version: 1.3

Additional context
The documentation did not mentioned anything about this scenario, it will be good to update it if a fix for this is not feasible.

[Shirueopseo]

I have a issue similar to that, using thevault.hashicorp.com/agent-init-first: 'true' annotation at the deployment which creates a backend that uses kv store in hashicorp vault (which works and injects those secrets without and problem. The thing is that sometimes randomly, when the backend re-deploys on k8s cluster (aws eks 1.30), the agent don't start first and the backend container throws the following error: secret file not found. I am using the latest helm release for hashicorp vault server and injector, but since this error paperas randomly, i am not able to reproduce it an neither apear anything on the logs that could indícate a injector error. Any clue about this? do i need to out the annotation at the top of the other annotations on the deployment, or it conflicts with something?