I created a simple Argo job to pull messages from a Google Cloud Pub/Sub topic. Permission has been given to the service account of GKE’s workload identity. But the Argo job failed with errors:
argo submit example.json -n argoproj
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
hello-world-pqbm5: return callable_(*args, **kwargs)
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 1030, in __call__
hello-world-pqbm5: return _end_unary_response_blocking(state, call, False, None)
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
hello-world-pqbm5: raise _InactiveRpcError(state) # pytype: disable=not-instantiable
hello-world-pqbm5: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
hello-world-pqbm5: status = StatusCode.PERMISSION_DENIED
hello-world-pqbm5: details = "User not authorized to perform this action."
hello-world-pqbm5: debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.69.95:443 {grpc_message:"User not authorized to perform this action.", grpc_status:7, created_time:"2023-05-15T01:10:43.128528579+00:00"}"
hello-world-pqbm5: >
hello-world-pqbm5:
hello-world-pqbm5: The above exception was the direct cause of the following exception:
hello-world-pqbm5:
hello-world-pqbm5: Traceback (most recent call last):
hello-world-pqbm5: File "<string>", line 26, in <module>
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/pubsub_v1/services/subscriber/client.py", line 1495, in pull
hello-world-pqbm5: response = rpc(
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
hello-world-pqbm5: return wrapped_func(*args, **kwargs)
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
hello-world-pqbm5: return retry_target(
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/retry.py", line 191, in retry_target
hello-world-pqbm5: return target()
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/timeout.py", line 120, in func_with_timeout
hello-world-pqbm5: return func(*args, **kwargs)
hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
hello-world-pqbm5: raise exceptions.from_grpc_error(exc) from exc
hello-world-pqbm5: google.api_core.exceptions.PermissionDenied: 403 User not authorized to perform this action.Thanks to my colleagues. They remind me that an Argo job needs to specify a service account when running in the workload identity namespace.
argo submit example.json -n argoproj --serviceaccount argo-workflow
Or, I can add this service account to the YAML file:
apiVersion: argoproj.io/v1alpha1 kind: Workflow # new type of k8s spec metadata: generateName: hello-world- # name of the workflow spec spec: entrypoint: whalesay # invoke the whalesay template serviceAccountName: argo-workflow