Skip to main content

How to integrate Google Cloud Platform

Send your Google Cloud audit logs to the Defense.com SIEM using Pub/Sub

Written by Daniel Sampson

Defense.com collects your Google Cloud logs by reading them from a Pub/Sub subscription in your environment. You route the logs you want monitored into a Pub/Sub topic using a Cloud Logging sink, and the Defense.com SIEM pulls them from a subscription using a dedicated service account with read-only access.

Cloud Logging → Log sink → Pub/Sub topic → Pub/Sub subscription → Defense.com SIEM (pulls)

Nothing in this setup gives Defense.com access to your wider Google Cloud environment. The service account can only view and consume messages from the one subscription you create.

What you'll create

Resource

Suggested name

Pub/Sub topic

defensecom-siem

Pub/Sub subscription

defensecom-siem-sub

Service account

defensecom-siem-sa

Log sink

defensecom-siem-sink

You can use different names – just send us the ones you choose (see Step 7).

Before you start

  • You'll need an account with permission to create projects (if needed), enable APIs, create Pub/Sub resources, create log sinks and manage IAM. Project Owner covers everything for a single project. For an organisation-wide sink (Step 6), you'll also need roles on the organisation itself.

  • You'll need access to the Google Cloud console at https://console.cloud.google.com.

⚠️ Service account keys may be blocked by default. Google Cloud organisations created on or after 3 May 2024 enforce the policy Disable service account key creation (iam.managed.disableServiceAccountKeyCreation), and some created between February and April 2024 may too. A JSON key is required for this integration, so if key creation is blocked, an Organisation Policy Administrator will need to allow it for the project you're using before you create the key in Step 3.

Step 1: Choose or create a project

We recommend a dedicated project (for example defensecom-logging) so the SIEM resources are kept separate. If you'd rather use an existing project, skip to Step 2.

  1. In the console, click the project picker at the top of the page.

  2. Click New project.

  3. Enter a Project name, select your Organisation and Location (organisation or folder), then click Create.

  4. Make sure the new project is selected in the project picker before continuing.

Note: we need the Project ID (shown under the name), not the display name. It can't be changed after creation.

Step 2: Create the Pub/Sub topic and subscription

  1. Search for Pub/Sub in the top search bar and open it. If prompted, click Enable to enable the Pub/Sub API.

  2. Go to Topics and click Create topic.

  3. Set Topic ID to defensecom-siem.

  4. Untick Add a default subscription. Leaving this ticked creates an extra, unused subscription that keeps storing messages.

  5. Leave the other options as default and click Create.

  6. Go to Subscriptions and click Create subscription.

  7. Set Subscription ID to defensecom-siem-sub.

  8. Under Select a Cloud Pub/Sub topic, choose projects/<PROJECT_ID>/topics/defensecom-siem.

  9. Set Delivery type to Pull.

  10. Leave the remaining settings as default and click Create.

Step 3: Create the service account

Create the account

  1. Go to IAM & Admin → Service accounts and click Create service account.

  2. Set Service account name to defensecom-siem-sa and Description to Defense.com SIEM – Pub/Sub log collection.

  3. Click Create and continue. Skip the optional Permissions and Principals with access steps and click Done. Access is granted on the subscription only, not across the whole project.

Grant access to the subscription

  1. Go to Pub/Sub → Subscriptions and tick defensecom-siem-sub.

  2. In the Permissions panel on the right (click Show info panel if it's hidden), click Add principal.

  3. Enter the service account email (replacing <PROJECT_ID> with your actual project ID): defensecom-siem-sa@<PROJECT_ID>.iam.gserviceaccount.com

  4. Select the role Pub/Sub Subscriber, then click Add another role and select Pub/Sub Viewer.

  5. Click Save.

Subscriber lets the SIEM pull messages; Viewer lets it confirm the subscription exists.

Create the key

  1. Go back to IAM & Admin → Service accounts and click defensecom-siem-sa.

  2. Open the Keys tab and click Add key → Create new key.

  3. Choose JSON and click Create. The key file downloads automatically.

⚠️ Keep the key file safe. It can't be downloaded again. Don't email it or store it in shared drives - see Step 7 for how to send it to us.

Step 4: Create the log sink

  1. Go to Logging → Log Router and click Create sink.

  2. Sink details: enter a name (e.g. defensecom-siem-sink) and an optional description. Click Next.

  3. Sink destination: set Select sink service to Cloud Pub/Sub topic, then choose projects/<PROJECT_ID>/topics/defensecom-siem. Click Next.

  4. Choose logs to include in sink: paste the inclusion filter below, then click Preview logs to check results are returned. Click Next.

log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/policy") OR (log_id("cloudaudit.googleapis.com/data_access")     AND protoPayload.serviceName=("secretmanager.googleapis.com" OR "iamcredentials.googleapis.com"))

  1. Choose logs to filter out of sink: leave blank unless advised by Defense.com.

  2. Click Create sink.

This filter sends only the security-relevant logs we need to monitor your environment:

  • Admin Activity, System Event and Policy Denied audit logs – changes to IAM, service accounts, firewall rules, logging and resources. These are always on and free to generate.

  • Secret access and service account impersonation – read events from Secret Manager and the IAM Service Account Credentials API. These need Data Access audit logging enabled (Step 5).

Everything else stays in your own Cloud Logging, so it's still available if we need it during an investigation.

Check the sink can publish to the topic

When the sink and topic are in the same project, the console grants the sink permission to publish automatically. To confirm:

  1. On the Log Router page, click the ⋮ menu next to the sink and select View sink details. Copy the Writer identity.

  2. Go to Pub/Sub → Topics, tick defensecom-siem and check that the writer identity is listed with the Pub/Sub Publisher role. If it isn't, click Add principal, paste it and grant Pub/Sub Publisher.

Step 5: Enable Data Access audit logs for two services

Data Access audit logs are off by default for most services. Enable them only for the two services below – not for all services, which would generate large volumes of logs.

  1. Go to IAM & Admin → Audit logs.

  2. Tick Secret Manager API. In the panel on the right, tick Data Read and click Save.

  3. Tick Identity and Access Management (IAM) API. In the panel on the right, tick Admin Read and Data Read and click Save.

Note: service account impersonation events (IAM Service Account Credentials API) don't have their own entry on this page – they're enabled through the IAM API. This also turns on some extra IAM read logs in your Cloud Logging, but the sink filter only sends the impersonation events to Defense.com.

Step 6 (optional): Collect logs from all projects with an organisation sink

If you have several projects, one aggregated sink at organisation (or folder) level avoids setting up a sink in each project.

Get the required organisation roles

Creating an organisation sink needs roles granted on the organisation. Roles you hold on a project – even Owner – don't carry upwards. You need one of these on the organisation:

  • Logs Configuration Writer (roles/logging.configWriter) – the minimum for creating sinks

  • Logging Admin (roles/logging.admin)

Enabling Data Access audit logs at organisation level also changes the organisation's IAM policy, so that step needs Organization Administrator (roles/resourcemanager.organizationAdmin) or Security Admin (roles/iam.securityAdmin) on the organisation. To check and grant these:

  1. Go to IAM & Admin → IAM and switch the resource picker at the top from your project to the organisation.

  2. Check whether your user is listed and which roles it has. A user who has only worked at project level often has no organisation roles at all.

  3. If you can grant roles at organisation level, click Grant access, add your user and assign Logs Configuration Writer. If the button is disabled, ask someone with Organization Administrator to do it. In a new Google Workspace or Cloud Identity domain, this is usually the super admin, who can grant themselves Organization Administrator first and then assign the logging role.

  4. Wait a minute or two, refresh the page and continue.

Create the organisation sink

  1. In the project picker, select your organisation (not a project).

  2. Go to Logging → Log Router and click Create sink.

  3. Set the destination to Cloud Pub/Sub topic, select Use a Cloud Pub/Sub topic in another project and enter: pubsub.googleapis.com/projects/<PROJECT_ID>/topics/defensecom-siem

  4. Use the same inclusion filter as in Step 4, then tick Include logs ingested by this organisation and all child resources (the wording may vary slightly). Also enable the Data Access audit logs from Step 5 with the organisation selected.

  5. Create the sink, then copy its Writer identity from View sink details.

  6. Grant that writer identity Pub/Sub Publisher on the defensecom-siem topic (as in Step 4). This is not done automatically for organisation sinks.

Step 7: Check logs are flowing and send us your details

Check logs are flowing

  1. Make a harmless change to generate an audit event (for example, add and remove a label on a project).

  2. Go to Pub/Sub → Subscriptions → defensecom-siem-sub and open the Metrics tab. Within a few minutes, Unacked message count should rise above zero.

Once Defense.com starts collecting, this count should stay near zero.

Send your details to Defense.com

We need:

  • Project ID (where the topic and subscription live)

  • Topic ID (e.g. defensecom-siem)

  • Subscription ID (e.g. defensecom-siem-sub)

  • The JSON key file for defensecom-siem-sa

Send the key using Onetimesecret – never by plain email or chat:

  1. Open the JSON key file in a text editor and copy the entire contents.

  2. Go to https://onetimesecret.com and paste the contents into the secret box.

  3. Set a passphrase and a short expiry (e.g. 24 hours), then click Create a secret link.

  4. Send us the link along with your Project, Topic and Subscription IDs, and share the passphrase separately (e.g. by phone or a different channel).

The link can only be opened once. If it has expired or been viewed before we receive it, just create a new one. Once we confirm logs are arriving, you can delete your local copy of the key file.

Prefer the command line?

Run these in Cloud Shell or with the gcloud CLI, replacing your-project-id.

PROJECT_ID="your-project-id" gcloud config set project "$PROJECT_ID"  # Enable the API gcloud services enable pubsub.googleapis.com  # Topic and subscription gcloud pubsub topics create defensecom-siem gcloud pubsub subscriptions create defensecom-siem-sub --topic=defensecom-siem  # Service account, scoped to the subscription only gcloud iam service-accounts create defensecom-siem-sa \   --display-name="Defense.com SIEM - Pub/Sub log collection"  gcloud pubsub subscriptions add-iam-policy-binding defensecom-siem-sub \   --member="serviceAccount:defensecom-siem-sa@${PROJECT_ID}.iam.gserviceaccount.com" \   --role="roles/pubsub.subscriber"  gcloud pubsub subscriptions add-iam-policy-binding defensecom-siem-sub \   --member="serviceAccount:defensecom-siem-sa@${PROJECT_ID}.iam.gserviceaccount.com" \   --role="roles/pubsub.viewer"  # Log sink (the output shows the writer identity) gcloud logging sinks create defensecom-siem-sink \   "pubsub.googleapis.com/projects/${PROJECT_ID}/topics/defensecom-siem" \   --log-filter='log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/policy") OR (log_id("cloudaudit.googleapis.com/data_access") AND protoPayload.serviceName=("secretmanager.googleapis.com" OR "iamcredentials.googleapis.com"))'  # Grant the sink permission to publish (use the writer identity from the # output above, including the "serviceAccount:" prefix) gcloud pubsub topics add-iam-policy-binding defensecom-siem \   --member="WRITER_IDENTITY" \   --role="roles/pubsub.publisher"  # JSON key gcloud iam service-accounts keys create defensecom-siem-sa-key.json \   --iam-account="defensecom-siem-sa@${PROJECT_ID}.iam.gserviceaccount.com"

For an organisation sink, add --organization=ORG_ID --include-children to the sinks create command. You'll still need to enable the Data Access audit logs in Step 5 through the console.

Troubleshooting

Problem

Likely cause

Fix

Error creating a key: Service account key creation is disabled

An organisation policy blocks keys

Ask your Organisation Policy Administrator to allow key creation for this project

No messages arriving on the subscription

The sink can't publish to the topic

Check the sink's writer identity has Pub/Sub Publisher on the topic (Step 4)

No messages arriving, permissions correct

The filter matches nothing

Run the filter in Logs Explorer and fix any typos

Permission error when creating a sink at organisation level

Missing organisation roles

See Get the required organisation roles in Step 6

Unacked message count keeps rising after go-live

Defense.com can't pull messages

Contact Defense.com support

And that's it! You've successfully integrated Google Cloud Platform 🎉

Did this answer your question?