Prevent Cloud CDN from serving cached elements?

Question

I have some problem with Cloud CDN. I've added a CDN to a bucket connected on an instance.

I need to get an element from CDN, but this element is updated each 1 sec from my server. What happen is that when from browser i get the first time the file I can get the correct one, then the file is never updated again (file is an m3u8 and i get always the same)

The m3u8 file is updated on the bucket but not updated on the cloud CDN. How can i prevent caching on some type of file (or all files) inside my cloud cdn?

UPDATE:

Honestly, by checking my response header directly from bucket, i get "cache-control: public, max-age=3600" I start to thinking the problem is the bucket, how can i set max-age to 0/1 for entire bucket?

P.s. my file are all public and i'm reading this (https://cloud.google.com/storage/docs/gsutil/commands/setmeta):

Note: By default, publicly readable objects are served with a Cache-Control header allowing such objects to be cached for 3600 seconds. For more details about this default behavior see the CACHE-CONTROL section of gsutil help metadata. If you need to ensure that updates become visible immediately, you should set a Cache-Control header of "Cache-Control:private, max-age=0, no-transform" on such objects.

I just want to avoid run a command like this from a cron each sec

    gsutil setmeta -h "Content-Type:text/html" \
  -h "Cache-Control:public, max-age=0, no-transform" gs://bucket/*.html

There is no general settings to manage an entire bucket directly? (not for each loaded content)

UPDATE: It's an nginx server for live streaming (rtmp).

I got an m3u8 with the indexis of others m3u8 for each quality.

If i mount a bucket on my GCE instance (with gcsfuse), the 1st m3u8 that i get from bucket from my browser will stay cached inside the bucket for 1 hour (max-age=3600)...because of caching.

Files from GCE are not updated with gsutil: are directly written on the mounted bucket with gcsfuse.

To make this files public available, I've setted the bucket to public -> this, for google bucket policy, make them cacheable.

Theorically I need to update all my m3u8 files each 1 sec.

To answer:

1. Please provide more details about what kind of data and in which way you write to the bucket -> m3u8

2. What do you want achieve by using Storage bucket -> shared resource for scalable infrastructure

3. Why do you need to update elements every 5 sec -> need update m3u8 each 1 sec (to update indexes)

That's the problem.

Answer 1

Please have a look at the documentation Cloud Storage FUSE section Key differences from a POSIX file system:

Metadata: Cloud Storage FUSE does not transfer metadata along with the file when uploading to Cloud Storage. This means that if you wish to use Cloud Storage FUSE as an uploading tool, you will not be able to set metadata such as content type and ACLs as you would with other uploading methods. If metadata properties are critical, considering using gsutil, the JSON API or the Google Cloud Console.

Meanwhile, as you can find in the documentation, to change default policies you should use metadata:

See gsutil help metadata for details about how you can set metadata while uploading objects, what metadata fields can be set and the meaning of these fields, use of custom metadata, and how to view currently set metadata.

Because of using Cloud Storage FUSE you're not able to set metadata while uploading objects and, as result, your m3u8 files affected by default cashing policy:

By default, publicly readable objects are served with a Cache-Control header allowing such objects to be cached for 3600 seconds.

As a temporary workaround, you can try to use setmeta command to set metadata on already uploaded objects:

gsutil setmeta -h "Content-Type:text/html" -h "Cache-Control:public, max-age=0, no-transform" gs://bucket/*.html

but it cause a lot of operations and, as result, increase cost of usage.

To solve this issue, you should change architecture of your service. For example, try to use Cloud Filestore to store your m3u8 files.

Answer 2

Set the Cache-Control header in the GCS object metadata to (e.g.) public, max-age=5

This will tell Cloud CDN to cache it for 5 seconds and then refetch from the origin (your bucket) after that time.

You can do this on upload/via the API when you are pushing the new object: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata