Skip to content

Original source: https://input.scs.community/how-to-s3#; written by Friedrich Zahn.

Swift+S3 on Rook

The official "guide" contains various inaccuracies and there are various caveats when deploying this setup on Yaook.

The Ceph Operator will not always immediately propagate config changes. Make sure to delete the rgw ReplicaSet often!

Preparations

Keystone user credentials

Copy the keystone-admin Secret from the yaook namespace to the rook-ceph namespace.

You MUST add OS_IDENTITY_API_VERSION: Mw== (decoded value: 3), otherwise the rgw will fall back to Keystone API v2.0, which is deprecated since ca. forever.

Alternatively, you can create a KeystoneUser, but that is discouraged by the docs.

CA bundle

Get the keystone-ca-certificates-... ConfigMap. Extract the value (loooong cert chain) for ca-bundle.crt

Create (not apply! Annotation may exceed 256 kiB) a Secret keystone-ca-certificates in the rook-ceph namespace with that chain (base64 encoded!) under the key cabundle

TLS Certificates for endpoints

We need a ClusterIssuer and Certificate:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: root-ca


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ceph-rgw-objectstorage
  namespace: rook-ceph
spec:
  commonName: ceph-rgw-objectstorage
  dnsNames:
  - rook-ceph-rgw-objectstorage.rook-ceph.svc
  duration: 720h
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  renewBefore: 168h
  revisionHistoryLimit: 3
  secretName: ceph-rgw-objectstorage-certificate
  subject:
    organizations:
    - yaook

This certificate is not recognized by the yaookctl openstack shell, thus all interaction with the endpoint has to be via http or with TLS verification disabled.

Roll-out

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: objectstorage
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 1
  dataPool:
    failureDomain: host
    replicated:
      size: 1
  auth:
    keystone:
      acceptedRoles:
        - admin
        - member
        - service
      implicitTenants: "true"
      revocationInterval: 1200
      serviceUserSecretName: keystone-admin
      tokenCacheSize: 1000
      url: https://keystone.yaook.svc:5000
  protocols:
    swift:
      accountInUrl: true
      urlPrefix: swift
    s3:
      authUseKeystone: true
    # note that s3 is enabled by default if protocols.s3.enabled is not explicitly set to false
  preservePoolsOnDelete: true
  gateway:
    caBundleRef: keystone-ca-certificates
    sslCertificateRef: ceph-rgw-objectstorage-certificate
    port: 80
    securePort: 443
    instances: 1

OpenStack integration

::: warn openstackclient caches the catalog. Do not use interactive mode to check it.

:::

Using openstackclient to add the service and endpoints (guide has wrong URLs):

openstack service create --name swift object-store

openstack endpoint create swift internal "https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s" --region YaookRegion

openstack endpoint create swift admin "https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s" --region YaookRegion

Alternatively you can create a KeystoneEndpoint, although that is discouraged by the docs:

apiVersion: yaook.cloud/v1
kind: KeystoneEndpoint
metadata:
  name: ceph-rgw-objectstorage-endpoint
  namespace: yaook
spec:
  description: OpenStack Object Storage
  endpoints:
    admin: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
    internal: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
    public: https://rook-ceph-rgw-objectstorage.rook-ceph.svc/swift/v1/AUTH_%(tenant_id)s
  keystoneRef:
    kind: KeystoneDeployment
    name: keystone
  region:
    name: YaookRegion
  servicename: swift
  servicetype: object-store

Testing

E.g. on a yaookctl openstack shell :

openstack --insecure container create foobar
openstack --insecure container list

openstack ec2 credentials create
pip install boto3
python
import boto3
s3 = boto3.client('s3', aws_access_key_id="<access>", aws_secret_access_key="<secret>", endpoint_url="https://rook-ceph-rgw-objectstorage.rook-ceph.svc", verify=False)
s3.create_bucket(Bucket="trollololol")
s3.list_buckets()

Findings

  • the Rook/Ceph guide on swift emulation does not work as-is, there are wrong URLs, missing components, too little explanation
  • it is way too easy to mess this up, and debugging is super painful due to the many layers of services and logs that are very verbose, but often contain very little actual information
  • scs-check in the end still fails due to self-signed certificates