CloudServer

Zenko CloudServer is an open-source Node.js object storage server handling the Amazon S3 protocols.

By providing a free-standing implementation of the S3 API, CloudServer offers developers the freedom to build S3 apps and run them either on-premises, in the AWS public cloud, or both—with no code changes. CloudServer is deployed in a Docker container.

Overview

Component	Description
S3 routes	The main S3 service that receives S3-protocol commands.
Backbeat routes	A special Backbeat-only S3 service that uses Backbeat routes to replicate data to other clouds and update the replication status of the local object, while being authenticated as the internal Backbeat service.
Management agent	CloudServer establishes an HTTPS connection to Orbit (API Push Server) and uses polling or websockets. The management agent stores the configuration, an in-memory-only overlay of the configuration, in the Metadata service. The same mechanism retrieves statistics from the Backbeat API and, later, to control the Replication service and do the same with other service components.
Prometheus client	(Not depicted) Monitoring information is maintained in a Prometheus endpoint. Prometheus polls this endpoint for monitoring.
Metadata backend	A multi-backend interface than can communicate with MongoDB.
Data backend	A multi-backend interface than can communicate with different clouds (S3, Azure, GCP) while preserving namespace.

Note

CloudServer also supports bucketd and sproxyd protocol for S3 Connector.

Use Cases

As currently implemented with Zenko, CloudServer supports the following use cases.

• Direct cloud storage

  Users can store data on the managed cloud locations using the S3 protocol, if a cloud location (AWS, Azure, GCP, etc.) and endpoints are configured (using Orbit or a configuration file).

• Managing a preferred location for PUTs and GETs

  When defining an endpoint, you can define and bind a preferred read location to it. This is a requirement for transient source support.

• Objects’ cloud location readable

  CloudServer can read objects’ location property.

• Direct RING storage (sproxydclient)

  CloudServer uses a library called sproxydclient to access the RING through the sproxy daemon (sproxyd).

• Direct SOFS storage (cdmiclient)

  CloudServer uses a library called cdmiclient to access the SOFS Dewpoint daemon through the CDMI protocol. Both the file system and the S3 environment have their own metadata. The CDMI protocol allows a user to attach custom metadata to an entity (directory/file). This feature is used to save S3 metadata: an entry named “s3metadata” is added to a metadata entity. Its value is the S3 metadata (JSON object). When an object is created from an S3 client, the cloud server produces all the S3 metadata. When a file is created using the file system interface (either using CDMI protocol or a traditional file system client on Dewpoint daemon fuse mountpoint), S3 metadata is reconstituted from POSIX information.

• Healthcheck

  Currently, a liveness probe calls /_/healthcheck/deep. Services that expose readiness can also get a readiness probe.

• Metrics Collection

  These metrics are valid on all Node.js-based services:

  Node.js Process General Metrics

  Metric	Description
  nodejs_version_info	Node.js version info
  nodejs_heap_space_size_available_bytes	Process heap space size available from node.js in bytes
  nodejs_heap_size_total_bytes	Process heap size from node.js in bytes
  nodejs_heap_size_used_bytes	Process heap size used from node.js in bytes
  nodejs_external_memory_bytes	Node.js external memory size in bytes
  nodejs_heap_space_size_total_bytes	Process heap space size total from node.js in bytes
  process_cpu_user_seconds_total	Total user CPU time spent in seconds
  process_cpu_system_seconds_total	Total system CPU time spent in seconds
  process_cpu_seconds_total	Total user and system CPU time spent in seconds
  process_start_time_seconds	Start time of the process since unix epoch in seconds
  process_resident_memory_bytes	Resident memory size in bytes
  nodejs_eventloop_lag_seconds	Lag of event loop in seconds
  nodejs_active_handles_total	Number of active handles
  nodejs_active_requests_total	Number of active requests
  nodejs_heap_space_size_used_bytes	Process heap space size used from node.js in bytes

  Cloud Server General Metrics

  Metric	Description
  cloud_server_number_of_buckets	Total number of buckets
  cloud_server_number_of_objects	Total number of objects
  cloud_server_data_disk_available	Available data disk storage in bytes
  cloud_server_data_disk_free	Free data disk storage in bytes
  cloud_server_data_disk_total	Total data disk storage in bytes

  Labeled Metrics

  Metric	Description
  cloud_server_http_requests_total	Total number of HTTP requests
  cloud_server_http_request_duration _microseconds	Duration of HTTP requests in microseconds
  cloud_server_http_request_size_bytes	The HTTP request sizes in bytes
  cloud_server_http_response_size_bytes	The HTTP response sizes in bytes