CloudServer¶
Zenko CloudServer is an open-source Node.js object storage server handling the Amazon S3 protocols.
By providing a free-standing implementation of the S3 API, CloudServer offers developers the freedom to build S3 apps and run them either on-premises, in the AWS public cloud, or both—with no code changes. CloudServer is deployed in a Docker container.
Overview¶
Component | Description |
---|---|
S3 routes | The main S3 service that receives S3-protocol commands. |
Backbeat routes | A special Backbeat-only S3 service that uses Backbeat routes to replicate data to other clouds and update the replication status of the local object, while being authenticated as the internal Backbeat service. |
Management agent | CloudServer establishes an HTTPS connection to Orbit (API Push Server) and uses polling or websockets. The management agent stores the configuration, an in-memory-only overlay of the configuration, in the Metadata service. The same mechanism retrieves statistics from the Backbeat API and, later, to control the Replication service and do the same with other service components. |
Prometheus client | (Not depicted) Monitoring information is maintained in a Prometheus endpoint. Prometheus polls this endpoint for monitoring. |
Metadata backend | A multi-backend interface than can communicate with MongoDB. |
Data backend | A multi-backend interface than can communicate with different clouds (S3, Azure, GCP) while preserving namespace. |
Note
CloudServer also supports bucketd and sproxyd protocol for S3 Connector.
Use Cases¶
As currently implemented with Zenko, CloudServer supports the following use cases.
Direct cloud storage
Users can store data on the managed cloud locations using the S3 protocol, if a cloud location (AWS, Azure, GCP, etc.) and endpoints are configured (using Orbit or a configuration file).
Managing a preferred location for PUTs and GETs
When defining an endpoint, you can define and bind a preferred read location to it. This is a requirement for transient source support.
Objects’ cloud location readable
CloudServer can read objects’ location property.
Direct RING storage (sproxydclient)
CloudServer uses a library called sproxydclient to access the RING through the sproxy daemon (sproxyd).
Direct SOFS storage (cdmiclient)
CloudServer uses a library called cdmiclient to access the SOFS Dewpoint daemon through the CDMI protocol. Both the file system and the S3 environment have their own metadata. The CDMI protocol allows a user to attach custom metadata to an entity (directory/file). This feature is used to save S3 metadata: an entry named “s3metadata” is added to a metadata entity. Its value is the S3 metadata (JSON object). When an object is created from an S3 client, the cloud server produces all the S3 metadata. When a file is created using the file system interface (either using CDMI protocol or a traditional file system client on Dewpoint daemon fuse mountpoint), S3 metadata is reconstituted from POSIX information.
Healthcheck
Currently, a liveness probe calls
/_/healthcheck/deep
. Services that expose readiness can also get a readiness probe.Metrics Collection
These metrics are valid on all Node.js-based services:
Metric | Description |
---|---|
nodejs_version_info | Node.js version info |
nodejs_heap_space_size_available_bytes | Process heap space size available from node.js in bytes |
nodejs_heap_size_total_bytes | Process heap size from node.js in bytes |
nodejs_heap_size_used_bytes | Process heap size used from node.js in bytes |
nodejs_external_memory_bytes | Node.js external memory size in bytes |
nodejs_heap_space_size_total_bytes | Process heap space size total from node.js in bytes |
process_cpu_user_seconds_total | Total user CPU time spent in seconds |
process_cpu_system_seconds_total | Total system CPU time spent in seconds |
process_cpu_seconds_total | Total user and system CPU time spent in seconds |
process_start_time_seconds | Start time of the process since unix epoch in seconds |
process_resident_memory_bytes | Resident memory size in bytes |
nodejs_eventloop_lag_seconds | Lag of event loop in seconds |
nodejs_active_handles_total | Number of active handles |
nodejs_active_requests_total | Number of active requests |
nodejs_heap_space_size_used_bytes | Process heap space size used from node.js in bytes |
Metric | Description |
---|---|
cloud_server_number_of_buckets | Total number of buckets |
cloud_server_number_of_objects | Total number of objects |
cloud_server_data_disk_available | Available data disk storage in bytes |
cloud_server_data_disk_free | Free data disk storage in bytes |
cloud_server_data_disk_total | Total data disk storage in bytes |
Metric | Description |
---|---|
cloud_server_http_requests_total | Total number of HTTP requests |
cloud_server_http_request_duration _microseconds | Duration of HTTP requests in microseconds |
cloud_server_http_request_size_bytes | The HTTP request sizes in bytes |
cloud_server_http_response_size_bytes | The HTTP response sizes in bytes |