Introduction
Errors may occur during data transmission between the client and the server. Cloud Object Storage (COS) can ensure the integrity of uploaded data through MD5 verification. Only when the MD5 checksum of the data received by the COS server matches that set by the user can the data be uploaded successfully.
Each object in COS corresponds to an ETag, which serves as an information identifier for the object's content at the time of creation. However, the ETag may not necessarily be equivalent to the MD5 checksum of the object's content. Therefore, it cannot be used to verify whether a downloaded object is consistent with the original one. Instead, users can utilize custom object metadata (x-cos-meta-*) to perform consistency checks between the downloaded object and the original.
Data Verification Method
- Verifying the uploaded object
- If users need to verify whether an object uploaded to COS matches the local object, they can set the Content-MD5 field in the HTTP request to the Base64-encoded MD5 checksum of the object's content during upload. In this case, the COS server will verify the uploaded object. The object will only be uploaded successfully if the MD5 checksum received by the COS server matches the user-set Content-MD5 value.
- Verifying the downloaded object
If users need to verify whether the downloaded object matches the original object, they can calculate the object checksum using a verification algorithm during upload, and set the checksum via custom metadata. After downloading the object, users can recalculate the checksum and compare it with the custom metadata for verification. In this approach, users can freely choose the verification algorithm, but for the same object, the algorithm used during upload and download should remain consistent.
API Example
Simple Upload Request
Below is an example request for a user to upload an object. When uploading an object, the user sets the Content-MD5 to the Base64-encoded MD5 checksum of the object content. In this case, the object can be uploaded successfully only when the MD5 checksum of the object received by the COS server matches the Content-MD5 set by the user. Additionally, the custom metadata x-cos-meta-md5 should be set to the checksum value of the object.
Note:
The example demonstrates obtaining the checksum of the object using the MD5 verification algorithm. Users may choose alternative verification algorithms according to their needs.
PUT /exampleobject HTTP/1.1
Host: <BucketName-APPID>.<Endpoint>
Date: Fri, 21 Jun 2019 09:24:28 GMT
Content-Type: image/jpeg
Content-Length: 13
Content-MD5: ti4QvKtVqIJAvZxDbP/c+Q==
Authorization: q-sign-algorithm=sha1&q-ak=************************************&q-sign-time=1561109068;1561116268&q-key-time=1561109068;1561116268&q-header-list=content-length;content-md5;content-type;date;host&q-url-param-list=&q-signature=****************************************
x-cos-meta-md5: b62e10bcab55a88240bd9c436cffdcf9
Connection: close
[Object Content]
Multipart Upload Request
Below is an example request for initializing a multipart upload. When uploading object parts, users can set custom metadata for the object through multipart upload initialization, for example, setting the custom metadata x-cos-meta-md5 as the object's checksum.
POST /exampleobject?uploads HTTP/1.1
Host: <BucketName-APPID>.<Endpoint>
Date: Fri, 21 Jun 2019 09:45:12 GMT
Authorization: q-sign-algorithm=sha1&q-ak=************************************&q-sign-time=1561109068;1561116268&q-key-time=1561109068;1561116268&q-header-list=content-length;content-md5;content-type;date;host&q-url-param-list=&q-signature=****************************************
x-cos-meta-md5: b62e10bcab55a88240bd9c436cffdcf9
Note:
For files uploaded via multipart upload, COS only verifies the MD5 value of each part and does not calculate the MD5 value of the complete file composed of merged parts.
Object Download Response
Below is an example response received after a user sends a request to download an object. From the response, users can obtain the custom metadata x-cos-meta-md5 of the object. By recalculating the checksum of the downloaded object and comparing it with this custom metadata, users can verify whether the downloaded object is consistent with the original object.
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 13
Connection: close
Accept-Ranges: bytes
Cache-Control: max-age=86400
Content-Disposition: attachment; filename=example.jpg
Date: Thu, 04 Jul 2019 11:33:00 GMT
ETag: "b62e10bcab55a88240bd9c436cffdcf9"
Last-Modified: Thu, 04 Jul 2019 11:32:55 GMT
x-cos-request-id: NWQxZGUzZWNfNjI4NWQ2NF9lMWYyXzk1NjFj****
x-cos-meta-md5: b62e10bcab55a88240bd9c436cffdcf9
[Object Content]