ti-chi-bot · GitHub

@gregwebs

@ti-chi-bot ti-chi-bot added the size/XXL

Denotes a PR that changes 1000+ lines, ignoring generated files.

label

Jan 27, 2021

@gregwebs

@gregwebs

@ti-chi-bot ti-chi-bot removed the needs-rebase

Indicates a PR cannot be merged because it has merge conflicts with HEAD.

label

Jan 28, 2021

@gregwebs

gregwebs

yiwu-arbug

@gregwebs

@gregwebs

yiwu-arbug

Merged

@gregwebs

ti-chi-bot pushed a commit that referenced this pull request

Feb 5, 2021

Greg Weber

### What problem does this PR solve?
In principle this util file shouldn't belong exclusively to external_storage and instead should be shareable among other crates.
In practice I needed to make this change for my cloud library refactoring [PR that is WIP](#9576): cloud code depends on this and external_storage will depend on cloud code. That PR is getting very large and I am trying to break out some separate changes from it.
We already have a future.rs file and perhaps the two of these should be re-organized appropriately. However, I am already working on massive refactoring so would like to leave that for someone else to do.
/cc @kennytm @yiwu-arbug @BusyJay
### What is changed and how it works?
move external_storage/util to tikv_util/stream
### Related changes
- #9576 still WIP will rebase to use this commit.
### Check List <!--REMOVE the items that are not applicable-->
Tests
- No code changes, so I am relying on the compiler and test suite
Side effects
None
### Release note
None

@gregwebs

@ti-chi-bot ti-chi-bot removed the needs-rebase

Indicates a PR cannot be merged because it has merge conflicts with HEAD.

label

Feb 7, 2021

Greg Weber and others added 10 commits

February 8, 2021 15:03
Signed-off-by: Greg Weber <greg@pingcap.com>
add a new create encryption/export
Signed-off-by: Greg Weber <greg@pingcap.com>
Signed-off-by: Greg Weber <greg@gregweber.info>
Cloud providers should import TiKV functionality from here.
Currently it only re-exports crates.
In the future it may provide some general cloud adapters.
Signed-off-by: Greg Weber <greg@pingcap.com>
Move KmsProvider into cloud.
Signed-off-by: Greg Weber <greg@pingcap.com>
Signed-off-by: Greg Weber <greg@gregweber.info>
Signed-off-by: Greg Weber <greg@pingcap.com>
add BlobStorage interface to cloud/blob
Signed-off-by: Greg Weber <greg@pingcap.com>
Signed-off-by: Greg Weber <greg@gregweber.info>
Signed-off-by: Greg Weber <greg@gregweber.info>

@gregwebs

Merged

ti-chi-bot pushed a commit that referenced this pull request

Feb 24, 2021
### What problem does this PR solve?
We are organizing the cloud code in a separate crate. This will facilitate the cloud team being able to own the code and modifying builds to include or exclude cloud providers or in the future dynamically link them.
This is the encryption portion split off from this cloud re-organization PR: #9576
After this PR is merged, I will create a second PR that organizes the external_storage code.
### What is changed and how it works?
* AWS code is moved from the encryption crate into the cloud/aws crate
* To further separate out the cloud crate, the encryption crate depends on the `KmsProvider` trait that it exposes. A few types (DataKeyPair, EncryptedKey) are duplicated in the cloud encryption crate to avoid any dependency.
* components/encryption/export is added to avoid cyclical dependencies. Cloud constructors of encryption traits are imported here and then re-exported along with the encryption crate for end users of encryption
* rusoto_util code is moved to aws/util. This is currently duplicated because external_storage still depends on rusoto_util. The next PR will organize the external_storage code and remove the rusoto_util crate.
/cc @yiwu-arbug
### Related changes
This is the encryption portion split off from this cloud re-organization PR: #9576
A second PR will be based off of that to organize the external_storage code.
### Check List
Tests
- Unit test: new unit tests covering the functionality
- Integration test: no new integration tests
- Manual test: basic test of running with encryption and rotating keys
Compatibility
- New cloud error codes are used for AWS KMS operations.
- Programs built from the TiKV lib (tikv-server, etc) now import the encryption_export crate instead of encryption directly.
### Release note
Move KMS cloud provider code out of encryption and into separate cloud provider crates.

Merged

5 tasks

Read the original on github.com ↗