Obnam
Goal Last time I ran out of time and didn't upload chunks with file content data. Today, I'll add that. I'll also upload the SQLite database that has the metadata for files in a backup. Plan When backing up a file, upload its content as a chunk. Add a type for splitting file content into chunks using the simplest possible chunking method, but with an API that allows other ways to split data into…
Goal Today I'll start the journey towards backing up sets of files and restoring them. It will be a long journey, but I have hopefully built a bicycle to make it nicer than having to walk all the way. Specifically, today I want to create a "files database" to contain all the files and their metadata, and the IDs of chunks for their content. The file content will be backed up as chunks. The files…
Goal Today's goal is to finish the work of having a unified way to access local and remote repositories. Plan Change the ClientRepository type to be an enum instead of struct . This will make it possible for obnam/src/bin/cmd/mod::open_repository to return values for local or remote repositories. The current approach of using a trait means only one concrete type can be returned. Change anything…
Goal Today's goal is to start the work of having a unified way to access local and remote repositories. Part of this will exploring what's in the way of that. Plan Simplify error types for the BackupReposiory trait. Decide how repository access needs to be configured, and implement that. Notes Currently the BackupRepository trait requires the implementation for a repository type to specify the…
Goal Today I'm going to do more tidying up. In the near future I'm going to need begin work on actual backups, but I'll want the foundation to be strong and clean by then. Also, it's warm and I'm tired. Plan Change the client so that the LocalRepositoryError error type never used except for local repositories, in the obnam/src/bin/cmd/mod.rs module. Change the client command line parser to be able…
Goal Today I will tidy up various small things. Plan Add clippy lints to prevent many causes of panic. Change chunk labels to be URL friendly in the same way identifiers are. Notes Enabled clippy lints: clippy::panic clippy::panicking_unwrap clippy::unwrap_in_result clippy::unwrap_used This required fixing many places, but all straightforward. Most annoying part of this is that the lints are meant…
Goal My goal today is to finish implementing the plan from last time . Plan See last time. Notes I seem to have made a good plan last time. Well, not all the way, though. The trait I specified for the client repository is fine, but could have a method for opening a client by name: fn open_client(&self, engine: &Engine, wanted: &str) -> Result<(Id, ClientChunk>), ClientRepositoryError>{...} In…
Goal My goal for today is to add an internal interface (Rust type) for access a backup repository, whether it's on local dis or on a server. Plan Define a type or trait for repository access. Implement it for local repository. Implement it for remote repository. Change the client to use new interface. Notes Refactor existing code The current type for using a local repository has the following…
Goal My goal for today is to write client-side code to use the server HTTP API to store chunks. I'll want the client to be able put and get chunks, find chunks by label, and delete chunks. Plan Add obnam api put CHUNKID LABEL --file FILE to upload a file as a chunk. Add obnam api get CHUNKID --output FILE to download a chunk and store it in a file. Add obnam api find LABEL to list IDs of chunks…
Goal The goal for today is to make the server persist chunks on disk. This continues from last time . Plan Start a new branch, cherry pick from the old branch what can needs to be carried over. Create a new module for storing chunks as opaque blogs, with separate metadata. Refactor the client repository module to use that. Change the server to provide persistency using the blob repository. Notes…
Goal The goal for today is to make the server persist chunks on disk. This continues from last time . Plan Change the server API to upload a chunk to accept just the chunk, and parse the chunk to get its metadata. Currently the request gives metadata separately. Change the server to use code from the client for the backup repository to store chunks. This will give us persistence. Add verification…
Goal Today's goal is to make chunk storage persistent in the server. At the moment chunks are stored in memory in the server. This was easy to do and is sufficient for light testing. It's time to make things persistent. I already have code in the client for this, and I can reuse that in the server. Plan Make terminology consistent: refer to where chunks are stored as "the backup repository"…
Goal Last time I started changing how chunk IDs are represented. Today I'll finish that work. Plan Change all code to use new Id and drop the old Id . Notes Reviewing what I did last time, I see I have no doc test that serves as an example for the new Id and IdFactory . That's easy to rectify. Next, the goal is to use the new stuff instead of the old stuff. Generating the old Id was infallible.…
Goal Today I'm a bit tired and don't want to tackle anything complicated. I'll ponder on ways on improving how to represent chunk IDs on the command line, in URLs, and in a database. This has recently been a problem so I'd like to find a solution that will suffice for a long time. Plan Consider needs and wants. Compare possible ways. Pick one. Notes Needs and wants Chunks have a unique identifier.…
Goal Last time I got some code written for the Obnam client, for using the API of the Obnam server. I ran out of time and couldn't finish it. I will continue today. Plan From last time: Implement obnam remote init to set up client for use with a server. Implement obnam remote add --file FILENAMe --id ID --label LABEL to store a chunk on the server. Implement obnam remote get --id ID to retrieve a…
Goal I have an HTTP API on the server, at least enough of one to be able to try to use the server for chunk storage. My goal for today is to change the Obnam client to be able to use that server at all. It doesn't need to be integrated yet, that can come later. However, the client should be able to store a chunk, retrieve it, delete it, and find chunks based on label metadata. Plan Implement obnam…
Goal The goal for today is to combine the separate client and server subplots into one, so that the client and server can be tested against each other. Plan Create at the root obnam.subplot as a union of the client and server ones. It can use separate Markdown files for client and server parts. Make sure make still passes. Notes Created a top level obnam.subplot that uses Markdown files etc at the…
Goal The goal for today is to implement API endpoints to upload and download chunks. Plan Add an endpoint for PUT /chunks/ID?label=LABEL . Add an endpoint for GET /chunks/ID Add verification scenario or scenarios for the endpoints. Notes Preparation: existing code builds and tests pass. in a checkout of main , run make it succeeds all good Create branch chunks . In obnam-server/src/api.rs add a…
Goal The goal of today is to change the acceptance test suite to run the Obnam server as a daemon so that HTTP requests can be made to it. Plan Prototype a Subplot library to manage a daemon: start on request, terminate when scenario ends. Implement this in Python. Notes The daemonize program for various Unix-like operating systems is helpful for this. A more thorough approach would be to use…
Goal The goal for today is to add a Subplot verification scenario for the /validate endpoint, and the scaffolding to have acceptance testing for the Obnam server. Plan Add Subplot scaffolding to obnam-server . Add scenario to verify that /validate accepts a valid API token. Add scenario to verify that /validate refuses an invalid API token. Add scenario to verify that /validate refuses an garbage…
Goal The goal for today is to implement an API end point that validates an access token. This requires adding all the infrastructure to actually provide an HTTP API. Plan Implement a GET /validate end point. The request MUST have an Authorization header with a valid API token. The successful response is a JSON representation of the token. For requests without a valid token, a 401 response is sent.…
Goal The goal for today is the same as last time: The goal for today is to start implementation of the Obnam server HTTP API. If all goes well, after this development session I can run the server, and use authenticated requests to upload chunks, download chunks, delete chunks, and search for chunks based on label. Plan Implement obnam-server token command, with mutually exclusive options…
Goal The goal for today is to start implementation of the Obnam server HTTP API. If all goes well, after this development session I can run the server, and use authenticated requests to upload chunks, download chunks, delete chunks, and search for chunks based on label. Plan I'll use the axum crate for the HTTP API. I've played with it a little bit, and it's pretty convenient to use. I will keep…
Goal Last time I pondered on how to implement remote storage in Obnam. I landed on having an Obnam server that guards access to the actual remote storage, which will be on a server with the S3 API. Alternatively, the Obnam serve can store the backups itself. I didn't say that very explicitly in the note for last time, but I think it's important to not require using an S3 instance, for operational…
Goal The next important thing to build for Obnam3 is support for both local and remote backup storage. There is initial support for local storage. That needs to be reviewed. Needs and wants for backup storage need to be identified and written down. An initial implementation of remote support needs to built. After that, further improvements can be done iteratively. The goal for today is to write…
Obnam will need support for online storage in some way. By that I mean that the backup repository can be on another computer and accessed over the network. The other computer might be in the same room or it might be somewhere distant. It might be owned by the same person running Obnam, or might be rented. It might even just be rented storage rather than a computer as such. I've looked at options,…
Goal Due to work, travel, and illness, I've not touched Obnam in over a month. The goal for today is to get back into it by doing some chores: making sure everything still works, update dependencies that need it, and fix some small issues. Plan Run tests, fix anything that's broken. Run cargo outdated -d1 and update anything that needs to be. Pick an issue to fix. Fix it. Repeat until I ran out of…
Goal Today I want to address some long-standing, but fairly minor issues in the Obnam project. Plan Add some logging and remove debug prints. This requires picking a logging system. That's a decision that can be changed later, so I'm going pick log and env_logger . It's possible we'll want tracing later, for structured logs if nothing else, but I want to start small. The obnam.org web site needs…
Goal I want to open up the project for contributions from others. I think the functionality is now there: there's a bunch of things I'd like to still do, but it might be better to not polish too much. Instead, I'll want to make it easy to start hacking on the code. My goal today is to review all documentation and update it as needed. I'll also want to make sure what is useful to publish is…
Goal Last time I implemented support for OpenPGP card credentials, but ran into problems with actually using them, and noticed that creating a client doesn't create an initial credential. Today I'll merge the card credential changes, and change obnam client init to create at least one credential. I'll remove credential removal from obnam store init . Roadmap I've decided to stop maintaining the…
Goal Today's goal is to add support for OpenPGP hardware card credentials. Roadmap Changes from last time: marked "create credential when init repo" and "initialize chunk repository" as finished picked "OpenPGP card credential" as next task Plan Implement a new credential using the rsoct program by Heiko Schaefer. Testing with hardware can be tricky to automate, but test at least manually.…
Goal Today I'll plan a road map for opening the Obnam 3 project up for contributions from others. I'll also change obnam store init to create at least one credential chunk, which is a big step forward to being able to using encryption for chunks in a convenient way. Plan Make a road map for opening up the project. Change obnam store init to optionally create a credential chunk. Notes Made a road…
Goal Last time I change the configuration file to be able to specify how to get secrets for unlocking credential chunks, and then using those to list only credentials that can be unlocked. Today I'll use that logic to get the client key to unlock the client chunk to get the chunk key to encrypt and decrypt chunks. Plan Tidy up what I did last time to list valid credentials only, by default, and…
Goal Last time I implemented command to create a credential chunk using an OpenPGP software key. Today I will change the commands to encrypt and decrypt chunks to get the client key from the credential. Plan I'll change in the Leaf trait in src/bin/cmd/mod.rs the provided function client_key to get it from one of the available credentials if not provided directly by the user. Before I do that I'll…
Goal Last time I changed things so that the chunk store can handle all kinds of chunks. Today I'll build on that to implement a command to create an OpenPGP software key credential. Plan Implement the obnam credential openpgp-soft command. It can assume an existing OpenPGP key. Notes Preliminary Since last time I've cheated and added a rudimentary architecture document and fixed some small things…
Goal Last time I attempted to change things so that all kinds of chunks can be stored in the chunk store. Status quo is that only data chunks can be, and credential chunks can't. Client chunks get encoded as data chunks, so they also can be. I tried to use a trait for chunks, but this failed, because I wanted the trait to provide serialization and deserialization, and that brought in a lifetime…
Goal Today I will continue on implementing credentials. I have most of the obnam credential openpgp-soft command implemented, but I ran into the problem that it's not of the type that the Store wants to put it in the store. Plan Rather than forcing myself forward, I'll take a step to the side. I'll refactor things so that the store abstraction deals in a new type, "chunk in the store", so that…
Goal Today I will continue on implementing credentials. I have the core functionality implemented, but in a library, which is not useful to users. Today I will add the minimal needed to manage credential chunks on the command line. Plan Implement obnam credential openppp-soft to create a new OpenPGP software key credential. Implement obnam credential list to list all credentials in the backup…
Goal The obnam invocations in the test suite are becoming quite long, because everything is expressed using command line options. This is about to get worse once I implement using credentials for opening a client chunk. I want to keep the options, but I also want to have a configuration file for convenience. Plan I will use my own clingwrap crate too configuration file support. In addition, I'll…
Goal Last time I added some scaffolding to run a SOP implementation. The time before that I rushed into implementing a SOP based credential, but I failed, because I was trying to achieve too much in one session, and I needed the scaffolding to happen first. Today I'll try again to implement a SOP based credential. Technically, it will be a credential using an OpenPGP software key. This is in…
Goal Last time I ran out of time implementing an OpenPGP credential, because I had to implement a bunch of scaffolding for executing programs in a convenient manner, and a SOP implementation in particular. Add tooling for running a SOP implementation to encrypt and decrypt data. Plan Add a wrapper around std::process::Command to make it easy to run a command and inspect the result easier.…
Goal Implement the first credential. Plan Verify that tests still pass, and fix any problems. Write up my thoughts on implementing credentials. Plan implementation of a minimum set of commands to manage credentials. obnam credential new to create a credential obnam credential show to show the client key in a credential Notes Verify tests pass Well, they don't. Since the last time, there's been a…
Goal Continue from where I left off last time and implement commands for client chunk and client key management. Plan Add subcommand obnam client list to list IDs of client chunks. Add subcommand obnam client show to decrypt a client chunk and show its contents. Add subcommand obnam client key generate to generate a new chunk key. Change obnam chnunk subcommands to be able to use keys from the…
Goal I have implemented encrypted chunks, and chunk storage. Next I want to add storing and management of the encryption keys for chunks. A client should be able to have any number of chunk keys. This opens up the possibility of sharing some of the keys with other clients, so that they can also share chunk and de-duplicate across clients. Chunk keys should be randomly generated, not chosen by…
Goal I was inspired to add chunk compression, after thinking about it yesterday. Plan Add compression using the flate2 crate , in a way that the compression can be undone transparently. Since compression can, at least in principle, fail, some interface changes may be necessary to make operations fallible. Notes Add a plain text data type, as an enum with compressed and uncompressed variants.…
Goal I wasn't able to develop Obnam 3 last week, so this week I'm doing a session on Saturday. Last time I implement subcommands for managing a chunk store. This time I'll integrate the chunk store to obnam chunk subcommands. Plan Add a global option obnam --chunks DIR that sets chunk store location. Change obnam store commands to use the location set with the global option. Remove any location…
Goal Last week I got started with a simple chunk repository implementation. It stores chunks in a directory on the local file system. I got as far as implementing a Rust type for this. Today, I will expose the chunk store at the command line. Plan Implement a command obnam store init DIR to initialize a directory as chunk storage. Implement a command obnam store is DIR to check if a directory is…
Goal I have at least initial support for chunk encryption working. It is very rudimentary, and there is no management of keys. My plan is to have a special per-client chunk where chunk encryption keys are stored. Before I start work on that I will want to have a convenient place to store chunks. Basically, I want to be able to say "decrypt the chunk with this ID using the key stored in the client…
Goal The previous eight weeks of developing encryption support has not resulted in success. Last week I came to the conclusion that I was building an internal API for this that was much too complicated. the cipher module needs to have an enum for cipher engines, and engine-independent ways to handle keys and errors, or otherwise hide the different implementations; the API caller should not need to…
Goal I've been hacking on chunk encryption for weeks now. Today I'll take stock and see what I've done and where I am and what I still need to do. Plan Make a new branch. Make sure the test suite passes. If not, fix it. Review the new branch, and rebase it to make sure it tells a coherent, sensible story. Review the test suite: am I missing verification of any important functionality? Can the test…