smruthi2187andTa-Ching Chen
31ba992726
All functions have a pkg reference. This can be a package with either source and a deploy archives, or, a deploy archive. Everytime a function is updated, a new package is created. With archive pruner, the archives that are pointed to by old pkg reference can be deleted from the storage. * High level spec for package pruning. * Skeleton for archive pruning * Adding meat 1 to skeleton. * Adding meat #2. Separated storage service into a httpHandler component and Storage Layer component. * Adding meat #3. getOrphanedArchives in pruner and getItems on stowClient. * Restructured archivePruner methods. * Commiting the day's work. Ready for testing #1. * Fixing compile errors. * Test ready. added a few logs for debugging. * Adding a filter for getItems in stowClient. * After testing. * Added a test for archivePruner. * Adding helm value pruneInterval for testing. * Modified test. * Final test. * Fixing interval from seconds to minutes. * Small change. * Changing debugs to info. * Removing the WIP design * Ran gofmt on all these files. * Fixing prune_interval as string in ENV var. * Addressing all comments, but one. * changing getFile method in stowClient to stream it into a response. * All comments incorporated. * Introducing a new flag for running archivePruner. 1. This flag is disabled for archivePruner to run in unit test. 2. This flag is enabled for archivePruner to run in production. 3. Also disabling test_archive_pruner.sh in this PR. Follow up with next PR to enable it. * Addressing review comments. * Changing the command to generate a file dynamically. * Enabling arching_pruner_test * giving execute permissions to test_archive_pruner.sh * Making changes of positional parameters after recent commit. Change test case permission and removing kubectlPortForward. * Adding debug to see why test_utils.sh passed junk pruneInterval. * shell needs special handling for positional parameters from 10.
StorageSvc
StorageSvc consists of 3 components
- Storage http request handler
- StowClient
- ArchivePruner
StorageSvc
This is the HTTP handler that serves requests to :
- upload archive into a storage
- fetch an archive from storage
- delete archive from storage
StowClient
This is the storage interface layer that interacts with stow package. It provides methods to:
- write a file to storage
- retrieve a file from storage
- delete a file from storage
- get all files on storage
ArchivePruner
This acts like a cron job to clean up orphaned archives from storage. By default configured to run every hour. The value can be set in Values.yaml to any preferred interval.