A fork of Gitea (see branch `mj`) adding Majority Judgment Polls 𐄷 over Issues and Merge Requests. https://git.mieuxvoter.fr
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

411 lines
12 KiB

[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
Move macaron to chi (#14293) Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR. - [x] Define `context.ResponseWriter` interface with an implementation `context.Response`. - [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before. - [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic . - [x] Use https://github.com/unrolled/render instead of macaron's internal render - [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip - [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK** - [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha - [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache - [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding - [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors - [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation` - [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle. - [x] Removed macaron log service because it's not need any more. **BREAK** - [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition. - [x] Move Git HTTP protocol implementation to use routers directly. - [x] Fix the problem that chi routes don't support trailing slash but macaron did. - [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render. Notices: - Chi router don't support request with trailing slash - Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI. Co-authored-by: 6543 <6543@obermui.de>
2 years ago
Move macaron to chi (#14293) Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR. - [x] Define `context.ResponseWriter` interface with an implementation `context.Response`. - [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before. - [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic . - [x] Use https://github.com/unrolled/render instead of macaron's internal render - [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip - [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK** - [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha - [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache - [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding - [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors - [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation` - [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle. - [x] Removed macaron log service because it's not need any more. **BREAK** - [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition. - [x] Move Git HTTP protocol implementation to use routers directly. - [x] Fix the problem that chi routes don't support trailing slash but macaron did. - [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render. Notices: - Chi router don't support request with trailing slash - Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI. Co-authored-by: 6543 <6543@obermui.de>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
9 years ago
Adopt repositories (#12920) * Don't automatically delete repository files if they are present Prior to this PR Gitea would delete any repository files if they are present during creation or migration. This can in certain circumstances lead to data-loss and is slightly unpleasant. This PR provides a mechanism for Gitea to adopt repositories on creation and otherwise requires an explicit flag for deletion. PushCreate is slightly different - the create will cause adoption if that is allowed otherwise it will delete the data if that is allowed. Signed-off-by: Andrew Thornton <art27@cantab.net> * Update swagger Signed-off-by: Andrew Thornton <art27@cantab.net> * Fix tests and migrate overwrite Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @lunny Only offer to adopt or overwrite if the user can do that. Allow the site administrator to adopt or overwrite in all circumstances Signed-off-by: Andrew Thornton <art27@cantab.net> * Use setting.Repository.DefaultBranch for the default branch Signed-off-by: Andrew Thornton <art27@cantab.net> * Always set setting.Repository.DefaultBranch Signed-off-by: Andrew Thornton <art27@cantab.net> * update swagger Signed-off-by: Andrew Thornton <art27@cantab.net> * update templates Signed-off-by: Andrew Thornton <art27@cantab.net> * ensure repo closed Signed-off-by: Andrew Thornton <art27@cantab.net> * Rewrite of adoption as per @6543 and @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * Apply suggestions from code review * update swagger Signed-off-by: Andrew Thornton <art27@cantab.net> * missing not Signed-off-by: Andrew Thornton <art27@cantab.net> * add modals and flash reporting Signed-off-by: Andrew Thornton <art27@cantab.net> * Make the unadopted page searchable Signed-off-by: Andrew Thornton <art27@cantab.net> * Add API Signed-off-by: Andrew Thornton <art27@cantab.net> * Fix swagger Signed-off-by: Andrew Thornton <art27@cantab.net> * fix swagger Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle empty and non-master branched repositories Signed-off-by: Andrew Thornton <art27@cantab.net> * placate lint Signed-off-by: Andrew Thornton <art27@cantab.net> * remove commented out code Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2 years ago
Move macaron to chi (#14293) Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR. - [x] Define `context.ResponseWriter` interface with an implementation `context.Response`. - [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before. - [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic . - [x] Use https://github.com/unrolled/render instead of macaron's internal render - [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip - [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK** - [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha - [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache - [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding - [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors - [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation` - [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle. - [x] Removed macaron log service because it's not need any more. **BREAK** - [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition. - [x] Move Git HTTP protocol implementation to use routers directly. - [x] Fix the problem that chi routes don't support trailing slash but macaron did. - [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render. Notices: - Chi router don't support request with trailing slash - Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI. Co-authored-by: 6543 <6543@obermui.de>
2 years ago
9 years ago
8 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
Add configurable Trust Models (#11712) * Add configurable Trust Models Gitea's default signature verification model differs from GitHub. GitHub uses signatures to verify that the committer is who they say they are - meaning that when GitHub makes a signed commit it must be the committer. The GitHub model prevents re-publishing of commits after revocation of a key and prevents re-signing of other people's commits to create a completely trusted repository signed by one key or a set of trusted keys. The default behaviour of Gitea in contrast is to always display the avatar and information related to a signature. This allows signatures to be decoupled from the committer. That being said, allowing arbitary users to present other peoples commits as theirs is not necessarily desired therefore we have a trust model whereby signatures from collaborators are marked trusted, signatures matching the commit line are marked untrusted and signatures that match a user in the db but not the committer line are marked unmatched. The problem with this model is that this conflicts with Github therefore we need to provide an option to allow users to choose the Github model should they wish to. Signed-off-by: Andrew Thornton <art27@cantab.net> * Adjust locale strings Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @6543 Co-authored-by: 6543 <6543@obermui.de> * Update models/gpg_key.go * Add migration for repository Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2 years ago
Template Repositories (#8768) * Start work on templates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Continue work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix IsTemplate vs IsGenerated Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tabs vs spaces * Tabs vs Spaces * Add templates to API & start adding tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix integration tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Remove unused User Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move template tests to existing repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Minor re-check updates and cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test cleanup Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix optionalbool Signed-off-by: jolheiser <john.olheiser@gmail.com> * make fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Test fixes and icon change Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add new user and repo for tests Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests (finally) Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update meta repo with env variables Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move generation to create page Combine with repo create template Modify API search to prioritize owner for repo Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix tests and coverage Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix swagger and JS lint Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix API searching for own private repos Signed-off-by: jolheiser <john.olheiser@gmail.com> * Change wording Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix repo search test. User had a private repo that didn't show up Signed-off-by: jolheiser <john.olheiser@gmail.com> * Another search test fix Signed-off-by: jolheiser <john.olheiser@gmail.com> * Clarify git content Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Feedback updates Signed-off-by: jolheiser <john.olheiser@gmail.com> * Add topics WIP Signed-off-by: jolheiser <john.olheiser@gmail.com> * Finish adding topics Signed-off-by: jolheiser <john.olheiser@gmail.com> * Update locale Signed-off-by: jolheiser <john.olheiser@gmail.com>
3 years ago
9 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
[RFC] Make archival asynchronous (#11296) * Make archival asynchronous The prime benefit being sought here is for large archives to not clog up the rendering process and cause unsightly proxy timeouts. As a secondary benefit, archive-in-progress is moved out of the way into a /tmp file so that new archival requests for the same commit will not get fulfilled based on an archive that isn't yet finished. This asynchronous system is fairly primitive; request comes in, we'll spawn off a new goroutine to handle it, then we'll mark it as done. Status requests will see if the file exists in the final location, and report the archival as done when it exists. Fixes #11265 * Archive links: drop initial delay to three-quarters of a second Some, or perhaps even most, archives will not take all that long to archive. The archive process starts as soon as the download button is initially clicked, so in theory they could be done quite quickly. Drop the initial delay down to three-quarters of a second to make it more responsive in the common case of the archive being quickly created. * archiver: restructure a little bit to facilitate testing This introduces two sync.Cond pointers to the archiver package. If they're non-nil when we go to process a request, we'll wait until signalled (at all) to proceed. The tests will then create the sync.Cond so that it can signal at-will and sanity-check the state of the queue at different phases. The author believes that nil-checking these two sync.Cond pointers on every archive processing will introduce minimal overhead with no impact on maintainability. * gofmt nit: no space around binary + operator * services: archiver: appease golangci-lint, lock queueMutex Locking/unlocking the queueMutex is allowed, but not required, for Cond.Signal() and Cond.Broadcast(). The magic at play here is just a little too much for golangci-lint, as we take the address of queueMutex and this is mostly used in archiver.go; the variable still gets flagged as unused. * archiver: tests: fix several timing nits Once we've signaled a cond var, it may take some small amount of time for the goroutines released to hit the spot we're wanting them to be at. Give them an appropriate amount of time. * archiver: tests: no underscore in var name, ungh * archiver: tests: Test* is run in a separate context than TestMain We must setup the mutex/cond variables at the beginning of any test that's going to use it, or else these will be nil when the test is actually ran. * archiver: tests: hopefully final tweak Things got shuffled around such that we carefully build up and release requests from the queue, so we can validate the state of the queue at each step. Fix some assertions that no longer hold true as fallout. * repo: Download: restore some semblance of previous behavior When archival was made async, the GET endpoint was only useful if a previous POST had initiated the download. This commit restores the previous behavior, to an extent; we'll now submit the archive request there and return a "202 Accepted" to indicate that it's processing if we didn't manage to complete the request within ~2 seconds of submission. This lets a client directly GET the archive, and gives them some indication that they may attempt to GET it again at a later time. * archiver: tests: simplify a bit further We don't need to risk failure and use time.ParseDuration to get 2 * time.Second. else if isn't really necessary if the conditions are simple enough and lead to the same result. * archiver: tests: resolve potential source of flakiness Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so there's no guarantee we'll actually take that long. If we need longer to not have a false-positive, then so be it. While here, various assert.{Not,}Equal arguments are flipped around so that the wording in error output reflects reality, where the expected argument is second and actual third. * archiver: setup infrastructure for notifying consumers of completion This API will *not* allow consumers to subscribe to specific requests being completed, just *any* request being completed. The caller is responsible for determining if their request is satisfied and waiting again if needed. * repo: archive: make GET endpoint synchronous again If the request isn't complete, this endpoint will now submit the request and wait for completion using the new API. This may still be susceptible to timeouts for larger repos, but other endpoints now exist that the web interface will use to negotiate its way through larger archive processes. * archiver: tests: amend test to include WaitForCompletion() This is a trivial one, so go ahead and include it. * archiver: tests: fix test by calling NewContext() The mutex is otherwise uninitialized, so we need to ensure that we're actually initializing it if we plan to test it. * archiver: tests: integrate new WaitForCompletion a little better We can use this to wait for archives to come in, rather than spinning and hoping with a timeout. * archiver: tests: combine numQueued declaration with next-instruction assignment * routers: repo: reap unused archiving flag from DownloadStatus() This had some planned usage before, indicating whether this request initiated the archival process or not. After several rounds of refactoring, this use was deemed not necessary for much of anything and got boiled down to !complete in all cases. * services: archiver: restructure to use a channel We now offer two forms of waiting for a request: - WaitForCompletion: wait for completion with no timeout - TimedWaitForCompletion: wait for completion with timeout In both cases, we wait for the given request's cchan to close; in the latter case, we do so with the caller-provided timeout. This completely removes the need for busy-wait loops in Download/InitiateDownload, as it's fairly clean to wait on a channel with timeout. * services: archiver: use defer to unlock now that we can This previously carried the lock into the goroutine, but an intermediate step just added the request to archiveInProgress outside of the new goroutine and removed the need for the goroutine to start out with it. * Revert "archiver: tests: combine numQueued declaration with next-instruction assignment" This reverts commit bcc52140238e16680f2e05e448e9be51372afdf5. Revert "archiver: tests: integrate new WaitForCompletion a little better" This reverts commit 9fc8bedb5667d24d3a3c7843dc28a229efffb1e6. Revert "archiver: tests: fix test by calling NewContext()" This reverts commit 709c35685eaaf261ebbb7d3420e3376a4ee8e7f2. Revert "archiver: tests: amend test to include WaitForCompletion()" This reverts commit 75261f56bc05d1fa8ff7e81dcbc0ccd93fdc9d50. * archiver: tests: first attempt at WaitForCompletion() tests * archiver: tests: slight improvement, less busy-loop Just wait for the requests to complete in order, instead of busy-waiting with a timeout. This is slightly less fragile. While here, reverse the arguments of a nearby assert.Equal() so that expected/actual are correct in any test output. * archiver: address lint nits * services: archiver: only close the channel once * services: archiver: use a struct{} for the wait channel This makes it obvious that the channel is only being used as a signal, rather than anything useful being piped through it. * archiver: tests: fix expectations Move the close of the channel into doArchive() itself; notably, before these goroutines move on to waiting on the Release cond. The tests are adjusted to reflect that we can't WaitForCompletion() after they've already completed, as WaitForCompletion() doesn't indicate that they've been released from the queue yet. * archiver: tests: set cchan to nil for comparison * archiver: move ctx.Error's back into the route handlers We shouldn't be setting this in a service, we should just be validating the request that we were handed. * services: archiver: use regex to match a hash This makes sure we don't try and use refName as a hash when it's clearly not one, e.g. heads/pull/foo. * routers: repo: remove the weird /archive/status endpoint We don't need to do this anymore, we can just continue POSTing to the archive/* endpoint until we're told the download's complete. This avoids a potential naming conflict, where a ref could start with "status/" * archiver: tests: bump reasonable timeout to 15s * archiver: tests: actually release timedReq * archiver: tests: run through inFlight instead of manually checking While we're here, add a test for manually re-processing an archive that's already been complete. Re-open the channel and mark it incomplete, so that doArchive can just mark it complete again. * initArchiveLinks: prevent default behavior from clicking * archiver: alias gitea's context, golang context import pending * archiver: simplify logic, just reconstruct slices While the previous logic was perhaps slightly more efficient, the new variant's readability is much improved. * archiver: don't block shutdown on waiting for archive The technique established launches a goroutine to do the wait, which will close a wait channel upon termination. For the timeout case, we also send back a value indicating whether the timeout was hit or not. The timeouts are expected to be relatively small, but still a multi- second delay to shutdown due to this could be unfortunate. * archiver: simplify shutdown logic We can just grab the shutdown channel from the graceful manager instead of constructing a channel to halt the caller and/or pass a result back. * Style issues * Fix mis-merge Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
2 years ago
  1. // Copyright 2014 The Gogs Authors. All rights reserved.
  2. // Copyright 2020 The Gitea Authors. All rights reserved.
  3. // Use of this source code is governed by a MIT-style
  4. // license that can be found in the LICENSE file.
  5. package repo
  6. import (
  7. "errors"
  8. "fmt"
  9. "strings"
  10. "time"
  11. "code.gitea.io/gitea/models"
  12. "code.gitea.io/gitea/modules/base"
  13. "code.gitea.io/gitea/modules/context"
  14. auth "code.gitea.io/gitea/modules/forms"
  15. "code.gitea.io/gitea/modules/log"
  16. "code.gitea.io/gitea/modules/setting"
  17. "code.gitea.io/gitea/modules/web"
  18. archiver_service "code.gitea.io/gitea/services/archiver"
  19. repo_service "code.gitea.io/gitea/services/repository"
  20. )
  21. const (
  22. tplCreate base.TplName = "repo/create"
  23. tplAlertDetails base.TplName = "base/alert_details"
  24. )
  25. // MustBeNotEmpty render when a repo is a empty git dir
  26. func MustBeNotEmpty(ctx *context.Context) {
  27. if ctx.Repo.Repository.IsEmpty {
  28. ctx.NotFound("MustBeNotEmpty", nil)
  29. }
  30. }
  31. // MustBeEditable check that repo can be edited
  32. func MustBeEditable(ctx *context.Context) {
  33. if !ctx.Repo.Repository.CanEnableEditor() || ctx.Repo.IsViewCommit {
  34. ctx.NotFound("", nil)
  35. return
  36. }
  37. }
  38. // MustBeAbleToUpload check that repo can be uploaded to
  39. func MustBeAbleToUpload(ctx *context.Context) {
  40. if !setting.Repository.Upload.Enabled {
  41. ctx.NotFound("", nil)
  42. }
  43. }
  44. func checkContextUser(ctx *context.Context, uid int64) *models.User {
  45. orgs, err := models.GetOrgsCanCreateRepoByUserID(ctx.User.ID)
  46. if err != nil {
  47. ctx.ServerError("GetOrgsCanCreateRepoByUserID", err)
  48. return nil
  49. }
  50. if !ctx.User.IsAdmin {
  51. orgsAvailable := []*models.User{}
  52. for i := 0; i < len(orgs); i++ {
  53. if orgs[i].CanCreateRepo() {
  54. orgsAvailable = append(orgsAvailable, orgs[i])
  55. }
  56. }
  57. ctx.Data["Orgs"] = orgsAvailable
  58. } else {
  59. ctx.Data["Orgs"] = orgs
  60. }
  61. // Not equal means current user is an organization.
  62. if uid == ctx.User.ID || uid == 0 {
  63. return ctx.User
  64. }
  65. org, err := models.GetUserByID(uid)
  66. if models.IsErrUserNotExist(err) {
  67. return ctx.User
  68. }
  69. if err != nil {
  70. ctx.ServerError("GetUserByID", fmt.Errorf("[%d]: %v", uid, err))
  71. return nil
  72. }
  73. // Check ownership of organization.
  74. if !org.IsOrganization() {
  75. ctx.Error(403)
  76. return nil
  77. }
  78. if !ctx.User.IsAdmin {
  79. canCreate, err := org.CanCreateOrgRepo(ctx.User.ID)
  80. if err != nil {
  81. ctx.ServerError("CanCreateOrgRepo", err)
  82. return nil
  83. } else if !canCreate {
  84. ctx.Error(403)
  85. return nil
  86. }
  87. } else {
  88. ctx.Data["Orgs"] = orgs
  89. }
  90. return org
  91. }
  92. func getRepoPrivate(ctx *context.Context) bool {
  93. switch strings.ToLower(setting.Repository.DefaultPrivate) {
  94. case setting.RepoCreatingLastUserVisibility:
  95. return ctx.User.LastRepoVisibility
  96. case setting.RepoCreatingPrivate:
  97. return true
  98. case setting.RepoCreatingPublic:
  99. return false
  100. default:
  101. return ctx.User.LastRepoVisibility
  102. }
  103. }
  104. // Create render creating repository page
  105. func Create(ctx *context.Context) {
  106. ctx.Data["Title"] = ctx.Tr("new_repo")
  107. // Give default value for template to render.
  108. ctx.Data["Gitignores"] = models.Gitignores
  109. ctx.Data["LabelTemplates"] = models.LabelTemplates
  110. ctx.Data["Licenses"] = models.Licenses
  111. ctx.Data["Readmes"] = models.Readmes
  112. ctx.Data["readme"] = "Default"
  113. ctx.Data["private"] = getRepoPrivate(ctx)
  114. ctx.Data["IsForcedPrivate"] = setting.Repository.ForcePrivate
  115. ctx.Data["default_branch"] = setting.Repository.DefaultBranch
  116. ctxUser := checkContextUser(ctx, ctx.QueryInt64("org"))
  117. if ctx.Written() {
  118. return
  119. }
  120. ctx.Data["ContextUser"] = ctxUser
  121. ctx.Data["repo_template_name"] = ctx.Tr("repo.template_select")
  122. templateID := ctx.QueryInt64("template_id")
  123. if templateID > 0 {
  124. templateRepo, err := models.GetRepositoryByID(templateID)
  125. if err == nil && templateRepo.CheckUnitUser(ctxUser, models.UnitTypeCode) {
  126. ctx.Data["repo_template"] = templateID
  127. ctx.Data["repo_template_name"] = templateRepo.Name
  128. }
  129. }
  130. ctx.Data["CanCreateRepo"] = ctx.User.CanCreateRepo()
  131. ctx.Data["MaxCreationLimit"] = ctx.User.MaxCreationLimit()
  132. ctx.HTML(200, tplCreate)
  133. }
  134. func handleCreateError(ctx *context.Context, owner *models.User, err error, name string, tpl base.TplName, form interface{}) {
  135. switch {
  136. case models.IsErrReachLimitOfRepo(err):
  137. ctx.RenderWithErr(ctx.Tr("repo.form.reach_limit_of_creation", owner.MaxCreationLimit()), tpl, form)
  138. case models.IsErrRepoAlreadyExist(err):
  139. ctx.Data["Err_RepoName"] = true
  140. ctx.RenderWithErr(ctx.Tr("form.repo_name_been_taken"), tpl, form)
  141. case models.IsErrRepoFilesAlreadyExist(err):
  142. ctx.Data["Err_RepoName"] = true
  143. switch {
  144. case ctx.IsUserSiteAdmin() || (setting.Repository.AllowAdoptionOfUnadoptedRepositories && setting.Repository.AllowDeleteOfUnadoptedRepositories):
  145. ctx.RenderWithErr(ctx.Tr("form.repository_files_already_exist.adopt_or_delete"), tpl, form)
  146. case setting.Repository.AllowAdoptionOfUnadoptedRepositories:
  147. ctx.RenderWithErr(ctx.Tr("form.repository_files_already_exist.adopt"), tpl, form)
  148. case setting.Repository.AllowDeleteOfUnadoptedRepositories:
  149. ctx.RenderWithErr(ctx.Tr("form.repository_files_already_exist.delete"), tpl, form)
  150. default:
  151. ctx.RenderWithErr(ctx.Tr("form.repository_files_already_exist"), tpl, form)
  152. }
  153. case models.IsErrNameReserved(err):
  154. ctx.Data["Err_RepoName"] = true
  155. ctx.RenderWithErr(ctx.Tr("repo.form.name_reserved", err.(models.ErrNameReserved).Name), tpl, form)
  156. case models.IsErrNamePatternNotAllowed(err):
  157. ctx.Data["Err_RepoName"] = true
  158. ctx.RenderWithErr(ctx.Tr("repo.form.name_pattern_not_allowed", err.(models.ErrNamePatternNotAllowed).Pattern), tpl, form)
  159. default:
  160. ctx.ServerError(name, err)
  161. }
  162. }
  163. // CreatePost response for creating repository
  164. func CreatePost(ctx *context.Context) {
  165. form := web.GetForm(ctx).(*auth.CreateRepoForm)
  166. ctx.Data["Title"] = ctx.Tr("new_repo")
  167. ctx.Data["Gitignores"] = models.Gitignores
  168. ctx.Data["LabelTemplates"] = models.LabelTemplates
  169. ctx.Data["Licenses"] = models.Licenses
  170. ctx.Data["Readmes"] = models.Readmes
  171. ctx.Data["CanCreateRepo"] = ctx.User.CanCreateRepo()
  172. ctx.Data["MaxCreationLimit"] = ctx.User.MaxCreationLimit()
  173. ctxUser := checkContextUser(ctx, form.UID)
  174. if ctx.Written() {
  175. return
  176. }
  177. ctx.Data["ContextUser"] = ctxUser
  178. if ctx.HasError() {
  179. ctx.HTML(200, tplCreate)
  180. return
  181. }
  182. var repo *models.Repository
  183. var err error
  184. if form.RepoTemplate > 0 {
  185. opts := models.GenerateRepoOptions{
  186. Name: form.RepoName,
  187. Description: form.Description,
  188. Private: form.Private,
  189. GitContent: form.GitContent,
  190. Topics: form.Topics,
  191. GitHooks: form.GitHooks,
  192. Webhooks: form.Webhooks,
  193. Avatar: form.Avatar,
  194. IssueLabels: form.Labels,
  195. }
  196. if !opts.IsValid() {
  197. ctx.RenderWithErr(ctx.Tr("repo.template.one_item"), tplCreate, form)
  198. return
  199. }
  200. templateRepo := getRepository(ctx, form.RepoTemplate)
  201. if ctx.Written() {
  202. return
  203. }
  204. if !templateRepo.IsTemplate {
  205. ctx.RenderWithErr(ctx.Tr("repo.template.invalid"), tplCreate, form)
  206. return
  207. }
  208. repo, err = repo_service.GenerateRepository(ctx.User, ctxUser, templateRepo, opts)
  209. if err == nil {
  210. log.Trace("Repository generated [%d]: %s/%s", repo.ID, ctxUser.Name, repo.Name)
  211. ctx.Redirect(setting.AppSubURL + "/" + ctxUser.Name + "/" + repo.Name)
  212. return
  213. }
  214. } else {
  215. repo, err = repo_service.CreateRepository(ctx.User, ctxUser, models.CreateRepoOptions{
  216. Name: form.RepoName,
  217. Description: form.Description,
  218. Gitignores: form.Gitignores,
  219. IssueLabels: form.IssueLabels,
  220. License: form.License,
  221. Readme: form.Readme,
  222. IsPrivate: form.Private || setting.Repository.ForcePrivate,
  223. DefaultBranch: form.DefaultBranch,
  224. AutoInit: form.AutoInit,
  225. IsTemplate: form.Template,
  226. TrustModel: models.ToTrustModel(form.TrustModel),
  227. })
  228. if err == nil {
  229. log.Trace("Repository created [%d]: %s/%s", repo.ID, ctxUser.Name, repo.Name)
  230. ctx.Redirect(setting.AppSubURL + "/" + ctxUser.Name + "/" + repo.Name)
  231. return
  232. }
  233. }
  234. handleCreateError(ctx, ctxUser, err, "CreatePost", tplCreate, &form)
  235. }
  236. // Action response for actions to a repository
  237. func Action(ctx *context.Context) {
  238. var err error
  239. switch ctx.Params(":action") {
  240. case "watch":
  241. err = models.WatchRepo(ctx.User.ID, ctx.Repo.Repository.ID, true)
  242. case "unwatch":
  243. err = models.WatchRepo(ctx.User.ID, ctx.Repo.Repository.ID, false)
  244. case "star":
  245. err = models.StarRepo(ctx.User.ID, ctx.Repo.Repository.ID, true)
  246. case "unstar":
  247. err = models.StarRepo(ctx.User.ID, ctx.Repo.Repository.ID, false)
  248. case "accept_transfer":
  249. err = acceptOrRejectRepoTransfer(ctx, true)
  250. case "reject_transfer":
  251. err = acceptOrRejectRepoTransfer(ctx, false)
  252. case "desc": // FIXME: this is not used
  253. if !ctx.Repo.IsOwner() {
  254. ctx.Error(404)
  255. return
  256. }
  257. ctx.Repo.Repository.Description = ctx.Query("desc")
  258. ctx.Repo.Repository.Website = ctx.Query("site")
  259. err = models.UpdateRepository(ctx.Repo.Repository, false)
  260. }
  261. if err != nil {
  262. ctx.ServerError(fmt.Sprintf("Action (%s)", ctx.Params(":action")), err)
  263. return
  264. }
  265. ctx.RedirectToFirst(ctx.Query("redirect_to"), ctx.Repo.RepoLink)
  266. }
  267. func acceptOrRejectRepoTransfer(ctx *context.Context, accept bool) error {
  268. repoTransfer, err := models.GetPendingRepositoryTransfer(ctx.Repo.Repository)
  269. if err != nil {
  270. return err
  271. }
  272. if err := repoTransfer.LoadAttributes(); err != nil {
  273. return err
  274. }
  275. if !repoTransfer.CanUserAcceptTransfer(ctx.User) {
  276. return errors.New("user does not have enough permissions")
  277. }
  278. if accept {
  279. if err := repo_service.TransferOwnership(repoTransfer.Doer, repoTransfer.Recipient, ctx.Repo.Repository, repoTransfer.Teams); err != nil {
  280. return err
  281. }
  282. ctx.Flash.Success(ctx.Tr("repo.settings.transfer.success"))
  283. } else {
  284. if err := models.CancelRepositoryTransfer(ctx.Repo.Repository); err != nil {
  285. return err
  286. }
  287. ctx.Flash.Success(ctx.Tr("repo.settings.transfer.rejected"))
  288. }
  289. ctx.Redirect(ctx.Repo.Repository.HTMLURL())
  290. return nil
  291. }
  292. // RedirectDownload return a file based on the following infos:
  293. func RedirectDownload(ctx *context.Context) {
  294. var (
  295. vTag = ctx.Params("vTag")
  296. fileName = ctx.Params("fileName")
  297. )
  298. tagNames := []string{vTag}
  299. curRepo := ctx.Repo.Repository
  300. releases, err := models.GetReleasesByRepoIDAndNames(models.DefaultDBContext(), curRepo.ID, tagNames)
  301. if err != nil {
  302. if models.IsErrAttachmentNotExist(err) {
  303. ctx.Error(404)
  304. return
  305. }
  306. ctx.ServerError("RedirectDownload", err)
  307. return
  308. }
  309. if len(releases) == 1 {
  310. release := releases[0]
  311. att, err := models.GetAttachmentByReleaseIDFileName(release.ID, fileName)
  312. if err != nil {
  313. ctx.Error(404)
  314. return
  315. }
  316. if att != nil {
  317. ctx.Redirect(att.DownloadURL())
  318. return
  319. }
  320. }
  321. ctx.Error(404)
  322. }
  323. // Download an archive of a repository
  324. func Download(ctx *context.Context) {
  325. uri := ctx.Params("*")
  326. aReq := archiver_service.DeriveRequestFrom(ctx, uri)
  327. if aReq == nil {
  328. ctx.Error(404)
  329. return
  330. }
  331. downloadName := ctx.Repo.Repository.Name + "-" + aReq.GetArchiveName()
  332. complete := aReq.IsComplete()
  333. if !complete {
  334. aReq = archiver_service.ArchiveRepository(aReq)
  335. complete = aReq.WaitForCompletion(ctx)
  336. }
  337. if complete {
  338. ctx.ServeFile(aReq.GetArchivePath(), downloadName)
  339. } else {
  340. ctx.Error(404)
  341. }
  342. }
  343. // InitiateDownload will enqueue an archival request, as needed. It may submit
  344. // a request that's already in-progress, but the archiver service will just
  345. // kind of drop it on the floor if this is the case.
  346. func InitiateDownload(ctx *context.Context) {
  347. uri := ctx.Params("*")
  348. aReq := archiver_service.DeriveRequestFrom(ctx, uri)
  349. if aReq == nil {
  350. ctx.Error(404)
  351. return
  352. }
  353. complete := aReq.IsComplete()
  354. if !complete {
  355. aReq = archiver_service.ArchiveRepository(aReq)
  356. complete, _ = aReq.TimedWaitForCompletion(ctx, 2*time.Second)
  357. }
  358. ctx.JSON(200, map[string]interface{}{
  359. "complete": complete,
  360. })
  361. }