Content
Updated by Markus Kahl 17 days ago
<figure class="table op-uc-figure_align-center op-uc-figure"><table class="op-uc-table"><tbody><tr class="op-uc-table--row"><td class="op-uc-table--cell" style="vertical-align:bottom;"><p class="op-uc-p"></p></td><td class="op-uc-table--cell" style="text-align:center;vertical-align:bottom;"><p class="op-uc-p"><strong>Time (CEST/Berlin) on 24.04.2024</strong></p></td><td class="op-uc-table--cell" style="text-align:center;vertical-align:bottom;"><p class="op-uc-p"><strong>Duration</strong></p></td></tr><tr class="op-uc-table--row"><td class="op-uc-table--cell" style="vertical-align:bottom;"><p class="op-uc-p"><strong>all shards</strong></p></td><td class="op-uc-table--cell" style="vertical-align:bottom;"><p class="op-uc-p">7:00 – 8:00</p></td><td class="op-uc-table--cell" style="vertical-align:bottom;"><p class="op-uc-p">60min</p></td></tr></tbody></table></figure>
We deploy all shards at once. The approximate timings for migrations at most 10 minutes across all shards.
We will try to get the progress update background jobs done within the announced maintenance window. These take about 10 minutes for most shards, but around 30 minutes for vips1 and 2. Accordingly these have priority and should be deployed first.
**Plan**
1. Prebuild docker image (##54315)
2. Clear dangling delayed jobs from imported schemas \[1\]
3. `convox env set -a <shard> OPENPROJECT_GOOD__JOB__MAX__THREADS=1` without promote on purpose
4. Deploy first: vips1, vips2, vips3, bim-trials1 (DO NOT SKIP SCALING)
5.
3. Enqueue next: trials1, trials2, trials3 (DO NOT SKIP SCALING)
6. Once done unset max threads
1. `convox env unset -a <shard> OPENPROJECT_GOOD__JOB__MAX__THREADS --promote`
In case jobs have to be run manually, you can perform only the progress update jobs like this. \[2\]
\[1\]
```ruby
def cleanup_delayed_jobs
accs = Multitenancy::Account.all.select do |acc|
acc.switch do
begin
count = ActiveRecord::Base
.connection
.execute("select count(*) as c from delayed_jobs")
.to_a
.first["c"]
count > 0
rescue ActiveRecord::StatementInvalid
false
end
end
end
puts "Cleaning up delayed jobs for #{accs.size} tenants"
accs.each do |acc|
acc.switch do
res = ActiveRecord::Base.connection.execute("delete from delayed_jobs;")
puts "Deleted #{res.cmd_tuples} delayed jobs for #{acc.domain}"
end
end
accs.size
end
```
\[2\]
```ruby
def run_update_progress_jobs
require 'benchmark'
job_query = GoodJob::Execution
.unfinished
.where("serialized_params ->> 'job_class' = 'WorkPackages::UpdateProgressJob'")
count = job_query.count
failed = []
ms = Benchmark.ms do
failed = job_query.select do |ex|
!ex.perform.succeeded?
end
end
puts "#{failed.size} / #{count} failed after #{(ms / 1000.0 / 60.0).round(2)} minutes"
failed
end
failed = run_update_progress_jobs; failed.size
```
We deploy all shards at once. The approximate timings for migrations at most 10 minutes across all shards.
We will try to get the progress update background jobs done within the announced maintenance window. These take about 10 minutes for most shards, but around 30 minutes for vips1 and 2. Accordingly these have priority and should be deployed first.
**Plan**
1. Prebuild docker image (##54315)
3. `convox env set -a <shard> OPENPROJECT_GOOD__JOB__MAX__THREADS=1` without promote on purpose
4. Deploy first: vips1, vips2, vips3, bim-trials1 (DO NOT SKIP SCALING)
5.
3.
6. Once done unset max threads
1. `convox env unset -a <shard> OPENPROJECT_GOOD__JOB__MAX__THREADS --promote`
In case jobs have to be run manually, you can perform only the progress update jobs like this. \[2\]
\[1\]
```ruby
def cleanup_delayed_jobs
accs = Multitenancy::Account.all.select do |acc|
acc.switch do
begin
count = ActiveRecord::Base
.connection
.execute("select count(*) as c from delayed_jobs")
.to_a
.first["c"]
count > 0
rescue ActiveRecord::StatementInvalid
false
end
end
end
puts "Cleaning up delayed jobs for #{accs.size} tenants"
accs.each do |acc|
acc.switch do
res = ActiveRecord::Base.connection.execute("delete from delayed_jobs;")
puts "Deleted #{res.cmd_tuples} delayed jobs for #{acc.domain}"
end
end
accs.size
end
```
\[2\]
```ruby
def run_update_progress_jobs
require 'benchmark'
job_query = GoodJob::Execution
.unfinished
.where("serialized_params ->> 'job_class' = 'WorkPackages::UpdateProgressJob'")
count = job_query.count
failed = []
ms = Benchmark.ms do
failed = job_query.select do |ex|
!ex.perform.succeeded?
end
end
puts "#{failed.size} / #{count} failed after #{(ms / 1000.0 / 60.0).round(2)} minutes"
failed
end
failed = run_update_progress_jobs; failed.size
```