Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Public Reverse Proxy #22

Open
wants to merge 139 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
6c2a5df
1/20 hanayo connections go to k8s
cmyui Nov 5, 2023
23b51c1
start the rollout
cmyui Nov 5, 2023
ea723e8
1/20 conns on akatsuki-api go to k8s
cmyui Nov 5, 2023
61882d8
roll out hanayo & api k8s to 10% of conns
cmyui Nov 5, 2023
ab0c28b
1/20 score service conns go to k8s
cmyui Nov 5, 2023
45f8139
remove least_conn and 2x score-submission k8s rate
cmyui Nov 5, 2023
9da30ca
2x k8s rate for score-service
cmyui Nov 5, 2023
62101f0
2x api, hanayo, score service conns to k8s
cmyui Nov 5, 2023
afbbbb7
/2 score service k8s exposure
cmyui Nov 5, 2023
4b3c909
fully roll out score-service to k8s
cmyui Nov 5, 2023
a05754f
fully roll out api & hanayo to k8s
cmyui Nov 5, 2023
9dda2f8
roll out performance service to 1/20 conns to perf service api
cmyui Nov 6, 2023
ab36847
fallback to k8s bancho services
cmyui Nov 6, 2023
3e291bb
2x performance service rate to k8s
cmyui Nov 6, 2023
e43ca92
send a fraction of connections to profile history to k8s
cmyui Nov 6, 2023
e7838ef
disable score-service k8s
cmyui Nov 9, 2023
69b5b29
add k8s-worker02 node
cmyui Nov 9, 2023
63ae7da
perf service k8s only as backup
cmyui Nov 9, 2023
098ac4d
roll out performance to some conns
cmyui Nov 10, 2023
0c7d3f1
roll out score-service to some conns
cmyui Nov 10, 2023
dcee758
k8s master for score & perf services
cmyui Nov 10, 2023
c9554c7
send small num of conns to score-service without cache
cmyui Nov 10, 2023
ab919e8
roll out no-cache score service to 30% of conns
cmyui Nov 10, 2023
cdd7fb1
score service ovh only as backup
cmyui Nov 10, 2023
b6b2ff1
send 5% of rework-fe calls to k8s
cmyui Nov 11, 2023
c284422
rework-fe all k8s with ovh backup
cmyui Nov 11, 2023
91eef8e
Revert "rework-fe all k8s with ovh backup"
cmyui Nov 11, 2023
00e04ee
leave rework-fe k8s for tomorrow after perf-service sql usage is opti…
cmyui Nov 11, 2023
e2332e3
roll out profile history api to k8s 100%
cmyui Nov 11, 2023
54e74e4
move sfo node to toronto
cmyui Nov 17, 2023
c2319a6
move payments to k8s
cmyui Nov 18, 2023
331fc3d
add & lb between another bancho instance on port=5002
cmyui Nov 19, 2023
36188c8
only do previous commit for 1/20 conns
cmyui Nov 19, 2023
a2b2c0e
double capacity to new bancho node
cmyui Nov 19, 2023
f416935
add 5003 bancho instance
cmyui Nov 19, 2023
95ebc01
even dist. between all bancho instances
cmyui Nov 19, 2023
a738c7e
expose air conditioning service on nginx
tsunyoku Dec 16, 2023
7f212be
set custom max body size
tsunyoku Dec 16, 2023
f2ea5e4
there is an attempt
tsunyoku Dec 16, 2023
840d534
attempts to allow loader-hub
tsunyoku Dec 16, 2023
aec8890
route profile backgrounds to k8s
tsunyoku Dec 19, 2023
b8e3717
temporarily prioritise bare metal hanayo
tsunyoku Mar 24, 2024
ff7f13c
move hanayo priority back to k8s
tsunyoku Mar 24, 2024
9dc3d15
web_zone ratelimit for hanayo requests
cmyui Mar 24, 2024
7ff7c00
disable web_zone ratelimit on /static
cmyui Mar 24, 2024
2783707
add beatmaps service
tsunyoku Mar 25, 2024
c7559c3
add k8s-worker02 node
cmyui Apr 14, 2024
e06aa95
remove master node
cmyui Apr 14, 2024
e054423
remove weird backups
cmyui Apr 14, 2024
13bddd0
temp log 2xx and 3xx
cmyui Apr 14, 2024
253dfbd
move phpmyadmin ref here from admin-panel
cmyui Apr 21, 2024
39ab810
proper:tm: php execution for phpmyadmin
cmyui Apr 21, 2024
6564b6d
optional trailing / for phpmyadmin
cmyui Apr 21, 2024
c7aa6bb
revert
cmyui Apr 21, 2024
a7b08a2
add some more bancho instances (mrekk)
tsunyoku May 11, 2024
8cc03f7
send rework-frontend conns to k8s
cmyui May 26, 2024
ed01ffc
redir `/api/v1/pp` to score-service
tsunyoku May 26, 2024
29507c3
route beatmaps-service to k8s (#5)
tsunyoku May 26, 2024
f8ec0b6
Add bare metal as backup (#6)
tsunyoku May 26, 2024
00aea01
remove bare metal backup for beatmaps-service (#7)
tsunyoku May 26, 2024
60d0e6b
phased k8s rollout for air conditioning (#9)
tsunyoku May 26, 2024
f5d7c3d
Merge branch 'main' into gradual-k8s-rollout
cmyui May 26, 2024
00899ac
point ac to baremetal
cmyui May 26, 2024
a6b680c
move branch to ovh-rev-proxy
cmyui May 26, 2024
aaf642a
deploy
cmyui May 26, 2024
adb0bc0
phased rollout ac to k8s (#11)
tsunyoku May 26, 2024
3937927
separate ac websocket upstream (#12)
tsunyoku May 26, 2024
fe863dd
attempt to websocket on k8s (#13)
tsunyoku May 26, 2024
b1adbee
remove bare metal backup for ac (#14)
tsunyoku May 26, 2024
83ce152
attempt to remove static from public rev proxy (#16)
tsunyoku May 26, 2024
1861789
separate pypi2 and expose pypi to k8s
cmyui May 26, 2024
284b341
remove /static from public rev proxy (#18)
tsunyoku May 26, 2024
5aedaa2
Remove pypi subdomains
cmyui May 26, 2024
c485410
old.cmyui.xyz to k8s admin panel
cmyui May 26, 2024
7bd9552
fix poor ref
cmyui May 26, 2024
b9d7681
Move vault location (#19)
tsunyoku May 27, 2024
afd2da5
Support akatsuki admin panel on k8s
cmyui May 27, 2024
9222add
upstream for phpmyadmin
cmyui May 30, 2024
bf24c02
New public-rev-proxy deploy on mysql-master01
cmyui Jun 2, 2024
8ec856f
Fix branch name
cmyui Jun 2, 2024
9cc852a
send bancho service to k8s
cmyui Jun 2, 2024
21b00c6
deploy
cmyui Jun 2, 2024
73cf286
Public Reverse Proxy
cmyui Jun 2, 2024
ffed115
add codeowners file
cmyui Jun 5, 2024
ce1e22c
Add new node
cmyui Jun 5, 2024
1ead774
Remove k8s-worker02 from the production cluster
cmyui Jun 5, 2024
f16824c
Remove k8s-worker03 from production cluster
cmyui Jun 5, 2024
764d320
Add k8s-worker02
cmyui Jun 8, 2024
979e668
Double API ratelimit
cmyui Jun 9, 2024
8ec9c64
Set infra CODEOWNERS
cmyui Jun 15, 2024
cb367c9
double down
cmyui Jun 19, 2024
312456b
10r/s beatmaps api ratelimit with burst=10
cmyui Jun 22, 2024
020a0b0
Rewrite when sending to beatmaps-service to only hit /public/* routes
cmyui Jun 24, 2024
ff476e0
Rewrite avatars and profile backgrounds to /public APIs
cmyui Jun 26, 2024
872c243
Rewrite proxy pass for avatars
cmyui Jun 26, 2024
51361fc
fix
cmyui Jun 26, 2024
5d6e510
Add $host to access logs
cmyui Jun 26, 2024
2149cf0
Move vault to mysql-master01
cmyui Jul 1, 2024
08f673d
auth.akatsuki.gg
cmyui Jul 6, 2024
65e3318
users.akatsuki.gg
cmyui Jul 6, 2024
25a4b15
next.akatsuki.gg
cmyui Jul 6, 2024
fe7951b
for now:tm:
cmyui Jul 7, 2024
4196a5f
add cors headers to bancho api
cmyui Jul 7, 2024
e5246bc
revert
cmyui Jul 7, 2024
e8fd0a7
add cors manually
cmyui Jul 7, 2024
6196a9c
avatars -> assets and support profile backgrounds
cmyui Jul 19, 2024
d9159c2
rewrite for profile backgrounds
cmyui Jul 19, 2024
713b662
avatars -> assets filename
cmyui Jul 19, 2024
591b2c5
More correct api cors headers
cmyui Jul 23, 2024
f364641
Fixes
cmyui Jul 23, 2024
3775be6
Access-Control-Allow-Credentials on akatsuki api
cmyui Jul 23, 2024
c712338
More restrictive cors rules
cmyui Jul 23, 2024
143f97e
Allow localhost:3000 cors
cmyui Jul 31, 2024
10ec8d2
try meme
cmyui Aug 1, 2024
95e643a
Undo
cmyui Aug 1, 2024
9a933f9
Support API routes on next.akatsuki.gg
cmyui Aug 1, 2024
cd5b8a5
Fix
cmyui Aug 1, 2024
f4fe167
Remove all cors cache
cmyui Aug 1, 2024
045e48b
next.akatsuki.gg/api/v1/authenticate
cmyui Aug 1, 2024
f0b2709
Simplify upstreams
cmyui Aug 1, 2024
d6826c7
Naming correction
cmyui Aug 1, 2024
577fb1b
Naming consistency
cmyui Aug 1, 2024
b636373
Handle options separately
cmyui Aug 1, 2024
1b93a73
support /api/v1/users
cmyui Aug 6, 2024
1a5668b
Disclude url
cmyui Aug 8, 2024
6d4ac7e
Maybe working regex
cmyui Aug 8, 2024
b46e7e6
Always send cors?
cmyui Aug 8, 2024
7a12073
Allow creds
cmyui Aug 8, 2024
0127edf
try
cmyui Aug 8, 2024
979af2b
re-add delete
cmyui Aug 8, 2024
b219701
more strict
cmyui Aug 8, 2024
d4027c4
wtf
cmyui Aug 8, 2024
e20c522
Maybe fix cors
cmyui Aug 8, 2024
a81f09d
Update CORS headers for next FE
cmyui Aug 8, 2024
7d04772
Forward logout api
cmyui Aug 8, 2024
b045ba8
Add overall-stats API
cmyui Aug 11, 2024
88ce468
Expose aggregate score stats api & fix user stats url
cmyui Aug 12, 2024
5236728
fix tabs
cmyui Oct 25, 2024
1b18b35
Update CODEOWNERS
cmyui Jan 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/production-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: production-deploy
on:
push:
branches:
- main
- public-rev-proxy

jobs:
deploy:
Expand All @@ -16,13 +16,13 @@ jobs:
- name: Run deploy on production
uses: appleboy/[email protected]
with:
host: ${{ secrets.SSH_HOST }}
username: ${{ secrets.SSH_USERNAME }}
key: ${{ secrets.SSH_KEY }}
port: ${{ secrets.SSH_PORT }}
host: ${{ secrets.NEW_SSH_HOST }}
username: ${{ secrets.NEW_SSH_USERNAME }}
key: ${{ secrets.NEW_SSH_KEY }}
port: ${{ secrets.NEW_SSH_PORT }}
script_stop: true
script: |
cd /home/akatsuki/nginx-conf
cd /home/programming/nginx-conf
git fetch origin
git checkout origin/${{ steps.extract_branch.outputs.branch }}
git pull origin ${{ steps.extract_branch.outputs.branch }}
Expand Down
6 changes: 6 additions & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
* @cmyui @infernalfire72

# Infrastructure
/.github/* @cmyui
/tf/* @cmyui
/chart/* @cmyui
20 changes: 17 additions & 3 deletions nginx.conf
Original file line number Diff line number Diff line change
Expand Up @@ -23,22 +23,33 @@ http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

# nginx uses http/1.0 by default
# but istio/envoy proxy requires 1.1
proxy_http_version 1.1;

# ssl settings
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;

# logging Settings
log_format main '[$time_local] $http_CF_Connecting_IP - '
'"$request_method $uri" $status $body_bytes_sent '
'"$request_method $host $uri" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';

map $status $loggable {
~^[23] 0;
~^[23] 1;
default 1;
}
access_log /var/log/nginx/access.log main if=$loggable;
error_log /var/log/nginx/error.log;

map $http_origin $cors_allowed_origin {
default "";
"https://akatsuki.gg" "https://akatsuki.gg";
"https://next.akatsuki.gg" "https://next.akatsuki.gg";
"http://localhost:3000" "http://localhost:3000";
}

# connection header for WebSocket reverse proxy
map $http_upgrade $connection_upgrade {
default upgrade;
Expand All @@ -56,8 +67,11 @@ http {
}

# define rate limiting zones
limit_req_zone $http_CF_Connecting_IP zone=api_zone:10m rate=10r/s;
limit_req_zone $http_CF_Connecting_IP zone=web_zone:10m rate=10r/s;
limit_req_zone $http_CF_Connecting_IP zone=api_zone:10m rate=20r/s;
limit_req_zone $http_CF_Connecting_IP zone=osu_zone:10m rate=20r/s;
limit_req_zone $http_CF_Connecting_IP zone=beatmaps_zone:10m rate=10r/s;
limit_req_zone $http_CF_Connecting_IP zone=users_zone:10m rate=10r/s;

# virtual host configs
include /etc/nginx/sites-enabled/*.conf;
Expand Down
29 changes: 10 additions & 19 deletions sites-enabled/admin_panel.conf
Original file line number Diff line number Diff line change
@@ -1,29 +1,20 @@
server {
listen 80;
server_name old.akatsuki.pw old.akatsuki.gg;
server_name old.akatsuki.gg old.akatsuki.pw;

root /home/akatsuki/admin-panel;
location /phpmyadmin {
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

location ~ \.php$ {
add_header Access-Control-Allow-Origin *;
try_files $uri =404;

fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

include fastcgi.conf;

}

location /.git/ {
return 200 "yes";
proxy_pass http://k8s_production;
}

location / {
add_header Access-Control-Allow-Origin *;
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

index index.php;
rewrite ^/(?:u|d)/\d+$ /rewrite.php;
proxy_pass http://k8s_production;
}
}
16 changes: 15 additions & 1 deletion sites-enabled/air_conditioning.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,25 @@ server {
listen 80;
server_name air_conditioning.akatsuki.pw air_conditioning.akatsuki.gg;

client_max_body_size 100M;

location /loader-hub {
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache_bypass $http_upgrade;
proxy_buffering off;

proxy_pass http://k8s_production;
}

location / {
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://air_conditioning_service;
proxy_pass http://k8s_production;
}
}
24 changes: 24 additions & 0 deletions sites-enabled/assets.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
server {
listen 80;
server_name a.akatsuki.pw a.akatsuki.gg;

location /profile-backgrounds {
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

rewrite /profile-backgrounds/(.*) /public/api/v1/profile-backgrounds/$1 break;
proxy_pass http://k8s_production;
}

# If they hit /{filename} directly, default to avatars
location / {
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
add_header Cache-Control no-cache;

rewrite /(.*) /public/api/v1/avatars/$1 break;
proxy_pass http://k8s_production;
}
}
12 changes: 0 additions & 12 deletions sites-enabled/avatars.conf

This file was deleted.

19 changes: 18 additions & 1 deletion sites-enabled/bancho.conf
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,24 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://bancho_server;
proxy_pass http://k8s_production;
}

location /api {
limit_req zone=osu_zone burst=10 nodelay;
limit_req_status 429;

proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

# Add CORS headers on /api routes

add_header 'Access-Control-Allow-Origin' $cors_allowed_origin always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, PATCH, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Headers' '*';

proxy_pass http://k8s_production;
}
}

Expand Down
16 changes: 16 additions & 0 deletions sites-enabled/beatmaps.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
server {
listen 80;
server_name beatmaps.akatsuki.gg;

location / {
limit_req zone=beatmaps_zone burst=10 nodelay;
limit_req_status 429;

proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

rewrite /(.*) /public/$1 break;
proxy_pass http://k8s_production;
}
}
34 changes: 13 additions & 21 deletions sites-enabled/frontend.conf
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://score_server;
proxy_pass http://k8s_production;
}

# /web/replays for replay downloads (hanayo)
Expand All @@ -38,7 +38,7 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://score_server;
proxy_pass http://k8s_production;
}

# Redirect osu! avatar edit to hanayo
Expand All @@ -51,44 +51,35 @@ server {
limit_req zone=api_zone burst=10 nodelay;
limit_req_status 429;

add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'POST, GET, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'X-Ripple-Token';
add_header 'Access-Control-Max-Age' '21600';
#add_header Access-Control-Allow-Origin "https://akatsuki.gg";
#add_header Access-Control-Allow-Origin "https://akatsuki.pw";
add_header 'Access-Control-Allow-Origin' $cors_allowed_origin always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, PATCH, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Headers' '*';
add_header 'Access-Control-Allow-Credentials' 'true';

proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://akatsuki_api;
proxy_pass http://k8s_production;
}

location /api/v1/profile-history {
limit_req zone=api_zone burst=10 nodelay;
limit_req_status 429;

add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'POST, GET, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'X-Ripple-Token';
add_header 'Access-Control-Max-Age' '21600';
add_header 'Access-Control-Allow-Origin' $cors_allowed_origin always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, PATCH, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Headers' '*';

proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass http://profile_history_service;
proxy_pass http://k8s_production;
}

# https://osu.ppy.sh/beatmapsets/1117775#osu/2334952 -> https://osu.ppy.sh/b/2334952
rewrite ^/beatmapsets/\d+#osu/(\d+)/?$ /b/$1 last;

location /static/ {
#add_header Access-Control-Allow-Origin "akatest.space";
alias /home/akatsuki/hanayo/web/static/;

}

location /discord {
return 302 $scheme://discord.gg/5cBtMPW;
}
Expand All @@ -98,10 +89,11 @@ server {
}

location / {
limit_req zone=web_zone burst=10 nodelay;
proxy_set_header X-Real-IP $http_CF_Connecting_IP;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_pass http://hanayo;
proxy_pass http://k8s_production;
}
}
12 changes: 0 additions & 12 deletions sites-enabled/irc.conf

This file was deleted.

Loading
Loading