Skip to content

Commit 9337c0c

Browse files
authored
Merge pull request #404 from constructive-io/anmol/jobs-refactor
feature: major jobs refactor 2
2 parents 4326e20 + b8a8e1d commit 9337c0c

File tree

45 files changed

+3767
-8027
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+3767
-8027
lines changed

DEVELOPMENT_JOBS.md

Lines changed: 127 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,10 @@
22

33
This guide covers a local development workflow for the jobs stack:
44

5-
- Postgres + `launchql-ext-jobs`
5+
- Postgres + `launchql-database-jobs`
66
- LaunchQL API server
77
- `simple-email` function
8+
- `send-email-link` function
89
- `knative-job-service`
910

1011
It assumes:
@@ -56,19 +57,19 @@ This ensures all subsequent `pgpm` and `psql` commands point at the same local d
5657

5758
Make sure `pgpm` is installed and up to date.
5859

59-
From the `constructive/` directory (with `pgenv` applied):
60+
From the `constructive-db/` directory (with `pgenv` applied):
6061

61-
1. Bootstrap admin users:
62+
1. Create the `launchql` database (if it does not already exist):
6263

6364
```sh
64-
pgpm admin-users bootstrap --yes
65-
pgpm admin-users add --test --yes
65+
createdb launchql
6666
```
6767

68-
2. Create the `launchql` database (if it does not already exist):
68+
2. Bootstrap admin users:
6969

7070
```sh
71-
createdb launchql
71+
pgpm admin-users bootstrap --yes
72+
pgpm admin-users add --test --yes
7273
```
7374

7475
3. Deploy the main app and jobs packages into `launchql`:
@@ -88,52 +89,125 @@ At this point, the app schema and `database-jobs` should be installed and `app_j
8889
With Postgres initialized, bring up the jobs-related services using `docker-compose.jobs.yml`:
8990

9091
```sh
92+
docker compose -f docker-compose.jobs.yml build
9193
docker compose -f docker-compose.jobs.yml up
9294
```
9395

9496
Or run detached:
9597

9698
```sh
97-
docker compose -f docker-compose.jobs.yml up -d
99+
docker compose -f docker-compose.jobs.yml up -d --build
98100
```
99101

100102
This starts:
101103

102104
- `launchql-server` – GraphQL API server
103105
- `simple-email` – Knative-style HTTP function
106+
- `send-email-link` – Knative-style HTTP function
104107
- `knative-job-service` – jobs runtime (callback server + worker + scheduler)
105108

106-
By default, all three services use the published image:
109+
---
110+
111+
### Switching dry run vs real Mailgun sending
112+
113+
By default, `docker-compose.jobs.yml` runs both email functions in dry-run mode (no real email is sent), and it uses placeholder Mailgun credentials unless you provide `MAILGUN_API_KEY` / `MAILGUN_KEY`.
114+
115+
Quick start commands:
116+
117+
Dry run:
118+
119+
```sh
120+
docker compose -f docker-compose.jobs.yml up -d --build --force-recreate
121+
```
122+
123+
Real sending (Mailgun):
124+
125+
```sh
126+
MAILGUN_API_KEY="your-mailgun-key" MAILGUN_KEY="your-mailgun-key" SIMPLE_EMAIL_DRY_RUN=false SEND_EMAIL_LINK_DRY_RUN=false docker compose -f docker-compose.jobs.yml up -d --build --force-recreate
127+
```
128+
129+
To use a real Mailgun key without editing `docker-compose.jobs.yml`, set these env vars before starting the stack (or put them in a local `.env` file in the `constructive/` directory). Don't commit your `.env`.
130+
131+
```sh
132+
export MAILGUN_API_KEY="your-mailgun-key"
133+
export MAILGUN_KEY="your-mailgun-key"
134+
```
135+
136+
If you're not using `mg.constructive.io`, also override `MAILGUN_DOMAIN`, `MAILGUN_FROM`, and `MAILGUN_REPLY` (for example in the override file below) to match your Mailgun setup.
137+
138+
To actually send email (instead of dry-run), set these env vars (or put them in your local `.env`):
107139

108-
```text
109-
ghcr.io/constructive-io/launchql:b88e3d1
140+
```sh
141+
export SIMPLE_EMAIL_DRY_RUN=false
142+
export SEND_EMAIL_LINK_DRY_RUN=false
110143
```
111144

112-
If you want to test a local build instead, build the image from the `constructive/` workspace and update `image:` in `docker-compose.jobs.yml` to point to your local tag, for example:
145+
Then recreate the stack so the new env is applied:
113146

114147
```sh
115-
docker build -t constructive-local .
148+
docker compose -f docker-compose.jobs.yml up -d --build --force-recreate
149+
```
150+
151+
If you prefer not to export env vars, create a local override file (don't commit it) at `docker-compose.jobs.override.yml`:
152+
153+
```yml
154+
services:
155+
simple-email:
156+
environment:
157+
SIMPLE_EMAIL_DRY_RUN: "false"
158+
159+
send-email-link:
160+
environment:
161+
SEND_EMAIL_LINK_DRY_RUN: "false"
116162
```
117163
118-
Then in `docker-compose.jobs.yml`:
164+
Start the stack with both files:
119165
120-
```yaml
121-
image: constructive-local
166+
```sh
167+
docker compose -f docker-compose.jobs.yml -f docker-compose.jobs.override.yml up -d --build --force-recreate
122168
```
123169

124-
All services are attached to the shared `constructive-net` network and talk to the `postgres` container by hostname `postgres`.
170+
To switch back to dry-run, set `SIMPLE_EMAIL_DRY_RUN=true` and `SEND_EMAIL_LINK_DRY_RUN=true` (or delete the override file) and recreate again.
125171

126172
---
127173

128-
## 5. Enqueue a test job (simple-email)
174+
## 5. Ensure GraphQL host routing works for `send-email-link`
175+
176+
LaunchQL selects the API by the HTTP `Host` header using rows in `meta_public.domains`.
177+
178+
For local development, `app-svc-local` seeds `admin.localhost` as the admin API domain. `docker-compose.jobs.yml` adds a Docker network alias so other containers can resolve `admin.localhost` to the `launchql-server` container, and `send-email-link` uses:
179+
180+
- `GRAPHQL_URL=http://admin.localhost:3000/graphql`
181+
182+
Quick check from your host (should return JSON, not HTML):
183+
184+
```sh
185+
curl -s -H 'Host: admin.localhost' \
186+
-H 'Content-Type: application/json' \
187+
-X POST http://localhost:3000/graphql \
188+
--data '{"query":"query { __typename }"}'
189+
```
190+
191+
If your GraphQL server requires auth, set `GRAPHQL_AUTH_TOKEN` before starting the jobs stack (it is passed through to the `send-email-link` container).
192+
193+
---
194+
195+
## 6. Enqueue a test job (simple-email)
129196

130197
With the jobs stack running, you can enqueue a test job from your host into the Postgres container:
131198

199+
First, grab a real `database_id` (required by `send-email-link`, optional for `simple-email`):
200+
201+
```sh
202+
DBID="$(docker exec -i postgres psql -U postgres -d launchql -Atc 'SELECT id FROM collections_public.database ORDER BY created_at LIMIT 1;')"
203+
echo "$DBID"
204+
```
205+
132206
```sh
133207
docker exec -it postgres \
134208
psql -U postgres -d launchql -c "
135209
SELECT app_jobs.add_job(
136-
'00000000-0000-0000-0000-000000000001'::uuid,
210+
'$DBID'::uuid,
137211
'simple-email',
138212
json_build_object(
139213
@@ -148,7 +222,39 @@ You should then see the job picked up by `knative-job-service` and the email pay
148222

149223
---
150224

151-
## 6. Inspect logs and iterate
225+
## 7. Enqueue a test job (`send-email-link`)
226+
227+
`send-email-link` queries GraphQL for site/database metadata, so it requires:
228+
229+
- The app/meta packages deployed in step 3 (`app-svc-local`, `db-meta`)
230+
- A real `database_id` (use `$DBID` above)
231+
- A GraphQL hostname that matches a seeded domain route (step 5)
232+
233+
With `SEND_EMAIL_LINK_DRY_RUN=true` (default in `docker-compose.jobs.yml`), enqueue a job:
234+
235+
```sh
236+
docker exec -it postgres \
237+
psql -U postgres -d launchql -c "
238+
SELECT app_jobs.add_job(
239+
'$DBID'::uuid,
240+
'send-email-link',
241+
json_build_object(
242+
'email_type', 'invite_email',
243+
'email', '[email protected]',
244+
'invite_token', 'invite123',
245+
'sender_id', '00000000-0000-0000-0000-000000000000'
246+
)::json
247+
);
248+
"
249+
```
250+
251+
You should see a log like:
252+
253+
- `[send-email-link] DRY RUN email (skipping send) ...`
254+
255+
---
256+
257+
## 8. Inspect logs and iterate
152258

153259
To watch logs while you develop:
154260

@@ -172,7 +278,7 @@ docker compose -f docker-compose.jobs.yml up --build
172278

173279
---
174280

175-
## 7. Stopping services
281+
## 9. Stopping services
176282

177283
To stop only the jobs stack:
178284

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ COPY . .
2323

2424
# Install and build all workspaces
2525
RUN set -eux; \
26-
pnpm install --frozen-lockfile; \
26+
CI=true pnpm install --frozen-lockfile; \
2727
pnpm run build
2828

2929
################################################################################

docker-compose.jobs.yml

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ services:
22
# LaunchQL API server (GraphQL)
33
launchql-server:
44
container_name: launchql-server
5-
image: ghcr.io/constructive-io/launchql:b88e3d1
5+
image: constructive-launchql:dev
6+
build:
7+
context: .
8+
dockerfile: ./Dockerfile
69
# The image entrypoint already runs the LaunchQL CLI (`lql`).
710
# We only need to provide the subcommand and flags here.
811
entrypoint: ["lql", "server", "--port", "3000", "--origin", "*", "--strictAuth", "false"]
@@ -29,36 +32,70 @@ services:
2932
ports:
3033
- "3000:3000"
3134
networks:
32-
- constructive-net
35+
constructive-net:
36+
aliases:
37+
# Let other containers call the admin API using the seeded domain route.
38+
- admin.localhost
3339

3440
# Simple email function (Knative-style HTTP function)
3541
simple-email:
3642
container_name: simple-email
37-
image: ghcr.io/constructive-io/launchql:b88e3d1
43+
image: constructive-launchql:dev
3844
# Override the image entrypoint (LaunchQL CLI) and run the Node function directly.
3945
entrypoint: ["node", "functions/simple-email/dist/index.js"]
4046
environment:
4147
NODE_ENV: development
4248
LOG_LEVEL: info
49+
SIMPLE_EMAIL_DRY_RUN: "${SIMPLE_EMAIL_DRY_RUN:-true}"
4350
# Mailgun / email provider configuration for @launchql/postmaster
4451
# Replace with real credentials for local testing.
45-
MAILGUN_API_KEY: "change-me-mailgun-api-key"
52+
MAILGUN_API_KEY: "${MAILGUN_API_KEY:-change-me-mailgun-api-key}"
53+
MAILGUN_KEY: "${MAILGUN_KEY:-change-me-mailgun-api-key}"
4654
MAILGUN_DOMAIN: "mg.constructive.io"
4755
MAILGUN_FROM: "[email protected]"
56+
MAILGUN_REPLY: "[email protected]"
4857
ports:
4958
# Expose function locally (optional)
5059
- "8081:8080"
5160
networks:
5261
- constructive-net
5362

63+
# Send email link function (invite, password reset, verification)
64+
send-email-link:
65+
container_name: send-email-link
66+
image: constructive-launchql:dev
67+
entrypoint: ["node", "functions/send-email-link/dist/index.js"]
68+
environment:
69+
NODE_ENV: development
70+
LOG_LEVEL: info
71+
DEFAULT_DATABASE_ID: "dbe"
72+
# LaunchQL selects the API by Host header; use a seeded domain route.
73+
GRAPHQL_URL: "http://admin.localhost:3000/graphql"
74+
META_GRAPHQL_URL: "http://admin.localhost:3000/graphql"
75+
# Optional: provide an existing API token (Bearer) if your server requires it.
76+
GRAPHQL_AUTH_TOKEN: "${GRAPHQL_AUTH_TOKEN:-}"
77+
# Mailgun / email provider configuration for @launchql/postmaster
78+
MAILGUN_API_KEY: "${MAILGUN_API_KEY:-change-me-mailgun-api-key}"
79+
MAILGUN_KEY: "${MAILGUN_KEY:-change-me-mailgun-api-key}"
80+
MAILGUN_DOMAIN: "mg.constructive.io"
81+
MAILGUN_FROM: "[email protected]"
82+
MAILGUN_REPLY: "[email protected]"
83+
SEND_EMAIL_LINK_DRY_RUN: "${SEND_EMAIL_LINK_DRY_RUN:-true}"
84+
ports:
85+
# Expose function locally (optional)
86+
- "8082:8080"
87+
networks:
88+
- constructive-net
89+
5490
# Jobs runtime: callback server + worker + scheduler
5591
knative-job-service:
5692
container_name: knative-job-service
57-
image: ghcr.io/constructive-io/launchql:b88e3d1
93+
image: constructive-launchql:dev
5894
# Override the image entrypoint and run the jobs runtime directly.
5995
entrypoint: ["node", "jobs/knative-job-service/dist/run.js"]
6096
depends_on:
6197
- simple-email
98+
- send-email-link
6299
environment:
63100
NODE_ENV: development
64101

@@ -72,22 +109,22 @@ services:
72109

73110
# Worker configuration
74111
JOBS_SUPPORT_ANY: "false"
75-
JOBS_SUPPORTED: "simple-email"
112+
JOBS_SUPPORTED: "simple-email,send-email-link"
76113
HOSTNAME: "knative-job-service-1"
77114

78115
# Callback HTTP server (job completion callbacks)
79116
INTERNAL_JOBS_CALLBACK_PORT: "8080"
80-
INTERNAL_JOBS_CALLBACK_URL: "http://knative-job-service:8080"
117+
INTERNAL_JOBS_CALLBACK_URL: "http://knative-job-service:8080/callback"
118+
# Hostname used by job-utils.getCallbackBaseUrl for callbacks
119+
JOBS_CALLBACK_HOST: "knative-job-service"
81120

82121
# Function gateway base URL (used by worker when no dev map is present)
83-
# Not used in dev when INTERNAL_GATEWAY_DEVELOPMENT_MAP is set, but required for validation.
84-
KNATIVE_SERVICE_URL: "dev.internal"
85122
INTERNAL_GATEWAY_URL: "http://simple-email:8080"
86123

87124
# Development-only map from task identifier -> function URL
88125
# Used by @launchql/knative-job-worker when NODE_ENV !== 'production'.
89-
# This lets the worker call the simple-email container directly in docker-compose.
90-
INTERNAL_GATEWAY_DEVELOPMENT_MAP: '{"simple-email":"http://simple-email:8080"}'
126+
# This lets the worker call the function containers directly in docker-compose.
127+
INTERNAL_GATEWAY_DEVELOPMENT_MAP: '{"simple-email":"http://simple-email:8080","send-email-link":"http://send-email-link:8080"}'
91128

92129
ports:
93130
- "8080:8080"

functions/send-email-link/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
"@launchql/mjml": "0.1.1",
1919
"@launchql/postmaster": "0.1.4",
2020
"@launchql/styled-email": "0.1.0",
21+
"@pgpmjs/env": "workspace:^",
2122
"graphql-request": "^7.1.2",
2223
"graphql-tag": "^2.12.6"
2324
}

0 commit comments

Comments
 (0)