feat: implement raw material grouping and cost management modules with associated gRPC services and database migrations#72
Merged
ilramdhan merged 6 commits intomutugading:mainfrom Apr 22, 2026
Conversation
…h associated gRPC services and database migrations
ilramdhan
added
documentation
There was a problem hiding this comment.
Pull request overview
This PR introduces end-to-end support for Raw Material (RM) grouping and RM landed-cost calculation in the Finance service, including database schema, domain/application layers, gRPC exposure (plus HTTP gateway), and worker-side job execution via RabbitMQ.
Changes:
- Added Finance DB migrations for RM group head/detail, RM cost tables, and append-only audit/history tables (plus index evolution for
(item_code, grade_code)). - Implemented rmgroup and rmcost domain + application handlers + Postgres repositories, and integrated new gRPC services + HTTP gateway registration.
- Extended RabbitMQ topology and publishing/consuming to support
rm_cost_calculation, and added worker support with Oracle-optional startup + Oracle-sync → RM-cost chaining.
Reviewed changes
Copilot reviewed 86 out of 89 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| services/iam/migrations/postgres/000026_seed_rm_grouping_menus.up.sql | Seeds IAM menus/permissions for RM Pricing screens and adds recalculate action type. |
| services/iam/migrations/postgres/000026_seed_rm_grouping_menus.down.sql | Rolls back RM Pricing menus/permissions and removes recalculate from action_type constraint. |
| services/finance/migrations/postgres/000010_create_cst_rm_group_head.up.sql | Adds RM group head table + constraints + search index. |
| services/finance/migrations/postgres/000010_create_cst_rm_group_head.down.sql | Drops RM group head table and indexes. |
| services/finance/migrations/postgres/000011_create_cst_rm_group_detail.up.sql | Adds RM group detail table and item-level uniqueness/indexes. |
| services/finance/migrations/postgres/000011_create_cst_rm_group_detail.down.sql | Drops RM group detail table and indexes. |
| services/finance/migrations/postgres/000012_create_cst_rm_cost.up.sql | Adds RM cost aggregate table with checks + indexes. |
| services/finance/migrations/postgres/000012_create_cst_rm_cost.down.sql | Drops RM cost table and indexes. |
| services/finance/migrations/postgres/000013_extend_job_execution_for_rmcost.up.sql | Adds job_type constraint and rm_cost_calculation + composite index. |
| services/finance/migrations/postgres/000013_extend_job_execution_for_rmcost.down.sql | Rolls back job_type constraint/index. |
| services/finance/migrations/postgres/000014_create_aud_rm_cost_history.up.sql | Adds append-only RM cost calculation history table + indexes. |
| services/finance/migrations/postgres/000014_create_aud_rm_cost_history.down.sql | Drops RM cost history table and indexes. |
| services/finance/migrations/postgres/000015_widen_rm_group_detail_text_cols.up.sql | Widens RM group detail text columns to match Oracle feed lengths. |
| services/finance/migrations/postgres/000015_widen_rm_group_detail_text_cols.down.sql | Narrows RM group detail text columns (rollback). |
| services/finance/migrations/postgres/000016_create_aud_rm_group.up.sql | Adds append-only audit tables for RM group head/detail mutations + indexes. |
| services/finance/migrations/postgres/000016_create_aud_rm_group.down.sql | Drops RM group audit tables. |
| services/finance/migrations/postgres/000017_widen_rm_group_text_cols_defensive.up.sql | Defensive widening to prevent varchar overflow + audit table alignment. |
| services/finance/migrations/postgres/000017_widen_rm_group_text_cols_defensive.down.sql | Rolls back defensive widenings (may fail if data too long). |
| services/finance/migrations/postgres/000018_rm_group_detail_grade_unique.up.sql | Evolves uniqueness rule to include grade_code and updates lookup index. |
| services/finance/migrations/postgres/000018_rm_group_detail_grade_unique.down.sql | Reverts uniqueness/index changes back to item-only. |
| services/finance/internal/infrastructure/rabbitmq/publisher.go | Extends job message payload with group_head_id + reason for RM cost jobs. |
| services/finance/internal/infrastructure/rabbitmq/job_publisher.go | Adds PublishRMCostCalculation message publishing helper. |
| services/finance/internal/infrastructure/rabbitmq/connection.go | Declares/binds RM cost calculation queue and routing key. |
| services/finance/internal/infrastructure/postgres/syncdata_ungrouped.go | Adds SQL for listing “Ungrouped Items” using (item_code, grade_code) join. |
| services/finance/internal/infrastructure/postgres/syncdata_rateinputs.go | Adds sync-data readers for RM cost calc inputs and UOM fallback lookups. |
| services/finance/internal/infrastructure/postgres/syncdata_item_lookup.go | Adds item lookup by (item_code, grade_code) with enrichment-friendly fallback. |
| services/finance/internal/infrastructure/postgres/rmgroup_item_rates.go | Adds join query for per-item stage rates display on group detail page. |
| services/finance/internal/infrastructure/postgres/rmgroup_head_repository.go | Implements RM group head persistence + audit insertions. |
| services/finance/internal/infrastructure/postgres/rmgroup_detail_repository.go | Implements RM group detail persistence + variant lookup by (item_code, grade_code). |
| services/finance/internal/infrastructure/postgres/rmgroup_audit.go | Shared audit helpers for RM group head/detail tables. |
| services/finance/internal/infrastructure/postgres/syncdata_repository.go | Adjusts sync-data search + scanning (null-safe numeric/time scanning). |
| services/finance/internal/domain/rmgroup/value_objects.go | Adds rmgroup value objects (code/item/flags) with validation. |
| services/finance/internal/domain/rmgroup/repository.go | Defines rmgroup repository contract and list filter types. |
| services/finance/internal/domain/rmgroup/errors.go | Defines rmgroup domain errors for validation and invariants. |
| services/finance/internal/domain/rmcost/repository.go | Defines rmcost repository contract and filter types. |
| services/finance/internal/domain/rmcost/errors.go | Defines rmcost domain errors. |
| services/finance/internal/domain/rmcost/entity.go | Adds rmcost entity + history model + period validation and trigger reasons. |
| services/finance/internal/domain/rmcost/calculation.go | Adds pure landed-cost engine (aggregate/select/cascade/compute). |
| services/finance/internal/domain/job/value_objects.go | Adds new job type constant rm_cost_calculation. |
| services/finance/internal/delivery/httpdelivery/gateway.go | Registers RMGroup and RMCost grpc-gateway handlers. |
| services/finance/internal/delivery/grpc/uom_handler.go | Expands conflict error mapping + prefers username/user_id from auth context. |
| services/finance/internal/delivery/grpc/metrics.go | Adds Prometheus counters for RM group/cost operations. |
| services/finance/internal/application/rmgroup/create_handler.go | Creates RM group head with uniqueness checks and optional field carry-over. |
| services/finance/internal/application/rmgroup/list_handler.go | Lists heads with filters/pagination and optional flag parsing. |
| services/finance/internal/application/rmgroup/get_handler.go | Fetches head and optional details (active-only option). |
| services/finance/internal/application/rmgroup/update_handler.go | Partial update handler with typed flag parsing + clear-init support. |
| services/finance/internal/application/rmgroup/delete_handler.go | Soft-delete handler with optional cost-data delete guard. |
| services/finance/internal/application/rmgroup/add_items_handler.go | Batch add items with “one variant, one active group” invariant + backfill. |
| services/finance/internal/application/rmgroup/remove_items_handler.go | Removes items via deactivate or soft-delete modes with ownership checks. |
| services/finance/internal/application/rmgroup/item_rates_handler.go | Retrieves per-item stage rates for a head/period. |
| services/finance/internal/application/rmgroup/export_handler.go | Exports groups/items to a two-sheet Excel workbook. |
| services/finance/internal/application/rmgroup/template_handler.go | Generates Excel import templates (full + per-group items). |
| services/finance/internal/application/rmgroup/ungrouped_handler.go | Lists ungrouped items with pagination metadata. |
| services/finance/internal/application/rmgroup/ungrouped_export_handler.go | Exports all ungrouped items by paging through the reader. |
| services/finance/internal/application/rmgroup/import_group_items_handler.go | Imports per-group items from one-sheet Excel, enriching via sync lookup. |
| services/finance/internal/application/rmgroup/mocks_test.go | Adds testify mocks for rmgroup application tests. |
| services/finance/internal/application/rmgroup/handlers_test.go | Adds unit tests covering rmgroup handlers and key invariants. |
| services/finance/internal/application/rmcost/trigger_handler.go | Creates job_execution rows and publishes rm cost calc jobs. |
| services/finance/internal/application/rmcost/calculate_handler.go | Calculates landed cost for one/all groups and persists cost + history. |
| services/finance/internal/application/rmcost/execute_handler.go | Worker-side job lifecycle wrapper around CalculateHandler. |
| services/finance/internal/application/rmcost/get_handler.go | Fetches cost by ID or (period, rm_code). |
| services/finance/internal/application/rmcost/list_handler.go | Lists costs with filters/pagination. |
| services/finance/internal/application/rmcost/history_handler.go | Lists calculation history with filters/pagination. |
| services/finance/internal/application/rmcost/periods_handler.go | Lists distinct periods that have cost rows. |
| services/finance/internal/application/rmcost/export_handler.go | Exports RM costs to a single-sheet Excel workbook. |
| services/finance/internal/application/rmcost/mocks_test.go | Adds testify mocks for rmcost application tests. |
| services/finance/internal/application/rmcost/handlers_test.go | Adds unit tests for rmcost handlers (trigger/list/history/get/calc). |
| services/finance/internal/application/oraclesync/sync_handler.go | Adds optional chaining to enqueue RM cost calc after successful Oracle sync. |
| services/finance/config.yaml | Updates RabbitMQ default URL configuration. |
| services/finance/cmd/worker/main.go | Adds RM cost consumers and makes Oracle optional with graceful degradation. |
| services/finance/cmd/server/main.go | Wires RMGroup/RMCost handlers into server startup and gRPC registration. |
| services/finance/Makefile | Adds build/run targets for the finance worker. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+105
to
+109
| syncMsgHandler := func(ctx context.Context, msg rabbitmq.JobMessage) error { | ||
| if syncHandler == nil { | ||
| log.Warn().Str("job_id", msg.JobID).Msg("Oracle sync job received but Oracle unavailable; skipping") | ||
| return nil | ||
| } |
There was a problem hiding this comment.
When Oracle is unavailable, the worker currently logs a warning and returns nil for oracle_sync messages. This will ack the message but leaves the corresponding job_execution stuck in QUEUED/PROCESSING with no terminal status. Consider marking the job as FAILED/CANCELLED (with a clear error message) via jobRepo.UpdateStatus, or returning an error so the message is retried/DLQ’d, depending on the desired semantics.
Comment on lines
176
to
183
| if filter.Search != "" { | ||
| conditions = append(conditions, fmt.Sprintf( | ||
| "to_tsvector('english', coalesce(item_code,'') || ' ' || coalesce(item_name,'') || ' ' || coalesce(grade_name,'')) @@ plainto_tsquery('english', $%d)", argIdx, | ||
| "(item_code ILIKE $%d OR item_name ILIKE $%d OR grade_name ILIKE $%d OR grade_code ILIKE $%d)", | ||
| argIdx, argIdx, argIdx, argIdx, | ||
| )) | ||
| args = append(args, filter.Search) | ||
| args = append(args, "%"+filter.Search+"%") | ||
| argIdx++ | ||
| } |
There was a problem hiding this comment.
The search filter was changed from a full-text query (backed by the idx_cst_micsp_search GIN index created in migration 000009) to multiple ILIKE predicates. This will bypass the existing index and can cause significant performance regression on large cst_item_cons_stk_po tables. Consider restoring FTS (and extending the tsvector/index to include grade_code if needed) or adding appropriate trigram indexes if ILIKE is required.
Comment on lines
+7
to
+13
|
|
||
| "github.com/google/uuid" | ||
|
|
||
| "github.com/mutugading/goapps-backend/services/finance/internal/application/oraclesync" | ||
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/job" | ||
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/rmcost" | ||
| "time" |
There was a problem hiding this comment.
time is imported in the non-standard section/order (it’s placed after project imports). This file doesn’t appear gofmt’ed; please run gofmt so imports are grouped consistently (std lib, third-party, local).
Suggested change
| "github.com/google/uuid" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/application/oraclesync" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/job" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/rmcost" | |
| "time" | |
| "time" | |
| "github.com/google/uuid" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/application/oraclesync" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/job" | |
| "github.com/mutugading/goapps-backend/services/finance/internal/domain/rmcost" |
Comment on lines
+148
to
+164
| func (h *SyncHandler) publishCostChain(ctx context.Context, period, createdBy string) { | ||
| if h.chainPub == nil { | ||
| return | ||
| } | ||
|
|
||
| chainExec, err := job.NewExecution(job.TypeRMCostCalculation, "landed_cost", period, createdBy, 5, nil) | ||
| if err != nil { | ||
| h.logger.Warn().Err(err).Str("period", period).Msg("Failed to build chained rm cost job") | ||
| return | ||
| } | ||
| if err := h.jobRepo.Create(ctx, chainExec); err != nil { | ||
| h.logger.Warn().Err(err).Str("period", period).Msg("Failed to persist chained rm cost job") | ||
| return | ||
| } | ||
|
|
||
| if err := h.chainPub.PublishRMCostCalculation(ctx, chainExec.ID().String(), period, nil, "oracle-sync-chain", createdBy); err != nil { | ||
| // Compensate: mark the created job as failed so it doesn't linger as queued. |
There was a problem hiding this comment.
publishCostChain creates and enqueues a new rm_cost_calculation job unconditionally after every successful Oracle sync. If operators rerun sync while a previous RM cost calc for the same period is still QUEUED/PROCESSING, this will create duplicate active jobs/messages for the same (type, period). Consider checking jobRepo.HasActiveJob(ctx, job.TypeRMCostCalculation, period) (and skipping/logging when true) before creating/publishing the chained job.
|
|
||
| rabbitmq: | ||
| url: "" # Override via RABBITMQ_URL env var (e.g. amqp://user:pass@host:5672/) | ||
| url: "amqp://guest:guest@localhost:5672/" # Override via RABBITMQ_URL env var |
There was a problem hiding this comment.
rabbitmq.url is now hardcoded to amqp://guest:guest@localhost:5672/. Even if intended for local dev, committing default credentials/connection strings increases the risk of accidentally running against an insecure broker. Consider leaving this empty (as before) or using a non-credential placeholder, relying on RABBITMQ_URL env var for actual values.
Suggested change
| url: "amqp://guest:guest@localhost:5672/" # Override via RABBITMQ_URL env var | |
| url: "" # Override via RABBITMQ_URL env var |
…ng across finance services
…minor internal helpers
… to colourant field labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces the gRPC server and worker integration for the RM Cost (Raw Material Cost) calculation pipeline in the finance service. It adds the full gRPC service definition and handlers for RM Cost operations, integrates the necessary repositories and handlers into the server startup, and updates the Makefile to support building and running the worker process. The worker is also updated for graceful degradation if Oracle is unavailable.
Type of Change
Service(s) Affected
Changes Made
The most important changes are:
gRPC Service and Handler Integration:
RMCostService, including client and server interfaces, method handlers, and service registration ingen/finance/v1/rm_cost_grpc.pb.go. This exposes endpoints for triggering, calculating, fetching, listing, and exporting RM Cost data.RMCostServiceandRMGroupServicehandlers to the gRPC server and wired up their dependencies incmd/server/main.go, ensuring RM Cost functionality is available via gRPC. [1] [2] [3] [4] [5] [6]Worker and Build Process Updates:
run-workerandbuild-workercommands for the finance worker process.These changes collectively enable full support for RM Cost calculation workflows through both gRPC server endpoints and background worker processing.
Testing Performed
Unit Tests
Integration Tests
Lint & Build
golangci-lint run ./...passesgo build ./...succeedsgo test -race ./...passesDatabase (if applicable)
Documentation
Pre-merge Checklist