Conversation
sky-0131
requested review from
jihoonkim501,
yeonjuncho and
yukyoungs
as code owners
July 17, 2026 15:42
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthrough원티드 공고 수집·정규화, 잡코리아 기업 보강, JSONL 생성·병합, 스킬 태그 보강과 Java 기반 실행·파싱·저장 및 관리자 API가 추가되었습니다. Changes원티드 수집 및 결과 생성
Java 적재 파이프라인
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant 관리자
participant JobCrawlerController
participant JobCrawlerManager
participant JobCrawlerExecutor
participant PythonCrawler
participant JobParserService
participant Database
관리자->>JobCrawlerController: POST /api/admin/crawler/run
JobCrawlerController->>JobCrawlerManager: runPipelineAndSave()
JobCrawlerManager->>JobCrawlerExecutor: executePipeline()
JobCrawlerExecutor->>PythonCrawler: 크롤링 프로세스 실행
PythonCrawler-->>JobCrawlerExecutor: companies.jsonl 및 job_postings.jsonl 생성
JobCrawlerManager->>JobParserService: parseAndSave()
JobParserService->>Database: 회사 및 공고 저장
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (6)
build.gradle (1)
46-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
spring-boot-starter-web의존성이 중복됩니다.Line 27에 이미
spring-boot-starter-webmvc가 선언되어 있으며, Spring Boot 4.x에서spring-boot-starter-webmvc는 Jackson(ObjectMapper)을 포함한 모든 웹 관련 의존성을 제공합니다.spring-boot-starter-web을 추가로 선언할 필요가 없습니다.♻️ 제안하는 변경
- - // JSONL 파일 파싱을 위해 사용한 ObjectMapper용 Jackson 라이브러리 - implementation 'org.springframework.boot:spring-boot-starter-web'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@build.gradle` around lines 46 - 48, Remove the redundant spring-boot-starter-web dependency from build.gradle; retain the existing spring-boot-starter-webmvc declaration as the provider of the required web and Jackson dependencies.src/main/java/com/leets7th/job_is_be/global/config/AppConfig.java (1)
10-13: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
ObjectMapper기본 설정 시 알 수 없는 프로퍼티 처리 구성을 고려하세요.Python 크롤러가 생성한 JSONL에 DTO에 없는 필드가 포함될 수 있습니다. 기본
ObjectMapper는 알 수 없는 프로퍼티에서 역직렬화 실패를 발생시킵니다.♻️ 제안하는 변경
+import com.fasterxml.jackson.databind.DeserializationFeature; + `@Bean` public ObjectMapper objectMapper() { - return new ObjectMapper(); + return new ObjectMapper() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/global/config/AppConfig.java` around lines 10 - 13, Update the objectMapper() bean to configure ObjectMapper to ignore unknown JSON properties during deserialization, allowing crawler-generated fields absent from DTOs without failing. Preserve the existing bean definition and return the configured mapper.src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java (1)
24-35: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
@PrePersist/@PreUpdate훅이@CreatedDate/@LastModifiedDate와 중복됩니다.
@EntityListeners(AuditingEntityListener.class)와@CreatedDate/@LastModifiedDate가 이미 동일한 필드를 자동 설정합니다. 수동 라이프사이클 훅을 추가하면 두 메커니즘이 모두 실행되어 불필요한 중복이 발생합니다. 하나의 방식만 선택하세요.♻️ 제안하는 변경 — auditing 방식만 유지
`@MappedSuperclass` `@Getter` `@EntityListeners`(AuditingEntityListener.class) public abstract class BaseEntity { `@CreatedDate` `@Column`(name = "created_at", nullable = false, updatable = false) private LocalDateTime createdAt; `@LastModifiedDate` `@Column`(name = "updated_at", nullable = false) private LocalDateTime updatedAt; - - // 저장 전 자동으로 호출되어 null 방지 - `@PrePersist` - public void onPrePersist() { - this.createdAt = LocalDateTime.now(); - this.updatedAt = LocalDateTime.now(); - } - - // 업데이트 전 자동으로 호출 - `@PreUpdate` - public void onPreUpdate() { - this.updatedAt = LocalDateTime.now(); - } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java` around lines 24 - 35, Remove the manual `@PrePersist` and `@PreUpdate` lifecycle methods on BaseEntity, including onPrePersist and onPreUpdate, and retain the existing `@EntityListeners`(AuditingEntityListener.class) with `@CreatedDate/`@LastModifiedDate auditing as the sole mechanism for setting timestamps.src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.java (2)
38-38: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
InputStreamReader에 문자셋을 지정하세요.Python 프로세스의 한글 출력이 플랫폼 기본 문자셋으로 해석되어 깨질 수 있습니다.
♻️ 제안하는 변경
+import java.nio.charset.StandardCharsets; + - try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) { + try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.java` at line 38, Update the InputStreamReader construction in JobCrawlerExecutor to specify the charset used by the Python process output instead of relying on the platform default, ensuring Korean text is decoded correctly. Reuse the project’s existing charset convention if one is defined.
46-46: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
process.waitFor()에 타임아웃이 없습니다.Python 프로세스가 무한 대기하면 요청 스레드가 영구적으로 블록됩니다.
♻️ 제안하는 변경
- int exitCode = process.waitFor(); - if (exitCode == 0) { + if (!process.waitFor(10, TimeUnit.MINUTES)) { + process.destroyForcibly(); + throw new RuntimeException("크롤링 프로세스 시간 초과"); + } + int exitCode = process.exitValue(); + if (exitCode == 0) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.java` at line 46, Update the process-waiting logic around process.waitFor() in JobCrawlerExecutor to use a bounded timeout instead of waiting indefinitely. Handle the timeout outcome by terminating the Python process and preserving the existing failure/cleanup flow, while retaining normal exit-code handling when the process finishes within the limit.src/main/java/com/leets7th/job_is_be/domain/job/controller/JobCrawlerController.java (1)
28-31: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift장기 실행 파이프라인을 동기식으로 처리하면 HTTP 타임아웃 위험이 있습니다.
Python 크롤링 + DB 적재는 수분이 소요될 수 있습니다. 동기식 처리 시 클라이언트 연결이 타임아웃될 수 있습니다. 비동기 실행(
@Async또는CompletableFuture)과 작업 상태 조회 엔드포인트 분리를 고려하세요.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/domain/job/controller/JobCrawlerController.java` around lines 28 - 31, Update the JobCrawlerController flow around runPipelineAndSave so the long-running crawling and database pipeline starts asynchronously instead of blocking the HTTP request. Return an immediate accepted response, and add a separate status-query endpoint backed by the pipeline’s existing execution state so clients can monitor completion without relying on the original request.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@database/scripts/build_pool.py`:
- Line 41: Validate the --shards argument in the argument parser so only
positive integers are accepted, preventing zero-division and negative-index
failures during pool construction. Update the argparse definition around
ap.add_argument("--shards") and preserve the existing default value and
downstream shard-processing behavior.
In `@database/scripts/run_pipeline.py`:
- Around line 51-54: Update the exception handling around enrich in
database/scripts/run_pipeline.py lines 51-54 to keep enrichment_status as
pending and log the error instead of recording not_found; apply the same status
and logging policy in database/scripts/shard_extract.py lines 33-36 so both
execution paths distinguish transient enrichment failures from genuine not_found
results.
- Around line 57-61: Update the metadata key list in the enrichment-to-company
JSONL flow to include rejected_name, so the value returned by enrich() and
preserved during shard execution is copied from meta into comp alongside the
other audit fields.
In `@database/scripts/skill_enrich.py`:
- Around line 119-125: Protect empty-result statistics from division by zero: in
database/scripts/skill_enrich.py lines 119-125, update the final coverage output
to print 0 when n == 0 while preserving the existing percentages otherwise; in
database/scripts/merge_shards.py lines 39-46, handle np == 0 and nc == 0
independently so each corresponding percentage or statistic outputs 0 instead of
dividing by zero.
In `@database/scripts/wanted.py`:
- Around line 122-124: Generate skill titles and IDs as a single parallel data
contract: in database/scripts/wanted.py lines 122-124, first build (title, id)
pairs from each valid API skill, then split those pairs into skill_titles and
skill_ids; in database/scripts/skill_enrich.py lines 109-110, stop filtering
None IDs independently and either preserve their positions or exclude the
corresponding skill name together so title/ID indexes remain aligned.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/controller/JobCrawlerController.java`:
- Around line 13-31: Update triggerCrawlerPipeline in JobCrawlerController to
require ROLE_ADMIN authorization for /api/admin/crawler/run, using the project’s
established method- or request-level security configuration. Ensure
authenticated non-admin users cannot invoke the crawler pipeline while
preserving the existing execution behavior for administrators.
In `@src/main/java/com/leets7th/job_is_be/domain/job/service/CompanyService.java`:
- Around line 18-35: CompanyService.java:18-35의 getOrCreateCompany에서
REQUIRES_NEW 전파를 제거해 회사 저장이 상위 트랜잭션에 참여하도록 변경하세요. JobParserService.java:92-99의
processFile에서는 저장 또는 파싱 예외를 삼키지 말고 호출자에게 전파하여 전체 작업이 롤백되고 실패 응답이 반환되도록 하세요.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.java`:
- Around line 47-51: Update the exit-code branching in JobCrawlerExecutor so
jobParserService.parseAndSave() runs only when exitCode == 0 after successful
crawling, and is not called in the failure branch. Preserve the existing success
and failure log messages, and avoid introducing another parse invocation because
JobCrawlerManager already handles its call.
- Around line 28-29: Update JobCrawlerExecutor so the --out argument uses the
output path from CrawlerProperties, honoring the configured company-output-path
and job-output-path values instead of the hardcoded Windows directory. Reuse the
existing CrawlerProperties accessors and preserve the crawler command
construction.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerManager.java`:
- Around line 19-34: Remove `@Transactional` from runPipelineAndSave so the
long-running executePipeline call does not hold a database transaction; apply
transactionality to parseAndSave instead, and update the catch block in
runPipelineAndSave to rethrow the exception after logging so failures propagate
and database rollback is triggered.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/service/JobParserService.java`:
- Around line 60-76: Update the job construction and persistence flow in
JobParserService so each Job receives the crawler’s stable source and externalId
values from the DTO. Before JobRepository.save, look up an existing job by the
composite (source, externalId) key and update it when found; otherwise create a
new entity, ensuring repeated administrator API executions do not insert
duplicates.
- Around line 60-74: Update the JobParserService job-building and persistence
flow to map CrawledJobDto.categories and skills into the Job entity’s tag
relationships or existing converters, instead of using only the single
categoryName value. Ensure both collected category and skill tag lists are
included when the job is saved and remain available to service views.
In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java`:
- Around line 16-22: BaseEntity의 createdAt 및 updatedAt 매핑에서 컬럼명을 기존 스키마와 일치하도록
각각 create_at에서 created_at으로, update_at에서 updated_at으로 수정하세요.
---
Nitpick comments:
In `@build.gradle`:
- Around line 46-48: Remove the redundant spring-boot-starter-web dependency
from build.gradle; retain the existing spring-boot-starter-webmvc declaration as
the provider of the required web and Jackson dependencies.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/controller/JobCrawlerController.java`:
- Around line 28-31: Update the JobCrawlerController flow around
runPipelineAndSave so the long-running crawling and database pipeline starts
asynchronously instead of blocking the HTTP request. Return an immediate
accepted response, and add a separate status-query endpoint backed by the
pipeline’s existing execution state so clients can monitor completion without
relying on the original request.
In
`@src/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.java`:
- Line 38: Update the InputStreamReader construction in JobCrawlerExecutor to
specify the charset used by the Python process output instead of relying on the
platform default, ensuring Korean text is decoded correctly. Reuse the project’s
existing charset convention if one is defined.
- Line 46: Update the process-waiting logic around process.waitFor() in
JobCrawlerExecutor to use a bounded timeout instead of waiting indefinitely.
Handle the timeout outcome by terminating the Python process and preserving the
existing failure/cleanup flow, while retaining normal exit-code handling when
the process finishes within the limit.
In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java`:
- Around line 24-35: Remove the manual `@PrePersist` and `@PreUpdate` lifecycle
methods on BaseEntity, including onPrePersist and onPreUpdate, and retain the
existing `@EntityListeners`(AuditingEntityListener.class) with
`@CreatedDate/`@LastModifiedDate auditing as the sole mechanism for setting
timestamps.
In `@src/main/java/com/leets7th/job_is_be/global/config/AppConfig.java`:
- Around line 10-13: Update the objectMapper() bean to configure ObjectMapper to
ignore unknown JSON properties during deserialization, allowing
crawler-generated fields absent from DTOs without failing. Preserve the existing
bean definition and return the configured mapper.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 037106c5-b2d2-48f7-b07a-37866877be62
⛔ Files ignored due to path filters (3)
database/scripts/__pycache__/common.cpython-314.pycis excluded by!**/*.pycdatabase/scripts/__pycache__/jobkorea_company.cpython-314.pycis excluded by!**/*.pycdatabase/scripts/__pycache__/wanted.cpython-314.pycis excluded by!**/*.pyc
📒 Files selected for processing (25)
build.gradledatabase/scripts/build_pool.pydatabase/scripts/common.pydatabase/scripts/jobkorea_company.pydatabase/scripts/merge_shards.pydatabase/scripts/run_pipeline.pydatabase/scripts/shard_extract.pydatabase/scripts/skill_enrich.pydatabase/scripts/wanted.pysrc/main/java/com/leets7th/job_is_be/domain/job/controller/JobCrawlerController.javasrc/main/java/com/leets7th/job_is_be/domain/job/dto/CrawledCompanyDto.javasrc/main/java/com/leets7th/job_is_be/domain/job/dto/CrawledJobDto.javasrc/main/java/com/leets7th/job_is_be/domain/job/entity/Job.javasrc/main/java/com/leets7th/job_is_be/domain/job/repository/CompanyRepository.javasrc/main/java/com/leets7th/job_is_be/domain/job/repository/JobCategoryRepository.javasrc/main/java/com/leets7th/job_is_be/domain/job/repository/JobRepository.javasrc/main/java/com/leets7th/job_is_be/domain/job/repository/RegionRepository.javasrc/main/java/com/leets7th/job_is_be/domain/job/service/CompanyService.javasrc/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerExecutor.javasrc/main/java/com/leets7th/job_is_be/domain/job/service/JobCrawlerManager.javasrc/main/java/com/leets7th/job_is_be/domain/job/service/JobParserService.javasrc/main/java/com/leets7th/job_is_be/global/base/BaseEntity.javasrc/main/java/com/leets7th/job_is_be/global/config/AppConfig.javasrc/main/java/com/leets7th/job_is_be/global/config/CrawlerProperties.javasrc/main/resources/application.yml
Collaborator
Author
|
Job Entity 관련해서 PM님께서 준비하신 https://github.com/Leets-Official/Job-is-data 일단 제멋대로 연결해놨습니다. |
11 tasks
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java (1)
16-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win감사 시간 갱신 로직을 하나로 통일하세요.
JpaConfig에서@EnableJpaAuditing이 켜져 있으므로@CreatedDate/@LastModifiedDate와@PrePersist/@PreUpdate가 같은 필드를 중복 관리합니다. auditing만 쓰거나 lifecycle callback만 남겨서 한쪽만 유지하세요.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java` around lines 16 - 34, Unify timestamp management in BaseEntity by removing either the JPA auditing annotations or the `@PrePersist/`@PreUpdate lifecycle callbacks, leaving only one mechanism to maintain createdAt and updatedAt. Since JpaConfig enables auditing, prefer retaining `@CreatedDate` and `@LastModifiedDate` and remove the redundant callback methods.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java`:
- Around line 24-29: BaseEntity의 onPrePersist에서 OffsetDateTime.now()를 한 번만 호출해
기준 시각을 저장한 뒤, createdAt과 updatedAt 모두 동일한 값으로 설정하세요.
---
Nitpick comments:
In `@src/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java`:
- Around line 16-34: Unify timestamp management in BaseEntity by removing either
the JPA auditing annotations or the `@PrePersist/`@PreUpdate lifecycle callbacks,
leaving only one mechanism to maintain createdAt and updatedAt. Since JpaConfig
enables auditing, prefer retaining `@CreatedDate` and `@LastModifiedDate` and remove
the redundant callback methods.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 894ddd23-a210-4938-805e-de4d3dc20125
📒 Files selected for processing (4)
src/main/java/com/leets7th/job_is_be/domain/job/entity/Company.javasrc/main/java/com/leets7th/job_is_be/domain/job/entity/Job.javasrc/main/java/com/leets7th/job_is_be/domain/job/repository/JobRepository.javasrc/main/java/com/leets7th/job_is_be/global/base/BaseEntity.java
🚧 Files skipped from review as they are similar to previous changes (3)
- src/main/java/com/leets7th/job_is_be/domain/job/repository/JobRepository.java
- src/main/java/com/leets7th/job_is_be/domain/job/entity/Company.java
- src/main/java/com/leets7th/job_is_be/domain/job/entity/Job.java
jihoonkim501
left a comment
jihoonkim501
left a comment
Member
There was a problem hiding this comment.
제 코멘트 확인해주시고 코드레빗의 major, critical한 리뷰 반영 부탁드립니다~!
Comment on lines
+12
to
+16
| public class CrawledCompanyDto { | ||
| private String name; // 기업명 | ||
| private String logoUrl; // 로고 이미지 URL | ||
| private String description; // 기업 한 줄 소개 또는 설명 | ||
| } |
Member
There was a problem hiding this comment.
우리가 DTO 형태를 record로 가져가고 있어서 record 형식으로 수정부탁해요~!
| @Setter | ||
| @NoArgsConstructor | ||
| @JsonIgnoreProperties(ignoreUnknown = true) | ||
| public class CrawledJobDto { |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
database/scripts/merge_shards.py (2)
18-21: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win공고 dedup 키에
source를 포함해야 합니다.Line 21은
external_id만 키로 사용하므로 서로 다른 source에서 같은 ID를 가진 공고가 병합 단계에서 유실됩니다.(p["source"], p["external_id"])를 키로 사용하세요. 이는database/load.py:66-95의(source, external_id)upsert 계약과 충돌합니다.제안
- posts.setdefault(p["external_id"], p) + key = (p["source"], p["external_id"]) + posts.setdefault(key, p)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@database/scripts/merge_shards.py` around lines 18 - 21, Update the deduplication key in the shard-merging loop to use both p["source"] and p["external_id"], preserving the first occurrence for each source/ID pair. Keep the existing sorted file traversal and post record behavior unchanged.
24-29: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win허용되지 않은
enrichment_status를 병합 전에 거부하세요.Line 28의
RANK.get(..., 0)은 우선순위 fallback일 뿐 입력 검증이 아닙니다. 오타·누락·새 상태값이 그대로 출력되면database/schema.sql:33-34의 CHECK 제약조건 위반으로 DB 적재가 실패할 수 있습니다.RANK에 없는 상태는 병합 단계에서 즉시 거부하거나 명시적으로 정규화하세요.제안
c = json.loads(line) k = c["normalized_name"] + status = c.get("enrichment_status") + if status not in RANK: + raise ValueError(f"지원하지 않는 enrichment_status: {status}") cur = comps.get(k)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@database/scripts/merge_shards.py` around lines 24 - 29, In the merge logic around RANK and the component selection, validate each component’s enrichment_status before comparing or storing it. Immediately reject or explicitly normalize any missing or value not present in RANK, while preserving the existing higher-enrichment-status selection for valid statuses.
🧹 Nitpick comments (2)
database/scripts/merge_shards.py (1)
19-24: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win입력 JSONL 파일도 컨텍스트 매니저로 여세요.
Line 19와 Line 23의
open()호출을with open(...)으로 감싸 파일 수가 많은 병합에서도 파일 디스크립터가 누적되지 않도록 하세요.제안
- for line in open(fp, encoding="utf-8"): - p = json.loads(line) + with open(fp, encoding="utf-8") as f: + for line in f: + p = json.loads(line)기업 파일 순회에도 동일하게 적용해야 합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@database/scripts/merge_shards.py` around lines 19 - 24, Update both JSONL-reading loops in the shard merge flow to open files with context managers: the posts loop processing `posts_*.jsonl` and the companies loop processing `companies_*.jsonl`. Ensure each file is closed after its iteration while preserving the existing JSON parsing and merge behavior.database/scripts/run_pipeline.py (1)
54-55: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win두 실행 경로의 기업 보강 예외 로그 처리를 통일하세요.
두 파일 모두 예외 메시지를 stdout으로 출력하고
str(e)를 f-string에 사용합니다. 로그는 stderr로 보내고 명시적 변환 플래그를 사용해야 합니다.
database/scripts/run_pipeline.py#L54-L55:print(..., file=sys.stderr)와{e!s}를 적용하세요.database/scripts/shard_extract.py#L36-L37: 동일하게file=sys.stderr와{e!s}를 적용하세요.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@database/scripts/run_pipeline.py` around lines 54 - 55, Unify exception logging in the enrichment paths: in database/scripts/run_pipeline.py lines 54-55 and database/scripts/shard_extract.py lines 36-37, update the error print calls to write to stderr via file=sys.stderr and use the explicit {e!s} conversion; leave the metadata handling unchanged.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@database/scripts/build_pool.py`:
- Around line 45-46: Remove the stray literal “+” character from the body of the
shards validation conditional in the build-pool argument handling, leaving
ap.error("--shards must be greater than zero") correctly indented beneath if
a.shards <= 0 so the script parses and executes.
---
Outside diff comments:
In `@database/scripts/merge_shards.py`:
- Around line 18-21: Update the deduplication key in the shard-merging loop to
use both p["source"] and p["external_id"], preserving the first occurrence for
each source/ID pair. Keep the existing sorted file traversal and post record
behavior unchanged.
- Around line 24-29: In the merge logic around RANK and the component selection,
validate each component’s enrichment_status before comparing or storing it.
Immediately reject or explicitly normalize any missing or value not present in
RANK, while preserving the existing higher-enrichment-status selection for valid
statuses.
---
Nitpick comments:
In `@database/scripts/merge_shards.py`:
- Around line 19-24: Update both JSONL-reading loops in the shard merge flow to
open files with context managers: the posts loop processing `posts_*.jsonl` and
the companies loop processing `companies_*.jsonl`. Ensure each file is closed
after its iteration while preserving the existing JSON parsing and merge
behavior.
In `@database/scripts/run_pipeline.py`:
- Around line 54-55: Unify exception logging in the enrichment paths: in
database/scripts/run_pipeline.py lines 54-55 and
database/scripts/shard_extract.py lines 36-37, update the error print calls to
write to stderr via file=sys.stderr and use the explicit {e!s} conversion; leave
the metadata handling unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 266f4285-a39e-4a9c-a072-772be7a998dd
📒 Files selected for processing (6)
database/scripts/build_pool.pydatabase/scripts/merge_shards.pydatabase/scripts/run_pipeline.pydatabase/scripts/shard_extract.pydatabase/scripts/skill_enrich.pydatabase/scripts/wanted.py
🚧 Files skipped from review as they are similar to previous changes (2)
- database/scripts/skill_enrich.py
- database/scripts/wanted.py
jihoonkim501
left a comment
jihoonkim501
left a comment
Member
There was a problem hiding this comment.
예외처리 부분만 수정부탁드릴게요~! 수고 많으셨습니다!
| } | ||
| } catch (Exception e) { | ||
| log.error("{} 처리 중 오류 발생: ", type, e); | ||
| throw new RuntimeException(type + " 처리 중 예외 발생으로 데이터 적재를 중단하고 롤백합니다.", e); |
Member
There was a problem hiding this comment.
에러 처리방식을 GeneralException 형식으로 반환해주시면 좋을 것 같습니다!
11 tasks
# Conflicts: # src/main/java/com/leets7th/job_is_be/domain/job/entity/Job.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#️⃣ 관련 이슈
closed #15
PR 유형
어떤 변경 사항이 있나요?
PR Checklist
PR이 다음 요구 사항을 충족하는지 확인하세요.
🧩 작업 내용 - 파일 추가 및 변경 사항
JobCrawlerExecutor.java: ProcessBuilder를 이용해 파이썬 크롤링 스크립트를 OS 프로세스로 실행하고 로그 모니터링.JobCrawlerService.java: 크롤링 실행(Executor)과 데이터 적재(Parser)를 순차적으로 제어하는 총괄 서비스.JobCrawlerController.java: 크롤링 및 적재 과정을 수동 트리거. 관리자용 API 엔드포인트.JobParserService.java:.jsonl파일을 파싱하여 DB에 중복 없이 저장하고, 기업/지역/카테고리 엔티티를 매핑.**CrawledJobDto.java / CrawledCompanyDto.java**: 파이썬 결과물 데이터를 자바 객체로 변환하기 위한 DTO.CompanyService.java: 크롤링된 데이터의 기업 정보를 조회. DB에 존재하지 않는 경우, 새로 적재하는 기업 관리 로직 구현.CrawlerProperties.java: application.yml의 크롤러 설정값(경로 등)을 매핑하는 설정.application.yml: 파이썬 인터프리터 경로 및 입출력 파일 경로 설정.build.gradle: JSONL 파일 파싱을 위해 사용한 ObjectMapper용 Jackon 라이브러리 추가.Job.java / Company.java: 채용 공고와 기업 엔티티 수정 및 필드 보완.JobRepository.java: 공고 관련 조회 및 저장 로직 구현.CompanyRepository.java: 기업명 조회 및 신규 기업 적재(임시) 로직 구현.JobCategoryRepository.java / RegionRepository.java: 카테고리 및 지역 데이터 조회를 위한 레포지토리 추가.extractor/scripts/:공고 수집을 위한 핵심 로직(wanted.py, skill_enrich.py 등) 및 파이프라인 실행 스크립트 배치.테스트 이미지
크롤링 기능 검증을 위해 Swagger를 통한 수동 실행 테스트 완료
intelliJ console창을 통해 확인. 1세트(30개)
크롤링된 공고 데이터.
📣 To Reviewers
전달사항
공고 크롤링 구조 요약:
자바에서 파이썬 크롤링 프로세스를 실행, 생성된 데이터를 JSONL 파일로 파싱하여 데이터베이스에 자동 적재하는 구조입니다.
파이썬 크롤링 로직을 사용하기 위해:
application.yml에서 물리적 경로를 명시해주세요.파이썬 인터프리터(python-path)/ 스크립트 실행 경로(script-path)/ 데이터 입출력 경로(company/job-output-path)의 절대 경로.
Summary by CodeRabbit