Skip to content

Commit 9fe55fc

Browse files
connerohnesorgeconnerohnesorge
authored
proto copier (#102)
* implemented ability to copy proto files declaritively * add commented out bufrnix URL in flake.nix
1 parent 25f3fb5 commit 9fe55fc

18 files changed

Lines changed: 1713 additions & 0 deletions

File tree

‎examples/proto-copier/README.md‎

Lines changed: 293 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
# Proto Copier Example
2+
3+
This example demonstrates the **Proto Copier** functionality in Bufrnix, which allows you to declaratively copy Protocol Buffer files to multiple destinations without code generation. This is useful for:
4+
5+
- **Distributing proto files** to different services or teams
6+
- **Creating client SDKs** with proto definitions
7+
- **Synchronizing proto files** across microservices
8+
- **Building proto file packages** for different environments
9+
- **Organizing proto files** by access level or team
10+
11+
## 🏗️ Project Structure
12+
13+
```
14+
proto-copier/
15+
├── proto/ # Source proto files
16+
│ ├── api/v1/ # Public API definitions
17+
│ │ └── user_service.proto # User service API
18+
│ ├── common/v1/ # Shared type definitions
19+
│ │ └── types.proto # Common enums and messages
20+
│ ├── external/v1/ # External integration APIs
21+
│ │ └── webhook.proto # Webhook service definitions
22+
│ ├── internal/v1/ # Internal-only APIs
23+
│ │ └── admin_service.proto # Admin service (excluded from public copies)
24+
│ └── test/ # Test proto files
25+
│ └── test_user.proto # Test utilities (excluded from production)
26+
├── flake.nix # Multiple example configurations
27+
└── README.md # This file
28+
```
29+
30+
## 📦 Available Examples
31+
32+
### 1. **Default** - Basic Proto Copying
33+
```bash
34+
nix run
35+
```
36+
37+
**Configuration:**
38+
- **Preserve structure**: ✅ Maintains directory hierarchy
39+
- **Multiple destinations**: `backend/proto` and `frontend/src/proto`
40+
- **Filters**: Excludes `test/**` and `internal/**` directories
41+
- **Files**: All `.proto` files in the source directory
42+
43+
### 2. **Advanced Copy** - Multiple Destinations with Fine-Grained Control
44+
```bash
45+
nix run .#advanced-copy
46+
```
47+
48+
**Configuration:**
49+
- **Three destinations**: `public-api`, `shared-types`, `mobile-client`
50+
- **Specific patterns**: Only API, common, and external proto files
51+
- **Debug mode**: Enabled with verbose output
52+
- **Smart filtering**: Excludes internal and test files
53+
54+
### 3. **Flattened Copy** - Single Directory with Transformations
55+
```bash
56+
nix run .#flattened-copy
57+
```
58+
59+
**Configuration:**
60+
- **Flattened structure**: All files copied to single directory
61+
- **File transformations**: Adds `proto_` prefix and `_v1` suffix
62+
- **Example**: `user_service.proto` → `proto_user_service_v1.proto`
63+
64+
### 4. **API-Only Copy** - Selective File Copying
65+
```bash
66+
nix run .#api-only-copy
67+
```
68+
69+
**Configuration:**
70+
- **Specific files**: Only `user_service.proto` and `types.proto`
71+
- **Multiple destinations**: `api-client` and `sdk-generator`
72+
- **Use case**: Creating minimal client libraries
73+
74+
### 5. **Proto + Go** - Multi-Language Generation
75+
```bash
76+
nix run .#proto-and-go
77+
```
78+
79+
**Configuration:**
80+
- **Proto copying**: Distributes proto files to multiple locations
81+
- **Go code generation**: Generates gRPC Go code simultaneously
82+
- **Demonstrates**: How proto copying works alongside other language modules
83+
84+
## 🚀 Quick Start
85+
86+
1. **Enter development environment:**
87+
```bash
88+
nix develop
89+
```
90+
91+
2. **Run basic example:**
92+
```bash
93+
nix run
94+
```
95+
96+
3. **View copied files:**
97+
```bash
98+
tree output/
99+
```
100+
101+
4. **Try different examples:**
102+
```bash
103+
nix run .#advanced-copy
104+
nix run .#flattened-copy
105+
nix run .#api-only-copy
106+
nix run .#proto-and-go
107+
```
108+
109+
## ⚙️ Configuration Reference
110+
111+
### Proto Copier Options
112+
113+
| Option | Type | Default | Description |
114+
|--------|------|---------|-------------|
115+
| `enable` | `bool` | `false` | Enable proto file copying |
116+
| `outputPath` | `string \| string[]` | `["proto/copy"]` | Output destination(s) |
117+
| `preserveStructure` | `bool` | `true` | Maintain directory hierarchy |
118+
| `flattenFiles` | `bool` | `false` | Copy all files to output root |
119+
| `includePatterns` | `string[]` | `["*.proto"]` | File patterns to include |
120+
| `excludePatterns` | `string[]` | `[]` | File patterns to exclude |
121+
| `filePrefix` | `string` | `""` | Prefix for copied file names |
122+
| `fileSuffix` | `string` | `""` | Suffix for copied file names |
123+
124+
### Example Configuration
125+
126+
```nix
127+
languages.proto = {
128+
enable = true;
129+
copier = {
130+
enable = true;
131+
outputPath = [
132+
"backend/proto"
133+
"frontend/src/proto"
134+
"mobile/proto"
135+
];
136+
preserveStructure = true;
137+
includePatterns = [
138+
"api/**/*.proto"
139+
"common/**/*.proto"
140+
];
141+
excludePatterns = [
142+
"**/internal/**"
143+
"**/test/**"
144+
"**/*_test.proto"
145+
];
146+
filePrefix = "";
147+
fileSuffix = "_copy";
148+
};
149+
};
150+
```
151+
152+
## 📁 Output Examples
153+
154+
### Preserved Structure (Default)
155+
```
156+
output/backend/proto/
157+
├── api/v1/
158+
│ └── user_service.proto
159+
├── common/v1/
160+
│ └── types.proto
161+
└── external/v1/
162+
└── webhook.proto
163+
```
164+
165+
### Flattened Structure
166+
```
167+
output/flattened/
168+
├── proto_user_service_v1.proto
169+
├── proto_types_v1.proto
170+
└── proto_webhook_v1.proto
171+
```
172+
173+
## 🎯 Use Cases
174+
175+
### 1. **Microservice Proto Distribution**
176+
Copy public API definitions to multiple services while excluding internal APIs.
177+
178+
### 2. **Client SDK Generation**
179+
Distribute proto files to client teams for SDK generation in different languages.
180+
181+
### 3. **Mobile Development**
182+
Copy mobile-relevant proto files to mobile development environments.
183+
184+
### 4. **Third-Party Integration**
185+
Provide external partners with webhook and integration proto definitions.
186+
187+
### 5. **Multi-Environment Deployment**
188+
Copy proto files with different filtering rules for development, staging, and production.
189+
190+
## 🔧 Advanced Features
191+
192+
### Pattern Matching
193+
- **Glob patterns**: Use `**` for recursive matching
194+
- **Negation**: Exclude patterns override include patterns
195+
- **Flexible**: Support standard shell glob syntax
196+
197+
### File Transformations
198+
- **Prefixes**: Add consistent prefixes to all copied files
199+
- **Suffixes**: Add version or environment suffixes
200+
- **Combine**: Use both prefixes and suffixes together
201+
202+
### Multiple Outputs
203+
- **Different rules**: Each output can have different filtering
204+
- **Parallel copying**: All destinations copied in single operation
205+
- **Independent**: Each destination maintains separate directory structure
206+
207+
## 🧪 Testing
208+
209+
### Verify Basic Functionality
210+
```bash
211+
# Run basic copy
212+
nix run
213+
214+
# Check output structure
215+
tree output/backend/proto
216+
tree output/frontend/src/proto
217+
218+
# Verify file content
219+
cat output/backend/proto/api/v1/user_service.proto
220+
```
221+
222+
### Test Advanced Features
223+
```bash
224+
# Test flattened copy with transformations
225+
nix run .#flattened-copy
226+
ls output/flattened/
227+
228+
# Test selective copying
229+
nix run .#api-only-copy
230+
find output/api-client -name "*.proto"
231+
```
232+
233+
### Debug Mode
234+
```bash
235+
# Enable debug output
236+
nix run .#advanced-copy
237+
# Check for debug messages in output
238+
```
239+
240+
## 🤝 Integration with Other Languages
241+
242+
The proto copier works seamlessly with other Bufrnix language modules:
243+
244+
```nix
245+
languages = {
246+
# Copy proto files for distribution
247+
proto.copier = {
248+
enable = true;
249+
outputPath = ["clients/proto"];
250+
};
251+
252+
# Generate Go code
253+
go = {
254+
enable = true;
255+
outputPath = "gen/go";
256+
grpc.enable = true;
257+
};
258+
259+
# Generate TypeScript code
260+
js = {
261+
enable = true;
262+
outputPath = "gen/js";
263+
es.enable = true;
264+
};
265+
};
266+
```
267+
268+
## 🔍 Troubleshooting
269+
270+
### Common Issues
271+
272+
1. **No files copied**: Check `includePatterns` and `excludePatterns`
273+
2. **Wrong directory structure**: Verify `preserveStructure` setting
274+
3. **File not found**: Ensure source files exist in `sourceDirectories`
275+
4. **Permission errors**: Check output directory permissions
276+
277+
### Debug Tips
278+
279+
1. **Enable debug mode**: Set `debug.enable = true`
280+
2. **Use verbose output**: Set `debug.verbosity = 3`
281+
3. **Check patterns**: Test glob patterns with standard shell tools
282+
4. **Verify paths**: Ensure all paths are relative to project root
283+
284+
---
285+
286+
## 📚 Further Reading
287+
288+
- [Bufrnix Documentation](https://conneroisu.github.io/bufrnix/)
289+
- [Language Modules Reference](https://conneroisu.github.io/bufrnix/reference/languages/)
290+
- [Configuration Guide](https://conneroisu.github.io/bufrnix/reference/configuration/)
291+
- [More Examples](https://github.com/conneroisu/bufrnix/tree/main/examples)
292+
293+
This example demonstrates the power and flexibility of Bufrnix's proto copier functionality. Use it as a starting point for your own proto file distribution needs!

0 commit comments

Comments
 (0)