Skip to content

Commit 06d02cc

Browse files
nonsenseLexLuthrdirkmcLexLuthr
authored
direct data onboarding - disabled (#1627)
* add dummy direct deals provider * add initial `import-direct` cmd * updated make gen and make docsgen * direct deals provider * steps to onboard direct data * sql table; direct data entry state; ORM * scaffolding up to Filecoin.SectorAddPieceToAny * rename DirectData to DirectDeal * remove docker-tc * fix AllocationId type to uint64 * remove VerifiedDeal field in DirectDeal table * update ifaces to add fast-retrieval and skip-ipni-announce * fast-retrieval becomes remove-unsealed-copy * add checkpoints to direct deal model * initial state into local db * add extra fields * feat: add `boost allocate` and `boost get-allocations` command (#1632) * add allocate & get-allocations command * comment out fullNode dependent lines * clarify error * update lotus, uncomment check code * go mod tidy * bump FFI, fix EthCall * docsgen * direct data onboarding: refactor/extract commp, use checkpoints for retries, retries on startup (#1650) * extract and refactor commp for direct deals * split Import into Import and Process, and Process all deals on startup * update Process with checkpoints check * handle commp size * devnet support for filplus (#1655) * comment * add signers and notaries * fixup * fixup * get allocation * lock lotus version * lock lotus to latest PR version * fill PieceDealInfo * bump up datacap * check for nil allocation; set ClientAddress and ProviderAddress in sqlitedb * take actor ID for a wallet from state * fixup * add start and end epoch * add startEpochSealingBuffer and compare startEpoch against chain head * comments and fixup * commpcalc; listActive * fix: doc gen * Web UI: direct deals list (#1665) * feat: ui - direct deals list * feat: direct deals ui - direct deal detail page (#1669) * fix: test compilation * fix: cbor gen * refactor: simplify db code * feat: ui - add direct deals link to menu * fix: direct deals menu item * feat: add more deal logs * feat: user defined allocation policy (#1668) * user defined allocation policy * refactor flags * fix typo * fix devnet LID, fix flag names * refactor: extract piece reading code into common function * feat: add deal to sector with retry * feat: add retryability to direct deals (#1670) * fix nil ptr exception due to missing datacap * fix: allocation bug, refactor func()s (#1673) * fix allocation bug, refactor func()s * fix lint err * feat: make retry / fail resolvers work with direct deals (#1675) * create DDProvider config (#1679) * add itest for direct data onboarding (#1684) * feat: add itest for direct data onboarding * refactor: move datacap code from ddo test function into itest framework * Watch sealing state until finalized (#1687) * feat: watch sealing state until finalized * fix: dont check deal id when watching sealing state for direct deals * fix: checking sealing sector state * refactor: use common commp throttle for direct and regular deals (#1695) * feat: watch sealing state until finalized * fix: dont check deal id when watching sealing state for direct deals * fix: checking sealing sector state * refactor: use common commp throttle for direct and regular deals * feat: add tests for direct deals db (#1700) * feat: index and announce direct deals (#1696) * index and announce * add IsDirectDeal to LID * fix new column type * add keyspace to cql * rm keyspace from add column * Use UUID for idxProv context * apply suggestions * add column in migration * fix lint err * update ipni announce log line * revert to lotus version without ddo * go.mod from main * comment out ddo-related code; revert itest framework * comment out itest for direct deals * cleanup * Update node/impl/boost.go Co-authored-by: LexLuthr <[email protected]> * fix commented out direct deals component * comment out direct deals cmd * fix cbor-gen --------- Co-authored-by: LexLuthr <[email protected]> Co-authored-by: Dirk McCormick <[email protected]> Co-authored-by: LexLuthr <[email protected]>
1 parent fc59a5d commit 06d02cc

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+3986
-367
lines changed

.circleci/config.yml

+5
Original file line numberDiff line numberDiff line change
@@ -343,6 +343,11 @@ workflows:
343343
suite: itest-disabled_markets_v1_deal
344344
target: "./itests/disabled_markets_v1_deal_test.go"
345345

346+
#- test:
347+
#name: test-itest-direct_deal
348+
#suite: itest-direct_deal
349+
#target: "./itests/ddo_test.go"
350+
346351
- test:
347352
name: test-all
348353
suite: all

api/api.go

+1
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ type Boost interface {
5050
BoostDagstorePiecesContainingMultihash(ctx context.Context, mh multihash.Multihash) ([]cid.Cid, error) //perm:read
5151
BoostDagstoreListShards(ctx context.Context) ([]DagstoreShardInfo, error) //perm:admin
5252
BoostMakeDeal(context.Context, smtypes.DealParams) (*ProviderDealRejectionInfo, error) //perm:write
53+
BoostDirectDeal(ctx context.Context, params smtypes.DirectDealParams) (*ProviderDealRejectionInfo, error) //perm:admin
5354

5455
// MethodGroup: Blockstore
5556
BlockstoreGet(ctx context.Context, c cid.Cid) ([]byte, error) //perm:read

api/proxy_gen.go

+13
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

build/openrpc/boost.json.gz

145 Bytes
Binary file not shown.

cmd/boost/direct_deal.god

+287
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,287 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
"strings"
7+
8+
bcli "github.com/filecoin-project/boost/cli"
9+
clinode "github.com/filecoin-project/boost/cli/node"
10+
"github.com/filecoin-project/boost/cmd"
11+
"github.com/filecoin-project/boost/cmd/boost/util"
12+
"github.com/filecoin-project/boost/cmd/lib"
13+
"github.com/filecoin-project/go-address"
14+
"github.com/filecoin-project/go-state-types/abi"
15+
verifregst "github.com/filecoin-project/go-state-types/builtin/v9/verifreg"
16+
lapi "github.com/filecoin-project/lotus/api"
17+
"github.com/filecoin-project/lotus/chain/actors/builtin/verifreg"
18+
"github.com/filecoin-project/lotus/chain/types"
19+
lcli "github.com/filecoin-project/lotus/cli"
20+
"github.com/filecoin-project/lotus/lib/tablewriter"
21+
"github.com/urfave/cli/v2"
22+
)
23+
24+
var directDealAllocate = &cli.Command{
25+
Name: "allocate",
26+
Usage: "Create new allocation[s] for verified deals",
27+
Flags: []cli.Flag{
28+
&cli.StringSliceFlag{
29+
Name: "miner",
30+
Usage: "storage provider address[es]",
31+
Required: true,
32+
Aliases: []string{"m", "provider", "p"},
33+
},
34+
&cli.StringSliceFlag{
35+
Name: "piece-info",
36+
Usage: "data piece-info[s] to create the allocation. The format must be --piece-info pieceCid1=pieceSize1 --piece-info pieceCid2=pieceSize2",
37+
Required: true,
38+
Aliases: []string{"pi"},
39+
},
40+
&cli.StringFlag{
41+
Name: "wallet",
42+
Usage: "the wallet address that will used create the allocation",
43+
},
44+
&cli.BoolFlag{
45+
Name: "quiet",
46+
Usage: "do not print the allocation list",
47+
Value: false,
48+
},
49+
&cli.Int64Flag{
50+
Name: "term-min",
51+
Usage: "The minimum duration which the provider must commit to storing the piece to avoid early-termination penalties (epochs).\n" +
52+
"Default is 180 days.",
53+
Aliases: []string{"tmin"},
54+
Value: verifregst.MinimumVerifiedAllocationTerm,
55+
},
56+
&cli.Int64Flag{
57+
Name: "term-max",
58+
Usage: "The maximum period for which a provider can earn quality-adjusted power for the piece (epochs).\n" +
59+
"Default is 5 years.",
60+
Aliases: []string{"tmax"},
61+
Value: verifregst.MaximumVerifiedAllocationTerm,
62+
},
63+
&cli.Int64Flag{
64+
Name: "expiration",
65+
Usage: "The latest epoch by which a provider must commit data before the allocation expires (epochs).\n" +
66+
"Default is 60 days.",
67+
Value: verifregst.MaximumVerifiedAllocationExpiration,
68+
},
69+
},
70+
Before: before,
71+
Action: func(cctx *cli.Context) error {
72+
ctx := bcli.ReqContext(cctx)
73+
74+
n, err := clinode.Setup(cctx.String(cmd.FlagRepo.Name))
75+
if err != nil {
76+
return err
77+
}
78+
79+
gapi, closer, err := lcli.GetGatewayAPI(cctx)
80+
if err != nil {
81+
return fmt.Errorf("can't setup gateway connection: %w", err)
82+
}
83+
defer closer()
84+
85+
// Get wallet address from input
86+
walletAddr, err := n.GetProvidedOrDefaultWallet(ctx, cctx.String("wallet"))
87+
if err != nil {
88+
return err
89+
}
90+
91+
log.Debugw("selected wallet", "wallet", walletAddr)
92+
93+
msg, err := util.CreateAllocationMsg(ctx, gapi, cctx.StringSlice("piece-info"), cctx.StringSlice("miner"), walletAddr, abi.ChainEpoch(cctx.Int64("term-min")), abi.ChainEpoch(cctx.Int64("term-max")), abi.ChainEpoch(cctx.Int64("expiration")))
94+
95+
if err != nil {
96+
return err
97+
}
98+
99+
oldallocations, err := gapi.StateGetAllocations(ctx, walletAddr, types.EmptyTSK)
100+
if err != nil {
101+
return fmt.Errorf("failed to get allocations: %w", err)
102+
}
103+
104+
mcid, sent, err := lib.SignAndPushToMpool(cctx, ctx, gapi, n, msg)
105+
if err != nil {
106+
return err
107+
}
108+
if !sent {
109+
return nil
110+
}
111+
112+
log.Infow("submitted data cap allocation message", "cid", mcid.String())
113+
log.Info("waiting for message to be included in a block")
114+
115+
res, err := gapi.StateWaitMsg(ctx, mcid, 1, lapi.LookbackNoLimit, true)
116+
if err != nil {
117+
return fmt.Errorf("waiting for message to be included in a block: %w", err)
118+
}
119+
120+
if !res.Receipt.ExitCode.IsSuccess() {
121+
return fmt.Errorf("failed to execute the message with error: %s", res.Receipt.ExitCode.Error())
122+
}
123+
124+
// Return early of quiet flag is set
125+
if cctx.Bool("quiet") {
126+
return nil
127+
}
128+
129+
newallocations, err := gapi.StateGetAllocations(ctx, walletAddr, types.EmptyTSK)
130+
if err != nil {
131+
return fmt.Errorf("failed to get allocations: %w", err)
132+
}
133+
134+
// Generate a diff to find new allocations
135+
for i := range newallocations {
136+
_, ok := oldallocations[i]
137+
if ok {
138+
delete(newallocations, i)
139+
}
140+
}
141+
142+
return printAllocation(newallocations, cctx.Bool("json"))
143+
},
144+
}
145+
146+
var directDealGetAllocations = &cli.Command{
147+
Name: "list-allocations",
148+
Usage: "Lists all allocations for a client address(wallet)",
149+
Flags: []cli.Flag{
150+
&cli.StringFlag{
151+
Name: "miner",
152+
Usage: "Storage provider address. If provided, only allocations against this minerID will be printed",
153+
Aliases: []string{"m", "provider", "p"},
154+
},
155+
&cli.StringFlag{
156+
Name: "wallet",
157+
Usage: "the wallet address that will used create the allocation",
158+
},
159+
},
160+
Before: before,
161+
Action: func(cctx *cli.Context) error {
162+
ctx := bcli.ReqContext(cctx)
163+
164+
n, err := clinode.Setup(cctx.String(cmd.FlagRepo.Name))
165+
if err != nil {
166+
return err
167+
}
168+
169+
gapi, closer, err := lcli.GetGatewayAPI(cctx)
170+
if err != nil {
171+
return fmt.Errorf("cant setup gateway connection: %w", err)
172+
}
173+
defer closer()
174+
175+
// Get wallet address from input
176+
walletAddr, err := n.GetProvidedOrDefaultWallet(ctx, cctx.String("wallet"))
177+
if err != nil {
178+
return err
179+
}
180+
181+
log.Debugw("selected wallet", "wallet", walletAddr)
182+
183+
allocations, err := gapi.StateGetAllocations(ctx, walletAddr, types.EmptyTSK)
184+
if err != nil {
185+
return fmt.Errorf("failed to get allocations: %w", err)
186+
}
187+
188+
if cctx.String("miner") != "" {
189+
// Get all minerIDs from input
190+
minerId := cctx.String("miner")
191+
maddr, err := address.NewFromString(minerId)
192+
if err != nil {
193+
return err
194+
}
195+
196+
// Verify that minerID exists
197+
_, err = gapi.StateMinerInfo(ctx, maddr, types.EmptyTSK)
198+
if err != nil {
199+
return err
200+
}
201+
202+
mid, err := address.IDFromAddress(maddr)
203+
if err != nil {
204+
return err
205+
}
206+
207+
for i, v := range allocations {
208+
if v.Provider != abi.ActorID(mid) {
209+
delete(allocations, i)
210+
}
211+
}
212+
}
213+
214+
return printAllocation(allocations, cctx.Bool("json"))
215+
},
216+
}
217+
218+
func printAllocation(allocations map[verifreg.AllocationId]verifreg.Allocation, json bool) error {
219+
// Map Keys. Corresponds to the standard tablewriter output
220+
allocationID := "AllocationID"
221+
client := "Client"
222+
provider := "Miner"
223+
pieceCid := "PieceCid"
224+
pieceSize := "PieceSize"
225+
tMin := "TermMin"
226+
tMax := "TermMax"
227+
expr := "Expiration"
228+
229+
// One-to-one mapping between tablewriter keys and JSON keys
230+
tableKeysToJsonKeys := map[string]string{
231+
allocationID: strings.ToLower(allocationID),
232+
client: strings.ToLower(client),
233+
provider: strings.ToLower(provider),
234+
pieceCid: strings.ToLower(pieceCid),
235+
pieceSize: strings.ToLower(pieceSize),
236+
tMin: strings.ToLower(tMin),
237+
tMax: strings.ToLower(tMax),
238+
expr: strings.ToLower(expr),
239+
}
240+
241+
var allocs []map[string]interface{}
242+
243+
for key, val := range allocations {
244+
alloc := map[string]interface{}{
245+
allocationID: key,
246+
client: val.Client,
247+
provider: val.Provider,
248+
pieceCid: val.Data,
249+
pieceSize: val.Size,
250+
tMin: val.TermMin,
251+
tMax: val.TermMax,
252+
expr: val.Expiration,
253+
}
254+
allocs = append(allocs, alloc)
255+
}
256+
257+
if json {
258+
// get a new list of wallets with json keys instead of tablewriter keys
259+
var jsonAllocs []map[string]interface{}
260+
for _, alloc := range allocs {
261+
jsonAlloc := make(map[string]interface{})
262+
for k, v := range alloc {
263+
jsonAlloc[tableKeysToJsonKeys[k]] = v
264+
}
265+
jsonAllocs = append(jsonAllocs, jsonAlloc)
266+
}
267+
// then return this!
268+
return cmd.PrintJson(jsonAllocs)
269+
} else {
270+
// Init the tablewriter's columns
271+
tw := tablewriter.New(
272+
tablewriter.Col(allocationID),
273+
tablewriter.Col(client),
274+
tablewriter.Col(provider),
275+
tablewriter.Col(pieceCid),
276+
tablewriter.Col(pieceSize),
277+
tablewriter.Col(tMin),
278+
tablewriter.Col(tMax),
279+
tablewriter.NewLineCol(expr))
280+
// populate it with content
281+
for _, alloc := range allocs {
282+
tw.Write(alloc)
283+
}
284+
// return the corresponding string
285+
return tw.Flush(os.Stdout)
286+
}
287+
}

cmd/boost/main.go

+4-2
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,11 @@ import (
44
"encoding/json"
55
"fmt"
66
"io"
7+
llog "log"
78
"os"
89

910
"github.com/filecoin-project/boost/cmd"
1011

11-
llog "log"
12-
1312
"github.com/filecoin-project/boost/build"
1413
"github.com/filecoin-project/boostd-data/shared/cliutil"
1514
logging "github.com/ipfs/go-log/v2"
@@ -41,6 +40,9 @@ func main() {
4140
offlineDealCmd,
4241
providerCmd,
4342
walletCmd,
43+
//TODO: enable when DDO ends up in a network upgrade
44+
//directDealAllocate,
45+
//directDealGetAllocations,
4446
},
4547
}
4648
app.Setup()

0 commit comments

Comments
 (0)