Update datasets chapter

Author: DerekMelchin

01 Cloud Platform/99 API Reference/08 Downloading Data/00.json

@@ -2,7 +2,7 @@
 	"type" : "landing",
 	"heading" : "API Reference",
 	"subHeading" : "",
-	"content" : "<p>The QuantConnect REST API lets you generate links to download data from our cloud servers through URL endpoints. To use the CLI to download data, see <a href='/docs/v2/lean-cli/datasets/downloading-quantconnect-data'>Downloading QuantConnect Data</a>.</p>",
+	"content" : "<p>The QuantConnect REST API lets you generate links to download data from our cloud servers through URL endpoints. To use the CLI to download data, see <a href='/docs/v2/lean-cli/datasets/downloading-data'>Downloading Data</a>.</p>",
 	"alsoLinks" : [
 	],
 	"featureShortDescription": {

05 Lean CLI/00.html

@@ -85,7 +85,7 @@ <h4>Initialization</h4>
         </div>
     </div>
 
-    <div class="internal-link" onclick="window.location.href = '/docs/v2/lean-cli/datasets/local-data'">
+    <div class="internal-link" onclick="window.location.href = '/docs/v2/lean-cli/datasets/format-and-storage'">
         <div class="internal-link-icon">
             <svg xmlns="http://www.w3.org/2000/svg" width="27" height="30" viewBox="0 0 27 30"><g transform="translate(-2 -1)"><path d="M12.5-1C19.209-1,26,.786,26,4.2S19.209,9.4,12.5,9.4-1,7.614-1,4.2,5.791-1,12.5-1Zm0,8.4a27.855,27.855,0,0,0,8.52-1.178C23.372,5.432,24,4.579,24,4.2s-.628-1.232-2.98-2.022A27.855,27.855,0,0,0,12.5,1,27.855,27.855,0,0,0,3.98,2.178C1.628,2.968,1,3.821,1,4.2s.628,1.232,2.98,2.022A27.855,27.855,0,0,0,12.5,7.4Z" transform="translate(3 2)" fill="#8f9ca3"/><path d="M15.5,17.2a29.9,29.9,0,0,1-9.173-1.281C2.751,14.716,2,13.124,2,12a1,1,0,0,1,2,0c0,.511.917,1.335,2.965,2.024A27.888,27.888,0,0,0,15.5,15.2a27.888,27.888,0,0,0,8.535-1.176C26.083,13.335,27,12.511,27,12a1,1,0,0,1,2,0c0,1.124-.751,2.716-4.327,3.919A29.9,29.9,0,0,1,15.5,17.2Z" transform="translate(0 4)" fill="#8f9ca3"/><path d="M15.5,29.8a29.9,29.9,0,0,1-9.173-1.281C2.751,27.316,2,25.724,2,24.6V5A1,1,0,0,1,4,5V24.6c0,.511.917,1.335,2.965,2.024A27.888,27.888,0,0,0,15.5,27.8a27.888,27.888,0,0,0,8.535-1.176C26.083,25.935,27,25.111,27,24.6V5a1,1,0,0,1,2,0V24.6c0,1.124-.751,2.716-4.327,3.919A29.9,29.9,0,0,1,15.5,29.8Z" transform="translate(0 1.2)" fill="#8f9ca3"/></g></svg>
         </div>

05 Lean CLI/01 Key Concepts/01 Getting Started/04 Basic Usage.html

@@ -42,7 +42,7 @@ <h4>Pull Projects From the Cloud</h4>
 <h4>Source Data</h4>
 <p>
     Run <code>lean data generate --start 20180101 --symbol-count 100</code> to generate realistic fake market data to test with.
-    You can also choose to <a href="/docs/v2/lean-cli/datasets/downloading-quantconnect-data">download data</a> from QuantConnect Datasets, or to convert your own data into LEAN-compatible data.
+    You can also choose to <a href="/docs/v2/lean-cli/datasets/downloading-data">download data</a> from QuantConnect Datasets or convert your own data into LEAN-compatible data.
 </p>
 
 <div class="cli section-example-container">

05 Lean CLI/05 Datasets/01 Format and Storage/01 Introduction.html

@@ -0,0 +1 @@
+<p>LEAN developed an open format based on open protocols (CSV, ZIP). The data format is simply compressed on disk.</p>

05 Lean CLI/05 Datasets/01 Local Data/02 Data Storage.html renamed to 05 Lean CLI/05 Datasets/01 Format and Storage/02 Default Location.html

@@ -4,6 +4,6 @@
 </p>
 
 <p>
-    For development purposes, it is also possible to <a href="/docs/v2/lean-cli/datasets/generating-random-data">generate data</a> using the CLI.
+    For development purposes, it is also possible to <a href="/docs/v2/lean-cli/datasets/generating-data">generate data</a> using the CLI.
     This generator uses a <a href="https://en.wikipedia.org/wiki/Brownian_model_of_financial_markets" target="_blank">Brownian motion model</a> to generate realistic market data, which might be helpful when you're testing strategies locally but don't have access to real market data.
 </p>

05 Lean CLI/05 Datasets/01 Local Data/03 Custom Data Directory.html renamed to 05 Lean CLI/05 Datasets/01 Format and Storage/03 Change Location.html

@@ -0,0 +1,12 @@
+{
+    "type": "metadata",
+    "values": {
+        "description": "LEAN developed an open format based on open protocols (CSV, ZIP). The data format is simply compressed on disk.",
+        "keywords": "csv files, zip compression, algo trading data structure, lean data formats",
+        "og:description": "LEAN developed an open format based on open protocols (CSV, ZIP). The data format is simply compressed on disk.",
+        "og:title": "Format and Storage - Documentation QuantConnect.com",
+        "og:type": "website",
+        "og:site_name": "Format and Storage - QuantConnect.com",
+        "og:image": "https://cdn.quantconnect.com/docs/i/lean-cli/datasets/format-and-storage.png"
+    }
+}

05 Lean CLI/05 Datasets/01 Local Data/05 Other Data Sources.html renamed to 05 Lean CLI/05 Datasets/01 Format and Storage/05 Other Data Sources.html

@@ -0,0 +1,13 @@
+{
+  "type" : "landing",
+  "heading" : "Downloading Data",
+  "subHeading" : "",
+  "content" : "<p>Running the LEAN engine locally with the CLI requires you to have your own local data. The recommended way to source local data is by purchasing and downloading it from our Dataset Market. This way, you can use the same data that is available to your algorithm when you run it in the cloud. There are two methods of downloading data. You can download smaller discrete datasets at a low cost or download complete collections in bulk to avoid selection bias.</p>",
+  "alsoLinks" : [
+    { "text" : "Datasets" , "href" : "/docs/v2/writing-algorithms/datasets/overview" }
+  ],
+  "featureShortDescription": {
+    "01" : "Low cost option for individual tickers",
+    "02" : "All tickers to avoid selection bias"
+  }
+}

05 Lean CLI/05 Datasets/01 Format and Storage/metadata.json

@@ -0,0 +1 @@
+<p>Downloading data by the ticker is a low cost option to aquiring local trading data. This technique is for smaller, discrete downloads instead of downloading an entire dataset. You can download our ticker data through the CLI or LEAN in exchange for some <a href='https://www.quantconnect.com/docs/v2/cloud-platform/organizations/credit'>QuantConnect Credit</a> (QCC). Before the CLI or LEAN download new files, they check if your local machine already stores the files.</p>

05 Lean CLI/05 Datasets/01 Local Data/01 Introduction.html

@@ -0,0 +1,183 @@
+<p>
+    Follow these steps to purchase and download data from the Dataset Market using the CLI:
+</p>
+
+<ol>
+    <li><a href="/docs/v2/lean-cli/initialization/authenticating-accounts#02-Log-In">Log in</a> to the CLI if you haven't done so already.</li>
+    <li>Open a terminal in one of your <a href='/docs/v2/lean-cli/initialization/workspace'>workspace directories</a>.</li>
+    <li>Run <code>lean data download</code> to start an interactive downloading wizard.</li>
+    <li>Go through the interactive wizard to configure the data you want to download.</li>
+    <li>After configuring the data you want to download, enter <span class='key-combinations'>N</span> when asked whether you want to download more data.
+<div class="cli section-example-container">
+<pre>$ lean data download
+Selected data:
+...
+Total price: 10 QCC
+Organization balance: 20,096,264 QCC
+Do you want to download more data? [y/N]: N</pre>
+</div>
+    </li>
+    <li>In the browser window that opens, read the <span class='document-title'>CLI API Access and Data Agreement</span> and click <span class='button-name'>I Accept The Data Agreement</span>.</li>
+    <li>Go back to the terminal and confirm the purchase to start downloading.
+<div class="cli section-example-container">
+<pre>$ lean data download
+You will be charged 10 QCC from your organization's QCC balance
+After downloading all files your organization will have 20,096,254 QCC left
+Continue? [y/N]: y
+[1/1] Downloading equity/usa/daily/spy.zip (10 QCC)</pre>
+</div>
+    </li>
+</ol>
+
+<p>Many of the datasets depend on the <a href='https://www.quantconnect.com/datasets/quantconnect-us-equity-security-master'>US Equity Security Master</a> dataset because the because the US Equity Security Master contains information on splits, dividends, and symbol changes. To check if a dataset depends the US Equity Security Master, see the listing in the <a href='https://www.quantconnect.com/datasets/'>Dataset Market</a>. If you try download a dataset through the CLI that depends on the US Equity Security Master and you don't have an active subscription to it, you'll get an error.</p>
+
+<p>Follow these steps to download US Equity data from the Dataset Market:</p>
+
+<ol>
+    <li>Open a terminal in one of your <a href='/docs/v2/lean-cli/initialization/workspace'>workspace directories</a>.</li>
+    <li>Run <code>lean data download</code> to start an interactive downloading wizard and then enter the dataset category number.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Select a category:
+1) Commerce Data (3 datasets)
+2) Consumer Data (2 datasets)
+3) Energy Data (2 datasets)
+4) Environmental Data (1 dataset)
+5) Financial Market Data (30 datasets)
+6) Industry Data (1 dataset)
+7) Legal Data (2 datasets)
+8) News and Events (4 datasets)
+9) Political Data (2 datasets)
+10) Transport and Logistics Data (1 dataset)
+11) Web Data (9 datasets)
+Enter an option: 5</pre>
+        </div>
+    </li>
+    <li>Enter the dataset number.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Select a dataset:
+1) Binance Crypto Future Margin Rate Data
+2) Binance Crypto Future Price Data
+3) Binance Crypto Price Data
+4) Binance US Crypto Price Data
+5) Bitcoin Metadata
+6) Bitfinex Crypto Price Data
+7) Brain Language Metrics on Company Filings
+8) Brain ML Stock Ranking
+9) CFD Data
+10) Coinbase Crypto Price Data
+11) Composite Factor Bundle
+12) Cross Asset Model
+13) FOREX Data
+14) Insider Trading
+15) Kraken Crypto Price Data
+16) NFT Sales
+17) Tactical
+18) Template (Do not Edit)
+19) US Coarse Universe
+20) US Congress Trading
+21) US ETF Constituents
+22) US Equities
+23) US Equity Options
+24) US Equity Security Master
+25) US Federal Reserve (FRED)
+26) US Futures Security Master
+27) US SEC Filings
+28) US Treasury Yield Curve
+29) VIX Central Contango
+30) VIX Daily Price
+Enter an option: 22</pre>
+        </div>
+
+        <p>If you don't have an active subscription to the US Equity Security Master, you'll get the following error message:</p>
+        <div class='error-messages'>Your organization needs to have an active Security Master subscription to download data from the 'US Equities' dataset<br>You can add the subscription at https://www.quantconnect.com/datasets/quantconnect-security-master/pricing</div>
+    </li>
+
+    <li>Enter the data type.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Data type:
+1) Trade
+2) Quote
+3) Bulk
+Enter an option: 1</pre>
+        </div>
+    </li>
+
+    <li>Enter the ticker(s).
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Ticker(s) (comma-separated): SPY</pre>
+        </div>
+    </li>
+
+    <li>Enter the resolution.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Resolution:
+1) Tick
+2) Second
+3) Minute
+4) Hour
+5) Daily
+Enter an option: 3</pre>
+        </div>
+    </li>
+
+    <li>If you selected tick, second, or minute resolution in the previous step, enter the start and end date.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Start date (yyyyMMdd): 20230101
+End date (yyyyMMdd): 20230105</pre>
+        </div>
+    </li>
+
+
+    <li>Review your selected data and enter whether you would like to download more data.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Selected data:
+┌─────────────┬──────────┬────────────────────────┬────────────┬────────┐
+│ Dataset     │ Vendor   │ Details                │ File count │ Price  │
+├─────────────┼──────────┼────────────────────────┼────────────┼────────┤
+│ US Equities │ AlgoSeek │ Data type: Trade       │ 3          │ 15 QCC │
+│             │          │ Ticker: SPY            │            │        │
+│             │          │ Resolution: Minute     │            │        │
+│             │          │ Start date: 2023-01-01 │            │        │
+│             │          │ End date: 2023-01-05   │            │        │
+└─────────────┴──────────┴────────────────────────┴────────────┴────────┘
+Total price: 15 QCC
+Organization balance: 100 QCC
+Do you want to download more data? [y/N]: n</pre>
+        </div>
+    </li>
+
+
+    <li>If you haven't already agreed to the <span class='document-title'>CLI API Access and Data Agreement</span>, in the browser window that opens, read the document and click <span class='button-name'>I Accept The Data Agreement</span>.</li>
+
+    <li>Confirm your data purchase.
+        <div class="cli section-example-container">
+            <pre>$ lean data download
+Data Terms of Use has been signed previously.
+Find full agreement at: https://www.quantconnect.com/terms/data/?organization=&lt;organizationId&gt;
+==========================================================================
+CLI API Access Agreement: On 2022-04-12 22:34:26 You Agreed:
+- Display or distribution of data obtained through CLI API Access is not permitted.
+- Data and Third Party Data obtained via CLI API Access can only be used for individual or internal employee's use.
+- Data is provided in LEAN format can not be manipulated for transmission or use in other applications.
+- QuantConnect is not liable for the quality of data received and is not responsible for trading losses.
+==========================================================================
+
+You will be charged 15 QCC from your organization's QCC balance
+After downloading all files your organization will have 85 QCC left
+Continue? [y/N]: y</pre>
+        </div>
+    </li>
+
+    <p>After you confirm your data purchsae, the CLI downloads the data and prints its progress.</p>
+    <div class="cli section-example-container">
+        <pre>$ lean data download
+ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% (3/3)</pre>
+    </div>
+</ol>

05 Lean CLI/05 Datasets/01 Local Data/04 QuantConnect Datasets.html

@@ -0,0 +1 @@
+<p>To view all of the supported datasets, see the <a href='https://www.quantconnect.com/datasets'>Dataset Market</a>.</p>