docs: update setting up and running in cloud (#936)

- I updated the "Setting up" and "Running in cloud" pages to make them
more beginner-friendly (hopefully) and also removed mentions of Poetry.
Including Poetry was unnecessary and could be confusing. Not to mention
in Python there are X other package managers that we could hypotetically
mention as well. So I decided to stay only with Pip and Pipx for
templates. People using a package manager other than Pip will likely
know how to handle the installation on their own.
- This was reported on Slack by @honzajavorek.

Author: vdusek

Makefile

@@ -1,7 +1,5 @@
 .PHONY: clean install-dev build publish-to-pypi lint type-check unit-tests unit-tests-cov integration-tests format check-code build-api-reference run-docs
 
-DIRS_WITH_CODE = src tests docs website
-
 # This is default for local testing, but GitHub workflows override it to a higher value in CI
 INTEGRATION_TESTS_CONCURRENCY = 1
 
@@ -22,11 +20,11 @@ publish-to-pypi:
 	poetry publish --no-interaction -vv
 
 lint:
-	poetry run ruff format --check $(DIRS_WITH_CODE)
-	poetry run ruff check $(DIRS_WITH_CODE)
+	poetry run ruff format --check
+	poetry run ruff check
 
 type-check:
-	poetry run mypy $(DIRS_WITH_CODE)
+	poetry run mypy
 
 unit-tests:
 	poetry run pytest --numprocesses=auto --verbose --cov=src/crawlee tests/unit
@@ -38,8 +36,8 @@ integration-tests:
 	poetry run pytest --numprocesses=$(INTEGRATION_TESTS_CONCURRENCY) tests/integration
 
 format:
-	poetry run ruff check --fix $(DIRS_WITH_CODE)
-	poetry run ruff format $(DIRS_WITH_CODE)
+	poetry run ruff check --fix
+	poetry run ruff format
 
 # The check-code target runs a series of checks equivalent to those performed by pre-commit hooks
 # and the run_checks.yaml GitHub Actions workflow.

README.md

@@ -38,10 +38,12 @@ We also have a TypeScript implementation of the Crawlee, which you can explore a
 
 We recommend visiting the [Introduction tutorial](https://crawlee.dev/python/docs/introduction) in Crawlee documentation for more information.
 
-Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package. The core functionality is included in the base package, with additional features available as optional extras to minimize package size and dependencies. To install Crawlee with all features, run the following command:
+Crawlee is available as [`crawlee`](https://pypi.org/project/crawlee/) package on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.
+
+To install Crawlee with all features, run the following command:
 
 ```sh
-pip install 'crawlee[all]'
+python -m pip install 'crawlee[all]'
 ```
 
 Then, install the [Playwright](https://playwright.dev/) dependencies:

docs/deployment/apify_platform.mdx

@@ -6,8 +6,6 @@ description: Apify platform - large-scale and high-performance web scraping
 
 import ApiLink from '@site/src/components/ApiLink';
 
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
 import CodeBlock from '@theme/CodeBlock';
 
 import LogWithConfigExample from '!!raw-loader!./code/apify/log_with_config_example.py';
@@ -27,7 +25,7 @@ We do not test Crawlee in other cloud environments such as Lambda or on specific
 
 ## Logging into Apify platform from Crawlee
 
-To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment variables.
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that either by utilizing [Apify CLI](https://docs.apify.com/cli/) or with environment variables.
 
 Once you provide credentials to your Apify CLI installation, you will be able to use all the Apify platform features, such as calling Actors, saving to cloud storages, using Apify proxies, setting up webhooks and so on.
 
@@ -142,7 +140,7 @@ If you don't plan to force usage of the platform storages when running the Actor
 {/*
 ### Getting public url of an item in the platform storage
 
-If you need to share a link to some file stored in a [Key-Value](https://docs.apify.com/sdk/python/reference/class/KeyValueStore) Store on Apify Platform, you can use [`get_public_url()`](https://docs.apify.com/sdk/python/reference/class/KeyValueStore#get_public_url) method. It accepts only one parameter: `key` - the key of the item you want to share.
+If you need to share a link to some file stored in a [Key-Value](https://docs.apify.com/sdk/python/reference/class/KeyValueStore) Store on Apify platform, you can use [`get_public_url()`](https://docs.apify.com/sdk/python/reference/class/KeyValueStore#get_public_url) method. It accepts only one parameter: `key` - the key of the item you want to share.
 
 <CodeBlock language="python">
     {GetPublicUrlSource}

docs/guides/http_clients.mdx

@@ -36,13 +36,13 @@ In Crawlee we currently have two HTTP clients: <ApiLink to="class/HttpxHttpClien
 Since <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink> is the default HTTP client, you don't need to install additional packages to use it. If you want to use <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>, you need to install `crawlee` with the `curl-impersonate` extra.
 
 ```sh
-pip install 'crawlee[curl-impersonate]'
+python -m pip install 'crawlee[curl-impersonate]'
 ```
 
 or install all available extras:
 
 ```sh
-pip install 'crawlee[all]'
+python -m pip install 'crawlee[all]'
 ```
 
 ## How HTTP clients work

docs/guides/storages.mdx

@@ -33,7 +33,7 @@ Crawlee offers multiple storage types for managing and persisting your crawling
 Storage clients in Crawlee are subclasses of <ApiLink to="class/BaseStorageClient">`BaseStorageClient`</ApiLink>. They handle interactions with different storage backends. For instance:
 
 - <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink>: Stores data in memory and persists it to the local file system.
-- [`ApifyStorageClient`](https://docs.apify.com/sdk/python/reference/class/ApifyStorageClient): Manages storage on the [Apify Platform](https://apify.com). Apify storage client is implemented in the [Apify SDK](https://github.com/apify/apify-sdk-python).
+- [`ApifyStorageClient`](https://docs.apify.com/sdk/python/reference/class/ApifyStorageClient): Manages storage on the [Apify platform](https://apify.com). Apify storage client is implemented in the [Apify SDK](https://github.com/apify/apify-sdk-python).
 
 Each storage client is responsible for maintaining the storages in a specific environment. This abstraction makes it easier to switch between different environments, e.g. between local development and cloud production setup.
 
@@ -52,7 +52,7 @@ where:
 - `{STORAGE_ID}`: The ID of the specific storage instance (default: `default`).
 
 :::info NOTE
-The current <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink> and its interface is quite old and not great. We plan to refactor it, together with the whole <ApiLink to="class/BaseStorageClient">`BaseStorageClient`</ApiLink> interface in the near future and it better and and easier to use. We also plan to introduce new storage clients for different storage backends - e.g. for [SQLLite](https://www.sqlite.org/).
+The current <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink> and its interface is quite old and not great. We plan to refactor it, together with the whole <ApiLink to="class/BaseStorageClient">`BaseStorageClient`</ApiLink> interface in the near future and it better and and easier to use. We also plan to introduce new storage clients for different storage backends - e.g. for [SQLLite](https://sqlite.org/).
 :::
 
 You can override default storage IDs using these environment variables: `CRAWLEE_DEFAULT_DATASET_ID`, `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID`, or `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID`.
@@ -117,7 +117,7 @@ The <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> implements the <Ap
 
 If you need custom functionality, you can create your own request storage by subclassing the <ApiLink to="class/RequestManager">`RequestManager`</ApiLink> class and implementing its required methods.
 
-For a detailed explanation of the <ApiLink to="class/RequestManager">`RequestManager`</ApiLink> and other related components, refer to the [Request loaders guide](https://www.crawlee.dev/python/docs/guides/request-loaders).
+For a detailed explanation of the <ApiLink to="class/RequestManager">`RequestManager`</ApiLink> and other related components, refer to the [Request loaders guide](https://crawlee.dev/python/docs/guides/request-loaders).
 
 ## Dataset
 

docs/introduction/01_setting_up.mdx

@@ -4,28 +4,43 @@ title: Setting up
 ---
 
 import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
 
-To run Crawlee on your computer, ensure you meet the following requirements:
+This guide will help you get started with Crawlee by setting it up on your computer. Follow the steps below to ensure a smooth installation process.
 
-1. [Python](https://www.python.org/) 3.9 or higher installed,
-2. [Pip](https://pip.pypa.io/en/stable/) installed.
+## Prerequisites
 
-You can verify these by running the following commands:
+Before installing Crawlee itself, make sure that your system meets the following requirements:
+
+- **Python 3.9 or higher**: Crawlee requires Python 3.9 or a newer version. You can download Python from the [official website](https://python.org/downloads/).
+- **Python package manager**: While this guide uses [pip](https://pip.pypa.io/) (the most common package manager), you can also use any package manager you want. You can download pip from the [official website](https://pip.pypa.io/en/stable/installation/).
+
+### Verifying prerequisites
+
+To check if Python and pip are installed, run the following commands:
 
 ```sh
 python --version
 ```
 
 ```sh
-pip --version
+python -m pip --version
 ```
 
-## Installation
+If these commands return the respective versions, you're ready to continue.
+
+## Installing Crawlee
+
+Crawlee is available as [`crawlee`](https://pypi.org/project/crawlee/) package on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.
 
-Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package. To install the core package, use:
+### Basic installation
+
+To install the core package, run:
 
 ```sh
-pip install crawlee
+python -m pip install crawlee
 ```
 
 After installation, verify that Crawlee is installed correctly by checking its version:
@@ -34,50 +49,40 @@ After installation, verify that Crawlee is installed correctly by checking its v
 python -c 'import crawlee; print(crawlee.__version__)'
 ```
 
-Crawlee offers several optional features through package extras. You can choose to install only the dependencies you need or install everything if you don't mind the package size.
-
-### Install all features
+### Full installation
 
-If you do not care about the package size, install Crawlee with all features:
+If you do not mind the package size, you can run the following command to install Crawlee with all optional features:
 
 ```sh
-pip install 'crawlee[all]'
+python -m pip install 'crawlee[all]'
 ```
 
-### Installing only specific extras
+### Installing specific extras
 
 Depending on your use case, you may want to install specific extras to enable additional functionality:
 
-#### BeautifulSoup
-
 For using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, install the `beautifulsoup` extra:
 
 ```sh
-pip install 'crawlee[beautifulsoup]'
+python -m pip install 'crawlee[beautifulsoup]'
 ```
 
-#### Parsel
-
 For using the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>, install the `parsel` extra:
 
 ```sh
-pip install 'crawlee[parsel]'
+python -m pip install 'crawlee[parsel]'
 ```
 
-#### Curl impersonate
-
 For using the <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>, install the `curl-impersonate` extra:
 
 ```sh
-pip install 'crawlee[curl-impersonate]'
+python -m pip install 'crawlee[curl-impersonate]'
 ```
 
-#### Playwright
-
 If you plan to use a (headless) browser with <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, install Crawlee with the `playwright` extra:
 
 ```sh
-pip install 'crawlee[playwright]'
+python -m pip install 'crawlee[playwright]'
 ```
 
 After installing the playwright extra, install the necessary Playwright dependencies:
@@ -91,29 +96,58 @@ playwright install
 You can install multiple extras at once by using a comma as a separator:
 
 ```sh
-pip install 'crawlee[beautifulsoup,curl-impersonate]'
+python -m pip install 'crawlee[beautifulsoup,curl-impersonate]'
 ```
 
-## With Crawlee CLI
+## Start a new project
+
+The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. The CLI helps you set up a new project in seconds.
+
+### Using Crawlee CLI with Pipx
 
-The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [Pipx](https://pipx.pypa.io/) installed:
+First, ensure you have Pipx installed. You can check if Pipx is installed by running:
 
 ```sh
-pipx --help
+pipx --version
 ```
 
-Then, run the CLI and choose from the available templates:
+If Pipx is not installed, follow the official [installation guide](https://pipx.pypa.io/stable/installation/).
+
+Then, run the Crawlee CLI using Pipx and choose from the available templates:
 
 ```sh
-pipx run crawlee create my-crawler
+pipx run crawlee create my_crawler
 ```
 
+### Using Crawlee CLI directly
+
 If you already have `crawlee` installed, you can spin it up by running:
 
 ```sh
-crawlee create my-crawler
+crawlee create my_crawler
 ```
 
+Follow the interactive prompts in the CLI to choose a crawler type and set up your new project.
+
+### Running your project
+
+To run your newly created project, navigate to the project directory, activate the virtual environment, and execute the Python interpreter with the project module:
+
+<Tabs>
+  <TabItem value="Linux" label="Linux" default>
+      <CodeBlock language="sh">cd my_crawler/</CodeBlock>
+      <CodeBlock language="sh">source .venv/bin/activate</CodeBlock>
+      <CodeBlock language="sh">python -m my_crawler</CodeBlock>
+  </TabItem>
+<TabItem value="Windows" label="Windows" default>
+      <CodeBlock language="sh">cd my_crawler/</CodeBlock>
+      <CodeBlock language="sh">venv\Scripts\activate</CodeBlock>
+      <CodeBlock language="sh">python -m my_crawler</CodeBlock>
+  </TabItem>
+</Tabs>
+
+Congratulations! You have successfully set up and executed your first Crawlee project.
+
 ## Next steps
 
 Next, you will learn how to create a very simple crawler and Crawlee components while building it.

docs/introduction/08_refactoring.mdx

@@ -67,10 +67,6 @@ Initially, using a simple `if` / `else` statement for selecting different logic
 
 It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `request_handler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
 
-{/* TODO: write this once SDK v2 is ready
-
 ## Next steps
 
-In the next and final step, you'll see how to deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
-
-*/}
+In the next and final step, you'll see how to deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a `Dockerfile` ready, and the next section will show you how to deploy it to the [Apify platform](../deployment/apify-platform) with ease.