' + - '' + - _("Hide Search Matches") + - "
" - ) - ); - }, - - /** - * helper function to hide the search marks again - */ - hideSearchWords: () => { - document - .querySelectorAll("#searchbox .highlight-link") - .forEach((el) => el.remove()); - document - .querySelectorAll("span.highlighted") - .forEach((el) => el.classList.remove("highlighted")); - localStorage.removeItem("sphinx_highlight_terms") - }, - - initEscapeListener: () => { - // only install a listener if it is really needed - if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; - - document.addEventListener("keydown", (event) => { - // bail for input elements - if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; - // bail with special keys - if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; - if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { - SphinxHighlight.hideSearchWords(); - event.preventDefault(); - } - }); - }, -}; - -_ready(SphinxHighlight.highlightSearchWords); -_ready(SphinxHighlight.initEscapeListener); diff --git a/docs/build/html/contact.html b/docs/build/html/contact.html deleted file mode 100644 index 0435713..0000000 --- a/docs/build/html/contact.html +++ /dev/null @@ -1,114 +0,0 @@ - - - - - - - - -This project is developed and maintained by the ODISSEI Social Data Science (SoDa) team and the FIRMBACKBONE Project.
-
-Do you have questions, suggestions, or remarks? File an issue in the issue tracker or feel free to contact the team of ODISSEI Social Data Science.
-This project is developed and maintained by the ODISSEI Social Data Science (SoDa) team and the FIRMBACKBONE Project.
-
-Any contributions you make to WebSweep are greatly appreciated.
-Whether you’re fixing a bug, adding a new feature, or improving the documentation, your contribution is welcome. For more information on how to report issues, submit pull requests, and contribute to WebSweep in general, please see our CONTRIBUTING file.
-The contribution file outlines the process we’d like contributors to follow to help maintain the quality and consistency of the project.
-For instructions on how to run WebSweep without installing it and how to install it from source, please refer to our DEVELOPMENT guide. This guide provides all the necessary steps to get you started with developing and testing WebSweep on your local environment.
-If you have any questions or need support, feel free to reach out. You can file an issue in our issue tracker or contact the team directly through the ODISSEI Social Data Science (SoDa) contact form.
-This page provides examples of how to use websweep both as a command line interface (CLI) tool and as a Python package.
-Initialize a new WebSweep instance:
-$ websweep init
- --headless
-Restore an existing WebSweep instance:
-$ websweep restore
- --headless
-Configure WebSweep settings:
-$ websweep config
- --delete-processed-files
- --target-folder-path /path/to/new/output
- --source-file-path /path/to/new/source/file.csv
-View active WebSweep instance:
-$ websweep instance
-Start crawling process:
-$ websweep crawl
- --sock-connect 150
-Start extracting process:
-$ websweep extract
- --start-date YYYY-MM-DD
- --end-date YYYY-MM-DD
-Scraping and extracting at the same time (no HTML saved):
-from websweep import Crawler
-
-urls = ['https://firmbackbone.nl', 'https://uu.nl/', 'https://odissei-soda.nl']
-
-crawler_unit = Crawler(target_folder_path=Path("your/data/folder"), save_html=False, extract=True)
-crawler_unit.crawl_base_urls(urls)
-Only scraping (saving HTML):
-from websweep import Crawler
-
-urls = ['https://firmbackbone.nl', 'https://uu.nl/', 'https://odissei-soda.nl']
-
-crawler_unit = Crawler(target_folder_path=Path("your/data/folder"))
-crawler_unit.crawl_base_urls(urls)
-Only extracting (using HTML):
-from websweep import FileExtractor
-
-extractor = FileExtractor(target_folder_path=Path("your/data/folder"))
-extractor.extract_urls()
-Only extracting with custom Extractor methods (using HTML):
-from websweep import FileExtractor
-from pathlib import Path
-import re
-
-class MyCustomFileExtractor(FileExtractor):
- def __init__(self, *args, **kwargs):
- super().__init__(*args, **kwargs)
-
- def _extract_something(self) -> list:
- """
- Extract something from the input file, and add found somethings to self.something in set form
- """
- pattern = re.compile(r'\b\d{5,}\b', re.VERBOSE)
- somethings = list(set(re.findall(pattern, self.text)))
- return somethings
-
-extractor_unit = MyCustomFileExtractor(target_folder_path=Path("your/data/folder"))
-extractor_unit.extract_urls()
-| - | - |
| - | - |
| - | - |
| - | - |
|
-
| - |
| - |
| - |
| - | - |
WebSweep is a powerful Python library crafted for web scraping projects that prioritize simplicity, modularity, and impressive speed.
-WebSweep is a joint initiative of FIRMBACKBONE and ODISSEI Social Data Science. WebSweep offers a suite of tools designed to make the extraction of information from the web as straightforward and efficient as possible. With WebSweep, developers and researchers can harness the power of web scraping to transform unstructured web data into valuable insights.
-Tracking Corporate Climate Responsibility: WebSweep enables analysts to efficiently evaluate corporate websites, measuring the frequency and sentiment of mentions related to green energies. This provides a quantifiable metric to gauge a company’s commitment to climate responsibility.
Analyzing Academic Collaboration Networks: By extracting data from various academic and research-oriented websites, WebSweep assists in uncovering patterns of academic collaboration. It serves as a tool for mapping research networks and identifying emerging trends in interdisciplinary studies.
Tracking Public Health Information: Utilizing WebSweep’s scraping capabilities on government and health organization websites, as well as medical journals, can significantly contribute to public health monitoring. It allows for real-time tracking of disease spread, assessment of public health initiatives, and analysis of the effects of healthcare policies on community wellness.
FIRMBACKBONE is an organically growing longitudinal data-infrastructure with information on Dutch companies for scientific research and education. Once it is ready, it will become available for researchers and students affiliated with Dutch member universities through the Open Data Infrastructure for Social Science and Economic Innovations (ODISSEI).
-FIRMBACKBONE is an initiative of Utrecht University (UU) and the Vrije Universiteit Amsterdam (VU Amsterdam) funded by the Platform Digital Infrastructure-Social Sciences and Humanities (PDI-SSH) for the period 2020-2025.
-High-Volume Crawling: WebSweep is designed to handle large-scale web scraping tasks, capable of downloading the full HTML content from a vast number of websites, scaling up to 10 million sites.
Research-Driven: The primary intended use of WebSweep is research, facilitating the collection and analysis of web data across various domains and disciplines.
User-Friendly for Beginners: With a focus on simplicity, WebSweep is accessible to beginners, offering an easy-to-use interface for web scraping tasks.
HTML Processing Flexibility: Users have the option to process HTML content either during the crawl or post-crawl, providing flexibility in how data is handled and analyzed.
Asynchronous Operation: WebSweep operates asynchronously, allowing for efficient utilization of resources and faster execution of scraping tasks.
Impressive Speed: On a consumer-grade laptop with a home internet connection, WebSweep can process between 50,000 and 100,000 pages per hour, demonstrating remarkable efficiency.
JavaScript Limitation: Currently, WebSweep does not process JavaScript, ensuring a focus on the raw HTML content for scraping tasks.
Domain-Level Consolidation: Results can be consolidated at the domain level, enabling a more organized and structured approach to data aggregation and analysis.
The package websweep is published under an MIT license. When using websweep for academic work, please cite:
-
XXXXX
-This section provides detailed instructions on how to install WebSweep.
-Before installing WebSweep, ensure you have the following prerequisites:
-Python 3.8
pip (Python package manager)
To install WebSweep, run the following command in your terminal:
-pip install websweep
-For advanced users who want to install from the source or customize their installation:
-Clone the repository:
-git clone https://github.com/sodascience/websweep.git
-Navigate to the WebSweep directory:
-cd websweep
-Install the package:
-pip install .
-After installing WebSweep, you can verify the installation by running:
-websweep --version
-| - w | ||
| - |
- websweep | - |
| - |
- websweep.config | - |
| - |
- websweep.crawler | - |
| - |
- websweep.crawler.crawler | - |
| - |
- websweep.extractor | - |
| - |
- websweep.extractor.add_host | - |
| - |
- websweep.extractor.extractor | - |
| - |
- websweep.extractor.websweep_extractor | - |
| - |
- websweep.main | - |
| - |
- websweep.utils | - |
| - |
- websweep.utils.utils | - |
- Searching for multiple words only shows matches that contain - all words. -
- - - - - - -The websweep tool can be used as a Python library or through its command line interface (CLI). Below are the basic usage instructions for both methods.
-To use websweep as a library, you can import it into your Python script and call its functions directly. Here is an example of how to use websweep in your code:
-
# Example Python code that uses websweep as a library
-from websweep import Crawler
-
-# Initialize the Scraper
-crawler = Crawler(target_folder_path = Path("your/data/folder"))
-# Use the Crawler to download HTML content
-crawler.crawl_base_urls(['https://firmbackbone.nl', 'https://uu.nl/'])
-
-# Process the content...
-# (your code here)
-websweep also provides a command line interface for easy access to its features directly from your terminal.
--v, --version Show the application's version and exit.
---install-completion Install completion for the current shell.
---show-completion Show completion for the current shell, to copy it or
- customize the installation.
---help Show this message and exit.
-$ websweep config Alter WebSweep configuration settings.
-$ websweep crawl Start caching websites.
-$ websweep extract Start extracting data from the fetched files.
-$ websweep init Initialise a new WebSweep instance.
-$ websweep instance Open configured WebSweep instance folder.
-$ websweep restore Restore configuration of existing WebSweep instance.
-Before using WebSweep, you need to initialize a new instance or you need to restore an instance. Note that only one instance can be active at any given time. However, you can switch between instances by restoring such sessions. -To initialise a new instance, you can use the init command:
-$ websweep init
-Run without GUI elements
-Run with GUI elements [default: no-headless]
-The initialisation process allows for the following configurations:
-WebSweep instance folder location
Crawl source file (csv)
Remove raw files after data extraction
Usage of SQL or CSV database
As mentioned, you may also configure WebSweep with an existing instance folder instead of creating a new one. When you restore a session, you should not initialise a new session with init. You can restore a session with the restore command:
-$ websweep restore
-Run without GUI elements
-Run with GUI elements [default: no-headless]
-You may alter some configurations of the restored or created WebSweep instance. You can do this with the config command:
-$ websweep config
-Delete extractor processed raw files
-Not-Delete extractor processed raw files
-Set new path for crawled data output
-Set new path for csv source file
-Other configurations such as the usage of database type and instance location cannot be altered.
-You can view the active WebSweep session. You can do this with the instance command:
-$ websweep instance
-Only when WebSweep is configured with a new or restored instance, the crawling functionality can be used. -The crawl command will start the crawling process of the urls provided in the source file and outputs the crawled data to the WebSweep instance folder under crawled_data.
-Complement the folder with failed pages, takes -the crawl date as argument
-Timeout value (ms) for establishing a connection -to remote server [default: 120]
-Extract files directly after crawl instead of saving HTML
-Save HTML [default: no-extract]
-Use a custom classification file with page title -terms (plain .txt with ‘;’ delimitation)
-Only when WebSweep is configured with a new or restored instance and the instance folder is populated with crawled data, the extracting functionality can be used. -The extract command will start the extracting process of the crawled html files and outputs the extracted data to the WebSweep instance folder under `extracted_data.
-Date on which the files are retreieved and extracted -should start extracting
-Date on which the files are retreieved and extractor -should stop extracting
-This module provides the Crawler model-controller.
-Bases: object
Parse code and return content and urls. Defined it here to be able to pickle it and process it in a thread pool.
-Create initial asynchronous task to fetch all urls
-urls – List of all level 0 urls to visit
-Bases: object
A class for extracting data from one specific file. -This class is used by the extractor pipeline. -Custom FileExtractor subclasses can be build, extending the data extracting functionalities.
-A tuple containing metadata about the file to extract data from, including the domain, level, website, date and path.
-Initiates the extracting of data from a file at the specified file path. -Calls extracting_default_metadata() and extract_extended_metadata().
-Defines methods that include the default extracting functionalities.
-Defines methods that include the extendable extracting functionalities in subclasses.
-Bases: object
A class for extracting data from files and storing it in the target folder.
-The path to the folder where the extracted data is stored.
-Whether or not to use SQLite to store the extracted data. Default is False.
-Whether or not to delete the original files after extracting data. Default is False.
-An custom instance of a FileExtractor class used to extract data from files. Default is None, in which case it will use the default FileExtractor class.
-Extracts the data from one specific file
-Start the extracting of all data from the files
-Bases: FileExtractor
This module provides the WebSweep config functionality.
-Return the current websweep location
-Initialize the application.
-Restore existing application.
-Return the current WebSweep instance location path
-Return the current source file path
-Return whether to delete processed raw files
-Return whether to use an SQL database raw files
-Restore configuration of existing WebSweep instance. -The exisiting location is stored in the application config file and the exisiting settings in the settings file are validated.
-Alter WebSweep configuration settings
-Open configured WebSweep instance folder
-Start caching websites
-Start extracting data from the fetched files
-Classify url based on level
-