Skip to content

Plugins

Jump to bottom
Andy edited this page Oct 18, 2022 · 23 revisions

You can develop your plugin for any site you want. I created a plugin environment with full SCrawler integration. You can display any class properties in SCrawler's internal edit forms.

The class that contains all the available objects is the Instagram settings:

You can look at them as an example.

You can also use the following plugins as an example:

  • XVIDEOS - development time 6.5 hours
  • LPSG - development time 1.5 hours

See how easy it is to create a plugin and make it for your site.

The SCrawler.PluginProvider.dll is fully compatible with the most popular language - CSharp.

Net.Framework version is 4.6.1.

Plugins environment

The plugin concept is based on two main interfaces. The first interface - ISiteSettings - is the major interface of the plugin. This class must provide all instances, validators, and other objects required by SCrawler. The second interface - IPluginContentProvider - is an instance of UserData.

How to make a plugin

    1. Create a new Net.Framework library project.
    2. Add a reference to SCrawler.PluginProvider.dll
    1. Create a new settings class (e.g. SiteSettings).
    2. Implement the ISiteSettings interface.
    3. Add the required Manifest class attribute.
    4. Add optional attributes if you need: SeparatedTasks, SavedPosts, SpecialForm
    • Set values for Site, Icon, Image.
    • Site is a required property (cannot be null)
    • If your plugin doesn't have an Icon or Image, these properties must return Nothing (null in C#).
    • Attention! Don't add any attributes to these properties!
  4. Define your own properties if you need to. To allow users to change the property values of your class and interact with SCrawler, this property must be an AutoProperty and must be declared as PropertyValue object.
  5. Write the code for the interface functions. If a function has a return type, then that function MUST return a value!
    • Develop a user instance of the IPluginContentProvider interface
    • This instance must be returned by the GetInstance function
  7. If you plugin has special options, develop an options exchange class and options editor form.
  8. It is done. Your plugin is ready to work. 😊

How it works

  1. When launched, SCrawler loads plugins from the Plugins folder.
  2. A new SettingsHost is created for each plugin.
    1. Get class attributes
    2. Invoke BeginInit function
    3. Sending XML data to your plugin class using the Load function.
    4. Get class members
    5. Create hosts for PropertyValue
    6. Adding property updaters to the property host
    7. Adding property providers to the property host
    8. Adding property data checkers to the property host
    9. Invoke EndInit function
    10. Get property values stored in XML (and create new ones if they don't exist) and replace default values with XML values.
  3. Creating a list of users using the GetInstance function
  4. When downloading
    1. If Available(Main) = false, users of your plugin will not be downloaded.
    2. Invoke DownloadStarted(Main)
    3. Per downloading user
      1. If ReadyToDownload(Main) = true
      2. Invoke BeforeStartDownload(User, Main)
      3. Downloading data
        1. Sending information to your instance:
          • Settings = your settings instance
          • Thrower
          • LogProvider
          • Name
          • ID
          • ParseUserMediaOnly
          • UserDescription
          • UserExists
          • UserSuspended
          • IsSavedPosts
          • SeparateVideoFolder
          • DataPath
          • PostsNumberLimit
          • ExistingContentList = list of existing content stored in user files
          • TempPostsList = list of post IDs stored in user files
        2. Invoke GetMedia
        3. Get the TempPostsList list back
        4. Get other parameters back
          • Name
          • ID
          • UserDescription
          • UserExists
          • UserSuspended
        5. If the instance does not have UseInternalDownloader attribute, invoke Download function
      4. Invoke AfterDownload(User, Main)
    4. Invoke DownloadDone(Main)
  5. When downloading saved posts
    1. If ReadyToDownload(SavedPosts) = true
    2. If Available(SavedPosts) = true
    3. Invoke DownloadStarted(SavedPosts)
    4. Get an instance with GetInstance(SavedPosts). The Instance must contain the UserName (Name)
    5. Set IsSavedPosts = true
    6. Invoke BeforeStartDownload(User, SavedPosts)
    7. The same steps as in: When downloading - Per downloading user - Downloading data
    8. Invoke AfterDownload(User, SavedPosts)
    9. Invoke DownloadDone(SavedPosts)
  6. When user creating / updating
    1. If your class has special options (SpecialForm(False)), the Options button will be enabled
    2. UserOptions(OptionsArg, False) will be invoked. If OptionsArg is Nothing (null in C#), you MUST create a new exchange object.
    3. By clicking on the Options button, UserOptions(OptionsArg, True) will be invoked. If OptionsArg is Nothing (null in C#), you MUST create a new exchange object. Then open the form to change the settings.
    4. When updating a user, ExchangeOptionsGet will be invoked. You MUST return a new exchange object with user-saved options. At the end of editing the user, ExchangeOptionsSet will be invoked with your exchange object. You MUST apply the options of your exchange object to the user.

Interfaces

ISiteSettings

Don't duplicate the Temporary, Favorite, Path and SavedPostsPath properties in the settings class. These properties are basic host data.

C# explanation

interface ISiteSettings
{
    Icon Icon {get;};
    Image Image {get;};
    string Site {get;};
    string GetUserUrl(string UserName,bool Channel);
    ExchangeOptions IsMyUser(string UserURL);
    ExchangeOptions IsMyImageVideo(string URL);
    IEnumerable GetSpecialData(string URL);
    IPluginContentProvider GetInstance(Download What);
    void Load(IEnumerable<KeyValuePair<string, string>> XMLValues);
    IEnumerable<KeyValuePair<string, string>> XMLFields();
    void BeginInit();
    void EndInit();
    bool Available();
    bool ReadyToDownload();
    void DownloadStarted(Download What);
    void BeforeStartDownload(object User, Download What);
    void AfterDownload(object User, Download What);
    void DownloadDone(Download What);
    void BeginEdit();
    void EndEdit();
    void Update();
    void Reset();
    void OpenSettingsForm();
    void UserOptions(ref object Options, bool OpenForm);
    string GetUserPostUrl(string UserID, string PostID);
}
interface IPluginContentProvider : IDisposable
{
    delegate void ProgressChangedEventHandler(int Count);
    event ProgressChangedEventHandler ProgressChanged;
    delegate void TotalCountChangedEventHandler(int Count);
    event TotalCountChangedEventHandler TotalCountChanged;
    IThrower Thrower {get; set;};
    ILogProvider LogProvider {get; set;};
    ISiteSettings Settings {get; set;};
    string Name {get; set;};
    string ID {get; set;};
    bool ParseUserMediaOnly {get; set;};
    string UserDescription {get; set;};
    List<PluginUserMedia> ExistingContentList {get; set;};
    List<string> TempPostsList {get; set;};
    List<PluginUserMedia> TempMediaList {get; set;};
    bool UserExists {get; set;};
    bool UserSuspended {get; set;};
    bool IsSavedPosts {get; set;};
    bool SeparateVideoFolder {get; set;};
    string DataPath {get; set;};
    Nullable<int> PostsNumberLimit {get; set;};
    Nullable<DateTime> DownloadDateFrom {get; set;};
    Nullable<DateTime> DownloadDateTo {get; set;};
    object ExchangeOptionsGet();
    void ExchangeOptionsSet(object Obj);
    void XmlFieldsSet(List<KeyValuePair<string, string>> Fields);
    List<KeyValuePair<string, string>> XmlFieldsGet();
    void GetMedia();
    void Download();
}

Site

Site name. This ReadOnly property must only provide the name of the site. For example: for instagram.com you should set the site name Instagram. The first letter must be uppercase.

Icon

Site icon for forms and other objects (can be null).

Image

Site image for controls (such as buttons) and other objects.

Logger

This is the SCrawler log provider. It is set automatically by SCrawler on initialization.

GetUserUrl

Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String

This function should return the user's URL based on the UserName argument. This function is called when the user clicks the Open site button in SCrawler.

The Channel argument specifies that the user is the a channel.

IsMyUser

Function IsMyUser(ByVal UserURL As String) As ExchangeOptions

This function should validate the user URL (UserURL argument) and return an ExchangeOptions object if it is your plugin user; otherwise, return a new empty ExchangeOptions

If this is the user of your plugin, you must extract the UserName from the URL and return a new ExchangeOptions instance with the UserName and Site name.

If it is a channel, you must also set the IsChannel field (in ExchangeOptions) to True

For your convenience, this structure provides two default initializers. Use them so you don't forget the fields.

IsMyImageVideo

Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions

Same as IsMyUser, but this function checks the URL argument to see if the URL is your site's media data. It is used for standalone video downloader.

Must return the same as the IsMyUser, but UserName can be any non-empty value.

GetSpecialData

Function GetSpecialData(ByVal URL As String) As IEnumerable

After IsMyImageVideo, if this is your plugin's media file, this function will be called.

This function must return an IEnumerable PluginUserMedia object. This object must contain the data URL and the file name (with extension). Any other fields are optional.

  • URL - media URL
  • Path - last destination path
  • AskForPath - if true, you need to ask the user to select a path

Only if the data is downloaded by your class:

  • If the returned data does not exists and AskForPath = true, you need to return a PluginUserMedia object with SpecialPath = selected path.
  • If the returned data exists, you need to replace PluginUserMedia.SpecialPath with the selected path.

GetInstance

Function GetInstance(ByVal What As Download) As IPluginContentProvider

This is the required main function that returns a UserData instance!

The What argument indicates who called this function.

  • Main - user instance
  • SavedPosts - instance of saved posts
  • Channel - an instance of a channel (main window).

You must return an initialized instance of the IPluginContentProvider object.

GetUserPostUrl

Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String

Return post URL based on user ID and post ID.

Load

Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))

This function sends the XML fields of your site node to your settings class when the host is initialized.

BeginInit

Called at the start of host initialization.

EndInit

Called at the end of host initialization.

BeginEdit

Called when opening the settings form in SCawler

EndEdit

Called when closing the settings form in SCawler

BeginUpdate

Called from the settings root when the settings BeginUpdate function is called.

EndUpdate

Called from the settings root when the settings EndUpdate function is called.

Available

Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean

If your plugin needs to be checked for availability (you can see an example in the API.Reddit.SiteSettings), place the check code here. Otherwise, this function MUST return true. If this function returns false, users of your plugin site will not be downloaded.

Silent stands for show notification or not.

ReadyToDownload

Function ReadyToDownload(ByVal What As Download) As Boolean

It is not the same as Available.

This function is an additional method for checking your class parameters. If your parameters don't need to be validated, just return true.

With this function, you can check the main settings parameters that authorization depends on (for example) or anything else your parser requires.

DownloadStarted

Sub DownloadStarted(ByVal What As Download)

This function will be called when the parser thread is initialized. Not for user parsing!

BeforeStartDownload

Sub BeforeStartDownload(ByVal User As Object, ByVal What As Download)

This function will be called before the user starts downloading. Here you can set some of user options you need.

AfterDownload

Sub AfterDownload(ByVal User As Object, ByVal What As Download)

This function will be called after the user download is complete.

Update

Sub Update()

This function will be called when saving site settings on the site settings form. You don't need to update the PropertyValue properties (these properties are updated automatically).

Reset

Sub Reset()

Not used. Added just in case.

DownloadDone

Sub DownloadDone(ByVal What As Download)

This function will be called when the parser thread is closed.

OpenSettingsForm

If your site settings have a special settings form, this function will be called by clicking on the button in the site settings form.

Your form must be opened with the ShowDialog method!

UserOptions

Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)

If your site settings have special user options, this function will be called by clicking on the Options button in the UserCreatorForm.

Options is an object parameter (reference). If it is null, you must create a new instance of your option exchange class. SCrawler does not know what class you are using to exchange. So SCrawler will exchange this class (as an object) between your settings class and user instance! So this class object cannot be null!

If the OpenForm argument = True, your options form must be opened (using the ShowDialog method). This form must provide controls for change parameters obtained using the Options.

IPluginContentProvider

ProgressChanged

Event ProgressChanged(ByVal Count As Integer)

Event for performing progress. When this event is raised, SCrawler downloading progress will be performed for this number (Count (default = 1)).

TotalCountChanged

Event TotalCountChanged(ByVal Count As Integer)

When you are starts downloading (before start), raise this event with the Count = total amount of media to download.

Thrower

This is an exception thrower.

While debugging, you can use any class you want, but SCrawler will replace that instance with my thrower. To work, you don't need to develop a special object of this interface.

Read more here

LogProvider

This is an interface for sending exceptions and messages to the program log.

While debugging, you can use any class you want, but SCrawler will replace that instance with my log provider. To work, you don't need to develop a special object of this interface.

Read more here

Settings

This object will be replaced by SCrawler with the active settings instance. ATTENTION! Don't instantiate settings!

Other properties

All of these properties will be delivered using SCrawler!

  • Name - user name
  • ID
  • ParseUserMediaOnly
  • UserDescription
  • ExistingContentList - collection of the existing downloaded data
  • TempPostsList - collection of the existing downloaded post id
  • TempMediaList - this is a list of new media to download. You must add data to this list!
  • UserExists - indicates that the user exists on the site
  • UserSuspended - indicates that the user's profile suspended (or blocked) on the site
  • IsSavedPosts - this is the instance for downloading saved posts
  • SeparateVideoFolder - if true, the videos must be saved separately from the images folder
  • DataPath - download path
  • PostsNumberLimit - the maximum number of posts a user has requested to download! This is a nullable argument. So, if it's Nothing (or null in C#), then no limited download is requested!
  • DownloadDateFrom, DownloadDateTo - the lowest and highest date of posts a user has requested to download! This is a nullable argument. So, if it's Nothing (or null in C#), then no limited download is requested!

ExchangeOptionsGet

Function ExchangeOptionsGet() As Object

This function is called by the UserCreatorForm when editing an existing user (not when creating a new one).

This function must return the ExchangeOptions (same as UserOptions function) with the values stored in this user instance.

ExchangeOptionsSet

Sub ExchangeOptionsSet(ByVal Obj As Object)

Set options in this instance with ExchangeOptions. Obj is the same object as in UserOptions function.

XmlFieldsSet

Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))

Delivery of XML fields stored in the user settings file, to the current instance.

  • Key - XML field name
  • Value - XML field value

XmlFieldsGet

Function XmlFieldsGet() As List(Of KeyValuePair(Of String, String))

You must return the XML fields that you want to store in the user settings file.

  • Key - XML field name
  • Value - XML field value

GetMedia

Parse site user data

Download

Download data using parsed information

IThrower

This is an exception thrower. If the user requested to cancel operation or delete user instance, an exception will be thrown. Use the IThrower.ThrowAny method in your code to stop executing when it's requested!

Exceptions:

  • OperationCanceledException - when user requested to cancel operation
  • ObjectDisposedException - when user deletes user instance

ILogProvider

This is an interface for sending exceptions and messages to the program log.

Functions:

  • Add(ByVal Message As String) - add Message to the program log
  • Add
    • ex - your exception
    • Message - additional message
    • ShowMainMsg - show main message
    • ShowErrorMsg - show error message
    • SendInLog - message and error message will be sent to the program log

Objects

PropertyValue

This is the base object for interacting with SCrawler and bi-directional communication.

There are three initialization constructors:

  • By initial value. If the value is null, this constructor will throw an error! Otherwise, the type will be extracted from the value.
  • By value and type.
  • By value, type and function. If you want, you can delegate a function to be called when the value changes. You can see an example in the Instagram SiteSettings class

If your initial value is null, you MUST set the type.

Only these types are available

IPropertyValue

Don't use this interface. This interface is only compatible with SCrawler. Always use the PropertyValue class!

PropertyData

This is a structure for exchanging properties between classes without saving. It is currently used for PropertiesDataChecker attribute.

  • Name - property name
  • Value - property value

ExchangeOptions

For your convenience, this structure provides two default initializers. Use them so you don't forget the fields.

Currently used in IsMyUser and IsMyImageVideo

  • UserName - user name or any other data (if specified in the method description)
  • SiteName - site name
  • HostKey - aet automatically via SettingsHost
  • IsChannel - this user is a channel

Attributes

PropertyOption

This attribute allows you to add your property to the Scrawler site settings form. Only works with PropertyValue object.

Options:

  • PropertyName - automatically set when attribute is initialized. Corresponds to the property name.
  • Type - property value type. Can be obtained automatically from PropertyValue.
  • ControlText - this text will be displayed on the control in the settings form.
  • ControlToolTip - Control ToolTip.
  • ThreeStates - CheckBox option. If true, then the CheckBox will be displayed in three states: Checked, Unchecked, Indeterminate. Default: false.
  • AllowNull - If false, then the settings form will display an error message when trying to save the property value if the property value is null. Default: true.
  • LeftOffset - Just a design option. This is just a control offset from the left border of the form (just for beauty).
  • IsAuth - Default: false. If at least one property has a PropertyOption with this parameter, then the controls in the settings form will be divided into two blocks: Authorization and Other. Just a design option.
  • IsInformationLabel - Just a design option. Specifies that this property is just information.
  • LabelTextAlign - The alignment of the control (label) text, if IsInformationLabel.

PXML

This attribute specifies that your property should be added to the SCrawler settings XML file. When launched, SCrawler will replace the value of your property with the value from the XML settings file.

Again. Only works with PropertyValue object.

PropertyUpdater

This attribute provides a mechanism for updating a specific property. Attributed can only be applied to a method. Allows multiple definitions. Function MUST return a Boolean (bool in C#) with true indicating that the value has been updated and false otherwise.

  • UpdatingPropertyName - the name of the property to be updated
  • Dependent - array of the property names on which update depends

Manifest

This attribute provides the plugin key to associate users with the plugin. I recommend using the pattern: DeveloperName_Site.

SpecialForm

I have developed a fully integrated plugin environment. This means that any of your class property can be displayed in the SCrawler internal forms (for example, SettingsForm, UserCreatorForm). But if your plugin requires authorization or you want to manipulate your settings in a separate form, you can use the SpecialForm class attribute. This attribute allows multiple definitions for the same class and two working modes.

SpecialForm(True), for class settings. This means that an additional button will be added to the site settings form to opening your settings form. In this case, it is necessary to develop an additional form that must be opened (using the ShowDialog method) by the OpenSettingsForm class function.

SpecialForm(False), for class settings. This means that the Options button will be enabled on the user creator form. In this case, you must develop an additional exchange class and form. This works with the UserOptions class function. This function has two arguments: Options (reference) and OpenForm.

Provider

This attribute indicates that the property is a provider. The provider must be an IFormatProvider.

There are two working ways: check fields (FieldsChecker) and not.

FieldsChecker = true. Convert text from TextBox in settings form to value

FieldsChecker = false. Converting a value by a specific converter. Also used to convert a value from XML to an object.

ControlNumber

Just a design attribute. Allows you to place controls on the settings form in a specific order. The number means the zero-based control position at the top of the form.

PropertiesDataChecker

An attribute for the validation method. Names is an array of property names to check values. The method must be a function with an IEnumerable argument of PropertyData and boolean return.

Returning true means the values are OK.

If you need to check the values of some properties before saving the settings, use this attribute.

SeparatedTasks

If your plugin needs to run on a separate thread add the SeparatedTasks class attribute. If you want to limit download tasks per your plugin, you can set number of tasks for this attribute. For example: SeparatedTasks(1) means that only one user will be downloaded per task; SeparatedTasks(2) means that two users will be downloaded at the same time for each task. The number argument is optional. This means that if you don't specify a tasks number, the global tasks number will be used instead. In additional, the TaskCounter property attribute has the highest priority. This means that if you make a property (to allow users to change the number of tasks) with a TaskCounter attribute, the number of SeparatedTasks will be ignored. But the SeparatedTasks attribute MUST be added to the class anyway if you want to run your plugin on the separate thread.

TaskCounter

If you want users to be able to control the number of download tasks in a thread, use this attribute. Only works with SeparatedTasks

SavedPosts

If your plugin provides downloading of saved posts, add the SavedPosts class attribute.

UseInternalDownloader

User data instance attribute (for IPluginContentProvider). Means that after parsing, SCrawler should use the internal downloader.

Github

Assembly attribute. Not used yet, but in future versions will allow SCrawler to check for updates to your plugin.

  • Name - your GitHub username
  • RepoName - your repository name

PropertyValue value types

vb.net C# NET
Boolean bool System.Boolean
Byte byte System.Byte
SByte sbyte System.SByte
Short short System.Int16
UShort ushort System.UInt16
Integer int System.Int32
UInteger uint System.UInt32
Long long System.Int64
ULong ulong System.UInt64
Double double System.Double
Single float System.Single
Decimal decimal System.Decimal
String string System.String
Date DateTime System.DateTime
TimeSpan TimeSpan System.TimeSpan

How to support the program

Clone this wiki locally