new doc structure (#361)

* reformat docs

* fix dead/wrong links in doc

* merge

Author: LordSimal

Diff for: docs/README.md

@@ -0,0 +1,26 @@
+# Basic Setup
+
+## Installation
+```
+composer require dereuromark/cakephp-queue
+```
+Load the plugin in your `src/Application.php`'s bootstrap() using:
+```php
+$this->addPlugin('Queue');
+```
+If you don't want to also access the backend controller (just using CLI), you need to use
+```php
+$this->addPlugin('Queue', ['routes' => false]);
+```
+
+Important: Make sure to use authentication if you are using the backend. You do not want visitors to be able to browse it.
+
+## Database migration
+
+Run the following command in the CakePHP console to create the tables using the Migrations plugin:
+```sh
+bin/cake migrations migrate -p Queue
+```
+
+Hint: use a native *nix-like or console and not the one provided like from Git (git-bash). This may lead to a non-working `migrations` command.
+It is also advised to have the `posix` PHP extension enabled.

Diff for: docs/bar_html.png renamed to docs/sections/bar_html.png

@@ -0,0 +1,165 @@
+# Configuration
+
+## Global configuration
+The plugin allows some simple runtime configuration.
+You may create a file called `app_queue.php` inside your `config` folder (NOT the plugins config folder) to set the following values:
+
+- Seconds to sleep() when no executable job is found:
+
+    ```php
+    $config['Queue']['sleeptime'] = 10;
+    ```
+
+- Probability in percent of an old job cleanup happening:
+
+    ```php
+    $config['Queue']['gcprob'] = 10;
+    ```
+
+- Default timeout after which a job is requeued if the worker doesn't report back:
+
+    ```php
+    $config['Queue']['defaultworkertimeout'] = 1800;
+    ```
+
+- Default number of retries if a job fails or times out:
+
+    ```php
+    $config['Queue']['defaultworkerretries'] = 3;
+    ```
+
+- Seconds of running time after which the worker will terminate (0 = unlimited):
+
+    ```php
+    $config['Queue']['workermaxruntime'] = 120;
+    ```
+
+  *Warning:* Do not use 0 if you are using a cronjob to permanantly start a new worker once in a while and if you do not exit on idle.
+
+- Seconds of running time after which the PHP process of the worker will terminate (0 = unlimited):
+
+    ```php
+    $config['Queue']['workertimeout'] = 120 * 100;
+    ```
+
+  *Warning:* Do not use 0 if you are using a cronjob to permanently start a new worker once in a while and if you do not exit on idle. This is the last defense of the tool to prevent flooding too many processes. So make sure this is long enough to never cut off jobs, but also not too long, so the process count stays in manageable range.
+
+- Should a worker process quit when there are no more tasks for it to execute (true = exit, false = keep running):
+
+    ```php
+    $config['Queue']['exitwhennothingtodo'] = false;
+    ```
+
+- Minimum number of seconds before a cleanup run will remove a completed task (set to 0 to disable):
+
+    ```php
+    $config['Queue']['cleanuptimeout'] = 2592000; // 30 days
+    ```
+
+- Max workers (per server):
+
+    ```php
+    $config['Queue']['maxworkers'] = 3 // Defaults to 1 (single worker can be run per server)
+    ```
+
+- Multi-server setup:
+
+    ```php
+    $config['Queue']['multiserver'] = true // Defaults to false (single server)
+    ```
+
+  For multiple servers running either CLI/web separately, or even multiple CLI workers on top, make sure to enable this.
+
+- Use a different connection:
+
+    ```php
+    $config['Queue']['connection'] = 'custom'; // Defaults to 'default'
+    ```
+
+- Ignore certain task classes to they don't end up in the generated SQL query. Can be used to filter out the example tasks classes shipped with the plugin, if you're not using them:
+
+    ```php
+    $config['Queue']['ignoredTasks'] = [
+        'Queue\Queue\Task\CostsExampleTask',
+        'Queue\Queue\Task\EmailTask',
+        'Queue\Queue\Task\ExampleTask',
+        'Queue\Queue\Task\ExceptionExampleTask',
+        'Queue\Queue\Task\ExecuteTask',
+        'Queue\Queue\Task\MonitorExampleTask',
+        'Queue\Queue\Task\ProgressExampleTask',
+        'Queue\Queue\Task\RetryExampleTask',
+        'Queue\Queue\Task\SuperExampleTask',
+        'Queue\Queue\Task\UniqueExampleTask',
+    ]; // Defaults to []
+    ```
+
+Don't forget to load that config file with `Configure::load('app_queue');` in your bootstrap.
+You can also use `$this->addPlugin('Queue', ['bootstrap' => true]);` which will load your `app_queue.php` config file automatically.
+
+Example `app_queue.php`:
+
+```php
+return [
+    'Queue' => [
+        'workermaxruntime' => 60,
+        'sleeptime' => 15,
+    ],
+];
+```
+
+You can also drop the configuration into an existing config file (recommended) that is already been loaded.
+The values above are the default settings which apply, when no configuration is found.
+
+### Serializer strategy
+By default, the payload data array will be serialized using PHPs native object serializer.
+It is recommended to switch to JSON serializing instead, if you don't need to send any actual objects, but only primitive values and arrays.
+```php
+'Queue' => [
+    ...
+    'serializerClass' => \Queue\Utility\JsonSerializer::class,
+    'serializerConfig' => [...],
+],
+```
+This is usually also safer from data perspective.
+Any update of your code would not break the process here for queued tasks as no objects could be outdated in serialized payload.
+
+You can also use a custom one as long as it implements the `Queue\Utility\SerializerInterface`.
+
+Note: When using JSON one, make sure to use Email and Mailer tasks only with JSON safe way (not passing actual objects).
+
+### Backend configuration
+
+- isSearchEnabled: Set to false if you do not want search/filtering capability.
+  This is auto-detected based on [Search](https://github.com/FriendsOfCake/search) plugin being available/loaded if not disabled.
+
+- isStatsEnabled: Set to true to enable. This requires [chart.js](https://github.com/chartjs/Chart.js) asset to be available.
+  You can also overwrite the template and as such change the asset library as well as the output/chart.
+
+
+### Configuration tips
+
+For the beginning maybe use not too many runners in parallel, and keep the runtimes rather short while starting new jobs every few minutes.
+You can then always increase spawning of runners if there is a shortage.
+
+## Task configuration
+
+You can set two main things on each task as property: timeout and retries.
+```php
+    /**
+     * Timeout for this task in seconds, after which the task is reassigned to a new worker.
+     *
+     * @var int
+     */
+    public $timeout = 120;
+
+    /**
+     * Number of times a failed instance of this task should be restarted before giving up.
+     *
+     * @var int
+     */
+    public $retries = 1;
+```
+Make sure you set the timeout high enough so that it could never run longer than this, otherwise you risk it being re-run while still being run.
+It is recommended setting it to at least 2x the maximum possible execution length. See [Concurrent workers](limitations.md)
+
+Set the retries to at least 1, otherwise it will never execute again after failure in the first run.

Diff for: docs/bar_text.png renamed to docs/sections/bar_text.png

@@ -0,0 +1,20 @@
+# Setting up the trigger cronjob
+
+As outlined in the [book](http://book.cakephp.org/3.0/en/console-and-shells/cron-jobs.html) you can easily set up a cronjob
+to start a new worker.
+
+The following example uses "crontab":
+
+    */10  *  *  *  *  cd /full/path/to/app && bin/cake queue run -q
+
+Make sure you use `crontab -e -u www-data` to set it up as `www-data` user, and not as root etc.
+
+This would start a new worker every 10 minutes. If you configure your max life time of a worker to 15 minutes, you
+got a small overlap where two workers would run simultaneously. If you lower the 10 minutes and raise the lifetime, you
+get quite a few overlapping workers and thus more "parallel" processing power.
+Play around with it, but just don't shoot over the top.
+
+Also don't forget to set Configure key `'Queue.maxworkers'` to a reasonable value per server.
+If, for any reason, some of the jobs should take way longer, you want to avoid additional x workers to be started.
+It will then just not start now ones beyond this count until the already running ones are finished.
+This is an important server protection to avoid overloading.

Diff for: docs/sections/basic_setup.md

@@ -0,0 +1,71 @@
+# Writing your own task
+
+## Baking new Queue task and test
+You can bake a new task and its test via
+```
+bin/cake bake queue_task MyTaskName [-p PluginName]
+```
+
+It will generate a `MyTaskNameTask` class in the right namespace.
+
+It will not overwrite existing classes unless you explicitly force this (after prompting).
+
+## Detailed explanation
+
+In most cases you wouldn't want to use the existing task, but just quickly build your own.
+Put it into `src/Queue/Task/` as `{YourNameForIt}Task.php`.
+
+You need to at least implement the `run()` method:
+```php
+namespace App\Queue\Task;
+
+use Queue\Queue\Task;
+
+class YourNameForItTask extends Task {
+
+    /**
+     * @var int
+     */
+    public $timeout = 20;
+
+    /**
+     * @var int
+     */
+    public $retries = 1;
+
+    /**
+     * @param array $data The array passed to QueuedJobsTable::createJob()
+     * @param int $jobId The id of the QueuedJob entity
+     * @return void
+     */
+    public function run(array $data, int $jobId): void {
+        $this->loadModel('FooBars');
+        if (!$this->FooBars->doSth()) {
+            throw new RuntimeException('Couldnt do sth.');
+        }
+    }
+
+}
+```
+Make sure it throws an exception with a clear error message in case of failure.
+
+Note: You can use the provided `Queue\Model\QueueException` if you do not need to include a strack trace.
+This is usually the default inside custom tasks.
+
+## DI Container Example
+
+If you use the [Dependency Injection Container](https://book.cakephp.org/4/en/development/dependency-injection.html) provided by CakePHP you can also use it inside your tasks.
+
+```php
+use Queue\Queue\ServicesTrait;
+
+class MyCustomTask extends Task {
+    use ServicesTrait;
+
+    public function run(array $data, int $jobId): void {
+        $myService = $this->getService(MyService::class);
+    }
+}
+```
+
+As you see here you have to add the [ServicesTrait](https://github.com/dereuromark/cakephp-queue/blob/master/src/Queue/ServicesTrait.php) to your task which then allows you to use the `$this->getService()` method.

Diff for: docs/sections/configuration.md

@@ -0,0 +1,9 @@
+# Known Limitations
+
+## Concurrent workers may execute the same job multiple times
+
+If you want to use multiple workers, please double-check that all jobs have a high enough timeout (>> 2x max possible execution time of a job). Currently it would otherwise risk the jobs being run multiple times!
+
+## Concurrent workers may execute the same job type multiple times
+
+If you need limiting of how many times a specific job type can be run in parallel, you need to find a custom solution here.

Diff for: docs/sections/cron.md

@@ -0,0 +1,64 @@
+# Misc
+
+## Logging
+
+By default errors are always logged, and with log enabled also the execution of a job.
+Make sure you add this to your config:
+```php
+'Log' => [
+    ...
+    'queue' => [
+        'className' => ...,
+        'type' => 'queue',
+        'levels' => ['info'],
+        'scopes' => ['queue'],
+    ],
+],
+```
+
+When debugging (not using `--quiet`/`-q`) on "run", it will also log the worker run and end.
+
+You can disable info logging by setting `Queue.log` to `false` in your config.
+
+## Resetting
+You can reset all failed jobs from CLI and web backend.
+With web backend you can reset specific ones, as well.
+
+From CLI you run this to reset all at once:
+```
+bin/cake queue reset
+```
+
+## Rerunning
+You can rerun successfully run jobs if they are not yet cleaned out. Make sure your cleanup timeout is high enough here.
+Usually weeks or months is a good balance to have those still stored for this case.
+
+This is especially useful for local development or debugging, though. As you would otherwise have to manually trigger or import the job all the time.
+
+From CLI you run this to rerun all of a specific job type at once:
+```
+bin/cake queue rerun FooBar
+```
+You can add an additional reference to rerun a specific job.
+
+## Using custom finder
+You can use a convenience finder for tasks that are still queued, that means not yet finished.
+```php
+$query = $this->QueuedJobs->find('queued')->...;
+```
+This includes also failed ones if not filtered further using `where()` conditions.
+
+## Notes
+
+`<TaskName>` is the complete class name without the Task suffix (eg. Example or PluginName.Example).
+
+Custom tasks should be placed in `src/Queue/Task/`.
+Tasks should be named `SomethingTask` and implement the Queue "Task".
+
+Plugin tasks go in `plugins/PluginName/src/Queue/Task/`.
+
+A detailed Example task can be found in `src/Queue/Task/QueueExampleTask.php` inside this plugin.
+
+Some more tips:
+- If you copy an example, do not forget to adapt the namespace ("App")!
+- For plugin tasks, make sure to load the plugin using the plugin prefix ("MyPlugin.MyName").