Cache API requests

[nmoinvaz]

It would be great if there was an option to cache API requests. Sometimes on large import it throws an error after a certain amount of time. Also API requests for most of these service desks are rate limited - for example Zendesk only allows 10 requests per minute! If the requests were cached on the file system it would make the next try quicker to pick up where it left off. Right now I am using something like this in my ApiController.

protected function getCached($url)
{
    $root = $_SERVER['DOCUMENT_ROOT'].'/importer-cache/tickets/';
    $filename = sha1($url);
    if (!property_exists($this, 'getCachedFiles')) {
        if (!file_exists($root)) {
            mkdir($root, 0777, true);
        }
        $this->getCachedFiles = scandir($root);
    } 
    if (in_array($filename, $this->getCachedFiles)) {
        $packet = file_get_contents($root.$filename);
    } else {
        $packet = $this->get($url);
        file_put_contents($root.$filename, $packet);
    }
    return $packet;
}

If there was a way to clear the cache that would be good too. And a way to reset the state of Awesome Support (delete all posts and all associated postmeta) just for the purposes of resetting an import.

[nmoinvaz]

I think caching all the content on the server before doing the actual import is the only thing that helped me finally import all 8 years worth of Zendesk data. Even with max_execution_time set to 0, the PHP requests would fail after a certain amount of time and it would half import the data and on the next try it would start from scratch.

In AttachmentMapper.php I used this replacement function:

protected function isReadable($url)
{
    // Check if the attachment is callable.
    $root = $_SERVER['DOCUMENT_ROOT'].'/importer-cache/attachments-readable/';
    $filename = sha1($url);
    if (!property_exists($this, 'isReadableCached')) {
        if (!file_exists($root)) {
            mkdir($root, 0777, true);
        }
        $this->isReadableCached = scandir($root);
    } 

    if (in_array($filename.'.good', $this->isReadableCached)) {
        return true;
    } else if (in_array($filename.'.bad', $this->isReadableCached)) {
        return false;
    }
    
    $handle = @fopen($url, 'rb', false);
    if (!$handle) {
        file_put_contents($root.$filename.'.bad', '');
        return false;
    }
    file_put_contents($root.$filename.'.good', '');
    fclose($handle);
    return true;
}

And in Inserter.php I used:

protected function getAttachmentCached($url)
{
    $root = $_SERVER['DOCUMENT_ROOT'].'/importer-cache/attachments-data/';
    $filename = sha1($url);
    if (!property_exists($this, 'getAttachmentCachedFiles')) {
        if (!file_exists($root)) {
            mkdir($root, 0777, true);
        }
        $this->getAttachmentCachedFiles = scandir($root);
    } 

    if (in_array($filename, $this->getAttachmentCachedFiles)) {
        return file_get_contents($root.$filename);
    }
    
    $data = file_get_contents($url);
    file_put_contents($root.$filename, $data);
    return $data;
}
public function insertAttachment($postId, array $attachment)
{
    if (!$attachment || !isset($attachment['url'])) {
        return;
    }
    
    $data = $this->getAttachmentCached($attachment['url']);
    $this->fileUpload->process_attachments($postId, [
        [
            'filename' => $attachment['filename'],
            'data'     => $data
        ],
    ]);
}

I think it would be great if there if the import was broken into two steps:

1. Copy all help desk data
2. Import all help desk data

If the copying took too long it could pick up where it left off easily. Import would also pick up where it left off by referencing help desk ids as it does now. The importer as it currently is, does not pick up where it left off as far as data retrieval from the help desk.