Skip to content

Commit b6fa9fe

Browse files
allenheltondevallenheltondev
committed
Initial commit
0 parents  commit b6fa9fe

22 files changed

+11488
-0
lines changed

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules/
2+
.aws-sam/
3+
samconfig.toml

‎README.md

+190
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Blog Crossposting Automation
2+
3+
Are you a blog writer? Hate cross-posting your content across the web? You're in luck!
4+
5+
This solution will hook into your blog creation process and automatically cross-post your content for you to Medium, Dev.to, and Hashnode!
6+
7+
Deploy into your AWS account and type away!
8+
9+
For a full summary of this solution [please refer to this blog post](https://www.readysetcloud.io/blog/allen.helton/how-i-built-a-serverless-automation-to-cross-post-my-blogs/) by [Allen Helton](https://twitter.com/allenheltondev).
10+
11+
## Prerequisites
12+
13+
For cross-posts to work successfully, there are a few prereqs that must be met in your setup.
14+
15+
* Your blog post must be written in [markdown](https://en.wikipedia.org/wiki/Markdown).
16+
* Content is checked into a repository in GitHub
17+
* You have an application in [AWS Amplify](https://aws.amazon.com/amplify/) that has a runnable CI pipeline
18+
* Blog posts have front matter in the format outlined in the [Blog Metadata](#blog-metadata) section
19+
20+
## How It Works
21+
22+
![](/docs/workflow.png)
23+
24+
The cross posting process is outlined below.
25+
26+
1. Completed blog post written in markdown is committed to main branch
27+
2. AWS Amplify CI pipeline picks up changes and runs build
28+
3. On success, Amplify publishes a `Amplify Deployment Status Change` event to EventBridge, triggering a Lambda function deployed in this stack
29+
4. The function uses your GitHub PAT to identify and load the blog post content and pass it into a Step Function workflow
30+
5. The workflow will do an idempotency check, and if it's ok to continue will transform and publish to Medium, Hashnode, and Dev.to in parallel
31+
6. After publish is complete, the workflow checks if there were any failures.
32+
* If there was a failure, it sends an email with a link to the execution for debugging
33+
* On success, it sends an email with links to the published content and updates the idempotency record and article catalog
34+
35+
*Note - If you do not provide a SendGrid API key, you will not receive email status updates*
36+
37+
## Platforms
38+
39+
This solution will take content you create and automatically cross-post it on three platforms:
40+
41+
* [Medium](https://medium.com)
42+
* [Dev.to](https://dev.to)
43+
* [Hashnode](https://hashnode.com)
44+
45+
46+
47+
## Deployment
48+
49+
The solution is built using AWS SAM. To deploy the resources into the cloud you must install the [SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html).
50+
51+
Once installed, run the following commands in the root folder of the solution.
52+
53+
```bash
54+
sam build --parallel
55+
sam deploy --guided
56+
```
57+
58+
This will walk you through deployment, prompting you for all the parameters necessary for proper use. Below are the parameters you must fill out on deploy.
59+
60+
|Parameter|Description|Required|
61+
|---------|-----------|--------|
62+
|TableName|Name of the DynamoDB table to create|No|
63+
|GSI1|Name of the GSI on the DDB table|No|
64+
|GitHubPAT|Personal Access Token to load newsletter content from your repository|Yes|
65+
|GitHubOwner|The GitHub user name that owns the repository for your content|Yes|
66+
|GitHubRepo|The repository name that contains your content|Yes|
67+
|AmplifyProjectId|Identifier of the Amplify project that builds your newsletter|Yes|
68+
|MediumApiKey|API key used to manipulate data in your Medium account|Yes|
69+
|MediumPublicationId|Identifier of the publication you wish to submit to on Medium|No|
70+
|MediumAuthorId|Identifier of your user on Medium|Yes if `MediumPublicationId` is not provided|
71+
|DevApiKey|API key used to manipulate data in your Dev.to account|Yes|
72+
|DevOrganizationId|Identifier of the organization you wish to submit to on Dev.to|No|
73+
|HashnodeApiKey|API key used to manipulate data in your Hashnode account|Yes|
74+
|HashnodePublicationId|Identifier for your blog publication on Hashnode|Yes|
75+
|HashnodeBlogUrl|Base url of your blog hosted in Hashnode|Yes|
76+
|BlogBaseUrl|Vase url of your blog on your personal site|Yes|
77+
|BlogContentPath|Relative path from the root directory to the blog content folder in your GitHub repo|Yes|
78+
|SendgridApiKey|Api Key of the SendGrid account that will send the status report when cross-posting is complete|No|
79+
|NotificationEmail|Email address to notify when cross posting is complete|No|
80+
|SendgridFromEmail|Email address for SendGrid that sends you the status email|No|
81+
82+
## Notification Emails
83+
84+
If you wish to get notification emails on the status of the cross posting, you must use [SendGrid](https://sendgrid.com). SendGrid offers a generous free tier for email messages and is quick to get started. To configure SendGrid to send you emails you must:
85+
86+
* [Create an API key](https://docs.sendgrid.com/ui/account-and-settings/api-keys)
87+
* [Create a sender](https://docs.sendgrid.com/ui/sending-email/senders)
88+
89+
Once you perform the above actions, you may use the values in the respective deployment variables listed above.
90+
91+
## Replay
92+
93+
In the event the cross-posting does not work, it can be safely retried without worrying about pushing your content multiple times. Each post will update the idempotency DynamoDB record for the cross-posting state machine. This record holds the status (*success/failure*) for each platform. If the article was successfully posted on a platform, it will be skipped on subsequent executions.
94+
95+
## Blog Metadata
96+
97+
Your blog must be written in Markdown for this solution to work appropriately. To save metadata about your post, you can add [front matter](https://gohugo.io/content-management/front-matter/) at the beginning of the file. This solution requires a specific set of metadata in order to function appropriately.
98+
99+
**Example**
100+
```yaml
101+
---
102+
title: My first blog!
103+
description: This is the subtitle that is used for SEO and visible in Medium and Hashnode posts.
104+
image: https://link-to-hero-image.png
105+
image_attribution: Any attribution required for hero image
106+
categories:
107+
- categoryOne
108+
tags:
109+
- serverless
110+
- other tag
111+
slug: /my-first-blog
112+
---
113+
```
114+
115+
|Field|Description|Required?|
116+
|-----|-----------|---------|
117+
|title|Title of the blog issue |Yes|
118+
|description| Brief summary of article. This shows up on Hashnode and Medium and is used in SEO previews|Yes|
119+
|image|Link to the hero image for your article|Yes|
120+
|image_attribution|Any attribution text needed for your hero image|No|
121+
|categories|Array of categories. This will be used as tags for Dev and Medium|No|
122+
|tags|Array of tags. Also used as tags for Dev and Medium|No|
123+
|slug|Relative url of your post. Used in the article catalog|Yes|
124+
125+
## Article Catalog
126+
127+
One of the neat features provided by this solution is substituting relative urls for the appropriate urls on a given page. For example, if you use a relative url to link to another blog post you've written on your site, this solution will replace that with the cross-posted version. So Medium articles will always point to Medium articles, Hashnode articles will always point to Hashnode, etc...
128+
129+
This is managed for you by the solution. It creates entries for your content in DynamoDB with the following format:
130+
131+
```json
132+
{
133+
"pk": "<article slug>",
134+
"sk": "article",
135+
"GSI1PK": "article",
136+
"GSI1SK": "<title of the post>",
137+
"links": {
138+
"url": "<article slug>",
139+
"devUrl": "<full path to article on dev.to>",
140+
"mediumUrl": "<full path to article on Medium>",
141+
"hashnodeUrl": "<full path to article on Hashnode>"
142+
}
143+
}
144+
```
145+
146+
When transforming your Markdown content, it will load all articles from DynamoDB, use a Regex to match on the article slug in your content, and replace with the url of appropriate site.
147+
148+
If you already have a number of articles and wish to seed the database with the cross references, you will have to compile the data manually and put it in the following format:
149+
150+
```json
151+
[
152+
{
153+
"title": "<title of article>",
154+
"devUrl": "<url of article on dev.to>",
155+
"url": "<relative url of article on your blog>",
156+
"mediumUrl": "<url of article on medium>",
157+
"hashnodeUrl": "<url of article on hashnode>"
158+
}
159+
]
160+
```
161+
162+
Take this data and update the [load-cross-posts](/functions/load-cross-posts/index.js) function to load and handle that data. Run the function manually to seed the data in your database table.
163+
164+
## Embeds
165+
166+
If you are embedding content in your posts, they might not work out of the box. *There is only support for Hugo twitter embeds.* The format of a Hugo Twitter embed is:
167+
168+
```
169+
{{<tweet user="" id="">}}
170+
```
171+
172+
If you include this in your content, it will be automatically transformed to the appropriate embed style on the appropriate platform.
173+
174+
## Limitations
175+
176+
Below are a list of known limitations:
177+
178+
* Your content must be written in Markdown with front matter describing the blog post.
179+
* Content must be hosted in GitHub.
180+
* You are required to post to Dev.to, Medium, and Hashnode. You cannot pick and choose which platforms you want to use.
181+
* Only Hugo style Twitter embeds are supported. Embeds for other content will not work.
182+
* This process is triggered on a successful build of an AWS Amplify project. Other triggers are not supported (but can easily be modified to add them).
183+
* Notifications are limited to sending emails in SendGrid.
184+
* The only way to deploy the solution is with AWS SAM.
185+
186+
## Contributions
187+
188+
Please feel free to contribute to this project! Bonus points if you can meaningfully address any of the limitations listed above :)
189+
190+
This is an AWS Community Builders project and is meant to help the community. If you see fit, please donate some time into making it better!

‎docs/workflow.png

341 KB
Loading

‎functions/identify-new-content/index.js

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
const { Octokit } = require('octokit');
2+
const { SFNClient, StartExecutionCommand } = require('@aws-sdk/client-sfn');
3+
const shared = require('/opt/nodejs/index');
4+
5+
const sfn = new SFNClient();
6+
let octokit;
7+
8+
exports.handler = async (event) => {
9+
try {
10+
await initializeOctokit();
11+
12+
const recentCommits = await getRecentCommits();
13+
if (recentCommits.length) {
14+
const newContent = await getNewContent(recentCommits);
15+
if (newContent.length) {
16+
const data = await getContentData(newContent);
17+
await processNewContent(data);
18+
}
19+
}
20+
} catch (err) {
21+
console.error(err);
22+
}
23+
};
24+
25+
const initializeOctokit = async () => {
26+
if (!octokit) {
27+
const gitHubSecret = await shared.getSecret('github');
28+
octokit = new Octokit({ auth: gitHubSecret });
29+
}
30+
};
31+
32+
const getRecentCommits = async () => {
33+
const timeTolerance = Number(process.env.COMMIT_TIME_TOLERANCE_MINUTES);
34+
const date = new Date();
35+
date.setMinutes(date.getMinutes() - timeTolerance);
36+
37+
const result = await octokit.rest.repos.listCommits({
38+
owner: process.env.OWNER,
39+
repo: process.env.REPO,
40+
path: process.env.PATH,
41+
since: date.toISOString()
42+
});
43+
44+
const newPostCommits = result.data.filter(c => c.commit.message.toLowerCase().startsWith(process.env.NEW_CONTENT_INDICATOR));
45+
return newPostCommits.map(d => d.sha);
46+
};
47+
48+
const getNewContent = async (commits) => {
49+
const newContent = await Promise.allSettled(commits.map(async (commit) => {
50+
const commitDetail = await octokit.rest.repos.getCommit({
51+
owner: process.env.OWNER,
52+
repo: process.env.REPO,
53+
ref: commit
54+
});
55+
56+
const newFiles = commitDetail.data.files.filter(f => f.status == 'added' && f.filename.startsWith(`${process.env.PATH}/`));
57+
return newFiles.map(p => {
58+
return {
59+
fileName: p.filename,
60+
commit: commit
61+
}
62+
});
63+
}));
64+
65+
let content = [];
66+
for (const result of newContent) {
67+
if (result.status == 'rejected') {
68+
console.error(result.reason);
69+
} else {
70+
content = [...content, ...result.value];
71+
}
72+
}
73+
74+
return content;
75+
};
76+
77+
const getContentData = async (newContent) => {
78+
const contentData = await Promise.allSettled(newContent.map(async (content) => {
79+
const postContent = await octokit.request('GET /repos/{owner}/{repo}/contents/{path}', {
80+
owner: process.env.OWNER,
81+
repo: process.env.REPO,
82+
path: content.fileName
83+
});
84+
85+
const buffer = Buffer.from(postContent.data.content, 'base64');
86+
const data = buffer.toString('utf8');
87+
88+
return {
89+
fileName: content.fileName,
90+
commit: content.commit,
91+
content: data,
92+
sendStatusEmail: process.env.SEND_STATUS_EMAIL == 'true'
93+
};
94+
}));
95+
96+
let allContent = [];
97+
for (const result of contentData) {
98+
if (result.status == 'rejected') {
99+
console.error(result.reason);
100+
} else {
101+
allContent.push(result.value);
102+
}
103+
}
104+
105+
return allContent;
106+
};
107+
108+
const processNewContent = async (newContent) => {
109+
const executions = await Promise.allSettled(newContent.map(async (content) => {
110+
const command = new StartExecutionCommand({
111+
stateMachineArn: process.env.STATE_MACHINE_ARN,
112+
input: JSON.stringify(content)
113+
});
114+
await sfn.send(command);
115+
}));
116+
117+
for (const execution of executions) {
118+
if (execution.status == 'rejected') {
119+
console.error(execution.reason);
120+
}
121+
}
122+
};

0 commit comments

Comments
 (0)