Package Exports
- notion-page-tree
- notion-page-tree/dist/index.js
This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (notion-page-tree) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.
Readme
Notion Page Tree
Fetch nested Notion pages from the root page/database/block.
Why Would I Want This?
π Use the official Notion API.
- Popular
/loadpagechunk/
endpoint is not public and may not be stable in future updates. - Official API can be integrated with private key, so you can keep your database private.
πΈ Fetch nested children pages.
- Pages inside non-page blocks are also fetched.
- Max search-depth can be set in your preference.
π Handle API Errors gracefully.
- Maximum fetch concurrency is set to avoid
rate_limited
error. - On
rate_limited
error, it stops and waits for some minutes. - Other errors are automatically retried. Max retry count can be set in your preference.
Other Features
πΎ It saves fetch results to your local disk.
- Set parameter
private_file_path
to your custom path.
π₯ It builds basic page server.
/page/:id/
endpoint for retrieving page and its childrens' id./tree/:id/
endpoint for retrieving all nested pages from the page.
π It builds basic page search indexes.
- Uses lunr.js.
- Page's properties and chilren are converted into plain text for building search index.
/search?keyword=
endpoint for searching page properties and retrieving page ids./suggestion?keyword=
endpoint for looking for search index's tokens.
Usage
.env
File Configuration
Write directly on <package_root>/.env
NOTION_ENTRY_ID = <root page/database/block's id>
NOTION_ENTRY_KEY = <root's integration key>
NOTION_ENTRY_TYPE = <page/database/block>
Create Instance
Interactively Ask for parameters.
const notionPageTree = new NotionPageTree({
fetchIntervalMinutes: 5,
private_file_path: path.resolve('./sample/')
});
Fetcher and Server Setup (sample)
Run yarn serve
to see fetcher running in action.
Sample database is here https://notion.so/2345a2ce3cdd48f183cca1f6d1ae25ca
./sample/index.ts
import NotionPageTree from 'notion-page-tree';
import path from 'path';
(async function main() {
// Create main class instance.
const notionPageTree = new NotionPageTree({
private_file_path: path.resolve('./sample/'),
createFetchQueueOptions: {
maxConcurrency: 3, // Current official rate limit is 3 requests per second. Notion api will throw error when you increase this value.
maxRetry: 2, // "rate_limited" error will not be retried and process will be exited immediately.
maxRequestDepth: 3, // Depth applied to all the entities.
maxBlockDepth: 2, // Depth applied to blocks (relative to nearest parent page).
databaseQueryFilter: {
// Database query filter.
property: 'isPublished',
checkbox: {
equals: true
}
}
}
});
notionPageTree.parseCachedDocument();
// Look for cached documents in `private_file_path`, assign to Page Data Variables.
const requestParameters = await notionPageTree.setRequestParameters({
forceRewrite: false,
prompt: true
});
console.log(
'Requesting',
requestParameters.entry_type,
'with id',
requestParameters.entry_id
);
// Set environment variables that are needed for Notion API.
const server = notionPageTree.setupServer({ port: 8888 });
// Setup servers for listing and searching pages. (will respond 503 if pages are undefined)
// await notionPageTree.fetchOnce();
// Fetch pages asynchronously, then assign results to variables.
notionPageTree.startFetchLoop(1000 * 10); // 10 seconds
// Create asynchronouse fetch loop, which waits for some milliseconds between each fetch.
setTimeout(() => {
// after 30 seconds
console.log('stopping fetch loop');
notionPageTree.stopFetchLoop();
// Stopping fetch loop after current fetch resolves.
console.log('closing servers');
server.close();
// Stopping servers immediately.
}, 1000 * 30);
})();
How It Works (Flowchart)
/******************************************************************************************************************************************************************************************************************************************************************************************************************************\
* *
* *
* *
* *
* ββββββββMain Routineββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ *
* β β *
* β β *
* β βββββββββββββββββββββββββ βββββββββββββββββββββββββ β *
* β βββββΆβ page_collection β βclearTimeout()ββββΆβ Request Promise Timer β β *
* β β βββββββββββββββββββββββββ β βββββββββββββββββββββββββ β *
* β β Ξ β β *
* β β βββββββββββββββββββββββββ β± β² β βββββββββββββββββββββββββ β *
* β βββββΆβ page_tree β β± β² βclearTimeout()ββββΆβ Request Ready Timer β β *
* β β βββββββββββββββββββββββββ β± β² β βββββββββββββββββββββββββ β *
* β β β± β² β β *
* β β βββββββββββββββββββββββββ β± check β² β βββββββββββββββββββ β *
* β βββββΆβ Request Promise Queue ββββββββ β± routine β² β ββββupdate ββββΆβ page_collection β β *
* β βββββββββββββββββββββββββ β βββββββββββββββββββββββββ β β± β» β² β β βββββββββββββββββββ β *
* β β Fetcher Routine ββββββ€ βββββββΆβ promise = 0 ββββ¬βtrueβββΌββββ€ β *
* β βββββββββββββββββββββββββ β βββββββββββββββββββββββββ β β² ready = 0 β± β β β βββββββββββββββββββ β *
* β β² βββββΆβ Request Ready Queue ββββββββ β² β» β± β β ββββupdateβββββΆβ page_tree β β *
* β β β βββββββββββββββββββββββββ β² 0ms β± β β βββββββββββββββββββ β *
* β β β β² β± β β β *
* β β β βββββββββββββββββββββββββ β² β± β β β *
* β β βββββΆβ Request Promise Timer β β² β± β ββββββββββββββββββββΆ wait for some minutes β β β β *
* β β β βββββββββββββββββββββββββ β² β± β β *
* β β β V β β β *
* β β β βββββββββββββββββββββββββ β² β β *
* β β βββββΆβ Request Ready Timer β βββfalseββββ β β *
* β β βββββββββββββββββββββββββ β *
* β β β β *
* β ββ β β β β β β β β β β β β β β β β β β β β β β β β β β β create new fetcher routine β β β β β β β β β β β β β β β β β β β β β β β β β β *
* β β *
* ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ *
* *
* β *
* *
* β *
* *
* ββββFetcher Routineββ»ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ *
* β ββ.plaintext += plaintextβββββββββββββββββββββββββββββββββββββββββββββββββββββ β *
* β β β β *
* β ββββββββββββββββββββββββββββ β ββββββββββββββββββββββββ β β *
* β βββfalseββββ β Promise (resolved) β ββ.children.push()β βConnector β β β *
* β ββββββββββββββββββββββββββββββββββ β β ββββββββββββββββββββββββββ β β β β ββββββββββββββββββββ β β β *
* β β Request Promise Queue β β β ββparentToAssign: Entity βββββ ββββββββββββββββββββ β βtoAssigned: itselfβ β β β *
* β β ββββββββββββββββββββββββββββ β β Ξ ββββββββββββββββββββββββββ β ββββββΆβ is Page/Database βββcreateβββΆβ ββββββββββββββββββββ ββββββββββββββββββββββββββββββββββ β *
* β β β Promise (pending) β β β β± β² ββββββββββββββββββββββββββ β β ββββββββββββββββββββ ββββββββββββββββββββββββ β β β *
* β βββββββββββββββββββββββββββββββββββ β β ββββββββββββββββββββββββββ β β β± β² ββparentToRequest: Entityβ β β β ββtoRequested: itself ββ β β β *
* β β New Requests ββββββββββββββββββββββββββββββββββββββββββββββββ¬βΆβparentToAssign: Entity ββ β β β± β² ββββββββββββββββ ββββββββββββββββββββββββββ β β β ββββββββββββββββββββββββ β β β *
* β β βββββββββββββββββββββββββββ β β ββββββββββββββββββββββββββββββββββββββββ β β ββββββββββββββββββββββββββ...β β β± β² ββββΆβ is Fulfilled ββββΆββββββββββββββββββββββββββ β β β ββββββββββββββββββββββββ β β β *
* β β βββββββββββββββββββββββββββ β β β β β β ββββββββββββββββββββββββββ β β β± set β² β ββββββββββββββββ ββ children: β β β β β β β *
* β β ββparentToAssign: Entity ββ ββββ β β βββββββββββββββ List Block βββββ¬βΆβparentToRequest: Entityββ β β β± interval β² β ββ QueryablePromise βββ¬ββββ€ β ββββββββββββββββββββ β β *
* β β βββββββββββββββββββββββββββ β ββββΆβis Block/PageβββββΆChildren()ββββββ β β ββββββββββββββββββββββββββ β β β± β» β² β ββ Entity[] β β β β extractPlainText β Concatenated β β β *
* β β βββββββββββββββββββββββββββ ... β β βββββββββββββββ β β β ββββββββββββββββββββββββββ βββ΄ββΆβ promise > 0 βββββ€ ββββββββββββββββββββββββββ β β β βββΆ FromBlock βββββΆ.reduce()ββββΆβ Plain Text β β β *
* β β ββparentToRequest: Entityββ ββββββββ .push()β β β children: ββ β β² isSetteled β± β ββββββββββββββββ ββββββββββββββββββββββββββ β β β β ββββββββββββββββββββ β β *
* β β βββββββββββββββββββββββββββ β β βββββββββββββββ Query ββββββ¬βΆβ QueryablePromise ββ β β² β» β± ββββΆβ is Rejected β ββ retry: number β β β β β β β *
* β β βββββββββββββββββββββββββββ β ββββΆβ is Database βββββΆDatabase()ββββββ β β β Entity[] ββ β β² 0ms β± ββββββββββββββββ ββββββββββββββββββββββββββ β β ββββββββββββββββββββ β β β *
* β β ββ retry: number ββ β βββββββββββββββ β β ββββββββββββββββββββββββββ β β² β± β ββββββββββββββββββββββββββββ ββββββΆβ is Block β ββββ€ ββββββββββββββββββββ β β *
* β β βββββββββββββββββββββββββββ β β β ββββββββββββββββββββββββββ β β² β± β ββββββββββββββββββββ β β is in Traverse β β β *
* β β βββββββββββββββββββββββββββ β β β β retry: number ββ β β² β± β set maxConcurrency to 0 β β ββββββΆβ Exclusion List β βββββββββββββ¬βββββββββββ β *
* β ββββββββββββββββ²βββββββββββββββββββ β β ββββββββββββββββββββββββββ β β² β± βΌ empty promise_queue β β β ββββββββββββββββββββ βConnector β β β *
* β β β ββββββββββββββββββββββββββββ β V rate_limitedββtrueβββΆ move promises to ready_queue β βββΆ.filter()βββββ€ ββββββββββββββββββββ β βββββββββββ΄βββββββββ β β *
* β β ββββββββββββββββββββββββββββββββββ β wait for some minutes β β βis NOT in Traverseβ β βtoAssigned: PARENTβ β β *
* β β β set maxConcurrency to 3 β ββββββΆβ Exclusion List βββcreateβββΆβ βββββββββββ¬βββββββββ β β *
* β β βββββββββββββββββββββββββββ false β ββββββββββββββββββββ βββββββββββββ΄βββββββββββ β *
* β β βββββββββββββββββββββββββββ β β ββtoRequested: itself ββ β *
* β .splice(concurrency - ββparentToAssign: Entity ββ β β βββββββββββββ¬βββββββββββ β *
* β promise.length) βββββββββββββββββββββββββββ βΌ β βββββββββββββ¬βββββββββββ β *
* β β βββββββββββββββββββββββββββ retry < β β β *
* β β βββββββββparentToRequest: Entityββββββtrueβββββ retryCount ββfalseβββΆ console.error β β β *
* β β β βββββββββββββββββββββββββββ β β β *
* β β β βββββββββββββββββββββββββββ β β β *
* β β βββfalseββββ β ββ retry: +=1 ββ β β β *
* β β β β β βββββββββββββββββββββββββββ β β β *
* β β β β .unshift() βββββββββββββββββββββββββββ β β β *
* β β Ξ β βΌ β β β *
* β β β± β² β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β *
* β β β± β² β β Request Ready Queue β β β β *
* β β β± β² β β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β β *
* β β β± β² β β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β β *
* β β β± check β² β β ββparentToAssign: Entity ββ ββparentToAssign: Entity ββ β β β β *
* β β β± routine β² β β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β β *
* β β β± β» β² β β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ...β β β β *
* β βββββββtrueββββββββ ready > 0 ββββ΄βββββββββββββ ββparentToRequest: Entityββ ββparentToRequest: Entityββ ββββββββββββββ.push()βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β *
* β β² promise < 3 β± β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β *
* β β² β± β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β *
* β β² β» β± β ββ retry: 0 ββ ββ retry: 0 ββ β β β *
* β β² 250ms β± β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β *
* β β² β± β βββββββββββββββββββββββββββ βββββββββββββββββββββββββββ β β β *
* β β² β± βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β *
* β β² β± β² β β *
* β V β ββββββββββββββββββββββββββββ β β *
* β (init) β Page Collection β β β *
* β β β ββββββ¦ββββββββββββββββββ β β β *
* β β β β id β page: Entity β β β β *
* β β β ββββββ©ββββββββββββββββββ β β β *
* β ββββββββββββββββββββ β ββββββ¦ββββββββββββββββββ β β β *
* β β ROOT ββββ(init)ββΆβ β id β page: Entity β ββββββββββββββassign with keyβββββββββββββββββββββββββββββββββββββββββ β *
* β ββββββββββββββββββββ β ββββββ©ββββββββββββββββββ β β *
* β β β ββββββ¦ββββββββββββββββββ β β *
* β β β id β page: Entity β β β *
* β β β ββββββ©ββββββββββββββββββ β β *
* β β .... β β *
* β β ββββββββββββββββββββββββββββ β *
* β β β *
* β β β β β β β β β β¬ β β β β β β β β β β *
* β β *
* β β β *
* β ββββββββββββββββββββββββ β *
* β β Entity β β *
* β ββββββββββββββββββββββββ β *
* β ββ id: string ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β ββ depth: number ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β ββ type: ββ β *
* β ββ 'page'|'database'| ββ β *
* β ββ 'block' ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β ββ metadata: ββ β *
* β ββGetBlockResponseWithββ β *
* β ββ Metadata ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β ββ plainText: ββ β *
* β ββextractPlainText(...ββ β *
* β ββ children) ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β ββ children: Entity[] ββ β *
* β ββββββββββββββββββββββββ β *
* β ββββββββββββββββββββββββ β *
* β β *
* β β *
* β β *
* βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ *
* *
* *
* *
* *
\******************************************************************************************************************************************************************************************************************************************************************************************************************************/