Represents a Story in the Editor. Please See Individual Parts of the Story.
Async function that Closes the author stream. See createAuthorStream for usage
Async function that Closes the section stream. See createSectionStream for usage
Async function that Closes the author stream. See createStoryAttributeStream for usage
Batch records in the stream. It will call your transform with a batch, and expect a batch of results to be returned Downstream from the pipe will see individual items, not a batch.
import { batchStream, Story } from '@quintype/migration-helpers';
async function mapRowToStories(rows: any, _batchNumber: number): Promise<ReadonlyArray<Story>> {
const someRelatedData = fetchInfoForStories(rows.map(r => r.storyId));
return rows.map(r => ({...rowToStory(r), ...someRelatedData(r.storyId)}));
}
writeStories(
readStoriesFromDatabase(conn)
.pipe(batchStream(100, mapRowToStories))
.on('end', () => conn.destroy())
);The maximum number of records in the batch
An optional function to transform the batch before being pushed onto the stream
Creates a writeable stream that can be used to pipe authors to. Authors that are piped to this stream are automatically writen to authors-*.txt.gz. The stream created with this function can be passed to writeStories
import { Author, createAuthorStream, endAuthorStream, writeStories } from "@quintype/migration-helpers";
async function main() {
const authorStream = createAuthorStream(mapAuthorIdToAuthor, opts);
await writeStories(getStoriesFromDB(), {...opts, authorStream});
await endAuthorStream(authorStream);
}An async function that maps from the external-ids given and the Author
Creates a writeable stream that can be used to pipe sections to. Sections that are piped to this stream are automatically writen to sections-*.txt.gz. The stream created with this function can be passed to writeStories
import { Section, createSectionStream, endSectionStream, writeStories } from "@quintype/migration-helpers";
async function main() {
const sectionStream = createSectionStream(mapSectionIdToSection, opts);
await writeStories(getStoriesFromDB(), {...opts, sectionStream});
await endSectionStream(sectionStream);
}An async function that maps from the external-ids given and the Section
Creates a writeable stream that can be used to pipe authors to. StoryAttributes that are piped to this stream are automatically writen to authors-*.txt.gz. The stream created with this function can be passed to writeStories
import { StoryAttribute, createStoryAttributeStream, endStoryAttributeStream, writeStories } from "@quintype/migration-helpers";
async function main() {
const authorStream = createStoryAttributeStream(mapStoryAttributeIdToStoryAttribute, opts);
await writeStories(getStoriesFromDB(), {...opts, authorStream});
await endStoryAttributeStream(authorStream);
}An async function that maps from the external-ids given and the StoryAttribute
Takes a generator of stories, and writes the stories into .txt.gz files. Each file will have 1000 stories.
You can provide a streams for various metadata to be written to. See createAuthorStream and createSectionStream for examples
import { batchStream, Story, writeStories } from '@quintype/migration-helpers';
async function* readStoriesFromDatabase(): AsyncIterableIterator<Story> {
const txn = createDbTxn();
const results = txn.runQuery("select * from stories");
while(results.hasNext()) {
yield rowToStory(results.next());
}
txn.close();
}
writeStories(readStoriesFromDatabase(), 'interviews')You may also use a stream to produce stories. Use batchStream to efficiently load stories.
import { Story, writeStories } from '@quintype/migration-helpers';
import { Transform } from 'stream'
const stream = conn.query("select * from stories").stream();
async function convertRowsToStories(rows: ReadonlyArray<any>): Promise<ReadonlyArray<Story>> {
const relatedData = await loadSomeData(rows.map(row => row.storyId));
return row.map(row => rowToStory(row, relatedData));
}
writeStories(stream.pipe(batchStream(100, convertRowsToStories)));A string describing where the stories come from. ex: interviews
Control some fine grained tuning
Generated using TypeDoc