article-parser

Extract main article, main image and meta data from URL

ndaidong/article-parserdemos.pwshub.com/article-parserarticle-parser

README

article-parser


Extract main article, main image and meta data from URL.
NPM
CI test Coverage Status
CodeQL JavaScript Style Guide


Intro


article-parser is a part of tool sets for content builder:

- feed-reader: extract & normalize RSS/ATOM/JSON feed
- article-parser: extract main article from given URL
- oembed-parser: extract oEmbed data from supported providers

You can use one or combination of these tools to build news sites, create automated content systems for marketing campaign or gather dataset for NLP projects...

  1. ```
  2.                                     ┌────────────────┐
  3.                             ┌───────► article-parser ├──────────┐
  4.                             │       └────────────────┘          │
  5. ┌─────────────┐   ┌─────────┴────┐                     ┌────────▼─────────┐   ┌─────────────┐
  6. │ feed-reader ├───► feed entries │                     │ content database ├───► public APIs │
  7. └─────────────┘   └─────────┬────┘                     └────────▲─────────┘   └─────────────┘
  8.                             │       ┌────────────────┐          │
  9.                             └───────► oembed-parser  ├──────────┘
  10.                                     └────────────────┘
  11. ```

Demo


- Give it a try!
- Example FaaS


Install & Usage


Node.js


  1. ``` sh
  2. npm i article-parser

  4. # pnpm
  5. pnpm i article-parser

  7. # yarn
  8. yarn add article-parser
  9. ```

  1. ```ts
  2. // es6 module
  3. import { extract } from 'article-parser'

  5. // CommonJS
  6. const { extract } = require('article-parser')

  8. // or specify exactly path to CommonJS variant
  9. const { extract } = require('article-parser/dist/cjs/article-parser.js')
  10. ```

Deno


  1. ```ts
  2. import { extract } from 'https://esm.sh/article-parser'
  3. ```

Browser


  1. ```ts
  2. import { extract } from 'https://unpkg.com/article-parser@latest/dist/article-parser.esm.js'
  3. ```

Please check the examples for reference.


Deta cloud


For Deta devs please refer the source code and guideline here or simply click the button below.
Deploy


APIs


- extract()
- Transformations
  - [transformation object](#transformation-object)
  - .addTransformations
  - .removeTransformations
  - Priority order
- [sanitize-html's options](#sanitize-htmls-options)


extract()


Load and extract article data. Return a Promise object.

Syntax


  1. ```ts
  2. extract(String input)
  3. extract(String input, Object parserOptions)
  4. extract(String input, Object parserOptions, Object fetchOptions)
  5. ```

Parameters


input required

URL string links to the article or HTML content of that web page.

For example:

  1. ``` js
  2. import { extract } from 'article-parser'

  4. const input = 'https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html'
  5. extract(input)
  6.   .then(article => console.log(article))
  7.   .catch(err => console.error(err))
  8. ```

The result - article - can be null or an object with the following structure:

  1. ```ts
  2. {
  3.   url: String,
  4.   title: String,
  5.   description: String,
  6.   image: String,
  7.   author: String,
  8.   content: String,
  9.   published: Date String,
  10.   source: String, // original publisher
  11.   links: Array, // list of alternative links
  12.   ttr: Number, // time to read in second, 0 = unknown
  13. }
  14. ```

Click here for seeing an actual result.


parserOptions optional

Object with all or several of the following properties:

  - wordsPerMinute: Number, to estimate time to read. Default 300.
  - descriptionTruncateLen: Number, max num of chars generated for description. Default 210.
  - descriptionLengthThreshold: Number, min num of chars required for description. Default 180.
  - contentLengthThreshold: Number, min num of chars required for content. Default 200.

For example:

  1. ``` js
  2. import { extract } from 'article-parser'

  4. extract('https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html', {
  5.   descriptionLengthThreshold: 120,
  6.   contentLengthThreshold: 500
  7. })
  8. ```

fetchOptions optional

You can use this param to set request headers to fetch.

For example:

  1. ``` js
  2. import { extract } from 'article-parser'

  4. const url = 'https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html'
  5. extract(url, null, {
  6.   headers: {
  7.     'user-agent': 'Opera/9.60 (Windows NT 6.0; U; en) Presto/2.1.1'
  8.   }
  9. })
  10. ```

You can also specify a proxy endpoint to load remote content, instead of fetching directly.

For example:

  1. ``` js
  2. import { extract } from 'article-parser'

  4. const url = 'https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html'

  6. extract(url, null, {
  7.   headers: {
  8.     'user-agent': 'Opera/9.60 (Windows NT 6.0; U; en) Presto/2.1.1'
  9.   },
  10.   proxy: {
  11.     target: 'https://your-secret-proxy.io/loadXml?url=',
  12.     headers: {
  13.       'Proxy-Authorization': 'Bearer YWxhZGRpbjpvcGVuc2VzYW1l...'
  14.     }
  15.   }
  16. })
  17. ```

Passing requests to proxy is useful while running article-parser on browser. View examples/browser-article-parser as reference example.

For more info about proxy authentication, please refer HTTP authentication

For a deeper customization, you can consider using Proxy to replacefetch behaviors with your own handlers.


Transformations


Sometimes the default extraction algorithm may not work well. That is the time when we need transformations.

By adding some functions before and after the main extraction step, we aim to come up with a better result as much as possible.

There are 2 methods to play with transformations:

- addTransformations(Object transformation | Array transformations)
- removeTransformations(Array patterns)

At first, let's talk about transformation object.

transformation object


In article-parser, transformation is an object with the following properties:

- patterns: required, a list of regexps to match the URLs
- pre: optional, a function to process raw HTML
- post: optional, a function to process extracted article

Basically, the meaning of transformation can be interpreted like this:

with the urls which match these patterns <br>

let's run pre function to normalize HTML content <br>

then extract main article content with normalized HTML, and if success <br>

let's run post function to normalize extracted article content


article-parser extraction process

Here is an example transformation:

  1. ``` js
  2. {
  3.   patterns: [
  4.     /([\w]+.)?domain.tld\/*/,
  5.     /domain.tld\/articles\/*/
  6.   ],
  7.   pre: (document) => {
  8.     // remove all .advertise-area and its siblings from raw HTML content
  9.     document.querySelectorAll('.advertise-area').forEach((element) => {
  10.       if (element.nodeName === 'DIV') {
  11.         while (element.nextSibling) {
  12.           element.parentNode.removeChild(element.nextSibling)
  13.         }
  14.         element.parentNode.removeChild(element)
  15.       }
  16.     })
  17.     return document
  18.   },
  19.   post: (document) => {
  20.     // with extracted article, replace all h4 tags with h2
  21.     document.querySelectorAll('h4').forEach((element) => {
  22.       const h2Element = document.createElement('h2')
  23.       h2Element.innerHTML = element.innerHTML
  24.       element.parentNode.replaceChild(h2Element, element)
  25.     })
  26.     // change small sized images to original version
  27.     document.querySelectorAll('img').forEach((element) => {
  28.       const src = element.getAttribute('src')
  29.       if (src.includes('domain.tld/pics/150x120/')) {
  30.         const fullSrc = src.replace('/pics/150x120/', '/pics/original/')
  31.         element.setAttribute('src', fullSrc)
  32.       }
  33.     })
  34.     return document
  35.   }
  36. }
  37. ```

- To write better transformation logic, please refer linkedom and Document Object.

addTransformations(Object transformation | Array transformations)


Add a single transformation or a list of transformations. For example:

  1. ``` js
  2. import { addTransformations } from 'article-parser'

  4. addTransformations({
  5.   patterns: [
  6.     /([\w]+.)?abc.tld\/*/
  7.   ],
  8.   pre: (document) => {
  9.     // do something with document
  10.     return document
  11.   },
  12.   post: (document) => {
  13.     // do something with document
  14.     return document
  15.   }
  16. })

  18. addTransformations([
  19.   {
  20.     patterns: [
  21.       /([\w]+.)?def.tld\/*/
  22.     ],
  23.     pre: (document) => {
  24.       // do something with document
  25.       return document
  26.     },
  27.     post: (document) => {
  28.       // do something with document
  29.       return document
  30.     }
  31.   },
  32.   {
  33.     patterns: [
  34.       /([\w]+.)?xyz.tld\/*/
  35.     ],
  36.     pre: (document) => {
  37.       // do something with document
  38.       return document
  39.     },
  40.     post: (document) => {
  41.       // do something with document
  42.       return document
  43.     }
  44.   }
  45. ])
  46. ````

The transformations without patterns will be ignored.

removeTransformations(Array patterns)


To remove transformations that match the specific patterns.

For example, we can remove all added transformations above:

  1. ``` js
  2. import { removeTransformations } from 'article-parser'

  4. removeTransformations([
  5.   /([\w]+.)?abc.tld\/*/,
  6.   /([\w]+.)?def.tld\/*/,
  7.   /([\w]+.)?xyz.tld\/*/
  8. ])
  9. ```

Calling removeTransformations() without parameter will remove all current transformations.

Priority order


While processing an article, more than one transformation can be applied.

Suppose that we have the following transformations:

  1. ```ts
  2. [
  3.   {
  4.     patterns: [
  5.       /http(s?):\/\/google.com\/*/,
  6.       /http(s?):\/\/goo.gl\/*/
  7.     ],
  8.     pre: function_one,
  9.     post: function_two
  10.   },
  11.   {
  12.     patterns: [
  13.       /http(s?):\/\/goo.gl\/*/,
  14.       /http(s?):\/\/google.inc\/*/
  15.     ],
  16.     pre: function_three,
  17.     post: function_four
  18.   }
  19. ]
  20. ```

As you can see, an article from goo.gl certainly matches both them.

In this scenario, article-parser will execute both transformations, one by one:

function_one -> function_three -> extraction -> function_two -> function_four


sanitize-html's options


article-parser uses sanitize-html to make a clean sweep of HTML content.

Here is the default options

Depending on the needs of your content system, you might want to gather some HTML tags/attributes, while ignoring others.

There are 2 methods to access and modify these options in article-parser.

- getSanitizeHtmlOptions()
- setSanitizeHtmlOptions(Object sanitizeHtmlOptions)

Read sanitize-html docs for more info.


Quick evaluation


  1. ``` sh
  2. git clone https://github.com/ndaidong/article-parser.git
  3. cd article-parser
  4. pnpm i

  6. npm run eval {URL_TO_PARSE_ARTICLE}
  7. ```

License

The MIT License (MIT)