# DataparkSearch HTML parser

## Tag parser

Tag parser understands the following tag notation:

- < ... parameter=value ... >

- < ... parameter="value" ... >

- < ... parameter='value' ... >

## Special characters

indexer understands the following special HTML characters:

- &lt; &gt; &amp; &nbsp; &quot;

- All HTML-4 character entities: &auml; &uuml; and other.

- Characters in their Unicode code notation: &#234;

## META tags

Indexer's HTML parser currently understands the following META tags. Note that `"HTTP-EQUIV"` may be used instead of `"NAME"` in all entries.

- `<META NAME="Content-Type" Content="text/html; charset=xxxx">` This is used to detect document character set if it is not specified in `Content-type` HTTP header.

- `<META NAME="REFRESH" Content="5; URL=http://www.somewhere.com">` URL value will be inserted in database.

- `<META NAME="Keywords" Content="xxx">`

- `<META NAME="Description" Content="xxx">`

- `<META NAME="Robots" Content="xxx">` with content value `ALL, NONE, INDEX, NOINDEX, FOLLOW, NOFOLLOW, NOARCHIVE`.

- `<META NAME="DP.PopRank" Content="xxx">` with Content value as a real number. This is used to assign the initial value of PopularityRank for the page.

## Links

HTML parser understand the following links:

- <A HREF="xxx">

- <IMG SRC="xxx">

- <LINK HREF="xxx">

- <FRAME SRC="xxx">

- <AREA HREF="xxx">

- <BASE HREF="xxx">

  If BASE HREF value has incorrectly formed URL, current one will be used instead to compose relative links.

## Comments

- Text inside the <!-- .... --> tag is recognized as HTML comment.

- You may use special <!--UdmComment--> .... <!--/UdmComment--> comment tags to exclude the text between from indexing. This may be useful to hide such things like menus and others from indexing.

- You may also use <NOINDEX> ... </NOINDEX> as a synonyms to <!--UdmComment--> and <!--/UdmComment-->

- For compatibility with ASPSeek, <!--noindex--> ... <!--/noindex--> are also equivalent to <!--UdmComment--> and <!--/UdmComment-->

## Body patterns

If you need index not whole page, for example, to exclude navigation, ads, etc., you may use `BodyPattern` command to specify a pattern to extract content of a page for indexing. For example:

    BodyPattern "<!--content-->(.*)<!--/content-->" "$1"

this pattern will extract content between special comments and only that content will be indexed for this page.

You may specify several `BodyPattern` commands, but only the first match will be applied to a page. These patterns are trying to apply to all pages indexed. Beware, huge number of such body patterns may hurt indexing speed.