mcp_server_webcrawl.models package

Submodules

mcp_server_webcrawl.models.resources module

class ResourceResultType

Bases: Enum

Enum representing different types of web resources.

UNDEFINED = ''

PAGE = 'html'

FRAME = 'iframe'

IMAGE = 'img'

AUDIO = 'audio'

VIDEO = 'video'

FONT = 'font'

CSS = 'style'

SCRIPT = 'script'

FEED = 'rss'

TEXT = 'text'

PDF = 'pdf'

DOC = 'doc'

OTHER = 'other'

classmethod values()

Return all values of the enum as a list.

Return type:

list[str]

classmethod to_int_map()

Return a dictionary mapping each enum value to its integer position.

Returns:

a dictionary with enum values as keys and their ordinal positions as values.

Return type:

dict

class ResourceResult

Bases: BaseModel

Represents a web resource result from a crawl operation.

Initialize a ResourceResult instance.

Parameters:

• id – resource identifier

• url – resource URL

• site – site identifier the resource belongs to

• crawl – crawl identifier the resource was found in

• type – type of resource

• name – resource name

• headers – HTTP headers

• content – resource content

• created – creation timestamp

• modified – last modification timestamp

• status – HTTP status code

• size – size in bytes

• time – response time in milliseconds

• thumbnail – base64 encoded thumbnail (experimental)

• metadata – additional metadata for the resource

__init__(id, url, site=None, crawl=None, type=ResourceResultType.UNDEFINED, name=None, headers=None, content=None, created=None, modified=None, status=None, size=None, time=None, metadata=None)

Initialize a ResourceResult instance.

Parameters:

• id (int) – resource identifier

• url (str) – resource URL

• site (int | None) – site identifier the resource belongs to

• crawl (int | None) – crawl identifier the resource was found in

• type (ResourceResultType) – type of resource

• name (str | None) – resource name

• headers (str | None) – HTTP headers

• content (str | None) – resource content

• created (datetime | None) – creation timestamp

• modified (datetime | None) – last modification timestamp

• status (int | None) – HTTP status code

• size (int | None) – size in bytes

• time (int | None) – response time in milliseconds

• thumbnail – base64 encoded thumbnail (experimental)

• metadata (dict[str, str | int | float | bool | datetime | Path | dict | list | None] | None) – additional metadata for the resource

to_dict()

Convert the object to a dictionary suitable for JSON serialization.

Return type:

dict[str, str | int | float | bool | datetime | Path | dict | list | None]

set_extra(extra_name, extra_value)

Parameters:

• extra_name (str) –

• extra_value (str | None | list[str] | list[dict[str, str | int | float]]) –

Return type:

None

get_extra(extra_name)

Parameters:

extra_name (str) –

Return type:

str | None | list[str] | list[dict[str, str | int | float]]

mcp_server_webcrawl.models.sites module

class SiteType

Bases: Enum

An enumeration.

UNDEFINED = 'undefined'

CRAWLED_URL = 'url'

CRAWLED_LIST = 'list'

class SiteResult

Bases: BaseModel

Represents a website or crawl directory result.

Initialize a SiteResult instance.

Parameters:

• id – site identifier

• name – site name, either a URL or a custom job

• urls – site URL(s), multiple for list type crawls

• path – path to site data, different from datasrc

• created – creation timestamp

• modified – last modification timestamp

• robots – robots.txt content

• metadata – additional metadata for the site

__init__(id, name=None, type=SiteType.CRAWLED_URL, urls=None, path=None, created=None, modified=None, robots=None, metadata=None)

Initialize a SiteResult instance.

Parameters:

• id (int) – site identifier

• name (str | None) – site name, either a URL or a custom job

• urls (list[str] | None) – site URL(s), multiple for list type crawls

• path (Path | None) – path to site data, different from datasrc

• created (datetime | None) – creation timestamp

• modified (datetime | None) – last modification timestamp

• robots (str | None) – robots.txt content

• metadata (dict[str, str | int | float | bool | datetime | Path | dict | list | None] | None) – additional metadata for the site

• type (SiteType) –

to_dict()

Convert the object to a dictionary suitable for JSON serialization.

Return type:

dict[str, str | int | float | bool | datetime | Path | dict | list | None]

Module contents