MCPcopy Create free account
hub / github.com/ScrapeGraphAI/Scrapegraph-ai / reduce_html

Function reduce_html

scrapegraphai/utils/cleanup_html.py:124–174  ·  view source on GitHub ↗

Reduces the size of the HTML content based on the specified level of reduction. Args: html (str): The HTML content to reduce. reduction (int): The level of reduction to apply to the HTML content. 0: minification only, 1: minification and removig unne

(html, reduction)

Source from the content-addressed store, hash-verified

122
123
124def reduce_html(html, reduction):
125 """
126 Reduces the size of the HTML content based on the specified level of reduction.
127
128 Args:
129 html (str): The HTML content to reduce.
130 reduction (int): The level of reduction to apply to the HTML content.
131 0: minification only,
132 1: minification and removig unnecessary tags and attributes,
133 2: minification, removig unnecessary tags and attributes,
134 simplifying text content, removing of the head tag
135
136 Returns:
137 str: The reduced HTML content based on the specified reduction level.
138 """
139 if reduction == 0:
140 return minify_html(html)
141
142 soup = BeautifulSoup(html, "html.parser")
143
144 for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
145 comment.extract()
146
147 for tag in soup(["style"]):
148 tag.string = ""
149
150 attrs_to_keep = ["class", "id", "href", "src", "type"]
151 for tag in soup.find_all(True):
152 for attr in list(tag.attrs):
153 if attr not in attrs_to_keep:
154 del tag[attr]
155
156 if reduction == 1:
157 return minify_html(str(soup))
158
159 for tag in soup(["style"]):
160 tag.decompose()
161
162 body = soup.body
163 if not body:
164 return "No <body> tag found in the HTML"
165
166 for tag in body.find_all(string=True):
167 if tag.parent.name not in ["script"]:
168 tag.replace_with(re.sub(r"\s+", " ", tag.strip())[:20])
169
170 reduced_html = str(body)
171
172 reduced_html = minify_html(reduced_html)
173
174 return reduced_html

Callers 5

test_reduce_html_no_bodyFunction · 0.90
executeMethod · 0.85

Calls 1

minify_htmlFunction · 0.85

Tested by 4

test_reduce_html_no_bodyFunction · 0.72