What scraping does for business
A scraper saves time when the alternative is manual copying. A sports nutrition store owner checks prices on 200 items across five competitors every week. Manually, that's two working days. A scraper does it in half an hour, including setup time.
Second scenario - data collection for analysis. A real estate agency scrapes listings from property platforms to understand price dynamics in a neighborhood. A marketer collects reviews from mapping services to analyze reputation.
Third - building a contact database. A B2B company collects emails and phone numbers from public directories to pitch services. This works if the data is public and the company posted it for contact purposes.
Scraping doesn't replace strategy, it speeds up routine work. If you don't know what to do with the data after collection, you don't need a scraper.
When scraping is legal
The law doesn't ban scraping as a technology. What's banned is violating a site's terms of service, stealing personal data, and circumventing technical protection measures.
Public data can be scraped. Prices in an online store, product descriptions, company addresses and phone numbers from directories - this is open information. If it's visible without registration or login, collecting it doesn't break the law.
Personal data is a separate story. Names, emails, phone numbers of individuals fall under data protection regulations. You can scrape them only if the person published the data for public use and gave consent for processing. Extracting a competitor's customer database from their CRM through a vulnerability - that's a criminal offense. Collecting business owner emails from their own websites for B2B outreach - legal, if you follow anti-spam requirements.
A site's terms of service may prohibit automated data collection. Formally, this is a contract, and violating it can lead to a lawsuit. In practice, courts rarely side with site owners if the data is public, but the risk exists. We usually recommend checking robots.txt and terms of use before launching a scraper for a client.
Three mistakes that get your scraper blocked within an hour
First - request frequency too high. A scraper hits a site 50 times per second, the server sees an attack and blocks the IP. Normal speed is 1-3 requests per second with pauses. If you need faster, use proxy pools and distributed load.
Second mistake - ignoring robots.txt. This file tells you which sections of a site can be scraped and which can't. If there's a ban on the section you need, you either don't run the scraper or prepare for blocking and figure out how much the site owner cares about enforcing it.
Third - scraping from one IP without rotation. The site sees thousands of requests from one address and cuts access. Solution - mobile proxies or residential IP addresses. This adds a monthly proxy fee to the scraper development cost, usually starting from a couple thousand rubles depending on volume.
When you don't need a scraper, you need an API
Many platforms provide APIs - an official way to get data programmatically. Marketplaces, social networks, messaging platforms - most have APIs for integrations.
APIs are faster, more stable, and don't violate platform rules. A scraper breaks with every site redesign, an API stays the same. If the task is integrating with a marketplace or social network, a scraper is a poor choice.
But APIs are often limited. An e-commerce platform might give you access only to your own products, but won't show competitor prices. Some platforms have paid APIs that don't cover all categories. In those cases, a scraper is the only way to get the data you need. We always check at the briefing stage whether a platform has an API before suggesting scraping.
How much scraper development costs
A simple scraper for one site with predictable structure - from 15-20 thousand rubles. This is a script that pulls data from several pages and puts it in Excel or Google Sheets. Good for one-time collection or weekly price monitoring.
A scraper with authorization, captcha bypass, proxy rotation, and database export - from 40-60 thousand. You need this if the site is protected, data is behind registration, or the volume is large.
Scraper maintenance is a separate expense. Sites change layout, add protection, redesigns break selectors. If the scraper is critical for business, budget 5-10 thousand rubles per month for updates. Plus proxy rental if needed.
A scraper pays off when it saves more time than it costs. If the task comes up once a quarter, it's easier to pay a freelancer for manual collection. If it's weekly routine taking several hours, the scraper pays for itself in a month.
How to accept a scraper and not end up with a broken script
First - check the result on real data, not test data. Ask to run the scraper on current site pages and manually verify several records. If prices, names, or contacts don't match, the scraper is configured wrong.
Second - make sure the scraper handles errors. The site might be unavailable, the page changed, the captcha not recognized. A good scraper logs errors and continues working, rather than crashing on the first problem.
Third - get launch instructions and code access. The scraper should be yours, not a black box with the developer. If the developer disappears, you can hand the code to another specialist for updates.
Fourth - agree on a warranty period. Usually a month after delivery, during which the developer fixes bugs for free and adapts the scraper to minor site changes.
Alternatives to scraping when it doesn't fit
If scraping is too expensive or risky, there are other ways to get data. First - buy a ready-made database from an aggregator. There are services that collect market data and sell subscription access. More expensive than a scraper, but without the headache of blocks.
Second - use monitoring services. For tracking competitor prices, there are specialized platforms that already scrape popular sites and provide ready reports. Works if your niche is covered.
Third - hire a VA (virtual assistant) for manual collection. If there's not much data and the task is irregular, a person will handle it faster and cheaper than developing and maintaining a scraper.
Scraping isn't a universal solution. Sometimes it's easier to negotiate with a platform owner for data access or find a source that already aggregates it.
How we work with scraping
At EFIMOV DEV we write scrapers in Node.js, usually as part of a broader task - business process automation or CRM integration. Before development, we check whether the platform has an API, assess blocking risk, and discuss the legality of the scenario with the client.
Cost depends on site complexity, data volume, and run frequency. We prepare an exact estimate after briefing, with starting prices for automation listed on the site. We can integrate the scraper with a Telegram bot, Google Sheets, your CRM, or database - whatever works for your workflow.
If you're thinking about automating data collection but aren't sure whether scraping fits, reach out. We'll review your task and suggest a solution that won't create unnecessary risks.
Scraping is a working tool for collecting public data if you use it sensibly. Check scenario legality, don't ignore robots.txt, budget for maintenance. A scraper pays off when it saves regular routine work, not when solving a one-time task. If you're unsure whether scraping fits your case, ask before starting development - it will save money and nerves.
Frequently asked
Can you scrape competitor prices?
Yes, if the prices are public and accessible without login. This doesn't break the law, but may contradict the site's terms of service. In practice, complaints are rare if scraping doesn't create server load.
Is it legal to scrape company contacts for cold outreach?
Yes, if the company published the contacts publicly for business communication. But outreach must follow anti-spam requirements: recipients must be able to unsubscribe, and you can't send to those who already opted out.
What to do if the scraper breaks after a site redesign?
The scraper needs updating - fix the selectors it uses to find data on the page. This is standard work, takes from an hour to several hours depending on the scale of changes. If the scraper is critical, budget for monthly maintenance.
Can you scrape data from a private account on someone else's site?
Technically yes, if you have legal access to that account (for example, you're a customer of the service). But this almost always violates the terms of service and can lead to account blocking. It's safer to use an API if available.
How long does scraper development take?
A simple scraper - 3 to 7 days. A complex one with authorization, captcha, proxies, and integrations - from two weeks. Exact timeline depends on site protection and data volume.