Hacker News

Subscribe to Hacker News feed
Hacker News RSS
Updated: 58 sec ago

50 years of text games – 1977: Zork

Wed, 04/02/2025 - 1:42pm
Categories: Hacker News

The MNT Difference

Wed, 04/02/2025 - 1:41pm

Article URL: https://mntre.com/about.html

Comments URL: https://news.ycombinator.com/item?id=43559215

Points: 1

# Comments: 0

Categories: Hacker News

Sonic Hedgehog Protein

Wed, 04/02/2025 - 1:22pm
Categories: Hacker News

Strategies to download data constantly changing via API

Wed, 04/02/2025 - 1:21pm

I have to download a dataset through one API (WFS provided by geoserver) that tells me the total amount of items and delivers at maximum 1000 items per request and I can sort by one field and offset the requests start index. The layer has ~1Million items. I can use at maximum 5 parallel request before API gets overloaded.

Problem is that items are being added and removed in real time, so at the end of the copy process I already have stale data copied and there are new items to be copied over. So what would you do, or have done in this situation? Start a never ending loop to crawl data all day long would be something evil or is it something to be fixed on provider side?

The api url is https://geoserver.car.gov.br/geoserver/sicar/wfs

Source data website: https://consultapublica.car.gov.br/publico/imoveis/index

Comments URL: https://news.ycombinator.com/item?id=43559034

Points: 2

# Comments: 0

Categories: Hacker News

Pages